diff --git a/attestor/README.md b/attestor/README.md index e82df80f3..c58203e18 100644 --- a/attestor/README.md +++ b/attestor/README.md @@ -35,14 +35,14 @@ point. | Variable | Meaning | |---|---| | `PENDULUM_WS` | WebSocket of your own Pendulum node, e.g. `ws://127.0.0.1:9944` | -| `PENDULUM_GENESIS_HASH` | Genesis hash that node must serve (default Pendulum's, `0x5d3c2986…462c86`); set it only for a local test chain | +| `PENDULUM_GENESIS_HASH` | Genesis hash that node must serve (default Pendulum's, `0x5d3c2986…462c86`); set it only for a local test chain. With `BASE_CHAIN_ID` 8453 (Base mainnet) any other value is refused at startup | | `BASE_RPC_URL` | Base JSON-RPC endpoint | | `VAULT_ADDRESS` | MigrationVault address on Base | | `ATTESTOR_PRIVATE_KEY` | This attestor's signing key (0x-prefixed) | | `CHECKPOINT_FILE` | Path persisting the last processed block (default `./checkpoint.json`) | | `START_BLOCK` | First Pendulum block to scan on the very first run | | `MIN_GAS_BALANCE_WEI` | Low-gas alert threshold (default 0.01 ETH) | -| `ALERT_WEBHOOK_URL` | Optional webhook receiving JSON alerts in Slack incoming-webhook form: `text` holds `[CRITICAL]` or `[WARNING]`, the attestor and the subject, then the detail; `severity`, `subject` and `detail` are also sent as fields. A non-2xx answer, a timeout (10 s) or a connection error is logged as a failed delivery | +| `ALERT_WEBHOOK_URL` | Optional webhook receiving JSON alerts in Slack incoming-webhook form: `text` holds `[CRITICAL]` or `[WARNING]`, the attestor and the subject, then the detail; `severity`, `subject` and `detail` are also sent as fields. A non-2xx answer, a timeout (10 s) or a connection error is logged as a failed delivery and tried again after 2 s and after 10 s more; a 4xx other than 408 and 429 is not retried, as repeating the request cannot change it | | `HEARTBEAT_URL` | Optional http(s) URL requested (GET) after a finalized head processed without error, at most once a minute; point it at a dead-man's switch that pages when the requests stop. Startup refuses any other value. A failed request is logged (error name and code only, never the URL), never fatal | | `BASE_CHAIN_ID` | Default 8453 (Base mainnet) | | `BASE_FINALITY_TAG` | Base confirmation boundary the checkpoint waits for: `safe` (default) or `finalized` | @@ -117,10 +117,12 @@ WantedBy=multi-user.target approval is recorded, a transaction still pending is no longer needed, but it is followed and bumped the same way until its nonce is used: every later approval waits behind that nonce, and mined it merely reverts. When no bump - is left, or the cap stops one, it pages `own approval not mined` once (with - the nonce, the account's next mined nonce, the fees and whether the migration - is already resolved) and keeps waiting, since later approvals would queue - behind that nonce anyway. Whenever it stops bumping, it re-sends the last + is left, or the cap stops one, it pages `own approval not mined` (with the + nonce, the account's next mined nonce, the fees and whether the migration is + already resolved) and keeps waiting, since later approvals would queue + behind that nonce anyway. While it stays stuck, the page repeats 10, 20 and + 40 minutes later and so on, at most every 4 hours, marked `still not mined + (page N)`. Whenever it stops bumping, it re-sends the last attempt unchanged, so a node that dropped the transaction while still counting its nonce gets it back instead of leaving a gap. An approval that an RPC error interrupted keeps its transaction, also one whose send got no @@ -129,8 +131,15 @@ WantedBy=multi-user.target counting `FEE_BUMP_AFTER_SECONDS` from its last send. A first attempt whose nonce is taken (mined already, or held by a transaction this process did not send) moves on to the next nonce at once, at most five - in a row, without spending a bump. A node refusing a send for its price, an - underpriced replacement or a full pool is retried at the next bump instead + in a row, without spending a bump. An approval that cannot be mined because + the node holds nothing at a nonce below it (a transaction of ours dropped by + a reorg, or a skipped one that left the pool) is sent again at that first + empty nonce, or the next approval is, so nothing waits on a nonce no + transaction will fill. It moves there, keeping its bumps and pages, only + once the node takes it there: a count from a lagging backend can show a + nonce empty that a transaction does hold, and a refused approval stays at + its nonce, bumped and paged as before. A node refusing a send for its + price, an underpriced replacement or a full pool is retried at the next bump instead of being fatal. Transactions in flight are kept in memory only: after a restart, an approval whose transaction is still pending is sent again at the next nonce, so a pending transaction stuck below the cap has to be replaced @@ -146,13 +155,17 @@ WantedBy=multi-user.target with no checkpoint and `START_BLOCK` before the oldest unreleased migration; a re-added key lowers `lastProcessedBlock` to such a block and sets `attestorGeneration` to the current value, keeping every other field. - Checkpoints written before these fields existed are bound to the first key - that starts on them. + A checkpoint without these fields, or with only one of them, was edited or + not written by this daemon and is refused; the error says how to recover. - The daemon verifies at startup that its address is in the vault's attestor set, that `BASE_RPC_URL` serves `BASE_CHAIN_ID` (`eth_chainId`, which the checkpoint's Base chain must match too) and that `PENDULUM_WS` serves `PENDULUM_GENESIS_HASH`, and refuses to run otherwise. These checks run on - every start, the first one without a checkpoint included. + every start, the first one without a checkpoint included. While running, it + re-reads its attestor membership and generation on every finalized head and + exits with `fatal error, exiting` when either changed (a key removed, or + removed and re-added): the next start then refuses the checkpoint until it + is rewound as described above. - After a Pendulum **runtime upgrade**, verify event decoding against the new metadata on a staging node before letting the fleet advance past the upgrade block. diff --git a/attestor/src/alert.test.ts b/attestor/src/alert.test.ts index 6da992645..8df13970f 100644 --- a/attestor/src/alert.test.ts +++ b/attestor/src/alert.test.ts @@ -42,24 +42,50 @@ test("a webhook error status, a timeout or a dead relay is a failed delivery, an for (const status of [500, 429, 404]) { const { url, server } = await webhook(status); try { - assert.equal(await postAlert(url, { subject: "s" }), false, `HTTP ${status}`); + assert.equal(await postAlert(url, { subject: "s" }, 10_000, []), false, `HTTP ${status}`); } finally { server.close(); } } const hanging = await webhook(null); try { - assert.equal(await postAlert(hanging.url, { subject: "s" }, 200), false, "timeout"); + assert.equal(await postAlert(hanging.url, { subject: "s" }, 200, []), false, "timeout"); } finally { hanging.server.closeAllConnections(); hanging.server.close(); } const closed = await webhook(200); closed.server.close(); - assert.equal(await postAlert(closed.url, { subject: "s" }), false, "connection refused"); + assert.equal(await postAlert(closed.url, { subject: "s" }, 10_000, []), false, "connection refused"); assert.equal(logged.mock.callCount(), 5); }); +test("a failed delivery is tried again after each wait, a request the relay refused is not", async (t) => { + t.mock.method(console, "error", () => {}); + let answers: number[] = []; + let posts = 0; + const server = createServer((request, response) => { + request.resume(); + request.on("end", () => response.writeHead(answers[posts++]).end()); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; + try { + for (const [statuses, delivered] of [ + [[503, 429, 200], true], + [[500, 502, 504], false], + [[404], false], + ] as const) { + answers = [...statuses]; + posts = 0; + assert.equal(await postAlert(url, { subject: "s" }, 10_000, [10, 10]), delivered, `${statuses}`); + assert.equal(posts, statuses.length, `${statuses}`); + } + } finally { + server.close(); + } +}); + test("a heartbeat requests its URL, and a failing endpoint is logged without the URL, never thrown", async (t) => { const logged = t.mock.method(console, "error", () => {}); const seen: string[] = []; diff --git a/attestor/src/alert.ts b/attestor/src/alert.ts index c5af62001..376769f49 100644 --- a/attestor/src/alert.ts +++ b/attestor/src/alert.ts @@ -66,24 +66,42 @@ export function throttledHeartbeat( }; } +/** A failed alert delivery is tried again after each of these waits: a relay + * that is briefly down or rate-limiting must not swallow a page. */ +const ALERT_RETRY_DELAYS_MS = [2_000, 10_000]; + /** * POST one alert to the webhook and report whether it was delivered. `fetch` * resolves on any HTTP answer, so a non-2xx status is checked explicitly: a * relay answering 4xx/5xx, a timeout and a connection failure are all logged - * and count as not delivered. + * and count as not delivered. Each is tried again after the next of + * `retryDelaysMs`, except a 4xx other than 408 and 429: the relay refused the + * request itself (a wrong URL, a rejected payload), which repeating it cannot + * change. */ -export async function postAlert(url: string, body: unknown, timeoutMs = 10_000): Promise { - try { - const response = await fetch(url, { - method: "POST", - headers: { "content-type": "application/json" }, - body: JSON.stringify(body), - signal: AbortSignal.timeout(timeoutMs), - }); - if (response.ok) return true; - console.error(`alert webhook failed: HTTP ${response.status}`); - } catch (webhookError) { - console.error("alert webhook failed", webhookError); +export async function postAlert( + url: string, + body: unknown, + timeoutMs = 10_000, + retryDelaysMs = ALERT_RETRY_DELAYS_MS, +): Promise { + for (let attempt = 0; ; attempt++) { + let status: number | undefined; + try { + const response = await fetch(url, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify(body), + signal: AbortSignal.timeout(timeoutMs), + }); + if (response.ok) return true; + status = response.status; + console.error(`alert webhook failed: HTTP ${response.status}`); + } catch (webhookError) { + console.error("alert webhook failed", webhookError); + } + const refused = status !== undefined && status < 500 && status !== 408 && status !== 429; + if (refused || attempt === retryDelaysMs.length) return false; + await new Promise((resolve) => setTimeout(resolve, retryDelaysMs[attempt])); } - return false; } diff --git a/attestor/src/approval.test.ts b/attestor/src/approval.test.ts index bc137daee..f2b7c6377 100644 --- a/attestor/src/approval.test.ts +++ b/attestor/src/approval.test.ts @@ -28,6 +28,7 @@ import { privateKeyToAccount } from "viem/accounts"; import { advanceCheckpoint, approve, + assertSignerUnchanged, bumpedFees, cappedFees, GAS_LIMIT_MULTIPLIER, @@ -39,7 +40,7 @@ import { type UnconfirmedBlock, type VaultChain, } from "./approval.js"; -import type { MigrationEvent } from "./checks.js"; +import { isTransientRpcError, type MigrationEvent } from "./checks.js"; import { vaultAbi } from "./vaultAbi.js"; const VAULT = "0x1111111111111111111111111111111111111111"; @@ -177,6 +178,14 @@ class FakeVault implements VaultChain { async boundaryBlock() { return this.boundary; } + member = true; + generation = 1; + async isAttestor() { + return this.member; + } + async attestorGeneration() { + return this.generation; + } /** Mine every pooled transaction that is next in line and pays the base fee. * An approval records ours unless the migration is already resolved. */ mine() { @@ -296,6 +305,14 @@ test("the viem chain reads the boundary at the finality tag and everything else const [simulated] = pick("simulateContract"); assert.deepEqual([simulated.account, simulated.functionName, simulated.args], [account, "approve", [EVENT.nonce, EVENT.recipient, EVENT.palletAmount]]); assert.deepEqual(pick("getTransactionCount"), [{ address: account.address, blockTag: "pending" }]); + answers.readContract = (args) => (args.functionName === "attestorGeneration" ? 2n : true); + assert.equal(await chain.isAttestor(), true); + assert.equal(await chain.attestorGeneration(), 2); + const [, , member, generation] = pick("readContract"); + assert.deepEqual( + [member.functionName, member.args, generation.functionName, generation.args, generation.address], + ["isAttestor", [account.address], "attestorGeneration", [account.address], VAULT], + ); answers.getBlock = () => ({ number: null }); await assert.rejects(chain.boundaryBlock(), /finalized Base block has no number/); @@ -663,6 +680,35 @@ test("after the last bump it pages once and keeps waiting until the approval is assert.equal(attestor.missedApprovals, 0); }); +test("an approval that stays stuck after its page pages again, at growing intervals", async () => { + const chain = new FakeVault(); + chain.baseFee = 100_000_000n; + const pages: { at: number; detail: string }[] = []; + const { attestor } = attestorOn( + chain, + { + alert: async (subject, detail) => { + assert.equal(subject, "own approval not mined"); + pages.push({ at: attestor.now(), detail: `${detail}` }); + }, + }, + (elapsed) => { + // Fees fall again two hours in. + if (elapsed >= 120 * 60_000) chain.baseFee = 5_000_000n; + chain.mine(); + }, + ); + assert.equal(await approve(attestor, EVENT), true); + // First paged 4 min after the send, then 10, 20 and 40 min apart. + assert.deepEqual( + pages.slice(1).map((page, i) => (page.at - pages[i].at) / 60_000), + [10, 20, 40], + ); + assert.match(pages[0].detail, /^3 fee bump\(s\) used up: transaction nonce 5, /); + assert.match(pages[1].detail, /^still not mined \(page 2\), 3 fee bump\(s\) used up: transaction nonce 5, /); + assert.ok(chain.approved.has(payloadHash(EVENT))); +}); + test("the fee cap stops a bump and pages once; no attempt offers more than the cap", async () => { const chain = new FakeVault(); chain.baseFee = 100_000_000n; @@ -1119,6 +1165,141 @@ test("the new transaction after another took the nonce starts from the current e assert.ok(chain.approved.has(payloadHash(EVENT))); }); +test("an own approval reorged out of Base leaves no empty nonce the next approval waits behind", async () => { + const chain = new FakeVault(); + const { attestor, alerts, logs } = attestorOn(chain); + assert.equal(await approve(attestor, EVENT), true); + // The block holding it is reorged away, and the transaction is not re-injected. + chain.minedNonce = 5; + chain.approved.delete(payloadHash(EVENT)); + chain.receipts.clear(); + assert.equal(await approve(attestor, OTHER), true); + assert.deepEqual( + chain.sent.map((tx) => [approveArgs(tx)[0], tx.nonce]), + [ + [EVENT.nonce, 5], + [OTHER.nonce, 6], + [OTHER.nonce, 5], + ], + ); + assert.ok(chain.approved.has(payloadHash(OTHER))); + assert.equal(chain.receipts.get(chain.sent[1].hash)?.status, "reverted", "the attempt left behind merely reverts"); + assert.ok(logs.some((line) => line.startsWith("nonce 5 below 6 holds no transaction; sent the approval again there: nonce=8 "))); + assert.ok(logs.some((line) => line.startsWith("approved: nonce=8 ") && line.endsWith(`tx=${chain.sent[2].hash}`))); + assert.equal(alerts.length, 0); +}); + +test("a nonce skipped for a transaction this process did not send is filled once that transaction leaves the pool", async () => { + const chain = new FakeVault(); + chain.pool.set(5, { hash: `0x${"ee".repeat(32)}`, nonce: 5, value: 0n, maxFeePerGas: 50_000_000n, maxPriorityFeePerGas: 2_000_000n }); + chain.pendingNonce = 5; // the backend answering the count has not seen it + chain.onSend = () => { + if (chain.sent.length === 2) { + // Evicted before it was mined. + chain.pool.delete(5); + chain.pendingNonce = undefined; + } + }; + const { attestor } = attestorOn(chain); + assert.equal(await approve(attestor, EVENT), true); + assert.deepEqual( + chain.sent.map((tx) => tx.nonce), + [5, 6, 5], + ); + assert.ok(chain.receipts.has(chain.sent[2].hash)); + assert.ok(chain.approved.has(payloadHash(EVENT))); +}); + +test("an approval no longer needed behind an empty nonce is not sent again, and the next approval fills that nonce", async () => { + const chain = new FakeVault(); + chain.pool.set(5, { hash: `0x${"ee".repeat(32)}`, nonce: 5, value: 0n, maxFeePerGas: 50_000_000n, maxPriorityFeePerGas: 2_000_000n }); + chain.pendingNonce = 5; + chain.onSend = () => { + if (chain.sent.length === 2) { + chain.pool.delete(5); + chain.pendingNonce = undefined; + chain.consumed.add(EVENT.nonce); // released by the others meanwhile + } + }; + const { attestor } = attestorOn(chain); + assert.equal(await approve(attestor, EVENT), true); + assert.equal(attestor.missedApprovals, 1); + assert.equal(await approve(attestor, OTHER), true); + assert.deepEqual( + chain.sent.map((tx) => [approveArgs(tx)[0], tx.nonce]), + [ + [EVENT.nonce, 5], + [EVENT.nonce, 6], + [OTHER.nonce, 5], + ], + ); + assert.ok(chain.approved.has(payloadHash(OTHER))); +}); + +test("a nonce below that the pending count shows empty but a stuck transaction holds leaves the approval to be bumped and paged", async () => { + const chain = new FakeVault(); + // A transaction from before a restart, stuck below the base fee with a higher tip than the estimate. + const stuck: Sent = { hash: `0x${"ee".repeat(32)}`, nonce: 5, value: 0n, maxFeePerGas: 50_000_000n, maxPriorityFeePerGas: 2_000_000n }; + chain.pool.set(5, stuck); + chain.baseFee = 100_000_000n; + chain.pendingNonce = 5; // the backend answering the count does not hold it + const { attestor, alerts, logs } = attestorOn(chain, {}, (elapsed) => { + if (elapsed >= 6 * 3600_000) chain.baseFee = 5_000_000n; // fees fall after 6 h + chain.mine(); + }); + assert.equal(await approve(attestor, EVENT), true); + assert.ok(chain.sent.every((tx) => tx.nonce === 5 || tx.nonce === 6)); + assert.ok(!logs.some((line) => line.includes("holds no transaction")), "never moved down to nonce 5"); + assert.equal(logs.filter((line) => /^fee bump \d at nonce 6: /.test(line)).length, 3); + assert.ok(alerts.length > 1 && alerts.every((a) => a.subject === "own approval not mined")); + assert.match(alerts[0].detail, /^3 fee bump\(s\) used up: transaction nonce 6, account's next mined nonce 5, /); + assert.ok(chain.receipts.has(stuck.hash), "the stuck transaction kept its nonce"); + assert.ok(chain.approved.has(payloadHash(EVENT))); +}); + +test("a latest count that lags behind keeps the approval at its nonce and its earlier attempts tracked", async () => { + const chain = new FakeVault(); + chain.baseFee = 9_000_000n; // above the first attempt and the first bump + // No room in the pool for a second bump. + chain.refuse = (_, tx) => (tx.maxFeePerGas > 8_400_000n ? "txpool is full" : undefined); + let lagging = false; + const accountNonce = chain.accountNonce.bind(chain); + // A backend a block behind: nonce 4 not yet mined there, and nothing of ours held. + chain.accountNonce = async (blockTag) => (lagging ? 4 : accountNonce(blockTag)); + const { attestor, logs } = attestorOn(chain, {}, (elapsed) => { + lagging = elapsed > 1.5 * AFTER_MS && elapsed <= 2 * AFTER_MS; + if (elapsed > 2 * AFTER_MS) chain.baseFee = 5_000_000n; + chain.mine(); + }); + assert.equal(await approve(attestor, EVENT), true); + assert.deepEqual( + chain.sent.map((tx) => [tx.nonce, tx.maxFeePerGas]), + [ + [5, 7_000_000n], + [5, 8_400_000n], + [4, 8_400_000n], + [5, 10_080_000n], + ], + ); + assert.ok(logs.some((line) => line.startsWith("approved: nonce=7 ") && line.endsWith(`tx=${chain.sent[1].hash}`))); +}); + +test("a signer removed, or removed and re-added, while running stops the daemon instead of going on", async () => { + const chain = new FakeVault(); + const { attestor } = attestorOn(chain); + await assertSignerUnchanged(attestor, 1); + // Removed and re-added between two heads: its generation-1 approvals no longer count. + chain.generation = 2; + await assert.rejects(assertSignerUnchanged(attestor, 1), (error: Error) => { + assert.match(error.message, /went from attestor generation 1 to 2 while running, .*\(runbook RB-6\)/); + assert.equal(isTransientRpcError(error), false, "fatal, not retried on the next head"); + return true; + }); + chain.generation = 1; + chain.member = false; + await assert.rejects(assertSignerUnchanged(attestor, 1), /is no longer an attestor of 0x1111/); +}); + test("unreleasable events are not tracked for durability; every other event is", async () => { const chain = new FakeVault(); const { attestor } = attestorOn(chain); diff --git a/attestor/src/approval.ts b/attestor/src/approval.ts index 8cef92230..eb607060f 100644 --- a/attestor/src/approval.ts +++ b/attestor/src/approval.ts @@ -42,6 +42,12 @@ export const MIN_REPLACEMENT_PERCENT = 10; /** How often a sent approval's receipt is polled: about one Base block. */ const RECEIPT_POLL_MS = 2000; +/** An own approval still stuck after its page pages again this much later, + * then at twice the previous interval up to `REPAGE_MAX_MS`: a single page + * whose delivery failed must not leave a stuck attestor silent. */ +const REPAGE_FIRST_MS = 10 * 60_000; +const REPAGE_MAX_MS = 4 * 3600_000; + /** How many nonces in a row a first attempt tries at once when each is taken: * enough for a backend whose pending count trails by a few blocks, and a * bound for one that refuses every nonce. */ @@ -76,14 +82,16 @@ export interface Submission { nonce?: number; /** The latest attempt's fees. */ fees?: Fees; - /** Every attempt at `nonce`; any of them may be the one that is mined. */ + /** Every attempt at `nonce`, and those left above it when the approval + * moved down to an empty nonce; any of them may be the one that is mined. */ hashes: Hash[]; /** When the latest attempt was sent: a follow-up an error interrupted * resumes the bump interval from there instead of starting it anew. */ sentAtMs?: number; bumps: number; - /** The page that it is not being mined went out. */ - paged: boolean; + /** How often the page that it is not being mined went out, and when last. */ + pages: number; + pagedAtMs?: number; } /** A block processed against the latest Base state whose releasable events are @@ -116,6 +124,10 @@ export interface VaultChain { receipt(hash: Hash): Promise; /** The newest block inside the Base finality boundary. */ boundaryBlock(): Promise; + /** Whether the signer is in the vault's attestor set, and its attestor + * generation there (bumped by every `addAttestor`). */ + isAttestor(): Promise; + attestorGeneration(): Promise; } /** `VaultChain` on a viem client, reading and sending as `account`. */ @@ -170,6 +182,17 @@ export function viemVaultChain( if (confirmed.number === null) throw new Error(`${finalityTag} Base block has no number`); return confirmed.number; }, + isAttestor: () => + client.readContract({ address: vaultAddress, abi: vaultAbi, functionName: "isAttestor", args: [account.address] }), + attestorGeneration: async () => + Number( + await client.readContract({ + address: vaultAddress, + abi: vaultAbi, + functionName: "attestorGeneration", + args: [account.address], + }), + ), }; } @@ -197,7 +220,9 @@ export interface Attestor { /** The lowest nonce no transaction of this process has been accepted at (or * sent at without an answer). The node's pending count can trail it (a * load-balanced endpoint, a pool that dropped the transaction), and a nonce - * one payload was sent at is never signed for another. */ + * one payload was sent at is never signed for another, unless an empty + * nonce below kept that attempt from ever being mined and the payload was + * sent again there (see `sendUntilMined`). */ nextNonce: number; } @@ -276,11 +301,13 @@ export function bumpedFees(last: Fees, estimate: Fees, policy: FeeBumpPolicy): F return replaces ? fees : undefined; } -/** Poll the receipts of `hashes` until one is mined or `timeoutMs` has passed. */ +/** Poll the receipts of `hashes` until one is mined or `timeoutMs` has passed. + * Newest first: an attempt moved down to an empty nonce is mined before the + * ones it left above, which then merely revert. */ async function minedReceipt(attestor: Attestor, hashes: Hash[], timeoutMs: number): Promise { const deadline = attestor.now() + timeoutMs; for (;;) { - for (const hash of hashes) { + for (const hash of [...hashes].reverse()) { const receipt = await attestor.chain.receipt(hash); if (receipt) return receipt; } @@ -367,9 +394,9 @@ async function resendLatest(attestor: Attestor, submission: Submission, nonce: n * An attempt still not mined `feeBump.afterMs` after it was sent, also across * a retry after an error, is re-sent: the identical call at the same nonce with * both fees raised (`bumpedFees`), at most `feeBump.max` times. When no bump is - * left or the cap stops one, it pages once - * and keeps waiting, re-sending the latest attempt unchanged: later approvals - * would queue behind this nonce anyway. An approval no longer needed (released, + * left or the cap stops one, it pages (again at growing intervals while it + * stays stuck) and keeps waiting, re-sending the latest attempt unchanged: + * later approvals would queue behind this nonce anyway. An approval no longer needed (released, * or ours recorded) is followed the same way while its nonce is pending, as * every later approval waits behind that nonce; mined, it merely reverts. * Returns undefined when the approval is no longer needed and no attempt holds @@ -399,22 +426,49 @@ async function sendUntilMined( if (late) return late; if (!needed) return undefined; attestor.log(`nonce ${nonce} was used by another transaction; sending the approval again: ${label}`); - Object.assign(submission, { nonce: undefined, fees: undefined, hashes: [], bumps: 0, paged: false }); + Object.assign(submission, { nonce: undefined, fees: undefined, hashes: [], bumps: 0, pages: 0 }); continue; } + if (nonce !== undefined && minedNonce !== undefined && minedNonce < nonce && (await attestor.chain.accountNonce("pending")) <= minedNonce) { + // The node holds nothing below our nonce, so ours can never be mined: + // a transaction of ours there was dropped (its block reorged away), or + // one this process skipped left the pool. Fill that nonce instead of + // waiting on it, for this approval or else the next one; the attempt + // left behind merely reverts should it ever be mined. + if (!needed) { + attestor.nextNonce = minedNonce; + return undefined; + } + // Counts from a backend that lags can hide a transaction that does + // hold that nonce, so the latest attempt moves down only once the node + // takes it there, with its bumps and pages; refused, it stays where it + // is and is bumped and paged as before. + const { hash, error } = await sendAttempt(attestor, submission, minedNonce, submission.fees as Fees); + if (error === undefined) { + attestor.log(`nonce ${minedNonce} below ${nonce} holds no transaction; sent the approval again there: ${label} tx=${hash}`); + submission.nonce = minedNonce; + submission.hashes.push(hash); + attestor.nextNonce = minedNonce + 1; + continue; + } + attestor.log(`approval not accepted at nonce ${minedNonce}, so it stays at nonce ${nonce}: ${label}`, error); + } const last = submission.fees as Fees; const fees = - submission.paged || submission.bumps >= policy.max + submission.pages > 0 || submission.bumps >= policy.max ? undefined : bumpedFees(last, await attestor.chain.fees(), policy); if (fees === undefined) { - if (!submission.paged) { - submission.paged = true; + const repageMs = Math.min(REPAGE_MAX_MS, REPAGE_FIRST_MS * 2 ** (submission.pages - 1)); + if (submission.pages === 0 || attestor.now() - (submission.pagedAtMs as number) >= repageMs) { + submission.pages += 1; + submission.pagedAtMs = attestor.now(); const reason = submission.bumps >= policy.max ? `${submission.bumps} fee bump(s) used up` : "the fee cap stops the next bump"; await attestor.alert( "own approval not mined", - `${reason}: transaction nonce ${nonce ?? "not accepted by the node"}, account's next mined nonce ${minedNonce ?? "unknown"}, ` + + `${submission.pages > 1 ? `still not mined (page ${submission.pages}), ` : ""}` + + `${reason}: transaction nonce ${nonce ?? "not accepted by the node"}, account's next mined nonce ${minedNonce ?? "unknown"}, ` + `maxFeePerGas ${formatGwei(last.maxFeePerGas)} gwei (cap ${formatGwei(policy.maxFeePerGas)} gwei)` + `${needed ? "" : ", migration already resolved"}; ${label}`, "critical", @@ -480,7 +534,7 @@ export async function approve(attestor: Attestor, event: MigrationEvent): Promis functionName: "approve", args: [event.nonce, event.recipient, event.palletAmount], }); - submission = { event, data, gas, hashes: [], bumps: 0, paged: false }; + submission = { event, data, gas, hashes: [], bumps: 0, pages: 0 }; attestor.submissions.set(key, submission); } const receipt = await sendUntilMined(attestor, event, submission, label); @@ -536,6 +590,25 @@ export async function approve(attestor: Attestor, event: MigrationEvent): Promis return true; } +/** Refuse to go on once the signer's standing in the vault changed since + * startup, when it was at `generation`: its key removed, or removed and + * re-added, while the daemon ran. Its approvals from before no longer count, + * so progress made under them has to be rewound first, which the checkpoint + * check at the next start asks for. */ +export async function assertSignerUnchanged(attestor: Attestor, generation: number): Promise { + const signer = attestor.account.address; + if (!(await attestor.chain.isAttestor())) { + throw new Error(`${signer} is no longer an attestor of ${attestor.vaultAddress}; stopping`); + } + const current = await attestor.chain.attestorGeneration(); + if (current !== generation) { + throw new Error( + `${signer} went from attestor generation ${generation} to ${current} while running, so its earlier approvals ` + + "no longer count; stopping: rewind the checkpoint as the next start asks (runbook RB-6)", + ); + } +} + /** Approve every event of one finalized Pendulum block and queue the block for * the durability check. */ export async function processBlock( diff --git a/attestor/src/config.test.ts b/attestor/src/config.test.ts index f1947499f..d1c6f0e21 100644 --- a/attestor/src/config.test.ts +++ b/attestor/src/config.test.ts @@ -36,14 +36,25 @@ test("the expected Pendulum genesis defaults to Pendulum's and must be a block h "0x5d3c298622d5634ed019bf61ea4b71655030015bde9beb0d6a24743714462c86", ); const local = `0x${"ab".repeat(32)}`; - assert.equal(loadConfig({ PENDULUM_GENESIS_HASH: local }, "config.pendulumGenesisHash").stdout, local); + assert.equal(loadConfig({ BASE_CHAIN_ID: "84532", PENDULUM_GENESIS_HASH: local }, "config.pendulumGenesisHash").stdout, local); for (const value of ["0x1234", "pendulum", `${"ab".repeat(32)}`]) { - const { status, stderr } = loadConfig({ PENDULUM_GENESIS_HASH: value }); + const { status, stderr } = loadConfig({ BASE_CHAIN_ID: "84532", PENDULUM_GENESIS_HASH: value }); assert.notEqual(status, 0, value); assert.match(stderr, /invalid PENDULUM_GENESIS_HASH/, value); } }); +test("on Base mainnet the expected genesis can only be Pendulum's", () => { + const local = `0x${"ab".repeat(32)}`; + for (const chain of [{}, { BASE_CHAIN_ID: "8453" }] as Record[]) { + const { status, stderr } = loadConfig({ ...chain, PENDULUM_GENESIS_HASH: local }); + assert.notEqual(status, 0); + assert.match(stderr, /PENDULUM_GENESIS_HASH 0xabab.* is not Pendulum's, which BASE_CHAIN_ID 8453 \(Base mainnet\) requires/); + } + const pendulum = "0x5D3C298622D5634ED019BF61EA4B71655030015BDE9BEB0D6A24743714462C86"; + assert.equal(loadConfig({ PENDULUM_GENESIS_HASH: pendulum }, "config.pendulumGenesisHash").stdout, pendulum); +}); + test("HEARTBEAT_URL must be an http(s) URL, and a bad one is refused without echoing it", () => { assert.equal(loadConfig({ HEARTBEAT_URL: "https://hc-ping.com/secret-token" }, "config.heartbeatUrl").stdout, "https://hc-ping.com/secret-token"); const { status, stderr } = loadConfig({ HEARTBEAT_URL: "hc-ping.com/secret-token" }); diff --git a/attestor/src/config.ts b/attestor/src/config.ts index 820d9ec7d..1b0cb4fe2 100644 --- a/attestor/src/config.ts +++ b/attestor/src/config.ts @@ -38,12 +38,19 @@ function finalityTag(): "safe" | "finalized" { const baseFinalityTag = finalityTag(); +const baseChainId = envNumber("BASE_CHAIN_ID", 8453); + /** Pendulum's genesis block hash. */ const PENDULUM_GENESIS_HASH = "0x5d3c298622d5634ed019bf61ea4b71655030015bde9beb0d6a24743714462c86"; function genesisHash(): `0x${string}` { const value = process.env.PENDULUM_GENESIS_HASH || PENDULUM_GENESIS_HASH; if (!/^0x[0-9a-fA-F]{64}$/.test(value)) throw new Error(`invalid PENDULUM_GENESIS_HASH ${value}`); + // Another genesis is for rehearsals against a test chain; the mainnet vault + // only ever pays for burns on Pendulum itself. + if (baseChainId === 8453 && value.toLowerCase() !== PENDULUM_GENESIS_HASH) { + throw new Error(`PENDULUM_GENESIS_HASH ${value} is not Pendulum's, which BASE_CHAIN_ID 8453 (Base mainnet) requires`); + } return value as `0x${string}`; } @@ -54,7 +61,7 @@ export const config = { * daemon outage longer than the pruning horizon wedges the catch-up. */ pendulumWs: required("PENDULUM_WS"), /** The genesis that node must serve; startup refuses any other chain. A - * local test chain sets its own. */ + * local test chain sets its own, never on Base mainnet. */ pendulumGenesisHash: genesisHash(), /** Base JSON-RPC endpoint. */ baseRpcUrl: required("BASE_RPC_URL"), @@ -77,7 +84,7 @@ export const config = { * requests stop. */ heartbeatUrl: parseHeartbeatUrl(process.env.HEARTBEAT_URL), /** Base chain id: 8453 mainnet. */ - baseChainId: envNumber("BASE_CHAIN_ID", 8453), + baseChainId, /** Confirmation boundary required before the checkpoint may pass a block. */ baseFinalityTag, /** How long a block's approvals may stay outside the finality boundary diff --git a/attestor/src/main.ts b/attestor/src/main.ts index 9baf6d1b5..6f9985ed1 100644 --- a/attestor/src/main.ts +++ b/attestor/src/main.ts @@ -25,7 +25,14 @@ import { ApiPromise, WsProvider } from "@polkadot/api"; import { createPublicClient, defineChain, http } from "viem"; import { privateKeyToAccount } from "viem/accounts"; import { alertText, postAlert, throttledHeartbeat, type Severity } from "./alert.js"; -import { advanceCheckpoint, processBlock, viemVaultChain, type Attestor, type UnconfirmedBlock } from "./approval.js"; +import { + advanceCheckpoint, + assertSignerUnchanged, + processBlock, + viemVaultChain, + type Attestor, + type UnconfirmedBlock, +} from "./approval.js"; import { decodeMigrationEvent, isTransientRpcError, type MigrationEvent } from "./checks.js"; import { config } from "./config.js"; import { @@ -36,7 +43,6 @@ import { saveCheckpoint, type Checkpoint, } from "./state.js"; -import { vaultAbi } from "./vaultAbi.js"; /** Repeated transient failures (a sustained RPC outage) page at most this * often; every occurrence is still logged. */ @@ -121,23 +127,10 @@ async function checkGasBalance(): Promise { async function main(): Promise { const baseChainId = await publicClient.getChainId(); assertBaseChain(baseChainId, config.baseChainId); - const isAttestor = await publicClient.readContract({ - address: config.vaultAddress, - abi: vaultAbi, - functionName: "isAttestor", - args: [account.address], - }); - if (!isAttestor) { + if (!(await attestor.chain.isAttestor())) { throw new Error(`${account.address} is not an attestor of ${config.vaultAddress}`); } - const attestorGeneration = Number( - await publicClient.readContract({ - address: config.vaultAddress, - abi: vaultAbi, - functionName: "attestorGeneration", - args: [account.address], - }), - ); + const attestorGeneration = await attestor.chain.attestorGeneration(); await checkGasBalance(); const api = await ApiPromise.create({ provider: new WsProvider(config.pendulumWs) }); @@ -159,10 +152,6 @@ async function main(): Promise { attestorAddress: account.address, attestorGeneration, }); - // A checkpoint written before the signer was recorded is bound to it from - // the next save on. - checkpoint.attestorAddress ??= account.address; - checkpoint.attestorGeneration ??= attestorGeneration; log(`attestor ${account.address} (generation ${attestorGeneration}) starting after block ${checkpoint.lastProcessedBlock}`); /** Blocks processed at the latest Base state, oldest first, that the @@ -180,6 +169,9 @@ async function main(): Promise { const finalized = head.number.toNumber(); // Serialize: a slow Base transaction must not let block processing overlap. processing = processing.then(async () => { + // Two reads per head: a key removed or re-added while running must + // not go on making progress that its new generation does not back. + await assertSignerUnchanged(attestor, attestorGeneration); for (let block = processedThrough + 1; block <= finalized; block++) { await processBlock(attestor, block, await migrationEventsInBlock(api, block), unconfirmed); processedThrough = block; diff --git a/attestor/src/state.test.ts b/attestor/src/state.test.ts index 6bd0b438d..c65ecb39f 100644 --- a/attestor/src/state.test.ts +++ b/attestor/src/state.test.ts @@ -66,15 +66,16 @@ test("a checkpoint made by another key or an earlier generation of this key is r ); }); -test("a checkpoint from before the signer was recorded still loads", () => { +test("a checkpoint without the signer fields is refused, not bound to the key that starts on it", () => { const directory = mkdtempSync(join(tmpdir(), "pen-attestor-state-")); const path = join(directory, "checkpoint.json"); try { - const { attestorAddress, attestorGeneration, ...legacy } = checkpoint; - writeFileSync(path, JSON.stringify(legacy)); - const loaded = loadCheckpoint(path); - assert.deepEqual(loaded, legacy); - assert.doesNotThrow(() => assertCheckpointIdentity(loaded!, { ...legacy, attestorAddress, attestorGeneration })); + const { attestorAddress, attestorGeneration, ...unbound } = checkpoint; + writeFileSync(path, JSON.stringify(unbound)); + assert.throws( + () => assertCheckpointIdentity(loadCheckpoint(path)!, { ...unbound, attestorAddress, attestorGeneration }), + /has neither attestorAddress nor attestorGeneration, .*\(runbook RB-6\): .*start with no checkpoint/, + ); writeFileSync(path, JSON.stringify({ ...checkpoint, attestorGeneration: 0 })); assert.throws(() => loadCheckpoint(path), /invalid or legacy schema/); writeFileSync(path, JSON.stringify({ ...checkpoint, attestorAddress: [checkpoint.attestorAddress] })); @@ -84,6 +85,30 @@ test("a checkpoint from before the signer was recorded still loads", () => { } }); +test("a checkpoint with only one of the signer fields is refused as edited", () => { + const directory = mkdtempSync(join(tmpdir(), "pen-attestor-state-")); + const path = join(directory, "checkpoint.json"); + try { + const { attestorAddress, attestorGeneration, ...unbound } = checkpoint; + writeFileSync(path, JSON.stringify({ ...unbound, attestorAddress })); + assert.throws( + () => assertCheckpointIdentity(loadCheckpoint(path)!, { ...checkpoint, attestorGeneration: 2 }), + /has attestorAddress but no attestorGeneration; .*\(runbook RB-6\)/, + ); + writeFileSync(path, JSON.stringify({ ...unbound, attestorGeneration })); + assert.throws( + () => + assertCheckpointIdentity(loadCheckpoint(path)!, { + ...checkpoint, + attestorAddress: "0x5555555555555555555555555555555555555555", + }), + /has attestorGeneration but no attestorAddress/, + ); + } finally { + rmSync(directory, { recursive: true, force: true }); + } +}); + test("the Base endpoint must serve the configured chain and the Pendulum node the pinned genesis", () => { assert.doesNotThrow(() => assertBaseChain(8453, 8453)); assert.throws(() => assertBaseChain(84532, 8453), /BASE_RPC_URL serves chain 84532, but BASE_CHAIN_ID is 8453/); diff --git a/attestor/src/state.ts b/attestor/src/state.ts index dfa3e1a0a..2ac89687d 100644 --- a/attestor/src/state.ts +++ b/attestor/src/state.ts @@ -15,8 +15,8 @@ export interface Checkpoint { vaultAddress: `0x${string}`; pendulumGenesisHash: `0x${string}`; /** The signer, and its vault attestor generation, whose approvals this - * progress stands for. Optional only so that checkpoints written before - * they were recorded still load; the daemon binds those on startup. */ + * progress stands for. Optional only so that a file without them loads + * and `assertCheckpointIdentity` refuses it with the way out. */ attestorAddress?: `0x${string}`; attestorGeneration?: number; lastProcessedBlock: number; @@ -136,17 +136,34 @@ export function assertCheckpointIdentity( // Progress made by another key, or by this key before it was removed and // re-added, says nothing about the approvals that count now: resuming from // it would silently skip every unreleased migration before it. - if ( - checkpoint.attestorAddress !== undefined && - checkpoint.attestorAddress.toLowerCase() !== expected.attestorAddress.toLowerCase() - ) { + // Both fields are always written together, so a checkpoint without them, + // or with only one of them, was edited or not written by this daemon. + if (checkpoint.attestorAddress === undefined && checkpoint.attestorGeneration === undefined) { + throw new Error( + "checkpoint has neither attestorAddress nor attestorGeneration, so nothing shows whose approvals its " + + "progress stands for: restore both if they were removed and follow the error that gives, or else " + + "backfill as a new key does (runbook RB-6): move the checkpoint file aside and start with no checkpoint " + + "and START_BLOCK before the oldest unreleased migration", + ); + } + if (checkpoint.attestorAddress === undefined || checkpoint.attestorGeneration === undefined) { + const [present, missing] = + checkpoint.attestorAddress === undefined + ? ["attestorGeneration", "attestorAddress"] + : ["attestorAddress", "attestorGeneration"]; + throw new Error( + `checkpoint has ${present} but no ${missing}; the two are always written together, so the file was edited: ` + + "restore the removed field and follow the error that gives (runbook RB-6), never remove either one", + ); + } + if (checkpoint.attestorAddress.toLowerCase() !== expected.attestorAddress.toLowerCase()) { throw new Error( `checkpoint belongs to attestor ${checkpoint.attestorAddress}, but this daemon signs as ${expected.attestorAddress}; ` + "a new key must backfill (runbook RB-6): move the checkpoint file aside and start with no checkpoint " + "and START_BLOCK before the oldest unreleased migration", ); } - if (checkpoint.attestorGeneration !== undefined && checkpoint.attestorGeneration !== expected.attestorGeneration) { + if (checkpoint.attestorGeneration !== expected.attestorGeneration) { throw new Error( `checkpoint was written under attestor generation ${checkpoint.attestorGeneration}, but ` + `${expected.attestorAddress} is now at generation ${expected.attestorGeneration}, so its earlier approvals ` + diff --git a/releaser/README.md b/releaser/README.md index 0ead43908..f5d060db4 100644 --- a/releaser/README.md +++ b/releaser/README.md @@ -34,15 +34,19 @@ consumed nonce and drops it. | `RELEASER_PRIVATE_KEY` | Gas-only signing key (no privileges) | | `START_BLOCK` | Vault deployment block — where log scanning begins on a first run | | `STATE_FILE` | Scan checkpoint + pending set (default `./releaser-state.json`) | -| `POLL_INTERVAL_MS` | Default 60s | -| `MAX_BLOCK_RANGE` | Max blocks per `eth_getLogs` (default 10,000) | +| `POLL_INTERVAL_MS` | Default 60s; at least 1000 | +| `MAX_BLOCK_RANGE` | Max blocks per `eth_getLogs` (default 10,000; a whole number of at least 1). A range the endpoint refuses as too large is read again at half the size, down to one block, and the size doubles again after 10 full-size reads in a row, never above this value; the current size is in each cycle's `ok:` log line. Set it to the endpoint's cap to avoid the refused reads (Base Sepolia's public endpoint allows 200) | | `MIN_GAS_BALANCE_WEI` | Low-gas alert threshold | -| `ALERT_WEBHOOK_URL` | Optional JSON alert webhook in Slack incoming-webhook form: `text` holds `[CRITICAL]` or `[WARNING]`, the releaser and the subject, then the detail; `severity`, `subject` and `detail` are also sent as fields. A non-2xx answer, a timeout (10 s) or a connection error is logged as a failed delivery | +| `ALERT_WEBHOOK_URL` | Optional JSON alert webhook in Slack incoming-webhook form: `text` holds `[CRITICAL]` or `[WARNING]`, the releaser and the subject, then the detail; `severity`, `subject` and `detail` are also sent as fields. A non-2xx answer, a timeout (10 s) or a connection error is logged as a failed delivery and tried again after 2 s and after 10 s more; a 4xx other than 408 and 429 is not retried, as repeating the request cannot change it | | `HEARTBEAT_URL` | Optional http(s) URL requested (GET) after a cycle that completed without error, at most once a minute; point it at a dead-man's switch that pages when the requests stop. Startup refuses any other value. A failed request is logged (error name and code only, never the URL), never fatal | | `BASE_CHAIN_ID` | Default 8453; startup refuses a `BASE_RPC_URL` that serves another chain (`eth_chainId`) | | `BASE_FINALITY_TAG` | `safe` (default) or `finalized`; the scan cursor stays inside this boundary and a pending entry is dropped only once its nonce is consumed there | -| `BLOCKED_ALERT_INTERVAL_MS` | Re-page interval for governance-blocked releases (default 6h — the condition needs a ≥48h timelocked action, so per-poll paging would bury the signal) | -| `READ_BATCH_SIZE` | Bound on Multicall calldata and individual fallback concurrency (default 100) | +| `BLOCKED_ALERT_INTERVAL_MS` | Re-page interval for the same page about one release, for the same reason: governance-blocked, or failing unexpectedly (default 6h — a blocked release needs a ≥48h timelocked action, so per-poll paging would bury the signal). A different page about it, or the same page for another reason, goes out at once; each reason keeps its own interval. A page whose webhook delivery failed goes out again after 10 minutes, or after this interval when it is shorter | +| `READ_BATCH_SIZE` | Bound on Multicall calldata and individual fallback concurrency (default 100; a whole number of at least 1) | +| `FEE_BUMP_AFTER_SECONDS` | Re-send a release transaction that is still not mined after this long (default 60) | +| `FEE_BUMP_PERCENT` | Raise both fees by at least this many percent per re-send (default 20; a whole number of at least 10, the minimum a node accepts for a replacement) | +| `FEE_BUMP_MAX` | Re-sends per transaction before the `release transaction not mined` page (default 3) | +| `MAX_FEE_PER_GAS_GWEI` | No release transaction ever offers a higher `maxFeePerGas` (default 1) | ## Run @@ -63,13 +67,29 @@ Failures are classified rather than treated alike: | `ExceedsDailyCap`, `EnforcedPause`, `NotEnoughApprovals` | **Retry quietly** — expected temporary conditions | | `NonceAlreadyConsumed` (at the finality boundary) | **Done** — drop it | | `ExceedsPerReleaseCap`, `InsufficientVaultBalance` | **Alert, throttled** to `BLOCKED_ALERT_INTERVAL_MS` — cannot self-heal without governance or operator action | -| anything else | **Alert** | +| anything else | **Alert** `unexpected release failure`, **throttled** to `BLOCKED_ALERT_INTERVAL_MS` per release and reason: the revert, or the kind of error, never its text (which names requests and endpoints), so a transient RPC error neither mutes a revert nor makes it page again | A successful `release()` does **not** drop the entry immediately: it leaves the durable pending set only once the nonce reads as consumed at the `safe`/`finalized` boundary, so a reorged-away release is still ours to retry. Until then re-attempts are gas-free simulations classified as pending finality. +Each release is signed at the account's pending nonce with the node's fee +estimate, capped at `MAX_FEE_PER_GAS_GWEI`. One not mined within +`FEE_BUMP_AFTER_SECONDS` is followed from cycle to cycle instead of being +signed again, and no other release is sent meanwhile, since it would only +queue behind that nonce. Every `FEE_BUMP_AFTER_SECONDS` it is re-sent as the +identical call at the same nonce with both fees raised by `FEE_BUMP_PERCENT`, +and at least to the current estimate, at most `FEE_BUMP_MAX` times. When no +bump is left, or the cap stops one, it pages `release transaction not mined` +(with the nonce, the account's next mined nonce and the fees; throttled to +`BLOCKED_ALERT_INTERVAL_MS`) and keeps waiting, sending the last attempt +again unchanged every cycle, so a pool that dropped it gets it back. Once +another transaction uses its nonce, or the node holds nothing at a nonce +below it, the release is simulated and sent again. The transaction in flight +is kept in memory only: after a restart, a pending one stuck below the cap +has to be replaced by hand. + State (safe/finalized scan checkpoint + pending set) is atomically persisted and bound to the configured chain and vault, so a restart resumes without trusting unsafe logs or losing pending work. Malformed state is fatal, and so is diff --git a/releaser/src/alert.test.ts b/releaser/src/alert.test.ts index 9a65363d2..bd4480cfa 100644 --- a/releaser/src/alert.test.ts +++ b/releaser/src/alert.test.ts @@ -42,24 +42,50 @@ test("a webhook error status, a timeout or a dead relay is a failed delivery, an for (const status of [500, 429, 404]) { const { url, server } = await webhook(status); try { - assert.equal(await postAlert(url, { subject: "s" }), false, `HTTP ${status}`); + assert.equal(await postAlert(url, { subject: "s" }, 10_000, []), false, `HTTP ${status}`); } finally { server.close(); } } const hanging = await webhook(null); try { - assert.equal(await postAlert(hanging.url, { subject: "s" }, 200), false, "timeout"); + assert.equal(await postAlert(hanging.url, { subject: "s" }, 200, []), false, "timeout"); } finally { hanging.server.closeAllConnections(); hanging.server.close(); } const closed = await webhook(200); closed.server.close(); - assert.equal(await postAlert(closed.url, { subject: "s" }), false, "connection refused"); + assert.equal(await postAlert(closed.url, { subject: "s" }, 10_000, []), false, "connection refused"); assert.equal(logged.mock.callCount(), 5); }); +test("a failed delivery is tried again after each wait, a request the relay refused is not", async (t) => { + t.mock.method(console, "error", () => {}); + let answers: number[] = []; + let posts = 0; + const server = createServer((request, response) => { + request.resume(); + request.on("end", () => response.writeHead(answers[posts++]).end()); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; + try { + for (const [statuses, delivered] of [ + [[503, 429, 200], true], + [[500, 502, 504], false], + [[404], false], + ] as const) { + answers = [...statuses]; + posts = 0; + assert.equal(await postAlert(url, { subject: "s" }, 10_000, [10, 10]), delivered, `${statuses}`); + assert.equal(posts, statuses.length, `${statuses}`); + } + } finally { + server.close(); + } +}); + test("a heartbeat requests its URL, and a failing endpoint is logged without the URL, never thrown", async (t) => { const logged = t.mock.method(console, "error", () => {}); const seen: string[] = []; diff --git a/releaser/src/alert.ts b/releaser/src/alert.ts index c5af62001..376769f49 100644 --- a/releaser/src/alert.ts +++ b/releaser/src/alert.ts @@ -66,24 +66,42 @@ export function throttledHeartbeat( }; } +/** A failed alert delivery is tried again after each of these waits: a relay + * that is briefly down or rate-limiting must not swallow a page. */ +const ALERT_RETRY_DELAYS_MS = [2_000, 10_000]; + /** * POST one alert to the webhook and report whether it was delivered. `fetch` * resolves on any HTTP answer, so a non-2xx status is checked explicitly: a * relay answering 4xx/5xx, a timeout and a connection failure are all logged - * and count as not delivered. + * and count as not delivered. Each is tried again after the next of + * `retryDelaysMs`, except a 4xx other than 408 and 429: the relay refused the + * request itself (a wrong URL, a rejected payload), which repeating it cannot + * change. */ -export async function postAlert(url: string, body: unknown, timeoutMs = 10_000): Promise { - try { - const response = await fetch(url, { - method: "POST", - headers: { "content-type": "application/json" }, - body: JSON.stringify(body), - signal: AbortSignal.timeout(timeoutMs), - }); - if (response.ok) return true; - console.error(`alert webhook failed: HTTP ${response.status}`); - } catch (webhookError) { - console.error("alert webhook failed", webhookError); +export async function postAlert( + url: string, + body: unknown, + timeoutMs = 10_000, + retryDelaysMs = ALERT_RETRY_DELAYS_MS, +): Promise { + for (let attempt = 0; ; attempt++) { + let status: number | undefined; + try { + const response = await fetch(url, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify(body), + signal: AbortSignal.timeout(timeoutMs), + }); + if (response.ok) return true; + status = response.status; + console.error(`alert webhook failed: HTTP ${response.status}`); + } catch (webhookError) { + console.error("alert webhook failed", webhookError); + } + const refused = status !== undefined && status < 500 && status !== 408 && status !== 429; + if (refused || attempt === retryDelaysMs.length) return false; + await new Promise((resolve) => setTimeout(resolve, retryDelaysMs[attempt])); } - return false; } diff --git a/releaser/src/checks.test.ts b/releaser/src/checks.test.ts index 89b065df6..a9a69276e 100644 --- a/releaser/src/checks.test.ts +++ b/releaser/src/checks.test.ts @@ -1,7 +1,16 @@ import { strict as assert } from "node:assert"; import { test } from "node:test"; -import { ContractFunctionRevertedError, encodeErrorResult } from "viem"; -import { blockRanges, chunks, classifyReleaseFailure, contractErrorName, toPalletAmount } from "./checks.js"; +import { + ContractFunctionRevertedError, + encodeErrorResult, + HttpRequestError, + InvalidInputRpcError, + InvalidParamsRpcError, + LimitExceededRpcError, + RpcRequestError, + TimeoutError, +} from "viem"; +import { chunks, classifyReleaseFailure, contractErrorName, isRangeTooLarge, readInRanges, toPalletAmount } from "./checks.js"; import { vaultAbi } from "./vaultAbi.js"; test("a cap-deferred release retries quietly — the case this service exists for", () => { @@ -46,16 +55,89 @@ test("a non-exact conversion throws rather than truncating into a wrong payload assert.throws(() => toPalletAmount(10n, 0n), /invalid conversionFactor/); }); -test("block ranges are chunked for RPC log limits, inclusive and gapless", () => { - assert.deepEqual(blockRanges(1n, 10n, 4n), [ - [1n, 4n], - [5n, 8n], - [9n, 10n], +/** An endpoint's refusal of a log range, as viem raises a JSON-RPC error. */ +const rpcError = (code: number, message: string) => new RpcRequestError({ body: {}, error: { code, message }, url: "http://rpc" }); + +test("an eth_getLogs refusal for the size of its range is recognised, other failures are not", () => { + // Base's public endpoints answer it with HTTP 400. + const base = { code: -32602, message: "block range greater than 200 max" }; + assert.equal(isRangeTooLarge(new HttpRequestError({ url: "http://rpc", status: 400, details: JSON.stringify(base) })), true); + assert.equal(isRangeTooLarge(new InvalidParamsRpcError(rpcError(base.code, base.message))), true); + for (const message of [ + "exceed maximum block range: 50000", + "query exceeds max block range 100000", + "eth_getLogs is limited to a 10,000 range", + "query returned more than 10000 results", + "Log response size exceeded.", + ]) { + assert.equal(isRangeTooLarge(new InvalidInputRpcError(rpcError(-32000, message))), true, message); + } + assert.equal(isRangeTooLarge(new HttpRequestError({ url: "http://rpc", status: 429, details: "Too Many Requests" })), false); + assert.equal(isRangeTooLarge(new LimitExceededRpcError(rpcError(-32005, "request limit reached"))), false); + assert.equal(isRangeTooLarge(new TimeoutError({ body: {}, url: "http://rpc" })), false); +}); + +test("log ranges are read in order and gapless, shrinking while the endpoint refuses their size and growing back to the maximum", async () => { + let cap = 200n; + const reads: [bigint, bigint][] = []; + const read = async (start: bigint, end: bigint) => { + if (end - start + 1n > cap) throw new InvalidParamsRpcError(rpcError(-32602, `block range greater than ${cap} max`)); + reads.push([start, end]); + }; + const span = { size: 10_000n, max: 10_000n, streak: 0 }; + await readInRanges(1n, 500n, span, read); + // 10,000 halved until accepted. + assert.deepEqual(reads, [ + [1n, 156n], + [157n, 312n], + [313n, 468n], + [469n, 500n], ]); - // A single block still produces one range. - assert.deepEqual(blockRanges(7n, 7n, 100n), [[7n, 7n]]); + // Ten full-size reads in a row try twice the size again, refused here. + reads.length = 0; + await readInRanges(501n, 2060n, span, read); + assert.deepEqual( + reads.map(([start, end]) => end - start + 1n), + Array(10).fill(156n), + ); + assert.equal(reads.at(-1)?.[1], 2060n); + assert.equal(span.size, 156n); + // On an endpoint without the cap it grows back, never above the maximum. + cap = 1_000_000n; + reads.length = 0; + await readInRanges(2061n, 1_000_000n, span, read); + const sizes = reads.map(([start, end]) => end - start + 1n); + assert.ok(sizes.every((size, i) => size <= 10_000n && (i === 0 || reads[i][0] === reads[i - 1][1] + 1n))); + assert.equal(span.size, 10_000n); + assert.equal(sizes.at(-2), 10_000n); // Nothing new to scan. - assert.deepEqual(blockRanges(11n, 10n, 100n), []); + reads.length = 0; + await readInRanges(11n, 10n, span, read); + assert.deepEqual(reads, []); +}); + +test("a refused single block and any other failure propagate", async () => { + const refused = async () => { + throw new InvalidParamsRpcError(rpcError(-32602, "block range greater than 0 max")); + }; + const tried: bigint[] = []; + const span = { size: 4n, max: 4n, streak: 0 }; + await assert.rejects( + readInRanges(1n, 10n, span, async (start, end) => { + tried.push(end - start + 1n); + await refused(); + }), + InvalidParamsRpcError, + ); + assert.deepEqual(tried, [4n, 2n, 1n]); + const unchanged = { size: 100n, max: 100n, streak: 0 }; + await assert.rejects( + readInRanges(1n, 10n, unchanged, async () => { + throw new HttpRequestError({ url: "http://rpc", status: 503 }); + }), + HttpRequestError, + ); + assert.equal(unchanged.size, 100n, "not a reason to shrink"); }); test("an amount larger than the daily cap itself is blocked, not a refill wait", () => { diff --git a/releaser/src/checks.ts b/releaser/src/checks.ts index d7d336f42..1b4a827a5 100644 --- a/releaser/src/checks.ts +++ b/releaser/src/checks.ts @@ -67,6 +67,21 @@ export function classifyReleaseFailure(errorName: string | undefined): ReleaseOu return "unexpected"; } +/** What a failed release is about, without the volatile parts of its text + * (request ids, URLs, block numbers): the vault's revert (the reason it gave, + * its error name, or the selector of an error the ABI lacks), or else the + * names of the errors it was raised through. */ +export function failureReason(error: unknown): string { + const names: string[] = []; + for (let cause: unknown = error, depth = 0; cause instanceof Error && depth < 10; cause = cause.cause, depth++) { + if (cause instanceof ContractFunctionRevertedError) { + return `reverted: ${cause.reason ?? cause.data?.errorName ?? cause.signature ?? "no reason"}`; + } + names.push(cause.name); + } + return names.join(" < ") || String(error); +} + /** Divide work into bounded batches. Both Multicall calldata and individual * fallback concurrency use the same explicit limit. */ export function chunks(items: T[], size: number): T[][] { @@ -94,19 +109,61 @@ export function toPalletAmount(tokenAmount: bigint, conversionFactor: bigint): b return tokenAmount / conversionFactor; } +/** How endpoints word a refused `eth_getLogs` span: Base's public endpoints + * (`block range greater than 200 max`), geth and reth (`exceed(s) maximum + * block range`), and hosted providers (`limited to a 10,000 range`, `ranges + * over 10000 blocks`, `query returned more than 10000 results`, `Log response + * size exceeded`). */ +const RANGE_TOO_LARGE = /\branges?\b|more than [\d,]+ results|response size/i; + +/** True when an endpoint refused an `eth_getLogs` call for the size of its + * block range (or of its answer), which a smaller range avoids. */ +export function isRangeTooLarge(error: unknown): boolean { + let depth = 0; + for (let cause: unknown = error; cause && depth < 10; cause = (cause as { cause?: unknown }).cause, depth++) { + for (const key of ["details", "shortMessage", "message"] as const) { + const text = (cause as Record)[key]; + if (typeof text === "string" && RANGE_TOO_LARGE.test(text)) return true; + } + } + return false; +} + +/** Full-size reads in a row after which a reduced span is tried at twice its + * size again. */ +const GROW_AFTER_READS = 10; + /** - * Split [fromBlock, toBlock] into chunks no larger than `maxRange` blocks. - * - * Public Base RPCs cap `eth_getLogs` ranges; a releaser restarting after a long - * outage would otherwise request a span the provider rejects and make no - * progress at all. Returns [] when there is nothing new to scan. + * Read [fromBlock, toBlock] in order, in consecutive ranges of at most + * `span.size` blocks. Public Base RPCs cap `eth_getLogs` ranges, some far below + * any sensible default (Base Sepolia's at 200 blocks), so a range the endpoint + * refuses as too large is read again at half the size, down to one block. After + * `GROW_AFTER_READS` full-size reads in a row succeed, the size doubles again, + * never above `span.max`. `span` persists across calls; any other failure, or + * a refused single block, propagates. */ -export function blockRanges(fromBlock: bigint, toBlock: bigint, maxRange: bigint): Array<[bigint, bigint]> { - if (maxRange <= 0n) throw new Error(`invalid maxRange ${maxRange}`); - const ranges: Array<[bigint, bigint]> = []; - for (let start = fromBlock; start <= toBlock; start += maxRange) { - const end = start + maxRange - 1n; - ranges.push([start, end > toBlock ? toBlock : end]); +export async function readInRanges( + fromBlock: bigint, + toBlock: bigint, + span: { size: bigint; max: bigint; streak: number }, + read: (start: bigint, end: bigint) => Promise, +): Promise { + if (span.size <= 0n) throw new Error(`invalid block range size ${span.size}`); + for (let start = fromBlock; start <= toBlock; ) { + const end = start + span.size - 1n < toBlock ? start + span.size - 1n : toBlock; + try { + await read(start, end); + } catch (error) { + if (span.size === 1n || !isRangeTooLarge(error)) throw error; + span.size /= 2n; + span.streak = 0; + continue; + } + // Only a read of the full size shows that the size is accepted. + if (end - start + 1n === span.size && ++span.streak >= GROW_AFTER_READS) { + span.size = span.size * 2n < span.max ? span.size * 2n : span.max; + span.streak = 0; + } + start = end + 1n; } - return ranges; } diff --git a/releaser/src/config.test.ts b/releaser/src/config.test.ts index 73fcbdbe7..29737a671 100644 --- a/releaser/src/config.test.ts +++ b/releaser/src/config.test.ts @@ -27,3 +27,49 @@ test("HEARTBEAT_URL must be an http(s) URL, and a bad one is refused without ech assert.match(stderr, /invalid HEARTBEAT_URL/); assert.doesNotMatch(stderr, /secret-token/); }); + +test("fee bumping defaults to 60 s, 20 %, 3 bumps and a 1 gwei cap, and refuses invalid settings", () => { + const print = "[config.feeBump.afterMs, config.feeBump.percent, config.feeBump.max, config.feeBump.maxFeePerGas].join(' ')"; + assert.equal(loadConfig({}, print).stdout, "60000 20 3 1000000000"); + assert.equal( + loadConfig({ FEE_BUMP_AFTER_SECONDS: "30", FEE_BUMP_PERCENT: "10", FEE_BUMP_MAX: "0", MAX_FEE_PER_GAS_GWEI: "0.25" }, print) + .stdout, + "30000 10 0 250000000", + ); + for (const [name, value] of [ + ["FEE_BUMP_PERCENT", "9"], + ["FEE_BUMP_PERCENT", "12.5"], + ["FEE_BUMP_MAX", "-1"], + ["FEE_BUMP_AFTER_SECONDS", "0"], + ["MAX_FEE_PER_GAS_GWEI", "0"], + ["MAX_FEE_PER_GAS_GWEI", "1e9"], + ]) { + const { status, stderr } = loadConfig({ [name]: value }); + assert.notEqual(status, 0, `${name}=${value}`); + assert.match(stderr, new RegExp(`invalid ${name}`), `${name}=${value}`); + } +}); + +test("a zero poll interval or read batch size is refused instead of looping or failing every cycle", () => { + assert.equal(loadConfig({}, "[config.pollIntervalMs, config.readBatchSize].join(' ')").stdout, "60000 100"); + for (const [name, value] of [ + ["POLL_INTERVAL_MS", "0"], + ["POLL_INTERVAL_MS", "999"], + ["READ_BATCH_SIZE", "0"], + ["READ_BATCH_SIZE", "2.5"], + ]) { + const { status, stderr } = loadConfig({ [name]: value }); + assert.notEqual(status, 0, `${name}=${value}`); + assert.match(stderr, new RegExp(`invalid ${name}`), `${name}=${value}`); + } +}); + +test("MAX_BLOCK_RANGE must be a whole number of at least 1", () => { + assert.equal(loadConfig({}, "String(config.maxBlockRange)").stdout, "10000"); + assert.equal(loadConfig({ MAX_BLOCK_RANGE: "200" }, "String(config.maxBlockRange)").stdout, "200"); + for (const value of ["0", "-1", "2.5", "lots"]) { + const { status, stderr } = loadConfig({ MAX_BLOCK_RANGE: value }); + assert.notEqual(status, 0, value); + assert.match(stderr, /invalid MAX_BLOCK_RANGE/, value); + } +}); diff --git a/releaser/src/config.ts b/releaser/src/config.ts index 3ed978da9..bcd66d9e1 100644 --- a/releaser/src/config.ts +++ b/releaser/src/config.ts @@ -1,4 +1,6 @@ +import { parseGwei } from "viem"; import { parseHeartbeatUrl } from "./alert.js"; +import { MIN_REPLACEMENT_PERCENT } from "./release.js"; function required(name: string): string { const value = process.env[name]; @@ -10,14 +12,23 @@ function required(name: string): string { /** A numeric environment value, validated at startup: a typo must fail loudly * here, not surface later as NaN-driven behavior. */ -function envNumber(name: string, fallback: number): number { +function envNumber(name: string, fallback: number, min = 0, integer = false): number { const raw = process.env[name]; if (raw === undefined || raw === "") return fallback; const value = Number(raw); - if (!Number.isFinite(value) || value < 0) throw new Error(`invalid ${name}: ${raw}`); + if (!Number.isFinite(value) || value < min || (integer && !Number.isInteger(value))) { + throw new Error(`invalid ${name}: ${raw}`); + } return value; } +/** A positive gwei amount given as a decimal, e.g. `0.5`. */ +function envGwei(name: string, fallback: string): bigint { + const raw = process.env[name] || fallback; + if (!/^\d+(\.\d{1,9})?$/.test(raw) || parseGwei(raw) === 0n) throw new Error(`invalid ${name}: ${raw}`); + return parseGwei(raw); +} + function finalityTag(): "safe" | "finalized" { const value = process.env.BASE_FINALITY_TAG ?? "safe"; if (value !== "safe" && value !== "finalized") throw new Error(`invalid BASE_FINALITY_TAG ${value}`); @@ -38,9 +49,10 @@ export const config = { /** Base block to begin scanning `ReleasePending` from on a first run — * set to the vault's deployment block. */ startBlock: BigInt(process.env.START_BLOCK ?? "0"), - pollIntervalMs: envNumber("POLL_INTERVAL_MS", 60_000), - /** Max blocks per `eth_getLogs` call; public RPCs cap this. */ - maxBlockRange: BigInt(process.env.MAX_BLOCK_RANGE ?? "10000"), + pollIntervalMs: envNumber("POLL_INTERVAL_MS", 60_000, 1000), + /** Max blocks per `eth_getLogs` call. Public RPCs cap this, often lower: + * a range the endpoint refuses is read again in smaller ones. */ + maxBlockRange: BigInt(envNumber("MAX_BLOCK_RANGE", 10_000, 1, true)), /** Alert when the gas wallet drops below this balance (wei). */ minGasBalanceWei: BigInt(process.env.MIN_GAS_BALANCE_WEI ?? "5000000000000000"), alertWebhookUrl: process.env.ALERT_WEBHOOK_URL, @@ -58,5 +70,17 @@ export const config = { * timelock, so per-poll paging would only bury the signal. */ blockedAlertIntervalMs: envNumber("BLOCKED_ALERT_INTERVAL_MS", 6 * 60 * 60 * 1000), /** Bound both Multicall calldata and individual fallback concurrency. */ - readBatchSize: envNumber("READ_BATCH_SIZE", 100), + readBatchSize: envNumber("READ_BATCH_SIZE", 100, 1, true), + /** A release transaction still not mined after `afterMs` is re-sent at the + * same nonce with both fees raised by `percent` (at least the 10 % a + * replacement needs, and at least to the current estimate), at most `max` + * times and never above `maxFeePerGas`; then it pages. No other release is + * sent meanwhile, as it would only queue behind that nonce. Same settings + * and defaults as the attestor's. */ + feeBump: { + afterMs: envNumber("FEE_BUMP_AFTER_SECONDS", 60, 1) * 1000, + percent: envNumber("FEE_BUMP_PERCENT", 20, MIN_REPLACEMENT_PERCENT, true), + max: envNumber("FEE_BUMP_MAX", 3, 0, true), + maxFeePerGas: envGwei("MAX_FEE_PER_GAS_GWEI", "1"), + }, }; diff --git a/releaser/src/main.ts b/releaser/src/main.ts index 383ef9224..d857caa4a 100644 --- a/releaser/src/main.ts +++ b/releaser/src/main.ts @@ -23,7 +23,7 @@ import { createPublicClient, createWalletClient, defineChain, http } from "viem"; import { privateKeyToAccount } from "viem/accounts"; import { alertText, postAlert, throttledHeartbeat, type Severity } from "./alert.js"; -import { blockRanges, chunks, toPalletAmount } from "./checks.js"; +import { chunks, readInRanges, toPalletAmount } from "./checks.js"; import { config } from "./config.js"; import { drainPending, type PendingRelease, type Releaser } from "./release.js"; import { @@ -57,7 +57,7 @@ const walletClient = createWalletClient({ account, chain: baseChain, transport: const pending = new Map(); let fromBlock = config.startBlock; -const blockedAlertedAt = new Map(); +const alertedAt: Releaser["alertedAt"] = new Map(); function log(message: string): void { console.log(`${new Date().toISOString()} ${message}`); @@ -69,11 +69,13 @@ function redact(text: string): string { return text.replace(/https?:\/\/[^\s"')]+/gi, ""); } -async function alert(subject: string, detail: string, severity: Severity = "warning"): Promise { +/** Log the alert and post it to the webhook; resolves to whether it was + * delivered. Without a webhook the log line is the delivery. */ +async function alert(subject: string, detail: string, severity: Severity = "warning"): Promise { console.error(`${new Date().toISOString()} ALERT: ${subject} — ${detail}`); - if (!config.alertWebhookUrl) return; + if (!config.alertWebhookUrl) return true; const redacted = redact(detail); - await postAlert(config.alertWebhookUrl, { + return postAlert(config.alertWebhookUrl, { text: alertText(severity, `pen-releaser ${account.address}`, subject, redacted), severity, service: "pen-releaser", @@ -112,9 +114,13 @@ function saveState(): void { writePersistedState(config.stateFile, state); } +/** The `eth_getLogs` span, MAX_BLOCK_RANGE at most: it shrinks while the + * endpoint refuses ranges as too large (see `readInRanges`). */ +const logSpan = { size: config.maxBlockRange, max: config.maxBlockRange, streak: 0 }; + /** Scan for newly deferred releases and add them to the pending set. */ async function ingestNewPending(toBlock: bigint, conversionFactor: bigint): Promise { - for (const [start, end] of blockRanges(fromBlock, toBlock, config.maxBlockRange)) { + await readInRanges(fromBlock, toBlock, logSpan, async (start, end) => { // Probe the range's end block first. Against a load-balanced endpoint, // `eth_getLogs` can be served by a node that has not reached `end` yet // and some providers then silently truncate rather than error — which @@ -140,7 +146,7 @@ async function ingestNewPending(toBlock: bigint, conversionFactor: bigint): Prom log(`deferred release observed: nonce=${nonce} recipient=${recipient}`); } fromBlock = end + 1n; - } + }); } /** Read `nonceConsumed` for many nonces, batched where possible. @@ -198,7 +204,7 @@ async function pruneConsumed(blockNumber: bigint): Promise { entries.forEach((entry, i) => { if (consumed[i]) { pending.delete(entry.nonce); - blockedAlertedAt.delete(entry.nonce); + alertedAt.delete(entry.nonce); } }); } @@ -211,9 +217,11 @@ const releaser: Releaser = { baseFinalityTag: config.baseFinalityTag, blockedAlertIntervalMs: config.blockedAlertIntervalMs, pending, - blockedAlertedAt, + alertedAt, + feeBump: config.feeBump, alert, log, + now: Date.now, }; async function checkGasBalance(): Promise { @@ -262,7 +270,10 @@ async function main(): Promise { }); await drainPending(releaser, conversionFactor, dailyCap); saveState(); - log(`ok: ${pending.size} pending, scanned ${config.baseFinalityTag} through block ${fromBlock - 1n}`); + log( + `ok: ${pending.size} pending, scanned ${config.baseFinalityTag} through block ${fromBlock - 1n} ` + + `(eth_getLogs span ${logSpan.size})`, + ); void heartbeat(); } catch (cycleError) { await alert("releaser cycle failed", `${cycleError}`); diff --git a/releaser/src/release.test.ts b/releaser/src/release.test.ts index c33f73aac..2d30cac60 100644 --- a/releaser/src/release.test.ts +++ b/releaser/src/release.test.ts @@ -4,10 +4,18 @@ import { ContractFunctionExecutionError, ContractFunctionRevertedError, encodeErrorResult, + getContractError, + HttpRequestError, + InvalidInputRpcError, + RpcRequestError, + TransactionReceiptNotFoundError, + WaitForTransactionReceiptTimeoutError, + type Hash, type PublicClient, } from "viem"; import { privateKeyToAccount } from "viem/accounts"; -import { drainPending, type PendingRelease, type Releaser } from "./release.js"; +import { getTransactionError } from "viem/utils"; +import { drainPending, type InFlightRelease, type PendingRelease, type Releaser } from "./release.js"; import { vaultAbi } from "./vaultAbi.js"; const RECIPIENT = "0x2222222222222222222222222222222222222222"; @@ -15,61 +23,302 @@ const RECIPIENT = "0x2222222222222222222222222222222222222222"; const CONVERSION_FACTOR = 1_000_000n; const DAILY_CAP = 10_000n * 10n ** 18n; -/** A releaser whose public client answers with `answers` (the finality - * boundary at block 100 and every nonce unconsumed unless overridden). */ -function releaserWith(answers: Record unknown>, entries: PendingRelease[]) { +type Answer = (args: Record) => unknown; +type Write = InFlightRelease["call"] & { nonce: number; maxFeePerGas: bigint; maxPriorityFeePerGas: bigint }; + +/** The hash the wallet answers for its nth send. */ +const txHash = (index: number): Hash => `0x${(index + 1).toString(16).padStart(64, "0")}`; + +/** A releaser whose public client answers each call with `answers`. Unless + * overridden: the finality boundary is at block 100, every nonce unconsumed, + * a release simulates fine, the account's nonce is 4, and a sent release is + * mined at once. A `writeContract` answer is the node's to each send, given + * the call and its hash: it may throw a refusal. */ +function releaserWith(answers: Record, entries: PendingRelease[]) { const alerts: string[] = []; const logs: string[] = []; - const writes: unknown[] = []; - const methods: Record unknown> = { getBlock: () => ({ number: 100n }), readContract: () => false, ...answers }; + const writes: Write[] = []; + const methods: Record = { + getBlock: () => ({ number: 100n }), + readContract: () => false, + simulateContract: () => ({}), + estimateContractGas: () => 50_000n, + getTransactionCount: () => 4, + estimateFeesPerGas: () => ({ maxFeePerGas: 7_000_000n, maxPriorityFeePerGas: 1_000_000n }), + waitForTransactionReceipt: ({ hash }) => ({ status: "success", transactionHash: hash }), + ...answers, + }; + let clock = 1_000_000; const releaser: Releaser = { publicClient: Object.fromEntries( - Object.entries(methods).map(([name, answer]) => [name, async () => answer()]), + Object.entries(methods).map(([name, answer]) => [name, async (args: Record = {}) => answer(args)]), ) as unknown as PublicClient, walletClient: { - writeContract: async (request: unknown) => { - writes.push(request); - return `0x${"ab".repeat(32)}`; + writeContract: async (call: Write) => { + writes.push(call); + const hash = txHash(writes.length - 1); + await methods.writeContract?.({ ...call, hash }); + return hash; }, } as unknown as Releaser["walletClient"], + feeBump: { afterMs: 60_000, percent: 20, max: 3, maxFeePerGas: 1_000_000_000n }, account: privateKeyToAccount(`0x${"11".repeat(32)}`), vaultAddress: "0x1111111111111111111111111111111111111111", baseFinalityTag: "safe", blockedAlertIntervalMs: 6 * 3600_000, pending: new Map(entries.map((entry) => [entry.nonce, entry])), - blockedAlertedAt: new Map(), + alertedAt: new Map(), alert: async (subject, detail, severity = "warning") => { alerts.push(`${severity} ${subject}: ${detail}`); + return true; }, log: (message) => logs.push(message), + now: () => clock, }; - return { releaser, alerts, logs, writes }; + return { releaser, alerts, logs, writes, advance: (ms: number) => (clock += ms) }; } -/** What simulation raises for the vault's `ExceedsDailyCap`. */ -function dailyCapRevert(): Error { - const data = encodeErrorResult({ abi: vaultAbi, errorName: "ExceedsDailyCap", args: [DAILY_CAP + 1n, 0n] }); +/** What simulation raises for a vault revert. */ +function vaultRevert( + error: + | { errorName: "ExceedsDailyCap" | "ExceedsPerReleaseCap"; args: [bigint, bigint] } + | { errorName: "NonceAlreadyConsumed"; args: [bigint] }, +): Error { + const data = encodeErrorResult({ abi: vaultAbi, ...error }); return new ContractFunctionExecutionError( new ContractFunctionRevertedError({ abi: vaultAbi, data, functionName: "release" }), { abi: vaultAbi, functionName: "release", args: [7n, RECIPIENT, 0n] }, ); } -test("a successful release keeps its entry until the nonce reads consumed at the finality boundary", async () => { - const entry = { nonce: 7n, recipient: RECIPIENT, palletAmount: 5_000n * 10n ** 12n } as const; - const { releaser, alerts, logs, writes } = releaserWith( - { - simulateContract: () => ({ request: { functionName: "release" } }), - estimateContractGas: () => 50_000n, - waitForTransactionReceipt: () => ({ status: "success" }), - }, - [entry], +const dailyCapRevert = () => vaultRevert({ errorName: "ExceedsDailyCap", args: [DAILY_CAP + 1n, 0n] }); + +/** What simulation raises for a revert with a reason string. */ +const revertedWith = (reason: string) => + new ContractFunctionRevertedError({ + abi: vaultAbi, + data: encodeErrorResult({ + abi: [{ type: "error", name: "Error", inputs: [{ name: "message", type: "string" }] }], + errorName: "Error", + args: [reason], + }), + functionName: "release", + }); + +/** What writeContract raises when the node refuses the transaction. */ +const sendRefusal = (message: string) => + getContractError( + getTransactionError(new InvalidInputRpcError(new RpcRequestError({ body: {}, error: { code: -32000, message }, url: "http://rpc" })), { + account: null, + }), + { abi: vaultAbi, functionName: "release", args: [] }, ); + +const ENTRY = { nonce: 7n, recipient: RECIPIENT, palletAmount: 5_000n * 10n ** 12n } as const; + +test("a successful release keeps its entry until the nonce reads consumed at the finality boundary", async () => { + const { releaser, alerts, logs, writes } = releaserWith({}, [ENTRY]); await drainPending(releaser, CONVERSION_FACTOR, DAILY_CAP); - assert.deepEqual(writes, [{ functionName: "release", gas: 100_000n }]); - assert.deepEqual([...releaser.pending.values()], [entry]); + // The vault's release with the entry's tuple, at the pending nonce, the + // estimated fees and a doubled gas limit. + assert.deepEqual( + writes.map(({ address, functionName, args, gas, nonce, maxFeePerGas, maxPriorityFeePerGas }) => ({ + address, + functionName, + args, + gas, + nonce, + maxFeePerGas, + maxPriorityFeePerGas, + })), + [ + { + address: releaser.vaultAddress, + functionName: "release", + args: [ENTRY.nonce, ENTRY.recipient, ENTRY.palletAmount], + gas: 100_000n, + nonce: 4, + maxFeePerGas: 7_000_000n, + maxPriorityFeePerGas: 1_000_000n, + }, + ], + ); + assert.deepEqual([...releaser.pending.values()], [ENTRY]); assert.ok(logs.some((line) => line.startsWith("released (awaiting safe): nonce=7 "))); assert.deepEqual(alerts, []); + assert.equal(releaser.inFlight, undefined); +}); + +/** A node that never mines what it is sent. */ +const notMined: Record = { + waitForTransactionReceipt: ({ hash }) => { + throw new WaitForTransactionReceiptTimeoutError({ hash: hash as Hash }); + }, + getTransactionReceipt: ({ hash }) => { + throw new TransactionReceiptNotFoundError({ hash: hash as Hash }); + }, +}; + +test("a release not mined in time is followed at its nonce: nothing is sent behind it, it is bumped, then paged and sent again unchanged", async () => { + let minedHash: Hash | undefined; + const { releaser, alerts, logs, writes, advance } = releaserWith( + { + ...notMined, + simulateContract: ({ args }) => { + // Once it is mined, the latest state has the nonce consumed. + if (minedHash && (args as bigint[])[0] === ENTRY.nonce) { + throw vaultRevert({ errorName: "NonceAlreadyConsumed", args: [ENTRY.nonce] }); + } + return {}; + }, + getTransactionReceipt: ({ hash }) => { + if (hash !== minedHash) throw new TransactionReceiptNotFoundError({ hash: hash as Hash }); + return { status: "success", transactionHash: hash }; + }, + // The node still holds the last bump, then has the second one mined. + writeContract: () => { + if (writes.length === 5) throw sendRefusal("already known"); + if (writes.length === 6) throw sendRefusal("nonce too low"); + }, + }, + [ENTRY, { ...ENTRY, nonce: 8n }], + ); + const drain = () => drainPending(releaser, CONVERSION_FACTOR, DAILY_CAP); + await drain(); + advance(30_000); + await drain(); + assert.equal(writes.length, 1, "not sent again before the bump interval, and nothing sent behind it"); + for (let cycle = 0; cycle < 5; cycle++) { + advance(60_000); + await drain(); + } + // The identical call at the same nonce, both fees raised, three times; then + // the last one again, unchanged, every cycle. + assert.deepEqual( + writes.map((w) => [w.args[0], w.nonce, w.gas, w.maxFeePerGas, w.maxPriorityFeePerGas]), + [ + [7n, 4, 100_000n, 7_000_000n, 1_000_000n], + [7n, 4, 100_000n, 8_400_000n, 1_200_000n], + [7n, 4, 100_000n, 10_080_000n, 1_440_000n], + [7n, 4, 100_000n, 12_096_000n, 1_728_000n], + [7n, 4, 100_000n, 12_096_000n, 1_728_000n], + [7n, 4, 100_000n, 12_096_000n, 1_728_000n], + ], + ); + // One page, not repeated within the interval; the node's answers to the + // sends again are no failure. + assert.deepEqual(alerts, [ + `critical release transaction not mined: 3 fee bump(s) used up: transaction nonce 4, account's next mined nonce 4, ` + + `maxFeePerGas 0.012096 gwei (cap 1 gwei); nonce=7 recipient=${RECIPIENT} amount=${ENTRY.palletAmount}`, + ]); + assert.ok(!logs.some((line) => line.startsWith("re-sending "))); + // The second bump is mined: its entry waits for the boundary, and the next release goes out. + minedHash = txHash(2); + await drain(); + assert.deepEqual( + writes.map((w) => w.args[0]), + [7n, 7n, 7n, 7n, 7n, 7n, 8n], + ); + assert.equal(releaser.inFlight?.entry.nonce, 8n); + assert.equal(releaser.pending.size, 2); + assert.equal(alerts.length, 1); +}); + +test("a release whose nonce another transaction used, or that an empty nonce below keeps from being mined, is sent anew", async () => { + let latest = 4; + let pendingCount = 5; + const { releaser, writes, logs } = releaserWith( + { ...notMined, getTransactionCount: ({ blockTag }) => (blockTag === "latest" ? latest : pendingCount) }, + [ENTRY], + ); + const drain = () => drainPending(releaser, CONVERSION_FACTOR, DAILY_CAP); + await drain(); + // Queued behind a transaction at nonce 4, which then leaves the pool unmined. + pendingCount = 4; + await drain(); + // Another transaction takes nonce 4. + latest = 5; + pendingCount = 5; + await drain(); + assert.deepEqual( + writes.map((w) => w.nonce), + [5, 4, 5], + ); + assert.equal(logs.filter((line) => line.startsWith("release transaction at nonce ")).length, 2); +}); + +test("the fee cap stops a release's bump and pages; no attempt offers more than the cap", async () => { + const { releaser, alerts, writes, advance } = releaserWith(notMined, [ENTRY]); + releaser.feeBump = { ...releaser.feeBump, maxFeePerGas: 8_000_000n }; + for (let cycle = 0; cycle < 3; cycle++) { + await drainPending(releaser, CONVERSION_FACTOR, DAILY_CAP); + advance(60_000); + } + // 7 → 8.4 is capped at 8; 8 → 8 is no replacement a pool takes, so the + // capped attempt is sent again unchanged. + assert.deepEqual( + writes.map((w) => [w.maxFeePerGas, w.maxPriorityFeePerGas]), + [ + [7_000_000n, 1_000_000n], + [8_000_000n, 1_200_000n], + [8_000_000n, 1_200_000n], + ], + ); + assert.equal(alerts.length, 1); + assert.match(alerts[0], /^critical release transaction not mined: the fee cap stops the next bump: transaction nonce 4, .*\(cap 0\.008 gwei\); nonce=7 /); +}); + +test("a release the pool dropped after its last bump is sent again unchanged, so the releases behind it go out", async () => { + const mined = new Set(); + const receipt = (hash: Hash, missing: Error) => { + if (!mined.has(hash)) throw missing; + return { status: "success", transactionHash: hash }; + }; + let feesFell = false; + const { releaser, alerts, logs, writes, advance } = releaserWith( + { + simulateContract: ({ args }) => { + if (mined.size > 0 && (args as bigint[])[0] === ENTRY.nonce) { + throw vaultRevert({ errorName: "NonceAlreadyConsumed", args: [ENTRY.nonce] }); + } + return {}; + }, + // Dropped from the pool: the node counts nothing pending at its nonce. + getTransactionCount: () => 4 + mined.size, + getTransactionReceipt: ({ hash }) => receipt(hash as Hash, new TransactionReceiptNotFoundError({ hash: hash as Hash })), + waitForTransactionReceipt: ({ hash }) => + receipt(hash as Hash, new WaitForTransactionReceiptTimeoutError({ hash: hash as Hash })), + // Nothing is mined until fees fall; then whatever is sent is. + writeContract: ({ hash }) => { + if (feesFell) mined.add(hash as Hash); + }, + }, + [ENTRY, { ...ENTRY, nonce: 8n }], + ); + for (let minute = 0; minute < 7; minute++) { + feesFell = minute >= 5; + await drainPending(releaser, CONVERSION_FACTOR, DAILY_CAP); + advance(60_000); + } + assert.deepEqual( + writes.map((w) => [w.args[0], w.nonce, w.maxFeePerGas]), + [ + [7n, 4, 7_000_000n], + [7n, 4, 8_400_000n], + [7n, 4, 10_080_000n], + [7n, 4, 12_096_000n], + [7n, 4, 12_096_000n], + [7n, 4, 12_096_000n], + [8n, 5, 7_000_000n], + ], + ); + assert.ok(logs.some((line) => line.startsWith("released (awaiting safe): nonce=7 ") && line.endsWith(`tx=${txHash(5)}`))); + assert.ok(logs.some((line) => line.startsWith("released (awaiting safe): nonce=8 "))); + assert.deepEqual( + alerts.map((a) => a.split(":")[0]), + ["critical release transaction not mined"], + ); + assert.equal(releaser.inFlight, undefined); }); test("a daily-cap revert pages as blocked when the token amount is above the daily cap itself, and waits quietly otherwise", async () => { @@ -90,3 +339,124 @@ test("a daily-cap revert pages as blocked when the token amount is above the dai ]); assert.equal(releaser.pending.size, 2); }); + +test("a consumption seen only at the latest state keeps the entry until the finality boundary shows it", async () => { + const atBoundary = (args: Record) => args.blockNumber === 100n; + for (const [simulated, consumedLatest] of [ + // Our release, or a peer's, is mined but not yet inside the boundary. + [new Error("execution reverted"), true], + // A simulation served by a node ahead of the ones the reads reach. + [vaultRevert({ errorName: "NonceAlreadyConsumed", args: [7n] }), false], + ] as const) { + const { releaser, alerts, writes } = releaserWith( + { + simulateContract: () => { + throw simulated; + }, + readContract: (args) => !atBoundary(args) && consumedLatest, + }, + [ENTRY], + ); + await drainPending(releaser, CONVERSION_FACTOR, DAILY_CAP); + assert.deepEqual([...releaser.pending.keys()], [7n], `${simulated.name}`); + assert.deepEqual([alerts, writes], [[], []]); + } + const { releaser, logs } = releaserWith( + { + simulateContract: () => { + throw vaultRevert({ errorName: "NonceAlreadyConsumed", args: [7n] }); + }, + readContract: atBoundary, + }, + [ENTRY], + ); + await drainPending(releaser, CONVERSION_FACTOR, DAILY_CAP); + assert.equal(releaser.pending.size, 0, "consumed at the boundary: dropped"); + assert.ok(logs.some((line) => line.startsWith("skip (already consumed): nonce=7 "))); +}); + +test("a blocked release pages once per interval, not once per cycle", async () => { + const { releaser, alerts, advance } = releaserWith( + { + simulateContract: () => { + throw vaultRevert({ errorName: "ExceedsPerReleaseCap", args: [ENTRY.palletAmount * CONVERSION_FACTOR, 1n] }); + }, + }, + [ENTRY], + ); + await drainPending(releaser, CONVERSION_FACTOR, DAILY_CAP); + await drainPending(releaser, CONVERSION_FACTOR, DAILY_CAP); + assert.deepEqual(alerts, [ + `critical release blocked and needs operator action: nonce=7 recipient=${RECIPIENT} amount=${ENTRY.palletAmount} — ExceedsPerReleaseCap; it cannot self-heal`, + ]); + advance(releaser.blockedAlertIntervalMs); + await drainPending(releaser, CONVERSION_FACTOR, DAILY_CAP); + assert.equal(alerts.length, 2); + assert.equal(releaser.pending.size, 1); +}); + +test("a page whose delivery failed goes out again ten minutes later, neither every cycle nor an interval later", async () => { + const { releaser, advance } = releaserWith( + { + simulateContract: () => { + throw vaultRevert({ errorName: "ExceedsPerReleaseCap", args: [ENTRY.palletAmount * CONVERSION_FACTOR, 1n] }); + }, + }, + [ENTRY], + ); + // The webhook is down for the first page only. + const start = releaser.now(); + const pages: { minute: number; delivered: boolean }[] = []; + releaser.alert = async () => { + const delivered = pages.length > 0; + pages.push({ minute: (releaser.now() - start) / 60_000, delivered }); + return delivered; + }; + for (let minute = 0; minute < 15; minute++) { + await drainPending(releaser, CONVERSION_FACTOR, DAILY_CAP); + advance(60_000); + } + assert.deepEqual(pages, [ + { minute: 0, delivered: false }, + { minute: 10, delivered: true }, + ]); +}); + +test("an unexpected release failure pages once per interval for each reason, and again at once when the failure changes", async () => { + let failure: Error = revertedWith("TokenNotSet"); + const { releaser, alerts, advance } = releaserWith( + { + simulateContract: () => { + throw failure; + }, + }, + [ENTRY], + ); + const drain = () => drainPending(releaser, CONVERSION_FACTOR, DAILY_CAP); + for (let cycle = 0; cycle < 5; cycle++) await drain(); + assert.equal(alerts.length, 1); + assert.match(alerts[0], /^critical unexpected release failure: nonce=7 .*reason:\nTokenNotSet\n/); + advance(releaser.blockedAlertIntervalMs); + await drain(); + assert.equal(alerts.length, 2); + // Another revert pages at once. + failure = revertedWith("Pausable: paused"); + await drain(); + assert.equal(alerts.length, 3); + // So does an RPC error, once, whatever endpoint and request its text names. + // Each reason keeps its own interval: the RPC error neither mutes the revert + // nor makes it page again. + for (const id of [1, 2]) { + failure = new HttpRequestError({ url: `http://rpc-${id}`, status: 503, details: `request ${id}` }); + await drain(); + } + failure = revertedWith("Pausable: paused"); + await drain(); + assert.equal(alerts.length, 4); + assert.match(alerts[3], /^critical unexpected release failure: nonce=7 .* — HTTP request failed\./); + failure = vaultRevert({ errorName: "ExceedsPerReleaseCap", args: [ENTRY.palletAmount * CONVERSION_FACTOR, 1n] }); + await drain(); + assert.equal(alerts.length, 5); + assert.match(alerts[4], /^critical release blocked and needs operator action: nonce=7 /); + assert.equal(releaser.pending.size, 1); +}); diff --git a/releaser/src/release.ts b/releaser/src/release.ts index a873f18e5..e34b664f7 100644 --- a/releaser/src/release.ts +++ b/releaser/src/release.ts @@ -1,21 +1,78 @@ /** * The releaser's release attempts: simulate and send `release()` for each - * pending entry, and classify what a failed attempt means. `main.ts` injects - * the clients and the pending set, so the decisions here are unit tested (see - * release.test.ts). + * pending entry, follow a sent transaction until it is mined (re-sent with + * higher fees at its nonce when it is not), and classify what a failed attempt + * means. `main.ts` injects the clients and the pending set, so the decisions + * here are unit tested (see release.test.ts). */ -import type { Account, Chain, PublicClient, Transport, WalletClient } from "viem"; +import { + BaseError, + formatGwei, + NonceTooLowError, + TransactionReceiptNotFoundError, + type Account, + type Chain, + type Hash, + type PublicClient, + type TransactionReceipt, + type Transport, + type WalletClient, +} from "viem"; import type { Severity } from "./alert.js"; -import { classifyReleaseFailure, contractErrorName } from "./checks.js"; +import { classifyReleaseFailure, contractErrorName, failureReason } from "./checks.js"; import { vaultAbi } from "./vaultAbi.js"; +/** op-geth replaces a pending transaction only when the new one raises both + * its fee cap and its tip by at least this many percent. */ +export const MIN_REPLACEMENT_PERCENT = 10; + export interface PendingRelease { nonce: bigint; recipient: `0x${string}`; palletAmount: bigint; } +export interface Fees { + maxFeePerGas: bigint; + maxPriorityFeePerGas: bigint; +} + +/** When, how often and how far a release transaction that is not mined is + * re-sent with higher fees. */ +export interface FeeBumpPolicy { + /** Re-send an attempt still not mined after this long. */ + afterMs: number; + /** Raise both fees by at least this many percent per re-send. */ + percent: number; + /** Re-sends per transaction. */ + max: number; + /** No attempt ever offers a higher maxFeePerGas. */ + maxFeePerGas: bigint; +} + +/** A release transaction sent and not seen mined: the identical call, kept at + * its nonce and re-sent there with higher fees until an attempt is mined. */ +export interface InFlightRelease { + entry: PendingRelease; + call: { + account: Account; + address: `0x${string}`; + abi: typeof vaultAbi; + functionName: "release"; + args: readonly [bigint, `0x${string}`, bigint]; + gas: bigint; + }; + nonce: number; + /** The latest attempt's fees. */ + fees: Fees; + /** Every attempt at `nonce`; any of them may be the one that is mined. */ + hashes: Hash[]; + /** When the latest attempt was sent. */ + sentAtMs: number; + bumps: number; +} + export interface Releaser { publicClient: PublicClient; walletClient: WalletClient; @@ -25,67 +82,225 @@ export interface Releaser { blockedAlertIntervalMs: number; /** nonce -> pending release. Keyed by nonce: the vault consumes a nonce once. */ pending: Map; - /** nonce -> when its "blocked" state was last alerted. A blocked release needs - * a timelocked governance action (>= 48h) to clear, so re-paging it every - * poll would emit thousands of identical alerts and train on-call to ignore - * them; re-alert at a bounded interval instead. */ - blockedAlertedAt: Map; - alert(subject: string, detail: string, severity?: Severity): Promise; + /** nonce -> each page about it, by subject and reason: when it last went + * out, and whether it was delivered. A blocked release needs a timelocked + * governance action (>= 48h) to clear, and an unexpected failure repeats + * every poll as well, so re-paging either every poll would emit thousands + * of identical alerts and train on-call to ignore them; the same page + * re-alerts at a bounded interval instead. */ + alertedAt: Map>; + feeBump: FeeBumpPolicy; + /** The release transaction not seen mined yet, if any. Kept in memory + * only. */ + inFlight?: InFlightRelease; + /** Resolves to whether the alert was delivered. */ + alert(subject: string, detail: string, severity?: Severity): Promise; log(message: string): void; + now(): number; +} + +/** A page whose delivery failed goes out again after this long, or after + * `blockedAlertIntervalMs` when that is shorter. Not on the next cycle: the + * cycle waits on every delivery attempt, up to about 42 s with its retries, + * so a webhook that is down would hold back the releases that can go. */ +const UNDELIVERED_REPAGE_MS = 10 * 60_000; + +/** Page `subject` about the release of `nonce`, unless the same page about it, + * for the same `reason`, went out less than `blockedAlertIntervalMs` ago, or + * less than `UNDELIVERED_REPAGE_MS` ago if its delivery failed. Each reason + * keeps its own interval, so one that comes and goes, such as a transient RPC + * error, neither mutes another nor makes it page again. */ +async function pageThrottled( + releaser: Releaser, + nonce: bigint, + subject: string, + reason: string, + detail: string, +): Promise { + const pages = releaser.alertedAt.get(nonce) ?? new Map(); + const key = `${subject}: ${reason}`; + const last = pages.get(key); + if (last !== undefined) { + const interval = last.delivered + ? releaser.blockedAlertIntervalMs + : Math.min(UNDELIVERED_REPAGE_MS, releaser.blockedAlertIntervalMs); + if (releaser.now() - last.at < interval) return; + } + const delivered = await releaser.alert(subject, detail, "critical"); + releaser.alertedAt.set(nonce, pages.set(key, { at: releaser.now(), delivered })); +} + +const min = (a: bigint, b: bigint) => (a < b ? a : b); +const max = (a: bigint, b: bigint) => (a > b ? a : b); + +/** `fee` raised by `percent`, rounded up and by at least 1 wei. */ +function raised(fee: bigint, percent: number): bigint { + return max(fee + 1n, (fee * BigInt(100 + percent) + 99n) / 100n); +} + +/** The first attempt's fees: the node's estimate, never above the cap. */ +export function cappedFees(estimate: Fees, cap: bigint): Fees { + return { maxFeePerGas: min(estimate.maxFeePerGas, cap), maxPriorityFeePerGas: min(estimate.maxPriorityFeePerGas, cap) }; +} + +/** The fees to re-send an attempt that was not mined with: both raised by the + * policy's percent, and at least to the node's current estimate, never above + * the cap. Undefined when the cap leaves no room for a replacement the pool + * accepts (both fees raised by at least `MIN_REPLACEMENT_PERCENT`). */ +export function bumpedFees(last: Fees, estimate: Fees, policy: FeeBumpPolicy): Fees | undefined { + const next = (fee: bigint, current: bigint) => min(policy.maxFeePerGas, max(raised(fee, policy.percent), current)); + const fees = { + maxFeePerGas: next(last.maxFeePerGas, estimate.maxFeePerGas), + maxPriorityFeePerGas: next(last.maxPriorityFeePerGas, estimate.maxPriorityFeePerGas), + }; + const replaces = + fees.maxFeePerGas >= raised(last.maxFeePerGas, MIN_REPLACEMENT_PERCENT) && + fees.maxPriorityFeePerGas >= raised(last.maxPriorityFeePerGas, MIN_REPLACEMENT_PERCENT); + return replaces ? fees : undefined; +} + +const releaseLabel = (p: PendingRelease) => `nonce=${p.nonce} recipient=${p.recipient} amount=${p.palletAmount}`; + +/** A mined release leaves the pending set only once its nonce reads consumed at + * the finality boundary (see `pruneConsumed`); until then a re-attempt is a + * cheap simulation that classifies as PendingFinality and stays quiet, and a + * release reorged away is still ours to retry. A mined revert carries no + * reason (usually a lost race: a peer released, or the daily allowance went + * between simulation and inclusion); the next cycle's simulation classifies + * it, and the prune drops it if a peer won. */ +function logMined(releaser: Releaser, receipt: TransactionReceipt, label: string): void { + releaser.log( + receipt.status === "success" + ? `released (awaiting ${releaser.baseFinalityTag}): ${label} tx=${receipt.transactionHash}` + : `release reverted on-chain, re-evaluating next cycle: ${label}`, + ); } -/** A release transaction that was mined and reverted. The receipt carries no - * revert data, so the reason is unknowable here; the next cycle's simulation - * classifies it properly (and pruneConsumed drops it if a peer won). */ -class MinedRevert extends Error {} +/** + * Follow the release transaction in flight and say whether it is settled: + * an attempt is mined, its nonce was used by another transaction, or the node + * holds nothing at a nonce below it, so nothing will ever let it be mined. The + * entry is then simulated again like any other. While it is pending, it is + * re-sent `feeBump.afterMs` after the last send: the identical call at the + * same nonce with both fees raised (`bumpedFees`), at most `feeBump.max` + * times. When no bump is left or the cap stops one, it pages (throttled like + * every page about a release) and keeps waiting, sending the latest attempt + * again unchanged each cycle: that is no bump, but it brings back a + * transaction the pool dropped, which nothing else would ever send again. + */ +async function followInFlight(releaser: Releaser, tx: InFlightRelease): Promise { + const { publicClient, walletClient, account, feeBump, log } = releaser; + const label = releaseLabel(tx.entry); + for (const hash of tx.hashes) { + const receipt = await publicClient.getTransactionReceipt({ hash }).catch((error: unknown) => { + if (error instanceof TransactionReceiptNotFoundError) return undefined; + throw error; + }); + if (receipt) { + releaser.inFlight = undefined; + logMined(releaser, receipt, label); + return true; + } + } + const count = (blockTag: "latest" | "pending") => publicClient.getTransactionCount({ address: account.address, blockTag }); + const mined = await count("latest"); + // Its nonce was used (by an attempt whose receipt is not visible yet, or by + // another transaction), or the node holds nothing below it: decide afresh. + if (mined > tx.nonce || (mined < tx.nonce && (await count("pending")) <= mined)) { + releaser.inFlight = undefined; + log( + `release transaction at nonce ${tx.nonce} can no longer be mined there ` + + `(account's next mined nonce ${mined}); re-evaluating: ${label}`, + ); + return true; + } + if (releaser.now() - tx.sentAtMs < feeBump.afterMs) return false; + const fees = tx.bumps < feeBump.max ? bumpedFees(tx.fees, await publicClient.estimateFeesPerGas(), feeBump) : undefined; + if (fees === undefined) { + const reason = tx.bumps >= feeBump.max ? `${tx.bumps} fee bump(s) used up` : "the fee cap stops the next bump"; + await pageThrottled( + releaser, + tx.entry.nonce, + "release transaction not mined", + reason, + `${reason}: transaction nonce ${tx.nonce}, account's next mined nonce ${mined}, ` + + `maxFeePerGas ${formatGwei(tx.fees.maxFeePerGas)} gwei (cap ${formatGwei(feeBump.maxFeePerGas)} gwei); ${label}`, + ); + try { + const hash = await walletClient.writeContract({ ...tx.call, nonce: tx.nonce, ...tx.fees }); + // New only when the last bump was refused: that is the attempt sent again. + if (!tx.hashes.includes(hash)) tx.hashes.push(hash); + } catch (error) { + // A node that still holds it answers that it knows it, and one that + // saw its nonce used that the nonce is too low; viem raises both as + // NonceTooLowError, and neither is a failure. + if (!(error instanceof BaseError && error.walk((cause) => cause instanceof NonceTooLowError))) { + log( + `re-sending the latest attempt at nonce ${tx.nonce} failed: ${label} — ` + + `${error instanceof BaseError ? error.shortMessage : error}`, + ); + } + } + return false; + } + tx.bumps += 1; + tx.fees = fees; + tx.sentAtMs = releaser.now(); + try { + const hash = await walletClient.writeContract({ ...tx.call, nonce: tx.nonce, ...fees }); + tx.hashes.push(hash); + log(`fee bump ${tx.bumps} at nonce ${tx.nonce}: ${label} tx=${hash}`); + } catch (error) { + // The earlier attempts at this nonce stay valid; the next bump goes higher. + const reason = error instanceof BaseError ? error.shortMessage : error; + log(`fee bump ${tx.bumps} at nonce ${tx.nonce} not accepted: ${label} — ${reason}`); + } + return false; +} -/** Try to push each pending release through; classify what comes back. */ +/** Try to push each pending release through; classify what comes back. One + * release transaction is in flight at a time: a later one would only queue + * behind its nonce. */ export async function drainPending(releaser: Releaser, conversionFactor: bigint, dailyCap: bigint): Promise { - const { publicClient, walletClient, account, vaultAddress, baseFinalityTag, pending, blockedAlertedAt, alert, log } = - releaser; + const { publicClient, walletClient, account, vaultAddress, baseFinalityTag, pending, alertedAt, log, feeBump } = releaser; + if (releaser.inFlight !== undefined && !(await followInFlight(releaser, releaser.inFlight))) return; for (const p of [...pending.values()]) { - const label = `nonce=${p.nonce} recipient=${p.recipient} amount=${p.palletAmount}`; + const label = releaseLabel(p); try { - // Simulate first so a still-capped release costs no gas. - const { request } = await publicClient.simulateContract({ + const release = { account, address: vaultAddress, abi: vaultAbi, functionName: "release", args: [p.nonce, p.recipient, p.palletAmount], - }); + } as const; + // Simulate first so a still-capped release costs no gas. + await publicClient.simulateContract(release); // Pad the gas limit: a release writes an ERC20Votes checkpoint for the // vault's vote sink, and whether that is an overwrite or a new entry // depends on the block timestamp the transaction lands in — a state // the estimate cannot know. An exact estimate can run out of gas in // that write (seen in phase 3); doubling it is cheap insurance. - const gasLimit = (await publicClient.estimateContractGas({ - account, - address: vaultAddress, - abi: vaultAbi, - functionName: "release", - args: [p.nonce, p.recipient, p.palletAmount], - })) * 2n; - const txHash = await walletClient.writeContract({ ...request, gas: gasLimit }); - const receipt = await publicClient.waitForTransactionReceipt({ hash: txHash }); - if (receipt.status !== "success") { - throw new MinedRevert(`release reverted on-chain: ${txHash}`); + const call = { ...release, gas: (await publicClient.estimateContractGas(release)) * 2n }; + const nonce = await publicClient.getTransactionCount({ address: account.address, blockTag: "pending" }); + const fees = cappedFees(await publicClient.estimateFeesPerGas(), feeBump.maxFeePerGas); + const sentAtMs = releaser.now(); + const hash = await walletClient.writeContract({ ...call, nonce, ...fees }); + let receipt: TransactionReceipt; + try { + receipt = await publicClient.waitForTransactionReceipt({ hash, timeout: feeBump.afterMs }); + } catch (waitError) { + // Sent, so it may be pending: followed from the next cycle on + // instead of signing another release behind its nonce. + releaser.inFlight = { entry: p, call, nonce, fees, hashes: [hash], sentAtMs, bumps: 0 }; + log( + `release not seen mined yet, following it at nonce ${nonce}: ${label} tx=${hash} — ` + + `${waitError instanceof BaseError ? waitError.shortMessage : waitError}`, + ); + return; } - // Deliberately NOT deleted here: the entry leaves the durable pending - // set only when `pruneConsumed` sees the nonce consumed at the - // finality boundary. Until then a re-attempt is a cheap simulation - // that classifies as PendingFinality and stays quiet — and if the - // release were reorged away, the entry is still ours to retry. - log(`released (awaiting ${baseFinalityTag}): ${label} tx=${txHash}`); + logMined(releaser, receipt, label); } catch (error) { - if (error instanceof MinedRevert) { - // Usually a benign race (a peer released, or the daily allowance - // was consumed between simulation and inclusion). Stay quiet: the - // next cycle prunes it if consumed and re-simulates otherwise, - // which yields a decodable, classifiable reason. - log(`release reverted on-chain, re-evaluating next cycle: ${label}`); - continue; - } const reason = error instanceof Error ? error.message : String(error); // A peer may have consumed the nonce after our simulation. Settle // that race from state before relying on decoded custom errors. @@ -118,28 +333,31 @@ export async function drainPending(releaser: Releaser, conversionFactor: bigint, switch (classifyReleaseFailure(errorName)) { case "done": pending.delete(p.nonce); - blockedAlertedAt.delete(p.nonce); + alertedAt.delete(p.nonce); log(`skip (already consumed): ${label}`); break; case "retry": // Expected while the daily bucket refills — stay quiet. break; - case "blocked": { + case "blocked": // Clearing this needs a timelocked governance action, so the // condition persists for days; page at a bounded interval. - const lastAlerted = blockedAlertedAt.get(p.nonce) ?? 0; - if (Date.now() - lastAlerted >= releaser.blockedAlertIntervalMs) { - blockedAlertedAt.set(p.nonce, Date.now()); - await alert( - "release blocked and needs operator action", - `${label} — ${errorName}; it cannot self-heal`, - "critical", - ); - } + await pageThrottled( + releaser, + p.nonce, + "release blocked and needs operator action", + `${errorName}`, + `${label} — ${errorName}; it cannot self-heal`, + ); break; - } case "unexpected": - await alert("unexpected release failure", `${label} — ${reason}`, "critical"); + await pageThrottled( + releaser, + p.nonce, + "unexpected release failure", + failureReason(error), + `${label} — ${reason}`, + ); break; } } diff --git a/releaser/src/state.test.ts b/releaser/src/state.test.ts index 020671935..4a225b41a 100644 --- a/releaser/src/state.test.ts +++ b/releaser/src/state.test.ts @@ -23,7 +23,10 @@ test("releaser state is atomically persisted and identity-bound", () => { assert.deepEqual(loaded, state); assert.doesNotThrow(() => assertStateIdentity(loaded!, state)); assert.throws(() => assertStateIdentity(loaded!, { ...state, baseChainId: 1 }), /does not match/); - } finally { + assert.throws( + () => assertStateIdentity(loaded!, { ...state, vaultAddress: "0x3333333333333333333333333333333333333333" }), + /state vault 0x1111111111111111111111111111111111111111 does not match configured vault 0x3333/, + ); } finally { rmSync(directory, { recursive: true, force: true }); } });