Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 24 additions & 11 deletions attestor/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,14 @@ point.
| Variable | Meaning |
|---|---|
| `PENDULUM_WS` | WebSocket of your own Pendulum node, e.g. `ws://127.0.0.1:9944` |
| `PENDULUM_GENESIS_HASH` | Genesis hash that node must serve (default Pendulum's, `0x5d3c2986…462c86`); set it only for a local test chain |
| `PENDULUM_GENESIS_HASH` | Genesis hash that node must serve (default Pendulum's, `0x5d3c2986…462c86`); set it only for a local test chain. With `BASE_CHAIN_ID` 8453 (Base mainnet) any other value is refused at startup |
| `BASE_RPC_URL` | Base JSON-RPC endpoint |
| `VAULT_ADDRESS` | MigrationVault address on Base |
| `ATTESTOR_PRIVATE_KEY` | This attestor's signing key (0x-prefixed) |
| `CHECKPOINT_FILE` | Path persisting the last processed block (default `./checkpoint.json`) |
| `START_BLOCK` | First Pendulum block to scan on the very first run |
| `MIN_GAS_BALANCE_WEI` | Low-gas alert threshold (default 0.01 ETH) |
| `ALERT_WEBHOOK_URL` | Optional webhook receiving JSON alerts in Slack incoming-webhook form: `text` holds `[CRITICAL]` or `[WARNING]`, the attestor and the subject, then the detail; `severity`, `subject` and `detail` are also sent as fields. A non-2xx answer, a timeout (10 s) or a connection error is logged as a failed delivery |
| `ALERT_WEBHOOK_URL` | Optional webhook receiving JSON alerts in Slack incoming-webhook form: `text` holds `[CRITICAL]` or `[WARNING]`, the attestor and the subject, then the detail; `severity`, `subject` and `detail` are also sent as fields. A non-2xx answer, a timeout (10 s) or a connection error is logged as a failed delivery and tried again after 2 s and after 10 s more; a 4xx other than 408 and 429 is not retried, as repeating the request cannot change it |
| `HEARTBEAT_URL` | Optional http(s) URL requested (GET) after a finalized head processed without error, at most once a minute; point it at a dead-man's switch that pages when the requests stop. Startup refuses any other value. A failed request is logged (error name and code only, never the URL), never fatal |
| `BASE_CHAIN_ID` | Default 8453 (Base mainnet) |
| `BASE_FINALITY_TAG` | Base confirmation boundary the checkpoint waits for: `safe` (default) or `finalized` |
Expand Down Expand Up @@ -117,10 +117,12 @@ WantedBy=multi-user.target
approval is recorded, a transaction still pending is no longer needed, but
it is followed and bumped the same way until its nonce is used: every later
approval waits behind that nonce, and mined it merely reverts. When no bump
is left, or the cap stops one, it pages `own approval not mined` once (with
the nonce, the account's next mined nonce, the fees and whether the migration
is already resolved) and keeps waiting, since later approvals would queue
behind that nonce anyway. Whenever it stops bumping, it re-sends the last
is left, or the cap stops one, it pages `own approval not mined` (with the
nonce, the account's next mined nonce, the fees and whether the migration is
already resolved) and keeps waiting, since later approvals would queue
behind that nonce anyway. While it stays stuck, the page repeats 10, 20 and
40 minutes later and so on, at most every 4 hours, marked `still not mined
(page N)`. Whenever it stops bumping, it re-sends the last
attempt unchanged, so a node that dropped the transaction while still
counting its nonce gets it back instead of leaving a gap. An approval that an
RPC error interrupted keeps its transaction, also one whose send got no
Expand All @@ -129,8 +131,15 @@ WantedBy=multi-user.target
counting `FEE_BUMP_AFTER_SECONDS` from its last send. A
first attempt whose nonce is taken (mined already, or held by a transaction
this process did not send) moves on to the next nonce at once, at most five
in a row, without spending a bump. A node refusing a send for its price, an
underpriced replacement or a full pool is retried at the next bump instead
in a row, without spending a bump. An approval that cannot be mined because
the node holds nothing at a nonce below it (a transaction of ours dropped by
a reorg, or a skipped one that left the pool) is sent again at that first
empty nonce, or the next approval is, so nothing waits on a nonce no
transaction will fill. It moves there, keeping its bumps and pages, only
once the node takes it there: a count from a lagging backend can show a
nonce empty that a transaction does hold, and a refused approval stays at
its nonce, bumped and paged as before. A node refusing a send for its
price, an underpriced replacement or a full pool is retried at the next bump instead
of being fatal. Transactions in flight are kept in memory only: after a
restart, an approval whose transaction is still pending is sent again at the
next nonce, so a pending transaction stuck below the cap has to be replaced
Expand All @@ -146,13 +155,17 @@ WantedBy=multi-user.target
with no checkpoint and `START_BLOCK` before the oldest unreleased migration;
a re-added key lowers `lastProcessedBlock` to such a block and sets
`attestorGeneration` to the current value, keeping every other field.
Checkpoints written before these fields existed are bound to the first key
that starts on them.
A checkpoint without these fields, or with only one of them, was edited or
not written by this daemon and is refused; the error says how to recover.
- The daemon verifies at startup that its address is in the vault's attestor
set, that `BASE_RPC_URL` serves `BASE_CHAIN_ID` (`eth_chainId`, which the
checkpoint's Base chain must match too) and that `PENDULUM_WS` serves
`PENDULUM_GENESIS_HASH`, and refuses to run otherwise. These checks run on
every start, the first one without a checkpoint included.
every start, the first one without a checkpoint included. While running, it
re-reads its attestor membership and generation on every finalized head and
exits with `fatal error, exiting` when either changed (a key removed, or
removed and re-added): the next start then refuses the checkpoint until it
is rewound as described above.
- After a Pendulum **runtime upgrade**, verify event decoding against the new
metadata on a staging node before letting the fleet advance past the
upgrade block.
32 changes: 29 additions & 3 deletions attestor/src/alert.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,24 +42,50 @@ test("a webhook error status, a timeout or a dead relay is a failed delivery, an
for (const status of [500, 429, 404]) {
const { url, server } = await webhook(status);
try {
assert.equal(await postAlert(url, { subject: "s" }), false, `HTTP ${status}`);
assert.equal(await postAlert(url, { subject: "s" }, 10_000, []), false, `HTTP ${status}`);
} finally {
server.close();
}
}
const hanging = await webhook(null);
try {
assert.equal(await postAlert(hanging.url, { subject: "s" }, 200), false, "timeout");
assert.equal(await postAlert(hanging.url, { subject: "s" }, 200, []), false, "timeout");
} finally {
hanging.server.closeAllConnections();
hanging.server.close();
}
const closed = await webhook(200);
closed.server.close();
assert.equal(await postAlert(closed.url, { subject: "s" }), false, "connection refused");
assert.equal(await postAlert(closed.url, { subject: "s" }, 10_000, []), false, "connection refused");
assert.equal(logged.mock.callCount(), 5);
});

test("a failed delivery is tried again after each wait, a request the relay refused is not", async (t) => {
t.mock.method(console, "error", () => {});
let answers: number[] = [];
let posts = 0;
const server = createServer((request, response) => {
request.resume();
request.on("end", () => response.writeHead(answers[posts++]).end());
});
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
const url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
try {
for (const [statuses, delivered] of [
[[503, 429, 200], true],
[[500, 502, 504], false],
[[404], false],
] as const) {
answers = [...statuses];
posts = 0;
assert.equal(await postAlert(url, { subject: "s" }, 10_000, [10, 10]), delivered, `${statuses}`);
assert.equal(posts, statuses.length, `${statuses}`);
}
} finally {
server.close();
}
});

test("a heartbeat requests its URL, and a failing endpoint is logged without the URL, never thrown", async (t) => {
const logged = t.mock.method(console, "error", () => {});
const seen: string[] = [];
Expand Down
46 changes: 32 additions & 14 deletions attestor/src/alert.ts
Original file line number Diff line number Diff line change
Expand Up @@ -66,24 +66,42 @@ export function throttledHeartbeat(
};
}

/** A failed alert delivery is tried again after each of these waits: a relay
* that is briefly down or rate-limiting must not swallow a page. */
const ALERT_RETRY_DELAYS_MS = [2_000, 10_000];

/**
* POST one alert to the webhook and report whether it was delivered. `fetch`
* resolves on any HTTP answer, so a non-2xx status is checked explicitly: a
* relay answering 4xx/5xx, a timeout and a connection failure are all logged
* and count as not delivered.
* and count as not delivered. Each is tried again after the next of
* `retryDelaysMs`, except a 4xx other than 408 and 429: the relay refused the
* request itself (a wrong URL, a rejected payload), which repeating it cannot
* change.
*/
export async function postAlert(url: string, body: unknown, timeoutMs = 10_000): Promise<boolean> {
try {
const response = await fetch(url, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(body),
signal: AbortSignal.timeout(timeoutMs),
});
if (response.ok) return true;
console.error(`alert webhook failed: HTTP ${response.status}`);
} catch (webhookError) {
console.error("alert webhook failed", webhookError);
export async function postAlert(
url: string,
body: unknown,
timeoutMs = 10_000,
retryDelaysMs = ALERT_RETRY_DELAYS_MS,
): Promise<boolean> {
for (let attempt = 0; ; attempt++) {
let status: number | undefined;
try {
const response = await fetch(url, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(body),
signal: AbortSignal.timeout(timeoutMs),
});
if (response.ok) return true;
status = response.status;
console.error(`alert webhook failed: HTTP ${response.status}`);
} catch (webhookError) {
console.error("alert webhook failed", webhookError);
}
const refused = status !== undefined && status < 500 && status !== 408 && status !== 429;
if (refused || attempt === retryDelaysMs.length) return false;
await new Promise((resolve) => setTimeout(resolve, retryDelaysMs[attempt]));
}
return false;
}
Loading
Loading