Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions attestor/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,15 @@ point.
| `BASE_FINALITY_TIMEOUT_MS` | How long a block's approvals may stay outside that boundary before an alert + idempotent re-submission (default 15 min for `safe`, 45 min for `finalized`) |
| `HEAD_STALL_ALERT_MS` | Page when no finalized Pendulum head has arrived for this long (default 5 min): the daemon is push-driven, so a node that stops finalizing would otherwise idle undetected |
| `MISSED_APPROVAL_ALERT_COUNT` | Page after this many consecutive migrations resolved on-chain without this attestor's own approval landing, and again every as many more (default 3, a whole number of at least 1) |
| `FEE_BUMP_AFTER_SECONDS` | Re-send an own approval that is still not mined after this long (default 60: Base mines a block every 2 s, so this means stuck, not slow) |
| `FEE_BUMP_AFTER_SECONDS` | Re-send an own approval that is still not mined after this long (default 60: Base mines a block every 200 ms, so this means stuck, not slow) |
| `FEE_BUMP_PERCENT` | Raise both fees by at least this many percent per re-send (default 20; a whole number of at least 10, the minimum a node accepts for a replacement) |
| `FEE_BUMP_MAX` | Re-sends per approval before the `own approval not mined` page (default 3) |
| `MAX_FEE_PER_GAS_GWEI` | No approval ever offers a higher `maxFeePerGas` (default 1, about 200 times Base's usual base fee; one approval uses at most about 300k gas) |

Log lines and alerts show `<url>` in place of every http(s) and ws(s) URL:
endpoint URLs often carry an access key. Startup refuses an endpoint setting
without the right scheme, and does not echo it.

## Run

```sh
Expand Down Expand Up @@ -161,7 +165,10 @@ WantedBy=multi-user.target
set, that `BASE_RPC_URL` serves `BASE_CHAIN_ID` (`eth_chainId`, which the
checkpoint's Base chain must match too) and that `PENDULUM_WS` serves
`PENDULUM_GENESIS_HASH`, and refuses to run otherwise. These checks run on
every start, the first one without a checkpoint included. While running, it
every start, the first one without a checkpoint included. A Base read they
need that fails, as during an RPC outage, is tried again every 5 s and pages
`Base read failed at startup, retrying` at most once a minute, so a restart
during an outage waits for the endpoint instead of exiting. While running, it
re-reads its attestor membership and generation on every finalized head and
exits with `fatal error, exiting` when either changed (a key removed, or
removed and re-added): the next start then refuses the checkpoint until it
Expand Down
15 changes: 14 additions & 1 deletion attestor/src/alert.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { createServer, type Server } from "node:http";
import type { AddressInfo } from "node:net";
import { test } from "node:test";
import { inspect } from "node:util";
import { alertText, parseHeartbeatUrl, pingHeartbeat, postAlert, throttledHeartbeat } from "./alert.js";
import { alertText, parseHeartbeatUrl, pingHeartbeat, postAlert, redactConsole, throttledHeartbeat } from "./alert.js";

/** A local webhook answering every POST with `status`, or never when null. */
async function webhook(status: number | null): Promise<{ url: string; server: Server }> {
Expand Down Expand Up @@ -166,3 +166,16 @@ test("heartbeats go out at most once a minute, however often they are due", asyn
assert.deepEqual(seen, ["/ping", "/ping"]);
assert.equal(throttledHeartbeat(undefined)(), undefined, "nothing without a URL");
});

test("console output shows <url> in place of each endpoint, also inside a logged error", (t) => {
const mocks = (["debug", "info", "log", "warn", "error"] as const).map((method) => t.mock.method(console, method, () => {}));
redactConsole();
console.log("scanning %s", "https://base.example/v2/secret-key");
console.error("cycle failed", new Error("HTTP request failed.\n\nURL: https://base.example/v2/secret-key"));
console.warn("API-WS: disconnected from wss://pendulum.example/ws?apikey=secret-key: 1006");
console.info("API-WS: disconnected from ws://127.0.0.1:9944/secret-key: 1006");
console.debug("posting to http://hooks.example/secret-key");
const lines = mocks.flatMap((mock) => mock.mock.calls.map((call) => `${call.arguments[0]}`));
assert.equal(lines.length, 5);
for (const line of lines) assert.ok(line.includes("<url>") && !line.includes("secret-key"), line);
});
35 changes: 30 additions & 5 deletions attestor/src/alert.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,23 @@
import { format } from "node:util";

export type Severity = "critical" | "warning";

/** Strip URLs before anything leaves the process: endpoints commonly embed API
* keys, and error texts quote them verbatim (viem's the RPC URL, polkadot.js
* the WebSocket one). */
export function redact(text: string): string {
return text.replace(/(?:https?|wss?):\/\/[^\s"')]+/gi, "<url>");
}

/** Pass all console output through `redact`, called once at startup: errors
* are logged whole, and libraries log on their own. */
export function redactConsole(): void {
for (const method of ["debug", "info", "log", "warn", "error"] as const) {
const write = console[method].bind(console);
console[method] = (...args: unknown[]) => write(redact(format(...args)));
}
}

/** Longer details are cut, well inside what Slack shows of one message. */
const MAX_TEXT = 3000;

Expand All @@ -19,20 +37,27 @@ export function alertText(severity: Severity, source: string, subject: string, d
* refuse more than a few requests a minute. */
const HEARTBEAT_INTERVAL_MS = 60_000;

/** HEARTBEAT_URL, checked once at startup: unset, or an http(s) URL. The value
* is never echoed, as it usually holds a token. */
export function parseHeartbeatUrl(value: string | undefined): string | undefined {
if (!value) return undefined;
/** The URL setting `name`, checked once at startup: an http(s) URL, or a ws(s)
* one when `scheme` says so. The value is never echoed, as it usually holds a
* token. */
export function checkUrl(name: string, value: string, scheme: "http" | "ws" = "http"): string {
let protocol: string | undefined;
try {
protocol = new URL(value).protocol;
} catch {
// Not a URL at all: refused below.
}
if (protocol !== "http:" && protocol !== "https:") throw new Error("invalid HEARTBEAT_URL: expected an http(s) URL");
if (protocol !== `${scheme}:` && protocol !== `${scheme}s:`) {
throw new Error(`invalid ${name}: expected ${scheme === "ws" ? "a ws(s)" : "an http(s)"} URL`);
}
return value;
}

/** HEARTBEAT_URL, checked once at startup: unset, or an http(s) URL. */
export function parseHeartbeatUrl(value: string | undefined): string | undefined {
return value ? checkUrl("HEARTBEAT_URL", value) : undefined;
}

/**
* Request the URL of an external dead-man's switch, which pages once these
* requests stop. A failed request is logged, never thrown: the missing
Expand Down
10 changes: 10 additions & 0 deletions attestor/src/config.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,16 @@ test("on Base mainnet the expected genesis can only be Pendulum's", () => {
assert.equal(loadConfig({ PENDULUM_GENESIS_HASH: pendulum }, "config.pendulumGenesisHash").stdout, pendulum);
});

test("an endpoint URL needs the right scheme, and a bad one is refused without echoing it", () => {
assert.equal(loadConfig({ ALERT_WEBHOOK_URL: "https://hooks.example/secret-key" }, "config.alertWebhookUrl").stdout, "https://hooks.example/secret-key");
for (const [name, value] of Object.entries({ PENDULUM_WS: "https://node.example/secret-key", BASE_RPC_URL: "base.example/secret-key", ALERT_WEBHOOK_URL: "hooks.example/secret-key" })) {
const { status, stderr } = loadConfig({ [name]: value });
assert.notEqual(status, 0, name);
assert.match(stderr, new RegExp(`invalid ${name}`), name);
assert.doesNotMatch(stderr, /secret-key/, name);
}
});

test("HEARTBEAT_URL must be an http(s) URL, and a bad one is refused without echoing it", () => {
assert.equal(loadConfig({ HEARTBEAT_URL: "https://hc-ping.com/secret-token" }, "config.heartbeatUrl").stdout, "https://hc-ping.com/secret-token");
const { status, stderr } = loadConfig({ HEARTBEAT_URL: "hc-ping.com/secret-token" });
Expand Down
10 changes: 5 additions & 5 deletions attestor/src/config.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { parseGwei } from "viem";
import { parseHeartbeatUrl } from "./alert.js";
import { checkUrl, parseHeartbeatUrl } from "./alert.js";
import { MIN_REPLACEMENT_PERCENT } from "./approval.js";

function required(name: string): string {
Expand Down Expand Up @@ -59,12 +59,12 @@ export const config = {
* Never point this at a public RPC: the attestor would inherit its honesty.
* The node must retain historical state (`--state-pruning archive`), or any
* daemon outage longer than the pruning horizon wedges the catch-up. */
pendulumWs: required("PENDULUM_WS"),
pendulumWs: checkUrl("PENDULUM_WS", required("PENDULUM_WS"), "ws"),
/** The genesis that node must serve; startup refuses any other chain. A
* local test chain sets its own, never on Base mainnet. */
pendulumGenesisHash: genesisHash(),
/** Base JSON-RPC endpoint. */
baseRpcUrl: required("BASE_RPC_URL"),
baseRpcUrl: checkUrl("BASE_RPC_URL", required("BASE_RPC_URL")),
/** MigrationVault contract address on Base. */
vaultAddress: required("VAULT_ADDRESS") as `0x${string}`,
/** This attestor's transaction-signing key (0x-prefixed, 32 bytes).
Expand All @@ -78,7 +78,7 @@ export const config = {
/** Alert when the gas wallet drops below this balance (wei). */
minGasBalanceWei: BigInt(process.env.MIN_GAS_BALANCE_WEI ?? "10000000000000000"), // 0.01 ETH
/** Optional webhook that receives JSON alerts (low gas, fatal errors). */
alertWebhookUrl: process.env.ALERT_WEBHOOK_URL,
alertWebhookUrl: process.env.ALERT_WEBHOOK_URL ? checkUrl("ALERT_WEBHOOK_URL", process.env.ALERT_WEBHOOK_URL) : undefined,
/** Optional http(s) URL requested after a finalized head processed without
* error, at most once a minute: a dead-man's switch pages when the
* requests stop. */
Expand Down Expand Up @@ -108,7 +108,7 @@ export const config = {
* nonce with both fees raised by `percent` (at least the 10 % a replacement
* needs, and at least to the current estimate), at most `max` times and
* never above `maxFeePerGas`; then it pages once. Base mines a block every
* 2 s and an approval priced at the estimate lands in the next one or two,
* 200 ms and an approval priced at the estimate lands in the next one or two,
* so 60 s means stuck, not slow. Since each bump also catches up with the
* estimate, one normally suffices; when three do not, the price is not the
* problem and the page (about 4 min after the first send) goes to a person.
Expand Down
53 changes: 53 additions & 0 deletions attestor/src/main.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
import assert from "node:assert/strict";
import { spawn } from "node:child_process";
import { once } from "node:events";
import { createServer } from "node:http";
import type { AddressInfo } from "node:net";
import { test } from "node:test";
import { fileURLToPath } from "node:url";

test("a failed Base read at startup is retried, paging once, instead of exiting", { timeout: 30_000 }, async () => {
// A Base endpoint that refuses connections, as during an outage.
const closed = createServer().listen(0, "127.0.0.1");
await once(closed, "listening");
const port = (closed.address() as AddressInfo).port;
closed.close();
const child = spawn(process.execPath, [fileURLToPath(new URL("./main.js", import.meta.url))], {
env: {
PATH: process.env.PATH,
PENDULUM_WS: "ws://127.0.0.1:9944",
BASE_RPC_URL: `http://127.0.0.1:${port}/v2/secret-key`,
VAULT_ADDRESS: "0x1111111111111111111111111111111111111111",
ATTESTOR_PRIVATE_KEY: `0x${"11".repeat(32)}`,
},
});
let stdout = "";
let stderr = "";
// A regression must fail the test, not leave the attestor running.
const kill = setTimeout(() => child.kill(), 20_000);
try {
// Until the second failed read, paged or logged, or an early exit.
await new Promise((resolve) => {
const seen = () => {
if (`${stdout}${stderr}`.split("Base read failed at startup").length > 2) resolve(undefined);
};
child.stdout.on("data", (chunk) => {
stdout += chunk;
seen();
});
child.stderr.on("data", (chunk) => {
stderr += chunk;
seen();
});
child.on("exit", resolve);
});
const all = `${stdout}${stderr}`;
assert.equal(child.exitCode, null, all);
assert.equal(stderr.split("ALERT: Base read failed at startup").length, 2, all);
assert.equal(stdout.split("Base read failed at startup").length, 2, all);
assert.ok(!all.includes("secret-key"), all);
} finally {
clearTimeout(kill);
child.kill();
}
});
67 changes: 47 additions & 20 deletions attestor/src/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
import { ApiPromise, WsProvider } from "@polkadot/api";
import { createPublicClient, defineChain, http } from "viem";
import { privateKeyToAccount } from "viem/accounts";
import { alertText, postAlert, throttledHeartbeat, type Severity } from "./alert.js";
import { alertText, postAlert, redact, redactConsole, throttledHeartbeat, type Severity } from "./alert.js";
import {
advanceCheckpoint,
assertSignerUnchanged,
Expand All @@ -44,9 +44,20 @@ import {
type Checkpoint,
} from "./state.js";

/** Repeated transient failures (a sustained RPC outage) page at most this
* often; every occurrence is still logged. */
redactConsole();
// Node prints an uncaught error or unhandled rejection itself, past the
// console: print it redacted instead, then exit as Node would.
process.on("uncaughtException", (error) => {
console.error(`${new Date().toISOString()} fatal error`, error);
process.exit(1);
});

/** Repeated failures that are retried (a sustained RPC outage, at startup or
* on a finalized head) page at most this often; every occurrence is still
* logged. */
const TRANSIENT_ALERT_INTERVAL_MS = 60_000;
/** A Base read at startup that fails is tried again after this long. */
const STARTUP_RETRY_MS = 5_000;

const baseChain = defineChain({
id: config.baseChainId,
Expand All @@ -62,12 +73,6 @@ function log(message: string, extra?: unknown): void {
console.log(`${new Date().toISOString()} ${message}`, extra ?? "");
}

/** Strip URLs before anything leaves the process: RPC endpoints commonly embed
* API keys, and viem error texts quote the endpoint verbatim. */
function redact(text: string): string {
return text.replace(/https?:\/\/[^\s"')]+/gi, "<url>");
}

async function alert(subject: string, detail: unknown, severity: Severity = "warning"): Promise<void> {
console.error(`${new Date().toISOString()} ALERT: ${subject}`, detail);
if (!config.alertWebhookUrl) return;
Expand All @@ -82,6 +87,34 @@ async function alert(subject: string, detail: unknown, severity: Severity = "war
});
}

let lastRetryAlertMs = 0;

/** Page a failure that will be retried, or only log it if such a page went out
* less than TRANSIENT_ALERT_INTERVAL_MS ago. */
async function alertRetried(subject: string, error: unknown): Promise<void> {
if (Date.now() - lastRetryAlertMs >= TRANSIENT_ALERT_INTERVAL_MS) {
lastRetryAlertMs = Date.now();
await alert(subject, error);
} else {
log(subject, error);
}
}

/** A Base read at startup, tried again every STARTUP_RETRY_MS until it answers:
* a restart during an RPC outage waits for the endpoint, paging as a running
* attestor does, instead of exiting and paging again on every restart.
* Nothing is processed before these reads, so no event can be skipped. */
async function startupRead<T>(read: () => Promise<T>): Promise<T> {
for (;;) {
try {
return await read();
} catch (error) {
await alertRetried("Base read failed at startup, retrying", error);
await new Promise((resolve) => setTimeout(resolve, STARTUP_RETRY_MS));
}
}
}

/** Extract MigrationInitiated events from one finalized Pendulum block. */
async function migrationEventsInBlock(api: ApiPromise, blockNumber: number): Promise<MigrationEvent[]> {
const blockHash = await api.rpc.chain.getBlockHash(blockNumber);
Expand Down Expand Up @@ -125,13 +158,13 @@ async function checkGasBalance(): Promise<void> {
}

async function main(): Promise<void> {
const baseChainId = await publicClient.getChainId();
const baseChainId = await startupRead(() => publicClient.getChainId());
assertBaseChain(baseChainId, config.baseChainId);
if (!(await attestor.chain.isAttestor())) {
if (!(await startupRead(() => attestor.chain.isAttestor()))) {
throw new Error(`${account.address} is not an attestor of ${config.vaultAddress}`);
}
const attestorGeneration = await attestor.chain.attestorGeneration();
await checkGasBalance();
const attestorGeneration = await startupRead(() => attestor.chain.attestorGeneration());
await startupRead(checkGasBalance);

const api = await ApiPromise.create({ provider: new WsProvider(config.pendulumWs) });
const pendulumGenesisHash = api.genesisHash.toHex();
Expand All @@ -158,7 +191,6 @@ async function main(): Promise<void> {
* checkpoint has not yet passed. Bounded by the Base finality lag. */
const unconfirmed: UnconfirmedBlock[] = [];
let processedThrough = checkpoint.lastProcessedBlock;
let lastTransientAlertMs = 0;
const heartbeat = throttledHeartbeat(config.heartbeatUrl);

let lastHeadAt = Date.now();
Expand All @@ -183,12 +215,7 @@ async function main(): Promise<void> {
// Not a statement about any event — resume from the in-memory
// position on the next finalized head. Sustained outages page
// at a bounded rate instead of once per head.
if (Date.now() - lastTransientAlertMs >= TRANSIENT_ALERT_INTERVAL_MS) {
lastTransientAlertMs = Date.now();
await alert("transient RPC failure, retrying on the next finalized head", error);
} else {
log("transient RPC failure, retrying on the next finalized head", error);
}
await alertRetried("transient RPC failure, retrying on the next finalized head", error);
return;
}
// PRD A5: never skip an event silently. Alert and exit; the process
Expand Down
14 changes: 12 additions & 2 deletions releaser/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@ consumed nonce and drops it.
| `FEE_BUMP_MAX` | Re-sends per transaction before the `release transaction not mined` page (default 3) |
| `MAX_FEE_PER_GAS_GWEI` | No release transaction ever offers a higher `maxFeePerGas` (default 1) |

Log lines and alerts show `<url>` in place of every http(s) and ws(s) URL:
endpoint URLs often carry an access key. Startup refuses an endpoint setting
without the right scheme, and does not echo it.

## Run

```sh
Expand All @@ -58,7 +62,12 @@ npm install && npm run build && npm start

Each cycle it scans new `ReleasePending` logs, drops nonces the vault has
already consumed, then simulates and submits `release()` for the rest.
Simulating first means a still-capped release costs no gas.
Simulating first means a still-capped release costs no gas. A cycle that
fails, on an RPC error for instance, is logged and runs again after
`POLL_INTERVAL_MS`, and so does a failed Base read at startup: a restart during
an outage waits for the endpoint instead of exiting. These failures page
(`releaser cycle failed`, `releaser startup read failed, retrying`) at most
once every 10 minutes.

Failures are classified rather than treated alike:

Expand Down Expand Up @@ -96,7 +105,8 @@ trusting unsafe logs or losing pending work. Malformed state is fatal, and so is
an endpoint that does not serve `BASE_CHAIN_ID`, checked before the state is
read on every start. Multicall
is disabled permanently only when an on-chain code check proves it absent; a
temporary provider error falls back in bounded batches for that cycle.
temporary provider error falls back in bounded batches for that cycle and is
logged, not paged.

**Known limitation:** the pending set is driven by `ReleasePending`, which the
vault emits when a threshold is crossed inside `approve()`. A payload made
Expand Down
Loading
Loading