Skip to content

fix(attestor,releaser): Keep endpoint URLs out of logs and limit repeated pages - #597

Merged
ebma merged 10 commits into
mainfrom
fix/daemon-alert-output
Nov 12, 2026
Merged

ebma merged 10 commits into
mainfrom
fix/daemon-alert-output

Conversation

@ebma

@ebma ebma commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

What changes

  • Console output, attestor and releaser. All console output now goes through the URL redaction that alerts already applied to their webhook payload. Log lines show <url> wherever they quoted an endpoint: the alert lines, logged errors (viem's error texts quote the RPC URL) and polkadot.js's WebSocket disconnect messages. The redaction now also covers ws(s) URLs, and it moves into the alert.ts module both services share. An uncaught exception or unhandled rejection, which Node prints itself, now goes through the same console; the process still exits with status 1.
  • Endpoint settings. BASE_RPC_URL, PENDULUM_WS and ALERT_WEBHOOK_URL must carry their scheme (http(s), and ws(s) for the Pendulum node). Startup refuses any other value without echoing it, as it already did for HEARTBEAT_URL. Such a value could not work, but it used to fail later with errors that quote it, and a webhook URL without a scheme let the daemon run with every alert failing delivery.
  • Releaser pages. releaser cycle failed pages on the first failure, then at most every 10 minutes; the failures in between are logged. It used to page on every poll while an endpoint stayed unreachable, and each page delayed the next cycle by its delivery attempts. The heartbeat still goes out only after a completed cycle. A failed Multicall batch is now logged instead of paged on every cycle: the per-nonce reads that follow it complete the cycle, and if they fail too, the cycle fails and pages.
  • Startup during an RPC outage. A Base read that failed at startup ended the process with a critical startup failed page, so a daemon restarted during an outage paged again on every restart (about every 11 s with the systemd example). Both daemons now retry their startup reads in the process and page the way they do while running: the releaser at most every 10 minutes, shared with its cycle failures, and the attestor at most once a minute, shared with its transient failures. Nothing is processed before these reads, so no event can be skipped. A wrong chain, a key that is not an attestor and unreadable state stay fatal.
  • Docs. Both READMEs describe the redaction, the scheme check and the startup retry; the releaser README also covers the page interval and the Multicall fallback, and the attestor docs now say Base mines a block every 200 ms.

Tests

  • alert.test.ts (both services): every console method shows <url> in place of http(s) and ws(s) URLs, also inside a logged error.
  • config.test.ts (both services): an endpoint setting without the right scheme is refused without echoing it.
  • attestor/src/main.test.ts (new): against a Base endpoint that refuses connections, the attestor keeps running, pages once and logs the next failed read.
  • releaser/src/main.test.ts: the releaser runs against a local RPC.
    • A failed startup read pages once, is retried, and startup completes once the endpoint answers.
    • When every cycle fails, it pages once and logs the next two failures.
    • Its output never contains the endpoint's path.
    • Removing the startup retry, the page interval or the console redaction makes these tests fail, and each test stops the releaser after 20 s rather than hang.
  • npm test: attestor 99/99, releaser 40/40.

ebma added 10 commits November 12, 2026 09:04
Alerts replaced URLs with <url> only in what they post to the webhook.
Their console line, other logged errors and library log lines still
printed endpoint URLs whole: viem's error texts quote the RPC URL, and
polkadot.js logs the WebSocket endpoint on every disconnect.

Route all console output through the same redaction, installed once at
startup, and extend it to ws(s) URLs. The redaction moves into the
alert module both services share.
Every failed cycle paged "releaser cycle failed", so an endpoint that
stays unreachable paged on every poll, and each page held the next
cycle back for its delivery attempts.

Page the first failure, then at most every 10 minutes, the same way the
attestor limits its transient-failure page; the failures in between are
logged. A dead-man's switch on HEARTBEAT_URL still notices a lasting
failure, as the heartbeat goes out only after a completed cycle.
…heme

BASE_RPC_URL, PENDULUM_WS and ALERT_WEBHOOK_URL were used as given. A
value without its scheme cannot work, but it failed later and echoed
the value: viem's and polkadot.js's errors quote it, and the URL
redaction cannot recognise a URL without a scheme. A webhook URL
without one let the daemon run with every alert failing delivery.

Check them at startup like HEARTBEAT_URL, without echoing the value:
http(s) for the RPC and webhook URLs, ws(s) for the Pendulum node.
…console

Node prints an uncaught exception or unhandled rejection itself, past
the console, so the URL redaction did not reach it. Every async path
has a handler today; this covers a later one. The process still exits
with status 1, as it does without the handler.
A failed Multicall batch paged on every cycle, once per batch, although
the bounded per-nonce reads that follow it still complete the cycle.
Log it instead: if those reads fail too, the cycle fails and pages.
The test waited for three failed cycles or the releaser's exit, with
nothing to stop the releaser, so a regression hung the test run, and
the CI job with it. Stop the releaser after 20 s, as the startup test
does, so the assertions run and fail with its output.
… redaction test

The test only logged https and wss URLs through log, error and warn, so
a redaction that missed http, ws, info or debug still passed it.
… of exiting

A Base read that failed at startup ended the process with a critical
"startup failed" page, so a daemon restarted during an RPC outage paged
again on every restart: about every 11 s with the systemd example.

Retry the startup reads in the process instead, paging the way the
running daemon does during the same outage: the releaser shares the
10-minute interval of its cycle-failure page, the attestor the one-minute
interval of its transient-failure page, and every failure is still
logged. Nothing is processed before these reads, so no event can be
skipped. A wrong chain, a key that is not an attestor and unreadable
state stay fatal.
@ebma ebma changed the title fix(attestor,releaser): Redact endpoint URLs from console output and limit cycle-failure pages fix(attestor,releaser): Keep endpoint URLs out of logs and limit repeated pages Nov 12, 2026
@ebma
ebma merged commit 27a2771 into main Nov 12, 2026
6 checks passed
@ebma
ebma deleted the fix/daemon-alert-output branch November 12, 2026 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant