Repository navigation
fix(attestor,releaser): Keep endpoint URLs out of logs and limit repeated pages - #597
Merged
Merged
Conversation
Alerts replaced URLs with <url> only in what they post to the webhook. Their console line, other logged errors and library log lines still printed endpoint URLs whole: viem's error texts quote the RPC URL, and polkadot.js logs the WebSocket endpoint on every disconnect. Route all console output through the same redaction, installed once at startup, and extend it to ws(s) URLs. The redaction moves into the alert module both services share.
Every failed cycle paged "releaser cycle failed", so an endpoint that stays unreachable paged on every poll, and each page held the next cycle back for its delivery attempts. Page the first failure, then at most every 10 minutes, the same way the attestor limits its transient-failure page; the failures in between are logged. A dead-man's switch on HEARTBEAT_URL still notices a lasting failure, as the heartbeat goes out only after a completed cycle.
…heme BASE_RPC_URL, PENDULUM_WS and ALERT_WEBHOOK_URL were used as given. A value without its scheme cannot work, but it failed later and echoed the value: viem's and polkadot.js's errors quote it, and the URL redaction cannot recognise a URL without a scheme. A webhook URL without one let the daemon run with every alert failing delivery. Check them at startup like HEARTBEAT_URL, without echoing the value: http(s) for the RPC and webhook URLs, ws(s) for the Pendulum node.
…console Node prints an uncaught exception or unhandled rejection itself, past the console, so the URL redaction did not reach it. Every async path has a handler today; this covers a later one. The process still exits with status 1, as it does without the handler.
A failed Multicall batch paged on every cycle, once per batch, although the bounded per-nonce reads that follow it still complete the cycle. Log it instead: if those reads fail too, the cycle fails and pages.
The test waited for three failed cycles or the releaser's exit, with nothing to stop the releaser, so a regression hung the test run, and the CI job with it. Stop the releaser after 20 s, as the startup test does, so the assertions run and fail with its output.
… redaction test The test only logged https and wss URLs through log, error and warn, so a redaction that missed http, ws, info or debug still passed it.
… of exiting A Base read that failed at startup ended the process with a critical "startup failed" page, so a daemon restarted during an RPC outage paged again on every restart: about every 11 s with the systemd example. Retry the startup reads in the process instead, paging the way the running daemon does during the same outage: the releaser shares the 10-minute interval of its cycle-failure page, the attestor the one-minute interval of its transient-failure page, and every failure is still logged. Nothing is processed before these reads, so no event can be skipped. A wrong chain, a key that is not an attestor and unreadable state stay fatal.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
<url>wherever they quoted an endpoint: the alert lines, logged errors (viem's error texts quote the RPC URL) and polkadot.js's WebSocket disconnect messages. The redaction now also covers ws(s) URLs, and it moves into thealert.tsmodule both services share. An uncaught exception or unhandled rejection, which Node prints itself, now goes through the same console; the process still exits with status 1.BASE_RPC_URL,PENDULUM_WSandALERT_WEBHOOK_URLmust carry their scheme (http(s), and ws(s) for the Pendulum node). Startup refuses any other value without echoing it, as it already did forHEARTBEAT_URL. Such a value could not work, but it used to fail later with errors that quote it, and a webhook URL without a scheme let the daemon run with every alert failing delivery.releaser cycle failedpages on the first failure, then at most every 10 minutes; the failures in between are logged. It used to page on every poll while an endpoint stayed unreachable, and each page delayed the next cycle by its delivery attempts. The heartbeat still goes out only after a completed cycle. A failed Multicall batch is now logged instead of paged on every cycle: the per-nonce reads that follow it complete the cycle, and if they fail too, the cycle fails and pages.startup failedpage, so a daemon restarted during an outage paged again on every restart (about every 11 s with the systemd example). Both daemons now retry their startup reads in the process and page the way they do while running: the releaser at most every 10 minutes, shared with its cycle failures, and the attestor at most once a minute, shared with its transient failures. Nothing is processed before these reads, so no event can be skipped. A wrong chain, a key that is not an attestor and unreadable state stay fatal.Tests
alert.test.ts(both services): every console method shows<url>in place of http(s) and ws(s) URLs, also inside a logged error.config.test.ts(both services): an endpoint setting without the right scheme is refused without echoing it.attestor/src/main.test.ts(new): against a Base endpoint that refuses connections, the attestor keeps running, pages once and logs the next failed read.releaser/src/main.test.ts: the releaser runs against a local RPC.npm test: attestor 99/99, releaser 40/40.