Repository navigation
feat(monitor): Verified chain view, governance and vault paging, Slack alerts - #575
Merged
Merged
Conversation
Once a queued operation's delay has passed, anyone may execute it, so the canceller has exactly the delay to veto it. With TIMELOCK_ADDRESS set, the monitor scans CallScheduled events up to its safe Base block at the start of each poll and pages each operation once, with its calls decoded and the time it becomes executable.
Every public proposal, the council's external proposal and every ongoing referendum pages once, with its call decoded through batches. Calls that mint, move or unlock balances, or change the runtime or the migration pallet, page as critical: minted PEN burned through the migration drains the fixed-supply vault, and only Pendulum governance can stop it before enactment. The watch is alert-only and cannot stop the conservation checks.
Split main() into start() and poll() so a test can run one poll of the real check code against a JSON-RPC stand-in for Base and an object shaped like the polkadot.js API for Pendulum, with the webhook recording every page.
alert() now resolves to whether the webhook answered 2xx in time and logs rejected or failed deliveries. Pages that must go out exactly once (scheduled timelock operations, Pendulum governance items, the first sighting of an unverified Base event, unreleasable burns) are keyed and held until delivered, and every poll retries the undelivered ones.
All Pendulum reads and the event ingest now run in readSource(). If it throws, the poll pages PENDULUM SOURCE UNREADABLE (throttled) and still runs the vault balance check and its auto-pause; reconciliation, the released-vs-migrated check and liveness wait for the source.
A governance item whose call cannot be read now pages as critical on its own and the watch continues with the next item, instead of the whole watch failing on that item every poll.
The Pendulum governance watch now reads a preimage's length from its lookup reference and neither fetches nor decodes a call over 64 KiB; such an item pages as unreadable. The call-tree walk stops past 256 calls or 16 levels of nesting. The watch runs after the conservation checks, with its own finalized view and error handling, and sends its pages together.
A governance item whose preimage is not noted yet, or whose call could not be fetched, now pages as critical once and is re-checked every poll; once its call can be read it pages again, as critical when the call is risky. Legacy call references are read through the noted preimage's length. A call that is too big or cannot be decoded pages as unreadable.
The Pendulum governance watch now also decodes every open council and technical-committee motion with the same size bound and risk rules, and the event ingest pages each MigrationPauseSet and TreasuryDestinationSet of the migration pallet as it is finalized.
Each poll compares the timestamp of the safe/finalized Base block with the wall clock and pages BASE VIEW STALLED, throttled like the Pendulum stall page, once it is older than BASE_STALE_ALERT_SECONDS (default 600 s for safe, 1800 s for finalized).
At startup and on every poll the monitor checks that the Base RPC serves BASE_CHAIN_ID and, with a guardian key configured, that the key parses, is vault.guardian() and holds at least GUARDIAN_MIN_BALANCE_WEI (default 0.001 ETH). Problems page as AUTO-PAUSE NOT READY, throttled; a monitor without a key pages AUTO-PAUSE DISABLED at startup, and the startup log states the auto-pause status.
The scan that runs ahead of the vault checks now also reads the vault's admin events (token, pause and unpause, attestors, threshold, caps, guardian, admin transfer, sweep, stale pending cleared) and pages each one once as VAULT ADMIN EVENT, so the persisted Base cursor never passes an unscanned one.
Stale migrations now page as one summary per kind and poll (count and the first nonces), with the detail per nonce in the log, instead of one awaited page per nonce; a nonce counts as paged only once its summary is delivered. The first sighting of an unmatched Base event pages without being awaited, so reconciliation and the pause decision never wait for the webhook, and the retry of once-only pages waits for one that is still in flight instead of sending it twice.
Reconciliation now stops at a block range that holds a Base event whose nonce the source view may still reveal (within the grace or past it) even when the poll found a violation, so the state persisted after the pause replays that range and matches the release once the burn is ingested. Events proven to have no source are still passed after a confirmed pause.
ACKNOWLEDGED_RELEASE_EXCESS (token units, default 0, validated and logged at startup) is added to the migrated total in the released-vs-migrated check, so once a loss has been quantified and accepted the vault can be unpaused without the check pausing it again; any further excess still fires.
Describe the Base view, auto-pause readiness, Pendulum source failure, vault and migration-pallet admin event checks, the motions and decoding bounds of the governance watch, the liveness summaries, webhook delivery and retry, the acknowledged release excess, and list every page subject.
A failed eth_chainId, guardian() or eth_getBalance read now counts as an auto-pause readiness problem and pages, throttled, like the others. The vault balance check and the rest of the poll keep running.
…n fails A failed scan of the vault admin events and the governance queue now pages BASE ADMIN SCAN FAILED, throttled, instead of aborting the poll: the vault balance check and its auto-pause still run. Reconciliation, M2a and liveness wait until the scan reads again, so the persisted Base cursor never passes a range the scan has not covered.
While the source read fails, Base events are now reconciled against the last Pendulum view read in full, as against a lagging source but without the fabrication proof: a mismatch on a known nonce pauses at once, and an event with an unknown nonce pages UNVERIFIED BASE EVENT and pauses once UNMATCHED_EVENT_GRACE_SECONDS has passed. The poll persists that state and skips only M2a and liveness, which need the current source.
…same call Council and technical-committee motions are now paged once per motion index instead of once per call hash, so a motion that was closed and proposed again with the same call pages again. The external proposal, which only its call identifies, pages again once it has left the queue and is made again.
The guardian balance floor is now parsed like the other amounts: a negative or non-numeric value fails at startup, and an empty one keeps the default, instead of silently switching the balance check off.
The startup page that says no guardian key is configured is now a once-only page, so a webhook that is down at startup no longer loses it: every poll sends it again until the webhook accepts it.
… map A legacy call reference is sized through preimage.requestStatusFor, but a preimage noted before the pallet's status migration stays in the deprecated statusFor until something touches it. The watcher now falls back to statusFor, so such a call is decoded instead of staying pending.
The governance watch cannot stop or delay a poll's conservation checks, and its cost to the next poll is bounded; the guardian key must be the vault guardian's, and any other key pages AUTO-PAUSE NOT READY.
…fields The page for an operation the governance timelock schedules formatted its ready time as a date, which throws for a time past the range a date can hold, and it read the call fields as strings. Such an operation now pages with its ready time in seconds, and missing call fields are shown as they are, so describing one operation never fails the scan or hides the operations after it.
Reconciliation, M2a and liveness waited for the scan of vault admin events and the governance queue, so a scan that kept failing switched them off. The scan now keeps its own cursor in the state file: while it fails, every other check runs, and it resumes where it stopped, also after a restart. State written before keeps working; its admin cursor starts at the Base cursor, which never passed an unscanned range. The PENDULUM SOURCE UNREADABLE page now says what happens until the source reads again.
… reads The vault balance check (M2b) ran after the Pendulum read and the Base log reconciliation, so a failing log read skipped it and a slow source delayed it. It reads Base only, so it now runs first in every poll, and its pause no longer ends the poll: reconciliation and the other checks still run after it.
An external proposal is identified only by its call, so the same call proposed again right after it became a referendum kept its key and did not page until it became a referendum too. The key now includes the newest ongoing referendum with the same call, so such a proposal pages again in the next poll.
With PENDULUM_WS_CROSSCHECK set, the second node must also agree with the source on tokenMigration.nextNonce and totalMigrated at the finalized head, and on the migration pallet events of every ingested block that has some. A difference pages CRITICAL: PENDULUM SOURCE CONTRADICTED and nothing from the source is ingested; a second node that cannot answer only warns.
M2a needs only totalReleased at the poll block and TotalMigrated, so a log range held for missing events no longer skips it. It still waits while a Base event's burn is not in the source view yet, where it would fire falsely.
While a block range is held for missing events, no later Base event is matched against its burn. Once a range has been held for longer than UNMATCHED_EVENT_GRACE_SECONDS, the throttled page turns into CRITICAL: BASE LOG SCAN INCOMPLETE; a range that reads in full ends the hold.
… busy A poll checks the vault balance first, but the next check waited for the rest of the poll: Pendulum reads, a long catch-up. main() now also runs the check on a timer whenever a poll is still busy a poll interval after its last check started. A violation found while a pause attempt is in flight joins that attempt instead of starting another.
At a vault threshold of two or less, one more approval releases a nonce, so a single attestor's approval that disagrees with the finalized burn now pauses the vault there. Above that, two distinct attestors are still needed and one pages. The threshold is read before the Base events are reconciled.
While the admin scan fails, vault admin events and scheduled governance operations are not paged, and the canceller's veto window keeps running. Once the scan has failed for longer than UNMATCHED_EVENT_GRACE_SECONDS, the throttled page turns into CRITICAL: BASE ADMIN SCAN FAILED; a scan that reads again ends it.
While the pause after a treasury destination violation is not confirmed, the monitor holds at that block. The hold was paged as an unreadable Pendulum source, which it is not; the violation's page now says that the monitor holds there until the vault is paused, and the hold is logged.
… read A range whose first log read misses an event and whose second accounts for the vault's totals advances the cursor without a page; a totals read that fails at a block where the vault has code fails the poll instead of counting as zero.
…e walk The second node's nonce and total are now each compared on their own, a relay head below the block ingested last is checked down the source's chain, and the tests of the vault balance timer wait until the poll holds on the source instead of for a fixed time.
Releases and sweeps in the first and the last block of a scanned range must account for the vault's totals, and a source header that names another block number pages critical. A test that waited for a page without a bound now fails after five seconds instead of hanging.
With PENDULUM_WS_CROSSCHECK set, the source's chain is compared with the second node at the second node's finalized head, or at the source's head once the second node has finalized past it, and no block above that height is ingested. The view the monitor checks against (its timestamp, nextNonce, totalMigrated and issuance) is read at the newest ingested block, and the second node's totals are compared there. A second node whose finalized head is older than SOURCE_STALE_ALERT_SECONDS counts as unavailable, like one that cannot be read: it pages a warning and holds nothing back. While the second node is unavailable, the migration events of the ingested blocks are not compared with it.
A relay RPC whose record does not anchor the source (not newer than the kept head, or not imported by the source yet) no longer ends the relay check: the next relay RPC in POLKADOT_WS is asked. A record that contradicts the source is outvoted only by another relay RPC's record that anchors the source at or above the contradicting record's height. That record then anchors the source, and RELAY RPCS DISAGREE asks operators to check both relay RPCs. Any other answer leaves the contradiction unconfirmed and pages RELAY ANCHOR UNAVAILABLE.
While a relay RPC answers, no Pendulum block above the newest relay-anchored head (this poll's, or the kept one) is ingested, and the view the checks use is read at the newest ingested block. Only while no relay RPC answers at all is the source used without the anchor, up to its finalized head. When the relay RPCs answer but none anchors the source's newer blocks (a head that is not newer, one the source has not imported, or a contradiction that stays unconfirmed), RELAY ANCHOR UNAVAILABLE pages at once instead of waiting for RELAY ANCHOR STALE. PENDULUM SOURCE VIEW STALLED now measures the age of the verified view and names the source's newer finalized head.
…blocks A relay-chain record of the Pendulum head that lies more than 100 blocks below the block ingested last is not walked down to: it does not anchor the source, and the next relay RPC is asked.
…an key The pause attempt reads paused() at the confirmed Base block before it looks for GUARDIAN_PRIVATE_KEY. A vault that is already paused counts as paused, so a monitor without a key moves past a treasury violation, and persists the state after a tuple violation, once operators paused the vault by hand. AUTO-PAUSE UNAVAILABLE pages only while the vault is not paused.
… expected one With EXPECTED_TREASURY_DESTINATION set, a finalized TreasuryDestinationSet to another address pages CRITICAL: MIGRATION PALLET ADMIN EVENT instead of the warning: each treasury migration to it would pause the vault. A destination set to the expected address still pages as a warning.
…or note While a poll is busy, two runs of the vault balance check can be about three poll intervals apart, not two. The helper that collects disagreeing attestors no longer describes the pause rule, which its caller states.
…and quiet relay polls Pin that the conservation check uses the total of the view the relay chain confirms, that a second node behind the block ingested last holds newer blocks back without a contradiction, that a second node that cannot be read is not asked for the ingested blocks' events, and that a relay record and a source that both stay put page nothing.
…nd a lagging source Pin that a pause mined by hand but not yet final keeps a treasury hold, that a tuple violation whose pause is not confirmed is found again in the next poll, that a kept relay record above the source's finalized head confirms the blocks below it, and that a source whose finalized head falls behind the block ingested last is read at that block.
The Base stand-in accepts the guardian's pause transaction where a test opts in, mined in the confirmed block. A test pins that the auto-pause is reported as confirmed only where paused() is true at the confirmed block after the transaction, and as failed otherwise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This extends the invariant monitor in four areas.
Verified view of Pendulum
POLKADOT_WS, one or more relay RPCs), and only once confirmed.PENDULUM_WS_CROSSCHECK) must agree on block hashes, migration totals and migration events.Base-side checks
Governance and admin visibility
TIMELOCK_ADDRESS) and every Pendulum public proposal, external proposal, referendum and council or technical-committee motion pages once, with its call decoded. Calls that mint or move balances, or change the runtime or the migration pallet, page as critical.EXPECTED_TREASURY_DESTINATIONpages as critical.Alerting and operations
HEARTBEAT_URLis called after each complete poll.ACKNOWLEDGED_RELEASE_EXCESSlets operators record a quantified excess.The README documents every setting and page.
Testing
npm testinmonitor/: 132 tests, including single-poll scenarios against stand-in chains and mutation checks of the pause decisions.