Skip to content

feat(monitor): Verified chain view, governance and vault paging, Slack alerts - #575

Merged
ebma merged 75 commits into
mainfrom
feat/monitor-timelock-watch
Oct 25, 2026
Merged

ebma merged 75 commits into
mainfrom
feat/monitor-timelock-watch

Conversation

@ebma

@ebma ebma commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

What

This extends the invariant monitor in four areas.

Verified view of Pendulum

  • Pendulum blocks are ingested along a parent-hash chain anchored to the Pendulum head recorded on the Polkadot relay chain (POLKADOT_WS, one or more relay RPCs), and only once confirmed.
  • An optional second node (PENDULUM_WS_CROSSCHECK) must agree on block hashes, migration totals and migration events.
  • A disagreement pages as critical and holds ingestion. Unavailable or stale checkers are reported without stopping the Base-side checks.

Base-side checks

  • Each scanned log range is checked against the vault's running totals and read again on a mismatch.
  • The vault balance check runs independently of Pendulum reads and of slow polls.
  • A Base view that stops advancing pages.
  • A mismatching approval pages; the vault pauses on mismatching approvals from two attestors or on a mismatching release.

Governance and admin visibility

  • Every timelock operation (TIMELOCK_ADDRESS) and every Pendulum public proposal, external proposal, referendum and council or technical-committee motion pages once, with its call decoded. Calls that mint or move balances, or change the runtime or the migration pallet, page as critical.
  • Vault admin events and migration pallet admin events page, and a treasury destination other than EXPECTED_TREASURY_DESTINATION pages as critical.

Alerting and operations

  • Pages are Slack incoming-webhook messages, retried until delivered. An optional HEARTBEAT_URL is called after each complete poll.
  • Auto-pause readiness is checked at startup and every poll; ACKNOWLEDGED_RELEASE_EXCESS lets operators record a quantified excess.

The README documents every setting and page.

Testing

  • npm test in monitor/: 132 tests, including single-poll scenarios against stand-in chains and mutation checks of the pause decisions.
  • The relay-chain anchoring was checked against public Polkadot RPCs and a Pendulum archive node.

ebma added 3 commits October 21, 2026 15:30
Once a queued operation's delay has passed, anyone may execute it, so the
canceller has exactly the delay to veto it. With TIMELOCK_ADDRESS set, the
monitor scans CallScheduled events up to its safe Base block at the start
of each poll and pages each operation once, with its calls decoded and
the time it becomes executable.
Every public proposal, the council's external proposal and every ongoing
referendum pages once, with its call decoded through batches. Calls that
mint, move or unlock balances, or change the runtime or the migration
pallet, page as critical: minted PEN burned through the migration drains
the fixed-supply vault, and only Pendulum governance can stop it before
enactment. The watch is alert-only and cannot stop the conservation checks.
@ebma ebma changed the title feat(monitor): Page every operation the governance timelock schedules feat(monitor): Page governance activity on Pendulum and the Base timelock Oct 21, 2026
ebma added 26 commits October 22, 2026 10:11
Split main() into start() and poll() so a test can run one poll of the
real check code against a JSON-RPC stand-in for Base and an object shaped
like the polkadot.js API for Pendulum, with the webhook recording every
page.
alert() now resolves to whether the webhook answered 2xx in time and logs
rejected or failed deliveries. Pages that must go out exactly once
(scheduled timelock operations, Pendulum governance items, the first
sighting of an unverified Base event, unreleasable burns) are keyed and
held until delivered, and every poll retries the undelivered ones.
All Pendulum reads and the event ingest now run in readSource(). If it
throws, the poll pages PENDULUM SOURCE UNREADABLE (throttled) and still
runs the vault balance check and its auto-pause; reconciliation, the
released-vs-migrated check and liveness wait for the source.
A governance item whose call cannot be read now pages as critical on its
own and the watch continues with the next item, instead of the whole
watch failing on that item every poll.
The Pendulum governance watch now reads a preimage's length from its
lookup reference and neither fetches nor decodes a call over 64 KiB; such
an item pages as unreadable. The call-tree walk stops past 256 calls or
16 levels of nesting. The watch runs after the conservation checks, with
its own finalized view and error handling, and sends its pages together.
A governance item whose preimage is not noted yet, or whose call could
not be fetched, now pages as critical once and is re-checked every poll;
once its call can be read it pages again, as critical when the call is
risky. Legacy call references are read through the noted preimage's
length. A call that is too big or cannot be decoded pages as unreadable.
The Pendulum governance watch now also decodes every open council and
technical-committee motion with the same size bound and risk rules, and
the event ingest pages each MigrationPauseSet and TreasuryDestinationSet
of the migration pallet as it is finalized.
Each poll compares the timestamp of the safe/finalized Base block with the
wall clock and pages BASE VIEW STALLED, throttled like the Pendulum stall
page, once it is older than BASE_STALE_ALERT_SECONDS (default 600 s for
safe, 1800 s for finalized).
At startup and on every poll the monitor checks that the Base RPC serves
BASE_CHAIN_ID and, with a guardian key configured, that the key parses,
is vault.guardian() and holds at least GUARDIAN_MIN_BALANCE_WEI (default
0.001 ETH). Problems page as AUTO-PAUSE NOT READY, throttled; a monitor
without a key pages AUTO-PAUSE DISABLED at startup, and the startup log
states the auto-pause status.
The scan that runs ahead of the vault checks now also reads the vault's
admin events (token, pause and unpause, attestors, threshold, caps,
guardian, admin transfer, sweep, stale pending cleared) and pages each one
once as VAULT ADMIN EVENT, so the persisted Base cursor never passes an
unscanned one.
Stale migrations now page as one summary per kind and poll (count and the
first nonces), with the detail per nonce in the log, instead of one
awaited page per nonce; a nonce counts as paged only once its summary is
delivered. The first sighting of an unmatched Base event pages without
being awaited, so reconciliation and the pause decision never wait for
the webhook, and the retry of once-only pages waits for one that is still
in flight instead of sending it twice.
Reconciliation now stops at a block range that holds a Base event whose
nonce the source view may still reveal (within the grace or past it) even
when the poll found a violation, so the state persisted after the pause
replays that range and matches the release once the burn is ingested.
Events proven to have no source are still passed after a confirmed pause.
ACKNOWLEDGED_RELEASE_EXCESS (token units, default 0, validated and logged
at startup) is added to the migrated total in the released-vs-migrated
check, so once a loss has been quantified and accepted the vault can be
unpaused without the check pausing it again; any further excess still
fires.
Describe the Base view, auto-pause readiness, Pendulum source failure,
vault and migration-pallet admin event checks, the motions and decoding
bounds of the governance watch, the liveness summaries, webhook delivery
and retry, the acknowledged release excess, and list every page subject.
A failed eth_chainId, guardian() or eth_getBalance read now counts as an
auto-pause readiness problem and pages, throttled, like the others. The
vault balance check and the rest of the poll keep running.
…n fails

A failed scan of the vault admin events and the governance queue now
pages BASE ADMIN SCAN FAILED, throttled, instead of aborting the poll:
the vault balance check and its auto-pause still run. Reconciliation,
M2a and liveness wait until the scan reads again, so the persisted Base
cursor never passes a range the scan has not covered.
While the source read fails, Base events are now reconciled against the
last Pendulum view read in full, as against a lagging source but without
the fabrication proof: a mismatch on a known nonce pauses at once, and an
event with an unknown nonce pages UNVERIFIED BASE EVENT and pauses once
UNMATCHED_EVENT_GRACE_SECONDS has passed. The poll persists that state
and skips only M2a and liveness, which need the current source.
…same call

Council and technical-committee motions are now paged once per motion
index instead of once per call hash, so a motion that was closed and
proposed again with the same call pages again. The external proposal,
which only its call identifies, pages again once it has left the queue
and is made again.
The guardian balance floor is now parsed like the other amounts: a
negative or non-numeric value fails at startup, and an empty one keeps
the default, instead of silently switching the balance check off.
The startup page that says no guardian key is configured is now a
once-only page, so a webhook that is down at startup no longer loses it:
every poll sends it again until the webhook accepts it.
… map

A legacy call reference is sized through preimage.requestStatusFor, but a
preimage noted before the pallet's status migration stays in the
deprecated statusFor until something touches it. The watcher now falls
back to statusFor, so such a call is decoded instead of staying pending.
The governance watch cannot stop or delay a poll's conservation checks,
and its cost to the next poll is bounded; the guardian key must be the
vault guardian's, and any other key pages AUTO-PAUSE NOT READY.
…fields

The page for an operation the governance timelock schedules formatted its
ready time as a date, which throws for a time past the range a date can
hold, and it read the call fields as strings. Such an operation now pages
with its ready time in seconds, and missing call fields are shown as they
are, so describing one operation never fails the scan or hides the
operations after it.
Reconciliation, M2a and liveness waited for the scan of vault admin events
and the governance queue, so a scan that kept failing switched them off.
The scan now keeps its own cursor in the state file: while it fails, every
other check runs, and it resumes where it stopped, also after a restart.
State written before keeps working; its admin cursor starts at the Base
cursor, which never passed an unscanned range.

The PENDULUM SOURCE UNREADABLE page now says what happens until the
source reads again.
… reads

The vault balance check (M2b) ran after the Pendulum read and the Base
log reconciliation, so a failing log read skipped it and a slow source
delayed it. It reads Base only, so it now runs first in every poll, and
its pause no longer ends the poll: reconciliation and the other checks
still run after it.
An external proposal is identified only by its call, so the same call
proposed again right after it became a referendum kept its key and did
not page until it became a referendum too. The key now includes the
newest ongoing referendum with the same call, so such a proposal pages
again in the next poll.
ebma added 27 commits October 22, 2026 13:57
With PENDULUM_WS_CROSSCHECK set, the second node must also agree with the
source on tokenMigration.nextNonce and totalMigrated at the finalized head,
and on the migration pallet events of every ingested block that has some.
A difference pages CRITICAL: PENDULUM SOURCE CONTRADICTED and nothing from
the source is ingested; a second node that cannot answer only warns.
M2a needs only totalReleased at the poll block and TotalMigrated, so a log
range held for missing events no longer skips it. It still waits while a
Base event's burn is not in the source view yet, where it would fire
falsely.
While a block range is held for missing events, no later Base event is
matched against its burn. Once a range has been held for longer than
UNMATCHED_EVENT_GRACE_SECONDS, the throttled page turns into CRITICAL: BASE
LOG SCAN INCOMPLETE; a range that reads in full ends the hold.
… busy

A poll checks the vault balance first, but the next check waited for the
rest of the poll: Pendulum reads, a long catch-up. main() now also runs the
check on a timer whenever a poll is still busy a poll interval after its
last check started. A violation found while a pause attempt is in flight
joins that attempt instead of starting another.
At a vault threshold of two or less, one more approval releases a nonce, so
a single attestor's approval that disagrees with the finalized burn now
pauses the vault there. Above that, two distinct attestors are still needed
and one pages. The threshold is read before the Base events are reconciled.
While the admin scan fails, vault admin events and scheduled governance
operations are not paged, and the canceller's veto window keeps running.
Once the scan has failed for longer than UNMATCHED_EVENT_GRACE_SECONDS, the
throttled page turns into CRITICAL: BASE ADMIN SCAN FAILED; a scan that
reads again ends it.
While the pause after a treasury destination violation is not confirmed,
the monitor holds at that block. The hold was paged as an unreadable
Pendulum source, which it is not; the violation's page now says that the
monitor holds there until the vault is paused, and the hold is logged.
… read

A range whose first log read misses an event and whose second accounts for
the vault's totals advances the cursor without a page; a totals read that
fails at a block where the vault has code fails the poll instead of
counting as zero.
…e walk

The second node's nonce and total are now each compared on their own, a
relay head below the block ingested last is checked down the source's
chain, and the tests of the vault balance timer wait until the poll holds
on the source instead of for a fixed time.
Releases and sweeps in the first and the last block of a scanned range
must account for the vault's totals, and a source header that names
another block number pages critical. A test that waited for a page
without a bound now fails after five seconds instead of hanging.
With PENDULUM_WS_CROSSCHECK set, the source's chain is compared with the
second node at the second node's finalized head, or at the source's head
once the second node has finalized past it, and no block above that height
is ingested. The view the monitor checks against (its timestamp, nextNonce,
totalMigrated and issuance) is read at the newest ingested block, and the
second node's totals are compared there.

A second node whose finalized head is older than SOURCE_STALE_ALERT_SECONDS
counts as unavailable, like one that cannot be read: it pages a warning and
holds nothing back. While the second node is unavailable, the migration
events of the ingested blocks are not compared with it.
A relay RPC whose record does not anchor the source (not newer than the
kept head, or not imported by the source yet) no longer ends the relay
check: the next relay RPC in POLKADOT_WS is asked.

A record that contradicts the source is outvoted only by another relay
RPC's record that anchors the source at or above the contradicting record's
height. That record then anchors the source, and RELAY RPCS DISAGREE asks
operators to check both relay RPCs. Any other answer leaves the
contradiction unconfirmed and pages RELAY ANCHOR UNAVAILABLE.
While a relay RPC answers, no Pendulum block above the newest relay-anchored
head (this poll's, or the kept one) is ingested, and the view the checks
use is read at the newest ingested block. Only while no relay RPC answers
at all is the source used without the anchor, up to its finalized head.

When the relay RPCs answer but none anchors the source's newer blocks (a
head that is not newer, one the source has not imported, or a contradiction
that stays unconfirmed), RELAY ANCHOR UNAVAILABLE pages at once instead of
waiting for RELAY ANCHOR STALE. PENDULUM SOURCE VIEW STALLED now measures
the age of the verified view and names the source's newer finalized head.
…blocks

A relay-chain record of the Pendulum head that lies more than 100 blocks
below the block ingested last is not walked down to: it does not anchor the
source, and the next relay RPC is asked.
…an key

The pause attempt reads paused() at the confirmed Base block before it
looks for GUARDIAN_PRIVATE_KEY. A vault that is already paused counts as
paused, so a monitor without a key moves past a treasury violation, and
persists the state after a tuple violation, once operators paused the vault
by hand. AUTO-PAUSE UNAVAILABLE pages only while the vault is not paused.
… expected one

With EXPECTED_TREASURY_DESTINATION set, a finalized TreasuryDestinationSet
to another address pages CRITICAL: MIGRATION PALLET ADMIN EVENT instead of
the warning: each treasury migration to it would pause the vault. A
destination set to the expected address still pages as a warning.
…or note

While a poll is busy, two runs of the vault balance check can be about
three poll intervals apart, not two. The helper that collects disagreeing
attestors no longer describes the pause rule, which its caller states.
…and quiet relay polls

Pin that the conservation check uses the total of the view the relay chain
confirms, that a second node behind the block ingested last holds newer
blocks back without a contradiction, that a second node that cannot be read
is not asked for the ingested blocks' events, and that a relay record and a
source that both stay put page nothing.
…nd a lagging source

Pin that a pause mined by hand but not yet final keeps a treasury hold, that
a tuple violation whose pause is not confirmed is found again in the next
poll, that a kept relay record above the source's finalized head confirms
the blocks below it, and that a source whose finalized head falls behind
the block ingested last is read at that block.
The Base stand-in accepts the guardian's pause transaction where a test
opts in, mined in the confirmed block. A test pins that the auto-pause is
reported as confirmed only where paused() is true at the confirmed block
after the transaction, and as failed otherwise.
@ebma ebma changed the title feat(monitor): Page governance activity on Pendulum and the Base timelock feat(monitor): Verified chain view, governance and vault paging, Slack alerts Oct 23, 2026
@ebma
ebma merged commit a95615b into main Oct 25, 2026
7 checks passed
@ebma
ebma deleted the feat/monitor-timelock-watch branch October 25, 2026 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant