feat(bot): add fast inbox bot with public/private consumption probes and OTel metrics

[spalladino]

Adds BOT_MODE=inbox to @aztec/bot: a continuous L1-to-L2 messaging probe that sends atomic Multicall3 batches through the Inbox and follows every message to consumption on L2, through both public and private execution, reporting latency and correctness as OTel metrics.

No new Noir contract — it reuses TestContract's consume_message_from_arbitrary_sender_public / _private. Every existing bot mode keeps its defaults and behaviour.

Fixes A-1986.

What it does

• Sends a batch (4 messages by default) every BOT_TX_INTERVAL_SECONDS. In inbox mode that clock drives L1 batch starts, not L2 consumption; readiness and receipts are polled independently at ~1s.
• Starts the public attempt as soon as the node observes the message on L1, with no historical-readiness gate and no simulation bypass, so same-block insertion races are exercised for real. Winning the race is measured, never required.
• Holds the private attempt until the message is present at the wallet's anchor, then re-checks it at a concrete pinned block and verifies the membership witness against that block's root.
• Once a day sends 257 messages in one atomic transaction — one more than a bucket holds — so a full bucket rolls over inside a single L1 block. BOT_INBOX_SATURATION_INTERVAL_SECONDS=0 disables it.
• Keeps per-batch and per-message state durable across restarts, uncertain L1 receipts, dropped and reverted L2 transactions, and reorgs. Claim secrets are never dropped and an uncertain L1 send is never blind-resent.
• Checks node index vs. the canonical L1 event, unknown-hash probes, fixed-block witnesses, consumption nullifiers, bucket rollover, and cross-domain replay rejection.

Configuration

Variable	Default	Meaning
BOT_MODE	transfer	Select inbox.
BOT_INBOX_MESSAGES_PER_BATCH	4	Messages per atomic L1 batch, in [1, 257].
BOT_INBOX_CONSUME_MODE	mixed	mixed, public or private.
BOT_INBOX_SATURATION_INTERVAL_SECONDS	86400	Full-bucket run cadence; 0 disables.

In inbox mode BOT_L1_TO_L2_SEED_COUNT defaults to 512 (an outstanding-message cap, not a seeded pipeline) and BOT_FOLLOW_CHAIN=NONE is rejected. Explicit overrides always win.

Telemetry

16 instruments under aztec.bot.inbox.* through the existing TelemetryClient and collector — no new collector, no push loop, no log scraping. Milestone counters, five stage-latency histograms, simulation/execution/check/failure counters, pending-work gauges and the saturation schedule. Attributes are bounded; no hash, address, index, block number, batch id, exception text or timestamp is ever a label. Instrument names, their Prometheus translations, example queries, worked configurations and the alerting recipe are in bot/README.md.

Measured L1 gas

Multicall3 aggregate3, gas from estimation, measured against this branch:

Messages	gasUsed	per message	buckets
4	134,103	33,525	1
256	3,682,558	14,384	1
257	3,741,197	14,557	2

The 257th message costs 58,639 gas because it opens a bucket. The historical 2,212,595 execution-gas estimate from the design notes is not a limit and is not comparable — it excludes intrinsic gas, calldata and wrapper overhead and predates current bytecode.

Two bugs found on the way

• BotRunner.update() called stop(), which closed the KV store before recreating the bot. Pre-existing for every mode; it only became visible once a bot persisted state. update() now stops the bot and leaves the store open.
• The inbox bot's dispatch used stale snapshot records and never re-checked message state, so a message already in sent could be dispatched a second time once readiness landed after a public consumption. Dispatch now re-reads each candidate.

Testing

• 154 unit tests in @aztec/bot, clock-controlled over a fake L1 producer, a modelled chain and an in-memory recording meter.
• end-to-end/src/single-node/cross-chain/inbox_bot.test.ts: a mixed batch through both domains with a cross-domain replay rejection, a controlled same-block public success (the race is removed rather than retried), the 257-message batch with contiguous indices and real bucket rollover, and the gas measurements above. Passes in ~200s.
• A collector export smoke test drives the real OTLP/HTTP exporter into a local collector stub.

---

Stacked on #25416.

🤖 Generated with Claude Code