feat(p2p): require the Inbox prefix reference on every block proposal

docs/docs-developers/docs/resources/migration_notes.md

@@ -71,6 +71,23 @@ In `@aztec/ethereum/contracts`, the top-level `MessageSentLog.l1BlockTimestamp`
 
 **Impact**: Only node-internal and tooling consumers are affected; `getL1ToL2MessageIndex`, `getL1ToL2MessageMembershipWitness` and the Aztec.nr consumption functions are unchanged. Operators upgrading a node: see the node changelog for the archiver store reset.
 
+### [Aztec Node] The Inbox prefix reference is a required proposal field, before the transaction bundle
+
+`BlockProposal.inboxPrefixRef` and `CheckpointProposal.lastBlock.inboxPrefixRef` are no longer optional, and the reference moves ahead of the optional `SignedTxs` bundle on the wire. Every proposed block carries one, including a block that consumes no new messages: such a block re-states the prefix its parent ended at, so the pair (the header's cumulative message count, the reference's rolling hash) is signed either way. The zero hash is a real value only for the empty prefix a chain starts from.
+
+The 32-byte reference is now written with no presence flag, immediately after the transaction hashes and before the `hasSignedTxs` flag, and it is always part of the signed payload. There is no end-of-buffer fallback: a proposal that omits or truncates the reference fails to decode instead of reading as a proposal that consumed nothing. A valid-reference proposal is four bytes smaller than before, because the presence flag is gone. The last block embedded in a `CheckpointProposal` uses the same layout, and its reference is checked against the checkpoint header's `inboxRollingHash` at construction and at decode. The embedded last block itself stays optional.
+
+This is a breaking peer-to-peer format change on top of the encoding change already noted above: a node on the previous format cannot decode a proposal from a node on this one, and vice versa. Every node that gossips, signs or validates proposals on a network has to run a matching version.
+
+**Migration:**
+
+```diff
+- new BlockProposal(header, index, archive, txHashes, signature, signatureContext, signedTxs, inboxPrefixRef)
++ new BlockProposal(header, index, archive, txHashes, signature, signatureContext, inboxPrefixRef, signedTxs)
+```
+
+`BlockProposal.createProposalFromSigner` takes `inboxPrefixRef` after `signatureContext` and before the signer callbacks, and `Validator.createBlockProposal` takes it after `proposerAddress` and before `options`. `ValidationService.createBlockProposal` keeps its parameter order but the reference is required. Historical blocks replayed from L1 blobs are unaffected: the reference is peer-to-peer only and is not part of any published block.
+
 ### [Aztec Node] Sequencer and validator streaming-Inbox internals
 
 For consumers wiring these packages directly. `Sequencer` takes an `InboxContract` constructor argument (after `rollupContract`) and `AutomineSequencerDeps` gains `inboxContract`; completion uses it to resolve a checkpoint's final message position to a live bucket. `NodePublicCallsSimulator` deps drop `l1Client` and `useAutomineSequencer`. `SequencerPublisher.validateCheckpointHeader` is replaced by `validateCheckpointHeaderAndInbox(header, { expectedTotal, expectedParentCheckpointNumber }, simulationOverridesPlan?)`, which returns the `bucketHint` to pass to `propose`. `@aztec/sequencer-client` no longer exports `InboxBucketConfirmationTracker`, `InboxBucketEligibility`, `L1BlockReader`, `immediateEligibility`, `ConsumedBucketCursor`, `InboxBucketSelection`, `InboxBucketSource`, `SelectInboxBucketInput` or `selectInboxBucketForBlock`; it exports `InboxConsumptionCaps`, `PROTOCOL_INBOX_CONSUMPTION_CAPS`, `InboxEndpointResolver`, `StreamingMessageSource`, `selectOrdinaryMessageEnd`, `selectSafeLocalEnd`, `getOrdinaryCeiling`, `mustQueryEndpoint`, `getEndpointUpperBound` and `resolveEndpoint` instead. These helpers are stateless: nothing is retained between blocks. `selectOrdinaryMessageEnd` returns the greedy end of the locally observed messages under the per-block and checkpoint caps; `getOrdinaryCeiling` is the threshold one bucket's worth of messages below the checkpoint cap, and `selectSafeLocalEnd` holds the greedy end down to it. `mustQueryEndpoint` decides, from that prospective end, whether the block has to resolve a live L1 bucket end at all, and `resolveEndpoint` performs the single Inbox lookup (bounded by `getEndpointUpperBound`) and authenticates the resolved end against the local message log in one snapshot.

docs/docs-operate/operators/reference/changelog/v6.md

@@ -16,6 +16,12 @@ The archiver now stores L1-to-L2 messages as a plain ordered log (compact index,
 
 The archiver is the only store the streaming Inbox changes; world state follows the archiver through the block stream, and the L1 transaction state store is unaffected. Other v6 work bumps other stores on its own schedule: `PXE_DATA_SCHEMA_VERSION` moves from 13 to 14 in this release for unrelated reasons, so a PXE reset in v6 is not caused by the streaming Inbox.
 
+### Block proposals must carry the Inbox prefix reference
+
+Every proposed block now carries a required 32-byte reference to the Inbox message prefix it consumed through, including a block that consumes no new messages, which re-states the prefix its parent ended at. The reference is serialized without a presence flag, ahead of the optional transaction bundle, and is always part of the signed payload; a proposal that omits or truncates it is rejected as malformed rather than read as a block that consumed nothing.
+
+**Migration**: this is a breaking peer-to-peer format change. A node on the previous format cannot decode a proposal from a node on this one, and vice versa, so upgrade the peer set together. There is no supported mode in which both encodings circulate at once. Nothing in the L1 contracts, the published block format or any node configuration changes, and a node that syncs historical blocks from L1 is unaffected.
+
 ### `maxBlocksPerCheckpoint` floor lowered from 7 to 4
 
 The network consensus configuration requires `maxBlocksPerCheckpoint` to be at least `MIN_BLOCKS_FOR_INBOX_CATCHUP`, so a checkpoint can always clear a cap-sized backlog of mandatory L1-to-L2 messages. Blocks now consume messages by count rather than by whole L1 bucket, so the floor is `ceil(1024 / 256) = 4` instead of 7. Existing configurations remain valid; networks that were forced to shorten their block duration to satisfy the old floor may relax it.

yarn-project/archiver/src/modules/l1_synchronizer.ts

@@ -680,6 +680,13 @@ export class ArchiverL1Synchronizer implements Traceable {
         // We must only set this block number based on actually retrieved logs.
         // TODO(#8621): Tackle this properly when we handle L1 Re-orgs.
         // await this.stores.blocks.setSynchedL1BlockNumber(currentL1BlockNumber);
+        //
+        // Known limitation: this short-circuit only compares the pending archive root, so a reconciliation that
+        // replaced the checkpoint at the same height leaves work outstanding that this pass skips, and the refetch
+        // waits for L1 to produce a block that moves the root again. A "checkpoint refetch still pending" flag
+        // carried out of reconciliation past this optimization would let the next pass fetch at the same head. It is
+        // general checkpoint-event syncing rather than an Inbox consumption rule, and is deferred:
+        // https://linear.app/aztec-labs/issue/A-1985
         this.log.debug(`No checkpoints to retrieve from ${blocksSynchedTo + 1n} to ${currentL1BlockNumber}`);
         return { rollupStatus, fetchCheckpoints: false, provenArchive };
       }

yarn-project/aztec-node/src/aztec-node/node_public_calls_simulator.ts

@@ -253,7 +253,9 @@ export class NodePublicCallsSimulator {
    * 400 messages observed and live buckets ending at 200 and 400, this appends 256 while a final block lands on
    * 200: a public call consuming message index 220 simulates successfully and then fails when it runs for real.
    * Callers that need certainty check inclusion at an L2 tip that already exists, with `isL1ToL2MessageReady` from
-   * `@aztec/aztec.js/messaging`.
+   * `@aztec/aztec.js/messaging`. Closing that gap would mean this node running the sequencer's live endpoint
+   * selection, including its Inbox reads, on every simulation; it is deferred:
+   * https://linear.app/aztec-labs/issue/A-1982
    *
    * Any failure, such as messages not synced yet or a torn archiver snapshot, leaves the fork at the tip state,
    * which is what the transaction sees if the next block consumes nothing.

yarn-project/end-to-end/src/composed/web3signer/e2e_multi_validator_node_key_store.test.ts

@@ -16,6 +16,7 @@ import { type EthPrivateKey, KeystoreManager, loadKeystores, mergeKeystores } fr
 import { StatefulTestContractArtifact } from '@aztec/noir-test-contracts.js/StatefulTest';
 import type { Sequencer, SequencerClient, SequencerPublisherFactory } from '@aztec/sequencer-client';
 import type { TestSequencer, TestSequencerClient } from '@aztec/sequencer-client/test';
+import type { InboxMessagePrefixRef } from '@aztec/stdlib/messaging';
 import type { BlockProposalOptions } from '@aztec/stdlib/p2p';
 import type { BlockHeader, Tx } from '@aztec/stdlib/tx';
 import { NodeKeystoreAdapter, ValidatorClient } from '@aztec/validator-client';
@@ -379,6 +380,7 @@ describe('e2e_multi_validator_node', () => {
       archive: Fr,
       txs: Tx[],
       proposerAddress: EthAddress | undefined,
+      inboxPrefixRef: InboxMessagePrefixRef,
       options: BlockProposalOptions,
     ) => {
       if (proposerAddress) {
@@ -399,6 +401,7 @@ describe('e2e_multi_validator_node', () => {
         archive,
         txs,
         proposerAddress,
+        inboxPrefixRef,
         options,
       );
     };

yarn-project/sequencer-client/src/sequencer/automine/automine_sequencer.ts

@@ -481,6 +481,12 @@ export class AutomineSequencer {
     // resolve the live L1 bucket end at or below it with one Inbox call, and authenticate the range to it against
     // the local log. Landing on that boundary can leave observed messages behind, so the block may consume fewer
     // than the caps allow. The parent total is the fork's L1-to-L2 leaf count (compact indexing).
+    //
+    // Known limitation: a single block can insert at most the per-block cap, so once more messages than that have
+    // aged past L1's censorship deadline, no single-block checkpoint can consume all the ones `propose` demands and
+    // automine stops making progress. An ordinary sequencer spreads the backlog over several blocks of one slot;
+    // automine has no second block to spread it over. Working around it by warping the L1 timestamp so the deadline
+    // moves is deferred: https://linear.app/aztec-labs/issue/A-1981
     const parentTotalMsgCount = (await fork.getTreeInfo(MerkleTreeId.L1_TO_L2_MESSAGE_TREE)).size;
     const cursor = await this.deps.l1ToL2MessageSource.getMessagePosition(parentTotalMsgCount);
     if (cursor === undefined) {

yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.test.ts

@@ -307,7 +307,7 @@ describe('CheckpointProposalJob', () => {
     validatorClient = mock<ValidatorClient>();
     validatorClient.collectAttestations.mockImplementation(() => Promise.resolve([]));
     validatorClient.createBlockProposal.mockImplementation(
-      async (blockHeader, _checkpointNumber, indexWithinCheckpoint, archiveRoot, txs) => {
+      async (blockHeader, _checkpointNumber, indexWithinCheckpoint, archiveRoot, txs, _proposer, inboxPrefixRef) => {
         const txHashes = await Promise.all((txs ?? []).map((tx: Tx) => tx.getTxHash()));
         return new BlockProposal(
           blockHeader,
@@ -316,6 +316,7 @@ describe('CheckpointProposalJob', () => {
           txHashes,
           mockedSig,
           signatureContext,
+          inboxPrefixRef,
         );
       },
     );
@@ -342,6 +343,7 @@ describe('CheckpointProposalJob', () => {
             indexWithinCheckpoint: lastBlockInfo.indexWithinCheckpoint,
             txHashes,
             signature: mockedSig,
+            inboxPrefixRef: lastBlockInfo.inboxPrefixRef,
             // Note: signedTxs omitted since publishTxsWithProposals is false in tests
           },
         );
@@ -1606,7 +1608,7 @@ describe('CheckpointProposalJob', () => {
     };
     const bundleLengths = () => checkpointBuilder.buildBlockCalls.map(call => call.opts.l1ToL2Messages?.length);
     const signedPrefixes = () =>
-      validatorClient.createBlockProposal.mock.calls.map(call => call[7]?.inboxRollingHash.toString());
+      validatorClient.createBlockProposal.mock.calls.map(call => call[6].inboxRollingHash.toString());
     const prefixAt = (count: number) => streamingInbox.positionAt(BigInt(count)).rollingHash.toString();
     const preflightTotals = () =>
       publisher.validateCheckpointHeaderAndInbox.mock.calls.map(call => call[1].expectedTotal);

yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.ts

@@ -1204,7 +1204,7 @@ export class CheckpointProposalJob implements Traceable {
     checkpointBuilder: CheckpointBuilder,
     timestamp: bigint,
     blockProposalOptions: BlockProposalOptions,
-    streamingState?: StreamingCheckpointState,
+    streamingState: StreamingCheckpointState,
   ): Promise<BlockBuildingResult> {
     const blocksInCheckpoint: L2Block[] = [];
     const txHashesAlreadyIncluded = new Set<string>();
@@ -1250,25 +1250,23 @@ export class CheckpointProposalJob implements Traceable {
       // includes the block that reaches the per-checkpoint block cap, not just the timetable's last sub-slot.
       const maxBlocks = Math.min(this.config.maxBlocksPerCheckpoint, this.timetable.getMaxBlocksPerCheckpoint());
       const isCheckpointFinalBlock = timingInfo.isLastBlock || blocksBuilt + 1 >= maxBlocks;
-      const selection = streamingState
-        ? await this.selectStreamingBundle(streamingState, {
-            isFinalBlock: isCheckpointFinalBlock,
-            buildDeadline: timingInfo.deadline,
-          })
-        : undefined;
+      const selection = await this.selectStreamingBundle(streamingState, {
+        isFinalBlock: isCheckpointFinalBlock,
+        buildDeadline: timingInfo.deadline,
+      });
 
       // No checkpoint ending on the messages this block could consume can be published: stop before signing
       // anything more and give up the slot.
-      if (selection?.kind === 'abort') {
-        this.reportStreamingAbort(streamingState!, selection.reason, {
+      if (selection.kind === 'abort') {
+        this.reportStreamingAbort(streamingState, selection.reason, {
           blocksBuilt,
           blockNumber,
           ...selection.context,
         });
         return { aborted: true };
       }
 
-      const streamingBundle = selection?.range.messages;
+      const streamingBundle = selection.range.messages;
 
       const buildResult = await this.buildSingleBlock(checkpointBuilder, {
         // Create all blocks with the same timestamp
@@ -1327,11 +1325,8 @@ export class CheckpointProposalJob implements Traceable {
 
       // Streaming Inbox: the block built successfully, so advance the cursor to the prefix it consumed through and
       // sign that prefix as this block's reference. A block that consumed nothing re-signs the cursor's prefix.
-      let blockPrefixRef: InboxMessagePrefixRef | undefined = undefined;
-      if (streamingState && selection) {
-        streamingState.cursor = selection.range.end;
-        blockPrefixRef = InboxMessagePrefixRef.fromPosition(streamingState.cursor);
-      }
+      streamingState.cursor = selection.range.end;
+      const blockPrefixRef = InboxMessagePrefixRef.fromPosition(streamingState.cursor);
 
       // Sign the block proposal. This will throw if HA signing fails.
       const proposal = await this.createBlockProposal(
@@ -1378,7 +1373,6 @@ export class CheckpointProposalJob implements Traceable {
     // block rather than lose the slot; this is the only place the loop overrides the timetable. A cursor still at the
     // checkpoint start needs nothing: the parent checkpoint already ended on a live bucket end.
     if (
-      streamingState &&
       ranOutOfSubslots &&
       blocksInCheckpoint.length > 0 &&
       streamingState.cursor.totalMessageCount > streamingState.checkpointStartTotalMsgCount
@@ -1524,7 +1518,7 @@ export class CheckpointProposalJob implements Traceable {
     block: L2Block,
     usedTxs: Tx[],
     blockProposalOptions: BlockProposalOptions,
-    inboxPrefixRef?: InboxMessagePrefixRef,
+    inboxPrefixRef: InboxMessagePrefixRef,
   ): Promise<BlockProposal | undefined> {
     if (this.config.fishermanMode) {
       this.log.info(`Skipping block proposal for block ${block.number} in fisherman mode`);
@@ -1537,8 +1531,8 @@ export class CheckpointProposalJob implements Traceable {
       block.archive.root,
       usedTxs,
       this.proposer,
-      blockProposalOptions,
       inboxPrefixRef,
+      blockProposalOptions,
     );
   }
 
@@ -2320,10 +2314,7 @@ export class CheckpointProposalJob implements Traceable {
    * and fee analysis only, and pushing them to the archiver causes spurious reorg cascades
    * whenever the real proposer's block arrives from L1.
    */
-  private async syncProposedBlockToArchiver(
-    block: L2Block,
-    inboxPrefixRef: InboxMessagePrefixRef | undefined,
-  ): Promise<void> {
+  private async syncProposedBlockToArchiver(block: L2Block, inboxPrefixRef: InboxMessagePrefixRef): Promise<void> {
     if (this.config.skipPushProposedBlocksToArchiver || this.config.fishermanMode) {
       this.log.warn(`Skipping push of proposed block ${block.number} to archiver`, {
         blockNumber: block.number,
@@ -2332,11 +2323,7 @@ export class CheckpointProposalJob implements Traceable {
       return;
     }
     // The archiver re-validates this reference against its own messages inside the insert transaction, which is what
-    // stops a block built before an L1 reorg from landing after the reorg pruned the chain it belongs to. Streaming
-    // block building always produces one, so a missing reference here is a wiring bug, not a compatibility case.
-    if (inboxPrefixRef === undefined) {
-      throw new Error(`Streaming inbox: proposed block ${block.number} has no signed Inbox prefix reference`);
-    }
+    // stops a block built before an L1 reorg from landing after the reorg pruned the chain it belongs to.
     this.log.debug(`Syncing proposed block ${block.number} to archiver`, {
       blockNumber: block.number,
       slot: block.header.globalVariables.slotNumber,