[multicast] Bitmap-based replication and bit-slice assignment for softnpu/a4x2 by zeeshanlakhani · Pull Request #240 · oxidecomputer/p4

zeeshanlakhani · 2026-03-17T16:09:43Z

This work includes and/or incorporates:

Bitmap-based packet replication for softnpu: a P4 program can make use of a port_bitmap field in its egress metadata and set the field during ingress (via table action, metadata assignment, etc.):
```
struct egress_metadata_t {
    ...
    bit<128> port_bitmap;
}

action set_port_bitmap(bit<128> bitmap) {
    egress.port_bitmap = bitmap;
}
```
- This bitmap is read after ingress, ahead of broadcast/unicast (drop > non-zero bitmap > broadcast > unicast), and it'll show up in dtrace metadata dumps for debugging, just like the other fields.
- Checks have been added as well, including a check_port_bitmap requiring a bit<N>.
- The p4rs::replicate() function skips the ingress port and bits at or above the given radix. If a non-zero bitmap filters down to no ports, the packet gets dropped. A zeroed bitmap falls back to broadcast/unicast handling. Bit N (value 2^N) is port N, which is the same encoding 128w1 << port produces via shl_le.
- Each selected port runs egress with its own copy of headers and metadata.
The pipeline codegen no longer hardcodes the ingress_metadata_t/egress_metadata_t types, which are now derived from the control parameters. The generated locals use fixed names (i.e., __ingress_metadata, __egress_metadata) to prevent P4 parameter names from shadowing pipeline locals.
Fixtures have been updated, including the in-repo sidecar-lite example gaining a mcast_replication_v6 table whose set_port_bitmap action drives port_bitmap.
- The mcast fixture demonstrates dendrite-style MAC splitting, i.e. unicast in the ingress pipeline and replicas in the egress pipeline.
Shift operators (<<, >>). Binary expressions follow operator precedence, and operators at equal precedence associate left to right. If the shift count is at least the left operand's bit width, the result is 0.
Bit-slice assignment (lval[hi:lo] = expr, P4-16 8.6) is added, and slice reads are fixed, both with byte-reversal-aware ranges and an arithmetic fallback when byte reversal makes slicing non-contiguous.
Fixes and tightening around unsigned comparisons, owned operands for slices in bit arithmetic, and the ^ marker's handling in diagnostics.
- Note: only bit<W> slices are lowered, whereas varbit slicing results in a language error (8.10), and int<W>/compile-time int slicing are reported as SoftNPU/x4c restrictions.
bit<W> -> bit<V> widening keeps the existing value (for sub-byte widths too).
- Implicit bit widening is an x4c extension. P4-16 8.11 requires explicit casts between different bit widths, but x4c does not parse explicit casts as of yet.
New HLIR checks on initializers, assignments, and slice bounds: runtime (unsized) int is rejected, and x4c/SoftNPU limitations get bubbled up instead of looking just like type errors.

…protocol This extends the management message protocol with operations for multicast group lifecycle: create, destroy, add port, remove port, and list. These dispatch to the corresponding [p4rs](https://github.com/oxidecomputer/p4) Pipeline trait methods (add_mcast_group, remove_mcast_group, add_mcast_port, etc.) that were added to support bifurcated multicast replication in the p4rs codegen. Mutating operations (create, destroy, add, remove) are fire-and-forget, matching the existing TableAdd/TableRemove pattern. `MulticastGroupList` is the only round-trip operation, returning a sorted group-to-ports mapping for use by dendrite's `AsicMulticastOps` queries. The `MulticastPortAdd` struct carries rid and level1_excl_id for API parity with dendrite's AsicMulticastOps trait, though softnpu does not use them. Tofino's PRE handles per-replica identification and exclusion via these fields. softnpu handles the equivalent via `McastReplicationTag` in the codegenning. This branch depends on the related oxidecomputer/p4#240.

This adds multicast replication to sidecar-lite, emulating DPD's sidecar.p4 We manage 2 bitmaps per multicast destination (external, underlay), in which replication is driven via the `Replicate` extern (from p4rs #240). The ingress control block suppresses (or doesn't) these bitmaps based on the geneve multicast option tag, and then merges both into a single replication bitmap. This tag is read-only, matching DPD's sidecar.p4. OPTE (would) stamp it before the packet reaches the switch. Egress decap is gated on the reserved underlay subnet (ff04::/64, per RFC 7346 admin-local scope + Omicron) and tag=2 (Both), matching DPD's egress mcast_tag_check. Ports in the decap table get geneve stripped, others keep encapsulation for OPTE to handle on the receiving sled. Multicast dst MAC derivation (RFC 1112 for IPv4, RFC 2464 for IPv6) and per-port source MAC rewrite run on every replicated copy. Egress validates the outer IP multicast range before any mcast processing. ### References This work depends on the P4(rs) work in [#240](oxidecomputer/p4#240).

…tnpu/a4x2 ## Replication This introduces bitmap-based packet replication for softnpu. Replication is opt-in via a Replicate extern that P4 programs declare and call explicitly: ```p4 extern Replicate { void replicate(in bit<128> bitmap); } Replicate() rep; rep.replicate(egress.bitmap_a | egress.bitmap_b); ``` The codegen scans the AST for the extern call, extracts the bitmap expression, and generates the replication loop at the pipeline level between ingress and egress. The call is elided from generated code after compile-time validation of the argument. `p4rs::replicate()` collects set bits from the bitmap expression, filtering out the ingress port to prevent self-replication. It interprets bitmaps in little-endian integer order: bit N (value 2^N) corresponds to port N. This matches the encoding produced by P4 shifting (`128w1 << port`) via `shl_le`, so replication bitmaps and shift-based bitmap checks use the same convention. The pipeline codegen no longer hardcodes metadata struct names (`ingress_metadata_t`, `egress_metadata_t`), deriving variable names and types from P4 control parameter declarations. Generated P4 structs now initialize `bit<N>` fields to properly-sized zero bitvecs (`bitvec![u8, Msb0; 0; N]`) instead of empty `BitVec` (length 0). This fixes bitmap comparisons and arithmetic on uninitialized metadata fields. ## Shift operators Adds left shift ("<<") and right shift (">>") support across the full compiler pipeline: lexer, parser, AST, HLIR, type checker, and codegen. The lexer previously tokenized "<<" but was not wired through the parser or AST (yet). ## Bit-slice assignment Adds `Statement::SliceAssignment` for P4-16 spec 8.6 `lval[hi:lo] = expr` syntax, including parser updates, HLIR bounds validation, type checking (RHS width must equal hi - lo + 1), and codegen with byte-reversal-aware bitvec range mapping. Non-contiguous slices after byte reversal fall back to arithmetic extraction. Slice reads assigned to local variables (e.g., `bit<16> lo = x[15:0]`) produce owned BitVec values via `.to_bitvec()`. ## Slice codegen handling - Fixes a latent issue where the slice-to-bitvec mapping ignored header byte reversal, producing incorrect ranges for sub-byte slices on multi-byte fields (e.g., field[31:28] on bit<32>). Fixes Varbit/Int slice reads, which were rejected due to swapped destructure naming. - Fixes single-bit slices (x[n:n]) rejected in the read context. ## Bitwise operators Clones operands for BitOr, BitAnd, Xor, and Mask in the Expression codegen. The previous generated code moved out of mutable references for BitVec operands, which does not implement Copy. ## Tests - Bitmap replication (port selection, self-replication filtering, empty bitmap, broadcast precedence), emulating multicast concepts. - Slice assignment with RFC 1112 MAC derivation and same-field aliasing. - Shift operators and width resizing. - Sub-byte slice reads verify byte-reversal correctness.

This wires up the softnpu ASIC backend to support multicast end-to-end by translating DPD's sidecar.p4 table operations into sidecar-lite.p4's simplified P4 pipeline. ## AsicMulticastOps We replace the stubbed AsicMulticastOps implementation (which returned "OperationUnsupported" for group creation and port addition with in-memory group tracking via McGroupData, following the tofino_stub pattern. Group membership is used by the table translation layer to build port bitmaps for sidecar-lite's Replicate extern. Ports >= 128 are rejected at add time to match sidecar-lite's 128-bit bitmap width. ## Table translation (asic/src/softnpu/table.rs) We map sidecar.p4 table names to sidecar-lite equivalents and translate action parameters where the designs differ for emulation. All multicast action arms are gated with #[cfg(feature = "multicast")]. ## References - [softnpu #183](oxidecomputer/softnpu#183) - [propolis #1093](oxidecomputer/propolis#1093) - [p4rs #240](oxidecomputer/p4#240) - [sidecar-lite #152](oxidecomputer/sidecar-lite#152) - tokio: 1.50 (due to softnpu) - oxide-tokio-rt: 0.1.3 (following-up from tokio's move to 1.50)

This wires up the softnpu ASIC backend to support multicast end-to-end by translating DPD's sidecar.p4 table operations into sidecar-lite.p4's simplified P4 pipeline. ## AsicMulticastOps We replace the stubbed AsicMulticastOps implementation (which returned "OperationUnsupported" for group creation and port addition with in-memory group tracking via McGroupData, following the tofino_stub pattern. Group membership is used by the table translation layer to build port bitmaps for sidecar-lite's Replicate extern. Ports >= 128 are rejected at add time to match sidecar-lite's 128-bit bitmap width. ## Table translation (asic/src/softnpu/table.rs) We map sidecar.p4 table names to sidecar-lite equivalents and translate action parameters where the designs differ for emulation. ## Fix update semantics We fix an issue with stale in-memory sources after update with absent source list(s). `modify_group_external` wrote back `new.sources.or(existing)` to the in-memory record while `update_external_tables` in the same call canonicalized `new.sources` directly to rewrite the P4 source-filter entries. A `None` update therefore cleared the P4 table but left a stale `Some([...])` in memory. All multicast action arms are gated with #[cfg(feature = "multicast")]. ## References and Deps - [softnpu #183](oxidecomputer/softnpu#183) - [propolis #1093](oxidecomputer/propolis#1093) - [p4rs #240](oxidecomputer/p4#240) - [sidecar-lite #152](oxidecomputer/sidecar-lite#152) - tokio: 1.50 (due to softnpu) - oxide-tokio-rt: 0.1.3 (following-up from tokio's move to 1.50) - bump oxide-tokio-rt to 0.1.4, tokio to 1.52.1 (matching Omicron)

This adds multicast replication to sidecar-lite, emulating DPD's sidecar.p4 We manage 2 bitmaps per multicast destination (external, underlay), in which replication is driven via the `Replicate` extern (from p4rs #240). The ingress control block suppresses (or doesn't) these bitmaps based on the geneve multicast option tag, and then merges both into a single replication bitmap. This tag is read-only, matching DPD's sidecar.p4. OPTE (would) stamp it before the packet reaches the switch. Egress decap is gated on the reserved underlay subnet (ff04::/64, per RFC 7346 admin-local scope + Omicron) and tag=2 (Both), matching DPD's egress mcast_tag_check. Ports in the decap table get geneve stripped, others keep encapsulation for OPTE to handle on the receiving sled. Multicast dst MAC derivation (RFC 1112 for IPv4, RFC 2464 for IPv6) and per-port source MAC rewrite run on every replicated copy. Egress validates the outer IP multicast range before any mcast processing. Decap port selection uses a RID-keyed bitmap table (tbl_decap_ports), matching Dendrite's tbl_decap_ports + port_bitmap_check. The egress shifts 1 << egress.port and ANDs against the bitmap to determine per-port decap, with optional VLAN insertion (following the current sidecar in DPD). We augmented the IPv4 header checksum after TTL decrement to use ones-complement addition with carry fold via 16-to-32-bit widening. ### References This work depends on the P4(rs) work in [#240](oxidecomputer/p4#240).

The mainline PR points softnpu at main to merge in pieces. a4x2 needs bitmap-based replication from softnpu zl/multicast, so this branch carries the pin until oxidecomputer/p4#240 and oxidecomputer/softnpu#183 land.

rcgoodfellow · 2026-09-29T20:49:50Z

What does the Replicate extern get us that a simple bitmap in egress metadata does not? I think this could be implemented in a simpler way by just using the port_bitmap metadata field introduced in this PR and then implementing multicast replication similar to how broadcast replication is already done.

Having replication state in metadata as a field also makes things a bit more debuggable, as then it's a concrete value in the state of the pipeline that things like dtrace scripts can observe.

zeeshanlakhani · 2026-09-29T22:29:24Z

What does the Replicate extern get us that a simple bitmap in egress metadata does not? I think this could be implemented in a simpler way by just using the port_bitmap metadata field introduced in this PR and then implementing multicast replication similar to how broadcast replication is already done.

Having replication state in metadata as a field also makes things a bit more debuggable, as then it's a concrete value in the state of the pipeline that things like dtrace scripts can observe.

Yep. I'm actually taking this approach in my cleanup work that will go in soon. The original thinking was to match and model the PRE being a fixed-fucntion block on hardware with an extern, while also avoiding some extra magic field naming.

The other thing pushing me in this direction is that sidecar.p4 already does it this way. The mcast_grp_a/mcast_grp_b fields are metadata and get matched on downstream in mcast_tag_check, and the egress keys-on tbl_decap_ports for eg_intr_md.egress_rid, allowing later stages to read the state back and invalidate a group. The extern doesn't give us this ability.

I'll continue to move to the simpler mechanism.

Includes: - Per Ry's review, this drops the `Replicate` extern and the AST scanning that went with it for a `port_bitmap` field in egress metadata, read after ingress, ahead of broadcast/unicast. It'll now show up in dtrace metadata dumps for debugging, like other fields. Each selected port runs egress with its own copy of headers and metadata. - The example/test sidecar-lite is modified to remove its `Replicate` extern. - The mcast fixture now carries dendrite-style MAC splitting, i.e. unicast in the ingress pipeline and replicas in the egress pipeline, while the shift, slice, and metadata-names tests check and assert on packet bytes and `process_packet_headers` together. - Fixes and tightening around expression precedence, shift typing (>= width returns zero), unsigned comparisons, slice reads/writes, and operator carets in diagnostics. - bit<W> -> bit<V> widening keeps the existing value (for sub-byte widths too). - This is an x4c extension at the moment since there's no casting yet, and narrowing is now rejected. - Added HLIR checks on initializers, assignments, and slice bounds: runtime (unsized) int is rejected, and x4c/SoftNPU limitations get bubbled up instead of looking just like type errors.

zeeshanlakhani · 2026-10-13T01:09:29Z

@rcgoodfellow back up for review.

This cleanup does two things: - The checker now errors appropriately when a table's default action isn't listed in its actions list (P4-16 14.2.1.3). I've updated the fixtures/tests. - `dynamic_router_noaddr.p4` failed to compile (on `main` even) because its `local` action shadowed the `local` table, but nothing built it, and it went unnoticed. This is now fixed.

When adding new tests to sidecar-lite (based on this dep), I started seeing things like VLAN 300 coming out as 44 (and 256 as 0) on a decap action. VLAN 100 in the tests hid this issue because it fit into 1 byte. This work fixes the issue in `extract_bit_action_parameter`, which used `resize(n)` and dropped the low nibble of the high byte. We now shift the partial last byte into the high bits, the same layout as `store_le` and the header fields, which works for any width. Plus, we also include the latent codegen offset (`n >> 3` -> `div_ceil(8)`), short `parameter_data` now being ignored vs panicking, and updated harness checks (vid, timeout, extra frames).

This adds multicast replication to sidecar-lite, emulating DPD's sidecar.p4 on dendrite @ zl/mcast-swadm, following RFD 488. We manage 2 bitmaps per underlay group (external and underlay), for which replication is driven by p4rs `replicate` (dep @ p4#240). Ingress zeroes out at most one of these bitmaps determined by the geneve multicast option tag (tag 0 zeroes out underlay; tag 1 zeroes out external), and merges both. Tag 0 (External) also decaps upon ingress. With no tag given, as on external NAT ingress packets, both bitmaps remain in place. Only geneve-encapsulated multicast commits to replication. Multicast packets that miss ingress NAT drop on the empty bitmap, matching DPD's `drop_mcastv{4,6}_no_group` semantics. Packet copies that remain encapsulated lose a hop on the outer IPv6 header, as modelled in `MulticastRouter6`. External multicast is NAT-encapsulated into the underlay via `nat_v{4,6}_mcast(_untagged)`, reusing `forward_to_sled`. Tagged and untagged variants exist as separate tables since x4c cannot key directly on header validity. The encap piece now zeroes out the outer src, traffic class, flow label, and inner eth src, like DPD's `add_encap_headers`. Egress decap checks `ff04::/64` and `tag=2`, matching DPD's egress `mcast_tag_check`, with per-port decap from a RID-keyed bitmap (`tbl_decap_ports`) and optional VLAN. `tag=2` decapsulation also rejects inner unicast. `tag=0` copies (already decapped in ingress) use the table for the VLAN. Dst MAC derivation (RFC 1112, RFC 2464) and per-port src MAC rewriting run on every packet copy. Includes: - The parser rejects `ff01` addresses outright, as well as `ff02` destined packets with a hop limit of 0 (`ff02` *does* still go to the scrimlet). - Multicast packets with a TTL/hop_limit <= 1 get dropped in ingress after the `fwd_from_scrimlet` return, letting scrimlet transmissions pass through. - Both decapsulation paths drop inner TTL <= 1. IPv4 checksums use RFC 1624 (eqn. 3): a checksum of 0xfeff turns into 0x0000 after decrementing the TTL (XOR used here due to `~` not being parsable by x4c). - scadm gets add/remove/dump for replication (v6), source filters, mcast NAT, decap ports, src MAC, and asic-id-to-port (for logical mapping). - Unicast modifications: - The router drops TTL/hop_limit <= 1 and decrements remaining packets on routes to nonzero ports, updating the IPv4 checksum. - Port 0 (the scrimlet) skips both the TTL check and decrement. - DPD's Router4/6 still decrement service-bound packets. - The NAT encap zeroing described above is shared with unicast NAT. - Packets with a parsed Geneve header skip ingress NAT, including unicast. ### References This work depends on the P4(rs) work in [#240](oxidecomputer/p4#240) and softnpu zl/multicast.

zeeshanlakhani requested review from FelixMcFelix, rcgoodfellow and taspelund March 17, 2026 18:05

zeeshanlakhani marked this pull request as ready for review March 18, 2026 05:32

zeeshanlakhani mentioned this pull request Mar 18, 2026

[multicast] softnpu with bitmap-based replication oxidecomputer/softnpu#183

Open

1 task

zeeshanlakhani force-pushed the zl/multicast branch from fc3bde8 to e07a034 Compare March 19, 2026 17:25

zeeshanlakhani force-pushed the zl/multicast branch 2 times, most recently from a0ee3d2 to 0e20707 Compare March 20, 2026 16:55

zeeshanlakhani changed the title ~~[multicast] Add multicast replication support for (upcoming) softnpu/a4x2~~ [multicast] Add multicast replication and bit-slice assignment for softnpu/a4x2 Mar 20, 2026

zeeshanlakhani force-pushed the zl/multicast branch 2 times, most recently from c6c1858 to 3df53f6 Compare March 25, 2026 03:38

zeeshanlakhani changed the title ~~[multicast] Add multicast replication and bit-slice assignment for softnpu/a4x2~~ [multicast] Bitmap-based replication and bit-slice assignment for softnpu/a4x2 Mar 25, 2026

zeeshanlakhani force-pushed the zl/multicast branch 2 times, most recently from 3df53f6 to 3e15edd Compare March 25, 2026 15:06

zeeshanlakhani mentioned this pull request Mar 25, 2026

[multicast] Bitmap-based multicast replication for sidecar-lite oxidecomputer/sidecar-lite#152

Open

zeeshanlakhani force-pushed the zl/multicast branch from 3e15edd to 06dd1de Compare March 27, 2026 06:16

zeeshanlakhani force-pushed the zl/multicast branch from 06dd1de to 0e8a28a Compare March 27, 2026 12:38

zeeshanlakhani mentioned this pull request Mar 30, 2026

[multicast] wire up softnpu backend to multicast table ops oxidecomputer/dendrite#242

Merged

zeeshanlakhani mentioned this pull request Jul 3, 2026

[multicast] default the multicast feature and wire SoftNPU end-to-end oxidecomputer/dendrite#224

Closed

zeeshanlakhani mentioned this pull request Aug 25, 2026

[multicast] Host-side plumbing for externally sourced multicast oxidecomputer/voxel#28

Closed

zeeshanlakhani marked this pull request as draft September 5, 2026 03:44

zeeshanlakhani added 2 commits September 9, 2026 01:10

[wip] bug handling and affirming top of function for replicate

c9031ee

[merge] merge main

5a2a481

zeeshanlakhani self-assigned this Oct 12, 2026

zeeshanlakhani marked this pull request as ready for review October 13, 2026 01:08

zeeshanlakhani added 2 commits October 13, 2026 01:37

zeeshanlakhani mentioned this pull request Oct 15, 2026

[softnpu, mcast] wire up the softnpu backend for multicast table ops oxidecomputer/dendrite#405

Open

Navigation Menu

Repository navigation

[multicast] Bitmap-based replication and bit-slice assignment for softnpu/a4x2 - #240

[multicast] Bitmap-based replication and bit-slice assignment for softnpu/a4x2#240
zeeshanlakhani wants to merge 6 commits into
mainfrom
zl/multicast

zeeshanlakhani commented Mar 17, 2026 •

edited

Loading

Uh oh!

rcgoodfellow commented Sep 29, 2026 •

edited

Loading

Uh oh!

zeeshanlakhani commented Sep 29, 2026

Uh oh!

zeeshanlakhani commented Oct 13, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

zeeshanlakhani commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rcgoodfellow commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zeeshanlakhani commented Sep 29, 2026

Uh oh!

zeeshanlakhani commented Oct 13, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zeeshanlakhani commented Mar 17, 2026 •

edited

Loading

rcgoodfellow commented Sep 29, 2026 •

edited

Loading