Skip to content

[multicast] Bitmap-based multicast replication for sidecar-lite - #152

Open
zeeshanlakhani wants to merge 1 commit into
mainfrom
zl/multicast
Open

zeeshanlakhani wants to merge 1 commit into
mainfrom
zl/multicast

Conversation

@zeeshanlakhani

@zeeshanlakhani zeeshanlakhani commented Mar 23, 2026 •

Copy link
Copy Markdown

This adds multicast replication to sidecar-lite, emulating DPD's sidecar.p4 on dendrite @ zl/mcast-swadm, following RFD 488.

We manage 2 bitmaps per underlay group (external and underlay), for which replication is driven by p4rs replicate (dep @ p4#240). Ingress zeroes out at most one of these bitmaps determined by the geneve multicast option tag (tag 0 zeroes out underlay; tag 1 zeroes out external), and merges both.

Tag 0 (External) also decaps upon ingress. With no tag given, as on external NAT ingress packets, both bitmaps remain in place.

Only geneve-encapsulated multicast commits to replication. Multicast packets that miss ingress NAT drop on the empty bitmap, matching DPD's drop_mcastv{4,6}_no_group semantics. Packet copies that remain encapsulated lose a hop on the outer IPv6 header, as modelled in MulticastRouter6.

External multicast is NAT-encapsulated into the underlay via nat_v{4,6_mcast(_untagged), reusing forward_to_sled. Tagged and untagged variants exist as separate tables since x4c cannot key directly on header validity. The encap piece now zeroes out the outer src, traffic class, flow label, and inner eth src, like DPD's add_encap_headers.

Egress decap checks ff04::/64 and tag=2, matching DPD's egress mcast_tag_check, with per-port decap from a RID-keyed bitmap (tbl_decap_ports) and optional VLAN. tag=2 decapsulation also rejects inner unicast. tag=0 copies (already decapped in ingress) use the table for the VLAN. Dst MAC derivation (RFC 1112, RFC 2464) and per-port src MAC rewriting run on every packet copy.

Includes:

  • The parser rejects ff01 addresses outright, as well as ff02 destined packets with a hop limit of 0 (ff02 does still go to the scrimlet).
  • Multicast packets with a TTL/hop_limit <= 1 get dropped in ingress after the fwd_from_scrimlet return, letting scrimlet transmissions pass through.
  • Both decapsulation paths drop inner TTL <= 1. IPv4 checksums use RFC 1624 (eqn. 3): a checksum of 0xfeff turns into 0x0000 after decrementing the TTL (XOR used here due to ~ not being parsable by x4c).
  • scadm gets add/remove/dump for replication (v6), source filters, mcast NAT, decap ports, src MAC, and asic-id-to-port (for logical mapping).
  • Unicast modifications:
    • The router drops TTL/hop_limit <= 1 and decrements remaining packets on routes to nonzero ports, updating the IPv4 checksum.
      • Port 0 (the scrimlet) skips both the TTL check and decrement.
      • DPD's Router4/6 still decrement service-bound packets.
    • The NAT encap zeroing described above is shared with unicast NAT.
    • Packets with a parsed Geneve header skip ingress NAT, including unicast.

References

This work depends on the P4(rs) work in #240 and softnpu's zl/multicast.

@zeeshanlakhani
zeeshanlakhani marked this pull request as ready for review March 24, 2026 00:40
@zeeshanlakhani zeeshanlakhani changed the title [multicast] adds multicast replication for sidecar-lite [multicast] Bitmap-based multicast replication for sidecar-lite Mar 25, 2026
@zeeshanlakhani
zeeshanlakhani marked this pull request as draft March 26, 2026 15:54
@zeeshanlakhani
zeeshanlakhani marked this pull request as ready for review March 30, 2026 05:48
zeeshanlakhani added a commit to oxidecomputer/dendrite that referenced this pull request Mar 30, 2026
This wires up the softnpu ASIC backend to support multicast end-to-end by
translating DPD's sidecar.p4 table operations into sidecar-lite.p4's
simplified P4 pipeline.

## AsicMulticastOps

We replace the stubbed AsicMulticastOps implementation (which returned
"OperationUnsupported" for group creation and port addition with in-memory group
tracking via McGroupData, following the tofino_stub pattern.

Group membership is used by the table translation layer to build port
bitmaps for sidecar-lite's Replicate extern. Ports >= 128 are
rejected at add time to match sidecar-lite's 128-bit bitmap width.

## Table translation (asic/src/softnpu/table.rs)

We map sidecar.p4 table names to sidecar-lite equivalents and translate
action parameters where the designs differ for emulation.

All multicast action arms are gated with #[cfg(feature = "multicast")].

## References

- [softnpu #183](oxidecomputer/softnpu#183)
- [propolis #1093](oxidecomputer/propolis#1093)
- [p4rs #240](oxidecomputer/p4#240)
- [sidecar-lite #152](oxidecomputer/sidecar-lite#152)
- tokio: 1.50 (due to softnpu)
- oxide-tokio-rt: 0.1.3 (following-up from tokio's move to 1.50)
zeeshanlakhani added a commit to oxidecomputer/dendrite that referenced this pull request Apr 1, 2026
This wires up the softnpu ASIC backend to support multicast end-to-end by
translating DPD's sidecar.p4 table operations into sidecar-lite.p4's
simplified P4 pipeline.

## AsicMulticastOps

We replace the stubbed AsicMulticastOps implementation (which returned
"OperationUnsupported" for group creation and port addition with in-memory group
tracking via McGroupData, following the tofino_stub pattern.

Group membership is used by the table translation layer to build port
bitmaps for sidecar-lite's Replicate extern. Ports >= 128 are
rejected at add time to match sidecar-lite's 128-bit bitmap width.

## Table translation (asic/src/softnpu/table.rs)

We map sidecar.p4 table names to sidecar-lite equivalents and translate
action parameters where the designs differ for emulation.

All multicast action arms are gated with #[cfg(feature = "multicast")].

## References

- [softnpu #183](oxidecomputer/softnpu#183)
- [propolis #1093](oxidecomputer/propolis#1093)
- [p4rs #240](oxidecomputer/p4#240)
- [sidecar-lite #152](oxidecomputer/sidecar-lite#152)
- tokio: 1.50 (due to softnpu)
- oxide-tokio-rt: 0.1.3 (following-up from tokio's move to 1.50)
zeeshanlakhani added a commit to oxidecomputer/dendrite that referenced this pull request Apr 28, 2026
This wires up the softnpu ASIC backend to support multicast end-to-end by translating DPD's sidecar.p4 table operations into sidecar-lite.p4's simplified P4 pipeline.

## AsicMulticastOps

We replace the stubbed AsicMulticastOps implementation (which returned "OperationUnsupported" for group creation and port addition with in-memory group tracking via McGroupData, following the tofino_stub pattern.

Group membership is used by the table translation layer to build port bitmaps for sidecar-lite's Replicate extern. Ports >= 128 are rejected at add time to match sidecar-lite's 128-bit bitmap width.

## Table translation (asic/src/softnpu/table.rs)

We map sidecar.p4 table names to sidecar-lite equivalents and translate action parameters where the designs differ for emulation.

## Fix update semantics

We fix an issue with stale in-memory sources after update with absent source list(s). `modify_group_external` wrote back `new.sources.or(existing)` to the in-memory record while `update_external_tables` in the same call canonicalized `new.sources` directly to rewrite the P4 source-filter entries. A `None` update therefore cleared the P4 table but left a stale `Some([...])` in memory.

All multicast action arms are gated with #[cfg(feature = "multicast")].



## References and Deps

- [softnpu #183](oxidecomputer/softnpu#183)
- [propolis #1093](oxidecomputer/propolis#1093)
- [p4rs #240](oxidecomputer/p4#240)
- [sidecar-lite #152](oxidecomputer/sidecar-lite#152)
- tokio: 1.50 (due to softnpu)
- oxide-tokio-rt: 0.1.3 (following-up from tokio's move to 1.50)
- bump oxide-tokio-rt to 0.1.4, tokio to 1.52.1 (matching Omicron)
@zeeshanlakhani zeeshanlakhani self-assigned this May 7, 2026
@zeeshanlakhani
zeeshanlakhani force-pushed the zl/multicast branch 2 times, most recently from 8757f58 to bcceb2d Compare June 4, 2026 03:42
This adds multicast replication to sidecar-lite, emulating DPD's
sidecar.p4 on dendrite @ zl/mcast-swadm, following RFD 488.

We manage 2 bitmaps per underlay group (external and underlay), for
which replication is driven by p4rs `replicate` (dep @ p4#240). Ingress
zeroes out at most one of these bitmaps determined by the geneve
multicast option tag (tag 0 zeroes out underlay; tag 1 zeroes out
external), and merges both.

Tag 0 (External) also decaps upon ingress. With no tag given, as on
external NAT ingress packets, both bitmaps remain in place.

Only geneve-encapsulated multicast commits to replication. Multicast
packets that miss ingress NAT drop on the empty bitmap, matching DPD's
`drop_mcastv{4,6}_no_group` semantics. Packet copies that
remain encapsulated lose a hop on the outer IPv6 header, as modelled in
`MulticastRouter6`.

External multicast is NAT-encapsulated into the underlay via
`nat_v{4,6}_mcast(_untagged)`, reusing `forward_to_sled`. Tagged and
untagged variants exist as separate tables since x4c cannot key
directly on header validity. The encap piece now zeroes out the outer
src, traffic class, flow label, and inner eth src, like DPD's
`add_encap_headers`.

Egress decap checks `ff04::/64` and `tag=2`, matching DPD's egress
`mcast_tag_check`, with per-port decap from a RID-keyed bitmap
(`tbl_decap_ports`) and optional VLAN. `tag=2` decapsulation also
rejects inner unicast. `tag=0` copies (already decapped in ingress)
use the table for the VLAN. Dst MAC derivation (RFC 1112, RFC 2464)
and per-port src MAC rewriting run on every packet copy.

Includes:

- The parser rejects `ff01` addresses outright, as well as `ff02` destined
  packets with a hop limit of 0 (`ff02` *does* still go to the scrimlet).
- Multicast packets with a TTL/hop_limit <= 1 get dropped in ingress
  after the `fwd_from_scrimlet` return, letting scrimlet transmissions
  pass through.
- Both decapsulation paths drop inner TTL <= 1. IPv4 checksums use
  RFC 1624 (eqn. 3): a checksum of 0xfeff turns into 0x0000 after
  decrementing the TTL (XOR used here due to `~` not being parsable by x4c).
- scadm gets add/remove/dump for replication (v6), source filters, mcast
  NAT, decap ports, src MAC, and asic-id-to-port (for logical mapping).
- Unicast modifications:
  - The router drops TTL/hop_limit <= 1 and decrements remaining packets
    on routes to nonzero ports, updating the IPv4 checksum.
    - Port 0 (the scrimlet) skips both the TTL check and decrement.
    - DPD's Router4/6 still decrement service-bound packets.
  - The NAT encap zeroing described above is shared with unicast NAT.
  - Packets with a parsed Geneve header skip ingress NAT, including
    unicast.

### References

This work depends on the P4(rs) work in
[#240](oxidecomputer/p4#240) and softnpu
zl/multicast.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant