Packet replication and VXLAN forwarding with XDP devmaps
This article is translation for Japanese my article takeio.hatenablog.com
Motivation
I wrote this article after reading XDP_CLONE: Efficient In-Driver Packet Duplication for eBPF-Accelerated Systems, a paper presented at eBPF '26.
In Section 1, on page 67, the authors describe packet replication with bpf_redirect_map() and BPF_F_BROADCAST.
They then make this claim about modifying the copies:
the program has no way to rewrite each copy independently
Section 3 repeats the claim that the replicated packets cannot be modified.
That claim is incorrect. With devmap broadcast and a devmap egress program, XDP can already replicate packets and rewrite each copy independently.
BUM stands for broadcast, unknown unicast, and multicast. For example, an L2 network built with VXLAN or SRv6 may distribute BUM traffic to several peers over unicast. That requires a different outer destination for each copy, while keeping the original Ethernet frame unchanged. I use VXLAN as the example here.
I already knew XDP could do this, so the paper made me suspect that this use of devmaps was not widely known.
When I asked an AI assistant, it suggested classifying packets in XDP and replicating them with bpf_clone_redirect() in TC.
That works too, but TC operates on skbs, so I wanted to compare its performance with replication handled entirely in XDP.
I decided to explain the existing XDP approach with working code and include measurements against an XDP + TC version.
For the code, see my goxdp-template repository. It started as a template for writing XDP programs and has gradually become a collection of runnable examples. It includes the VXLAN implementation used in this article, which may help when implementing BUM forwarding or giving an AI assistant concrete code to work with. If you find it useful, I'd appreciate a star.
Let's start with the simplest case: replicating a received frame without changing it.
How does basic broadcast work in XDP?
To broadcast a received frame, replicate it to the output interfaces registered in a devmap.
Create a BPF_MAP_TYPE_DEVMAP map and redirect to it with BPF_F_BROADCAST:
#include <linux/bpf.h> #include <linux/if_ether.h> #include <bpf/bpf_helpers.h> struct { __uint(type, BPF_MAP_TYPE_DEVMAP); __uint(max_entries, 64); __type(key, __u32); __type(value, __u32); } tx_ports SEC(".maps"); SEC("xdp") int xdp_broadcast(struct xdp_md* ctx) { void* data = (void*)(long)ctx->data; void* data_end = (void*)(long)ctx->data_end; if (data + sizeof(struct ethhdr) > data_end) return XDP_DROP; return bpf_redirect_map(&tx_ports, 0, BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS); } char __license[] SEC("license") = "Dual MIT/GPL";
User space populates the slots in tx_ports with the output interface indices.
With two outputs other than the ingress interface, the kernel sends one identical copy to each.
This program leaves the frame's MAC addresses, IP addresses, and TTL unchanged.
The two flags have different roles:
BPF_F_BROADCASTtells the kernel to replicate the frame to the outputs registered in the map. The key passed as the second argument is unused in this mode.BPF_F_EXCLUDE_INGRESSexcludes the receiving interface from the outputs. Even if that interface belongs to the group, the kernel does not send a copy back to it.
Broadcast here means replication to the outputs in the map. The flag does not change the Ethernet destination MAC address, so it also works with a unicast input frame. Separate devmaps for separate groups can provide multicast forwarding with a selected set of outputs. The kernel's devmap documentation describes this use.
Applying this to VXLAN requires choosing the replication targets and rewriting each copy. A VNI identifies a logical VXLAN network, and a VTEP is a VXLAN tunnel endpoint. Below, Outer means the outer VXLAN encapsulation headers, and Inner means the encapsulated Ethernet frame.
- How do we select the output ports for each VNI?
- Putting all ports in one devmap would also send copies to ports belonging to other VNIs.
- How do we change the VXLAN destination for each copy?
- Delivering BUM traffic to several VTEPs over unicast requires an identical Inner and a different Outer destination for each copy.
Let's work through these two questions in order.
How do we select the output ports for each VNI?
Create a devmap for each VNI and select the one matching the received packet's VNI. This limits replication to that VNI's members. To select the devmap, use a map-in-map, whose values reference other maps.
A frame in VNI 100 should reach the members of VNI 100, and a frame in VNI 200 should reach the members of VNI 200.
Select the devmap before calling bpf_redirect_map() with BPF_F_BROADCAST:
- The outer map,
vni_groups, is aBPF_MAP_TYPE_HASH_OF_MAPSkeyed by VNI. - Each inner map is a
BPF_MAP_TYPE_DEVMAP_HASHcontaining only that VNI's members.- This example uses the output ifindex as the inner key and stores the output ifindex and egress program in the value.
The map definitions are in xdp_vxlan.c. The Go implementation of UpdateMembers() creates and registers the per-VNI devmaps.
During forwarding, look up the VNI's devmap, then broadcast through it:
void *members = bpf_map_lookup_elem(&vni_groups, &vni); if (!members) return XDP_DROP; return bpf_redirect_map(members, 0, BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS);
This call sends copies to the outputs registered in the devmap referenced by members.
Although the redirect call ignores its key argument, selecting the map by VNI has already restricted the outputs.
The kernel's map-in-map documentation explains how to obtain a reference to an inner map.
In the diagram below, access1 and access2 face endpoints, while uplink0 carries VXLAN traffic.
The outer map references a separate devmap for each VNI.
Registering uplink0 in both maps does not create a path from VNI 100 to access2.
flowchart LR
accTitle: VNI-scoped BUM replication with map-in-map
accDescr: The outer vni_groups hash maps VNI 100 and VNI 200 to separate devmaps. Each devmap contains one access port and the shared uplink0. Broadcast uses only the selected devmap, so the shared uplink does not merge the VNI groups. The arrows show map references and membership.
subgraph outer["vni_groups: HASH_OF_MAPS"]
v100["key: VNI 100"]
v200["key: VNI 200"]
end
v100 -->|map reference| a["DEVMAP_HASH A<br/>VNI 100 members"]
v200 -->|map reference| b["DEVMAP_HASH B<br/>VNI 200 members"]
a --> access1["access1<br/>VNI 100"]
a --> uplink["uplink0<br/>shared tunnel interface"]
b --> uplink
b --> access2["access2<br/>VNI 200"]
classDef default fill:#ffffff,stroke:#666666,color:#222222;
class v100,v200,a,b map;
classDef map fill:#edf4ee,stroke:#7b9180,color:#222222;
The same output ifindex can belong to several per-VNI devmaps without merging their replication groups.
For example, VNI 100 and VNI 200 can share uplink0 while retaining separate access ports.
The destination VTEP on that uplink can differ by VNI, so choosing a port also requires rewriting the Outer headers.
How do we change the VXLAN destination for each copy?
Use an egress program associated with the devmap entry to rewrite each copy. In the basic example, each entry contained only an output ifindex, and the kernel sent the frame unchanged. Now register an XDP program alongside that ifindex to run before transmission. This is a devmap egress program.
The ingress XDP program still requests replication with bpf_redirect_map() and BPF_F_BROADCAST.
After distributing the frame to the outputs, the kernel runs the egress program registered in each output's entry before sending that copy.
The program can rewrite the Outer destination there.
The same egress program can serve several entries.
It reads ctx->egress_ifindex to identify the output interface for the current copy.
Consider two output interfaces named p2 and p3.
When the kernel replicates a frame in one VNI to both outputs, this field contains p2's ifindex for the p2 copy and p3's ifindex for the p3 copy.
The program can use that value to set the Outer destination IP to 192.0.2.2 for p2 and 192.0.3.2 for p3.
Rewriting the p2 copy does not affect the p3 copy.
Change the devmap value type from __u32 to struct bpf_devmap_val, which holds both the ifindex and the program.
Its bpf_prog.fd field takes the file descriptor of an XDP program with attach type BPF_XDP_DEVMAP.
On the BPF side, define that program with SEC("xdp/devmap").
In Go, the map value consists of two 32-bit values: the output ifindex and the program FD.
Here, members is the selected VNI's devmap, and egress is the loaded egress program:
value := [2]uint32{uint32(ifindex), uint32(egress.FD())} if err := members.Update(value[0], value, ebpf.UpdateAny); err != nil { return err }
The sample performs this registration in UpdateMembers().
Existing kernels already support running a separate XDP program on the replicated frames.
For the replication and program execution, see dev_map_enqueue_multi() and dev_map_bpf_prog_run() in Linux 7.3.
This implementation looks up the peer VTEP by VNI and ctx->egress_ifindex, then rewrites the Outer source and destination addresses.
The following excerpt from vxlan_rewrite() runs after header validation and the peer lookup:
__builtin_memcpy(eth->h_source, port.mac, ETH_ALEN); __builtin_memcpy(eth->h_dest, target.mac, ETH_ALEN); ip->saddr = port.local_ip; ip->daddr = target.remote_ip; checksum(ip); return XDP_PASS;
Returning XDP_PASS from a devmap egress program continues transmission through the output selected by the devmap.
Unlike XDP_PASS at ingress, it does not pass the packet to the Linux network stack.
Returning XDP_DROP discards only the copy for that output.
The program rewrites only Outer; Inner stays identical across the copies. The UDP and VXLAN headers can also share common values. The Outer UDP source port comes from an Inner flow hash, keeping it stable within a flow while allowing different flows to use different values.
Each tunnel egress adds Outer after replication and sends the same Inner to a different VTEP:
flowchart TB
accTitle: VXLAN replication with per-output rewriting
accDescr: One inner frame is replicated to two tunnel outputs and one access output. Tunnel copies get different outer IP addresses while the inner frame remains identical.
inner["Inner F"] --> prepare["Prepare Outer template<br/>header_cache[key]"]
prepare --> group["DEVMAP broadcast<br/>Inner F + metadata"]
group --> b["p2: egress rewrite<br/>attach Outer"]
group --> c["p3: egress rewrite<br/>attach Outer"]
group --> access["Access egress<br/>remove metadata"]
b --> packetB["Outer dst: 192.0.2.2<br/>UDP / VXLAN / Inner F"]
c --> packetC["Outer dst: 192.0.3.2<br/>UDP / VXLAN / Inner F"]
access --> local["Access output<br/>Inner F"]
classDef default fill:#ffffff,stroke:#666666,color:#222222;
class prepare,b,c,access process;
classDef process fill:#edf3fa,stroke:#74849a,color:#222222;
class group map;
classDef map fill:#edf4ee,stroke:#7b9180,color:#222222;
This sample configures one unicast peer for each (VNI, output ifindex) pair. It therefore uses different output interfaces to replicate traffic in the same VNI to several peers. That is a constraint of this sample's design; the kernel does not require devmap keys to be ifindices. Replicating traffic in the same VNI to several VTEPs through a single output interface would require an additional way to identify the destination VTEP for each copy.
We now have both output selection and per-copy rewriting. The complete forwarding path puts them together.
The replication and rewrite pipeline
- A per-VNI devmap restricts the outputs, and the egress program registered in each entry rewrites its copy to send the same Inner to several VTEPs.
- Ingress selects the replication group; devmap egress assembles the packet for each output.
flowchart TB
accTitle: VNI selection, replication, and per-copy rewriting
accDescr: Ingress selects a VNI and its devmap. Broadcast replicates the Inner frame to the group's members. Each tunnel egress adds Outer headers and rewrites its destination, while access egress sends the original Inner frame.
input["Received frame"] --> ingress["Ingress XDP<br/>validate / select VNI"]
ingress --> group["DEVMAP for selected VNI<br/>replicate to members"]
group -->|"tunnel copy"| tunnel["Devmap egress<br/>attach Outer<br/>rewrite destination"]
group -->|"access copy"| access["Devmap egress<br/>keep Inner unchanged"]
tunnel --> peers["Unicast to each VTEP<br/>same Inner / different Outer"]
access --> endpoint["Access output<br/>original Inner"]
classDef default fill:#ffffff,stroke:#666666,color:#222222;
class ingress,tunnel,access process;
classDef process fill:#edf3fa,stroke:#74849a,color:#222222;
class group map;
classDef map fill:#edf4ee,stroke:#7b9180,color:#222222;
This leaves two implementation questions: where to prepare the common Outer headers, and how to pass the VNI and processing instructions from ingress to each egress program. The sample uses the following design; other implementations are possible.
Passing data and processing instructions between programs
Each egress program needs header data for rewriting, along with the operation to perform and its arguments.
Before replication, the sample saves a common header template in a map and places the operation ID and arguments in 32 bytes of data_meta.
| Location | Contents |
|---|---|
First 16 bytes of data_meta |
program (operation ID) and magic (validation marker), 64 bits each. |
Last 16 bytes of data_meta |
vni, flags, inner_len, and flow_hash, 32 bits each. |
header_cache |
The common Ethernet, IPv4, UDP, and VXLAN header template. |
Ingress starts forwarding with PROGRAM_FORWARD and switches to PROGRAM_REWRITE before replication.
At egress, a shared dispatcher validates the command length, magic, and operation ID, then tail-calls the rewrite handler.
It drops the packet if the command is invalid or the handler is missing.
The packet body remains Inner during replication. Only the tunnel copies receive the 50-byte Outer template retrieved from the map:
%%{init: {"theme":"base","look":"classic","fontFamily":"DejaVu Sans, sans-serif","themeVariables":{"lineColor":"#666666","clusterBkg":"#fafafa","clusterBorder":"#aaaaaa","edgeLabelBackground":"#ffffff"},"flowchart":{"htmlLabels":false,"curve":"linear"},"themeCSS":".edgeLabel rect, .edgeLabel .background { fill: #ffffff !important; opacity: 1 !important; } .labelBkg { background-color: #ffffff !important; }"}}%%
flowchart TB
accTitle: Shared Outer template and metadata handoff
accDescr: A 1514-byte Inner frame stays unchanged while its shared 50-byte Outer template is saved in a Per-CPU LRU map. Devmap checks output MTUs and replicates the Inner frame with a 32-byte command. Access egress keeps the Inner frame. Tunnel egress retrieves or rebuilds the template, checks the final length, prepends Outer, and rewrites addresses.
inner["Inner frame: 1514 B"] --> prepare["Ingress forwarding<br/>prepare shared Outer template"]
prepare -. "store template" .-> cache[("Per-CPU LRU: header_cache<br/>key: (VNI, Inner length)<br/>Outer template: 50 B")]
prepare -->|"32 B metadata<br/>REWRITE + arguments"| group["DEVMAP<br/>check MTU per output<br/>replicate Inner frame"]
group -->|"access MTU: 1500"| access["Access egress<br/>Inner frame: 1514 B"]
group -->|"tunnel MTU: 1600"| tunnel["Tunnel egress<br/>check final length<br/>prepend 50 B / rewrite"]
cache -. "read / validate<br/>rebuild on miss" .-> tunnel
tunnel --> output["Outer 50 B + Inner 1514 B<br/>frame: 1564 B<br/>Outer IP packet: 1550 B"]
classDef default fill:#ffffff,stroke:#666666,color:#222222;
class prepare,access,tunnel process;
classDef process fill:#edf3fa,stroke:#74849a,color:#222222;
class group,cache map;
classDef map fill:#edf4ee,stroke:#7b9180,color:#222222;
The devmap checks the output MTU before running the egress program, so the sample leaves Outer off the packet until after replication.
Access copies undergo the MTU check at their Inner length.
For tunnel copies, the egress program also checks the final length including Outer.
The 50-byte VXLAN_OVERHEAD consists of 14 bytes of Ethernet, 20 bytes of IPv4, 8 bytes of UDP, and 8 bytes of VXLAN.
In vxlan_rewrite(), the program removes the 32-byte command, reserves 50 bytes at the front, and copies in the template.
In this excerpt, header already contains the retrieved and validated template:
if (bpf_xdp_adjust_meta(ctx, sizeof(struct xdp_command)) || bpf_xdp_adjust_head(ctx, -VXLAN_OVERHEAD)) return XDP_DROP; data = (void*)(long)ctx->data; end = (void*)(long)ctx->data_end; if (data + VXLAN_OVERHEAD > end) return XDP_DROP; __builtin_memcpy(data, header.bytes, VXLAN_OVERHEAD);
So far, the forwarding path has assumed that the VNI is already known. A VLAN trunk needs an input classification step before selecting the replication group.
BUM isolation with VLANs and VNIs
For a trunk carrying several VLANs, the sample selects a VNI using the ingress ifindex and VLAN ID (VID), then replicates BUM traffic through that VNI's devmap. One trunk can map VID 10 to VNI 100 and VID 20 to VNI 200. Because the lookup includes the ingress ifindex, another port can map VID 10 to VNI 300 without mixing its replication targets with VNI 100.
Configure these mappings in each port's vlans list.
The following trunk accepts two VIDs.
Each VNI's group includes the trunk and the required output ports:
{ "device": "trunk0", "vlans": [ {"vid": 10, "vni": 100}, {"vid": 20, "vni": 200} ] }
Ingress validates the 802.1Q tag, extracts the VID, and looks up the VNI in vlan_bindings:
struct vlan_key key = {.ifindex = ingress, .vid = vid}; __u32* binding = bpf_map_lookup_elem(&vlan_bindings, &key); if (!binding) return XDP_DROP; vni = *binding;
It then selects the devmap from vni_groups using that VNI.
The VNI travels in metadata, while egress reads the VID from the tag retained in Inner.
This keeps the command within the same 32-byte metadata area.
Before sending through a shared trunk, egress checks both VNI membership and Inner's VID.
Even if the trunk belongs to VNI 100 and VNI 200, a VNI 100 copy must carry VID 10.
It will not send a VNI 100 packet carrying VID 20 through that trunk.
The check uses the VNI, output ifindex, and expected VID registered in port_vnis.
For received VXLAN traffic, ingress removes Outer and forwards Inner to access members of the same VNI. Split horizon prevents loops by keeping that traffic from being forwarded to other tunnels. The same egress checks apply to the decapsulated Inner. Before decapsulation, ingress also checks that the receiving tunnel's ifindex and the VXLAN VNI form an allowed pair. Configuring a VNI somewhere on the host does not authorize receiving it on a different tunnel port.
%%{init: {"theme":"base","look":"classic","fontFamily":"DejaVu Sans, sans-serif","themeVariables":{"lineColor":"#666666","clusterBkg":"#fafafa","clusterBorder":"#aaaaaa","edgeLabelBackground":"#ffffff"},"flowchart":{"htmlLabels":false,"curve":"linear"},"themeCSS":".edgeLabel rect, .edgeLabel .background { fill: #ffffff !important; opacity: 1 !important; } .labelBkg { background-color: #ffffff !important; }"}}%%
flowchart TB
accTitle: VLAN-scoped BUM forwarding across VXLAN
accDescr: A tagged frame on trunk0 selects VNI 100 for VID 10 or VNI 200 for VID 20. Each devmap sends local copies and a tunnel copy with its original inner tag. The remote VTEP checks tunnel admission, decapsulates, and validates output membership and the inner VID before sending to a shared trunk.
input["trunk0: one 802.1Q frame<br/>Inner VID 10 or VID 20"] --> classify["vlan_bindings<br/>(ingress ifindex, VID) → VNI"]
classify -->|VID 10| a["vni_groups[100]<br/>DEVMAP broadcast"]
classify -->|VID 20| b["vni_groups[200]<br/>DEVMAP broadcast"]
a --> local10["Local VID 10 output<br/>Inner unchanged"]
b --> local20["Local VID 20 output<br/>Inner unchanged"]
a --> tunnel10["Shared uplink<br/>Outer VNI 100 / Inner VID 10"]
b --> tunnel20["Shared uplink<br/>Outer VNI 200 / Inner VID 20"]
decap["Remote VTEP ingress<br/>check allowed tunnel / VNI<br/>then decapsulate"] --> policy["Remote egress: port_vnis<br/>(VNI, egress ifindex)<br/>require expected Inner VID"]
policy -->|Match| output["Remote trunk<br/>Inner tag unchanged"]
policy -->|Mismatch| drop["Drop this output copy"]
tunnel10 --> decap
tunnel20 --> decap
class drop rejected;
classDef default fill:#ffffff,stroke:#666666,color:#222222;
classDef rejected fill:#ffffff,stroke:#666666,color:#222222,stroke-dasharray:4 3;
class classify,decap,policy process;
classDef process fill:#edf3fa,stroke:#74849a,color:#222222;
class a,b map;
classDef map fill:#edf4ee,stroke:#7b9180,color:#222222;
The branches represent separate input frames with VID 10 or VID 20; each frame selects one VNI. Tunnel copies retain the Inner tag, and the receiving VTEP checks the output port's VNI and VID.
This implementation forwards frames with a single 802.1Q tag and preserves that tag. VLAN-aware ports in the same VNI must use the same VID, and the peer VTEP must accept it. The sample does not add or remove tags or translate VIDs. It also rejects configurations that mix fixed-VNI ports and VLAN-aware ports within one VNI. Configurations using only fixed-VNI ports continue to forward Inner tags unchanged.
VLAN-aware ports drop unregistered VIDs, untagged frames, VID 0 and 4095, truncated tags, 802.1ad, and double tagging. They preserve the PCP and DEI bits without using them for classification. If RX VLAN stripping on a physical NIC removes the tag, the program treats the frame as untagged and drops it. Forwarding through a VLAN-aware port therefore requires disabling stripping.
UpdateMembers() can remove members from an initially configured group and restore them later.
These updates do not change VLAN classification or tunnel admission.
Changing the VID-to-VNI mappings requires reloading the configuration.
The VLAN documentation contains the complete JSON configuration and admission rules.
With the forwarding path defined, we can check it in a network namespace topology built from veth pairs.
Testing in network namespaces
I ran the sample with three VTEPs and endpoints in VNI 100 and VNI 200, connected by veth pairs. XDP handles both encapsulation and decapsulation, without a Linux bridge or Linux VXLAN device. Each VTEP and endpoint runs in its own network namespace:
%%{init: {"theme":"base","look":"classic","fontFamily":"DejaVu Sans, sans-serif","themeVariables":{"lineColor":"#666666","clusterBkg":"#fafafa","clusterBorder":"#aaaaaa","edgeLabelBackground":"#ffffff"},"flowchart":{"htmlLabels":false,"curve":"linear"},"themeCSS":".edgeLabel rect, .edgeLabel .background { fill: #ffffff !important; opacity: 1 !important; } .labelBkg { background-color: #ffffff !important; }"}}%%
flowchart LR
accTitle: Three-VTEP VXLAN network namespace topology
accDescr: VTEP A connects to VTEPs B and C through two direct veth underlay links. Each VTEP has separate VNI 100 and VNI 200 access endpoints, and A has one extra VNI 100 access endpoint. Every endpoint and VTEP has its own namespace.
a100["a-100 namespace<br/>VNI 100"] ---|"eth0 / p0"| A["VTEP A namespace<br/>goxdp / XDP VXLAN"]
a200["a-200 namespace<br/>VNI 200"] ---|"eth0 / p1"| A
alocal["a-local namespace<br/>VNI 100"] ---|"eth0 / p4"| A
A ---|"p2 / p2<br/>192.0.2.1 / 192.0.2.2<br/>VXLAN over veth<br/>MTU 1600"| B["VTEP B namespace<br/>goxdp / XDP VXLAN"]
A ---|"p3 / p2<br/>192.0.3.1 / 192.0.3.2<br/>VXLAN over veth<br/>MTU 1600"| C["VTEP C namespace<br/>goxdp / XDP VXLAN"]
B ---|"p0 / eth0"| b100["b-100 namespace<br/>VNI 100"]
B ---|"p1 / eth0"| b200["b-200 namespace<br/>VNI 200"]
C ---|"p0 / eth0"| c100["c-100 namespace<br/>VNI 100"]
C ---|"p1 / eth0"| c200["c-200 namespace<br/>VNI 200"]
classDef default fill:#ffffff,stroke:#666666,color:#222222;
class A,B,C process;
classDef process fill:#edf3fa,stroke:#74849a,color:#222222;
I confirmed BUM delivery within each VNI, including delivery over VXLAN to endpoints behind the remote VTEPs.
The test environment used Linux 7.0.0-38-generic and veth.
The testing documentation includes the test details and commands.
With BUM forwarding working in XDP, the next question is how it performs against replication in TC.
Performance against XDP + TC
I compared the default --backend xdp with --backend tc using the same forwarding configuration.
The XDP + TC version validates and decapsulates packets in XDP, returns XDP_PASS to hand them off as skbs, then replicates them in TC.
TC sends the access copies first, adds Outer once for the tunnel copies, and rewrites the destination for each peer. Both versions share the logic for VLAN-to-VNI classification and VNI/VID validation.
%%{init: {"theme":"base","look":"classic","fontFamily":"DejaVu Sans, sans-serif","themeVariables":{"lineColor":"#666666","clusterBkg":"#fafafa","clusterBorder":"#aaaaaa","edgeLabelBackground":"#ffffff"},"flowchart":{"htmlLabels":false,"curve":"linear"},"themeCSS":".edgeLabel rect, .edgeLabel .background { fill: #ffffff !important; opacity: 1 !important; } .labelBkg { background-color: #ffffff !important; }"}}%%
flowchart TB
accTitle: XDP and XDP plus TC VXLAN replication paths
accDescr: Both backends classify and decapsulate in XDP. The XDP backend replicates through a devmap and rewrites each tunnel copy in an egress program. The TC backend passes an skb to TC, sends access copies first, prepends Outer once, and rewrites each tunnel destination. Both preserve the Inner frame.
subgraph xdp["XDP backend"]
direction TB
xi["Ingress XDP<br/>validate / decap"] --> group["DEVMAP broadcast"]
group --> rewrite["Per-copy egress XDP<br/>attach Outer / rewrite"]
rewrite --> xo["Tunnel copies<br/>same Inner / varied Outer"]
end
subgraph tc["XDP + TC backend"]
direction TB
ti["Ingress XDP<br/>validate / decap"] --> pass["XDP_PASS<br/>create skb"]
pass --> clone["TC ingress<br/>access copies first<br/>prepend Outer once"]
clone --> target["Rewrite each destination<br/>N-1 clones + redirect"]
target --> tout["Tunnel copies<br/>same Inner / varied Outer"]
end
classDef default fill:#ffffff,stroke:#666666,color:#222222;
class xi,rewrite,ti,pass,clone,target process;
classDef process fill:#edf3fa,stroke:#74849a,color:#222222;
class group map;
classDef map fill:#edf4ee,stroke:#7b9180,color:#222222;
The XDP version replicates through a devmap and processes each copy in its egress program. The XDP + TC version performs the rewriting and replication at TC ingress.
In this veth setup, the XDP version achieved about 1.6 to 1.8 times the aggregate receive throughput of XDP + TC. Using four XDPerf workers and four veth queues, I replicated 64 UDP flows in one VNI to two unicast VTEPs. The table lists the median aggregate receive rate across three runs, measured with binaries built before VLAN support was added.
| Input frame length | XDP (Mpps) | XDP + TC (Mpps) | XDP / TC |
|---|---|---|---|
| 64 bytes | 1.861 | 1.056 | 1.76x |
| 1514 bytes | 1.454 | 0.896 | 1.62x |
These measurements include the sender, forwarding process, and receivers running on the same KVM host. They do not establish physical NIC performance, a loss-free forwarding limit, or a performance difference against XDP_CLONE, which I have not measured. The benchmark report contains the setup and raw data.
Closing thoughts
Existing XDP supports BUM replication with devmap broadcast and an egress program, including sending the same Inner with different Outer headers. I confirmed that behavior in network namespaces. The paper states that each copy cannot be rewritten independently. A devmap egress program makes this possible with existing XDP.
XDP_CLONE may still be more convenient in some cases, with direct access to a copy index and the ability to pass the original packet directly to the stack. I'd like to see future comparisons of performance and usability against the existing approach.
