[daniel-noland]

The DPDK driver binds the gateway's NICs to vfio-pci, and vfio-pci needs an IOMMU group to bind to. VLAB gateway VMs have no IOMMU, and the Flatcar kernel is built without CONFIG_VFIO_NOIOMMU, so with --gateway-driver=dpdk the bind fails with EINVAL (vfio-pci 0000:02:01.0: probe with driver vfio-pci failed with error -22) and the dataplane never starts.

Gateway VMs whose Gateway uses the DPDK driver (an interface given by PCI address, the way the gateway API defaults its driver) now get:

• an emulated Intel IOMMU with interrupt remapping (-device intel-iommu,intremap=on, kernel-irqchip=split), ahead of every PCI device. The Flatcar kernel has CONFIG_INTEL_IOMMU_DEFAULT_ON=y, so the guest needs no kernel arguments.
• their dataplane NICs on the root bus, from slot 0x10 + NIC ID, so each is its own IOMMU group. Behind the shared conventional PCI bridge all NICs land in one group, vfio-pci takes a group whole, and DPDK rejects it (VFIO group is not viable) while the management NIC is still on e1000. The management NIC stays behind the bridge, so it is still enp2s0.
• Gateway PCI addresses generated from the same rule (0000:00:11.0, 0000:00:12.0, ...).
• e1000e (82574L) dataplane NICs instead of e1000 (82540EM). The DPDK driver gives each port a transmit queue per worker plus one for the control plane. The 82540EM has a single transmit ring, though DPDK's em driver advertises two, so control-plane frames sent on the second never left the VM: ARP, BGP and BFD all failed while the dataplane counted them as sent.

Kernel-driver gateways, switches, servers and the control node are unchanged.

Testing

• New TestCreateVLABConfigDPDKGatewayIOMMU: for both drivers, only DPDK gateways get the IOMMU, the management NIC stays behind the bridge, and every Gateway interface's PCI address is where the VM's NIC is. It was checked for vacuity: with the builder put back on 0000:02:%02d.0, it fails.

• go test ./pkg/fab/... ./pkg/hhfab/... passes, and golangci-lint v1.64.8 reports 0 issues on pkg/hhfab.

• VLAB, using this commit cherry-picked onto v0.50.4 with vlab gen --gateway-driver=dpdk and vlab up --recreate, running dataplane 0.30.0:

  • The gateway QEMU gets q35,...,kernel-irqchip=split and intel-iommu,intremap=on, with e1000 on pcibr0,addr=0x0 (management) and on pcie.0,addr=0x11/0x12. Switches don't get a vIOMMU.
  • Guest: DMAR-IR: Enabled IRQ remapping in x2apic mode. enp2s0 is up with its management address, and the IOMMU groups are {00:04.0, 01:00.0, 02:00.0}, {00:11.0} and {00:12.0}.
  • The Gateway gets port1=pci@0000:00:11.0, port2=pci@0000:00:12.0. dataplane-init binds both to vfio-pci, and EAL reports Using IOMMU type 1 (Type 1) and probes the device. The dataplane then stops at its own mlx5-only check (see below). Without this change it stops much earlier, at the vfio-pci bind (EINVAL), and with the vIOMMU alone it hits VFIO group is not viable.

• Follow-up run on the same VLAB, with githedgehog/dataplane's vfio-pci port support (Bump to August Flatcar #1976) and an experimental fabric controller adding hugepage requests and the interface MTU (see below): BGP (IPv4 and EVPN) and BFD up on all four gateway links, LLDP seen by the spines, setup-vpcs + setup-peerings 1+2:gw + test-connectivity passing, and iperf3 between vpc-01 and vpc-02 through the gateway at about 11 Mbit/s each way.

Not in this PR

• Fabric's gateway controller needs two changes for a DPDK gateway to work (gw-ctrl: give a DPDK dataplane its MTU and hugepages fabric#1704): hugepage requests on the dataplane pod (without them its init stops with "cgroup permits 0 additional bytes"), and the interface MTU passed as /mtu=N (without it the DPDK ports stay at 1500, and full-size VXLAN packets are dropped). The gateway VMs also need hugepages reserved, which this PR does not do yet.
• The dataplane's DPDK driver needs vfio-pci port support, which is dataplane: support vfio-pci NICs dataplane#1976.

Draft: the commit still needs the author's DCO sign-off.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}