AF_XDP ====== AF_XDP (eXpress Data Path) is a high speed capture framework for Linux that was introduced in Linux v4.18. AF_XDP aims at improving capture performance by redirecting ingress frames to user-space memory rings, thus bypassing the network stack. Note that during ``af_xdp`` operation the selected interface cannot be used for regular network usage. Further reading: - https://www.kernel.org/doc/html/latest/networking/af_xdp.html Compiling Suricata ------------------ Linux ~~~~~ libxdp and libpbf are required for this feature. When building from source the development files will also be required. Example:: dnf -y install libxdp-devel libbpf-devel This feature is enabled provided the libraries above are installed, the user does not need to add any additional command line options. The command line option ``--disable-af-xdp`` can be used to disable this feature. Example:: ./configure --disable-af-xdp Starting Suricata ----------------- IDS ~~~ Suricata can be started as follows to use af-xdp: :: af-xdp: suricata --af-xdp= suricata --af-xdp=igb0 In the above example Suricata will start reading from the `igb0` network interface. AF_XDP Configuration -------------------- Each of these settings can be configured under ``af-xdp`` within the "Configure common capture settings" section of suricata.yaml configuration file. The number of threads created can be configured in the suricata.yaml configuration file. It is recommended to use threads equal to NIC queues/CPU cores. Another option is to select ``auto`` which will allow Suricata to configure the number of threads based on the number of RSS queues available on the NIC. With ``auto`` selected, Suricata spawns receive threads equal to the number of configured RSS queues on the interface. :: af-xdp: threads: threads: auto threads: 8 Advanced setup --------------- af-xdp capture source will operate using the default configuration settings. However, these settings are available in the suricata.yaml configuration file. Available configuration options are: force-xdp-mode ~~~~~~~~~~~~~~ There are two operating modes employed when loading the XDP program, these are: - XDP_DRV: Mode chosen when the driver supports AF_XDP - XDP_SKB: Mode chosen when no AF_XDP support is unavailable XDP_DRV mode is the preferred mode, used to ensure best performance. :: af-xdp: force-xdp-mode: where: value = force-xdp-mode: drv force-bind-mode ~~~~~~~~~~~~~~~ During binding the kernel will first attempt to use zero-copy (preferred). If zero-copy support is unavailable it will fallback to copy mode, copying all packets out to user space. :: af-xdp: force-bind-mode: where: value = force-bind-mode: zero For both options, the kernel will attempt the 'preferred' option first and fallback upon failure. Therefore the default (none) means the kernel has control of which option to apply. By configuring these options the user is forcing said option. Note that if enabled, the bind will only attempt this option, upon failure the bind will fail i.e. no fallback. mem-unaligned ~~~~~~~~~~~~~~~~ AF_XDP can operate in two memory alignment modes, these are: - Aligned chunk mode - Unaligned chunk mode Aligned chunk mode is the default option which ensures alignment of the data within the UMEM. Unaligned chunk mode uses hugepages for the UMEM. Hugepages start at the size of 2MB but they can be as large as 1GB. Lower count of pages (memory chunks) allows faster lookup of page entries. The hugepages need to be allocated on the NUMA node where the NIC and CPU resides. Otherwise, if the hugepages are allocated only on NUMA node 0 and the NIC is connected to NUMA node 1, then the application will fail to start. Therefore, it is recommended to first find out to which NUMA node the NIC is connected to and only then allocate hugepages and set CPU cores affinity to the given NUMA node. Memory assigned per socket/thread is 16MB, so each worker thread requires at least 16MB of free space. As stated above hugepages can be of various sizes, consult the OS to confirm with ``cat /proc/meminfo``. Example :: 8 worker threads * 16Mb = 128Mb hugepages = 2048 kB so: pages required = 62.5 (63) pages See https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt for detailed description. To enable unaligned chunk mode: :: af-xdp: mem-unaligned: mem-unaligned: yes Introduced from Linux v5.11 a ``SO_PREFER_BUSY_POLL`` option has been added to AF_XDP that allows a true polling of the socket queues. This feature has been introduced to reduce context switching and improve CPU reaction time during traffic reception. Enabled by default, this feature will apply the following options, unless disabled (see below). The following options are used to configure this feature. enable-busy-poll ~~~~~~~~~~~~~~~~ Enables or disables busy polling. :: af-xdp: enable-busy-poll: enable-busy-poll: yes busy-poll-time ~~~~~~~~~~~~~~ Sets the approximate time in microseconds to busy poll on a ``blocking receive`` when there is no data. :: af-xdp: busy-poll-time: