1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
|
/*! \page page_dpdk DPDK, Data Plane Development Kit
\tableofcontents
\section dpdk_overview DPDK Overview
Data Plane Development Kit (DPDK) is a set of libraries that allows network
interface controller (NIC) drivers to use user space memory buffers to send and
receive data over a network. These libraries underpin one of the network
transport options in UHD.
In UHD, the DPDK-based transport will fork off I/O threads that implement the
network services, and these I/O threads will service the NICs on cores provided
in your configuration. The cores will be completely consumed by the I/O thread.
Together with DPDK's polling-mode drivers, this virtually eliminates context
switching in UHD's transport layer, which enables us to stream higher sample
rates.
\section dpdk_setup DPDK Setup
DPDK is currently only available on Linux platforms, requires an input-output
memory management unit (IOMMU), and must be run on a multicore processor. The
following subsections will talk through the steps required to setup DPDK on your
computer.
\subsection dpdk_installation DPDK Installation Instructions
As a new and developing technology, the DPDK APIs are unstable. UHD requires
version 18.11(deprecated), 19.11, 20.11 or 21.11. Adjacent non-LTS releases of
DPDK may work, but are not recommended.
On Ubuntu 20.04, Fedora 33-36, or Debian Bullseye/Buster (via backports),
DPDK is available in your distribution's repositories. For example, on Debian
systems, it can be obtained with the following command:
sudo apt install dpdk dpdk-dev
Otherwise, you'll need to follow the build guide at
https://doc.dpdk.org/guides-21.11/linux_gsg/build_dpdk.html . The software
releases can be found at https://core.dpdk.org/download/. Version >=20.11 builds
with appropriate options when using the default build procedure.
Note that if you are using a Mellanox NIC, the MLNX-OFED drivers available from
https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed may be
needed to support dpdk.
Note that if you are building and installing older versions of DPDK from source,
you will need to change its configuration to build shared libraries in order for
UHD to link successfully. After running `make config` or `make defconfig`, open
the `build/.config` file in an editor and find the following line:
CONFIG_RTE_BUILD_SHARED_LIB=n
Change the `n` to a `y` to enable the building of shared libraries, then type
`make` to start the build.
If you are using a Mellanox ConnectX 4 or later NIC, you also need to update the
following line:
CONFIG_RTE_LIBRTE_MLX5_PMD=n
\subsection dpdk_system_configuration System Configuration
The official documentation regarding system configuration can be found at
https://doc.dpdk.org/guides-21.11/linux_gsg/sys_reqs.html.
First, you'll need to enable the IOMMU and set up some hugepages. DPDK will
completely take over all available hugepages, so don't allocate all your memory
to them- the rest of UHD and the application need memory too.
For example, on a system with 16 GB of RAM, a generous appropriation of
512x 2 MiB pages was more than sufficient, and you likely won't need that much.
For best results, hugepages should be enabled at boot. For example, using an
Intel IOMMU with Ubuntu 19.04 IOMMU drivers, the following line was needed in
our Grub config.
iommu=pt intel_iommu=on hugepages=2048
The setup of the IOMMU and hugepages is system-specific, so consult the kernel
documentation for more info. After you reboot, you should see
`/sys/kernel/iommu_groups` populated.
Next, many of the NIC drivers are implemented atop `vfio-pci`, so you'll need to
load that driver with the following command:
modprobe vfio-pci
Mellanox ConnectX 4 and later NICs do not use the vfio-pci driver, so this step
is not necessary for them. They will use the standard driver in conjunction with
libibverbs.
For NICs that require vfio-pci (like Intel's X520), you'll want to use the
`dpdk-devbind.py` script to the vfio-pci driver. This script is shipped with
DPDK and installed to `$prefix/share/dpdk/usertools`. If the NIC uses the
vfio-pci driver, and the package was installed apt-get, then a typical invocation
might be
/usr/share/dpdk/usertools/dpdk-devbind.py --bind=vfio-pci ens6f0
If successful, the script might provide an updated status like this:
/usr/share/dpdk/usertools/dpdk-devbind.py -s
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
[...]
See https://doc.dpdk.org/guides-21.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules
for more details.
With the hugepages, IOMMU, and drivers set up, the system is ready for DPDK to
use.
\subsection dpdk_nic_config NIC Configuration
Configuration of the NIC can be controlled via device arguments via the usual
methods, but the \ref page_configfiles "UHD configuration file" is the
recommended location.
In order to run, you'll need to set the permissions for your user to take over
the vfio-pci devices, the hugepages, and the scheduler's settings for the
threads (at a minimum). You may consider running you applications as root, at
least while becoming familiar with DPDK. If you use a per-user config file, make
sure it's in the correct location.
The config file will have 2 different components. First are the global DPDK
options:
;When present in device args, use_dpdk indicates you want DPDK to take over the UDP transports
;The value here represents a config, so you could have another section labeled use_dpdk=myconf
;instead and swap between them
[use_dpdk=1]
;dpdk_mtu is the NIC's MTU setting
;This is separate from MPM's maximum packet size
dpdk_mtu=9000
;dpdk_driver is the -d flag for the DPDK EAL. If DPDK doesn't pick up the driver for your NIC
;automatically, you may need this argument to point it to the folder where it can find the drivers
;Note that DPDK will attempt to load _everything_ in that folder as a driver, so you may want to
;create a separate folder with symlinks to the librte_pmd_* and librte_mempool_* libraries.
dpdk_driver=/usr/local/lib/x86_64-linux-gnu/dpdk/pmds-21.0/
;dpdk_corelist is the -l flag for the DPDK EAL. See more at the link
; https://doc.dpdk.org/guides-21.11/linux_gsg/build_sample_apps.html#running-a-sample-application
;Note if you use multiple SFP ports in a streaming application simultaneously,
;you can specify multiple cores in the core list (e.g. 0, 1, 2) and then assign
;them each to the separate SFP port/NIC.
dpdk_corelist=0,1
;dpdk_num_mbufs is the total number of packet buffers allocated
;to each direction's packet buffer pool
;This will be multiplied by the number of NICs, but NICs on the same
;CPU socket share a pool. When using Mellanox NICs, this value must be greater
;than the dpdk_num_desc value in the next section.
dpdk_num_mbufs=4096
;dpdk_mbuf_cache_size is the number of buffers to cache for a CPU
;The cache reduces the interaction with the global pool
dpdk_mbuf_cache_size=64
The other sections fall under per-NIC arguments. The key for NICs is the MAC
address, and it must be in a particular format. Hex digits must all be lower
case, and octets must be separated by colons. Here is an example:
[dpdk_mac=3c:fd:fe:a2:a9:09]
;dpdk_lcore selects the lcore that this NIC's driver will run on
;Multiple NICs may occupy one lcore, but the I/O thread will completely
;consume that lcore's CPU. When using dual SFP configuration, using a
;different dpdk_lcore value for each SFP connection is recommended and
;will result in better streaming performance. Also, 0 is reserved for
;the master thread (i.e.the initial UHD thread that calls init() for DPDK).
;Attempting to use it as an I/O thread will only result in hanging.
;Note also that by default, the lcore ID will be the same as the CPU ID.
dpdk_lcore = 1
;dpdk_ipv4 specifies the IPv4 address, and both the address and
;subnet mask are required (and in this format!). DPDK uses the
;netmask to create a basic routing table. Routing to other networks
;(i.e. via gateways) is not permitted.
dpdk_ipv4 = 192.168.10.1/24
;dpdk_num_desc is the number of descriptors in each DMA ring.
;Must be a power of 2.
dpdk_num_desc=4096
[dpdk_mac=3c:fd:fe:a2:a9:0a]
;Using a separate dpdk_lcore value for each SFP connection/MAC entry
;can possibly result in improved streaming performance. E.g. dpdk_lcore = 2.
dpdk_lcore = 1
dpdk_ipv4 = 192.168.20.1/24
\section dpdk_using Using DPDK in UHD
Once DPDK is installed and configured on your system, it can be used with UHD.
The following steps will describe how to stream using DPDK. DPDK is currently
only available on the following devices:
- \ref page_usrp_e3xx "E320 (but not E31x) device"
- \ref page_usrp_n3xx "N3xx devices"
- \ref page_usrp_x3x0 "X3xx devices"
- \ref page_usrp_x4xx "X4xx devices"
\subsection dpdk_device_args Enabling DPDK with UHD Device Args
Add the following to your device args in order to indicate that a DPDK-based UDP
transport shall be used instead of the kernel's UDP stack.
--args="use_dpdk=1"
Device discovery via DPDK is not currently implemented, so the device args
`mgmt_addr`, `addr`, and `second_addr` (if applicable) must all be specified at
runtime. There is no mechanism for MPM's TCP/IP control traffic to flow over a
link that is occupied by DPDK, so mgmt_addr must point to a link that is not
used for CHDR, such as N310's RJ45 port.
\subsection dpdk_link_detection DPDK Link Detection
When DPDK is enabled and the driver is initializing, the status of all
DPDK-enabled links is checked to verify that all links are active before the
driver proceeds. The links are checked every 250 ms until all links have
reported that they are up or until the link timeout expires, which by default
is 1000 ms. If any of the links have not reported as being up by the time the
timeout expires, a runtime error is thrown.
Users may choose to override the link timeout in cases where particular
systems and/or network cards take longer to establish stable DPDK links.
The timeout can be overridden by passing `dpdk_link_timeout=N` in the device
arguments, where N is the desired timeout time in <b>milliseconds</b>, or
by adding a `dpdk_link_timeout` entry to the
\ref page_configfiles "UHD configuration file".
\subsection dpdk_troubleshooting Troubleshooting
With Linux kernels 5.10 and beyond, we have observed periodic underruns on systems
that otherwise have no issues. These Linux kernel versions are the default for Ubuntu
20.04.3 LTS and later. The underrun issue is due to the RT_RUNTIME_SHARE feature
being disabled by default in newer versions of the Linux kernel. The following
procedure can be used to enable RT_RUNTIME_SHARE. Note, this process was tested on
Linux kernel version 5.13. The procedure may be slightly different on other kernel
versions. To determine the Linux kernel version of your system, in a terminal issue
the command `uname -r`.
sudo -s
cd /sys/kernel/debug/sched/
cat features
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI **NO_RT_RUNTIME_SHARE** NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE
echo RT_RUNTIME_SHARE > features
cat features
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_HRTICK_DL NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI **RT_RUNTIME_SHARE** NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP NO_LATENCY_WARN ALT_PERIOD BASE_SLICE
*/
// vim:ft=doxygen:
|
