[ovs-discuss] Open vSwitch fails to allocate memory pool for DPDK port

Ian Stokes ian.stokes at intel.com
Wed Apr 3 13:00:23 UTC 2019


On 4/3/2019 1:04 AM, Tobias Hofmann -T (tohofman - AAP3 INC at Cisco) 
via discuss wrote:
> Hello,
> 
> I’m trying to attach a DPDK port with an mtu_size of 9216 to a bridge. 
> For this purpose, I have allocated 512 HugePages of size 2MB for OVS 
> (1GB in total).

Hi,

I couldn't reproduce the behavior above on my own system with 512 x 2MB 
hugepages. Ports were successfully configured with MTU 9216. Perhaps 
some more detail as regards your setup will help reproduce/root cause.

Questions inline below.

What commands have you used to configure the hugepage memory on your system?

Before you start OVS with DPDK, if you execute cat /proc/meminfo how 
many hugepages do you see available and how many free? (for 2mb I would 
assume 512 in both cases).

What memory commands are you passing to OVS with DPDK (e.g. 
dpdk-socket-mem parameter etc.)?

Is it just 1 bridge and a single DPDK interface you are adding or are 
there more than 1 DPDK interface attached?

> 
> Doing so will constantly fail, two workarounds to get it working were 
> either to decrease the MTU size to 1500 or to increase the total amount 
> of HugePage memory to 3GB.
> 
> Actually, I did expect the setup to also work with just 1GB because if 
> the amount of memory is not sufficient, OVS will try to halve the number 
> of buffers until 16K.
> 
> However, inside the logs I couldn’t find any details regarding this. The 
> only error message I observed was:
> 
> netdev_dpdk|ERR|Failed to create memory pool for netdev dpdk-p0, with 
> MTU 9216 on socket 0: Invalid argument

Can you provide the entire log? I'd be interested in seeing the memory 
info at initialization of OVS DPDK.

> 
> That log message is weird as I would have expected an error message 
> saying something like ‘could not reserve memory’ but not ‘Invalid argument’.
> 
> I then found this very similar bug on Openstack: 
> https://bugs.launchpad.net/starlingx/+bug/1796380
> 
> After having read this, I tried the exact same setup as described above 
> but this time with HugePages of size 1GB instead of 2MB. In this 
> scenario, it also worked with just 1GB of memory reserved for OVS.
> 
> Inside the logs I could observe this time:
> 
> 2019-04-02T22:55:31.849Z|00098|dpdk|ERR|RING: Cannot reserve memory
> 
> 2019-04-02T22:55:32.019Z|00099|dpdk|ERR|RING: Cannot reserve memory
> 
> 2019-04-02T22:55:32.200Z|00100|netdev_dpdk|INFO|Virtual function 
> detected, HW_CRRC_STRIP will be enabled
> 

What type of DPDK device are you adding? It seems to be a Virtual 
function from the log above, can you provide more detail as regards the 
underlying NIC type the VF is associated with?

> 2019-04-02T22:55:32.281Z|00101|netdev_dpdk|INFO|Port 0: f6:e9:29:4d:f9:cf
> 
> 2019-04-02T22:55:32.281Z|00102|dpif_netdev|INFO|Core 1 on numa node 0 
> assigned port 'dpdk-p0' rx queue 0 (measured processing cycles 0).
> 
> The two times where OVS cannot reserve memory are, I guess, the two 
> times where it has to halve the number of buffers to get it working.

Yes this is correct. For example in my setup with 512 x 2MB pages I see 
"Cannot reserve memory" message 4 times before it completes configuration.

> 
> My question now is, is the fact that it does not work for 2MB HugePages 
> a bug? Also, is the error message in the first log extract the intended one?
> 

Yes, it seems like a bug if it can be reproduced. The invalid argument 
in this case would refer to the number of mbufs being requested by OVS 
DPDK being less than the minimum allowed (4096 * 64). i.e. the amount of 
memory available cannot support the minimum 262,144 mbufs OVS DPDK 
requires. The memory configuration at the system level is outside of OVS 
control however.

> *My version numbers:*
> 
>   * CentOS 7.6
>   * Open vSwitch version: 2.9.3
>   * DPDK version: 17.11
Is this DPDK 17.11.0? As an FYI The latest DPDK 17.11.x version 
recommended for OVS is 17.11.4 with OVS 2.9.4, these typically include 
bug fixes in DPDK so we recommend moving to them if possible..

Ian
>   * System has a single NUMA node.
> 
> Thank you
> 
> Tobias
> 
> 
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> 



More information about the discuss mailing list