[ovs-discuss] Open vSwitch fails to allocate memory pool for DPDK port
Tobias Hofmann -T (tohofman - AAP3 INC at Cisco)
tohofman at cisco.com
Mon May 20 21:13:44 UTC 2019
Hi Ian,
Just to close on this.
Using openvswitch with version 2.9.3 and DPDK 17.11.5 fixed the issue.
Thanks for your help!
On 4/16/19, 12:46 PM, "Ian Stokes" <ian.stokes at intel.com> wrote:
On 4/3/2019 5:15 PM, Tobias Hofmann -T (tohofman - AAP3 INC at Cisco) wrote:
> Sorry I just accidentally sent out the last mail too early...
>
> Hi Ian,
>
> To answer your questions, I just reproduced the issue on my system:
>
> What commands have you used to configure the hugepage memory on your system?
> - I have added the following kernel parameters: hugepagesz=2M hugepages=512 and then rebooted the system. In other scenarios I allocated the HugePages by writing into /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages. The HugePages are also mounted on: hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
>
Hi Tobias,
I was able to reproduce the issue today.
The error that occurs actually happens inside DPDK so as such it's not a
bug in OVS.
I was able to reproduce the issue with DPDK 17.11.0, 17.11.1 and
17.11.2. The issue itself seems to be resolved on my system by the
following commit in DPDK 17.11.3:
222f91da4714 ("mem: do not use physical addresses in IOVA as VA mode")
Since you are using OVS 2.9.3 I would suggest moving to use DPDK 17.11.4
at least.
The mapping for OVS to DPDK versions is available in the OVS release notes:
http://docs.openvswitch.org/en/latest/faq/releases/
Can you give this a try and see if it resolves the issue for you?
HTH
Ian
> Before you start OVS with DPDK, if you execute cat /proc/meminfo how many hugepages do you see available and how many free? (for 2mb I would assume 512 in both cases).
> - In this specific case, I only saw 512 HugePages_Total and 0 free because after the restart OVS was already using the 512 pages.
>
> What memory commands are you passing to OVS with DPDK (e.g. dpdk-socket-mem parameter etc.)?
> - Nothing, meaning the default memory of 1GB.
>
> Is it just 1 bridge and a single DPDK interface you are adding or are there more than 1 DPDK interface attached?
> - There are 4 bridges in total but only one is using netdev datapath and DPDK ports. Here is an extract of that one DPDK bridge:
> Bridge lan-br
> Port lan-br
> Interface lan-br
> type: internal
> Port "dpdk-p0"
> Interface "dpdk-p0"
> type: dpdk
> options: {dpdk-devargs="0000:08:0b.2"}
> error: "could not add network device dpdk-p0 to ofproto (No such device)"
>
> Can you provide the entire log? I'd be interested in seeing the memory info at initialization of OVS DPDK.
> - I attached it to that mail.
>
> What type of DPDK device are you adding? It seems to be a Virtual function from the log above, can you provide more detail as regards the underlying NIC type the VF is associated with?
> - It's a VF. NIC type is Intel 710 and driver is i40
>
> DPDK Version is 17.11.0
>
> Thanks
> Tobias
>
> On 4/3/19, 9:07 AM, "Tobias Hofmann -T (tohofman - AAP3 INC at Cisco)" <tohofman at cisco.com> wrote:
>
> Hi Ian,
>
> To answer your questions, I just reproduced the issue on my system:
>
> What commands have you used to configure the hugepage memory on your system?
> - I have added the following kernel parameters: hugepagesz=2M hugepages=512 and then rebooted the system. In other scenarios I allocated the HugePages by writing into /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages.
> The HugePages are also mounted on: hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
>
> Before you start OVS with DPDK, if you execute cat /proc/meminfo how many hugepages do you see available and how many free? (for 2mb I would assume 512 in both cases).
> - In this specific case, I only saw 512 HugePages_Total and 0 free because after the restart OVS was already using the 512 pages.
>
> What memory commands are you passing to OVS with DPDK (e.g. dpdk-socket-mem parameter etc.)?
>
> On 4/3/19, 6:00 AM, "Ian Stokes" <ian.stokes at intel.com> wrote:
>
> On 4/3/2019 1:04 AM, Tobias Hofmann -T (tohofman - AAP3 INC at Cisco)
> via discuss wrote:
> > Hello,
> >
> > I’m trying to attach a DPDK port with an mtu_size of 9216 to a bridge.
> > For this purpose, I have allocated 512 HugePages of size 2MB for OVS
> > (1GB in total).
>
> Hi,
>
> I couldn't reproduce the behavior above on my own system with 512 x 2MB
> hugepages. Ports were successfully configured with MTU 9216. Perhaps
> some more detail as regards your setup will help reproduce/root cause.
>
> Questions inline below.
>
> What commands have you used to configure the hugepage memory on your system?
>
> Before you start OVS with DPDK, if you execute cat /proc/meminfo how
> many hugepages do you see available and how many free? (for 2mb I would
> assume 512 in both cases).
>
> What memory commands are you passing to OVS with DPDK (e.g.
> dpdk-socket-mem parameter etc.)?
>
> Is it just 1 bridge and a single DPDK interface you are adding or are
> there more than 1 DPDK interface attached?
>
> >
> > Doing so will constantly fail, two workarounds to get it working were
> > either to decrease the MTU size to 1500 or to increase the total amount
> > of HugePage memory to 3GB.
> >
> > Actually, I did expect the setup to also work with just 1GB because if
> > the amount of memory is not sufficient, OVS will try to halve the number
> > of buffers until 16K.
> >
> > However, inside the logs I couldn’t find any details regarding this. The
> > only error message I observed was:
> >
> > netdev_dpdk|ERR|Failed to create memory pool for netdev dpdk-p0, with
> > MTU 9216 on socket 0: Invalid argument
>
> Can you provide the entire log? I'd be interested in seeing the memory
> info at initialization of OVS DPDK.
>
> >
> > That log message is weird as I would have expected an error message
> > saying something like ‘could not reserve memory’ but not ‘Invalid argument’.
> >
> > I then found this very similar bug on Openstack:
> > https://bugs.launchpad.net/starlingx/+bug/1796380
> >
> > After having read this, I tried the exact same setup as described above
> > but this time with HugePages of size 1GB instead of 2MB. In this
> > scenario, it also worked with just 1GB of memory reserved for OVS.
> >
> > Inside the logs I could observe this time:
> >
> > 2019-04-02T22:55:31.849Z|00098|dpdk|ERR|RING: Cannot reserve memory
> >
> > 2019-04-02T22:55:32.019Z|00099|dpdk|ERR|RING: Cannot reserve memory
> >
> > 2019-04-02T22:55:32.200Z|00100|netdev_dpdk|INFO|Virtual function
> > detected, HW_CRRC_STRIP will be enabled
> >
>
> What type of DPDK device are you adding? It seems to be a Virtual
> function from the log above, can you provide more detail as regards the
> underlying NIC type the VF is associated with?
>
> > 2019-04-02T22:55:32.281Z|00101|netdev_dpdk|INFO|Port 0: f6:e9:29:4d:f9:cf
> >
> > 2019-04-02T22:55:32.281Z|00102|dpif_netdev|INFO|Core 1 on numa node 0
> > assigned port 'dpdk-p0' rx queue 0 (measured processing cycles 0).
> >
> > The two times where OVS cannot reserve memory are, I guess, the two
> > times where it has to halve the number of buffers to get it working.
>
> Yes this is correct. For example in my setup with 512 x 2MB pages I see
> "Cannot reserve memory" message 4 times before it completes configuration.
>
> >
> > My question now is, is the fact that it does not work for 2MB HugePages
> > a bug? Also, is the error message in the first log extract the intended one?
> >
>
> Yes, it seems like a bug if it can be reproduced. The invalid argument
> in this case would refer to the number of mbufs being requested by OVS
> DPDK being less than the minimum allowed (4096 * 64). i.e. the amount of
> memory available cannot support the minimum 262,144 mbufs OVS DPDK
> requires. The memory configuration at the system level is outside of OVS
> control however.
>
> > *My version numbers:*
> >
> > * CentOS 7.6
> > * Open vSwitch version: 2.9.3
> > * DPDK version: 17.11
> Is this DPDK 17.11.0? As an FYI The latest DPDK 17.11.x version
> recommended for OVS is 17.11.4 with OVS 2.9.4, these typically include
> bug fixes in DPDK so we recommend moving to them if possible..
>
> Ian
> > * System has a single NUMA node.
> >
> > Thank you
> >
> > Tobias
> >
> >
> > _______________________________________________
> > discuss mailing list
> > discuss at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> >
>
>
>
>
>
More information about the discuss
mailing list