[ovs-dev] Memory pool issue on delete and re-add of DPDK ports with jumbo MTU >1894

Kavanagh, Mark B mark.b.kavanagh at intel.com
Tue Feb 28 09:34:38 UTC 2017


>
>Hi Mark,
>Is there any patch I can expect for this issue?

Hi Kapil,

I flagged the issue to our onsite DPDK memory expert (Sergio, cc'd). Sergio worked with us on the issue, and wrote a simple DPDK application in an attempt to reproduce the issue, and localize it to DPDK.

Unfortunately, the issue was not observed/reproducible in that DPDK app. I intend to take another look at it today from an OvS-DPDK perspective - I'll provide an update as and when available.

Thanks,
Mark

>
>On Tue, Feb 21, 2017, 7:28 PM Kapil Adhikesavalu <kapil20084 at gmail.com> wrote:
>Hi Mark,
>
>Thanks for the detailed analysis, i will wait for further updates.
>
>Btw on the mailing list, i seem to have clicked 'reply' instead of 'replyall' :)
>
>Regards
>Kapil.
>
>On Tue, Feb 21, 2017 at 3:26 PM, Kavanagh, Mark B <mark.b.kavanagh at intel.com> wrote:
>+ dev - please keep the list included in any discussions going forward folks :)
>
>Hi Kapil,
>
>I've managed to reproduce the issue of removing and re-adding ports with MTU > 1894, using
>the following configuration:
>
>        DPDK:           v16.07
>        OVS:            f922f0f1c9
>        dpdk-socket-mem: 1024
>        hugepage-sz:    2M
>        port MTU:               1920
>
>The same behavior is observed with both Phy ports and vhost-user ports; when I add a port
>with MTU 1920, subsequently remove it, and then re-add it, I receive an error message stating
>that insufficient memory is available to create a mempool for that port.
>
>A few notes on this:
>        - this behavior is observed only when attempting to re-add the same port (more on
>this later); if I add a new port with MTU 1920, the issue doesn't occur, indicating that lack
>of available memory is not the root-cause
>        - the reason that you don't see the issue when you add a 'dummy' port is because that
>port is also of type dpdkvhostuser and shares the same MTU - and thus the same mempool - as
>the other dpdkvhostuser ports that you've added. A mempool can only be deleted when all ports
>that use it are deleted; since you never delete the 'dummy' port, the mempool is never
>destroyed, and so when you re-add the other dpdkvhostuser ports, they simply use the existing
>mempool that they previously did (and which is still in use by 'dummy').
>
>In the course of debugging, I traced the issue to the DPDK function
>rte_memzone_reserve_aligned_thread_unsafe, which is invoked in the call hierarchy of
>rte_mempool_create (in turn, invoked by dpdk_mp_get). By instrumenting the code, I observed
>that when I attempt to re-add a port with MTU 1920, a check for the pre-existence of one
>particular memzone fails in this function:
>
>        ovs-vswitchd[59273]: EAL: memzone_reserve_aligned_thread_unsafe(): reserving memzone
>for mempool: === MP_ovs_mp_3054_0_262144_130 ===
>        ovs-vswitchd[59273]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone
><MP_ovs_mp_3054_0_262144_130> already exists
>        ovs-vswitchd[59273]: ovs|00069|dpdk|ERR|Insufficient memory to create memory pool for
>netdev dpdk0, with MTU 1920 on socket 0
>
>Since that function fails, it triggers the 'insufficient memory' log in OvS-DPDK - this is
>certainly something that we can remedy, as it is misleading.
>
>Curiously, I don't observe this behavior when dpdk-socket-mem=2048, in which case the port
>may be deleted and re-added without issue. I'll consult my DPDK colleagues on this, and get
>back to you with an answer in the coming days.
>
>Thanks,
>Mark
>
>>
>>Hi Ian,
>>
>>Thanks for looking into it.
>>
>>Please find the information below and let me know if anything else is needed.
>>
>>•        OVS version: 2.6.1
>>•        DPDK version: 16.07
>>• Was OvS installed through a package manager or built from source?  Built from source.
>>• If OVS is a release version, is it the latest version of that release?  branch-2.6 with
>>head commit f922f0f1c926bb7596ba4e0971960dc89e39f0e7
>>• Author: Thadeu Lima de Souza Cascardo <cascardo at redhat.com>
>>• Date:   Wed Oct 19 13:32:57 2016 -0200
>>• OS Version: ?  It's a yocto based OS with 4.1 Linux kernel uname -r > 4.1.26-yocto-
>standard
>>• HW Platform: ? HP gen9 DL20 proliant
>>• CPU version and frequency: ?   Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz, 4 cores
>>•        Commands & parameters used to launch OVS (In particular any command related to
>>memory setup)
>>
>>root at hp-dl20:/# cat /proc/cmdline
>>BOOT_IMAGE=/dl20.bin root=/dev/ram0 ip=dhcp default_hugepagesz=2M hugepagesz=2M
>>hugepages=1536 isolcpus=1-3
>>
>># Create ovs config
>>mkdir -p /var/log/openvswitch
>>ovsdb-tool create $ovsdir/etc/openvswitch/conf.db
>>$ovsdir/usr/share/openvswitch/vswitch.ovsschema
>>
>># Bring up ovsdb-server daemon
>>mkdir -p $ovsdir/var/run/openvswitch
>>/usr/sbin/ovsdb-server --remote=punix:$ovsdir/var/run/openvswitch/db.sock \
>>               --remote=db:Open_vSwitch,Open_vSwitch,manager_options \
>>               --private-key=db:Open_vSwitch,SSL,private_key \
>>               --certificate=db:Open_vSwitch,SSL,certificate \
>>               --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \
>>               --pidfile --detach --verbose=err
>>
>># Intialize the ovs database
>>/usr/bin/ovs-vsctl --no-wait init
>>
>>  dpdk_socket_mem="1024,0"
>>  dpdk_lcore_mask=0x1
>>
>>  # Specify DPDK options for very newest as-of-yet-unreleased rev of OVS
>>  /usr/bin/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
>>
>>  # Number of memory channels on targeted platform
>>  /usr/bin/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-extra="-n 4"
>>
>>  # Set dpdk-socket-mem
>>  /usr/bin/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-
>>mem=$dpdk_socket_mem
>>
>>  # Set dpdk-lcore-mask
>>  /usr/bin/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-
>>mask=$dpdk_lcore_mask
>>
>>To collect the below logs in attachment, i enabled --verbose and ran following commands to
>>hit the issue
>>
>>root at hp-dl20:/# ovs-vsctl add-br trail -- set bridge trail datapath_type=netdev
>>root at hp-dl20:/# ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
>>type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1920
>>root at hp-dl20:/# ovs-vsctl get interface dpdkvhostuser0 mtu
>>1920
>>root at hp-dl20:/# ovs-vsctl del-port trail dpdkvhostuser0
>>root at hp-dl20:/# ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
>>type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1920
>>root at hp-dl20:/# ovs-vsctl get interface dpdkvhostuser0 mtu
>>1500
>>
>>Regards
>>Kapil.
>>
>>On Mon, Feb 20, 2017 at 2:27 PM, Stokes, Ian <ian.stokes at intel.com> wrote:
>>Hi Kapil,
>>
>>Myself and my colleague Mark Kavanagh (the author of the Jumbo Frames implementation) are
>>planning to take a look at this today.
>>
>>We were unable to reproduce the issue itself last week as again the port always reports an
>>error when we attempt to set MTU > 1894 so in effect we never get to the stage where we can
>>delete and re-request the MTU.
>>
>>To help with this can you confirm/provide the following data:
>>
>>•        OVS version: 2.6.1
>>•        DPDK version: 16.07
>>• Was OvS installed through a package manager or built from source?
>>• If OVS is a release version, is it the latest version of that release?
>>• OS Version: ?
>>• Kernel Version: ?
>>• HW Platform: ?
>>• CPU version and frequency: ?
>>•        Commands & parameters used to launch OVS (In particular any command related to
>>memory setup)
>>
>>
>>Could you also provide the vswitch logs? There might be something there that could help us
>>root cause.
>>
>>Thanks
>>Ian
>>
>>
>>
>>From: Kapil Adhikesavalu [mailto:kapil20084 at gmail.com]
>>Sent: Monday, February 20, 2017 7:08 AM
>>To: Stokes, Ian <ian.stokes at intel.com>
>>Subject: Re: [ovs-dev] Memory pool issue on delete and re-add of DPDK ports with jumbo MTU
>>>1894
>>
>>Hi Ian,
>>
>>Any thoughts on my observation ? Looks like a possible bug with MTU implementation.
>>
>>Regards
>>Kapil.
>>
>>On Thu, Feb 16, 2017 at 11:49 AM, Kapil Adhikesavalu <kapil20084 at gmail.com> wrote:
>>Hi Ian,
>>
>>Thanks for the information.
>>
>>I already figured increasing the dpdk-socket-mem to 2048M from 1024M is allowing me to use
>>1920B as MTU(1920B is all i need) consistently across port add/delete. But i have a memory
>>constrain on my system, so all that i could afford is 1024M for socket memory.
>>
>>Coming back to the issue;
>>
>>The very first time after starting OVS, i am able to create ports and assign MTU as1920(with
>>1024M socket mem) without any issues, no failure logs and i am able read back MTU as1920 in
>>get mtu and send packets size of 1920B size.
>>I have been using it this way for about 2 months without any issues.
>>
>>Problem is seen only when i clean up all the ports on the bridges and add back a new port
>>with the same 1920B MTU.
>>Considering that i am able to assign 1920B the very first time and as well as when i restart
>>the OVS process, i suspect there is some issue with OVS/DPDK in freeing up the allocated
>>space on port deletion.
>>
>>Other strange observation i made is:
>>Along with all the ports and bridges i needed to use,  if i add a dummy port with 1920
>>MTU..Now i am able to remove and re-add all my intended ports without any issues by keeping
>>this dummy port always existing.
>>
>>Logs from my trail:
>>=============
>>
>>Trail 1 with dummy port MTU as 1920, helps to remove and add back my intended port with
>>desired MTU
>>============================================================================
>>
>>Note: in my actual setup, i have 10 ports in total and observed the same behavior, just
>>picking 2 ports here for the trail
>>
>>pkill ovs
>>./lib/systemd/ovs-start.sh
>>
>>tail -f /var/log/openvswitch/ovs-vswitchd.log &
>>
>>ovs-vsctl add-br trail -- set bridge trail datapath_type=netdev
>>ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0 type=dpdkvhostuser -
>-
>>set Interface dpdkvhostuser0 mtu_request=1920
>>ovs-vsctl get interface dpdkvhostuser0 mtu
>>1920
>>ovs-vsctl add-port trail dpdkvhostuser1 -- set Interface dpdkvhostuser1 type=dpdkvhostuser -
>-
>>set Interface dpdkvhostuser1 mtu_request=1920
>>ovs-vsctl get interface dpdkvhostuser1 mtu
>>1920
>>
>>ovs-vsctl add-port trail dummy -- set Interface dummy type=dpdkvhostuser -- set Interface
>>dummy mtu_request=1920
>>ovs-vsctl get interface dummy mtu
>>1920
>>
>>ovs-vsctl del-port trail dpdkvhostuser0
>>ovs-vsctl del-port trail dpdkvhostuser1
>>
>>ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0 type=dpdkvhostuser -
>-
>>set Interface dpdkvhostuser0 mtu_request=1920
>>ovs-vsctl get interface dpdkvhostuser0 mtu
>>1920
>>ovs-vsctl add-port trail dpdkvhostuser1 -- set Interface dpdkvhostuser1 type=dpdkvhostuser -
>-
>>set Interface dpdkvhostuser1 mtu_request=1920
>>ovs-vsctl get interface dpdkvhostuser1 mtu
>>1920
>>
>>ovs-vsctl del-port trail dpdkvhostuser0
>>ovs-vsctl del-port trail dpdkvhostuser1
>>ovs-vsctl del-port trail dummy
>>
>>After removing the last dummy port, same issue
>>====================================
>>
>>ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0 type=dpdkvhostuser -
>-
>>set Interface dpdkvhostuser0 mtu_request=1920
>>|dpdk|ERR|Insufficient memory to create memory pool for netdev dpdkvhostuser0, with MTU 1920
>>on socket 0
>>ovs-vsctl get interface dpdkvhostuser0 mtu
>>1500
>>ovs-vsctl add-port trail dpdkvhostuser1 -- set Interface dpdkvhostuser1 type=dpdkvhostuser -
>-
>>set Interface dpdkvhostuser1 mtu_request=1920
>>|dpdk|ERR|Insufficient memory to create memory pool for netdev dpdkvhostuser1, with MTU 1920
>>on socket 0
>>ovs-vsctl get interface dpdkvhostuser1 mtu
>>1500
>>
>>Trail 2 with dummy port MTU as 1500, has the same issue when i remove and add back my
>>intended port with desired MTTU
>>========================================================================================
>>
>>pkill ovs
>>./lib/systemd/scripts/bristol-ovs.sh
>>
>>tail -f /var/log/openvswitch/ovs-vswitchd.log &
>>
>>
>>ovs-vsctl add-br trail -- set bridge trail datapath_type=netdev
>>ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0 type=dpdkvhostuser -
>-
>>set Interface dpdkvhostuser0 mtu_request=1920
>>ovs-vsctl get interface dpdkvhostuser0 mtu
>>1920
>>ovs-vsctl add-port trail dpdkvhostuser1 -- set Interface dpdkvhostuser1 type=dpdkvhostuser -
>-
>>set Interface dpdkvhostuser1 mtu_request=1920
>>ovs-vsctl get interface dpdkvhostuser1 mtu
>>1920
>>
>>ovs-vsctl add-port trail dummy -- set Interface dummy type=dpdkvhostuser -- set Interface
>>dummy mtu_request=1500
>>ovs-vsctl get interface dummy mtu
>>1500
>>
>>ovs-vsctl del-port trail dpdkvhostuser0
>>ovs-vsctl del-port trail dpdkvhostuser1
>>
>>ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0 type=dpdkvhostuser -
>-
>>set Interface dpdkvhostuser0 mtu_request=1920
>>|dpdk|ERR|Insufficient memory to create memory pool for netdev dpdkvhostuser0, with MTU 1920
>>on socket 0
>>ovs-vsctl get interface dpdkvhostuser0 mtu
>>1500
>>ovs-vsctl add-port trail dpdkvhostuser1 -- set Interface dpdkvhostuser1 type=dpdkvhostuser -
>-
>>set Interface dpdkvhostuser1 mtu_request=1920
>>|dpdk|ERR|Insufficient memory to create memory pool for netdev dpdkvhostuser1, with MTU 1920
>>on socket 0
>>ovs-vsctl get interface dpdkvhostuser1 mtu
>>1500
>>
>>
>>Regards
>>Kapil.
>>
>>On Thu, Feb 16, 2017 at 3:47 AM, Stokes, Ian <ian.stokes at intel.com> wrote:
>>> Hi,
>>>
>>> WIth OVS + DPDK, after i add a port with MTU size greater than >1894, if i
>>> do port delete and re-add the port with same MTU, it leads to following
>>> memory pool error and MTU is set to 1500.
>>> While re-adding the port it the MTU is configured less than 1894, issue is
>>> not seen. To recover, i need to kill the OVS process and start it again.
>>
>>What I suspect is happening here is that you are allocating too little hugepage memory to
>OVS
>>with DPDK. From the details below it looks like you are assigning 1024 MB, can you confirm
>>this?
>>
>>I see the same error message you describe when requesting an MTU >= 1895 and with only 1024
>>hugepages memory set (although I see the error the first time the MTU is requested).
>>
>>I'm surprised you only see the issue after deleting the port and re-adding it, can you
>>confirm in your logs that you don't see the error when the port is added the first time with
>>the 1895 MTU request?
>>
>>Also can you increase the hugepage memory being assigned to OVS with DPDK, it should allow
>>you to request/set a larger MTU.
>>
>>Below is a link to a very good article that explains the memory requirement's when using
>>Jumbo frames with OVS with DPDK if you are interested
>>
>>https://software.intel.com/en-us/articles/jumbo-frames-in-open-vswitch-with-dpdk?language=ru
>>
>>Ian
>>
>>>
>>> ovs-vswitchd[14264]: ovs|00002|dpdk|ERR|Insufficient memory to create
>>> memory pool for netdev dpdkvhostuser0, with MTU 1895 on socket 0
>>>
>>> Issue is not seen when ports use MTU size <=1894. Let me know if
>>> additional logs are needed.
>>>
>>> OVS version: 2.6.1 and DPDK: 16.07
>>> ===========================
>>>
>>> root at hp-dl20:/# ovs-vsctl --version
>>> ovs-vsctl (Open vSwitch) 2.6.1
>>> DB Schema 7.14.0
>>>
>>> root at hp-dl20:/# ovs-vsctl show
>>> 176cae66-f14b-436e-9cb2-a7caa054c481
>>>
>>> Issue case with MTU 1895:  (when port with mtu 1895 is readded error is
>>> seen, able to configure MTU <=1894) ==================== ovs-vsctl add-br
>>> trail -- set bridge trail datapath_type=netdev ovs-vsctl add-port trail
>>> dpdkvhostuser0 -- set Interface dpdkvhostuser0 type=dpdkvhostuser -- set
>>> Interface dpdkvhostuser0 mtu_request=1895 root at hp-dl20:/# ovs-vsctl get
>>> interface dpdkvhostuser0 mtu
>>> 1895
>>>
>>> root at hp-dl20:/# ovs-vsctl show
>>> d5b44a7b-62c1-4096-82a4-722833d3d154
>>>     Bridge trail
>>>         Port "dpdkvhostuser0"
>>>             Interface "dpdkvhostuser0"
>>>                 type: dpdkvhostuser
>>>         Port trail
>>>             Interface trail
>>>                 type: internal
>>>
>>> ovs-vsctl del-port trail dpdkvhostuser0
>>> ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
>>> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1895
>>>
>>> 2017-02-15T15:39:23.830Z|00002|dpdk|ERR|Insufficient memory to create
>>> memory pool for netdev dpdkvhostuser0, with MTU 1895 on socket 0
>>>
>>> ovs-vsctl get interface dpdkvhostuser0 mtu
>>> 1500
>>>
>>> root at hp-dl20:/# ovs-vsctl set Interface dpdkvhostuser0 mtu_request=1894
>>> root at hp-dl20:/# ovs-vsctl get interface dpdkvhostuser0 mtu
>>> 1894
>>>
>>> Issue case with MTU 1895:
>>> ====================
>>> pkill ovs and restart OVS
>>>
>>> root     15206  0.0  0.0  23180  4788 ?        Ss   15:42   0:00
>>> /usr/sbin/ovsdb-server --remote=punix:/var/run/openvswitch/db.sock
>>> --remote=db:Open_vSwitch,Open_vSwitch,manager_options
>>> --private-key=db:Open_vSwitch,SSL,private_key
>>> --certificate=db:Open_vSwitch,SSL,certificate
>>> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach --
>>> verbose=err
>>> root     15213 10.1  0.0 1836764 5960 ?        Ssl  15:42   0:01
>>> /usr/sbin/ovs-vswitchd --pidfile --log-file --verbose=err --detach
>>>
>>> ovs-vsctl add-br trail -- set bridge trail datapath_type=netdev ovs-vsctl
>>> add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
>>> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1894
>>> root at hp-dl20:/# ovs-vsctl get interface dpdkvhostuser0 mtu
>>> 1894
>>>
>>> ovs-vsctl del-port trail dpdkvhostuser0
>>> ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
>>> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1894 ovs-
>>> vsctl get interface dpdkvhostuser0 mtu
>>> 1894
>>>
>>> When moved from 1894 to 1895
>>> ========================
>>> pkill ovs and restart OVS
>>>
>>> ovs-vsctl add-br trail -- set bridge trail datapath_type=netdev ovs-vsctl
>>> add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
>>> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1894 ovs-
>>> vsctl get interface dpdkvhostuser0 mtu
>>> 1894
>>>
>>> ovs-vsctl del-port trail dpdkvhostuser0
>>> ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
>>> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1895 ovs-
>>> vsctl get interface dpdkvhostuser0 mtu
>>> 1895
>>> ovs-vsctl del-port trail dpdkvhostuser0
>>> ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
>>> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1895 2017-
>>> 02-15T15:43:28.066Z|00002|dpdk|ERR|Insufficient memory to create memory
>>> pool for netdev dpdkvhostuser0, with MTU 1895 on socket 0 ovs-vsctl get
>>> interface dpdkvhostuser0 mtu
>>> 1500
>>>
>>> root at hp-dl20:/# uname -r
>>> 4.1.26-yocto-standard
>>>
>>> root at hp-dl20:/# cat /proc/meminfo
>>> HugePages_Total:    1536
>>> HugePages_Free:     1024
>>> HugePages_Rsvd:        0
>>> HugePages_Surp:        0
>>> Hugepagesize:       2048 kB
>>>
>>> Regards
>>> Kapil.
>>> _______________________________________________
>>> dev mailing list
>>> dev at openvswitch.org
>>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>>
>>
>
>--
>Thanks
>Kapil


More information about the dev mailing list