[ovs-dev] Memory pool issue on delete and re-add of DPDK ports with jumbo MTU >1894

Kapil Adhikesavalu kapil20084 at gmail.com
Tue Feb 28 03:59:52 UTC 2017


Hi Mark,

Is there any patch I can expect for this issue?

On Tue, Feb 21, 2017, 7:28 PM Kapil Adhikesavalu <kapil20084 at gmail.com>
wrote:

> Hi Mark,
>
> Thanks for the detailed analysis, i will wait for further updates.
>
> Btw on the mailing list, i seem to have clicked 'reply' instead of
> 'replyall' :)
>
> Regards
> Kapil.
>
> On Tue, Feb 21, 2017 at 3:26 PM, Kavanagh, Mark B <
> mark.b.kavanagh at intel.com> wrote:
>
> + dev - please keep the list included in any discussions going forward
> folks :)
>
> Hi Kapil,
>
> I've managed to reproduce the issue of removing and re-adding ports with
> MTU > 1894, using the following configuration:
>
>         DPDK:           v16.07
>         OVS:            f922f0f1c9
>         dpdk-socket-mem: 1024
>         hugepage-sz:    2M
>         port MTU:               1920
>
> The same behavior is observed with both Phy ports and vhost-user ports;
> when I add a port with MTU 1920, subsequently remove it, and then re-add
> it, I receive an error message stating that insufficient memory is
> available to create a mempool for that port.
>
> A few notes on this:
>         - this behavior is observed only when attempting to re-add the
> same port (more on this later); if I add a new port with MTU 1920, the
> issue doesn't occur, indicating that lack of available memory is not the
> root-cause
>         - the reason that you don't see the issue when you add a 'dummy'
> port is because that port is also of type dpdkvhostuser and shares the same
> MTU - and thus the same mempool - as the other dpdkvhostuser ports that
> you've added. A mempool can only be deleted when all ports that use it are
> deleted; since you never delete the 'dummy' port, the mempool is never
> destroyed, and so when you re-add the other dpdkvhostuser ports, they
> simply use the existing mempool that they previously did (and which is
> still in use by 'dummy').
>
> In the course of debugging, I traced the issue to the DPDK function
> rte_memzone_reserve_aligned_thread_unsafe, which is invoked in the call
> hierarchy of rte_mempool_create (in turn, invoked by dpdk_mp_get). By
> instrumenting the code, I observed that when I attempt to re-add a port
> with MTU 1920, a check for the pre-existence of one particular memzone
> fails in this function:
>
>         ovs-vswitchd[59273]: EAL: memzone_reserve_aligned_thread_unsafe():
> reserving memzone for mempool: === MP_ovs_mp_3054_0_262144_130 ===
>         ovs-vswitchd[59273]: EAL: memzone_reserve_aligned_thread_unsafe():
> memzone <MP_ovs_mp_3054_0_262144_130> already exists
>         ovs-vswitchd[59273]: ovs|00069|dpdk|ERR|Insufficient memory to
> create memory pool for netdev dpdk0, with MTU 1920 on socket 0
>
> Since that function fails, it triggers the 'insufficient memory' log in
> OvS-DPDK - this is certainly something that we can remedy, as it is
> misleading.
>
> Curiously, I don't observe this behavior when dpdk-socket-mem=2048, in
> which case the port may be deleted and re-added without issue. I'll consult
> my DPDK colleagues on this, and get back to you with an answer in the
> coming days.
>
> Thanks,
> Mark
>
> >
> >Hi Ian,
> >
> >Thanks for looking into it.
> >
> >Please find the information below and let me know if anything else is
> needed.
> >
> >•        OVS version: 2.6.1
> >•        DPDK version: 16.07
> >• Was OvS installed through a package manager or built from source?
> Built from source.
> >• If OVS is a release version, is it the latest version of that release?
>  branch-2.6 with
> >head commit f922f0f1c926bb7596ba4e0971960dc89e39f0e7
> >• Author: Thadeu Lima de Souza Cascardo <cascardo at redhat.com>
> >• Date:   Wed Oct 19 13:32:57 2016 -0200
> >• OS Version: ?  It's a yocto based OS with 4.1 Linux kernel uname -r >
> 4.1.26-yocto-standard
> >• HW Platform: ? HP gen9 DL20 proliant
> >• CPU version and frequency: ?   Intel(R) Xeon(R) CPU E3-1220 v5 @
> 3.00GHz, 4 cores
> >•        Commands & parameters used to launch OVS (In particular any
> command related to
> >memory setup)
> >
> >root at hp-dl20:/# cat /proc/cmdline
> >BOOT_IMAGE=/dl20.bin root=/dev/ram0 ip=dhcp default_hugepagesz=2M
> hugepagesz=2M
> >hugepages=1536 isolcpus=1-3
> >
> ># Create ovs config
> >mkdir -p /var/log/openvswitch
> >ovsdb-tool create $ovsdir/etc/openvswitch/conf.db
> >$ovsdir/usr/share/openvswitch/vswitch.ovsschema
> >
> ># Bring up ovsdb-server daemon
> >mkdir -p $ovsdir/var/run/openvswitch
> >/usr/sbin/ovsdb-server --remote=punix:$ovsdir/var/run/openvswitch/db.sock
> \
> >               --remote=db:Open_vSwitch,Open_vSwitch,manager_options \
> >               --private-key=db:Open_vSwitch,SSL,private_key \
> >               --certificate=db:Open_vSwitch,SSL,certificate \
> >               --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \
> >               --pidfile --detach --verbose=err
> >
> ># Intialize the ovs database
> >/usr/bin/ovs-vsctl --no-wait init
> >
> >  dpdk_socket_mem="1024,0"
> >  dpdk_lcore_mask=0x1
> >
> >  # Specify DPDK options for very newest as-of-yet-unreleased rev of OVS
> >  /usr/bin/ovs-vsctl --no-wait set Open_vSwitch .
> other_config:dpdk-init=true
> >
> >  # Number of memory channels on targeted platform
> >  /usr/bin/ovs-vsctl --no-wait set Open_vSwitch .
> other_config:dpdk-extra="-n 4"
> >
> >  # Set dpdk-socket-mem
> >  /usr/bin/ovs-vsctl --no-wait set Open_vSwitch .
> other_config:dpdk-socket-
> >mem=$dpdk_socket_mem
> >
> >  # Set dpdk-lcore-mask
> >  /usr/bin/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-
> >mask=$dpdk_lcore_mask
> >
> >To collect the below logs in attachment, i enabled --verbose and ran
> following commands to
> >hit the issue
> >
> >root at hp-dl20:/# ovs-vsctl add-br trail -- set bridge trail
> datapath_type=netdev
> >root at hp-dl20:/# ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface
> dpdkvhostuser0
> >type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1920
> >root at hp-dl20:/# ovs-vsctl get interface dpdkvhostuser0 mtu
> >1920
> >root at hp-dl20:/# ovs-vsctl del-port trail dpdkvhostuser0
> >root at hp-dl20:/# ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface
> dpdkvhostuser0
> >type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1920
> >root at hp-dl20:/# ovs-vsctl get interface dpdkvhostuser0 mtu
> >1500
> >
> >Regards
> >Kapil.
> >
> >On Mon, Feb 20, 2017 at 2:27 PM, Stokes, Ian <ian.stokes at intel.com>
> wrote:
> >Hi Kapil,
> >
> >Myself and my colleague Mark Kavanagh (the author of the Jumbo Frames
> implementation) are
> >planning to take a look at this today.
> >
> >We were unable to reproduce the issue itself last week as again the port
> always reports an
> >error when we attempt to set MTU > 1894 so in effect we never get to the
> stage where we can
> >delete and re-request the MTU.
> >
> >To help with this can you confirm/provide the following data:
> >
> >•        OVS version: 2.6.1
> >•        DPDK version: 16.07
> >• Was OvS installed through a package manager or built from source?
> >• If OVS is a release version, is it the latest version of that release?
> >• OS Version: ?
> >• Kernel Version: ?
> >• HW Platform: ?
> >• CPU version and frequency: ?
> >•        Commands & parameters used to launch OVS (In particular any
> command related to
> >memory setup)
> >
> >
> >Could you also provide the vswitch logs? There might be something there
> that could help us
> >root cause.
> >
> >Thanks
> >Ian
> >
> >
> >
> >From: Kapil Adhikesavalu [mailto:kapil20084 at gmail.com]
> >Sent: Monday, February 20, 2017 7:08 AM
> >To: Stokes, Ian <ian.stokes at intel.com>
> >Subject: Re: [ovs-dev] Memory pool issue on delete and re-add of DPDK
> ports with jumbo MTU
> >>1894
> >
> >Hi Ian,
> >
> >Any thoughts on my observation ? Looks like a possible bug with MTU
> implementation.
> >
> >Regards
> >Kapil.
> >
> >On Thu, Feb 16, 2017 at 11:49 AM, Kapil Adhikesavalu <
> kapil20084 at gmail.com> wrote:
> >Hi Ian,
> >
> >Thanks for the information.
> >
> >I already figured increasing the dpdk-socket-mem to 2048M from 1024M is
> allowing me to use
> >1920B as MTU(1920B is all i need) consistently across port add/delete.
> But i have a memory
> >constrain on my system, so all that i could afford is 1024M for socket
> memory.
> >
> >Coming back to the issue;
> >
> >The very first time after starting OVS, i am able to create ports and
> assign MTU as1920(with
> >1024M socket mem) without any issues, no failure logs and i am able read
> back MTU as1920 in
> >get mtu and send packets size of 1920B size.
> >I have been using it this way for about 2 months without any issues.
> >
> >Problem is seen only when i clean up all the ports on the bridges and add
> back a new port
> >with the same 1920B MTU.
> >Considering that i am able to assign 1920B the very first time and as
> well as when i restart
> >the OVS process, i suspect there is some issue with OVS/DPDK in freeing
> up the allocated
> >space on port deletion.
> >
> >Other strange observation i made is:
> >Along with all the ports and bridges i needed to use,  if i add a dummy
> port with 1920
> >MTU..Now i am able to remove and re-add all my intended ports without any
> issues by keeping
> >this dummy port always existing.
> >
> >Logs from my trail:
> >=============
> >
> >Trail 1 with dummy port MTU as 1920, helps to remove and add back my
> intended port with
> >desired MTU
>
> >============================================================================
> >
> >Note: in my actual setup, i have 10 ports in total and observed the same
> behavior, just
> >picking 2 ports here for the trail
> >
> >pkill ovs
> >./lib/systemd/ovs-start.sh
> >
> >tail -f /var/log/openvswitch/ovs-vswitchd.log &
> >
> >ovs-vsctl add-br trail -- set bridge trail datapath_type=netdev
> >ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> type=dpdkvhostuser --
> >set Interface dpdkvhostuser0 mtu_request=1920
> >ovs-vsctl get interface dpdkvhostuser0 mtu
> >1920
> >ovs-vsctl add-port trail dpdkvhostuser1 -- set Interface dpdkvhostuser1
> type=dpdkvhostuser --
> >set Interface dpdkvhostuser1 mtu_request=1920
> >ovs-vsctl get interface dpdkvhostuser1 mtu
> >1920
> >
> >ovs-vsctl add-port trail dummy -- set Interface dummy type=dpdkvhostuser
> -- set Interface
> >dummy mtu_request=1920
> >ovs-vsctl get interface dummy mtu
> >1920
> >
> >ovs-vsctl del-port trail dpdkvhostuser0
> >ovs-vsctl del-port trail dpdkvhostuser1
> >
> >ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> type=dpdkvhostuser --
> >set Interface dpdkvhostuser0 mtu_request=1920
> >ovs-vsctl get interface dpdkvhostuser0 mtu
> >1920
> >ovs-vsctl add-port trail dpdkvhostuser1 -- set Interface dpdkvhostuser1
> type=dpdkvhostuser --
> >set Interface dpdkvhostuser1 mtu_request=1920
> >ovs-vsctl get interface dpdkvhostuser1 mtu
> >1920
> >
> >ovs-vsctl del-port trail dpdkvhostuser0
> >ovs-vsctl del-port trail dpdkvhostuser1
> >ovs-vsctl del-port trail dummy
> >
> >After removing the last dummy port, same issue
> >====================================
> >
> >ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> type=dpdkvhostuser --
> >set Interface dpdkvhostuser0 mtu_request=1920
> >|dpdk|ERR|Insufficient memory to create memory pool for netdev
> dpdkvhostuser0, with MTU 1920
> >on socket 0
> >ovs-vsctl get interface dpdkvhostuser0 mtu
> >1500
> >ovs-vsctl add-port trail dpdkvhostuser1 -- set Interface dpdkvhostuser1
> type=dpdkvhostuser --
> >set Interface dpdkvhostuser1 mtu_request=1920
> >|dpdk|ERR|Insufficient memory to create memory pool for netdev
> dpdkvhostuser1, with MTU 1920
> >on socket 0
> >ovs-vsctl get interface dpdkvhostuser1 mtu
> >1500
> >
> >Trail 2 with dummy port MTU as 1500, has the same issue when i remove and
> add back my
> >intended port with desired MTTU
>
> >========================================================================================
> >
> >pkill ovs
> >./lib/systemd/scripts/bristol-ovs.sh
> >
> >tail -f /var/log/openvswitch/ovs-vswitchd.log &
> >
> >
> >ovs-vsctl add-br trail -- set bridge trail datapath_type=netdev
> >ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> type=dpdkvhostuser --
> >set Interface dpdkvhostuser0 mtu_request=1920
> >ovs-vsctl get interface dpdkvhostuser0 mtu
> >1920
> >ovs-vsctl add-port trail dpdkvhostuser1 -- set Interface dpdkvhostuser1
> type=dpdkvhostuser --
> >set Interface dpdkvhostuser1 mtu_request=1920
> >ovs-vsctl get interface dpdkvhostuser1 mtu
> >1920
> >
> >ovs-vsctl add-port trail dummy -- set Interface dummy type=dpdkvhostuser
> -- set Interface
> >dummy mtu_request=1500
> >ovs-vsctl get interface dummy mtu
> >1500
> >
> >ovs-vsctl del-port trail dpdkvhostuser0
> >ovs-vsctl del-port trail dpdkvhostuser1
> >
> >ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> type=dpdkvhostuser --
> >set Interface dpdkvhostuser0 mtu_request=1920
> >|dpdk|ERR|Insufficient memory to create memory pool for netdev
> dpdkvhostuser0, with MTU 1920
> >on socket 0
> >ovs-vsctl get interface dpdkvhostuser0 mtu
> >1500
> >ovs-vsctl add-port trail dpdkvhostuser1 -- set Interface dpdkvhostuser1
> type=dpdkvhostuser --
> >set Interface dpdkvhostuser1 mtu_request=1920
> >|dpdk|ERR|Insufficient memory to create memory pool for netdev
> dpdkvhostuser1, with MTU 1920
> >on socket 0
> >ovs-vsctl get interface dpdkvhostuser1 mtu
> >1500
> >
> >
> >Regards
> >Kapil.
> >
> >On Thu, Feb 16, 2017 at 3:47 AM, Stokes, Ian <ian.stokes at intel.com>
> wrote:
> >> Hi,
> >>
> >> WIth OVS + DPDK, after i add a port with MTU size greater than >1894,
> if i
> >> do port delete and re-add the port with same MTU, it leads to following
> >> memory pool error and MTU is set to 1500.
> >> While re-adding the port it the MTU is configured less than 1894, issue
> is
> >> not seen. To recover, i need to kill the OVS process and start it again.
> >
> >What I suspect is happening here is that you are allocating too little
> hugepage memory to OVS
> >with DPDK. From the details below it looks like you are assigning 1024
> MB, can you confirm
> >this?
> >
> >I see the same error message you describe when requesting an MTU >= 1895
> and with only 1024
> >hugepages memory set (although I see the error the first time the MTU is
> requested).
> >
> >I'm surprised you only see the issue after deleting the port and
> re-adding it, can you
> >confirm in your logs that you don't see the error when the port is added
> the first time with
> >the 1895 MTU request?
> >
> >Also can you increase the hugepage memory being assigned to OVS with
> DPDK, it should allow
> >you to request/set a larger MTU.
> >
> >Below is a link to a very good article that explains the memory
> requirement's when using
> >Jumbo frames with OVS with DPDK if you are interested
> >
> >
> https://software.intel.com/en-us/articles/jumbo-frames-in-open-vswitch-with-dpdk?language=ru
> >
> >Ian
> >
> >>
> >> ovs-vswitchd[14264]: ovs|00002|dpdk|ERR|Insufficient memory to create
> >> memory pool for netdev dpdkvhostuser0, with MTU 1895 on socket 0
> >>
> >> Issue is not seen when ports use MTU size <=1894. Let me know if
> >> additional logs are needed.
> >>
> >> OVS version: 2.6.1 and DPDK: 16.07
> >> ===========================
> >>
> >> root at hp-dl20:/# ovs-vsctl --version
> >> ovs-vsctl (Open vSwitch) 2.6.1
> >> DB Schema 7.14.0
> >>
> >> root at hp-dl20:/# ovs-vsctl show
> >> 176cae66-f14b-436e-9cb2-a7caa054c481
> >>
> >> Issue case with MTU 1895:  (when port with mtu 1895 is readded error is
> >> seen, able to configure MTU <=1894) ==================== ovs-vsctl
> add-br
> >> trail -- set bridge trail datapath_type=netdev ovs-vsctl add-port trail
> >> dpdkvhostuser0 -- set Interface dpdkvhostuser0 type=dpdkvhostuser -- set
> >> Interface dpdkvhostuser0 mtu_request=1895 root at hp-dl20:/# ovs-vsctl get
> >> interface dpdkvhostuser0 mtu
> >> 1895
> >>
> >> root at hp-dl20:/# ovs-vsctl show
> >> d5b44a7b-62c1-4096-82a4-722833d3d154
> >>     Bridge trail
> >>         Port "dpdkvhostuser0"
> >>             Interface "dpdkvhostuser0"
> >>                 type: dpdkvhostuser
> >>         Port trail
> >>             Interface trail
> >>                 type: internal
> >>
> >> ovs-vsctl del-port trail dpdkvhostuser0
> >> ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> >> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1895
> >>
> >> 2017-02-15T15:39:23.830Z|00002|dpdk|ERR|Insufficient memory to create
> >> memory pool for netdev dpdkvhostuser0, with MTU 1895 on socket 0
> >>
> >> ovs-vsctl get interface dpdkvhostuser0 mtu
> >> 1500
> >>
> >> root at hp-dl20:/# ovs-vsctl set Interface dpdkvhostuser0 mtu_request=1894
> >> root at hp-dl20:/# ovs-vsctl get interface dpdkvhostuser0 mtu
> >> 1894
> >>
> >> Issue case with MTU 1895:
> >> ====================
> >> pkill ovs and restart OVS
> >>
> >> root     15206  0.0  0.0  23180  4788 ?        Ss   15:42   0:00
> >> /usr/sbin/ovsdb-server --remote=punix:/var/run/openvswitch/db.sock
> >> --remote=db:Open_vSwitch,Open_vSwitch,manager_options
> >> --private-key=db:Open_vSwitch,SSL,private_key
> >> --certificate=db:Open_vSwitch,SSL,certificate
> >> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach --
> >> verbose=err
> >> root     15213 10.1  0.0 1836764 5960 ?        Ssl  15:42   0:01
> >> /usr/sbin/ovs-vswitchd --pidfile --log-file --verbose=err --detach
> >>
> >> ovs-vsctl add-br trail -- set bridge trail datapath_type=netdev
> ovs-vsctl
> >> add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> >> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1894
> >> root at hp-dl20:/# ovs-vsctl get interface dpdkvhostuser0 mtu
> >> 1894
> >>
> >> ovs-vsctl del-port trail dpdkvhostuser0
> >> ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> >> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1894 ovs-
> >> vsctl get interface dpdkvhostuser0 mtu
> >> 1894
> >>
> >> When moved from 1894 to 1895
> >> ========================
> >> pkill ovs and restart OVS
> >>
> >> ovs-vsctl add-br trail -- set bridge trail datapath_type=netdev
> ovs-vsctl
> >> add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> >> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1894 ovs-
> >> vsctl get interface dpdkvhostuser0 mtu
> >> 1894
> >>
> >> ovs-vsctl del-port trail dpdkvhostuser0
> >> ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> >> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1895 ovs-
> >> vsctl get interface dpdkvhostuser0 mtu
> >> 1895
> >> ovs-vsctl del-port trail dpdkvhostuser0
> >> ovs-vsctl add-port trail dpdkvhostuser0 -- set Interface dpdkvhostuser0
> >> type=dpdkvhostuser -- set Interface dpdkvhostuser0 mtu_request=1895
> 2017-
> >> 02-15T15:43:28.066Z|00002|dpdk|ERR|Insufficient memory to create memory
> >> pool for netdev dpdkvhostuser0, with MTU 1895 on socket 0 ovs-vsctl get
> >> interface dpdkvhostuser0 mtu
> >> 1500
> >>
> >> root at hp-dl20:/# uname -r
> >> 4.1.26-yocto-standard
> >>
> >> root at hp-dl20:/# cat /proc/meminfo
> >> HugePages_Total:    1536
> >> HugePages_Free:     1024
> >> HugePages_Rsvd:        0
> >> HugePages_Surp:        0
> >> Hugepagesize:       2048 kB
> >>
> >> Regards
> >> Kapil.
> >> _______________________________________________
> >> dev mailing list
> >> dev at openvswitch.org
> >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> >
> >
>
>
> --

Thanks
Kapil


More information about the dev mailing list