[ovs-dev] Openvswitch crash when bringing down the dpdk bond port using "ovs-ofctl mod-port br-prv dpdk1 down"

[Ben Pfaff]

OK, it's a reasonable point that DPDK was "experimental" then.

On Mon, Jul 31, 2017 at 03:21:02PM +0000, Darrell Ball wrote:
> OVS 2.6 DPDK is labelled as ‘experimental’
> There were lots of fixes and quick fixes in 2.7 that could be backported, but how to verify those effectively in 2.6 ?
> 
> 
> 
> -----Original Message-----
> From: <ovs-dev-bounces at openvswitch.org> on behalf of Ben Pfaff <blp at ovn.org>
> Date: Monday, July 31, 2017 at 6:05 AM
> To: Ilya Maximets <i.maximets at samsung.com>
> Cc: "ovs-dev at openvswitch.org" <ovs-dev at openvswitch.org>, Keshav Gupta <keshav.gupta at ericsson.com>
> Subject: Re: [ovs-dev] Openvswitch crash when bringing down the dpdk bond port using "ovs-ofctl mod-port br-prv dpdk1 down"
> 
>     Ilya, should we apply this patch to branch-2.6?  Are there other patches
>     that should be backported?
>     
>     On Wed, Jul 26, 2017 at 03:28:12PM +0300, Ilya Maximets wrote:
>     > Hi.
>     > 
>     > You need to backport at least following patch:
>     > 
>     > commit 3b1fb0779b87788968c1a6a9ff295a9883547485
>     > Author: Daniele Di Proietto <diproiettod at vmware.com>
>     > Date:   Tue Nov 15 15:40:49 2016 -0800
>     > 
>     >     netdev-dpdk: Don't call rte_dev_stop() in update_flags().
>     >     
>     >     Calling rte_eth_dev_stop() while the device is running causes a crash.
>     >     
>     >     We could use rte_eth_dev_set_link_down(), but not every PMD implements
>     >     that, and I found one NIC where that has no effect.
>     >     
>     >     Instead, this commit checks if the device has the NETDEV_UP flag when
>     >     transmitting or receiving (similarly to what we do for vhostuser). I
>     >     didn't notice any performance difference with this check in case the
>     >     device is up.
>     >     
>     >     An alternative would be to remove the device queues from the pmd threads
>     >     tx and receive cache, but that requires reconfiguration and I'd prefer
>     >     to avoid it, because the change can come from OpenFlow.
>     >     
>     >     Signed-off-by: Daniele Di Proietto <diproiettod at vmware.com>
>     >     Acked-by: Ilya Maximets <i.maximets at samsung.com>
>     > 
>     > This should fix your issue.
>     > In general, I'm suggesting to use stable 2.7 OVS, there was too many DPDK
>     > related changes including stability fixes since 2.6.
>     > 
>     > Best regards, Ilya Maximets.
>     > 
>     > > Hi
>     > >   We are experiencing a openvswitch crash when bringing down the dpdk bond port using "ovs-ofctl mod-port br-prv dpdk1 down".
>     > > 
>     > > backtrace of core is like below. Is there any issue reported earlier  for this type of crash in openvswitch community.
>     > > 
>     > > (gdb) bt
>     > > #0  ixgbe_rxq_rearm (rxq=0x7fa45061f800) at /home/sdn/new_cloud_sdn_switch_2/cloud-sdn-switch/dpdk/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c:98
>     > > #1  _recv_raw_pkts_vec (split_packet=0x0, nb_pkts=32, rx_pkts=<optimized out>, rxq=0x7fa45061f800)
>     > >     at /home/sdn/new_cloud_sdn_switch_2/cloud-sdn-switch/dpdk/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c:290
>     > > #2  ixgbe_recv_pkts_vec (rx_queue=0x7fa45061f800, rx_pkts=<optimized out>, nb_pkts=<optimized out>)
>     > >     at /home/sdn/new_cloud_sdn_switch_2/cloud-sdn-switch/dpdk/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c:474
>     > > #3  0x000000e5000000e4 in ?? ()
>     > > #4  0x00000046000000e6 in ?? ()
>     > > #5  0x0000006a00000069 in ?? ()
>     > > #6  0x0000006c0000006b in ?? ()
>     > > #7  0x000000ec0000006d in ?? ()
>     > > #8  0x000000ee000000ed in ?? ()
>     > > #9  0x00000001537f5780 in ?? ()
>     > > #10 0x0000000000000000 in ?? ()
>     > > (gdb)
>     > > 
>     > > 
>     > > I have analyzed the core and it seems it is a result of device stop and packet receive from the port happening at same time by two thread
>     > > OVS main thread(device stop) and PMD thread(pkt receive). More precisely main thread cleaning the packet buffer from rxq sw_ring to avoid the
>     > > packet buffer leak while in parallel PMD thread is filling the packet buffer in sw_ring/descriptor ring as part of ixgbe_recv_pkts_vec.
>     > > 
>     > > version used is: openvswitch (2.6.1) with dpdk (16.11).
>     > > 
>     > > This crash is not every time reproducible but frequency seems to be high.
>     > > 
>     > > I am new to openvswitch community and this is first time I am posting a query. let me know if anything you require from my side.
>     > > 
>     > > Thanks
>     > > Keshav
>     > 
>     > _______________________________________________
>     > dev mailing list
>     > dev at openvswitch.org
>     > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.openvswitch.org_mailman_listinfo_ovs-2Ddev&d=DwICAg&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-uZnsw&m=_h2I15eFvYAbv0-w1inBv7SkZmqeC964EejrWik1IS4&s=boUF1mM55SnZvzZ2aoe3e7YsMCVOFLl4kf49Xh5Nxkg&e= 
>     _______________________________________________
>     dev mailing list
>     dev at openvswitch.org
>     https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.openvswitch.org_mailman_listinfo_ovs-2Ddev&d=DwICAg&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-uZnsw&m=_h2I15eFvYAbv0-w1inBv7SkZmqeC964EejrWik1IS4&s=boUF1mM55SnZvzZ2aoe3e7YsMCVOFLl4kf49Xh5Nxkg&e= 
>     
>