[ovs-discuss] OVS misconfiguration issue

[Gregory Rose]

On 5/1/2018 10:35 PM, Neelakantam Gaddam wrote:
> Hi All,
>
> The issue here is we are trying to send the packets on the same device 
> in a loop. While sending on a device, a spinlock for the tx queue has 
> to be acquired in dev_queue_xmit function. This is where we are trying 
> to acquire the same lock again, which is leading to the kernel crash. 
> This issue becomes worse if internal ports are involved in the 
> configuration.
>
> I think, we should avoid these loops in the vport send functions. But 
> in case of tunneling, the tunnel send function should take care of 
> these checks.
>
> Please share your thoughts on this issue.

Hello Neelakantam,

Please provide the full kernel panic backtrace and I'll have a look at 
the problem.  I'm rather busy
dealing with some other critical issues but will try to get back to you 
next week.

Thanks,

- Greg

>
> On Mon, Apr 30, 2018 at 11:17 AM, Neelakantam Gaddam 
> <neelugaddam at gmail.com <mailto:neelugaddam at gmail.com>> wrote:
>
>     Hi All,
>
>     OVS misconfiguration leading to spinlock recursion in dev_queue_xmit.
>
>     We are running ovs-2.8.1 with openvswitch kernel modules on two
>     hosts connected back to back. We are running OVS on MIPS64 platform.
>
>
>
>     We are using the below configuration.
>
>     ovs-vsctl add-br br0
>
>     ovs-vsctl add-bond br0 bond0 p1p1 p1p2
>
>     ovs-vsctl set port bond0 lacp=active bond_mode=balance-tcp
>
>     ifconfig br0 100.0.0.1 up
>
>     ovs-vsctl add-port br0 veth0
>
>     ovs-vsctl add-port br0 vx0 -- set interface vx0 type=vxlan
>     options:local_ip=100.0.0.1 options:remote_ip=100.0.0.2 option:key=flow
>
>     ovs-ofctl add-flow br0 "table=0, priority=1, cookie=100,
>     tun_id=100, in_port=4, action=output:3"
>
>     ovs-ofctl add-flow br0 "table=0, priority=1, cookie=100,
>     in_port=3, actions=set_field:100->tun_id output:4"
>
>     When this configuration is applied on both hosts, we are seeing
>     the below spinlock recursion bug.
>
>     [<ffffffff80864cd4>] show_stack+0x6c/0xf8
>
>     [<ffffffff80ad1628>] do_raw_spin_lock+0x168/0x170
>
>     [<ffffffff80bf7b1c>] dev_queue_xmit+0x43c/0x470
>
>     [<ffffffff80c32c08>] ip_finish_output+0x250/0x490
>
>     [<ffffffffc0115664>] rpl_iptunnel_xmit+0x134/0x218 [openvswitch]
>
>     [<ffffffffc0120f28>] rpl_vxlan_xmit+0x430/0x538 [openvswitch]
>
>     [<ffffffffc00f9de0>] do_execute_actions+0x18f8/0x19e8 [openvswitch]
>
>     [<ffffffffc00fa2b0>] ovs_execute_actions+0x90/0x208 [openvswitch]
>
>     [<ffffffffc0101860>] ovs_dp_process_packet+0xb0/0x1a8 [openvswitch]
>
>     [<ffffffffc010c5d8>] ovs_vport_receive+0x78/0x130 [openvswitch]
>
>     [<ffffffffc010ce6c>] internal_dev_xmit+0x34/0x98 [openvswitch]
>
>     [<ffffffff80bf74d0>] dev_hard_start_xmit+0x2e8/0x4f8
>
>     [<ffffffff80c10e48>] sch_direct_xmit+0xf0/0x238
>
>     [<ffffffff80bf78b8>] dev_queue_xmit+0x1d8/0x470
>
>     [<ffffffff80c5ffe4>] arp_process+0x614/0x628
>
>     [<ffffffff80bf0cb0>] __netif_receive_skb_core+0x2e8/0x5d8
>
>     [<ffffffff80bf4770>] process_backlog+0xc0/0x1b0
>
>     [<ffffffff80bf501c>] net_rx_action+0x154/0x240
>
>     [<ffffffff8088d130>] __do_softirq+0x1d0/0x218
>
>     [<ffffffff8088d240>] do_softirq+0x68/0x70
>
>     [<ffffffff8088d3a0>] local_bh_enable+0xa8/0xb0
>
>     [<ffffffff80bf5c88>] netif_rx_ni+0x20/0x30
>
>     The packet path traced is : netif_rx->arp->dev_queue_xmit(internal
>     port)->vxlan_xmit->dev_queue_xmit(internal port). According to the
>     configuration, this packet path is valid. But we should not hit
>     the crash.
>
>
>     Questions:
>
>
>       * Is it a kernel bug or ovs bug ?
>       * How OVS handles these kind of misconfigurations especially
>         packet loops involved?
>
>     Any suggestion or help is greatly appreciated.
>
>
>
>     Thanks
>
>
>
>     -- 
>     Thanks & Regards
>     Neelakantam Gaddam
>
>
>
>
> -- 
> Thanks & Regards
> Neelakantam Gaddam
>
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20180503/801e0415/attachment-0001.html>