[ovs-discuss] ovs-master crashes due to double locking of ofproto_mutex.

Vishal Deep Ajmera vishal.deep.ajmera at ericsson.com
Wed Apr 3 13:10:02 UTC 2019


Hi Ben, Jarno,

I am not sure why we have to take a lock (ofproto_mutex) for processing packet-out action. I think this action does not modify any open flow table. I could see that lock is required for actions like flow_add/flow_mod etc. as they are modifying the existing flow tables.

Any pointers for reasoning behind this will be very helpful. We are frequently hitting this scenario in our deployment and as you see below, it is quite easy to reproduce this by a set of open flow rules.

Warm Regards,
Vishal Ajmera
----
Following commit id introduced this lock in handle_packet_out:

commit 1f4a893366826e392722d5b1ba59e94331bfe5c9
Author: Jarno Rajahalme <jarno at ovn.org>
Date:   Wed Sep 14 16:51:27 2016 -0700

    ofproto: Refactor packet_out handling.

    Refactor handle_packet_out() to prepare for bundle support for packet
    outs in a later patch.

    Two new callbacks are introduced in ofproto-provider class:
    ->packet_xlate() and ->packet_execute().  ->packet_xlate() translates
    the packet using the flow and actions provided by the caller, but
    defers all OpenFlow-visible side-effects (stats, learn actions, actual
    packet output, etc.) to be explicitly executed with the
    ->packet_execute() call.

    Adds a new ofproto_rule_reduce_timeouts__() that must be called with
    'ofproto_mutex' held.  This is used in the next patch.

    Signed-off-by: Jarno Rajahalme <jarno at ovn.org>
    Acked-by: Ben Pfaff <blp at ovn.org>


From: ovs-discuss-bounces at openvswitch.org <ovs-discuss-bounces at openvswitch.org> On Behalf Of Anil Kumar Koli via discuss
Sent: Wednesday, April 3, 2019 3:37 PM
To: ovs-discuss at openvswitch.org
Subject: [ovs-discuss] ovs-master crashes due to double locking of ofproto_mutex.

Hello OVS team,

OVS crash is observed in ovs-master when controller sends a packet with packet-out (output port as OFPP_TABLE) and any of the openflow table entry which gets hit results into a learn action.

Steps to reproduce:
1. Start the ovs-vswitchd in dpdk mode.
2. Configure the following flows
ovs-ofctl -OOpenflow13 add-flow br-int "table=0, priority=50, ct_state=-trk,ip, in_port=10 actions=ct(table=0)"
ovs-ofctl -OOpenflow13 add-flow br-int "table=0, priority=50, ct_state=+trk,ip, in_port=10 actions=ct(commit),resubmit(,1)"
ovs-ofctl -OOpenflow13 add-flow br-int "table=1 actions=learn(table=2,NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:NXM_OF_IN_PORT[]->NXM_NX_REG0[0..15], output:NXM_NX_REG0[0..15]),resubmit(,2)"
3. Send a packet with output as OFPP_TABLE
ovs-ofctl -OOpenflow13 packet-out br-int 'in_port=10 packet=505400000007101111111111080045000028000000004006f97cc0a80001c0a800020008000a0000000000000000500200002e7d0000, actions=TABLE'

This leads to a crash which is a case of double locking of oproto_mutex in handle_packet_out and ofproto_flow_mod_learn.
Can some one provide more insight of introducing ofproto_mutex lock in handle_packet_out which is the cause for above crash?

Please find the backtrace in the attachment.

Thanks & Regards,
Anil Kumar.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20190403/3bfa8d72/attachment.html>


More information about the discuss mailing list