[ovs-discuss] OVS 2.4 and 2.4 Flow deletion issue
Maurya, Alok Kumar (HP Networking)
alok-kumar.maurya at hpe.com
Mon May 9 15:15:19 UTC 2016
Hi Jarno,
Have we managed to merge these changes ?
Thanks & Regards,
Alok
-----Original Message-----
From: Maurya, Alok Kumar (HP Networking)
Sent: Thursday, April 14, 2016 5:46 PM
To: 'Jarno Rajahalme' <jarno at ovn.org>
Cc: Ben Pfaff <blp at ovn.org>; discuss at openvswitch.org
Subject: RE: [ovs-discuss] OVS 2.4 and 2.4 Flow deletion issue
I have tried the fix on master , it solved the problem .
Thanks & Regards,
Alok
-----Original Message-----
From: Jarno Rajahalme [mailto:jarno at ovn.org]
Sent: Thursday, April 14, 2016 7:43 AM
To: Maurya, Alok Kumar (HP Networking) <alok-kumar.maurya at hpe.com>
Cc: Ben Pfaff <blp at ovn.org>; discuss at openvswitch.org
Subject: Re: [ovs-discuss] OVS 2.4 and 2.4 Flow deletion issue
I just sent a fix for this issue in OVS master to the OVS dev list. You can see the patches also in patchwork:
http://patchwork.ozlabs.org/patch/610337/
http://patchwork.ozlabs.org/patch/610338/
http://patchwork.ozlabs.org/patch/610339/
http://patchwork.ozlabs.org/patch/610340/
Jarno
> On Apr 13, 2016, at 12:36 PM, Maurya, Alok Kumar (HP Networking) <alok-kumar.maurya at hpe.com> wrote:
>
> Hi Jarno,
>
> I have 4 CPU , 8 GB RAM VM .
> I am running Ubuntu 14.04 on the VM .
>
> With 200K flows , yes in my system also its 2-3 minutes , however when I have around 350K flows , it took 29-30 mins (sorry the data which I had given was with 350K flows not with 200K flows) .
> When 350 Flows ae there in the system try to delete 17K flows in batch .
>
> I can try to remove in_port=1 from match and see if that makes any difference , but flows will have same in_port as we expect packets with match to come from this port .
>
>
> ### script ####
> echo "Adding 200K Flows"
> date
> for v in {1..20}
>
> do
>
> for i in {1..250}
> do
>
> for j in {1..70}
> do
>
> ovs-ofctl add-flow br-int
> hard_timeout=0,idle_timeout=0,priority=10,ip,cookie=0x5c38cfa3a668$v,t
> able=0,dl_dst=fa:16:3e:af:69:a3,dl_vlan=$v,nw_src=9.16.$i.$j/32,in_por
> t=1,actions=output:1
>
> done
> done
>
> done
> date
> echo "Flows are added"
>
> echo "Dump-Flows"
> ovs-ofctl dump-flows br-int |wc
> date
>
> #### end script #####
>
> Thanks & Regards,
> Alok
>
> -----Original Message-----
> From: Jarno Rajahalme [mailto:jarno at ovn.org]
> Sent: Thursday, April 14, 2016 12:44 AM
> To: Maurya, Alok Kumar (HP Networking) <alok-kumar.maurya at hpe.com>
> Cc: Ben Pfaff <blp at ovn.org>; discuss at openvswitch.org
> Subject: Re: [ovs-discuss] OVS 2.4 and 2.4 Flow deletion issue
>
>
>> On Apr 12, 2016, at 1:05 AM, Maurya, Alok Kumar (HP Networking) <alok-kumar.maurya at hpe.com> wrote:
>>
>> Thanks Jarno and Ben for looking into this issue .
>>
>> I had tried to backtrace this from 2.3 to 2.4 .
>>
>>
>> I found that I had started seeing this issue after f2c214029e1ac0c9bc0f7c2013fe178a6b08dd7f commit , however in that
>> ovs-ofctl del-flows br-int dl_vlan=1
>>
>
> I was able to reproduce the issue with the exact flow pattern you reported, thanks!
>
> The issue is that each of the flows has exactly the same metadata portion in the match (in_port=1). I’m looking into how to alleviate this, but in the meanwhile you could try to remove fields from the matches that make no difference to the outcome of the match.
>
>> took long time to execute ( more than 30 mins) .
>>
>
> On my 4-core VM the delete of the 5% of the flows took almost two minutes, so 30 minutes sounds like a long time. Could you tell a little bit more about your setup where you are executing this? OS, memory, # of cores, etc?
>
>> In case of flow delete now , looks like ovs-vswitchd is sending "barrier-reply" even before flow deletion is completed .
>>
>
> The barrier-reply is returned as soon as the flows have been made invisible, but some of the clean-up work is postponed and can safely happen while following OpenFlow messages are processed. However, in this case the clean-up takes a mutex for a duration of removing the 10000 rules, and due to all the 200000 rules having the exactly same metadata match this now takes a long time.
>
> Regards,
>
> Jarno
>
>> Thanks & Regards,
>> Alok
>>
>> -----Original Message-----
>> From: Jarno Rajahalme [mailto:jarno at ovn.org]
>> Sent: Tuesday, April 12, 2016 7:02 AM
>> To: Ben Pfaff <blp at ovn.org>
>> Cc: Maurya, Alok Kumar (HP Networking) <alok-kumar.maurya at hpe.com>;
>> discuss at openvswitch.org
>> Subject: Re: [ovs-discuss] OVS 2.4 and 2.4 Flow deletion issue
>>
>> I tried with a bit simpler flows, like this:
>>
>> ovs-ofctl add-flow br-int
>> hard_timeout=0,idle_timeout=0,priority=10,ip,cookie=0x$v,table=0,dl_v
>> l
>> an=$v,nw_src=9.16.$i.$j,actions=output:1
>>
>> And while adding them one-by-one in my VM took 12 minutes (my dev VM has been very slow recently), the runtime of the deletion was not noticeable, i.e., a following flow dump returned right away. Deleting all 200 flows took maybe 2-3 seconds.
>>
>> I’m trying again with the exact flow reported, but I’d be surprised if the additional match fields would make a difference.
>>
>> This on tip of master. I’ll have to try again with releases 2.4 and 2.5.
>>
>> Jarno
>>
>>> On Apr 11, 2016, at 5:15 PM, Jarno Rajahalme <jarno at ovn.org> wrote:
>>>
>>> I’m looking into this,
>>>
>>> Jarno
>>>
>>>> On Apr 11, 2016, at 3:41 PM, Ben Pfaff <blp at ovn.org> wrote:
>>>>
>>>> Jarno, it seems likely that this has something to do with the
>>>> transactional classifier implementation. Do you think so? Do you
>>>> have any idea why deletions would be so slow?
>>>>
>>>> Thanks,
>>>>
>>>> Ben.
>>>>
>>>> On Mon, Apr 11, 2016 at 09:51:45PM +0000, Maurya, Alok Kumar (HP Networking) wrote:
>>>>> Hi All,
>>>>>
>>>>> I am seeing that in OVS 2.4 and ovs 2.5 Bulk flow deletion is taking huge time more than (30 mins) compared to ovs 2.3 where it used to take 1-2 secs .
>>>>>
>>>>> Below are steps which I am following to reproduce the issue :
>>>>>
>>>>>
>>>>> 1. Add 200K Flows
>>>>>
>>>>> ####### Sample Script to add 200 K Flows ######
>>>>>
>>>>>
>>>>> echo "Adding 200K Flows"
>>>>> date
>>>>> for v in {1..20}
>>>>>
>>>>> do
>>>>>
>>>>> for i in {1..250}
>>>>> do
>>>>>
>>>>> for j in {1..40}
>>>>> do
>>>>>
>>>>> ovs-ofctl add-flow br-int
>>>>> hard_timeout=0,idle_timeout=0,priority=10,ip,cookie=0x5c38cfa3a668
>>>>> $
>>>>> v
>>>>> ,table=0,dl_dst=fa:16:3e:af:69:a3,dl_vlan=$v,nw_src=9.16.$i.$j/32,
>>>>> i
>>>>> n
>>>>> _port=1,actions=output:1
>>>>>
>>>>> done
>>>>> done
>>>>>
>>>>> done
>>>>> date
>>>>> echo "Flows are added"
>>>>>
>>>>> echo "Dump-Flows"
>>>>> ovs-ofctl dump-flows br-int |wc
>>>>> date
>>>>>
>>>>>
>>>>> ######## End Script ####
>>>>>
>>>>>
>>>>> 2. Try to delete 10 k Flows ()
>>>>>
>>>>> ovs-ofctl del-flows br-int dl_vlan=1
>>>>>
>>>>> 3. Above command gets executed successfully , but looks like deletion keep on happening in background .
>>>>> 4. It takes more than 30 mins to complete it , during this period if anyone executes any of below command it hangs without any output .
>>>>>
>>>>>
>>>>> ovs-ofctl dump-flows br-int
>>>>>
>>>>> or
>>>>>
>>>>> ovs-ofctl show br-int
>>>>>
>>>>>
>>>>> 5. This issue is not seen in ovs 2.3
>>>>>
>>>>>
>>>>> Is there any solution for this issue ?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks & Regards,
>>>>> Alok
>>>>
>>>>> _______________________________________________
>>>>> discuss mailing list
>>>>> discuss at openvswitch.org
>>>>> http://openvswitch.org/mailman/listinfo/discuss
>>>>
>>>
>>
>
More information about the discuss
mailing list