[ovs-discuss] OVS 2.4 and 2.4 Flow deletion issue
Maurya, Alok Kumar (HP Networking)
alok-kumar.maurya at hpe.com
Wed Apr 13 19:36:01 UTC 2016
Hi Jarno,
I have 4 CPU , 8 GB RAM VM .
I am running Ubuntu 14.04 on the VM .
With 200K flows , yes in my system also its 2-3 minutes , however when I have around 350K flows , it took 29-30 mins (sorry the data which I had given was with 350K flows not with 200K flows) .
When 350 Flows ae there in the system try to delete 17K flows in batch .
I can try to remove in_port=1 from match and see if that makes any difference , but flows will have same in_port as we expect packets with match to come from this port .
### script ####
echo "Adding 200K Flows"
date
for v in {1..20}
do
for i in {1..250}
do
for j in {1..70}
do
ovs-ofctl add-flow br-int hard_timeout=0,idle_timeout=0,priority=10,ip,cookie=0x5c38cfa3a668$v,table=0,dl_dst=fa:16:3e:af:69:a3,dl_vlan=$v,nw_src=9.16.$i.$j/32,in_port=1,actions=output:1
done
done
done
date
echo "Flows are added"
echo "Dump-Flows"
ovs-ofctl dump-flows br-int |wc
date
#### end script #####
Thanks & Regards,
Alok
-----Original Message-----
From: Jarno Rajahalme [mailto:jarno at ovn.org]
Sent: Thursday, April 14, 2016 12:44 AM
To: Maurya, Alok Kumar (HP Networking) <alok-kumar.maurya at hpe.com>
Cc: Ben Pfaff <blp at ovn.org>; discuss at openvswitch.org
Subject: Re: [ovs-discuss] OVS 2.4 and 2.4 Flow deletion issue
> On Apr 12, 2016, at 1:05 AM, Maurya, Alok Kumar (HP Networking) <alok-kumar.maurya at hpe.com> wrote:
>
> Thanks Jarno and Ben for looking into this issue .
>
> I had tried to backtrace this from 2.3 to 2.4 .
>
>
> I found that I had started seeing this issue after f2c214029e1ac0c9bc0f7c2013fe178a6b08dd7f commit , however in that
> ovs-ofctl del-flows br-int dl_vlan=1
>
I was able to reproduce the issue with the exact flow pattern you reported, thanks!
The issue is that each of the flows has exactly the same metadata portion in the match (in_port=1). I’m looking into how to alleviate this, but in the meanwhile you could try to remove fields from the matches that make no difference to the outcome of the match.
> took long time to execute ( more than 30 mins) .
>
On my 4-core VM the delete of the 5% of the flows took almost two minutes, so 30 minutes sounds like a long time. Could you tell a little bit more about your setup where you are executing this? OS, memory, # of cores, etc?
> In case of flow delete now , looks like ovs-vswitchd is sending "barrier-reply" even before flow deletion is completed .
>
The barrier-reply is returned as soon as the flows have been made invisible, but some of the clean-up work is postponed and can safely happen while following OpenFlow messages are processed. However, in this case the clean-up takes a mutex for a duration of removing the 10000 rules, and due to all the 200000 rules having the exactly same metadata match this now takes a long time.
Regards,
Jarno
> Thanks & Regards,
> Alok
>
> -----Original Message-----
> From: Jarno Rajahalme [mailto:jarno at ovn.org]
> Sent: Tuesday, April 12, 2016 7:02 AM
> To: Ben Pfaff <blp at ovn.org>
> Cc: Maurya, Alok Kumar (HP Networking) <alok-kumar.maurya at hpe.com>;
> discuss at openvswitch.org
> Subject: Re: [ovs-discuss] OVS 2.4 and 2.4 Flow deletion issue
>
> I tried with a bit simpler flows, like this:
>
> ovs-ofctl add-flow br-int
> hard_timeout=0,idle_timeout=0,priority=10,ip,cookie=0x$v,table=0,dl_vl
> an=$v,nw_src=9.16.$i.$j,actions=output:1
>
> And while adding them one-by-one in my VM took 12 minutes (my dev VM has been very slow recently), the runtime of the deletion was not noticeable, i.e., a following flow dump returned right away. Deleting all 200 flows took maybe 2-3 seconds.
>
> I’m trying again with the exact flow reported, but I’d be surprised if the additional match fields would make a difference.
>
> This on tip of master. I’ll have to try again with releases 2.4 and 2.5.
>
> Jarno
>
>> On Apr 11, 2016, at 5:15 PM, Jarno Rajahalme <jarno at ovn.org> wrote:
>>
>> I’m looking into this,
>>
>> Jarno
>>
>>> On Apr 11, 2016, at 3:41 PM, Ben Pfaff <blp at ovn.org> wrote:
>>>
>>> Jarno, it seems likely that this has something to do with the
>>> transactional classifier implementation. Do you think so? Do you
>>> have any idea why deletions would be so slow?
>>>
>>> Thanks,
>>>
>>> Ben.
>>>
>>> On Mon, Apr 11, 2016 at 09:51:45PM +0000, Maurya, Alok Kumar (HP Networking) wrote:
>>>> Hi All,
>>>>
>>>> I am seeing that in OVS 2.4 and ovs 2.5 Bulk flow deletion is taking huge time more than (30 mins) compared to ovs 2.3 where it used to take 1-2 secs .
>>>>
>>>> Below are steps which I am following to reproduce the issue :
>>>>
>>>>
>>>> 1. Add 200K Flows
>>>>
>>>> ####### Sample Script to add 200 K Flows ######
>>>>
>>>>
>>>> echo "Adding 200K Flows"
>>>> date
>>>> for v in {1..20}
>>>>
>>>> do
>>>>
>>>> for i in {1..250}
>>>> do
>>>>
>>>> for j in {1..40}
>>>> do
>>>>
>>>> ovs-ofctl add-flow br-int
>>>> hard_timeout=0,idle_timeout=0,priority=10,ip,cookie=0x5c38cfa3a668$
>>>> v
>>>> ,table=0,dl_dst=fa:16:3e:af:69:a3,dl_vlan=$v,nw_src=9.16.$i.$j/32,i
>>>> n
>>>> _port=1,actions=output:1
>>>>
>>>> done
>>>> done
>>>>
>>>> done
>>>> date
>>>> echo "Flows are added"
>>>>
>>>> echo "Dump-Flows"
>>>> ovs-ofctl dump-flows br-int |wc
>>>> date
>>>>
>>>>
>>>> ######## End Script ####
>>>>
>>>>
>>>> 2. Try to delete 10 k Flows ()
>>>>
>>>> ovs-ofctl del-flows br-int dl_vlan=1
>>>>
>>>> 3. Above command gets executed successfully , but looks like deletion keep on happening in background .
>>>> 4. It takes more than 30 mins to complete it , during this period if anyone executes any of below command it hangs without any output .
>>>>
>>>>
>>>> ovs-ofctl dump-flows br-int
>>>>
>>>> or
>>>>
>>>> ovs-ofctl show br-int
>>>>
>>>>
>>>> 5. This issue is not seen in ovs 2.3
>>>>
>>>>
>>>> Is there any solution for this issue ?
>>>>
>>>>
>>>>
>>>>
>>>> Thanks & Regards,
>>>> Alok
>>>
>>>> _______________________________________________
>>>> discuss mailing list
>>>> discuss at openvswitch.org
>>>> http://openvswitch.org/mailman/listinfo/discuss
>>>
>>
>
More information about the discuss
mailing list