[ovs-discuss] High CPU Usage by ovs-vswitchd and resulting packet loss

Oliver Francke Oliver.Francke at filoo.de
Fri Jun 8 09:39:21 UTC 2012


Another good thing...

I have two test-VM's running on two different nodes... The VM on node 
with ovs 1.4.1 gives me throuput of about 5MB/s, the other one on the 
1.7.90 about 8MB/s, so that customers def. should see a better overall 
performance.
Will continue to monitor ;)

Oliver.

On 06/08/2012 10:01 AM, Oliver Francke wrote:
> Hi,
>
> change done this morning, but to typical times I still see a raise in 
> CPU-load, though the number of flows kept being low:
>
> 20120608-094530: lookups: hit:21369696007 missed:12901230041 
> lost:210760086 flows: 588 SHORT_FLOWS=329, TOP=mem: 6848 cpu: 47
> 20120608-094540: lookups: hit:21369710090 missed:12901242161 
> lost:210760086 flows: 515 SHORT_FLOWS=253, TOP=mem: 6848 cpu: 45
> 20120608-094550: lookups: hit:21369723892 missed:12901279759 
> lost:210760086 flows: 538 SHORT_FLOWS=299, TOP=mem: 6848 cpu: 98
> 20120608-094600: lookups: hit:21369738394 missed:12901351923 
> lost:210760086 flows: 573 SHORT_FLOWS=309, TOP=mem: 6848 cpu: 100
> 20120608-094610: lookups: hit:21369756196 missed:12901416693 
> lost:210760086 flows: 524 SHORT_FLOWS=284, TOP=mem: 6848 cpu: 96
> 20120608-094620: lookups: hit:21369770278 missed:12901482470 
> lost:210760086 flows: 609 SHORT_FLOWS=378, TOP=mem: 6848 cpu: 96
> 20120608-094630: lookups: hit:21369784860 missed:12901548691 
> lost:210760086 flows: 627 SHORT_FLOWS=346, TOP=mem: 6848 cpu: 98
> 20120608-094640: lookups: hit:21369798128 missed:12901611209 
> lost:210760086 flows: 558 SHORT_FLOWS=317, TOP=mem: 6848 cpu: 97
> 20120608-094650: lookups: hit:21369812453 missed:12901670167 
> lost:210760086 flows: 661 SHORT_FLOWS=387, TOP=mem: 6848 cpu: 98
> 20120608-094700: lookups: hit:21369827834 missed:12901730154 
> lost:210760086 flows: 796 SHORT_FLOWS=531, TOP=mem: 6848 cpu: 100
> .
> .
> .
>
> ( ovs-vsctl (Open vSwitch) 1.7.90)
>
> it's now at these figures for about 12 minutes, calming down as I'm 
> writing, without changing the no. of flows etc...
>
> Any idea? I have started with the file-log option, so I could get some 
> debug-logging next time this is happening, if it's useful.
>
> Thnx n regards,
>
> Oliver.
>
> On 06/07/2012 12:22 PM, Oliver Francke wrote:
>> Hi Justin,
>>
>> thnx for the explanations.
>>
>> Here is an excerpt of a scenario, when CPU-load goes up, though 
>> within our network the lost-figures don't normally change:
>>
>> 20120607-114420: lookups: hit:21149788628 missed:12736368714 
>> lost:210746961 flows: 3280 SHORT_FLOWS=2451, TOP=mem: 19m cpu: 39
>> 20120607-114425: lookups: hit:21149799408 missed:12736372737 
>> lost:210746961 flows: 4654 SHORT_FLOWS=3831, TOP=mem: 19m cpu: 45
>> 20120607-114430: lookups: hit:21149810014 missed:12736374681 
>> lost:210746961 flows: 2769 SHORT_FLOWS=1907, TOP=mem: 19m cpu: 18
>> 20120607-114435: lookups: hit:21149821758 missed:12736378269 
>> lost:210746961 flows: 4160 SHORT_FLOWS=3231, TOP=mem: 19m cpu: 49
>> 20120607-114440: lookups: hit:21149831635 missed:12736381096 
>> lost:210746961 flows: 3871 SHORT_FLOWS=2995, TOP=mem: 19m cpu: 18
>> 20120607-114445: lookups: hit:21149842697 missed:12736384730 
>> lost:210746961 flows: 4099 SHORT_FLOWS=3241, TOP=mem: 19m cpu: 35
>> 20120607-114450: lookups: hit:21149853788 missed:12736387554 
>> lost:210746961 flows: 3769 SHORT_FLOWS=2944, TOP=mem: 19m cpu: 12
>> 20120607-114455: lookups: hit:21149862246 missed:12736389740 
>> lost:210746961 flows: 2589 SHORT_FLOWS=1809, TOP=mem: 19m cpu: 18
>> 20120607-114500: lookups: hit:21149875807 missed:12736392636 
>> lost:210746961 flows: 3311 SHORT_FLOWS=2489, TOP=mem: 19m cpu: 29
>> 20120607-114505: lookups: hit:21149893590 missed:12736396998 
>> lost:210746961 flows: 5187 SHORT_FLOWS=4066, TOP=mem: 19m cpu: 53
>> 20120607-114510: lookups: hit:21149904797 missed:12736402095 
>> lost:210746961 flows: 6230 SHORT_FLOWS=5171, TOP=mem: 19m cpu: 37
>> 20120607-114515: lookups: hit:21149915723 missed:12736407377 
>> lost:210746961 flows: 6054 SHORT_FLOWS=4950, TOP=mem: 19m cpu: 45
>> 20120607-114520: lookups: hit:21149928325 missed:12736412748 
>> lost:210746961 flows: 6422 SHORT_FLOWS=5326, TOP=mem: 19m cpu: 31
>> 20120607-114525: lookups: hit:21149938705 missed:12736415973 
>> lost:210746961 flows: 4072 SHORT_FLOWS=2993, TOP=mem: 19m cpu: 43
>> 20120607-114530: lookups: hit:21149949606 missed:12736422759 
>> lost:210746961 flows: 7633 SHORT_FLOWS=6338, TOP=mem: 19m cpu: 94
>> 20120607-114535: lookups: hit:21149964017 missed:12736452506 
>> lost:210746961 flows: 11739 SHORT_FLOWS=10993, TOP=mem: 19m cpu: 96
>> 20120607-114540: lookups: hit:21149976648 missed:12736480881 
>> lost:210746961 flows: 15925 SHORT_FLOWS=15143, TOP=mem: 19m cpu: 98
>> 20120607-114545: lookups: hit:21149988896 missed:12736508350 
>> lost:210746961 flows: 13592 SHORT_FLOWS=12888, TOP=mem: 19m cpu: 98
>> 20120607-114550: lookups: hit:21150002168 missed:12736538481 
>> lost:210746961 flows: 15581 SHORT_FLOWS=14800, TOP=mem: 19m cpu: 100
>> 20120607-114555: lookups: hit:21150016018 missed:12736566873 
>> lost:210746961 flows: 11541 SHORT_FLOWS=10865, TOP=mem: 19m cpu: 98
>> 20120607-114600: lookups: hit:21150029226 missed:12736594616 
>> lost:210746961 flows: 14313 SHORT_FLOWS=15555, TOP=mem: 19m cpu: 100
>> 20120607-114605: lookups: hit:21150049470 missed:12736623781 
>> lost:210746961 flows: 14113 SHORT_FLOWS=13341, TOP=mem: 19m cpu: 100
>> 20120607-114610: lookups: hit:21150061782 missed:12736651311 
>> lost:210746961 flows: 13490 SHORT_FLOWS=12613, TOP=mem: 19m cpu: 99
>> 20120607-114615: lookups: hit:21150074821 missed:12736677656 
>> lost:210746961 flows: 12209 SHORT_FLOWS=11518, TOP=mem: 19m cpu: 97
>> 20120607-114620: lookups: hit:21150087942 missed:12736704949 
>> lost:210746961 flows: 11863 SHORT_FLOWS=11182, TOP=mem: 19m cpu: 84
>> 20120607-114625: lookups: hit:21150101016 missed:12736731540 
>> lost:210746961 flows: 11214 SHORT_FLOWS=10475, TOP=mem: 19m cpu: 97
>> 20120607-114630: lookups: hit:21150114324 missed:12736758289 
>> lost:210746961 flows: 10456 SHORT_FLOWS=10931, TOP=mem: 19m cpu: 98
>> 20120607-114635: lookups: hit:21150128318 missed:12736785776 
>> lost:210746961 flows: 11338 SHORT_FLOWS=10645, TOP=mem: 19m cpu: 98
>>
>> this now last for a couple of minutes, then falls down again. Nothing 
>> critical so far.
>>
>> Regards,
>>
>> Oliver.
>>
>> Am 07.06.2012 um 09:52 schrieb Justin Pettit:
>>
>>> On Jun 6, 2012, at 2:52 AM, Oliver Francke wrote:
>>>
>>>> @Justin: Any other recommendations?
>>> Are you also having many short-lived flows?  If you're in the range 
>>> I mentioned in my response to Kaushal (roughly 120,000 flow setups 
>>> per second), then the forthcoming 1.7.0 release may be enough for you.
>>>
>>>> If it's worth, I could try to start a new thread, but talking about 
>>>> high CPU-load, how do you all handle something like SYN-FLOOD 
>>>> attacks and stuff like that?
>>> Each datapath has 16 queues that connect the kernel to userspace.  
>>> We assign each port to one of those queues, which will help prevent 
>>> a port from starving the other ports.  Our use-case is to prevent 
>>> one VM from starving out the others.  In Kaushal's case, he using 
>>> OVS more like a bump-in-the-wire than a vswitch, meaning that he's 
>>> not concerned with a bad actor at the port level.
>>>
>>> We've got a couple of people traveling this week, but when they get 
>>> back, I plan to discuss how we may be able to provide finer-grained 
>>> control over flow setups for vswitch deployments, since our current 
>>> approach is rather coarse and can lead to queue collisions.  I've 
>>> also written Kaushal off-line to see if I can get more information 
>>> about his situation.
>>>
>>> --Justin
>>>
>>>
>>> _______________________________________________
>>> discuss mailing list
>>> discuss at openvswitch.org
>>> http://openvswitch.org/mailman/listinfo/discuss
>> _______________________________________________
>> discuss mailing list
>> discuss at openvswitch.org
>> http://openvswitch.org/mailman/listinfo/discuss
>
>


-- 

Oliver Francke

filoo GmbH
Moltkestraße 25a
33330 Gütersloh
HRB4355 AG Gütersloh

Geschäftsführer: S.Grewing | J.Rehpöhler | C.Kunz

Folgen Sie uns auf Twitter: http://twitter.com/filoogmbh




More information about the discuss mailing list