[ovs-discuss] 回复: small traffic but high CPU load for ovs-vswitchd

Henry 342619517 at qq.com
Sat Jan 30 00:05:28 UTC 2016


does any one can give me some hint?

---原始邮件---
发件人: "Henry"<342619517 at qq.com>
发送时间: 2016年1月15日 01:06:00
收件人: "discuss"<discuss at openvswitch.org>;
主题: [ovs-discuss] small traffic but high CPU load for ovs-vswitchd


Hi experts,


My company use OVS 2.3.1 in OpenStack deployment, there is something strange about the ovs-vswitchd process:


1. The traffic is small(as you can see in the following nmon result, the bond port eth2 & eth3 totoally has 200KBps), but the ovs-vswitchd CPU usage is high(>100%).
2. Not all ovs-vswitchd eat so much CPU, I have more than hundreds host act as compute node, all installed OVS 2.3.1,  and all have the similar traffic load, most ovs-vswitchd only use 10% CPU, but dozens use more than 100%.


What I want to know is: 
1. What's the potential cause can explain such issue?
2. What kind of debug I can do to figure out why small traffic with high CPU load?




Here is the nmon check for traffic load:


+nmon-14i------[H for help]---Hostname=node00234Refresh= 2secs ---09:27.03-------------------------------------------------------------------+
| Network I/O -----------------------------------------------------------------------------------------------------------------------------------|
|I/F Name Recv=KB/s Trans=KB/s packin packout insize outsize Peak->Recv Trans                                                                    |
|      lo     0.0     0.0       0.0      0.0     0.0    0.0       58.4    58.4                                                                   |
|    eth0     0.8    11.7       7.0     63.4   121.1  188.9    120158.2   616.4                                                                  |
|    eth1     1.3     0.0       9.5      0.0   137.5    0.0        1.3     0.0                                                                   |
|    eth2    99.1    33.7    1270.2    201.8    79.9  171.0      129.9    36.9                                                                   |
|    eth3   100.1    51.6    1294.7    227.3    79.2  232.6      132.3    56.2                                                                   |
|ovs-system     0.0     0.0       0.0      0.0     0.0    0.0        0.0     0.0                                                                 |
|  br-int     0.1     0.0       2.5      0.0    56.8    0.0        0.2     0.0                                                                   |
|br-bond1    49.9     0.0     490.5      0.0   104.2    0.0       73.2     0.0                                                                   |
|   bond0     2.1    11.7      16.5     63.4   130.5  188.9    120160.3   616.6                                                                  |
|ovirtmgmt     1.9    11.7      16.5     63.4   116.5  188.9    115238.5   616.5           




The CPU usage for the ovs-vswitchd:


top - 09:27:33 up 281 days, 13:16,  2 users,  load average: 2.74, 2.70, 2.72
Tasks:  14 total,   1 running,  13 sleeping,   0 stopped,   0 zombie
Cpu(s):  8.1%us, 14.2%sy,  0.0%ni, 76.7%id,  0.1%wa,  0.0%hi,  0.8%si,  0.0%st
Mem:  132109680k total, 85618248k used, 46491432k free,   714884k buffers
Swap:        0k total,        0k used,        0k free, 67934440k cached


  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                              
14833 root      10 -10 1004m 157m 7684 R 99.8  0.1  43033:59 ovs-vswitchd                                                                         
14523 root      10 -10 1004m 157m 7684 S  8.3  0.1 669:07.90 handler26                                                                            
14531 root      10 -10 1004m 157m 7684 S  1.3  0.1 104:36.08 revalidator34                                                                        
14532 root      10 -10 1004m 157m 7684 S  1.0  0.1  97:09.84 revalidator35                                                                        
14533 root      10 -10 1004m 157m 7684 S  0.7  0.1  96:36.16 revalidator36                                                                        
14534 root      10 -10 1004m 157m 7684 S  0.7  0.1  96:47.22 revalidator37                                                                        
14843 root      10 -10 1004m 157m 7684 S  0.0  0.1   0:00.02 urcu8                                                                                
14524 root      10 -10 1004m 157m 7684 S  0.0  0.1   0:00.00 handler28                                                                            
14525 root      10 -10 1004m 157m 7684 S  0.0  0.1   0:00.00 handler27                                                                            
14526 root      10 -10 1004m 157m 7684 S  0.0  0.1   0:00.00 handler29                                                                            
14527 root      10 -10 1004m 157m 7684 S  0.0  0.1   0:00.00 handler30                                                                            
14528 root      10 -10 1004m 157m 7684 S  0.0  0.1   0:00.00 handler31                                                                            
14529 root      10 -10 1004m 157m 7684 S  0.0  0.1   0:00.00 handler32                                                                            
14530 root      10 -10 1004m 157m 7684 S  0.0  0.1   0:00.00 handler33  


No cpu affinity setting:


[root at node00234 ~]# taskset -cp 14833
pid 14833's current affinity list: 0-23     


More info about dp, and I'm sure megaflow is enabled.


[root at node00234 ~]# ovs-dpctl show
system at ovs-system:
        lookups: hit:15858768478 missed:47078328606 lost:52900178
        flows: 1716
        port 0: ovs-system (internal)
        port 1: br-int (internal)
        port 2: br-bond1 (internal)
        port 3: eth3
        port 4: eth2
        port 5: qvo75ba34b4-fd
        port 6: qvo9f41063a-06
        port 7: qvod0752feb-d2
        port 8: qvo1eeb697a-f8
        port 9: qvof9c1a38d-f7
        port 10: qvocd8761b5-ed
        port 12: qvo19b7291b-2b
        port 13: qvo1b683926-41
        port 14: qvofde29ea5-be
        port 15: qvoae1324aa-b9
        port 16: qvo443b155c-13
        port 17: qvo9b7a893c-03
        port 18: qvoa5421574-e5
        port 20: qvod707714a-51
        port 21: qvo9244013b-e6
        port 22: qvo2ac86832-a7
        port 23: qvo4080273b-10
        port 24: qvob14d48f1-55
        port 25: br-ha (internal)


[root at node00234 ~]# ovs-appctl upcall/show
system at ovs-system:
        flows         : (current 59) (avg 716) (max 13893) (limit 200000)
        dump duration : 5ms


        34: (keys 0)
        35: (keys 0)
        36: (keys 0)
        37: (keys 0)


[root at node00234 ~]# ovs-appctl bond/show
---- bond1 ----
bond_mode: balance-slb
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 1392 ms
lacp_status: off
active slave mac: 90:1b:0e:24:70:92(eth3)


slave eth2: enabled
        may_enable: true
        hash 51: 72 kB load
        hash 54: 517 kB load


slave eth3: enabled
        active slave
        may_enable: true
        hash 10: 1 kB load
        hash 48: 1 kB load
        hash 96: 936 kB load
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20160130/3109844b/attachment-0002.html>


More information about the discuss mailing list