[ovs-discuss] network bandwidth in Openstack when using OVS+VLAN
Li, Chen
chen.li at intel.com
Fri Aug 2 06:10:50 UTC 2013
I'm already using :
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchVirtualPortDriver
And I think the flow is goes with tapxx -> br-int (bridge) -> int-br-eth4 -> phy-br-eth4 -> br-eth4 (bridge) -> eth4
ovs-vsctl show :
f0fa9384-f6d0-4071-a5ac-53245eb2d68d
Bridge br-int
Port "tapc447a6c8-18"
tag: 2
Interface "tapc447a6c8-18"
Port "tap4f625dbc-21"
tag: 2
Interface "tap4f625dbc-21"
Port "tape9c77763-3f"
tag: 2
Interface "tape9c77763-3f"
Port "tap607a56d5-47"
tag: 2
Interface "tap607a56d5-47"
Port "tap72cebb96-c9"
tag: 2
Interface "tap72cebb96-c9"
Port "tap4270f5ae-37"
tag: 2
Interface "tap4270f5ae-37"
Port "tap9d74b661-42"
tag: 2
Interface "tap9d74b661-42"
Port "tap8781a0ba-2d"
tag: 2
Interface "tap8781a0ba-2d"
Port "int-br-eth4"
Interface "int-br-eth4"
Port "tapa5d4f8bd-bd"
tag: 2
Interface "tapa5d4f8bd-bd"
Port "tap4f893e9f-de"
tag: 2
Interface "tap4f893e9f-de"
Port br-int
Interface br-int
type: internal
Bridge "br-eth4"
Port "eth4"
Interface "eth4"
Port "br-eth4"
Interface "br-eth4"
type: internal
Port "phy-br-eth4"
Interface "phy-br-eth4"
ovs_version: "1.4.0+build0"
From: Aaron Rosen [mailto:arosen at nicira.com]
Sent: Friday, August 02, 2013 2:01 PM
To: Li, Chen
Cc: Luca Giraudo; discuss at openvswitch.org
Subject: Re: [ovs-discuss] network bandwidth in Openstack when using OVS+VLAN
Hi,
What's your configuration in nova.conf for: libvirt_vif_driver ? If this is set to: nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver I'd guess this is the culprit. The OVS plugin for openstack leverages linuxbridges as well in order to use iptables for it's security group implementation. The bridge configuration on the host would look like this:
br-int(ovs_bridge) --- qvo-(veth)-qvb ----qbr(linux_bridge)--tap(vm)
If you are not using security groups you could try setting: libvirt_vif_driver to nova.virt.libvirt.vif.LibvirtOpenVswitchVirtualPortDriver or nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
Aaron
On Thu, Aug 1, 2013 at 8:43 PM, Li, Chen <chen.li at intel.com<mailto:chen.li at intel.com>> wrote:
* interrupts: cat /proc/interrupts | grep <your physical interface>
I tested for 60 seconds, and get the flowing data by using end - start.
Already normalize to 1 second:
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 CPU24 CPU25 CPU26 CPU27 CPU28 CPU29 CPU30 CPU31
126 1441 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-0
127 0 1099 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-1
128 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-2
129 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-3
130 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-4
131 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-5
132 0 0 0 0 0 0 166 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-6
133 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-7
134 0 0 0 0 0 0 0 0 12989 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-8
135 0 0 0 0 0 0 0 0 0 532 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-9
136 0 0 0 0 0 0 0 0 0 0 455 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-10
137 0 0 0 0 0 0 0 0 0 0 0 36 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-11
138 0 0 0 0 0 0 0 0 0 0 0 0 5285 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-12
139 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-13
140 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-14
141 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-15
142 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1679 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-16
143 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1770 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-17
144 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-18
145 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-19
146 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-20
147 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 eth4-TxRx-21
148 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 612 0 0 0 0 0 0 0 0 0 eth4-TxRx-22
149 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 eth4-TxRx-23
150 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 27646 0 0 0 0 0 0 0 eth4-TxRx-24
151 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 eth4-TxRx-25
152 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 eth4-TxRx-26
153 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 eth4-TxRx-27
154 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 518 0 0 0 eth4-TxRx-28
155 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 eth4-TxRx-29
156 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 eth4-TxRx-30
157 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 eth4-TxRx-31
158 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth4
* processes that are consuming more CPU. Take a look at kvm, vhost, ovs-vswitchd. If you use top, press "1" and take a look at the idle time of CPUs to see what are the most overloaded cores.
top - 11:32:12 up 77 days, 2:16, 2 users, load average: 0.89, 1.02, 0.97
Tasks: 280 total, 1 running, 279 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.3%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.7%us, 1.0%sy, 0.0%ni, 98.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.0%us, 9.0%sy, 0.0%ni, 76.0%id, 0.0%wa, 0.0%hi, 15.0%si, 0.0%st
Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu8 : 0.0%us, 0.0%sy, 0.0%ni, 97.7%id, 2.3%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 : 0.7%us, 0.0%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 : 1.3%us, 0.0%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 : 0.0%us, 0.7%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu12 : 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu13 : 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu14 : 82.7%us, 2.7%sy, 0.0%ni, 14.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu15 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu16 : 5.9%us, 1.3%sy, 0.0%ni, 92.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu17 : 4.7%us, 1.3%sy, 0.0%ni, 94.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu18 : 1.7%us, 0.7%sy, 0.0%ni, 97.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu19 : 0.0%us, 27.1%sy, 0.0%ni, 27.1%id, 0.0%wa, 0.0%hi, 45.8%si, 0.0%st
Cpu20 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu21 : 6.0%us, 1.7%sy, 0.0%ni, 92.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu22 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu23 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu24 : 0.3%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu25 : 24.7%us, 5.4%sy, 0.0%ni, 69.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu26 : 8.3%us, 3.6%sy, 0.0%ni, 88.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu27 : 0.0%us, 0.7%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu28 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu29 : 10.0%us, 2.0%sy, 0.0%ni, 88.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu30 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu31 : 5.0%us, 1.3%sy, 0.0%ni, 93.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 131985576k total, 9768740k used, 122216836k free, 309352k buffers
Swap: 975868k total, 0k used, 975868k free, 4573908k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
34364 root 20 0 0 0 0 S 92 0.0 9:03.54 vhost-34350
34350 libvirt- 20 0 4851m 550m 7548 S 85 0.4 14:53.36 kvm
83 root 20 0 0 0 0 S 4 0.0 2:28.41 ksoftirqd/19
114986 quantum 20 0 101m 29m 4352 S 2 0.0 160:36.58 python
19 root 20 0 0 0 0 S 1 0.0 0:10.68 ksoftirqd/3
1621 root 10 -10 19772 2232 1340 S 1 0.0 115:56.88 ovsdb-server
150 root 25 5 0 0 0 S 0 0.0 126:52.72 ksmd
179 root 20 0 0 0 0 S 0 0.0 4:49.86 kworker/14:1
303 root 20 0 0 0 0 S 0 0.0 12:48.86 kworker/19:1
1656 root 10 -10 25364 4152 1252 S 0 0.0 145:04.35 ovs-vswitchd
38641 root 20 0 0 0 0 S 0 0.0 0:23.83 kworker/18:2
59338 libvirt- 20 0 4850m 545m 7548 S 0 0.4 7:00.51 kvm
81053 root 20 0 898m 15m 6964 S 0 0.0 163:00.38 libvirtd
125121 root 20 0 17472 1552 1048 R 0 0.0 0:00.22 top
126197 root 20 0 11436 1112 852 S 0 0.0 0:00.08 sadc
1 root 20 0 27796 5732 1356 S 0 0.0 8:44.68 init
2 root 20 0 0 0 0 S 0 0.0 0:00.11 kthreadd
3 root 20 0 0 0 0 S 0 0.0 1:26.52 ksoftirqd/0
6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
7 root RT 0 0 0 0 S 0 0.0 0:22.08 watchdog/0
* /etc/init.d/irqbalance status
service irqbalance status
irqbalance start/running, process 1665
* what CPU are you using and how many sockets/cores you have
Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
2 sockets, 8*2 physical core, 32 logical core (HT enabled)
* did you enable power savings mechanisms in the BIOS?
Yes. I have both p-sate and c-stat enabled.
While idle, CPU frequency is 1200 MHz.
From: Luca Giraudo [mailto:lgiraudo at nicira.com<mailto:lgiraudo at nicira.com>]
Sent: Friday, August 02, 2013 12:31 AM
To: Li, Chen
Cc: discuss at openvswitch.org<mailto:discuss at openvswitch.org>
Subject: Re: [ovs-discuss] network bandwidth in Openstack when using OVS+VLAN
Can you repeat the VM-VM test and monitor the RX hypervisor please?
I am interested in the following parameters:
* interrupts: cat /proc/interrupts | grep <your physical interface>
* processes that are consuming more CPU. Take a look at kvm, vhost, ovs-vswitchd. If you use top, press "1" and take a look at the idle time of CPUs to see what are the most overloaded cores.
* /etc/init.d/irqbalance status
* what CPU are you using and how many sockets/cores you have
* did you enable power savings mechanisms in the BIOS?
I suppose that the performance is limited by the receiving side so I would like to understand where is the problem (CPU limitations, NIC, kvm/vhost, OVS, etc.)
Luca
On Tue, Jul 30, 2013 at 8:37 PM, Li, Chen <chen.li at intel.com<mailto:chen.li at intel.com>> wrote:
Yes. For one pair test, rx side is the bottleneck currently.
I changed one side of the test to a physical machine.
Then I get:
One instance -> physical machine (instance send out packages) = > 5.71 Gbit/sec
physical machine -> One instance (physical machine send out packages) = > 1.79 Gbit/sec
The CPU% in client side:
Average CPU% is still low, CPU7 is the CPU core with the highest CPU%:
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
Average: all 1.47 0.00 1.85 0.03 0.00 1.77 0.00 2.61 92.26
Average: 7 0.17 0.00 0.92 0.00 0.00 0.00 0.00 28.79 70.13
The CPU% time line for CPU7:
CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
04:00:28 PM 7 1.00 0.00 3.00 0.00 0.00 0.00 0.00 90.00 6.00
04:00:29 PM 7 0.00 0.00 1.98 0.00 0.00 0.00 0.00 91.09 6.93
04:00:30 PM 7 0.00 0.00 3.06 0.00 0.00 0.00 0.00 91.84 6.12
04:00:31 PM 7 0.00 0.00 2.02 0.00 0.00 0.00 0.00 90.91 7.07
04:00:32 PM 7 0.00 0.00 3.00 0.00 0.00 0.00 0.00 90.00 7.00
04:00:33 PM 7 1.01 0.00 3.03 0.00 0.00 0.00 0.00 88.89 7.07
04:00:34 PM 7 0.00 0.00 2.97 0.00 0.00 0.00 0.00 91.09 6.93
04:00:35 PM 7 0.00 0.00 2.00 0.00 0.00 0.00 0.00 90.00 8.00
04:00:36 PM 7 0.00 0.00 3.00 0.00 0.00 0.00 0.00 90.00 7.00
04:00:37 PM 7 0.96 0.00 2.88 0.00 0.00 0.00 0.00 89.42 6.73
04:00:38 PM 7 0.00 0.00 4.00 0.00 0.00 0.00 0.00 89.00 8.00
04:00:39 PM 7 0.00 0.00 1.04 0.00 0.00 0.00 0.00 91.67 7.29
04:00:40 PM 7 0.99 0.00 1.98 0.00 0.00 0.00 0.00 88.12 8.91
04:00:41 PM 7 0.00 0.00 3.03 0.00 0.00 0.00 0.00 91.92 6.06
04:00:42 PM 7 0.00 0.00 2.94 0.00 0.00 0.00 0.00 90.20 6.86
04:00:43 PM 7 0.00 0.00 3.96 0.00 0.00 0.00 0.00 89.11 6.93
04:00:44 PM 7 0.00 0.00 5.05 0.00 0.00 0.00 0.00 88.89 6.06
04:00:45 PM 7 0.99 0.00 1.98 0.00 0.00 0.00 0.00 90.10 6.93
04:00:46 PM 7 0.00 0.00 1.96 0.00 0.00 0.00 0.00 92.16 5.88
04:00:47 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10.00 91.00
04:00:48 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:49 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:50 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:51 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:52 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:53 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:54 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:55 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:56 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:57 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:58 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:59 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:00 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:01 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:02 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:03 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:04 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:05 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:06 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:07 PM 7 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.01
04:01:08 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:09 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:10 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:11 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:12 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:13 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:14 PM 7 7.92 0.00 1.98 0.00 0.00 0.00 0.00 0.00 90.10
04:01:15 PM 7 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00
04:01:16 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:17 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:18 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:19 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:20 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:21 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:22 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:23 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:24 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:25 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 99.01
04:01:26 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:27 PM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
Another CPU core : CPU12:
CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
04:00:28 PM 12 0.00 0.00 38.54 0.00 0.00 60.42 0.00 0.00 1.04
04:00:29 PM 12 0.00 0.00 40.40 0.00 0.00 58.59 0.00 0.00 1.01
04:00:30 PM 12 0.00 0.00 51.49 0.00 0.00 46.53 0.00 0.00 1.98
04:00:31 PM 12 0.00 0.00 34.02 0.00 0.00 63.92 0.00 0.00 2.06
04:00:32 PM 12 0.00 0.00 36.00 0.00 0.00 49.00 0.00 0.00 15.00
04:00:33 PM 12 0.00 0.00 40.59 0.00 0.00 57.43 0.00 0.00 1.98
04:00:34 PM 12 0.00 0.00 46.46 0.00 0.00 52.53 0.00 0.00 1.01
04:00:35 PM 12 0.00 0.00 36.73 0.00 0.00 61.22 0.00 0.00 2.04
04:00:36 PM 12 0.00 0.00 14.85 0.00 0.00 19.80 0.00 0.00 65.35
04:00:37 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:38 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:39 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:40 PM 12 0.00 0.00 28.12 0.00 0.00 38.54 0.00 0.00 33.33
04:00:41 PM 12 0.00 0.00 35.71 0.00 0.00 62.24 0.00 0.00 2.04
04:00:42 PM 12 0.00 0.00 39.39 0.00 0.00 51.52 0.00 0.00 9.09
04:00:43 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:44 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:45 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:46 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:47 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:48 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:49 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:50 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:51 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:52 PM 12 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02
04:00:53 PM 12 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 99.00
04:00:54 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:55 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:56 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:57 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:00:58 PM 12 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00
04:00:59 PM 12 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00
04:01:00 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:01 PM 12 1.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.99
04:01:02 PM 12 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01
04:01:03 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:04 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:05 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:06 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:07 PM 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:01:08 PM 12 0.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 98.00
04:01:09 PM 12 2.02 0.00 1.01 0.00 0.00 0.00 0.00 0.00 96.97
04:01:10 PM 12 0.00 0.00 2.02 0.00 0.00 0.00 0.00 0.00 97.98
04:01:11 PM 12 1.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 98.00
04:01:12 PM 12 0.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 98.00
04:01:13 PM 12 0.00 0.00 1.01 0.00 0.00 0.00 0.00 0.00 98.99
04:01:14 PM 12 1.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 97.00
04:01:15 PM 12 0.00 0.00 2.02 0.00 0.00 0.00 0.00 0.00 97.98
04:01:16 PM 12 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 99.00
04:01:17 PM 12 0.00 0.00 1.01 0.00 0.00 0.00 0.00 0.00 98.99
04:01:18 PM 12 0.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 98.00
04:01:19 PM 12 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02
04:01:20 PM 12 0.00 0.00 1.01 0.00 0.00 0.00 0.00 0.00 98.99
04:01:21 PM 12 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 99.00
04:01:22 PM 12 0.00 0.00 1.01 0.00 0.00 0.00 0.00 0.00 98.99
04:01:23 PM 12 0.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 98.00
04:01:24 PM 12 1.01 0.00 2.02 0.00 0.00 0.00 0.00 0.00 96.97
04:01:25 PM 12 1.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 96.00
04:01:26 PM 12 0.00 0.00 2.94 0.00 0.00 0.00 0.00 13.73 83.33
04:01:27 PM 12 0.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 98.00
The service irqbalance is running, but it looks not working well ????
Thanks.
-chen
From: Luca Giraudo [mailto:lgiraudo at nicira.com<mailto:lgiraudo at nicira.com>]
Sent: Tuesday, July 30, 2013 8:34 PM
To: Li, Chen
Cc: discuss at openvswitch.org<mailto:discuss at openvswitch.org>
Subject: RE: [ovs-discuss] network bandwidth in Openstack when using OVS+VLAN
Can you monitor the CPU in the receiving node? RX is the difficult part and in general there will be some process taking 100% of a CPU core.
Luca
On Jul 30, 2013 1:47 AM, "Li, Chen" <chen.li at intel.com<mailto:chen.li at intel.com>> wrote:
For 6 Pairs CPU% in 1 second:
04:11:08 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
04:11:36 PM all 0.42 0.00 4.22 0.03 0.00 8.56 0.00 7.82 78.95
04:11:36 PM 0 0.00 0.00 7.69 0.00 0.00 13.19 0.00 13.19 65.93
04:11:36 PM 1 0.00 0.00 8.42 0.00 0.00 18.95 0.00 8.42 64.21
04:11:36 PM 2 1.02 0.00 6.12 0.00 0.00 13.27 0.00 17.35 62.24
04:11:36 PM 3 0.00 0.00 10.10 0.00 0.00 24.24 0.00 10.10 55.56
04:11:36 PM 4 0.00 0.00 10.42 0.00 0.00 30.21 0.00 4.17 55.21
04:11:36 PM 5 0.00 0.00 10.64 0.00 0.00 17.02 0.00 10.64 61.70
04:11:36 PM 6 0.00 0.00 9.68 0.00 0.00 18.28 0.00 16.13 56.99
04:11:36 PM 7 0.00 0.00 9.78 0.00 0.00 27.17 0.00 4.35 58.70
04:11:36 PM 8 1.08 0.00 10.75 0.00 0.00 20.43 0.00 10.75 56.99
04:11:36 PM 9 1.04 0.00 9.38 0.00 0.00 17.71 0.00 11.46 60.42
04:11:36 PM 10 0.99 0.00 7.92 0.00 0.00 9.90 0.00 13.86 67.33
04:11:36 PM 11 1.04 0.00 4.17 0.00 0.00 5.21 0.00 27.08 62.50
04:11:36 PM 12 0.00 0.00 3.09 0.00 0.00 10.31 0.00 14.43 72.16
04:11:36 PM 13 0.00 0.00 11.00 0.00 0.00 15.00 0.00 18.00 56.00
04:11:36 PM 14 0.00 0.00 6.45 0.00 0.00 17.20 0.00 15.05 61.29
04:11:36 PM 15 0.00 0.00 7.29 0.00 0.00 19.79 0.00 21.88 52.08
04:11:36 PM 16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:11:36 PM 17 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 99.00
04:11:36 PM 18 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:11:36 PM 19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:11:36 PM 20 6.00 0.00 0.00 0.00 0.00 0.00 0.00 16.00 78.00
04:11:36 PM 21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:11:36 PM 22 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 99.00
04:11:36 PM 23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:11:36 PM 24 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00
04:11:36 PM 25 0.00 0.00 1.00 0.00 0.00 2.00 0.00 0.00 97.00
04:11:36 PM 26 4.95 0.00 0.99 0.00 0.00 0.00 0.00 0.00 94.06
04:11:36 PM 27 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:11:36 PM 28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:11:36 PM 29 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
04:11:36 PM 30 0.00 0.00 0.99 0.00 0.00 0.00 0.00 10.89 88.12
04:11:36 PM 31 0.00 0.00 0.99 0.00 0.00 0.00 0.00 7.92 92.08
From: discuss-bounces at openvswitch.org<mailto:discuss-bounces at openvswitch.org> [mailto:discuss-bounces at openvswitch.org<mailto:discuss-bounces at openvswitch.org>] On Behalf Of Li, Chen
Sent: Tuesday, July 30, 2013 4:42 PM
To: Luca Giraudo
Cc: discuss at openvswitch.org<mailto:discuss at openvswitch.org>
Subject: Re: [ovs-discuss] network bandwidth in Openstack when using OVS+VLAN
* are you using KVM? If yes, can you verify that your VMs are using vhost_net, please? In the KVM process related to the VM there should be a vhost=on parameter. If not, modprobe vhost_net.
Yes.
And after enable vhost_net, I can get higher bandwidth now:
One pair, increase from 1.18 Gbits/sec => 2.38 Gbits/sec
Six pairs, increase from 4.25 Gbits/sec => 6.98 Gbits/sec
* I saw different throughput with different OS in the VM. Did you try different VMs?
No, but I guess different OS in VM might cause performance difference like 5Gbit/sec vs. 6Gbit/sec, but not 2Gbit/s vs. 6Gbit/sec
* what bandwidth do you get among two VMs in the same compute node?
14.7 Gbits/sec
* can you monitor the CPU usage in the compute node and look out for big CPU consumer, please?
I'm running on a SNB-EP server, so I have 32 cores (HT enabled) for one compute node.
Only observed the compute node that run iperf client (send packages out)
1. The average CPU%
Total average CPU% = 6.66 %
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
Average: all 1.66 0.00 1.38 0.05 0.00 1.83 0.00 1.74 93.34
Per-core :
The highest CPU% = 21.65%
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
Average: 0 0.30 0.00 5.29 0.02 0.00 11.29 0.00 4.75 78.35
2. CPU% time line : only check the CPU core with the highest CPU%
02:23:29 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
02:23:29 PM 0 0.00 0.00 2.06 0.00 0.00 4.12 0.00 0.00 93.81
02:23:30 PM 0 0.00 0.00 25.77 0.00 0.00 67.01 0.00 0.00 7.22
02:23:31 PM 0 0.00 0.00 26.04 0.00 0.00 59.38 0.00 0.00 14.58
02:23:32 PM 0 0.00 0.00 23.00 0.00 0.00 52.00 0.00 0.00 25.00
02:23:33 PM 0 0.00 0.00 28.28 0.00 0.00 52.53 0.00 0.00 19.19
02:23:34 PM 0 0.00 0.00 10.89 0.00 0.00 19.80 0.00 0.00 69.31
02:23:35 PM 0 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00
02:23:36 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
02:23:37 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 99.00
02:23:38 PM 0 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.01
02:23:39 PM 0 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 99.00
02:23:40 PM 0 1.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.99
02:23:41 PM 0 2.06 0.00 15.46 0.00 0.00 36.08 0.00 0.00 46.39
02:23:42 PM 0 1.98 0.00 12.87 0.00 0.00 34.65 0.00 0.00 50.50
02:23:43 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 15.15 84.85
02:23:44 PM 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02
02:23:45 PM 0 0.00 0.00 2.00 0.00 0.00 0.00 0.00 19.00 80.00
02:23:46 PM 0 0.99 0.00 0.00 0.00 0.00 0.00 0.00 23.76 75.25
02:23:47 PM 0 0.00 0.00 24.49 0.00 0.00 40.82 0.00 0.00 34.69
02:23:48 PM 0 0.00 0.00 23.96 0.00 0.00 60.42 0.00 0.00 15.62
02:23:49 PM 0 0.00 0.00 23.96 0.00 0.00 60.42 0.00 0.00 15.62
02:23:50 PM 0 0.00 0.00 6.12 0.00 0.00 18.37 0.00 0.00 75.51
02:23:51 PM 0 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00
From: Luca Giraudo [mailto:lgiraudo at nicira.com]
Sent: Tuesday, July 30, 2013 1:37 PM
To: Li, Chen
Cc: discuss at openvswitch.org<mailto:discuss at openvswitch.org>; Gurucharan Shetty
Subject: Re: [ovs-discuss] network bandwidth in Openstack when using OVS+VLAN
Few more things to check:
* are you using KVM? If yes, can you verify that your VMs are using vhost_net, please? In the KVM process related to the VM there should be a vhost=on parameter. If not, modprobe vhost_net.
* I saw different throughput with different OS in the VM. Did you try different VMs?
* what bandwidth do you get among two VMs in the same compute node?
* can you monitor the CPU usage in the compute node and look out for big CPU consumer, please?
Thanks,
Luca
On Jul 29, 2013 6:26 PM, "Li, Chen" <chen.li at intel.com<mailto:chen.li at intel.com>> wrote:
* Is the VM ethernet driver a para-virtual driver? Para-virtual drivers give a good performance boost.
I have used openstack default parameters, it is virtio, I think virtio should have a good performance:
<interface type='bridge'>
<mac address='fa:16:3e:ca:4a:86'/>
<source bridge='br-int'/>
<virtualport type='openvswitch'>
<parameters interfaceid='3213dbec-f2ea-462f-818b-e07b76a1752c'/>
</virtualport>
<target dev='tap3213dbec-f2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
* Is TSO ON in the VM and the Hypervisor?
The VM:
Features for eth0:
rx-checksumming: off [fixed]
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: on
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
The hypervisor:
Offload parameters for eth4:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
* What throughput do you get while using Linux bridge instead of OVS?
Currently, I don't have a linux bridge environment.
But, I remember in virtio test, while I create bridge by hand and assigned it to an instance, I can always get near hardware limitation bandwidth if I have enough threads.
* Are you using tunnels? If you are using a tunnel like GRE, you will see a throughput drop.
No, I'm working under Quantum+OVS+VLAN.
Thanks.
-chen
From: Gurucharan Shetty [mailto:shettyg at nicira.com<mailto:shettyg at nicira.com>]
Sent: Tuesday, July 30, 2013 12:06 AM
To: Li, Chen
Cc: discuss at openvswitch.org<mailto:discuss at openvswitch.org>
Subject: Re: [ovs-discuss] network bandwidth in Openstack when using OVS+VLAN
There could be multiple reasons for the low throughput. I would probably look at the following.
* Is the VM ethernet driver a para-virtual driver? Para-virtual drivers give a good performance boost.
* Is TSO ON in the VM and the Hypervisor?
* What throughput do you get while using Linux bridge instead of OVS?
* Are you using tunnels? If you are using a tunnel like GRE, you will see a throughput drop.
On Mon, Jul 29, 2013 at 1:48 AM, Li, Chen <chen.li at intel.com<mailto:chen.li at intel.com>> wrote:
Hi list,
I'm a new user to OVS.
I installed OpenStack Grizzly, and using Quantum + OVS + VLAN for network.
I have two compute nodes with 10 Gb NICs, and the bandwidth between them is about 8.49 Gbits/sec (tested by iperf).
I started one instance at each compute node:
instance-a => compute1
instance-b=> compute2
The bandwidth between this two virtual machine is only 1.18 Gbits/sec.
Then I start 6 instances at each compute node:
( instance-a => compute1 ) ----- iperf------ > (instance-b=> compute2)
( instance-c => compute1 ) ----- iperf------ > (instance-d=> compute2)
( instance-e => compute1 ) ----- iperf------ > (instance-f=> compute2)
( instance-g => compute1 ) ----- iperf------ > (instance-h=> compute2)
( instance-i => compute1 ) ----- iperf------ > (instance-j=> compute2)
( instance-k => compute1 ) ----- iperf------ > (instance-l=> compute2)
The total bandwidth is only 4.25 Gbits/sec.
Anyone know why the performance is this low ?
Thanks.
-chen
_______________________________________________
discuss mailing list
discuss at openvswitch.org<mailto:discuss at openvswitch.org>
http://openvswitch.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list
discuss at openvswitch.org<mailto:discuss at openvswitch.org>
http://openvswitch.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list
discuss at openvswitch.org<mailto:discuss at openvswitch.org>
http://openvswitch.org/mailman/listinfo/discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20130802/5c6e7d16/attachment.html>
More information about the discuss
mailing list