[ovs-discuss] network bandwidth in Openstack when using OVS+VLAN
Aaron Rosen
arosen at nicira.com
Fri Aug 2 06:00:48 UTC 2013
Hi,
What's your configuration in nova.conf for: libvirt_vif_driver ? If this is
set to: nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver I'd guess this
is the culprit. The OVS plugin for openstack leverages linuxbridges as well
in order to use iptables for it's security group implementation. The bridge
configuration on the host would look like this:
br-int(ovs_bridge) --- qvo-(veth)-qvb ----qbr(linux_bridge)--tap(vm)
If you are not using security groups you could try setting:
libvirt_vif_driver to
nova.virt.libvirt.vif.LibvirtOpenVswitchVirtualPortDriver or
nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
Aaron
On Thu, Aug 1, 2013 at 8:43 PM, Li, Chen <chen.li at intel.com> wrote:
> ** **
>
> * interrupts: cat /proc/interrupts | grep <your physical interface>****
>
> ** **
>
> I tested for 60 seconds, and get the flowing data by using end – start.**
> **
>
> Already normalize to 1 second:****
>
> ** **
>
> CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
> CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12
> CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21
> CPU22 CPU23 CPU24 CPU25 CPU26 CPU27 CPU28
> CPU29 CPU30 CPU31 ****
>
> 126 1441 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-0****
>
> 127 0 1099 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-1****
>
> 128 0 0 1 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-2****
>
> 129 0 0 0 1
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-3****
>
> 130 0 0 0 0
> 1 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-4****
>
> 131 0 0 0 0
> 0 1 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-5****
>
> 132 0 0 0 0
> 0 0 166 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-6****
>
> 133 0 0 0 0
> 0 0 0 1 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-7****
>
> 134 0 0 0 0
> 0 0 0 0 *12989*
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-8****
>
> 135 0 0 0 0
> 0 0 0 0 0
> 532 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 eth4-TxRx-9***
> *
>
> 136 0 0 0 0
> 0 0 0 0 0
> 0 455 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 eth4-TxRx-10**
> **
>
> 137 0 0 0 0
> 0 0 0 0 0
> 0 0 36 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 eth4-TxRx-11**
> **
>
> 138 0 0 0 0
> 0 0 0 0 0
> 0 0 0 5285
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-12****
>
> 139 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 4 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-13****
>
> 140 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 1 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-14****
>
> 141 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 1 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-15****
>
> 142 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 1679 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-16****
>
> 143 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 1770
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-17****
>
> 144 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 1 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-18****
>
> 145 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 1 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-19****
>
> 146 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 1 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-20****
>
> 147 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 1 0
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-21****
>
> 148 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 612
> 0 0 0 0
> 0 0 0 0 0
> eth4-TxRx-22****
>
> 149 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 1 0 0 0
> 0 0 0 0 0
> eth4-TxRx-23****
>
> 150 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 *27646* 0 0 0
> 0 0 0 0 eth4-TxRx-24**
> **
>
> 151 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 5 0
> 0 0 0 0 0
> eth4-TxRx-25****
>
> 152 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 1
> 0 0 0 0 0
> eth4-TxRx-26****
>
> 153 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 1 0 0 0 0
> eth4-TxRx-27****
>
> 154 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 518 0 0 0
> eth4-TxRx-28****
>
> 155 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 1 0 0
> eth4-TxRx-29****
>
> 156 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 1 0
> eth4-TxRx-30****
>
> 157 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 1
> eth4-TxRx-31****
>
> 158 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0 0
> 0 0 0 0
> 0 0 0 0 0
> eth4****
>
> ** **
>
> * processes that are consuming more CPU. Take a look at kvm, vhost,
> ovs-vswitchd. If you use top, press "1" and take a look at the idle time of
> CPUs to see what are the most overloaded cores.****
>
> ** **
>
> top - 11:32:12 up 77 days, 2:16, 2 users, load average: 0.89, 1.02, 0.97
> ****
>
> Tasks: 280 total, 1 running, 279 sleeping, 0 stopped, 0 zombie****
>
> Cpu0 : 0.3%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu2 : 0.7%us, 1.0%sy, 0.0%ni, 98.3%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu3 : 0.0%us, 9.0%sy, 0.0%ni, 76.0%id, 0.0%wa, 0.0%hi, 15.0%si,
> 0.0%st****
>
> Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu7 : 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu8 : 0.0%us, 0.0%sy, 0.0%ni, 97.7%id, 2.3%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu9 : 0.7%us, 0.0%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu10 : 1.3%us, 0.0%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu11 : 0.0%us, 0.7%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu12 : 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu13 : 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu14 : 82.7%us, 2.7%sy, 0.0%ni, 14.7%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu15 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu16 : 5.9%us, 1.3%sy, 0.0%ni, 92.7%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu17 : 4.7%us, 1.3%sy, 0.0%ni, 94.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu18 : 1.7%us, 0.7%sy, 0.0%ni, 97.7%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu19 : 0.0%us, 27.1%sy, 0.0%ni, 27.1%id, 0.0%wa, 0.0%hi, 45.8%si,
> 0.0%st****
>
> Cpu20 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu21 : 6.0%us, 1.7%sy, 0.0%ni, 92.4%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu22 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu23 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu24 : 0.3%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu25 : 24.7%us, 5.4%sy, 0.0%ni, 69.9%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu26 : 8.3%us, 3.6%sy, 0.0%ni, 88.1%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu27 : 0.0%us, 0.7%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu28 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu29 : 10.0%us, 2.0%sy, 0.0%ni, 88.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu30 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Cpu31 : 5.0%us, 1.3%sy, 0.0%ni, 93.7%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st****
>
> Mem: 131985576k total, 9768740k used, 122216836k free, 309352k buffers
> ****
>
> Swap: 975868k total, 0k used, 975868k free, 4573908k cached***
> *
>
> ** **
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND****
>
> 34364 root 20 0 0 0 0 S 92 0.0 9:03.54 vhost-34350**
> **
>
> 34350 libvirt- 20 0 4851m 550m 7548 S 85 0.4 14:53.36 kvm****
>
> 83 root 20 0 0 0 0 S 4 0.0 2:28.41 ksoftirqd/19
> ****
>
> 114986 quantum 20 0 101m 29m 4352 S 2 0.0 160:36.58 python****
>
> 19 root 20 0 0 0 0 S 1 0.0 0:10.68 ksoftirqd/3*
> ***
>
> 1621 root 10 -10 19772 2232 1340 S 1 0.0 115:56.88 ovsdb-server
> ****
>
> 150 root 25 5 0 0 0 S 0 0.0 126:52.72 ksmd****
>
> 179 root 20 0 0 0 0 S 0 0.0 4:49.86 kworker/14:1
> ****
>
> 303 root 20 0 0 0 0 S 0 0.0 12:48.86 kworker/19:1
> ****
>
> 1656 root 10 -10 25364 4152 1252 S 0 0.0 145:04.35 ovs-vswitchd
> ****
>
> 38641 root 20 0 0 0 0 S 0 0.0 0:23.83 kworker/18:2*
> ***
>
> 59338 libvirt- 20 0 4850m 545m 7548 S 0 0.4 7:00.51 kvm****
>
> 81053 root 20 0 898m 15m 6964 S 0 0.0 163:00.38 libvirtd****
>
> 125121 root 20 0 17472 1552 1048 R 0 0.0 0:00.22 top****
>
> 126197 root 20 0 11436 1112 852 S 0 0.0 0:00.08 sadc****
>
> 1 root 20 0 27796 5732 1356 S 0 0.0 8:44.68 init****
>
> 2 root 20 0 0 0 0 S 0 0.0 0:00.11 kthreadd****
>
> 3 root 20 0 0 0 0 S 0 0.0 1:26.52 ksoftirqd/0*
> ***
>
> 6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0*
> ***
>
> 7 root RT 0 0 0 0 S 0 0.0 0:22.08 watchdog/0**
> **
>
> ** **
>
> ** **
>
> * /etc/init.d/irqbalance status****
>
> service irqbalance status****
>
> irqbalance start/running, process 1665****
>
> ** **
>
> * what CPU are you using and how many sockets/cores you have****
>
> Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz****
>
> 2 sockets, 8*2 physical core, 32 logical core (HT enabled)****
>
> ** **
>
> * did you enable power savings mechanisms in the BIOS?****
>
> Yes. I have both p-sate and c-stat enabled.****
>
> While idle, CPU frequency is 1200 MHz.****
>
> ** **
>
> ** **
>
> *From:* Luca Giraudo [mailto:lgiraudo at nicira.com]
> *Sent:* Friday, August 02, 2013 12:31 AM
> *To:* Li, Chen
> *Cc:* discuss at openvswitch.org
> *Subject:* Re: [ovs-discuss] network bandwidth in Openstack when using
> OVS+VLAN****
>
> ** **
>
> Can you repeat the VM-VM test and monitor the RX hypervisor please?****
>
> ** **
>
> I am interested in the following parameters:****
>
> ** **
>
> * interrupts: cat /proc/interrupts | grep <your physical interface>****
>
> * processes that are consuming more CPU. Take a look at kvm, vhost,
> ovs-vswitchd. If you use top, press "1" and take a look at the idle time of
> CPUs to see what are the most overloaded cores.****
>
> * /etc/init.d/irqbalance status****
>
> * what CPU are you using and how many sockets/cores you have****
>
> * did you enable power savings mechanisms in the BIOS?****
>
> ** **
>
> I suppose that the performance is limited by the receiving side so I would
> like to understand where is the problem (CPU limitations, NIC, kvm/vhost,
> OVS, etc.)****
>
> ** **
>
> Luca****
>
> ** **
>
> On Tue, Jul 30, 2013 at 8:37 PM, Li, Chen <chen.li at intel.com> wrote:****
>
> Yes. For one pair test, rx side is the bottleneck currently.****
>
> I changed one side of the test to a physical machine.****
>
> Then I get:****
>
> One instance -> physical machine (instance send out packages) = > *
> 5.71* Gbit/sec****
>
> physical machine -> One instance (physical machine send out
> packages) = > *1.79* Gbit/sec****
>
> ****
>
> The CPU% in client side:****
>
> Average CPU% is still low, CPU7 is the CPU core with the highest CPU%:
> ****
>
> Average: CPU %usr
> %nice %sys %iowait %irq
> %soft %steal %guest %idle****
>
> Average: all 1.47
> 0.00 1.85 0.03
> 0.00 1.77 0.00
> 2.61 92.26****
>
> Average: 7 0.17
> 0.00 0.92 0.00
> 0.00 0.00 0.00 28.79
> 70.13****
>
> ****
>
> The CPU% time line for CPU7:****
>
> CPU %usr
> %nice %sys %iowait %irq
> %soft %steal %guest %idle****
>
> *04:00:28 PM 7 1.00
> 0.00 3.00 0.00
> 0.00 0.00 0.00 90.00
> 6.00*****
>
> *04:00:29 PM 7 0.00
> 0.00 1.98 0.00
> 0.00 0.00 0.00
> 91.09 6.93*****
>
> *04:00:30 PM 7 0.00
> 0.00 3.06 0.00
> 0.00 0.00 0.00
> 91.84 6.12*****
>
> *04:00:31 PM 7 0.00
> 0.00 2.02 0.00
> 0.00 0.00 0.00
> 90.91 7.07*****
>
> *04:00:32 PM 7 0.00
> 0.00 3.00 0.00
> 0.00 0.00 0.00
> 90.00 7.00*****
>
> *04:00:33 PM 7 1.01
> 0.00 3.03 0.00
> 0.00 0.00 0.00
> 88.89 7.07*****
>
> *04:00:34 PM 7 0.00
> 0.00 2.97 0.00
> 0.00 0.00 0.00
> 91.09 6.93*****
>
> *04:00:35 PM 7 0.00
> 0.00 2.00 0.00
> 0.00 0.00 0.00
> 90.00 8.00*****
>
> *04:00:36 PM 7 0.00
> 0.00 3.00 0.00
> 0.00 0.00 0.00
> 90.00 7.00*****
>
> *04:00:37 PM 7 0.96
> 0.00 2.88 0.00
> 0.00 0.00 0.00
> 89.42 6.73*****
>
> *04:00:38 PM 7 0.00
> 0.00 4.00 0.00
> 0.00 0.00 0.00
> 89.00 8.00*****
>
> *04:00:39 PM 7 0.00
> 0.00 1.04 0.00 0.00
> 0.00 0.00 91.67 7.29*****
>
> *04:00:40 PM 7 0.99
> 0.00 1.98 0.00
> 0.00 0.00 0.00
> 88.12 8.91*****
>
> *04:00:41 PM 7 0.00
> 0.00 3.03 0.00
> 0.00 0.00 0.00
> 91.92 6.06*****
>
> *04:00:42 PM 7 0.00
> 0.00 2.94 0.00
> 0.00 0.00 0.00
> 90.20 6.86*****
>
> *04:00:43 PM 7 0.00 0.00
> 3.96 0.00 0.00
> 0.00 0.00 89.11 6.93*****
>
> *04:00:44 PM 7 0.00
> 0.00 5.05 0.00
> 0.00 0.00 0.00
> 88.89 6.06*****
>
> *04:00:45 PM 7 0.99
> 0.00 1.98 0.00
> 0.00 0.00 0.00
> 90.10 6.93*****
>
> *04:00:46 PM 7 0.00
> 0.00 1.96 0.00
> 0.00 0.00 0.00
> 92.16 5.88*****
>
> 04:00:47 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 10.00
> 91.00****
>
> 04:00:48 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:49 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:50 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:51 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:52 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:53 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:54 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:55 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:56 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:57 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:58 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:59 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:00 PM 7 0.00
> 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00****
>
> 04:01:01 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:02 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:03 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:04 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:05 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:06 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:07 PM 7 0.99
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 99.01****
>
> 04:01:08 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:09 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:10 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:11 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:12 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:13 PM 7 0.00
> 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00****
>
> 04:01:14 PM 7 7.92
> 0.00 1.98 0.00
> 0.00 0.00 0.00 0.00
> 90.10****
>
> 04:01:15 PM 7 1.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 99.00****
>
> 04:01:16 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:17 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:18 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:19 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:20 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:21 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:22 PM 7 0.00 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00****
>
> 04:01:23 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:24 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:25 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.99
> 99.01****
>
> 04:01:26 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:27 PM 7 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> ****
>
> Another CPU core : CPU12:****
>
> CPU %usr
> %nice %sys %iowait %irq
> %soft %steal %guest %idle****
>
> *04:00:28 PM 12 0.00
> 0.00 38.54 0.00 0.00
> 60.42 0.00 0.00 1.04*****
>
> *04:00:29 PM 12 0.00
> 0.00 40.40 0.00 0.00
> 58.59 0.00 0.00 1.01*****
>
> *04:00:30 PM 12 0.00
> 0.00 51.49 0.00 0.00
> 46.53 0.00 0.00 1.98*****
>
> *04:00:31 PM 12 0.00
> 0.00 34.02 0.00 0.00
> 63.92 0.00 0.00 2.06*****
>
> *04:00:32 PM 12 0.00
> 0.00 36.00 0.00 0.00
> 49.00 0.00 0.00 15.00*****
>
> *04:00:33 PM 12 0.00
> 0.00 40.59 0.00
> 0.00 57.43 0.00
> 0.00 1.98*****
>
> *04:00:34 PM 12 0.00
> 0.00 46.46 0.00 0.00
> 52.53 0.00 0.00 1.01*****
>
> *04:00:35 PM 12 0.00
> 0.00 36.73 0.00 0.00
> 61.22 0.00 0.00 2.04*****
>
> *04:00:36 PM 12 0.00
> 0.00 14.85 0.00 0.00
> 19.80 0.00 0.00 65.35*****
>
> *04:00:37 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00*****
>
> *04:00:38 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00*****
>
> *04:00:39 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00*****
>
> *04:00:40 PM 12 0.00
> 0.00 28.12 0.00 0.00
> 38.54 0.00 0.00 33.33*****
>
> *04:00:41 PM 12 0.00
> 0.00 35.71 0.00 0.00
> 62.24 0.00 0.00 2.04*****
>
> *04:00:42 PM 12 0.00
> 0.00 39.39 0.00 0.00
> 51.52 0.00 0.00 9.09*****
>
> 04:00:43 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:44 PM 12 0.00
> 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00****
>
> 04:00:45 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:46 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:47 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:48 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:49 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:50 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:51 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:52 PM 12 0.99
> 0.00 0.99 0.00
> 0.00 0.00 0.00 0.00
> 98.02****
>
> 04:00:53 PM 12 0.00
> 0.00 1.00 0.00
> 0.00 0.00 0.00 0.00
> 99.00****
>
> 04:00:54 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:55 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:56 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:57 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:00:58 PM 12 1.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 99.00****
>
> 04:00:59 PM 12 1.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 99.00****
>
> 04:01:00 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:01 PM 12 1.01
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 98.99****
>
> 04:01:02 PM 12 0.00
> 0.00 0.99 0.00
> 0.00 0.00 0.00 0.00
> 99.01****
>
> 04:01:03 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:04 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:05 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:06 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:07 PM 12 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:01:08 PM 12 0.00
> 0.00 2.00 0.00
> 0.00 0.00 0.00 0.00
> 98.00****
>
> 04:01:09 PM 12 2.02
> 0.00 1.01 0.00
> 0.00 0.00 0.00
> 0.00 96.97****
>
> 04:01:10 PM 12 0.00
> 0.00 2.02 0.00
> 0.00 0.00 0.00 0.00
> 97.98****
>
> 04:01:11 PM 12 1.00
> 0.00 1.00 0.00
> 0.00 0.00 0.00 0.00
> 98.00****
>
> 04:01:12 PM 12 0.00
> 0.00 2.00 0.00
> 0.00 0.00 0.00 0.00
> 98.00****
>
> 04:01:13 PM 12 0.00
> 0.00 1.01 0.00
> 0.00 0.00 0.00 0.00
> 98.99****
>
> 04:01:14 PM 12 1.00
> 0.00 2.00 0.00
> 0.00 0.00 0.00 0.00
> 97.00****
>
> 04:01:15 PM 12 0.00
> 0.00 2.02 0.00
> 0.00 0.00 0.00 0.00
> 97.98****
>
> 04:01:16 PM 12 0.00
> 0.00 1.00 0.00
> 0.00 0.00 0.00 0.00
> 99.00****
>
> 04:01:17 PM 12 0.00
> 0.00 1.01 0.00
> 0.00 0.00 0.00 0.00
> 98.99****
>
> 04:01:18 PM 12 0.00
> 0.00 2.00 0.00
> 0.00 0.00 0.00 0.00
> 98.00****
>
> 04:01:19 PM 12 0.99
> 0.00 0.99 0.00
> 0.00 0.00 0.00 0.00
> 98.02****
>
> 04:01:20 PM 12 0.00
> 0.00 1.01 0.00
> 0.00 0.00 0.00 0.00
> 98.99****
>
> 04:01:21 PM 12 0.00
> 0.00 1.00 0.00
> 0.00 0.00 0.00 0.00
> 99.00****
>
> 04:01:22 PM 12 0.00
> 0.00 1.01 0.00
> 0.00 0.00 0.00 0.00
> 98.99****
>
> 04:01:23 PM 12 0.00
> 0.00 2.00 0.00
> 0.00 0.00 0.00 0.00
> 98.00****
>
> 04:01:24 PM 12 1.01
> 0.00 2.02 0.00
> 0.00 0.00 0.00 0.00
> 96.97****
>
> 04:01:25 PM 12 1.00
> 0.00 3.00 0.00
> 0.00 0.00 0.00 0.00
> 96.00****
>
> 04:01:26 PM 12 0.00
> 0.00 2.94 0.00
> 0.00 0.00 0.00 13.73
> 83.33****
>
> 04:01:27 PM 12 0.00
> 0.00 2.00 0.00
> 0.00 0.00 0.00 0.00
> 98.00****
>
> ****
>
> ****
>
> The service *irqbalance* is running, but it looks not working well ????**
> **
>
> ****
>
> Thanks.****
>
> -chen****
>
> ****
>
> *From:* Luca Giraudo [mailto:lgiraudo at nicira.com]
> *Sent:* Tuesday, July 30, 2013 8:34 PM
> *To:* Li, Chen
> *Cc:* discuss at openvswitch.org
> *Subject:* RE: [ovs-discuss] network bandwidth in Openstack when using
> OVS+VLAN****
>
> ****
>
> Can you monitor the CPU in the receiving node? RX is the difficult part
> and in general there will be some process taking 100% of a CPU core.****
>
> Luca****
>
> On Jul 30, 2013 1:47 AM, "Li, Chen" <chen.li at intel.com> wrote:****
>
> For 6 Pairs CPU% in 1 second:****
>
> ****
>
> 04:11:08 PM CPU %usr %nice
> %sys %iowait %irq %soft
> %steal %guest %idle****
>
> 04:11:36 PM all 0.42 0.00
> 4.22 0.03 0.00
> 8.56 0.00 7.82 78.95****
>
> 04:11:36 PM 0 0.00
> 0.00 7.69 0.00 0.00
> 13.19 0.00 13.19 65.93****
>
> 04:11:36 PM 1 0.00
> 0.00 8.42 0.00
> 0.00 18.95 0.00 8.42
> 64.21****
>
> 04:11:36 PM 2 1.02
> 0.00 6.12 0.00 0.00
> 13.27 0.00 17.35 62.24****
>
> 04:11:36 PM 3 0.00
> 0.00 10.10 0.00 0.00
> 24.24 0.00 10.10 55.56****
>
> 04:11:36 PM 4 0.00
> 0.00 10.42 0.00 0.00
> 30.21 0.00 4.17 55.21****
>
> 04:11:36 PM 5 0.00
> 0.00 10.64 0.00 0.00
> 17.02 0.00 10.64 61.70****
>
> 04:11:36 PM 6 0.00
> 0.00 9.68 0.00
> 0.00 18.28 0.00 16.13 56.99
> ****
>
> 04:11:36 PM 7 0.00
> 0.00 9.78 0.00 0.00
> 27.17 0.00 4.35 58.70****
>
> 04:11:36 PM 8 1.08
> 0.00 10.75 0.00 0.00
> 20.43 0.00 10.75 56.99****
>
> 04:11:36 PM 9 1.04
> 0.00 9.38 0.00 0.00
> 17.71 0.00 11.46 60.42****
>
> 04:11:36 PM 10 0.99
> 0.00 7.92 0.00
> 0.00 9.90 0.00 13.86
> 67.33****
>
> 04:11:36 PM 11 1.04 0.00
> 4.17 0.00 0.00
> 5.21 0.00 27.08 62.50****
>
> 04:11:36 PM 12 0.00
> 0.00 3.09 0.00 0.00
> 10.31 0.00 14.43 72.16****
>
> 04:11:36 PM 13 0.00
> 0.00 11.00 0.00 0.00
> 15.00 0.00 18.00 56.00****
>
> 04:11:36 PM 14 0.00
> 0.00 6.45 0.00 0.00
> 17.20 0.00 15.05 61.29****
>
> 04:11:36 PM 15 0.00
> 0.00 7.29 0.00 0.00
> 19.79 0.00 21.88 52.08****
>
> 04:11:36 PM 16 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:11:36 PM 17 0.00
> 0.00 1.00 0.00
> 0.00 0.00 0.00 0.00
> 99.00****
>
> 04:11:36 PM 18 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:11:36 PM 19 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:11:36 PM 20 6.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 16.00
> 78.00****
>
> 04:11:36 PM 21 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:11:36 PM 22 0.00
> 0.00 1.00 0.00
> 0.00 0.00 0.00 0.00
> 99.00****
>
> 04:11:36 PM 23 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:11:36 PM 24 1.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 99.00****
>
> 04:11:36 PM 25 0.00
> 0.00 1.00 0.00
> 0.00 2.00 0.00 0.00
> 97.00****
>
> 04:11:36 PM 26 4.95
> 0.00 0.99 0.00
> 0.00 0.00 0.00 0.00
> 94.06****
>
> 04:11:36 PM 27 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:11:36 PM 28 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:11:36 PM 29 0.00
> 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> 100.00****
>
> 04:11:36 PM 30 0.00
> 0.00 0.99 0.00
> 0.00 0.00 0.00 10.89
> 88.12****
>
> 04:11:36 PM 31 0.00
> 0.00 0.99 0.00
> 0.00 0.00 0.00 7.92
> 92.08****
>
> ****
>
> ****
>
> *From:* discuss-bounces at openvswitch.org [mailto:
> discuss-bounces at openvswitch.org] *On Behalf Of *Li, Chen
> *Sent:* Tuesday, July 30, 2013 4:42 PM
> *To:* Luca Giraudo
> *Cc:* discuss at openvswitch.org
> *Subject:* Re: [ovs-discuss] network bandwidth in Openstack when using
> OVS+VLAN****
>
> ****
>
> ** are you using KVM? If yes, can you verify that your VMs are using
> vhost_net, please? In the KVM process related to the VM there should be a
> vhost=on parameter. If not, modprobe vhost_net.*****
>
> Yes.****
>
> And after enable vhost_net, I can get higher bandwidth now:****
>
> One pair, increase from *1.18* Gbits/sec => *2.38* Gbits/sec ****
>
> Six pairs, increase from *4.25 *Gbits/sec => *6.98* Gbits/sec ** **
>
> ****
>
> ** I saw different throughput with different OS in the VM. Did you try
> different VMs?*****
>
> No, but I guess different OS in VM might cause performance difference like
> 5Gbit/sec vs. 6Gbit/sec, but not 2Gbit/s vs. 6Gbit/sec****
>
> ** what bandwidth do you get among two VMs in the same compute node?*****
>
> *14.7 *Gbits/sec****
>
> ** can you monitor the CPU usage in the compute node and look out for big
> CPU consumer, please?*****
>
> I’m running on a SNB-EP server, so I have 32 cores (HT enabled) for one
> compute node.****
>
> ****
>
> *Only observed the compute node that run iperf client (send packages out)*
> ****
>
> * *****
>
> 1. The average CPU%****
>
> Total average CPU% = 6.66 %****
>
> Average: CPU %usr %nice %sys %iowait %irq %soft
> %steal %guest %idle****
>
> Average: all 1.66 0.00 1.38 0.05 0.00
> 1.83 0.00 1.74 93.34****
>
> ****
>
> Per-core :****
>
> The highest CPU% = 21.65%****
>
> Average: CPU %usr %nice %sys %iowait %irq %soft
> %steal %guest %idle****
>
> Average: 0 0.30 0.00 5.29 0.02 0.00
> *11.29* 0.00 4.75 78.35****
>
> ****
>
> 2. CPU% time line : only check the CPU core with the highest CPU% ***
> *
>
> 02:23:29 PM CPU %usr %nice %sys %iowait %irq %soft
> %steal %guest %idle****
>
> 02:23:29 PM 0 0.00 0.00 2.06 0.00 0.00
> 4.12 0.00 0.00 93.81****
>
> 02:23:30 PM 0 0.00 0.00 25.77 0.00 0.00
> 67.01 0.00 0.00 *7.22*****
>
> 02:23:31 PM 0 0.00 0.00 26.04 0.00 0.00
> 59.38 0.00 0.00 14.58****
>
> 02:23:32 PM 0 0.00 0.00 23.00 0.00 0.00
> 52.00 0.00 0.00 25.00****
>
> 02:23:33 PM 0 0.00 0.00 28.28 0.00 0.00
> 52.53 0.00 0.00 19.19****
>
> 02:23:34 PM 0 0.00 0.00 10.89 0.00 0.00
> 19.80 0.00 0.00 69.31****
>
> 02:23:35 PM 0 1.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 99.00****
>
> 02:23:36 PM 0 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00****
>
> 02:23:37 PM 0 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 1.00 99.00****
>
> 02:23:38 PM 0 0.99 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 99.01****
>
> 02:23:39 PM 0 0.00 0.00 1.00 0.00 0.00
> 0.00 0.00 0.00 99.00****
>
> 02:23:40 PM 0 1.01 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 98.99****
>
> 02:23:41 PM 0 2.06 0.00 15.46 0.00 0.00
> 36.08 0.00 0.00 46.39****
>
> 02:23:42 PM 0 1.98 0.00 12.87 0.00 0.00
> 34.65 0.00 0.00 50.50****
>
> 02:23:43 PM 0 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 15.15 84.85****
>
> 02:23:44 PM 0 0.99 0.00 0.99 0.00 0.00
> 0.00 0.00 0.00 98.02****
>
> 02:23:45 PM 0 0.00 0.00 2.00 0.00 0.00
> 0.00 0.00 19.00 80.00****
>
> 02:23:46 PM 0 0.99 0.00 0.00 0.00 0.00
> 0.00 0.00 23.76 75.25****
>
> 02:23:47 PM 0 0.00 0.00 24.49 0.00 0.00
> 40.82 0.00 0.00 34.69****
>
> 02:23:48 PM 0 0.00 0.00 23.96 0.00 0.00
> 60.42 0.00 0.00 15.62****
>
> 02:23:49 PM 0 0.00 0.00 23.96 0.00 0.00
> 60.42 0.00 0.00 15.62****
>
> 02:23:50 PM 0 0.00 0.00 6.12 0.00 0.00
> 18.37 0.00 0.00 75.51****
>
> 02:23:51 PM 0 1.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 99.00****
>
> ****
>
> ****
>
> *From:* Luca Giraudo [mailto:lgiraudo at nicira.com <lgiraudo at nicira.com>]
> *Sent:* Tuesday, July 30, 2013 1:37 PM
> *To:* Li, Chen
> *Cc:* discuss at openvswitch.org; Gurucharan Shetty
> *Subject:* Re: [ovs-discuss] network bandwidth in Openstack when using
> OVS+VLAN****
>
> ****
>
> Few more things to check:****
>
> * are you using KVM? If yes, can you verify that your VMs are using
> vhost_net, please? In the KVM process related to the VM there should be a
> vhost=on parameter. If not, modprobe vhost_net.****
>
> * I saw different throughput with different OS in the VM. Did you try
> different VMs?****
>
> * what bandwidth do you get among two VMs in the same compute node?****
>
> * can you monitor the CPU usage in the compute node and look out for big
> CPU consumer, please?****
>
> Thanks,
> Luca****
>
> On Jul 29, 2013 6:26 PM, "Li, Chen" <chen.li at intel.com> wrote:****
>
> ** Is the VM ethernet driver a para-virtual driver? Para-virtual drivers
> give a good performance boost.*****
>
> I have used openstack default parameters, it is virtio, I think virtio
> should have a good performance:****
>
> <interface type='bridge'>****
>
> <mac address='fa:16:3e:ca:4a:86'/>****
>
> <source bridge='br-int'/>****
>
> <virtualport type='openvswitch'>****
>
> <parameters interfaceid='3213dbec-f2ea-462f-818b-e07b76a1752c'/>**
> **
>
> </virtualport>****
>
> <target dev='tap3213dbec-f2'/>****
>
> <model type='virtio'/>****
>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>****
>
> </interface>****
>
> ****
>
> ****
>
> ** Is TSO ON in the VM and the Hypervisor?*****
>
> * *****
>
> The VM:****
>
> Features for eth0:****
>
> rx-checksumming: off [fixed]****
>
> tx-checksumming: on****
>
> tx-checksum-ipv4: off [fixed]****
>
> tx-checksum-ip-generic: on****
>
> tx-checksum-ipv6: off [fixed]****
>
> tx-checksum-fcoe-crc: off [fixed]****
>
> tx-checksum-sctp: off [fixed]****
>
> scatter-gather: on****
>
> tx-scatter-gather: on****
>
> tx-scatter-gather-fraglist: on****
>
> *tcp-segmentation-offload: on*****
>
> tx-tcp-segmentation: on****
>
> tx-tcp-ecn-segmentation: on****
>
> tx-tcp6-segmentation: on****
>
> udp-fragmentation-offload: on****
>
> generic-segmentation-offload: on****
>
> generic-receive-offload: on****
>
> large-receive-offload: off [fixed]****
>
> rx-vlan-offload: off [fixed]****
>
> tx-vlan-offload: off [fixed]****
>
> ntuple-filters: off [fixed]****
>
> receive-hashing: off [fixed]****
>
> highdma: on [fixed]****
>
> rx-vlan-filter: on [fixed]****
>
> vlan-challenged: off [fixed]****
>
> tx-lockless: off [fixed]****
>
> netns-local: off [fixed]****
>
> tx-gso-robust: off [fixed]****
>
> tx-fcoe-segmentation: off [fixed]****
>
> fcoe-mtu: off [fixed]****
>
> tx-nocache-copy: on****
>
> loopback: off [fixed]****
>
> rx-fcs: off [fixed]****
>
> rx-all: off [fixed]****
>
> The hypervisor:****
>
> Offload parameters for eth4:****
>
> rx-checksumming: on****
>
> tx-checksumming: on****
>
> scatter-gather: on****
>
> *tcp-segmentation-offload: on*****
>
> udp-fragmentation-offload: off****
>
> generic-segmentation-offload: on****
>
> generic-receive-offload: on****
>
> large-receive-offload: off****
>
> rx-vlan-offload: on****
>
> tx-vlan-offload: on****
>
> ntuple-filters: off****
>
> receive-hashing: on****
>
> ****
>
> ** What throughput do you get while using Linux bridge instead of OVS?****
> *
>
> Currently, I don’t have a linux bridge environment.****
>
> But, I remember in virtio test, while I create bridge by hand and assigned
> it to an instance, I can always get near hardware limitation bandwidth if I
> have enough threads.****
>
> ****
>
> ** Are you using tunnels? If you are using a tunnel like GRE, you will
> see a throughput drop.*****
>
> No, I’m working under Quantum+OVS+*VLAN*.****
>
> ****
>
> Thanks.****
>
> -chen****
>
> ****
>
> *From:* Gurucharan Shetty [mailto:shettyg at nicira.com]
> *Sent:* Tuesday, July 30, 2013 12:06 AM
> *To:* Li, Chen
> *Cc:* discuss at openvswitch.org
> *Subject:* Re: [ovs-discuss] network bandwidth in Openstack when using
> OVS+VLAN****
>
> ****
>
> There could be multiple reasons for the low throughput. I would probably
> look at the following.****
>
> ****
>
> * Is the VM ethernet driver a para-virtual driver? Para-virtual drivers
> give a good performance boost.****
>
> * Is TSO ON in the VM and the Hypervisor?****
>
> * What throughput do you get while using Linux bridge instead of OVS?****
>
> * Are you using tunnels? If you are using a tunnel like GRE, you will see
> a throughput drop.****
>
> ****
>
> ****
>
> On Mon, Jul 29, 2013 at 1:48 AM, Li, Chen <chen.li at intel.com> wrote:****
>
> Hi list,****
>
> ****
>
> I’m a new user to OVS.****
>
> ****
>
> I installed OpenStack Grizzly, and using Quantum + OVS + VLAN for network.
> ****
>
> ****
>
> I have two compute nodes with 10 Gb NICs, and the bandwidth between them
> is about *8.49* Gbits/sec (tested by iperf).****
>
> ****
>
> I started one instance at each compute node:****
>
> instance-a => compute1****
>
> instance-b=> compute2****
>
> The bandwidth between this two virtual machine is only *1.18* Gbits/sec.**
> **
>
> ****
>
> Then I start 6 instances at each compute node:****
>
> ( instance-a => compute1 ) ----- iperf------ > (instance-b=>
> compute2)****
>
> ( instance-c => compute1 ) ----- iperf------ >
> (instance-d=> compute2)****
>
> ( instance-e => compute1 ) ----- iperf------ >
> (instance-f=> compute2)****
>
> ( instance-g => compute1 ) ----- iperf------ >
> (instance-h=> compute2)****
>
> ( instance-i => compute1 ) ----- iperf------ >
> (instance-j=> compute2)****
>
> ( instance-k => compute1 ) ----- iperf------ >
> (instance-l=> compute2)****
>
> The total bandwidth is only *4.25 *Gbits/sec.****
>
> ****
>
> ****
>
> Anyone know why the performance is this low ?****
>
> ****
>
> Thanks.****
>
> -chen****
>
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> http://openvswitch.org/mailman/listinfo/discuss****
>
> ****
>
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> http://openvswitch.org/mailman/listinfo/discuss****
>
> ** **
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> http://openvswitch.org/mailman/listinfo/discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20130801/bf6b8dcc/attachment.html>
More information about the discuss
mailing list