[ovs-discuss] network bandwidth in Openstack when using OVS+VLAN

Li, Chen chen.li at intel.com
Fri Aug 2 06:10:50 UTC 2013


I'm already using :
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchVirtualPortDriver

And I think the  flow is goes with  tapxx -> br-int (bridge) -> int-br-eth4  ->  phy-br-eth4 -> br-eth4 (bridge) -> eth4

ovs-vsctl show :
f0fa9384-f6d0-4071-a5ac-53245eb2d68d
    Bridge br-int
        Port "tapc447a6c8-18"
            tag: 2
            Interface "tapc447a6c8-18"
        Port "tap4f625dbc-21"
            tag: 2
            Interface "tap4f625dbc-21"
        Port "tape9c77763-3f"
            tag: 2
            Interface "tape9c77763-3f"
        Port "tap607a56d5-47"
            tag: 2
            Interface "tap607a56d5-47"
        Port "tap72cebb96-c9"
            tag: 2
            Interface "tap72cebb96-c9"
        Port "tap4270f5ae-37"
            tag: 2
            Interface "tap4270f5ae-37"
        Port "tap9d74b661-42"
            tag: 2
            Interface "tap9d74b661-42"
        Port "tap8781a0ba-2d"
            tag: 2
            Interface "tap8781a0ba-2d"
        Port "int-br-eth4"
            Interface "int-br-eth4"
        Port "tapa5d4f8bd-bd"
            tag: 2
            Interface "tapa5d4f8bd-bd"
        Port "tap4f893e9f-de"
            tag: 2
            Interface "tap4f893e9f-de"
        Port br-int
            Interface br-int
                type: internal
    Bridge "br-eth4"
        Port "eth4"
            Interface "eth4"
        Port "br-eth4"
            Interface "br-eth4"
                type: internal
        Port "phy-br-eth4"
            Interface "phy-br-eth4"
    ovs_version: "1.4.0+build0"

From: Aaron Rosen [mailto:arosen at nicira.com]
Sent: Friday, August 02, 2013 2:01 PM
To: Li, Chen
Cc: Luca Giraudo; discuss at openvswitch.org
Subject: Re: [ovs-discuss] network bandwidth in Openstack when using OVS+VLAN

Hi,

What's your configuration in nova.conf for: libvirt_vif_driver ? If this is set to: nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver I'd guess this is the culprit. The OVS plugin for openstack leverages linuxbridges as well in order to use iptables for it's security group implementation. The bridge configuration on the host would look like this:

br-int(ovs_bridge) --- qvo-(veth)-qvb ----qbr(linux_bridge)--tap(vm)

If you are not using security groups you could try setting: libvirt_vif_driver to  nova.virt.libvirt.vif.LibvirtOpenVswitchVirtualPortDriver or  nova.virt.libvirt.vif.LibvirtOpenVswitchDriver

Aaron




On Thu, Aug 1, 2013 at 8:43 PM, Li, Chen <chen.li at intel.com<mailto:chen.li at intel.com>> wrote:

* interrupts: cat /proc/interrupts | grep <your physical interface>

I tested for 60 seconds, and get the flowing  data by using end - start.
Already normalize to 1 second:

                CPU0     CPU1     CPU2     CPU3     CPU4     CPU5     CPU6     CPU7     CPU8     CPU9     CPU10   CPU11                CPU12   CPU13   CPU14   CPU15   CPU16   CPU17   CPU18   CPU19   CPU20   CPU21   CPU22   CPU23                CPU24   CPU25   CPU26   CPU27   CPU28   CPU29   CPU30   CPU31
126        1441       0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-0
127        0              1099       0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-1
128        0              0              1              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-2
129        0              0              0              1              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-3
130        0              0              0              0              1              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-4
131        0              0              0              0              0              1              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-5
132        0              0              0              0              0              0              166         0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-6
133        0              0              0              0              0              0              0              1              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-7
134        0              0              0              0              0              0              0              0              12989    0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-8
135        0              0              0              0              0              0              0              0              0              532         0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-9
136        0              0              0              0              0              0              0              0              0              0              455         0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-10
137        0              0              0              0              0              0              0              0              0              0              0              36           0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-11
138        0              0              0              0              0              0              0              0              0              0              0              0              5285                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-12
139        0              0              0              0              0              0              0              0              0              0              0              0              0                4              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-13
140        0              0              0              0              0              0              0              0              0              0              0              0              0                0              1              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-14
141        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              1              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-15
142        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              1679       0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-16
143        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              1770       0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-17
144        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              1              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-18
145        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              1              0              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-19
146        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              1              0              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-20
147        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              1              0              0              0              0                0              0              0              0              0              0              eth4-TxRx-21
148        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              612         0              0              0                0              0              0              0              0              0              eth4-TxRx-22
149        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              1              0              0                0              0              0              0              0              0              eth4-TxRx-23
150        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              27646    0                0              0              0              0              0              0              eth4-TxRx-24
151        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              5                0              0              0              0              0              0              eth4-TxRx-25
152        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                1              0              0              0              0              0              eth4-TxRx-26
153        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              1              0              0              0              0              eth4-TxRx-27
154        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              518         0              0              0              eth4-TxRx-28
155        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              1              0              0              eth4-TxRx-29
156        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              1              0              eth4-TxRx-30
157        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              1              eth4-TxRx-31
158        0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              0              0              0              0              0              0              0                0              0              0              0              0              0              eth4

* processes that are consuming more CPU. Take a look at kvm, vhost, ovs-vswitchd. If you use top, press "1" and take a look at the idle time of CPUs to see what are the most overloaded cores.

top - 11:32:12 up 77 days,  2:16,  2 users,  load average: 0.89, 1.02, 0.97
Tasks: 280 total,   1 running, 279 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.3%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.7%us,  1.0%sy,  0.0%ni, 98.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.0%us,  9.0%sy,  0.0%ni, 76.0%id,  0.0%wa,  0.0%hi, 15.0%si,  0.0%st
Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu8  :  0.0%us,  0.0%sy,  0.0%ni, 97.7%id,  2.3%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu9  :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu10 :  1.3%us,  0.0%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu11 :  0.0%us,  0.7%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu12 :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu13 :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu14 : 82.7%us,  2.7%sy,  0.0%ni, 14.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu15 :  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu16 :  5.9%us,  1.3%sy,  0.0%ni, 92.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu17 :  4.7%us,  1.3%sy,  0.0%ni, 94.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu18 :  1.7%us,  0.7%sy,  0.0%ni, 97.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu19 :  0.0%us, 27.1%sy,  0.0%ni, 27.1%id,  0.0%wa,  0.0%hi, 45.8%si,  0.0%st
Cpu20 :  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu21 :  6.0%us,  1.7%sy,  0.0%ni, 92.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu22 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu23 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu24 :  0.3%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu25 : 24.7%us,  5.4%sy,  0.0%ni, 69.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu26 :  8.3%us,  3.6%sy,  0.0%ni, 88.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu27 :  0.0%us,  0.7%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu28 :  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu29 : 10.0%us,  2.0%sy,  0.0%ni, 88.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu30 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu31 :  5.0%us,  1.3%sy,  0.0%ni, 93.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  131985576k total,  9768740k used, 122216836k free,   309352k buffers
Swap:   975868k total,        0k used,   975868k free,  4573908k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
34364 root      20   0     0    0    0 S   92  0.0   9:03.54 vhost-34350
34350 libvirt-  20   0 4851m 550m 7548 S   85  0.4  14:53.36 kvm
    83 root      20   0     0    0    0 S    4  0.0   2:28.41 ksoftirqd/19
114986 quantum   20   0  101m  29m 4352 S    2  0.0 160:36.58 python
    19 root      20   0     0    0    0 S    1  0.0   0:10.68 ksoftirqd/3
  1621 root      10 -10 19772 2232 1340 S    1  0.0 115:56.88 ovsdb-server
   150 root      25   5     0    0    0 S    0  0.0 126:52.72 ksmd
   179 root      20   0     0    0    0 S    0  0.0   4:49.86 kworker/14:1
   303 root      20   0     0    0    0 S    0  0.0  12:48.86 kworker/19:1
  1656 root      10 -10 25364 4152 1252 S    0  0.0 145:04.35 ovs-vswitchd
38641 root      20   0     0    0    0 S    0  0.0   0:23.83 kworker/18:2
59338 libvirt-  20   0 4850m 545m 7548 S    0  0.4   7:00.51 kvm
81053 root      20   0  898m  15m 6964 S    0  0.0 163:00.38 libvirtd
125121 root      20   0 17472 1552 1048 R    0  0.0   0:00.22 top
126197 root      20   0 11436 1112  852 S    0  0.0   0:00.08 sadc
     1 root      20   0 27796 5732 1356 S    0  0.0   8:44.68 init
     2 root      20   0     0    0    0 S    0  0.0   0:00.11 kthreadd
     3 root      20   0     0    0    0 S    0  0.0   1:26.52 ksoftirqd/0
     6 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0
     7 root      RT   0     0    0    0 S    0  0.0   0:22.08 watchdog/0


* /etc/init.d/irqbalance status
service irqbalance status
irqbalance start/running, process 1665

* what CPU are you using and how many sockets/cores you have
Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
2 sockets, 8*2 physical core, 32 logical core (HT enabled)

* did you enable power savings mechanisms in the BIOS?
Yes. I have both p-sate and c-stat enabled.
While idle, CPU frequency is 1200 MHz.


From: Luca Giraudo [mailto:lgiraudo at nicira.com<mailto:lgiraudo at nicira.com>]
Sent: Friday, August 02, 2013 12:31 AM
To: Li, Chen
Cc: discuss at openvswitch.org<mailto:discuss at openvswitch.org>
Subject: Re: [ovs-discuss] network bandwidth in Openstack when using OVS+VLAN

Can you repeat the VM-VM test and monitor the RX hypervisor please?

I am interested in the following parameters:

* interrupts: cat /proc/interrupts | grep <your physical interface>
* processes that are consuming more CPU. Take a look at kvm, vhost, ovs-vswitchd. If you use top, press "1" and take a look at the idle time of CPUs to see what are the most overloaded cores.
* /etc/init.d/irqbalance status
* what CPU are you using and how many sockets/cores you have
* did you enable power savings mechanisms in the BIOS?

I suppose that the performance is limited by the receiving side so I would like to understand where is the problem (CPU limitations, NIC, kvm/vhost, OVS, etc.)

Luca

On Tue, Jul 30, 2013 at 8:37 PM, Li, Chen <chen.li at intel.com<mailto:chen.li at intel.com>> wrote:
Yes. For one pair test, rx side is the bottleneck currently.
I changed one side of the test to a physical machine.
Then I get:
     One instance -> physical machine (instance send out packages) = > 5.71 Gbit/sec
     physical machine ->  One instance (physical machine send out packages) = >  1.79 Gbit/sec

The CPU% in client side:
     Average CPU% is still low, CPU7 is the CPU core with the highest CPU%:
Average:                    CPU                %usr            %nice                %sys    %iowait                %irq            %soft        %steal        %guest            %idle
Average:                    all                1.47                0.00                1.85                0.03                0.00                1.77                0.00                2.61            92.26
Average:                     7                0.17                0.00                0.92                0.00                0.00                0.00                0.00            28.79            70.13

The CPU% time line for CPU7:
                                     CPU                %usr            %nice                %sys    %iowait                %irq            %soft        %steal        %guest            %idle
04:00:28    PM                7                1.00                0.00                3.00                0.00                0.00                0.00                0.00            90.00                6.00
04:00:29    PM                7                0.00                0.00                1.98                0.00                0.00                0.00                0.00            91.09                6.93
04:00:30    PM                7                0.00                0.00                3.06                0.00                0.00                0.00                0.00            91.84                6.12
04:00:31    PM                7                0.00                0.00                2.02                0.00                0.00                0.00                0.00            90.91                7.07
04:00:32    PM                7                0.00                0.00                3.00                0.00                0.00                0.00                0.00            90.00                7.00
04:00:33    PM                7                1.01                0.00                3.03                0.00                0.00                0.00                0.00            88.89                7.07
04:00:34    PM                7                0.00                0.00                2.97                0.00                0.00                0.00                0.00            91.09                6.93
04:00:35    PM                7                0.00                0.00                2.00                0.00                0.00                0.00                0.00            90.00                8.00
04:00:36    PM                7                0.00                0.00                3.00                0.00                0.00                0.00                0.00            90.00                7.00
04:00:37    PM                7                0.96                0.00                2.88                0.00                0.00                0.00                0.00            89.42                6.73
04:00:38    PM                7                0.00                0.00                4.00                0.00                0.00                0.00                0.00            89.00                8.00
04:00:39    PM                7                0.00                0.00                1.04                0.00                0.00                0.00                0.00            91.67                7.29
04:00:40    PM                7                0.99                0.00                1.98                0.00                0.00                0.00                0.00            88.12                8.91
04:00:41    PM                7                0.00                0.00                3.03                0.00                0.00                0.00                0.00            91.92                6.06
04:00:42    PM                7                0.00                0.00                2.94                0.00                0.00                0.00                0.00            90.20                6.86
04:00:43    PM                7                0.00                0.00                3.96                0.00                0.00                0.00                0.00            89.11                6.93
04:00:44    PM                7                0.00                0.00                5.05                0.00                0.00                0.00                0.00            88.89                6.06
04:00:45    PM                7                0.99                0.00                1.98                0.00                0.00                0.00                0.00            90.10                6.93
04:00:46    PM                7                0.00                0.00                1.96                0.00                0.00                0.00                0.00            92.16                5.88
04:00:47    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00            10.00            91.00
04:00:48    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:49    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:50    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:51    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:52    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:53    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:54    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:55    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:56    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:57    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:58    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:59    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:00    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:01    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:02    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:03    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:04    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:05    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:06    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:07    PM                7                0.99                0.00                0.00                0.00                0.00                0.00                0.00                0.00            99.01
04:01:08    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:09    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:10    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:11    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:12    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:13    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:14    PM                7                7.92                0.00                1.98                0.00                0.00                0.00                0.00                0.00            90.10
04:01:15    PM                7                1.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00            99.00
04:01:16    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:17    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:18    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:19    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:20    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:21    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:22    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:23    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:24    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:25    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.99            99.01
04:01:26    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:27    PM                7                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00

Another CPU core : CPU12:
                                        CPU                %usr            %nice                %sys    %iowait                %irq            %soft        %steal        %guest            %idle
04:00:28    PM            12                0.00                0.00            38.54                0.00                0.00            60.42                0.00                0.00                1.04
04:00:29    PM            12                0.00                0.00            40.40                0.00                0.00            58.59                0.00                0.00                1.01
04:00:30    PM            12                0.00                0.00            51.49                0.00                0.00            46.53                0.00                0.00                1.98
04:00:31    PM            12                0.00                0.00            34.02                0.00                0.00            63.92                0.00                0.00                2.06
04:00:32    PM            12                0.00                0.00            36.00                0.00                0.00            49.00                0.00                0.00            15.00
04:00:33    PM            12                0.00                0.00            40.59                0.00                0.00            57.43                0.00                0.00                1.98
04:00:34    PM            12                0.00                0.00            46.46                0.00                0.00            52.53                0.00                0.00                1.01
04:00:35    PM            12                0.00                0.00            36.73                0.00                0.00            61.22                0.00                0.00                2.04
04:00:36    PM            12                0.00                0.00            14.85                0.00                0.00            19.80                0.00                0.00            65.35
04:00:37    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:38    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:39    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:40    PM            12                0.00                0.00            28.12                0.00                0.00            38.54                0.00                0.00            33.33
04:00:41    PM            12                0.00                0.00            35.71                0.00                0.00            62.24                0.00                0.00                2.04
04:00:42    PM            12                0.00                0.00            39.39                0.00                0.00            51.52                0.00                0.00                9.09
04:00:43    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:44    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:45    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:46    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:47    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:48    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:49    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:50    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:51    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:52    PM            12                0.99                0.00                0.99                0.00                0.00                0.00                0.00                0.00            98.02
04:00:53    PM            12                0.00                0.00                1.00                0.00                0.00                0.00                0.00                0.00            99.00
04:00:54    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:55    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:56    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:57    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:00:58    PM            12                1.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00            99.00
04:00:59    PM            12                1.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00            99.00
04:01:00    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:01    PM            12                1.01                0.00                0.00                0.00                0.00                0.00                0.00                0.00            98.99
04:01:02    PM            12                0.00                0.00                0.99                0.00                0.00                0.00                0.00                0.00            99.01
04:01:03    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:04    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:05    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:06    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:07    PM            12                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:01:08    PM            12                0.00                0.00                2.00                0.00                0.00                0.00                0.00                0.00            98.00
04:01:09    PM            12                2.02                0.00                1.01                0.00                0.00                0.00                0.00                0.00            96.97
04:01:10    PM            12                0.00                0.00                2.02                0.00                0.00                0.00                0.00                0.00            97.98
04:01:11    PM            12                1.00                0.00                1.00                0.00                0.00                0.00                0.00                0.00            98.00
04:01:12    PM            12                0.00                0.00                2.00                0.00                0.00                0.00                0.00                0.00            98.00
04:01:13    PM            12                0.00                0.00                1.01                0.00                0.00                0.00                0.00                0.00            98.99
04:01:14    PM            12                1.00                0.00                2.00                0.00                0.00                0.00                0.00                0.00            97.00
04:01:15    PM            12                0.00                0.00                2.02                0.00                0.00                0.00                0.00                0.00            97.98
04:01:16    PM            12                0.00                0.00                1.00                0.00                0.00                0.00                0.00                0.00            99.00
04:01:17    PM            12                0.00                0.00                1.01                0.00                0.00                0.00                0.00                0.00            98.99
04:01:18    PM            12                0.00                0.00                2.00                0.00                0.00                0.00                0.00                0.00            98.00
04:01:19    PM            12                0.99                0.00                0.99                0.00                0.00                0.00                0.00                0.00            98.02
04:01:20    PM            12                0.00                0.00                1.01                0.00                0.00                0.00                0.00                0.00            98.99
04:01:21    PM            12                0.00                0.00                1.00                0.00                0.00                0.00                0.00                0.00            99.00
04:01:22    PM            12                0.00                0.00                1.01                0.00                0.00                0.00                0.00                0.00            98.99
04:01:23    PM            12                0.00                0.00                2.00                0.00                0.00                0.00                0.00                0.00            98.00
04:01:24    PM            12                1.01                0.00                2.02                0.00                0.00                0.00                0.00                0.00            96.97
04:01:25    PM            12                1.00                0.00                3.00                0.00                0.00                0.00                0.00                0.00            96.00
04:01:26    PM            12                0.00                0.00                2.94                0.00                0.00                0.00                0.00            13.73            83.33
04:01:27    PM            12                0.00                0.00                2.00                0.00                0.00                0.00                0.00                0.00            98.00


The service irqbalance  is running, but it looks not working well ????

Thanks.
-chen

From: Luca Giraudo [mailto:lgiraudo at nicira.com<mailto:lgiraudo at nicira.com>]
Sent: Tuesday, July 30, 2013 8:34 PM
To: Li, Chen
Cc: discuss at openvswitch.org<mailto:discuss at openvswitch.org>
Subject: RE: [ovs-discuss] network bandwidth in Openstack when using OVS+VLAN


Can you monitor the CPU in the receiving node? RX is the difficult part and in general there will be some process taking 100% of a CPU core.

Luca
On Jul 30, 2013 1:47 AM, "Li, Chen" <chen.li at intel.com<mailto:chen.li at intel.com>> wrote:
For 6 Pairs CPU% in 1 second:

04:11:08    PM      CPU                 %usr           %nice              %sys               %iowait       %irq                %soft            %steal          %guest        %idle
04:11:36    PM        all                0.42                0.00                   4.22                0.03                0.00                8.56                0.00                7.82            78.95
04:11:36    PM                0                0.00                0.00                7.69                0.00                0.00            13.19                0.00            13.19            65.93
04:11:36    PM                1                0.00                0.00                8.42                0.00                0.00            18.95                0.00                8.42            64.21
04:11:36    PM                2                1.02                0.00                6.12                0.00                0.00            13.27                0.00            17.35            62.24
04:11:36    PM                3                0.00                0.00            10.10                0.00                0.00            24.24                0.00            10.10            55.56
04:11:36    PM                4                0.00                0.00            10.42                0.00                0.00            30.21                0.00                4.17            55.21
04:11:36    PM                5                0.00                0.00            10.64                0.00                0.00            17.02                0.00            10.64            61.70
04:11:36    PM                6                0.00                0.00                9.68                0.00                0.00            18.28                0.00            16.13            56.99
04:11:36    PM                7                0.00                0.00                9.78                0.00                0.00            27.17                0.00                4.35            58.70
04:11:36    PM                8                1.08                0.00            10.75                0.00                0.00            20.43                0.00            10.75            56.99
04:11:36    PM                9                1.04                0.00                9.38                0.00                0.00            17.71                0.00            11.46            60.42
04:11:36    PM            10                0.99                0.00                7.92                0.00                0.00                9.90                0.00            13.86            67.33
04:11:36    PM            11                1.04                0.00                4.17                0.00                0.00                5.21                0.00            27.08            62.50
04:11:36    PM            12                0.00                0.00                3.09                0.00                0.00            10.31                0.00            14.43            72.16
04:11:36    PM            13                0.00                0.00            11.00                0.00                0.00            15.00                0.00            18.00            56.00
04:11:36    PM            14                0.00                0.00                6.45                0.00                0.00            17.20                0.00            15.05            61.29
04:11:36    PM            15                0.00                0.00                7.29                0.00                0.00            19.79                0.00            21.88            52.08
04:11:36    PM            16                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:11:36    PM            17                0.00                0.00                1.00                0.00                0.00                0.00                0.00                0.00            99.00
04:11:36    PM            18                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:11:36    PM            19                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:11:36    PM            20                6.00                0.00                0.00                0.00                0.00                0.00                0.00            16.00            78.00
04:11:36    PM            21                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:11:36    PM            22                0.00                0.00                1.00                0.00                0.00                0.00                0.00                0.00            99.00
04:11:36    PM            23                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:11:36    PM            24                1.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00            99.00
04:11:36    PM            25                0.00                0.00                1.00                0.00                0.00                2.00                0.00                0.00            97.00
04:11:36    PM            26                4.95                0.00                0.99                0.00                0.00                0.00                0.00                0.00            94.06
04:11:36    PM            27                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:11:36    PM            28                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:11:36    PM            29                0.00                0.00                0.00                0.00                0.00                0.00                0.00                0.00        100.00
04:11:36    PM            30                0.00                0.00                0.99                0.00                0.00                0.00                0.00            10.89            88.12
04:11:36    PM            31                0.00                0.00                0.99                0.00                0.00                0.00                0.00                7.92            92.08


From: discuss-bounces at openvswitch.org<mailto:discuss-bounces at openvswitch.org> [mailto:discuss-bounces at openvswitch.org<mailto:discuss-bounces at openvswitch.org>] On Behalf Of Li, Chen
Sent: Tuesday, July 30, 2013 4:42 PM
To: Luca Giraudo
Cc: discuss at openvswitch.org<mailto:discuss at openvswitch.org>
Subject: Re: [ovs-discuss] network bandwidth in Openstack when using OVS+VLAN


* are you using KVM? If yes, can you verify that your VMs are using vhost_net, please? In the KVM process related to the VM there should be a vhost=on parameter. If not, modprobe vhost_net.

Yes.

And after enable vhost_net, I can get higher bandwidth now:

One pair, increase from 1.18 Gbits/sec  => 2.38 Gbits/sec

Six pairs,  increase from  4.25 Gbits/sec  =>  6.98 Gbits/sec



* I saw different throughput with different OS in the VM. Did you try different VMs?

No, but I guess different OS in VM might cause performance difference like 5Gbit/sec vs. 6Gbit/sec, but not 2Gbit/s vs. 6Gbit/sec

* what bandwidth do you get among two VMs in the same compute node?

14.7 Gbits/sec

* can you monitor the CPU usage in the compute node and look out for big CPU consumer, please?

I'm running on a SNB-EP server, so I have 32 cores (HT enabled) for one compute node.



Only observed the compute node that run iperf client (send packages out)



1.      The average CPU%

Total average CPU% = 6.66 %

Average:     CPU    %usr   %nice    %sys   %iowait    %irq   %soft  %steal  %guest   %idle

Average:     all         1.66    0.00      1.38      0.05         0.00    1.83    0.00       1.74      93.34



Per-core :

The highest CPU% = 21.65%

Average:     CPU    %usr   %nice    %sys    %iowait     %irq   %soft    %steal    %guest   %idle

Average:       0          0.30     0.00      5.29       0.02        0.00   11.29      0.00       4.75      78.35



2.      CPU% time line : only check the CPU core with the highest CPU%

02:23:29 PM  CPU    %usr   %nice    %sys     %iowait    %irq        %soft    %steal   %guest   %idle

02:23:29 PM    0         0.00    0.00      2.06       0.00         0.00         4.12      0.00        0.00       93.81

02:23:30 PM    0         0.00    0.00     25.77       0.00         0.00        67.01     0.00        0.00      7.22

02:23:31 PM    0         0.00    0.00     26.04       0.00         0.00        59.38    0.00        0.00       14.58

02:23:32 PM    0         0.00    0.00     23.00       0.00         0.00        52.00    0.00         0.00       25.00

02:23:33 PM    0         0.00    0.00     28.28       0.00         0.00        52.53    0.00        0.00      19.19

02:23:34 PM    0         0.00    0.00     10.89       0.00         0.00        19.80    0.00        0.00       69.31

02:23:35 PM    0         1.00    0.00      0.00       0.00         0.00         0.00      0.00        0.00      99.00

02:23:36 PM    0         0.00    0.00      0.00       0.00         0.00         0.00     0.00        0.00       100.00

02:23:37 PM    0         0.00    0.00      0.00       0.00         0.00         0.00     0.00        1.00        99.00

02:23:38 PM    0         0.99    0.00      0.00       0.00         0.00         0.00     0.00        0.00       99.01

02:23:39 PM    0         0.00    0.00      1.00       0.00         0.00         0.00     0.00        0.00       99.00

02:23:40 PM    0         1.01    0.00      0.00       0.00         0.00         0.00     0.00        0.00       98.99

02:23:41 PM    0         2.06    0.00     15.46       0.00         0.00        36.08    0.00        0.00      46.39

02:23:42 PM    0         1.98    0.00     12.87       0.00         0.00        34.65    0.00        0.00      50.50

02:23:43 PM    0         0.00    0.00      0.00       0.00         0.00         0.00      0.00       15.15      84.85

02:23:44 PM    0         0.99    0.00      0.99       0.00         0.00         0.00     0.00        0.00       98.02

02:23:45 PM    0         0.00    0.00      2.00       0.00         0.00         0.00     0.00       19.00       80.00

02:23:46 PM    0         0.99    0.00      0.00       0.00         0.00         0.00     0.00       23.76       75.25

02:23:47 PM    0         0.00    0.00     24.49       0.00         0.00        40.82    0.00        0.00       34.69

02:23:48 PM    0         0.00    0.00     23.96       0.00         0.00        60.42    0.00        0.00      15.62

02:23:49 PM    0         0.00    0.00     23.96       0.00         0.00        60.42    0.00        0.00      15.62

02:23:50 PM    0         0.00    0.00      6.12       0.00         0.00        18.37     0.00       0.00       75.51

02:23:51 PM    0         1.00    0.00      0.00       0.00         0.00         0.00      0.00       0.00       99.00


From: Luca Giraudo [mailto:lgiraudo at nicira.com]
Sent: Tuesday, July 30, 2013 1:37 PM
To: Li, Chen
Cc: discuss at openvswitch.org<mailto:discuss at openvswitch.org>; Gurucharan Shetty
Subject: Re: [ovs-discuss] network bandwidth in Openstack when using OVS+VLAN


Few more things to check:

* are you using KVM? If yes, can you verify that your VMs are using vhost_net, please? In the KVM process related to the VM there should be a vhost=on parameter. If not, modprobe vhost_net.

* I saw different throughput with different OS in the VM. Did you try different VMs?

* what bandwidth do you get among two VMs in the same compute node?

* can you monitor the CPU usage in the compute node and look out for big CPU consumer, please?

Thanks,
Luca
On Jul 29, 2013 6:26 PM, "Li, Chen" <chen.li at intel.com<mailto:chen.li at intel.com>> wrote:
* Is the VM ethernet driver a para-virtual driver? Para-virtual drivers give a good performance boost.
I have used openstack default parameters, it is virtio, I think virtio should have a good performance:
    <interface type='bridge'>
      <mac address='fa:16:3e:ca:4a:86'/>
      <source bridge='br-int'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='3213dbec-f2ea-462f-818b-e07b76a1752c'/>
      </virtualport>
      <target dev='tap3213dbec-f2'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>


* Is TSO ON in the VM and the Hypervisor?

The VM:
Features for eth0:
rx-checksumming: off [fixed]
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: on
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
The hypervisor:
Offload parameters for eth4:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on

* What throughput do you get while using Linux bridge instead of OVS?
Currently, I don't have a linux bridge environment.
But, I remember in virtio test, while I create bridge by hand and assigned it to an instance, I can always get near hardware limitation bandwidth if I have enough threads.

* Are you using tunnels? If you are using a tunnel like GRE, you will see a throughput drop.
No, I'm working under Quantum+OVS+VLAN.

Thanks.
-chen

From: Gurucharan Shetty [mailto:shettyg at nicira.com<mailto:shettyg at nicira.com>]
Sent: Tuesday, July 30, 2013 12:06 AM
To: Li, Chen
Cc: discuss at openvswitch.org<mailto:discuss at openvswitch.org>
Subject: Re: [ovs-discuss] network bandwidth in Openstack when using OVS+VLAN

There could be multiple reasons for the low throughput. I would probably look at the following.

* Is the VM ethernet driver a para-virtual driver? Para-virtual drivers give a good performance boost.
* Is TSO ON in the VM and the Hypervisor?
* What throughput do you get while using Linux bridge instead of OVS?
* Are you using tunnels? If you are using a tunnel like GRE, you will see a throughput drop.


On Mon, Jul 29, 2013 at 1:48 AM, Li, Chen <chen.li at intel.com<mailto:chen.li at intel.com>> wrote:
Hi list,

I'm a new user to OVS.

I installed OpenStack Grizzly, and  using Quantum + OVS + VLAN for network.

I have two compute nodes with 10 Gb NICs, and the bandwidth between them is about  8.49 Gbits/sec (tested by iperf).

I started one instance at each compute node:
instance-a => compute1
instance-b=> compute2
The bandwidth between this two virtual machine is only 1.18 Gbits/sec.

Then I start 6 instances at each compute node:
          (   instance-a => compute1 ) ----- iperf------ > (instance-b=> compute2)
                          (   instance-c => compute1 ) ----- iperf------ > (instance-d=> compute2)
                          (   instance-e => compute1 ) ----- iperf------ > (instance-f=> compute2)
                          (   instance-g => compute1 ) ----- iperf------ > (instance-h=> compute2)
                          (   instance-i => compute1 ) ----- iperf------ > (instance-j=> compute2)
                          (   instance-k => compute1 ) ----- iperf------ > (instance-l=> compute2)
The total bandwidth is only 4.25 Gbits/sec.


Anyone know why the performance is this low ?

Thanks.
-chen

_______________________________________________
discuss mailing list
discuss at openvswitch.org<mailto:discuss at openvswitch.org>
http://openvswitch.org/mailman/listinfo/discuss


_______________________________________________
discuss mailing list
discuss at openvswitch.org<mailto:discuss at openvswitch.org>
http://openvswitch.org/mailman/listinfo/discuss


_______________________________________________
discuss mailing list
discuss at openvswitch.org<mailto:discuss at openvswitch.org>
http://openvswitch.org/mailman/listinfo/discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20130802/5c6e7d16/attachment.html>


More information about the discuss mailing list