[ovs-discuss] Question in regard to packet errors with OVS

Morgan Yang Morgan.Yang at radisys.com
Thu Oct 24 01:11:27 UTC 2013


Just to elaborate, I don't think the physical interface is a bottleneck, but I think because OVS can't switch packets fast enough between the virual QUEUE's, the physical interface is now the one dropping packets.

From: discuss-bounces at openvswitch.org [mailto:discuss-bounces at openvswitch.org] On Behalf Of Morgan Yang
Sent: Wednesday, October 23, 2013 6:08 PM
To: discuss at openvswitch.org
Subject: [ovs-discuss] Question in regard to packet errors with OVS

Hi All:

I am doing testing with Mellanox's CX3 40G NIC.

The setup without OVS runs about 37~38 Gbps between multiple threads of IPERF, and no errors are noticed on the Ethernet interfaces

Host A Eth4(80.1.1.1) <-> Host B Eth4(80.2.1.1)

After I introduce OVS (running 2.0.0), I still get 37~38 Gbps, but I begin to see errors  and overruns on the physical NIC

[cid:image003.png at 01CED01B.49E1C3F0]
OVSFA(80.1.1.1) <-> Host A Eth4 <-> Host B Eth4 <-> OVSFB(80.2.1.1)

root:~/mlnx# ovs-ofctl dump-ports ovsfa
OFPST_PORT reply (xid=0x2): 2 ports
  port  1: rx pkts=7262324, bytes=451601757688, drop=0, errs=363, frame=0, over=470, crc=0
           tx pkts=5624590, bytes=303750392, drop=0, errs=0, coll=0
  port LOCAL: rx pkts=5624537, bytes=303747170, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=7262379, bytes=451601761078, drop=0, errs=0, coll=0

This is only showing up on the physical port and not on the LOCAL switch port

eth4      Link encap:Ethernet  HWaddr 00:00:50:A4:5F:AC
          inet6 addr: fe80::200:50ff:fea4:5fac/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:309077941 errors:363 dropped:0 overruns:470 frame:470
          TX packets:5624597 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:467899917070 (435.7 GiB)  TX bytes:337492360 (321.8 MiB)

ovsfa Link encap:Ethernet  HWaddr 00:00:50:A4:5F:AC
          inet addr:80.1.1.1  Bcast:80.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::1452:ceff:fe82:f794/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:7262782 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5624537 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:451601898904 (420.5 GiB)  TX bytes:303747170 (289.6 MiB)

This gets really bad when I bring up multiple VM's and run an aggregated performance test

When I bring up 8 VM's, each VM will have a VNET to OVSFA. When I try to run an aggregated iperf test, I can only get an aggregated throughput of 20~21 Gbps

Examining the port stats, it seems the physical link to OVSFA interface is now a bottleneck (see port 1 stats)

OFPST_PORT reply (xid=0x2): 10 ports
  port 14: rx pkts=993698, bytes=54019132, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=1001557, bytes=26452089418, drop=0, errs=0, coll=0
  port 16: rx pkts=1018316, bytes=55415244, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=1024662, bytes=24373022960, drop=0, errs=0, coll=0
  port 10: rx pkts=467571, bytes=25466650, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=470419, bytes=11241911122, drop=0, errs=0, coll=0
  port  1: rx pkts=14665755, bytes=623148271848, drop=0, errs=85734, frame=0, over=76354, crc=1
           tx pkts=12988799, bytes=704654302, drop=0, errs=0, coll=0
  port 17: rx pkts=915354, bytes=49815016, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=919841, bytes=21658413792, drop=0, errs=0, coll=0
  port 12: rx pkts=1338285, bytes=72922342, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=1343566, bytes=27474695496, drop=0, errs=0, coll=0
  port 13: rx pkts=1338687, bytes=72961466, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=1343787, bytes=27303337788, drop=0, errs=0, coll=0
  port 11: rx pkts=277265, bytes=15097566, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=278658, bytes=6790283224, drop=0, errs=0, coll=0
  port 15: rx pkts=1015033, bytes=55206482, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=1021695, bytes=26252916396, drop=0, errs=0, coll=0
  port LOCAL: rx pkts=5624537, bytes=303747170, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=7262947, bytes=451601929606, drop=0, errs=0, coll=0

Is there anything I can do to reduce errors and overruns? It seems the QUEUE between Eth4 and the OVS interface gets congested and OVS can't switch traffic between too many VM's at a very high speed.


Much Thanks
Morgan Yang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20131024/75cb8e39/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 9673 bytes
Desc: image003.png
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20131024/75cb8e39/attachment-0002.png>


More information about the discuss mailing list