[ovs-discuss] OVS-DPDK in OpenStack network node

Пономарёв Вадим velizarx at gmail.com
Thu May 25 21:08:39 UTC 2017


Hi all.

I am setting up a network node for the OpenStack cluster (mitaka) in the
classic scenario with OpenvSwitch (
https://docs.openstack.org/liberty/networking-guide/scenario-classic-ovs.html#openstack-services-network-node).
On the network node a neutron is launched. The neutron controls the
OpenFlow and creates separate namespaces for routers and dhcp servers. The
basic scheme with standard network card drivers and OVS kernel space
datapath has a low performance (300k pps with small packets). So now I'm
trying to set up a OVS-DPDK scheme. Test network node configuration:

1. OVS 2.7
2. DPDK 16.11.1 with igb_uio, 2048 hugepages (2M), 8 CPU cores isolated
3. Ubuntu 14.04

# ovs-vsctl -V
ovs-vsctl (Open vSwitch) 2.7.0
DB Schema 7.14.0

# ovsdb-server -V
ovsdb-server (Open vSwitch) 2.7.0

# /usr/src/dpdk-stable-16.11.1/tools/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio
unused=
0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio
unused=

# /usr/src/dpdk-stable-16.11.1/tools/cpu_layout.py
============================================================
Core and Socket Information (as reported by '/proc/cpuinfo')
============================================================
cores =  [0, 1, 2, 3]
sockets =  [0, 1]
       Socket 0        Socket 1
       --------        --------
Core 0 [0, 8]          [4, 12]
Core 1 [1, 9]          [5, 13]
Core 2 [2, 10]         [6, 14]
Core 3 [3, 11]         [7, 15]

# grep "isol" /boot/grub/grub.cfg  | grep `uname -r`
linux /boot/vmlinuz-4.4.0-75-generic
root=UUID=6d19b3f6-e48d-4ba3-9585-36e534bf0568
ro isolcpus=0-7 quiet consoleblank=0 nomdmonddf nomdmonisw
linux /boot/vmlinuz-4.4.0-75-generic
root=UUID=6d19b3f6-e48d-4ba3-9585-36e534bf0568
ro isolcpus=0-7 quiet consoleblank=0 nomdmonddf nomdmonisw
linux /boot/vmlinuz-4.4.0-75-generic
root=UUID=6d19b3f6-e48d-4ba3-9585-36e534bf0568
ro recovery nomodeset isolcpus=0-7

# ovs-vsctl list Open_vSwitch
_uuid               : db2fd30f-6c56-4331-adaa-20a6cac84dc1
bridges             : [f5104e4a-8466-4594-8251-03e22a7e9d62]
cur_cfg             : 105
datapath_types      : [netdev, system]
db_version          : []
external_ids        : {}
iface_types         : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient,
geneve, gre, internal, lisp, patch, stt, system, tap, vxlan]
manager_options     : []
next_cfg            : 105
other_config        : {dpdk-hugepage-dir="/mnt/huge", dpdk-init="true",
pmd-cpu-mask="0xff"}
ovs_version         : []
ssl                 : []
statistics          : {}
system_type         : []
system_version      : []

# ovs-vsctl list interface | grep -E "dpdk|affinity"
name                : "dpdk-eth2"
options             : {dpdk-devargs="0000:04:00.0", n_rxq="4"}
other_config        : {pmd-rxq-affinity="0:0,1:1,2:2,3:3"}
type                : dpdk
name                : "dpdk-eth3"
options             : {dpdk-devargs="0000:04:00.1", n_rxq="4"}
other_config        : {pmd-rxq-affinity="0:4,1:5,2:6,3:7"}
type                : dpdk

# ovs-vsctl show
db2fd30f-6c56-4331-adaa-20a6cac84dc1
    Bridge br-ex
        Port dpdk-bond
            trunks: [462, 465]
            Interface "dpdk-eth2"
                type: dpdk
                options: {dpdk-devargs="0000:04:00.0", n_rxq="4"}
            Interface "dpdk-eth3"
                type: dpdk
                options: {dpdk-devargs="0000:04:00.1", n_rxq="4"}
        Port br-ex
            Interface br-ex
                type: internal
        Port qr-test
            tag: 462
            Interface qr-test
                type: internal
        Port qg-test
            tag: 465
            Interface qg-test
                type: internal

# ip netns exec test-router0 ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
136: qg-test: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UNKNOWN group default qlen 1000
    link/ether be:56:e7:26:0c:64 brd ff:ff:ff:ff:ff:ff
    inet 10.10.0.1/24 scope global qg-test
       valid_lft forever preferred_lft forever
    inet6 fe80::bc56:e7ff:fe26:c64/64 scope link
       valid_lft forever preferred_lft forever
137: qr-test: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UNKNOWN group default qlen 1000
    link/ether 6e:e4:96:21:07:2d brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/24 scope global qr-test
       valid_lft forever preferred_lft forever
    inet6 fe80::6ce4:96ff:fe21:72d/64 scope link
       valid_lft forever preferred_lft forever

# ovs-ofctl dump-flows br-ex
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=20358.846s, table=0, n_packets=137021466,
n_bytes=22735026192, idle_age=1, priority=0 actions=NORMAL

The router currently only routes the packets from Floating IP network (
10.10.0.0/24) to tenant network (192.168.0.0/24) iptables rules are not
configured. The performance of which I was able to achieve did not exceed
what was without DPDK. With traffic 500Kpps small ICMP packets:

# ip netns exe test-router0 tcpdump -i qg-test -n -e -c 5
15:12:34.048947 e6:91:73:d1:b7:09 > be:56:e7:26:0c:64, ethertype IPv4
(0x0800), length 60: (tos 0x10, ttl 63, id 0, offset 0, flags [DF], proto
ICMP (1), length 46)
    10.10.0.2 > 192.168.0.2: ICMP echo request, id 0, seq 16766, length 26
15:12:34.048960 be:56:e7:26:0c:64 > e6:91:73:d1:b7:09, ethertype IPv4
(0x0800), length 60: (tos 0x10, ttl 64, id 827, offset 0, flags [none],
proto ICMP (1), length 46)
    192.168.0.2 > 10.10.0.2: ICMP echo reply, id 0, seq 16766, length 26
15:12:34.048948 e6:91:73:d1:b7:09 > be:56:e7:26:0c:64, ethertype IPv4
(0x0800), length 60: (tos 0x10, ttl 63, id 0, offset 0, flags [DF], proto
ICMP (1), length 46)
    10.10.0.2 > 192.168.0.2: ICMP echo request, id 0, seq 16766, length 26
15:12:34.048968 be:56:e7:26:0c:64 > e6:91:73:d1:b7:09, ethertype IPv4
(0x0800), length 60: (tos 0x10, ttl 64, id 828, offset 0, flags [none],
proto ICMP (1), length 46)
    192.168.0.2 > 10.10.0.2: ICMP echo reply, id 0, seq 16766, length 26
15:12:34.048949 e6:91:73:d1:b7:09 > be:56:e7:26:0c:64, ethertype IPv4
(0x0800), length 60: (tos 0x10, ttl 63, id 0, offset 0, flags [DF], proto
ICMP (1), length 46)
    10.10.0.2 > 192.168.0.2: ICMP echo request, id 0, seq 16766, length 26

I received such indicators:
# ip netns exe test-router0 atop -n 1
NET | network       | ipi   197166  | ipo   197252  | ipfrw 197167  | deliv
     0  |               |               |               |               |
icmpi      0  | icmpo      0  |
NET | qg-test 900%  | pcki  *187649*  | pcko       1  | sp   10 Mbps  | si
  90 Mbps  | so    0 Kbps  | mlti       0  | erri       0  | erro       0
 | drpi       0  | drpo       0  |
NET | qr-test  46%  | pcki    9511  | pcko    *9639*  | sp   10 Mbps  | si
4565 Kbps  | so 4627 Kbps  | mlti       0  | erri       0  | erro       0
 | drpi       0  | drpo  178095  |

That is, in the namespace of the router received only ~190Kpps of 500Kpps.

Questions:
1. The reason for such packet loss is that the namespace of the router
works through kernel space and OVS copy packets?
2. Is there support for another type of interface that can be assigned to
the router in its namespace?
3. And the theoretical question: does such a scheme have a right to life?
Or are there better options for this task?

Thanks for the help!

-- 
Best regards,
Vadim Ponomarev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20170526/059b86d6/attachment-0001.html>


More information about the discuss mailing list