[ovs-dev] [PATCH 00/11] RFC: Improvements in upcall processing code path.

Jarno Rajahalme jarno.rajahalme at nsn.com
Mon Feb 11 14:46:16 UTC 2013


This series tries to improve the kernel datapath - ofproto upcall processing
code path.  Some of these are local changes that may be easier to adopt,
while others will require more discussion (i.e. modifications to the kernel
interface).

'make check' passes at each stage, but otherwise I have tested these only as
described below.

Summary: The share of packets reaching the userspace flow table doubles.
By far the most of the improvement comes from netlink upcall request pooling,
which is a change to the dpif-provider interface.

Note: The test described below almost exclusively runs on the "without_facet"
code path, so it does not measure the benefit for kernel flow setup.

Test protocol:
- 'udpping' sends 10000000 UDP packets with a unique source and destination
  port pair to br0 in bursts of 97 packets.  Between bursts it nanosleeps
  for 50 nanoseconds.  This results into ~80% CPU (core) utilization.
- Start test when CPU >96.5% idle
- Collect 5 flow dumps after runs during which the ovs-vswitchd process
  remains >97.5% CPU utilization throughout the test.
- Keep 3 best out of 5 collected for each test case.
- Compute percentage of mean packets processed by the flow entry.

Test system:
Linux debian-jr 3.2.0-4-amd64 #1 SMP Debian 3.2.35-2 x86_64 GNU/Linux
2GB VM on VirtualBox 4.2.6
Host: Intel Core2 Duo 2.93 GHz OSX 10.6.8

In 1st shell:

# rmmod openvswitch
# ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,manager_options --bootstrap-ca-cert=db:SSL,ca_cert --pidfile --detach
# insmod datapath/linux/openvswitch.ko
# ovs-vsctl --no-wait del-br br0
# ovs-vsctl --no-wait add-br br0
# chrt 99 ovs-vswitchd --pidfile -verr

In 2nd shell:

# ovs-ofctl del-flows br0
# ovs-ofctl add-flow br0 "in_port=LOCAL actions="
# nice -n -5 ./udpping

Results:

---

ovs git master (84507691a510efa76a0f063ee04a06cf19081ff1):

n_packets=3103133
n_packets=3085670
n_packets=3274467

Mean: 3154423 = 31.5 %

---

01/11: dpif: Take upcall netlink pooling down to the system call interface.

n_packets=5540243
n_packets=5804791
n_packets=5782079

Mean: 5709038 = 57.1 %

---

02/11: netlink-socket: Use recvmmsg() system call with kernels since 2.6.32.

Note: This may be Linux version dependent.

n_packets=5568893
n_packets=5897427
n_packets=5893462

Mean: 5786594 = 57.9 %

---

03/11: datapath: Fold key_len and hash fields into struct sw_flow_key.

Note: This patch saves unnecessary hash calculations on kernel datapath flow
setup. This test does not stress that code path.

n_packets=5946011
n_packets=5855520
n_packets=5936731

Mean: 5912754 = 59.1 %

---

06/11: ofproto-dpif: Deferred setting of packet layer pointers.

n_packets=6053026
n_packets=6005442
n_packets=5997440

Mean: 6018636 = 60.2 %

---

09/11: datapath: Pass more packet metadata to userspace.
(dpif-linux: Use kernel provided layer pointers)

n_packets=5937665
n_packets=6354151
n_packets=6195069

Mean: 6162295 = 61.6 %

---

10/11: ofproto-dpif: Use kernel provided flow hash in upcall pooling.

Note: Hashes are also used when creating facets and subfacets. This test does
not stress that code path.

n_packets=6430066
n_packets=6416697
n_packets=6071671

Mean: 6306144 = 63.1 %

---

Jarno Rajahalme (11):
  dpif: Pool dpif_recv() calls.
  netlink-socket: Use recvmmsg() with kernels since 2.6.32.
  datapath: Fold key_len and hash into sw_flow_key.
  lib/flow: Split flow_extract().
  ofproto-dpif: Collect TCP flags only when needed.
  ofproto-dpif: Deferred setting of packet layer pointers.
  lib/packets: Remove the unused l7 setting for ICMP.
  dpif: Prepare for additional kernel provided packet metadata.
  datapath: Pass more packet metadata to userspace.
  ofproto-dpif: Use kernel provided flow hash in upcall pooling.
  dpif-netdev: pass layer pointers and key hash like dpif-linux.

 datapath/datapath.c         |   61 ++++++++------
 datapath/flow.c             |  176 +++++++++++++++++++++------------------
 datapath/flow.h             |   29 ++++---
 datapath/vport-vxlan.c      |    2 +-
 include/linux/openvswitch.h |   10 +++
 lib/dpif-linux.c            |  161 +++++++++++++++++++++++++-----------
 lib/dpif-netdev.c           |   56 ++++++++++---
 lib/dpif-provider.h         |   32 +++++---
 lib/dpif.c                  |   57 ++++++++-----
 lib/dpif.h                  |    6 +-
 lib/flow.c                  |   42 +++++++---
 lib/flow.h                  |    1 +
 lib/netlink-socket.c        |   96 ++++++++++++++++++++++
 lib/netlink-socket.h        |   11 +++
 lib/odp-util.c              |   16 ++++
 lib/odp-util.h              |    3 +-
 lib/packets.c               |   44 +++++-----
 lib/packets.h               |   23 +++++-
 ofproto/in-band.c           |   30 ++++---
 ofproto/ofproto-dpif.c      |  192 ++++++++++++++++++++++++++-----------------
 20 files changed, 720 insertions(+), 328 deletions(-)

-- 
1.7.10.4




More information about the dev mailing list