[ovs-dev] [RFC 0/4] Status of mmap upcall work

Thomas Graf tgraf at suug.ch
Thu May 22 23:15:57 UTC 2014

A quick status update and request for comments on the future
direction of this work.

Changes since last post (tl;dr)
  * Reduced ring size from 4M to 1M
  * Added proper handling for socket errors (EPOLLERR)
  * More benchmarking

Numbers, first:

Test: * Avg pps as reported by pktgen for a single pktgen thread
      * qemu, vcpu=2, mem=2GB, megaflows disabled

                 Orig [0]  ZC [1]    mmap [2]  shared-ofpbuf [3]
 64B (linear)    624K      564K      962K      900K
1.5K (linear)    612K      629K      800K      891K
  9K (linear)    476K      474K      508K      513K
  9K (4 frags)   365K      411K      503K      631K

[0] git master branch modified to disable zerocopy
[1] git master branch (18a6a6cd9)
    A single copy is avoided when constructing message in the
[2] Patch 1-3 applied
    Separate shared memory based rx/tx ring buffer is used to
    carry upcall messages. ovs-vswitchd copies data off ring.
[3] Patch 4 applied
    ovs-vswitchd reuses the shared memory and avoid another
Numbers on ring buffer size:
  ring-size:  512K    1M    2M    4M
        pps:  596K  914K  926K  900K

Q1: The gains are significant and can be optimized further. But even
    with the ring size reduced from 8Mb to 2MB per socket (rx+tx),
    the resource consumption will not allow for a separate ring for
    every port * handler.

      1. Reuse same ring for multiple vport
      2. Require manual enablement per port
Q2: How have you been testing fairness so far?

Thoughts? Comments?

Thomas Graf (4):
  netlink: Support for memory mapped Netlink sockets
  nlmon: Enable memory mapped Netlink socket
  upcall: Enable memory mapped Netlink sockets
  netlink: COW mmap Netlink messages to avoid memcpy()

 lib/dpif-linux.c       |   7 +-
 lib/netlink-protocol.h |  39 ++++++
 lib/netlink-socket.c   | 369 ++++++++++++++++++++++++++++++++++++++++++++-----
 lib/netlink-socket.h   |  10 +-
 lib/ofpbuf.c           |   6 +
 lib/ofpbuf.h           |   2 +
 utilities/nlmon.c      |   2 +-
 7 files changed, 396 insertions(+), 39 deletions(-)


More information about the dev mailing list