[ovs-dev] Statistics Collection Performance Impact

[Simon Horman]

Hi,

I have been doing some work with the people at Midokura on evaluating
the performance of Open vSwtich. At this time we are focusing on the
rate at which flows can be created.

A key observation that has been made so far is that the collection of
statistics has a significant impact on flow create rate once the number
of flows becomes large, where large is somewhere around 500k. This problem
was alluded to in an earlier email thread on the dev at openvswitch.org list
in which Ben Pfaff outlined O(n) areas of the OVS code.

http://www.mail-archive.com/dev@openvswitch.org/msg03242.html

The attached graph illustrates the problem.

* "1000ms Facet Stats" is the current behaviour of Open vSwitch
  The flat spots are where update_stats() is running.

  This is the baseline performance.

  In my test environment it is not possible to add more than about 900k
  flows.

* "5000ms Facet Stats" extends the interval between calls to
  update_stats() from 1000ms to 5000ms by having expire() return
  a maximum value of 5000 rather than 1000.

  In this case there is a significant increase in the rate at which
  flows are created.

* "No Facet Stats or Expiry" disables stats collection. It also
  disables expiry of facets as that no longer works correctly.

  This ideal performance (though with significant loss of functionality)

  In my test environment I have been able to add more than 10M flows.

Although a very simple and possibly naïve approach, I wonder if
dynamically extending the interval at which statistics are collected is
a worthwhile approach to mitigating the performance impact of statistics
collection.


---

The test I ran involves using the in-kernel packet generator to
generate 768k flows. Packets for each flow are sent in turn in
a round-robin fashion. 128 byte packets are sent at a rate of
450,000 packets/s.

I am using Open vSwitch 1.2.2 with the following patches applied:

* [PATCH] datapath: Allow table to expand to have TBL_MAX_BUCKETS buckets
* [PATCH] datapath: Allow the number of hash entries to exceed TBL_MAX_BUCKETS
* [PATCH] [patch v2] ovs-dpctl: Show number of flows

And the following run-time tweaks.

ovs-vsctl set bridge br3 other-config:flow-eviction-threshold=1000000
ovs-ofctl add-flow br3 "in_port=1 ip nw_dst=10.0.0.0/8 idle_timeout=0 action=output:2"

Port 2 of br3 is a dummy network device. The link of the device is down.
The other device enslaved to br3 is an intel 82576 gigabit network device
which receives packets from another host running the packet generator.

Both machines have 2 x Intel E5520 processors @ 2.27GHz.
Each processor has 4 cores. Each core has 2 threads.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: avg.pdf
Type: application/pdf
Size: 74242 bytes
Desc: not available
URL: <http://mail.openvswitch.org/pipermail/ovs-dev/attachments/20111107/f8ada6ca/attachment-0004.pdf>