[ovs-dev] [PATCH v5 0/5] OVS-DPDK flow offload with rte_flow

Thu Jan 18 08:38:19 UTC 2018

Yes, I have done review on this patch and gave comments back on it a while ago.
I also have done a lot of testing as a result of the performance analysis I made for the PV and PVP cases.
These tests includes full integration of the feature. Meaning the hw-offload follows the add/deletion of 
flows in the megaflow cache. So, add and deletion I a single/multi-queue setup has been tested.
Furthermore, I'm working on this patchset for continued development, so it is continuously being 
used by me as a basic functionality.
So far, it has been running stable.

Finn

>-----Original Message-----
>From: Yuanhan Liu [mailto:yliu at fridaylinux.org]
>Sent: 18. januar 2018 04:39
>To: Ian Stokes <ian.stokes at intel.com>
>Cc: Finn Christensen <fc at napatech.com>; dev at openvswitch.org; Darrell Ball
><dball at vmware.com>; Chandran Sugesh <sugesh.chandran at intel.com>;
>Simon Horman <simon.horman at netronome.com>; Thomas Monjalon
><thomas at monjalon.net>; Olga Shern <olgas at mellanox.com>; Shahaf Shuler
><shahafs at mellanox.com>
>Subject: Re: [PATCH v5 0/5] OVS-DPDK flow offload with rte_flow
>
>I was told more tests and reviews are needed from yesterday's meeting.
>
>For testing, as you can see below, Finn already have done some performance
>testing covering quite few testcases. I have also done some very basic
>performance testing, basically just with rxonly.
>
>Besides that, I have actually spent a lot of effort for doing functional testing,
>to make sure it doesn't break anything. The limited testing
>including:
>
>- 1 queue
>- 2 queues
>- 1 flow
>- 1000 flows
>- adding/deleting flows without traffic
>- adding/deleting flows with traffic
>- keep adding and then deleting flows with traffic
>
>For review, I think Finn from Napatech should also have done it. He has
>provided some valuable comments after all :)
>
>Thanks.
>
>	--yliu
>
>On Fri, Dec 22, 2017 at 11:28:45AM +0000, Finn Christensen wrote:
>> Hi,
>>
>> The addition of the offload thread is an great addition, furthermore, RSS
>action usage for target queues, works now well with our NICs, I just had to do
>some additional drive work, to make it all work.
>>
>> The support for RSS removed the problem with rte-flow queue selection,
>but RSS in general added an additional challenge as now multiple pmd's may
>request the same flow offload (as Yuanhan pointed out). The solution has
>worked without issues for me. I have run tests with 1 and 2 queues in a PV
>and PVP setup going over virtio.
>> Yuanhan mentioned tests in a P2P setup with high throughput acceleration. I
>have focused on a PVP scenario, which, in addition, will crosses virtio, and
>most importantly, has acceleration in Rx direction only (seen from VM). It still
>gives fairly good performance improvements.
>>
>> Here are my initial test results.
>>
>> Percentage improvements:
>> 1 RX/Tx queue:
>> 1     megaflow - 10K flowsPV: 73%
>> 10  megaflows - 100K flowsPVP: 50%PV: 56%
>> 100 megaflows - 1M flowsPVP: 42%PV: 49%
>> 1000 megaflows - 10M flowsPVP: 40%PV: 42%
>>
>> With RSS: 2 Rx/Tx queue pairs:
>> 1     megaflow - 10K flowsPV: 55%
>> 10  megaflows - 100K flowsPVP: 36%PV: 38%
>> 100 megaflows - 1M flowsPVP: 27%PV: 24%
>> 1000 megaflows - 10M flowsPVP: 21%PV: 24%
>>
>> PVP for 1 megaflow offload is omitted because my VM-app imposed a
>bottle-neck.
>>
>> Regards,
>> Finn
>>
>>     -----Original Message-----
>>     From: Yuanhan Liu [mailto:yliu at fridaylinux.org]
>>     Sent: 20. december 2017 15:45
>>     To: dev at openvswitch.org
>>     Cc: Finn Christensen <fc at napatech.com>; Darrell Ball
>>     <dball at vmware.com>; Chandran Sugesh <sugesh.chandran at intel.com>;
>>     Simon Horman <simon.horman at netronome.com>; Ian Stokes
>>     <ian.stokes at intel.com>; Yuanhan Liu <yliu at fridaylinux.org>
>>     Subject: [PATCH v5 0/5] OVS-DPDK flow offload with rte_flow
>>
>>     Hi,
>>
>>     Here is a joint work from Mellanox and Napatech, to enable the flow hw
>>     offload with the DPDK generic flow interface (rte_flow).
>>
>>     The basic idea is to associate the flow with a mark id (a unit32_t number).
>>     Later, we then get the flow directly from the mark id, which could bypass
>>     some heavy CPU operations, including but not limiting to mini flow extract,
>>     emc lookup, dpcls lookup, etc.
>>
>>     The association is done with CMAP in patch 1. The CPU workload
>bypassing
>>     is done in patch 2. The flow offload is done in patch 3, which mainly does
>>     two things:
>>
>>     - translate the ovs match to DPDK rte flow patterns
>>     - bind those patterns with a RSS + MARK action.
>>
>>     Patch 5 makes the offload work happen in another thread, for leaving the
>>     datapath as light as possible.
>>
>>     A PHY-PHY forwarding with 1000 mega flows (udp,tp_src=1000-1999) and 1
>>     million streams (tp_src=1000-1999, tp_dst=2000-2999) show more than
>>     260% performance boost.
>>
>>     Note that it's disabled by default, which can be enabled by:
>>
>>         $ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
>>
>>
>>     v5: - fixed an issue that it took too long if we do flow add/remove
>>           repeatedly.
>>         - removed an unused mutex lock
>>         - turned most of the log level to DBG
>>         - rebased on top of the latest code
>>
>>     v4: - use RSS action instead of QUEUE action with MARK
>>         - make it work with multiple queue (see patch 1)
>>         - rebased on top of latest code
>>
>>     v3: - The mark and id association is done with array instead of CMAP.
>>         - Added a thread to do hw offload operations
>>         - Removed macros completely
>>         - dropped the patch to set FDIR_CONF, which is a workround some
>>           Intel NICs.
>>         - Added a debug patch to show all flow patterns we have created.
>>         - Misc fixes
>>
>>     v2: - workaround the queue action issue
>>         - fixed the tcp_flags being skipped issue, which also fixed the
>>           build warnings
>>         - fixed l2 patterns for Intel nic
>>         - Converted some macros to functions
>>         - did not hardcode the max number of flow/action
>>         - rebased on top of the latest code
>>
>>     Thanks.
>>
>>         --yliu
>>
>>
>>     ---
>>     Finn Christensen (1):
>>       netdev-dpdk: implement flow offload with rte flow
>>
>>     Yuanhan Liu (4):
>>       dpif-netdev: associate flow with a mark id
>>       dpif-netdev: retrieve flow directly from the flow mark
>>       netdev-dpdk: add debug for rte flow patterns
>>       dpif-netdev: do hw flow offload in a thread
>>
>>      lib/dp-packet.h   |  13 +
>>      lib/dpif-netdev.c | 473 ++++++++++++++++++++++++++++++++++-
>>      lib/flow.c        | 155 +++++++++---
>>      lib/flow.h        |   1 +
>>      lib/netdev-dpdk.c | 732
>>     +++++++++++++++++++++++++++++++++++++++++++++++++++++-
>>      lib/netdev.h      |   6 +
>>      6 files changed, 1341 insertions(+), 39 deletions(-)
>>
>>     --
>>     2.7.4
>>
>> Disclaimer: This email and any files transmitted with it may contain
>confidential information intended for the addressee(s) only. The information
>is not to be surrendered or copied to unauthorized persons. If you have
>received this communication in error, please notify the sender immediately
>and delete this e-mail from your system.