[ovs-dev] [PATCH v5 0/5] OVS-DPDK flow offload with rte_flow

Fri Dec 22 11:28:45 UTC 2017

Hi,

The addition of the offload thread is an great addition, furthermore, RSS action usage for target queues, works now well with our NICs, I just had to do some additional drive work, to make it all work.

The support for RSS removed the problem with rte-flow queue selection, but RSS in general added an additional challenge as now multiple pmd's may request the same flow offload (as Yuanhan pointed out). The solution has worked without issues for me. I have run tests with 1 and 2 queues in a PV and PVP setup going over virtio. 
Yuanhan mentioned tests in a P2P setup with high throughput acceleration. I have focused on a PVP scenario, which, in addition, will crosses virtio, and most importantly, has acceleration in Rx direction only (seen from VM). It still gives fairly good performance improvements.

Here are my initial test results.

Percentage improvements:
1 RX/Tx queue:
1     megaflow - 10K flows		PV: 73%
10  megaflows - 100K flows	PVP: 50%	PV: 56%
100 megaflows - 1M flows	PVP: 42%	PV: 49%
1000 megaflows - 10M flows	PVP: 40%	PV: 42%

With RSS: 2 Rx/Tx queue pairs:
1     megaflow - 10K flows		PV: 55%
10  megaflows - 100K flows	PVP: 36%	PV: 38%
100 megaflows - 1M flows	PVP: 27%	PV: 24%
1000 megaflows - 10M flows	PVP: 21%	PV: 24%

PVP for 1 megaflow offload is omitted because my VM-app imposed a bottle-neck.

Regards,
Finn

    -----Original Message-----
    From: Yuanhan Liu [mailto:yliu at fridaylinux.org]
    Sent: 20. december 2017 15:45
    To: dev at openvswitch.org
    Cc: Finn Christensen <fc at napatech.com>; Darrell Ball
    <dball at vmware.com>; Chandran Sugesh <sugesh.chandran at intel.com>;
    Simon Horman <simon.horman at netronome.com>; Ian Stokes
    <ian.stokes at intel.com>; Yuanhan Liu <yliu at fridaylinux.org>
    Subject: [PATCH v5 0/5] OVS-DPDK flow offload with rte_flow

    Hi,

    Here is a joint work from Mellanox and Napatech, to enable the flow hw
    offload with the DPDK generic flow interface (rte_flow).

    The basic idea is to associate the flow with a mark id (a unit32_t number).
    Later, we then get the flow directly from the mark id, which could bypass
    some heavy CPU operations, including but not limiting to mini flow extract,
    emc lookup, dpcls lookup, etc.

    The association is done with CMAP in patch 1. The CPU workload bypassing
    is done in patch 2. The flow offload is done in patch 3, which mainly does
    two things:

    - translate the ovs match to DPDK rte flow patterns
    - bind those patterns with a RSS + MARK action.

    Patch 5 makes the offload work happen in another thread, for leaving the
    datapath as light as possible.

    A PHY-PHY forwarding with 1000 mega flows (udp,tp_src=1000-1999) and 1
    million streams (tp_src=1000-1999, tp_dst=2000-2999) show more than
    260% performance boost.

    Note that it's disabled by default, which can be enabled by:

        $ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true

    v5: - fixed an issue that it took too long if we do flow add/remove
          repeatedly.
        - removed an unused mutex lock
        - turned most of the log level to DBG
        - rebased on top of the latest code

    v4: - use RSS action instead of QUEUE action with MARK
        - make it work with multiple queue (see patch 1)
        - rebased on top of latest code

    v3: - The mark and id association is done with array instead of CMAP.
        - Added a thread to do hw offload operations
        - Removed macros completely
        - dropped the patch to set FDIR_CONF, which is a workround some
          Intel NICs.
        - Added a debug patch to show all flow patterns we have created.
        - Misc fixes

    v2: - workaround the queue action issue
        - fixed the tcp_flags being skipped issue, which also fixed the
          build warnings
        - fixed l2 patterns for Intel nic
        - Converted some macros to functions
        - did not hardcode the max number of flow/action
        - rebased on top of the latest code

    Thanks.

        --yliu

    ---
    Finn Christensen (1):
      netdev-dpdk: implement flow offload with rte flow

    Yuanhan Liu (4):
      dpif-netdev: associate flow with a mark id
      dpif-netdev: retrieve flow directly from the flow mark
      netdev-dpdk: add debug for rte flow patterns
      dpif-netdev: do hw flow offload in a thread

     lib/dp-packet.h   |  13 +
     lib/dpif-netdev.c | 473 ++++++++++++++++++++++++++++++++++-
     lib/flow.c        | 155 +++++++++---
     lib/flow.h        |   1 +
     lib/netdev-dpdk.c | 732
    +++++++++++++++++++++++++++++++++++++++++++++++++++++-
     lib/netdev.h      |   6 +
     6 files changed, 1341 insertions(+), 39 deletions(-)

    --
    2.7.4