[ovs-dev] [RFC v2 PATCH 0/4] XDP offload using flow API provider

[Toshiaki Makita]

On 2020/05/05 1:24, William Tu wrote:
> On Tue, Apr 21, 2020 at 11:47:00PM +0900, Toshiaki Makita wrote:
>> This patch adds an XDP-based flow cache using the OVS netdev-offload
>> flow API provider.  When an OVS device with XDP offload enabled,
>> packets first are processed in the XDP flow cache (with parse, and
>> table lookup implemented in eBPF) and if hits, the action processing
>> are also done in the context of XDP, which has the minimum overhead.
>>
>> This provider is based on top of William's recently posted patch for
>> custom XDP load.  When a custom XDP is loaded, the provider detects if
>> the program supports classifier, and if supported it starts offloading
>> flows to the XDP program.
>>
>> The patches are derived from xdp_flow[1], which is a mechanism similar to
>> this but implemented in kernel.
>>
>>
>> * Motivation
>>
>> While userspace datapath using netdev-afxdp or netdev-dpdk shows good
>> performance, there are use cases where packets better to be processed in
>> kernel, for example, TCP/IP connections, or container to container
>> connections.  Current solution is to use tap device or af_packet with
>> extra kernel-to/from-userspace overhead.  But with XDP, a better solution
>> is to steer packets earlier in the XDP program, and decides to send to
>> userspace datapath or stay in kernel.
>>
>> One problem with current netdev-afxdp is that it forwards all packets to
>> userspace, The first patch from William (netdev-afxdp: Enable loading XDP
>> program.) only provides the interface to load XDP program, howerver users
>> usually don't know how to write their own XDP program.
>>
>> XDP also supports HW-offload so it may be possible to offload flows to
>> HW through this provider in the future, although not currently.
>> The reason is that map-in-map is required for our program to support
>> classifier with subtables in XDP, but map-in-map is not offloadable.
>> If map-in-map becomes offloadable, HW-offload of our program will also
>> be doable.
>>
>>
>> * How to use
>>
>> 1. Install clang/llvm >= 9, libbpf >= 0.0.4, and kernel >= 5.3.
>>
>> 2. make with --enable-afxdp --enable-bpf
>> --enable-bpf will generate XDP program "bpf/flowtable_afxdp.o".  Note that
>> the BPF object will not be installed anywhere by "make install" at this point.
> 
> When configure, there is a missing include, causing error due to __u64
> checking bpf/bpf_helpers.h usability... no
> checking bpf/bpf_helpers.h presence... yes
> configure: WARNING: bpf/bpf_helpers.h: present but cannot be compiled
> configure: WARNING: bpf/bpf_helpers.h:     check for missing prerequisite headers?
> 
> configure:18876: gcc -c -g -O2  conftest.c >&5
> In file included from /usr/local/include/bpf/bpf_helpers.h:5:0,
>                   from conftest.c:73:
> /usr/local/include/bpf/bpf_helper_defs.h:55:82: error: unknown type name '__u64'
>   static int (*bpf_map_update_elem)(void *map, const void *key, const void *value, __u64 flags) = (void *) 2;
>                                                                                    ^
> /usr/local/include/bpf/bpf_helper_defs.h:79:41: error: unknown type name '__u32'
>   static int (*bpf_probe_read)(void *dst, __u32 size, const void *unsafe_ptr) = (void *) 4;
> 
> I applied this to fix it:
> diff --git a/acinclude.m4 b/acinclude.m4
> index 5eeab6feb9cc..39dfce565182 100644
> --- a/acinclude.m4
> +++ b/acinclude.m4
> @@ -326,7 +326,8 @@ AC_DEFUN([OVS_CHECK_LINUX_BPF], [
>         [AC_MSG_ERROR([unable to find llc to compile BPF program])])
>   
>       AC_CHECK_HEADER([bpf/bpf_helpers.h], [],
> -      [AC_MSG_ERROR([unable to find bpf/bpf_helpers.h to compile BPF program])])
> +      [AC_MSG_ERROR([unable to find bpf/bpf_helpers.h to compile BPF program])],
> +        [#include <linux/types.h>])
>   
>       AC_CHECK_HEADER([linux/bpf.h], [],
>         [AC_MSG_ERROR([unable to find linux/bpf.h to compile BPF program])])

Oh thanks, not sure why I didn't hit this problem.

>> 3. Load custom XDP program
>> E.g.
>> $ ovs-vsctl add-port ovsbr0 veth0 -- set int veth0 options:xdp-mode=native \
>>    options:xdp-obj="path/to/ovs/bpf/flowtable_afxdp.o"
>> $ ovs-vsctl add-port ovsbr0 veth1 -- set int veth1 options:xdp-mode=native \
>>    options:xdp-obj="path/to/ovs/bpf/flowtable_afxdp.o"
>>
>> 4. Enable XDP_REDIRECT
>> If you use veth devices, make sure to load some (possibly dummy) programs
>> on the peers of veth devices.
>>
>> 5. Enable hw-offload
>> $ ovs-vsctl set Open_vSwitch . other_config:offload-driver=linux_xdp
>> $ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
>> This will starts offloading flows to the XDP program.
>>
> Thanks, I can succefully get it working...
> When applying your patch on current master, I hit a bug.
> I will send a patch to fix it.
> 
> diff --git a/lib/netdev-linux.c b/lib/netdev-linux.c
> index 40d0cc1105ea..b52071e92ec7 100644
> --- a/lib/netdev-linux.c
> +++ b/lib/netdev-linux.c
> @@ -3588,6 +3588,7 @@ const struct netdev_class netdev_internal_class = {
>   
>   #ifdef HAVE_AF_XDP
>   #define NETDEV_AFXDP_CLASS_COMMON                               \
> +    .init = netdev_afxdp_init,                                  \
>       .construct = netdev_afxdp_construct,                        \
>       .destruct = netdev_afxdp_destruct,                          \
>       .get_stats = netdev_afxdp_get_stats,                        \

Seem you have already fixed it. Thanks.

> Other part works fine.
> I'm planning to play with more rules and performance.

Thank you for trying!

Toshiaki Makita