[ovs-dev] [ovs-dev, PATCHv6] netdev-afxdp: add new netdev type for AF_XDP.

Ilya Maximets i.maximets at samsung.com
Tue Apr 30 13:14:47 UTC 2019


On 27.04.2019 16:28, William Tu wrote:
>     >  if WIN32
>     >  lib_libopenvswitch_la_SOURCES += \
>     > diff --git a/lib/dp-packet.c b/lib/dp-packet.c
>     > index 0976a35e758b..a61552f72988 100644
>     > --- a/lib/dp-packet.c
>     > +++ b/lib/dp-packet.c
>     > @@ -22,6 +22,9 @@
>     >  #include "netdev-dpdk.h"
>     >  #include "openvswitch/dynamic-string.h"
>     >  #include "util.h"
>     > +#ifdef HAVE_AF_XDP
>     > +#include "xdpsock.h"
>     > +#endif
>>     >  static void
>     >  dp_packet_init__(struct dp_packet *b, size_t allocated, enum dp_packet_source source)
>     > @@ -122,6 +125,16 @@ dp_packet_uninit(struct dp_packet *b)
>     >               * created as a dp_packet */
>     >              free_dpdk_buf((struct dp_packet*) b);
>     >  #endif
>     > +        } else if (b->source == DPBUF_AFXDP) {
>     > +#ifdef HAVE_AF_XDP
>     > +            struct dp_packet_afxdp *xpacket;
>     > +
>     > +            xpacket = dp_packet_cast_afxdp(b);
>     > +            if (xpacket->mpool) {
>     > +                umem_elem_push(xpacket->mpool, dp_packet_base(b));
>     > +            }
>     > +#endif
> 
>     Why not making the same trick as we have for DPDK few lines above?
>     i.e. wrap this part in a function like 'free_afxdp_buf' and move it
>     to the netdev-afxdp.c ? You will not need to expose so many internals
>     to generic code. dp_packet_cast_afxdp() will also be moved there along
>     with 'struct dp_packet_afxdp'.
> 
>     BTW, I hope, someday, I'll finally implement 'dp-packet-memory-provider'
>     abstraction for OVS.
> 
> 
> Hi Ilya,
> 
> Can you share more detail about this idea, dp-packet-memory-provider?
> Why do we need it?

Hi.

OVS uses way too different memory sources for 'dp-packet's. They are defined
in 'enum dp_packet_source'. Here is some list of issues (in my opinion) we
have:

* Some of dp-packet APIs are independent from the memory source, but all the
  APIs that able to alloc/free/resize depends on the memory source and includes
  all these ugly switches/ifs guarded by ifdefs to do completely different
  things on packets from different sources.

* Some specific management of memory pools resides inside implementations of
  netdevs, which is not good from the point of OVS design. This is moslty
  about managing DPDK memory pools, but the same is applicable to your umem
  memory pools. There was also attempt to implement netmap based netdev and
  there was own memory pool implementation too.

* We even have DPDK specific code in the generic datapath code to prevent
  having packets with different memory sources in a single batch.

* netdevs that requires specific memory sources (DPDK) has to reallocate and
  copy all the packets that has different source. And this is done on netdev
  level while handling send().

* 'dp_packet_new()' is able to allocate new packet only with DPBUF_MALLOC.
  This triggers the previous issue with re-allocating on send. This is really
  bad thing because 'dp_packet_new()' used by 'dp_packet_clone()'. i.e. to
  clone dp-packet with DPDK source we're reallocating it on heap and again
  re-allocating it to send to DPDK netdev.

* netdevs usually new packets with dp_packet_new() on malloced memory even
  if they have no strict requirements about memory sources.
  For example, netdev-linux is not able to receive packets to dp-packet
  with DPBUF_DPDK/AFXDP source.

* Probably, there are more issues that I don't remember right now.


Solution is to have 'dp-packet-memory-provider' abstraction layer:

* Each memory-provider will implement specific alloc/free/etc functions.

* Each netdev might have preferred memory-provider that must be used while
  sending packets to it.

* Datapath will be able to make decision about dp-packet reallocations in
  case of memory-provider mismatch on a higher level decreasing the munber
  of reallocations.

* Packet clone could take into account the memory-provider of the original
  packet with ability to reallocate from the same provider.

* We might have globally preferred memory provider for all cases where
  memory source is not important. For example, we could use dpdk memory
  provider for packets received from netdev-linux to not re-allocate them
  on send().

* Memory pools' management will be encapsulated inside specific providers.

* It might be possible even for AFXDP netdev to use DPDK mempools created
  on top of memory shared with kernel. (just an idea)


Drawbacks:

* There are always performance concerns because this will have significant
  impact on the hot path --> could produce performance issues. Extensive
  testing required.

Best regards, Ilya Maximets.


More information about the dev mailing list