[ovs-dev] [PATCH 0/2] Stop configuring '--socket-mem'/'--socket-limit' by default for DPDK if not requested.

[David Marchand]

On Thu, Jul 1, 2021 at 7:46 PM Ilya Maximets <i.maximets at ovn.org> wrote:
>
> On 6/30/21 10:48 PM, Rosemarie O'Riorden wrote:
> > From: Rosemarie O'Riorden <roriorde at redhat.com>
> >
> > Currently, there is a default value of 1024 for socket-mem if not
> > configured. socket-limit automatically takes on the value of socket-mem
> > unless otherwise specified. With these changes, memory allocation will
> > be dynamically managed by DPDK, meaning that by default,  no memory will
> > be pre-allocated on startup, and there will be no limit to how much
> > memory can be used. Either or both of these values can be set by the
> > user.
> >
> > The EAL arguments will look like this:
> >
> > - dpdk-socket-mem=<not set>, dpdk-socket-limit=<not set>
> >   current: "--scket-mem=1024,1024 --socket-limit=1024,1024"
> >   patch 1: ""
> >   patch 2: ""
> >
> > - dpdk-socket-mem=<MEM>, dpdk-socket-limit=<not set>
> >   current: "--scket-mem=MEM --socket-limit=MEM"
> >   patch 1: "--scket-mem=MEM --socket-limit=MEM"
> >   patch 2: "--scket-mem=MEM"
> >
> > - dpdk-socket-mem=<not set>, dpdk-socket-limit=<LIMIT>
> >   current: "--scket-mem=1024,1024 --socket-limit=LIMIT"
> >   patch 1: "--socket-limit=LIMIT"
> >   patch 2: "--socket-limit=LIMIT"
> >
> > - dpdk-socket-mem=<MEM>, dpdk-socket-limit=<LIMIT>
> >   current: "--scket-mem=MEM --socket-limit=LIMIT"
> >   patch 1: "--scket-mem=MEM --socket-limit=LIMIT"
> >   patch 2: "--scket-mem=MEM --socket-limit=LIMIT"
> >
> > Rosemarie O'Riorden (2):
> >   dpdk: Remove default values for socket-mem and limit.
> >   dpdk: Stop configuring socket-limit with the value of socket-mem.
> >
> >  Documentation/intro/install/dpdk.rst |  3 +--
> >  NEWS                                 |  4 ++++
> >  lib/dpdk.c                           |  6 +-----
> >  vswitchd/vswitch.xml                 | 13 ++++++-------
> >  4 files changed, 12 insertions(+), 14 deletions(-)
> >
>
> Hi, Ian, everyone.
>
> This is the series I barely mentioned on yesterday's public
> meeting.  It seems like a good thing to have in 2.16 release,
> but we definitely need an input from different sides on this
> kind of small, but user-visible change.
>
> In general, I think, it's good to stop configuring these
> unnecessary defaults that only limit users.  1GB of memory is
> barely enough for a very small setup and, most likely, most
> of users are setting dpdk-socket-mem anyway and doesn't rely
> on the default option (AFAICT, Michael is checking this with
> OpenStack folks).  In any case, the upgrade path should not
> be hard, as it's enough to just set current values to the
> database before OVS upgrade.

Afaik, setting memory configuration is mandatory in OSP.
Here, the changes mean that OSP would have to set a limit too, as they
were relying on this so far (some users are pick about how much memory
is available for the rest of the system and for vms).

But otherwise the current parameter changes seems ok.


>
> Historically, it was there just to allow OVS run with DPDK
> "out-of-the-box" with a very minimal manual configuration,
> but current DPDK works without any memory configuration, so
> the original purpose of these defaults is not relevant anymore.

Just a warning.

Explicitely setting the memory configuration gives determinism.
Hugepages allocations are done once at EAL init, and then OVS picks
into them for building mempools and for rte_malloc's.
As long as those operations are done in control path, this is not
really a concern.

But iirc, we have rte_malloc calls in the datapath for TSO support
(external buffer) and here, it means syscalls in the datapath when
growing "DPDK heap".
I agree, this might not happen often with 1G hugepages and once
reaching a steady state.


-- 
David Marchand