[ovs-dev] [PATCH 0/2] Stop configuring '--socket-mem'/'--socket-limit' by default for DPDK if not requested.

[Aaron Conole]

David Marchand <david.marchand at redhat.com> writes:

> On Thu, Jul 1, 2021 at 7:46 PM Ilya Maximets <i.maximets at ovn.org> wrote:
>>
>> On 6/30/21 10:48 PM, Rosemarie O'Riorden wrote:
>> > From: Rosemarie O'Riorden <roriorde at redhat.com>
>> >
>> > Currently, there is a default value of 1024 for socket-mem if not
>> > configured. socket-limit automatically takes on the value of socket-mem
>> > unless otherwise specified. With these changes, memory allocation will
>> > be dynamically managed by DPDK, meaning that by default,  no memory will
>> > be pre-allocated on startup, and there will be no limit to how much
>> > memory can be used. Either or both of these values can be set by the
>> > user.
>> >
>> > The EAL arguments will look like this:
>> >
>> > - dpdk-socket-mem=<not set>, dpdk-socket-limit=<not set>
>> >   current: "--scket-mem=1024,1024 --socket-limit=1024,1024"
>> >   patch 1: ""
>> >   patch 2: ""
>> >
>> > - dpdk-socket-mem=<MEM>, dpdk-socket-limit=<not set>
>> >   current: "--scket-mem=MEM --socket-limit=MEM"
>> >   patch 1: "--scket-mem=MEM --socket-limit=MEM"
>> >   patch 2: "--scket-mem=MEM"
>> >
>> > - dpdk-socket-mem=<not set>, dpdk-socket-limit=<LIMIT>
>> >   current: "--scket-mem=1024,1024 --socket-limit=LIMIT"
>> >   patch 1: "--socket-limit=LIMIT"
>> >   patch 2: "--socket-limit=LIMIT"
>> >
>> > - dpdk-socket-mem=<MEM>, dpdk-socket-limit=<LIMIT>
>> >   current: "--scket-mem=MEM --socket-limit=LIMIT"
>> >   patch 1: "--scket-mem=MEM --socket-limit=LIMIT"
>> >   patch 2: "--scket-mem=MEM --socket-limit=LIMIT"
>> >
>> > Rosemarie O'Riorden (2):
>> >   dpdk: Remove default values for socket-mem and limit.
>> >   dpdk: Stop configuring socket-limit with the value of socket-mem.
>> >
>> >  Documentation/intro/install/dpdk.rst |  3 +--
>> >  NEWS                                 |  4 ++++
>> >  lib/dpdk.c                           |  6 +-----
>> >  vswitchd/vswitch.xml                 | 13 ++++++-------
>> >  4 files changed, 12 insertions(+), 14 deletions(-)
>> >
>>
>> Hi, Ian, everyone.
>>
>> This is the series I barely mentioned on yesterday's public
>> meeting.  It seems like a good thing to have in 2.16 release,
>> but we definitely need an input from different sides on this
>> kind of small, but user-visible change.
>>
>> In general, I think, it's good to stop configuring these
>> unnecessary defaults that only limit users.  1GB of memory is
>> barely enough for a very small setup and, most likely, most
>> of users are setting dpdk-socket-mem anyway and doesn't rely
>> on the default option (AFAICT, Michael is checking this with
>> OpenStack folks).  In any case, the upgrade path should not
>> be hard, as it's enough to just set current values to the
>> database before OVS upgrade.
>
> Afaik, setting memory configuration is mandatory in OSP.
> Here, the changes mean that OSP would have to set a limit too, as they
> were relying on this so far (some users are pick about how much memory
> is available for the rest of the system and for vms).
>
> But otherwise the current parameter changes seems ok.
>
>
>>
>> Historically, it was there just to allow OVS run with DPDK
>> "out-of-the-box" with a very minimal manual configuration,
>> but current DPDK works without any memory configuration, so
>> the original purpose of these defaults is not relevant anymore.
>
> Just a warning.
>
> Explicitely setting the memory configuration gives determinism.
> Hugepages allocations are done once at EAL init, and then OVS picks
> into them for building mempools and for rte_malloc's.
> As long as those operations are done in control path, this is not
> really a concern.
>
> But iirc, we have rte_malloc calls in the datapath for TSO support
> (external buffer) and here, it means syscalls in the datapath when
> growing "DPDK heap".
> I agree, this might not happen often with 1G hugepages and once
> reaching a steady state.

I guess it will be important to inform OSP folks about this change.
There's no reason OSP cannot also start configuring the socket-limit
already (since the code will use whatever OSP configured rather than
providing a value).

CC'd Assaf, Terry, and Jakub.