[ovs-dev] OVS-DPDK public meeting
Ilya Maximets
i.maximets at samsung.com
Fri Nov 16 10:10:12 UTC 2018
>
> On 15 Nov 2018, at 15:18, Kevin Traynor wrote:
>
>> Next meeting 12th December
>> (May cancel if no agenda items as DPDK/OVS confs the previous week)
>>
>> Minutes for 14th November.
>> Attendees: Pieter, Asaf, Timothy, Ophir, Ian, Thomas, David, Matteo,
>> Johann, Aaron, Tiago, Kevin.
>>
>> =======
>> GENERAL
>> =======
>> - Ian is one of the core committers :-)
>> -- He will still use his private branch for validation
>> -- Patches will be applied directly into main tree
>> -- i.e. no more pull requests
>>
>> ====================
>> PERFORMANCE/FEATURES
>> ====================
>> - DPDK 18.08/18.11 update (Kevin)
>> -- sent patches for update to 18.11
>> -- Ian found an issue with CRC strip, to be fixed
>> -- Ophir also sent a patch prior to meeting
>> -- Thomas M, explained some of the new features of hotplug
>> -- Discussed integration of changes and if it should be broken up
>> into:
>> -- 1. Minimal update to use 18.11
>> -- 2. Extend OVS to take advantage of new features in 18.11
>> -- Discussion to be continued on ML
>>
>> - CI lab (Ophir/Aaron/Ian)
>> -- Not much changed
>> -- Testing done in VM
>> -- Mlx to check about Hw for Aaron
>>
>> - 17.11.4 (Eelco/Aaron)
>> -- Slow start up issue reported because of some patches in 17.11.4
>> -- Alejandro sent patch for DPDK 17.11 branch
>> -- https://mails.dpdk.org/archives/dev/2018-November/118770.html
>> -- Eelco will add some more details
>
> This issue got introduced due to commit 293c0c4b9. This change will try
> to map the huge page memory we need starting at 0x100000000. If it fails
> the hint, it will retry increasing the address, until it gets the
> mapping at the requested address. This to minimise the chance to map it
> outside the HWs DMA address space.
>
> However just before doing this DPDK also mmaps each individual huge page
> in the system. It starts low, but it will collide with the 0x100000000
> address.
>
> When this happens there a quite some retries until we find the mmaps we
> want. So the more memory in huge pages, the more time it takes. It
> depends on the number of huge pages in the system, not the memory
> configured for OVS.
>
> FYI this mapping on a 32G huge page system:
> 40000000-80000000 rw-s 00000000 00:23 206083
> /dev/hugepages/rtemap_1
> 80000000-c0000000 rw-s 00000000 00:23 206084
> /dev/hugepages/rtemap_2
> c0000000-100000000 rw-s 00000000 00:23 206085
> /dev/hugepages/rtemap_3
> 100000000-140000000 rw-s 00000000 00:23 206086
> /dev/hugepages/rtemap_4
> 140000000-180000000 rw-s 00000000 00:23 206087
> /dev/hugepages/rtemap_5
> 180000000-1c0000000 rw-s 00000000 00:23 206088
> /dev/hugepages/rtemap_6
> 1c0000000-200000000 rw-s 00000000 00:23 206089
> /dev/hugepages/rtemap_7
> 200000000-240000000 rw-s 00000000 00:23 206090
> /dev/hugepages/rtemap_8
> 240000000-280000000 rw-s 00000000 00:23 206091
> /dev/hugepages/rtemap_9
> 280000000-2c0000000 rw-s 00000000 00:23 206092
> /dev/hugepages/rtemap_10
> 2c0000000-300000000 rw-s 00000000 00:23 206093
> /dev/hugepages/rtemap_11
> 300000000-340000000 rw-s 00000000 00:23 206094
> /dev/hugepages/rtemap_12
> 340000000-380000000 rw-s 00000000 00:23 206095
> /dev/hugepages/rtemap_13
> 380000000-3c0000000 rw-s 00000000 00:23 206096
> /dev/hugepages/rtemap_14
> 3c0000000-400000000 rw-s 00000000 00:23 206097
> /dev/hugepages/rtemap_15
> 400000000-440000000 rw-s 00000000 00:23 206098
> /dev/hugepages/rtemap_16
> 440000000-480000000 rw-s 00000000 00:23 206099
> /dev/hugepages/rtemap_17
> 480000000-4c0000000 rw-s 00000000 00:23 206100
> /dev/hugepages/rtemap_18
> 4c0000000-500000000 rw-s 00000000 00:23 206101
> /dev/hugepages/rtemap_19
> 500000000-540000000 rw-s 00000000 00:23 206102
> /dev/hugepages/rtemap_20
> 540000000-580000000 rw-s 00000000 00:23 206103
> /dev/hugepages/rtemap_21
> 580000000-5c0000000 rw-s 00000000 00:23 206104
> /dev/hugepages/rtemap_22
> 5c0000000-600000000 rw-s 00000000 00:23 206105
> /dev/hugepages/rtemap_23
> 600000000-640000000 rw-s 00000000 00:23 206106
> /dev/hugepages/rtemap_24
> 640000000-680000000 rw-s 00000000 00:23 206107
> /dev/hugepages/rtemap_25
> 680000000-6c0000000 rw-s 00000000 00:23 206108
> /dev/hugepages/rtemap_26
> 6c0000000-700000000 rw-s 00000000 00:23 206109
> /dev/hugepages/rtemap_27
> 700000000-740000000 rw-s 00000000 00:23 206110
> /dev/hugepages/rtemap_28
> 740000000-780000000 rw-s 00000000 00:23 206111
> /dev/hugepages/rtemap_29
> 780000000-7c0000000 rw-s 00000000 00:23 206112
> /dev/hugepages/rtemap_30
> 7c0000000-800000000 rw-s 00000000 00:23 206113
> /dev/hugepages/rtemap_31
> 2aaac0000000-2aab00000000 rw-s 00000000 00:23 206082
> /dev/hugepages/rtemap_0
>
> As you can see it needs to retry a couple of times until it finds a free
> space at 0x100000000. In this example, OVS startup time went from 7
> seconds to 20.
>
>
> This happens on all OVS versions using DPDK 17.11.4, and the more memory
> you have the longer it takes. I guess most people test with a couple of
> 1G huge pages, but the more you add the longer it takes. We saw this at
> a customer taking 4+ minutes.
Hint: As OVS usually doesn't need so much memory you could limit the
hugepage allocation by cgroups to some reasonable number. This will
significantly decrease the start time and will help to avoid described issue.
This works just fine since CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is
enabled by default.
Best regards, Ilya Maximets.
More information about the dev
mailing list