[ovs-dev] [PATCH] Do RCU synchronization at fixed interval in PMD main loop.
i.maximets at samsung.com
Tue Aug 13 15:18:30 UTC 2019
On 07.08.2019 17:21, Nitin Katiyar wrote:
> Each PMD updates the global sequence number for RCU synchronization
> purpose with other OVS threads. This is done at every 1025th iteration
> in PMD main loop.
> If the PMD thread is responsible for polling large number of queues
> that are carrying traffic, it spends a lot of time processing packets
> and this results in significant delay in performing the housekeeping
> If the OVS main thread is waiting to synchronize with the PMD threads
> and if those threads delay performing housekeeping activities for
> more than 3 sec then LACP processing will be impacted and it will lead
> to LACP flaps. Similarly, other controls protocols run by OVS main
> thread are impacted.
> For e.g. a PMD thread polling 200 ports/queues with average of 1600
> processing cycles per packet with batch size of 32 may take 10240000
> (200 * 1600 * 32) cycles per iteration. In system with 2.0 GHz CPU
> it means more than 5 ms per iteration. So, for 1024 iterations to
> complete it would be more than 5 seconds.
> This gets worse when there are PMD threads which are less loaded.
> It reduces possibility of getting mutex lock in ovsrcu_try_quiesce()
> by heavily loaded PMD and next attempt to quiesce would be after 1024
> With this patch, PMD RCU synchronization will be performed after fixed
> interval instead after a fixed number of iterations. This will ensure
> that even if the packet processing load is high the RCU synchronization
> will not be delayed long.
> Co-authored-by: Anju Thomas <anju.thomas at ericsson.com>
> Signed-off-by: Nitin Katiyar <nitin.katiyar at ericsson.com>
> Signed-off-by: Anju Thomas <anju.thomas at ericsson.com>
Hi. Thanks for working on this.
Your setup seems a bit strange to me (thread will be able to handle only 6Kpps
per port), but I'm tend to agree that RCU synchronization needs to be fixed.
Not sure about implementation. At least, I'd suggest using time instead of TSC
Each PMD thread has 'ctx.now' time context measured in microseconds, so you may
schedule next synchronization to exactly ctx.now + 10ms. See 'next_optimization'
and 'rxq_next_cycle_store' as a reference. This will save few instructions and
also will help to avoid issues in non-DPDK (e.g. netdev-afxdp) case.
BTW, It's better to add 'dpif-netdev:' prefix to the patch name.
Best regards, Ilya Maximets.
More information about the dev