[ovs-dev] 10-25 packet drops every few (10-50) seconds TCP (iperf3)

Flavio Leitner fbl at sysclose.org
Tue Jun 2 19:06:38 UTC 2020


On Mon, Jun 01, 2020 at 07:27:09PM -0400, Shahaji Bhosle via dev wrote:
> Hi Ben/Ilya,
> Hope you guys are doing well and staying safe. I have been chasing a weird
> problem with small drops and I think that is causing lots of TCP
> retransmission.
> 
> Setup details
> iPerf3(1k-5K
> Servers)<--DPDK2:OvS+DPDK(VxLAN:BOND)[DPDK0+DPDK1)<====2x25G<====
> [DPDK0+DPDK1)(VxLAN:BOND)OVS+DPDKDPDK2<---iPerf3(Clients)
> 
> All the Drops are ring drops on BONDed functions on the server side.  I
> have 4 CPUs each with 3PMD threads, DPDK0, DPDK1 and DPDK2 all running with
> 4 Rx rings each.
> 
> What is interesting is when I give each Rx rings its own CPU the drops go
> away. Or if I set cother_config:emc-insert-inv-prob=1 the drops go away.
> But I need to scale up the number of flows so trying to run this with EMC
> disabled.
> 
> I can tell that the rings are not getting serviced for 30-40usec because of
> some kind context switch or interrupts on these cores. I have tried to do
> the usual isolation, nohz_full rcu_nocbs etc. Move all the interrupts away
> from these cores etc. But nothing helps. I mean it improves, but the drops
> still happen.

When you disable the EMC (or reduce its efficiency) the per packet cost
increases, then it becomes more sensitive to variations. If you share
a CPU with multiple queues, you decrease the amount of time available
to process the queue. In either case, there will be less room to tolerate
variations.

Well, you might want to use 'perf' and monitor for the scheduling events
and then based on the stack trace see what is causing it and try to
prevent it.

For example:
# perf record -e sched:sched_switch -a -g sleep 1

For instance, you might see that another NIC used for management has
IRQs assigned to one isolated CPU. You can move it to another CPU to
reduce the noise, etc...

Another suggestion is look at PMD thread idle statistics because it
will tell you how much "extra" room you have left. As it approaches
to 0, more fine tuned your setup needs to be to avoid drops.

HTH,
-- 
fbl


More information about the dev mailing list