[ovs-dev] [PATCH] netdev-dpdk: Change eth rx burst size.

[Traynor, Kevin]

> -----Original Message-----
> From: Daniele Di Proietto [mailto:diproiettod at vmware.com]
> Sent: Friday, April 10, 2015 2:01 PM
> To: Traynor, Kevin
> Cc: Pravin Shelar; dev at openvswitch.org
> Subject: Re: [ovs-dev] [PATCH] netdev-dpdk: Change eth rx burst size.
> 
> 
> > On 10 Apr 2015, at 00:22, Traynor, Kevin <kevin.traynor at intel.com> wrote:
> >
> >
> >> -----Original Message-----
> >> From: Pravin Shelar [mailto:pshelar at nicira.com]
> >> Sent: Thursday, April 9, 2015 10:50 PM
> >> To: Traynor, Kevin
> >> Cc: Daniele Di Proietto; dev at openvswitch.org
> >> Subject: Re: [ovs-dev] [PATCH] netdev-dpdk: Change eth rx burst size.
> >>
> >> On Thu, Apr 9, 2015 at 2:36 PM, Traynor, Kevin <kevin.traynor at intel.com>
> >> wrote:
> >>>
> >>>> -----Original Message-----
> >>>> From: Pravin Shelar [mailto:pshelar at nicira.com]
> >>>> Sent: Thursday, April 9, 2015 9:10 PM
> >>>> To: Traynor, Kevin; Daniele Di Proietto
> >>>> Cc: dev at openvswitch.org
> >>>> Subject: Re: [ovs-dev] [PATCH] netdev-dpdk: Change eth rx burst size.
> >>>>
> >>>> On Wed, Apr 8, 2015 at 8:43 AM, Kevin Traynor <kevin.traynor at intel.com>
> >>>> wrote:
> >>>>> Change eth rx burst size from 192 to 32. This significantly
> >>>>> improves performance for packets that will be forwarded
> >>>>> through dpdkvhost ports, as the max dpdkvhost tx burst
> >>>>> size (32) will not be exceeded. There are negligible
> >>>>> effects in other scenarios.
> >>>>>
> >>>>> Signed-off-by: Kevin Traynor <kevin.traynor at intel.com>
> >>>>
> >>>> Daniele,
> >>>> You mentioned that you are going to fix the issue by handling
> >>>> different burst size of vhost devices. Do you have the fix?
> >>>
> >>> It wasn't clear in the commit message but just to clarify, the
> >>> dpdkvhost max burst size is set in the DPDK vhost library and if
> >>> we try and enqueue >32 packets only 32 will be enqueued. We can
> >>> add a back-off/retry but it will give better performance to
> >>> ensure that the vhost burst size is not exceeded.
> >>>
> >>
> >> How about adding for loop in vhost_send() to handle burst of packets
> >> larger than 32?
> >
> > Yeah, the DPDK sample app uses a backoff(small sleep)/retry loop but
> > you could put it in a busy loop too. I was planning to add some retry
> > code anyway as it's possible that you could get bursts of >32 from a
> > different interface, so it's good to handle this.
> >
> > The virtqueues are of very limited size (128 packets I think), so they
> > are not ideally suited to bursty traffic which I guess is why DPDK have
> > set the max burst size to 32. Time the PMD spends waiting/retrying is
> > time it could be doing something else, so I think it is better to reduce
> > the burst size where possible. I haven't seen any advantage to the 192 eth
> > rx burst size in my testing as changing to 32 I get negligible difference
> > on phy2phy tests (~1%) and I get a 2x-3x performance boost for vhost
> > loopback tests, but perhaps there are some use cases where it is
> > advantageous?
> 
> I agree with you guys that we should use a loop here,
> like in dpdk_queue_flush__(). How much slower would that be?
> 
> 

I've tested a simple retry loop with the 192 burst size and the performance
is very poor. With the retry loop and a burst size of 32, I see good
performance. I think this is because with the bigger burst sizes we are
hitting limits/contention in the vhost interface and we have to loop until
they are resolved. 

I think we need a retry loop, but it's not a performant replacement for
optimizing the burst size where we can at the moment. I can share the code
if you want to test it out on your systems.