[ovs-dev] [PATCH 2/2] netdev-dpdk: Retry tx/rx queue setup until we don't get any failure.

Ethan Jackson ethan at nicira.com
Thu Aug 6 21:25:02 UTC 2015


Merged and backported to 2.4.  Thanks

Ethan

On Thu, Aug 6, 2015 at 11:23 AM, Ethan Jackson <ethan at nicira.com> wrote:
> Great, I'll merge both today.
>
> Thanks a lot,
> Ethan
>
> On Thu, Aug 6, 2015 at 9:00 AM, Stokes, Ian <ian.stokes at intel.com> wrote:
>> No problem,
>>
>> I've sent a patch to the mailing list with these changes for INSTALL.DPDK.md.
>>
>> Thanks
>> Ian
>>
>>> -----Original Message-----
>>> From: Ethan Jackson [mailto:ethan at nicira.com]
>>> Sent: Tuesday, August 04, 2015 10:46 PM
>>> To: Stokes, Ian
>>> Cc: Ben Pfaff; Justin Pettit; Pravin Shelar; Traynor, Kevin; Daniele Di
>>> Proietto; dev at openvswitch.org
>>> Subject: Re: [ovs-dev] [PATCH 2/2] netdev-dpdk: Retry tx/rx queue setup
>>> until we don't get any failure.
>>>
>>> Sure let's do that.  If you write an INSTALL.DPDK note, and post it
>>> here.  I'll incorporate it into the patch and merge.  Thanks for your
>>> patience.
>>>
>>> Ethan
>>>
>>> On Tue, Aug 4, 2015 at 4:48 AM, Stokes, Ian <ian.stokes at intel.com>
>>> wrote:
>>> > This issue currently affects Fortville cards on systems with a logical
>>> core count greater than 64.
>>> >
>>> > Just to call out however that the number of cores setup by default to
>>> forward traffic to a NIC using OVS with DPDK will be the total number of
>>> cores detected on the system. This is the default behavior, thus there
>>> is no way user can "devote" less cores to forwarding on the NIC.
>>> >
>>> > This means they will always hit this issue when using a system with 72
>>> logical cores and a Fortville card. There is no way they can work around
>>> the issue with tx configuration changes.
>>> >
>>> > If this preferable then I can write up a note for theINSTALL.DPDK.md
>>> explaining the issue.
>>> >
>>> > Thanks
>>> > Ian
>>> >
>>> >> -----Original Message-----
>>> >> From: Ben Pfaff [mailto:blp at nicira.com]
>>> >> Sent: Monday, August 03, 2015 6:11 PM
>>> >> To: Ethan Jackson
>>> >> Cc: Stokes, Ian; Justin Pettit; Pravin Shelar; Traynor, Kevin;
>>> Daniele
>>> >> Di Proietto; dev at openvswitch.org
>>> >> Subject: Re: [ovs-dev] [PATCH 2/2] netdev-dpdk: Retry tx/rx queue
>>> setup
>>> >> until we don't get any failure.
>>> >>
>>> >> If that's an accurate description of the problem then that seems fine
>>> to
>>> >> me.  We could add a note to INSTALL.DPDK.md describing the issue I
>>> >> suppose, if someone wants to write one up.
>>> >>
>>> >> On Sat, Aug 01, 2015 at 02:40:16PM -0700, Ethan Jackson wrote:
>>> >> > I personally am fine with waiting on this being fixed until
>>> December.
>>> >> > My reading is, it only happens on one specific NIC, and even then
>>> only
>>> >> > if you devote a huge number of cores to forwarding on that NIC.
>>> >> >
>>> >> > That said, I won't block this if another committer disagrees with
>>> me.
>>> >> > Ben Justin Pravin?  What do you think?
>>> >> >
>>> >> > Ethan
>>> >> >
>>> >> > On Thu, Jul 30, 2015 at 1:13 AM, Stokes, Ian <ian.stokes at intel.com>
>>> >> wrote:
>>> >> > > So ideally this will be fixed in a future release of DPDK. We
>>> have
>>> >> flagged this. However that solution will not be in place until the
>>> DPDK
>>> >> 2.2 release in December at the earliest (DPDK 2.1 is currently in
>>> >> release candidate mode at the moment so it won't make it to that).
>>> When
>>> >> this has been changed in DPDK we can revisit the OVS code.
>>> >> > >
>>> >> > > Technically DPDK is doing what it is supposed to with the current
>>> >> implementation i.e. it is returning the max number of queues it
>>> >> supports. From the OVS side I think we need to understand that this
>>> has
>>> >> a different connotation to what it had with previously with NICS in
>>> >> terms of how many of those queues are usable.
>>> >> > >
>>> >> > > Unfortunately I don’t see another way to negotiate the tx queue
>>> >> initialization without something like the patch below.
>>> >> > > Not until we have more explicit configuration details available
>>> for
>>> >> the HW device from DPDK.
>>> >> > >
>>> >> > > Thanks
>>> >> > > Ian
>>> >> > >
>>> >> > >> -----Original Message-----
>>> >> > >> From: Ethan Jackson [mailto:ethan at nicira.com]
>>> >> > >> Sent: Wednesday, July 29, 2015 10:13 PM
>>> >> > >> To: Stokes, Ian
>>> >> > >> Cc: Traynor, Kevin; Daniele Di Proietto; dev at openvswitch.org
>>> >> > >> Subject: Re: [ovs-dev] [PATCH 2/2] netdev-dpdk: Retry tx/rx
>>> queue
>>> >> setup
>>> >> > >> until we don't get any failure.
>>> >> > >>
>>> >> > >> Sorry for taking so long to get to this.  The one question I
>>> have
>>> >> is:
>>> >> > >> Is OVS the right layer to be fixing this?  Isn't this really an
>>> >> issue
>>> >> > >> of DPDK reporting a number of available queues that for
>>> practical
>>> >> > >> purposes is wrong?  I.E. Shouldn't this be fixed by the DPDK
>>> driver
>>> >> of
>>> >> > >> this system?  This patch feels like a hack to me . . .
>>> >> > >>
>>> >> > >> Ethan
>>> >> > >>
>>> >> > >> On Tue, Jul 28, 2015 at 2:36 AM, Stokes, Ian
>>> <ian.stokes at intel.com>
>>> >> > >> wrote:
>>> >> > >> > Hi all,
>>> >> > >> >
>>> >> > >> > Just wondering what the status of this patch is? Is there any
>>> >> feedback
>>> >> > >> > or queries we can answer to help?
>>> >> > >> >
>>> >> > >> > Thanks
>>> >> > >> > Ian
>>> >> > >> >
>>> >> > >> >> -----Original Message-----
>>> >> > >> >> From: dev [mailto:dev-bounces at openvswitch.org] On Behalf Of
>>> >> Traynor,
>>> >> > >> >> Kevin
>>> >> > >> >> Sent: Thursday, July 23, 2015 11:28 AM
>>> >> > >> >> To: Daniele Di Proietto; dev at openvswitch.org
>>> >> > >> >> Subject: Re: [ovs-dev] [PATCH 2/2] netdev-dpdk: Retry tx/rx
>>> >> queue
>>> >> > >> setup
>>> >> > >> >> until we don't get any failure.
>>> >> > >> >>
>>> >> > >> >>
>>> >> > >> >> > -----Original Message-----
>>> >> > >> >> > From: dev [mailto:dev-bounces at openvswitch.org] On Behalf Of
>>> >> Daniele
>>> >> > >> Di
>>> >> > >> >> > Proietto
>>> >> > >> >> > Sent: Thursday, July 16, 2015 7:48 PM
>>> >> > >> >> > To: dev at openvswitch.org
>>> >> > >> >> > Subject: [ovs-dev] [PATCH 2/2] netdev-dpdk: Retry tx/rx
>>> queue
>>> >> setup
>>> >> > >> >> until we
>>> >> > >> >> > don't get any failure.
>>> >> > >> >> >
>>> >> > >> >> > It has been observed that some DPDK device (e.g intel
>>> xl710)
>>> >> report
>>> >> > >> an
>>> >> > >> >> > high number of queues but make some of them available only
>>> for
>>> >> > >> special
>>> >> > >> >> > functions (SRIOV).  Therefore the queues will be counted in
>>> >> > >> >> > rte_eth_dev_info_get(), but rte_eth_tx_queue_setup() will
>>> >> fail.
>>> >> > >> >> >
>>> >> > >> >> > This commit works around the issue by retrying the device
>>> >> > >> >> initialization
>>> >> > >> >> > with a smaller number of queues, if a queue fails to setup.
>>> >> > >> >> >
>>> >> > >> >> > Reported-by: Ian Stokes <ian.stokes at intel.com>
>>> >> > >> >> > Signed-off-by: Daniele Di Proietto <diproiettod at vmware.com>
>>> >> > >> >> > ---
>>> >> > >> >> >  lib/netdev-dpdk.c | 100
>>> >> +++++++++++++++++++++++++++++++++++++++---
>>> >> > >> ---
>>> >> > >> >> -------
>>> >> > >> >> > --
>>> >> > >> >> >  1 file changed, 73 insertions(+), 27 deletions(-)
>>> >> > >> >>
>>> >> > >> >>
>>> >> > >> >> Acked-by: Kevin Traynor <kevin.traynor at intel.com>
>>> >> > >> >> _______________________________________________
>>> >> > >> >> dev mailing list
>>> >> > >> >> dev at openvswitch.org
>>> >> > >> >> http://openvswitch.org/mailman/listinfo/dev
>>> >> > >> > _______________________________________________
>>> >> > >> > dev mailing list
>>> >> > >> > dev at openvswitch.org
>>> >> > >> > http://openvswitch.org/mailman/listinfo/dev



More information about the dev mailing list