[ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does not work

Loftus, Ciara ciara.loftus at intel.com
Tue Apr 5 09:59:58 UTC 2016


> 
> Hi,
> 
> with the branch-2.5 ovs and n-dpdk-rxqs option i was able to get four queues
> with four pmd host threads, which currently is my limit for vm queues in
> terms of usable cores and the command itself.
> 
> To my understanding, with ovs post 2.5 (latest git master) i should be able to
> use two n-dpdk-rxqs queues for the host and use ovs-vsctl set Interface
> vhost-user-0 options:n_rxq=8 to get eight queues inside the vm, is this
> correct?

Hi Felix,

Hopefully my explanation below will help clear things up.

Post-2.5 the 'n-dpdk-rxqs' option is not available to use any more. Here is a snippet from the commit message that removes this option (commit id a14b8947fd13d4c587addbffd24eedc7bb48ee2b)

"dpif-netdev: Allow different numbers of rx queues for different ports.

Currently, all of the PMD netdevs can only have the same number of
rx queues, which is specified in other_config:n-dpdk-rxqs.

Fix that by introducing of new option for PMD interfaces: 'n_rxq', which
specifies the maximum number of rx queues to be created for this
interface.

Example:
    ovs-vsctl set Interface dpdk0 options:n_rxq=8

Old 'other_config:n-dpdk-rxqs' deleted."

In your case, now on latest master, if you want 8 queues assigned to vhost-user-0 in the guest you need to do the following:

On the host:
1. ovs-vsctl set Interface vhost-user-0 options:n_rxq=8
2. QEMU: -chardev socket,id=char0,path=/path/to/vhost-user-0 -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce,queues=8  -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=off,mq=on,vectors=18
Or if you're using libvirt I think the equivalent would be: <driver queues='8'/>

On the VM:
3. Check queues available:
[root at localhost ~]# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:             0
TX:             0
Other:          0
Combined:       8
Current hardware settings:
RX:             0
TX:             0
Other:          0
Combined:       1
# Enable 8 queues
[root at localhost ~]# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:             0
TX:             0
Other:          0
Combined:       8
Current hardware settings:
RX:             0
TX:             0
Other:          0
Combined:       8

At this point your vhost-user-0 interface (eth0) on the guest can use 8 queues.

If you want a pmd to service each of the 8 queues, you can set the number of PMD threads via:
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=FF
This will set up 8 pmd threads, on cores 0-7.

If you have other interfaces that you want to increase the number of rxqs for, you may do so like so:
ovs-vsctl set Interface <iface> options:n_rxq=X

Thanks,
Ciara

> 
> If so, im experiencing a problem were only two queues out of the eight are
> used for traffic.
> 
> Best regards
> Felix
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: discuss [mailto:discuss-bounces at openvswitch.org] Im Auftrag von Felix
> Brucker
> Gesendet: Dienstag, 5. April 2016 09:48
> An: Loftus, Ciara <ciara.loftus at intel.com>; Christian Ehrhardt
> <christian.ehrhardt at canonical.com>
> Cc: Daniele Di Proietto <diproiettod at vmware.com>;
> discuss at openvswitch.org
> Betreff: [MASSMAIL] Re: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> queues does not work
> 
> Hi Ciara,
> 
> thanks that clarified it, i got confused by > Also this does NOT set the
> multiqueues the guest shall get i read the Install md from here
> http://openvswitch.org/support/dist-docs/INSTALL.DPDK.md.txt
> i thought this is related to the download on the same site
> (http://openvswitch.org/releases/openvswitch-2.5.0.tar.gz), but it seems
> not to.
> With the n-dpdk-rxqs=2 option i was able to get 2 queues inside the vm and
> working communication, too.
> After testing i will try to get the latest (post 2.5) version of ovs to get a more
> fine grained control over the queues.
> Thanks all!
> 
> Best regards
> Felix
> 
> -----Ursprüngliche Nachricht-----
> Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
> Gesendet: Montag, 4. April 2016 18:11
> An: Felix Brucker <FBrucker at xantaro.net>; Christian Ehrhardt
> <christian.ehrhardt at canonical.com>
> Cc: Daniele Di Proietto <diproiettod at vmware.com>;
> discuss at openvswitch.org
> Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does
> not work
> 
> > yes that part works, but for communication to work between the guest
> > and host OVS has to use 2 queues as well, which currently does not work.
> > So how does one set multiple queues for vhostuser in OVS 2.5.0 or below?
> > Im not talking about libvirt or qemu regarding the above question, but OVS.
> 
> Hi Felix,
> 
> As we've mentioned before, you need to use the following command:
> 
> ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2
> 
> ... to assign two rx queues to the vhost-user ports in OVS.
> 
> This is clearly stated in INSTALL.DPDK.md on the 2.5 branch. I suspect you
> were previously looking at the latest INSTALL guide which pointed you to use
> the n_rxq option, which is not available on the older branch-2.5.
> 
> Essentially, if your bridge has two vhost-user ports eg. vhost-user-0 and
> vhost-user-1, the effect of ' ovs-vsctl set Open_vSwitch . other_config:n-
> dpdk-rxqs=2'
> is the same as
> ovs-vsctl set Interface vhost-user-0 options:n_rxq=2 ovs-vsctl set Interface
> vhost-user-1 options:n_rxq=2
> 
> On branch-2.5, you need to use the former command.
> 
> Thanks,
> Ciara
> 
> >
> > Grüße
> > Felix
> >
> > Von: Christian Ehrhardt [mailto:christian.ehrhardt at canonical.com]
> > Gesendet: Montag, 4. April 2016 17:35
> > An: Felix Brucker <FBrucker at xantaro.net>
> > Cc: Daniele Di Proietto <diproiettod at vmware.com>; Loftus, Ciara
> > <ciara.loftus at intel.com>; discuss at openvswitch.org
> > Betreff: Re: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues
> > does not work
> >
> > Hi Felix,
> > here you already do the right thing:
> >
> >     <interface type='vhostuser'>
> > [...]
> >       <driver queues='2'/>
> >
> > Given you have the recent libvirt and qemu versions that translates to
> > the right qemu parameters as you have seen in my initial posts.
> >
> > You can then log into the guest and check with "ethtool -l" if the
> > guest really "sees" its multiple queues (also shown in my first mail
> > that this fails for me)
> >
> > Kind Regards,
> > Christian
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> http://openvswitch.org/mailman/listinfo/discuss


More information about the discuss mailing list