[ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does not work
Loftus, Ciara
ciara.loftus at intel.com
Wed Apr 6 16:26:09 UTC 2016
>
> i noticed my dpdk devices are configured to have only one queue (default),
> so i tried with 2,4,8 queues for the dpdk devices themselves.
> when configuring more than one queue on the dpdk device the vm has four
> queues transferring traffic instead of only two, however if configuring 4
> queues on the dpdk devices the vm does not get 8 queues working, it stays
> at four (that is virtio0->queue0+1 and virtio1->queue0+1).
That looks like one rx and one tx queue per vHost device, so essentially still no multiqueue in the VM.
Can you post the output of ethtool -l <vhost_dev> please.
> My current setup hast wo pmd threads which serve two nics with 4 queues
> each. The dpdk devices are attached to an ovs dpdk enabled bridge each,
> which has one vhost-user interface each, configured with eight queues, like
> so:
>
> ovs-appctl dpif-netdev/pmd-rxq-show
> pmd thread numa_id 0 core_id 4:
> port: dpdk0 queue-id: 0 2
> port: vhost-user-1 queue-id: 0 2 4 6
> port: vhost-user-0 queue-id: 0 2 4 6
> port: dpdk1 queue-id: 0 2
> pmd thread numa_id 0 core_id 5:
> port: dpdk0 queue-id: 1 3
> port: vhost-user-1 queue-id: 1 3 5 7
> port: vhost-user-0 queue-id: 1 3 5 7
> port: dpdk1 queue-id: 1 3
>
> ovs-vsctl show
> 88b09698-f11b-4f4c-ab2c-45b455d6a2d1
> Bridge "br0"
> Port "br0"
> Interface "br0"
> type: internal
> Port "dpdk0"
> Interface "dpdk0"
> type: dpdk
> options: {n_rxq="4"}
> Port "vhost-user-0"
> Interface "vhost-user-0"
> type: dpdkvhostuser
> options: {n_rxq="8"}
> Bridge "br1"
> Port "vhost-user-1"
> Interface "vhost-user-1"
> type: dpdkvhostuser
> options: {n_rxq="8"}
> Port "dpdk1"
> Interface "dpdk1"
> type: dpdk
> options: {n_rxq="4"}
> Port "br1"
> Interface "br1"
> type: internal
>
>
> Maybe you or someone else on the list knows why that is?
Hi,
I reproduced your set-up but did not get the same result. I can see 8 queue pairs (8 x rx and 8 x tx) on the guest for each vHost device. Like you said perhaps somebody else on the list might be able to provide more insight.
Thanks,
Ciara
>
> Best regards
> Felix
>
> -----Ursprüngliche Nachricht-----
> Von: Felix Brucker
> Gesendet: Dienstag, 5. April 2016 13:07
> An: 'Loftus, Ciara' <ciara.loftus at intel.com>
> Cc: discuss at openvswitch.org
> Betreff: AW: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues
> does not work
>
> Hi Ciara,
>
> yes i do see them, although i dont need to enter the ethtool command as the
> ubuntu image or qemu does this automatically it seems (ethtool -l eth0
> always shows combined 8 even after boot) for me they look like this:
>
> Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: VHOST_CONFIG: read
> message VHOST_USER_SET_VRING_ENABLE Apr 5 12:23:36 dpdk-test ovs-
> vswitchd[16128]: VHOST_CONFIG: set queue enable: 1 to qp idx: 0 Apr 5
> 12:23:36 dpdk-test ovs-vswitchd[16128]:
> ovs|00142|dpdk(vhost_thread2)|INFO|State of queue 0 ( tx_qid 0 ) of vhost
> device '/usr/local/var/run/openvswitch/vhost-user-0' 0 changed to 'enabled'
> Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: VHOST_CONFIG: read
> message VHOST_USER_SET_VRING_ENABLE Apr 5 12:23:36 dpdk-test ovs-
> vswitchd[16128]: VHOST_CONFIG: set queue enable: 1 to qp idx: 1 [...]
>
> I do see the other queues as well but they dont transport packets
>
> Best regards
> Felix
>
>
> -----Ursprüngliche Nachricht-----
> Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
> Gesendet: Dienstag, 5. April 2016 12:54
> An: Felix Brucker <FBrucker at xantaro.net>
> Cc: discuss at openvswitch.org
> Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does
> not work
>
> >
> > Hi Ciara,
> >
> > ok so it should work!
> > for me the traffic also reaches the vm, but i only see two queues utilized.
> >
> > Configs used:
> >
> > Ovs:
> >
> > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=30030
> > //corresponds to cores 4+5 and their siblings, so 4 pmd threads
> > ovs-vsctl set Interface vhost-user-0 options:n_rxq=8 //8
> > queues for each vhost-user interface
> > ovs-vsctl set Interface vhost-user-1 options:n_rxq=8
> >
> > libvirt:
> >
> > <vcpu placement='static' cpuset='0-3,12-15'>8</vcpu>
> //cores 0-3
> > and their siblings, so 8 vcpu
> > <cputune>
> > <vcpupin vcpu='0' cpuset='0'/>
> > <vcpupin vcpu='1' cpuset='1'/>
> > <vcpupin vcpu='2' cpuset='2'/>
> > <vcpupin vcpu='3' cpuset='3'/>
> > <vcpupin vcpu='4' cpuset='12'/>
> > <vcpupin vcpu='5' cpuset='13'/>
> > <vcpupin vcpu='6' cpuset='14'/>
> > <vcpupin vcpu='7' cpuset='15'/>
> > </cputune>
> > [...]
> > <driver queues='8'/>
> > //used on both virtio nics
> >
> >
> > Output of ovs-appctl dpif-netdev/pmd-rxq-show:
> >
> > pmd thread numa_id 0 core_id 16:
> > port: vhost-user-0 queue-id: 2 6
> > port: vhost-user-1 queue-id: 0 4
> > pmd thread numa_id 0 core_id 17:
> > port: vhost-user-0 queue-id: 3 7
> > port: vhost-user-1 queue-id: 1 5
> > pmd thread numa_id 0 core_id 4:
> > port: vhost-user-0 queue-id: 0 4
> > port: dpdk0 queue-id: 0
> > port: vhost-user-1 queue-id: 2 6
> > pmd thread numa_id 0 core_id 5:
> > port: vhost-user-0 queue-id: 1 5
> > port: dpdk1 queue-id: 0
> > port: vhost-user-1 queue-id: 3 7
> >
> >
> > upon starting traffic and having cleared the stats:
> > ovs-appctl dpif-netdev/pmd-stats-show
> > main thread:
> > emc hits:0
> > megaflow hits:0
> > miss:0
> > lost:0
> > polling cycles:953844 (100.00%)
> > processing cycles:0 (0.00%)
> > pmd thread numa_id 0 core_id 16:
> > emc hits:11227924
> > megaflow hits:1
> > miss:231
> > lost:0
> > polling cycles:18074152077 (66.87%)
> > processing cycles:8955323625 (33.13%)
> > avg cycles per packet: 2407.29 (27029475702/11228156)
> > avg processing cycles per packet: 797.58 (8955323625/11228156)
> > pmd thread numa_id 0 core_id 17:
> > emc hits:1774
> > megaflow hits:37
> > miss:32
> > lost:0
> > polling cycles:20839820676 (99.99%)
> > processing cycles:1977540 (0.01%)
> > avg cycles per packet: 11308626.27 (20841798216/1843)
> > avg processing cycles per packet: 1073.00 (1977540/1843) pmd
> > thread numa_id 0 core_id 4:
> > emc hits:11541300
> > megaflow hits:108
> > miss:24
> > lost:0
> > polling cycles:503392599 (1.55%)
> > processing cycles:32036341425 (98.45%)
> > avg cycles per packet: 2819.38 (32539734024/11541432)
> > avg processing cycles per packet: 2775.77
> > (32036341425/11541432) pmd thread numa_id 0 core_id 5:
> > emc hits:25307647
> > megaflow hits:0
> > miss:226
> > lost:0
> > polling cycles:2461529511 (7.57%)
> > processing cycles:30065889423 (92.43%)
> > avg cycles per packet: 1285.27 (32527418934/25307873)
> > avg processing cycles per packet: 1188.01
> > (30065889423/25307873)
> >
> > inside the vm there are only two out of the eight cores utilized.
> > when manually setting the irq affinity inside the guest and disableing
> > irqbalance like so:
> >
> > echo 0 > /proc/irq/45/smp_affinity_list //virtio0-
> input.0
> > echo 0 > /proc/irq/46/smp_affinity_list //virtio0-
> output.0
> > echo 1 > /proc/irq/47/smp_affinity_list //virtio0-
> input.1
> > echo 1 > /proc/irq/48/smp_affinity_list //...
> > echo 2 > /proc/irq/49/smp_affinity_list echo 2 >
> > /proc/irq/50/smp_affinity_list echo 3 > /proc/irq/51/smp_affinity_list
> > echo 3 > /proc/irq/52/smp_affinity_list echo 4 >
> > /proc/irq/53/smp_affinity_list echo 4 > /proc/irq/54/smp_affinity_list
> > echo 5 > /proc/irq/55/smp_affinity_list echo 5 >
> > /proc/irq/56/smp_affinity_list echo 6 > /proc/irq/57/smp_affinity_list
> > echo 6 > /proc/irq/58/smp_affinity_list echo 7 >
> > /proc/irq/59/smp_affinity_list echo 7 > /proc/irq/60/smp_affinity_list
> >
> > echo 0 > /proc/irq/62/smp_affinity_list //virtio1-
> input.0
> > echo 0 > /proc/irq/63/smp_affinity_list //...
> > echo 1 > /proc/irq/64/smp_affinity_list echo 1 >
> > /proc/irq/65/smp_affinity_list echo 2 > /proc/irq/66/smp_affinity_list
> > echo 2 > /proc/irq/67/smp_affinity_list echo 3 >
> > /proc/irq/68/smp_affinity_list echo 3 > /proc/irq/69/smp_affinity_list
> > echo 4 > /proc/irq/70/smp_affinity_list echo 4 >
> > /proc/irq/71/smp_affinity_list echo 5 > /proc/irq/72/smp_affinity_list
> > echo 5 > /proc/irq/73/smp_affinity_list echo 6 >
> > /proc/irq/74/smp_affinity_list echo 6 > /proc/irq/75/smp_affinity_list
> > echo 7 > /proc/irq/76/smp_affinity_list echo 7 >
> > /proc/irq/77/smp_affinity_list
> >
> > im getting the same result, only two cores used inside the vm.
> > Inside the vm: ifconfig does not return any dropped packets, so it
> > seems the queues are not used.
> >
> > Cat /proc/interrupts shows only
> > virtio0-input.0
> > virtio0-output.0
> > virtio0-output.1
> > virtio1-output.0
> > virtio1-input.1
>
> Hi Felix,
>
> That's strange. I can see:
> virtio0-input.0
> virtio0-output.0
> virtio0-input.1
> virtio0-output.1
> virtio0-input.2
> virtio0-output.2
> virtio0-input.3
> virtio0-output.3
> virtio0-input.4
> virtio0-output.4
> virtio0-input.5
> virtio0-output.5
> virtio0-input.6
> virtio0-output.6
> virtio0-input.7
> virtio0-output.7
>
> Do you get the following logs or similar when you set " ethtool -L eth0
> combined 8" in the VM?:
> 2016-04-05T03:40:54Z|00010|dpdk(vhost_thread2)|INFO|State of queue 0 (
> tx_qid 0 ) of vhost device '/usr/local/var/run/openvswitch/dpdkvhostuser0'
> 0 changed to 'enabled'
> 2016-04-05T03:40:54Z|00011|dpdk(vhost_thread2)|INFO|State of queue 2 (
> tx_qid 1 ) of vhost device '/usr/local/var/run/openvswitch/dpdkvhostuser0'
> 0 changed to 'enabled'
> ....
> 2016-04-05T03:40:54Z|00017|dpdk(vhost_thread2)|INFO|State of queue 14
> ( tx_qid 7 ) of vhost device '/usr/local/var/run/openvswitch/dpdkvhostuser0'
> 0 changed to 'enabled'
>
> Thanks,
> Ciara
>
> >
> > generating interrupts, all other queues have generated zero or one
> > interrupt since start.
> >
> >
> > Best regards
> > Felix
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
> > Gesendet: Dienstag, 5. April 2016 12:14
> > An: Felix Brucker <FBrucker at xantaro.net>
> > Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues
> > does not work
> >
> > Hi Felix,
> >
> > It should work with as many pmd threads. I tested 8 queues with 1,2,4
> > and 8 pmd threads and each time traffic was able to reach the VM.
> >
> > For example with 1 PMD thread
> > # sudo ./utilities/ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-
> > mask=1 # sudo ./utilities/ovs-appctl dpif-netdev/pmd-rxq-show pmd
> > thread numa_id 0 core_id 0:
> > port: dpdkvhostuser0 queue-id: 0 1 2 3 4 5 6 7
> > port: dpdk0 queue-id: 0
> >
> > You will get better performance if you assign a core to each queue though.
> >
> > Thanks,
> > Ciara
> >
> > > -----Original Message-----
> > > From: Felix Brucker [mailto:FBrucker at xantaro.net]
> > > Sent: Tuesday, April 05, 2016 11:06 AM
> > > To: Loftus, Ciara <ciara.loftus at intel.com>
> > > Subject: AW: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> > > queues does not work
> > >
> > > Hi Ciara,
> > >
> > > thanks, one question though:
> > >
> > > you wrote:
> > >
> > > If you want a pmd to service each of the 8 queues, you can set the
> > > number of PMD threads via:
> > > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=FF This will
> > > set up
> > > 8 pmd threads, on cores 0-7.
> > >
> > > Does this mean i have to use as many host pmd threads as i want to
> > > use queues for the vhost-user?
> > > So using 2 host pmd threads and 8 vhost-user queues does not work?
> > >
> > > Best regards
> > > Felix
> > >
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
> > > Gesendet: Dienstag, 5. April 2016 12:00
> > > An: Felix Brucker <FBrucker at xantaro.net>; Christian Ehrhardt
> > > <christian.ehrhardt at canonical.com>
> > > Cc: discuss at openvswitch.org
> > > Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> > > queues does not work
> > >
> > > >
> > > > Hi,
> > > >
> > > > with the branch-2.5 ovs and n-dpdk-rxqs option i was able to get
> > > > four queues with four pmd host threads, which currently is my
> > > > limit for vm queues in terms of usable cores and the command itself.
> > > >
> > > > To my understanding, with ovs post 2.5 (latest git master) i
> > > > should be able to use two n-dpdk-rxqs queues for the host and use
> > > > ovs-vsctl set Interface
> > > > vhost-user-0 options:n_rxq=8 to get eight queues inside the vm, is
> > > > this correct?
> > >
> > > Hi Felix,
> > >
> > > Hopefully my explanation below will help clear things up.
> > >
> > > Post-2.5 the 'n-dpdk-rxqs' option is not available to use any more.
> > > Here is a snippet from the commit message that removes this option
> > > (commit id
> > > a14b8947fd13d4c587addbffd24eedc7bb48ee2b)
> > >
> > > "dpif-netdev: Allow different numbers of rx queues for different ports.
> > >
> > > Currently, all of the PMD netdevs can only have the same number of
> > > rx queues, which is specified in other_config:n-dpdk-rxqs.
> > >
> > > Fix that by introducing of new option for PMD interfaces: 'n_rxq',
> > > which specifies the maximum number of rx queues to be created for
> > > this
> > interface.
> > >
> > > Example:
> > > ovs-vsctl set Interface dpdk0 options:n_rxq=8
> > >
> > > Old 'other_config:n-dpdk-rxqs' deleted."
> > >
> > > In your case, now on latest master, if you want 8 queues assigned to
> > > vhost-
> > > user-0 in the guest you need to do the following:
> > >
> > > On the host:
> > > 1. ovs-vsctl set Interface vhost-user-0 options:n_rxq=8 2. QEMU:
> > > -chardev
> > > socket,id=char0,path=/path/to/vhost-user-0 -netdev type=vhost-
> > > user,id=mynet1,chardev=char0,vhostforce,queues=8 -device
> > > virtio-net-
> > >
> >
> pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=off,mq=on,vectors=1
> > > 8
> > > Or if you're using libvirt I think the equivalent would be: <driver
> > > queues='8'/>
> > >
> > > On the VM:
> > > 3. Check queues available:
> > > [root at localhost ~]# ethtool -l eth0
> > > Channel parameters for eth0:
> > > Pre-set maximums:
> > > RX: 0
> > > TX: 0
> > > Other: 0
> > > Combined: 8
> > > Current hardware settings:
> > > RX: 0
> > > TX: 0
> > > Other: 0
> > > Combined: 1
> > > # Enable 8 queues
> > > [root at localhost ~]# ethtool -l eth0
> > > Channel parameters for eth0:
> > > Pre-set maximums:
> > > RX: 0
> > > TX: 0
> > > Other: 0
> > > Combined: 8
> > > Current hardware settings:
> > > RX: 0
> > > TX: 0
> > > Other: 0
> > > Combined: 8
> > >
> > > At this point your vhost-user-0 interface (eth0) on the guest can
> > > use
> > > 8 queues.
> > >
> > > If you want a pmd to service each of the 8 queues, you can set the
> > > number of PMD threads via:
> > > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=FF This will
> > > set up
> > > 8 pmd threads, on cores 0-7.
> > >
> > > If you have other interfaces that you want to increase the number of
> > > rxqs for, you may do so like so:
> > > ovs-vsctl set Interface <iface> options:n_rxq=X
> > >
> > > Thanks,
> > > Ciara
> > >
> > > >
> > > > If so, im experiencing a problem were only two queues out of the
> > > > eight are used for traffic.
> > > >
> > > > Best regards
> > > > Felix
> > > >
> > > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: discuss [mailto:discuss-bounces at openvswitch.org] Im Auftrag
> > > > von Felix Brucker
> > > > Gesendet: Dienstag, 5. April 2016 09:48
> > > > An: Loftus, Ciara <ciara.loftus at intel.com>; Christian Ehrhardt
> > > > <christian.ehrhardt at canonical.com>
> > > > Cc: Daniele Di Proietto <diproiettod at vmware.com>;
> > > > discuss at openvswitch.org
> > > > Betreff: [MASSMAIL] Re: [ovs-discuss] [OVS-DPDK] vhost-user with
> > > > multiple queues does not work
> > > >
> > > > Hi Ciara,
> > > >
> > > > thanks that clarified it, i got confused by > Also this does NOT
> > > > set the multiqueues the guest shall get i read the Install md from
> > > > here http://openvswitch.org/support/dist-docs/INSTALL.DPDK.md.txt
> > > > i thought this is related to the download on the same site
> > > > (http://openvswitch.org/releases/openvswitch-2.5.0.tar.gz), but it
> > > > seems not to.
> > > > With the n-dpdk-rxqs=2 option i was able to get 2 queues inside
> > > > the vm and working communication, too.
> > > > After testing i will try to get the latest (post 2.5) version of
> > > > ovs to get a more fine grained control over the queues.
> > > > Thanks all!
> > > >
> > > > Best regards
> > > > Felix
> > > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
> > > > Gesendet: Montag, 4. April 2016 18:11
> > > > An: Felix Brucker <FBrucker at xantaro.net>; Christian Ehrhardt
> > > > <christian.ehrhardt at canonical.com>
> > > > Cc: Daniele Di Proietto <diproiettod at vmware.com>;
> > > > discuss at openvswitch.org
> > > > Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> > > > queues does not work
> > > >
> > > > > yes that part works, but for communication to work between the
> > > > > guest and host OVS has to use 2 queues as well, which currently
> > > > > does
> > not work.
> > > > > So how does one set multiple queues for vhostuser in OVS 2.5.0
> > > > > or
> > > below?
> > > > > Im not talking about libvirt or qemu regarding the above
> > > > > question, but
> > > OVS.
> > > >
> > > > Hi Felix,
> > > >
> > > > As we've mentioned before, you need to use the following command:
> > > >
> > > > ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2
> > > >
> > > > ... to assign two rx queues to the vhost-user ports in OVS.
> > > >
> > > > This is clearly stated in INSTALL.DPDK.md on the 2.5 branch. I
> > > > suspect you were previously looking at the latest INSTALL guide
> > > > which pointed you to use the n_rxq option, which is not available
> > > > on the
> > older branch-2.5.
> > > >
> > > > Essentially, if your bridge has two vhost-user ports eg.
> > > > vhost-user-0 and vhost-user-1, the effect of ' ovs-vsctl set
> > > > Open_vSwitch
> > .
> > > > other_config:n- dpdk-rxqs=2'
> > > > is the same as
> > > > ovs-vsctl set Interface vhost-user-0 options:n_rxq=2 ovs-vsctl set
> > > > Interface
> > > > vhost-user-1 options:n_rxq=2
> > > >
> > > > On branch-2.5, you need to use the former command.
> > > >
> > > > Thanks,
> > > > Ciara
> > > >
> > > > >
> > > > > Grüße
> > > > > Felix
> > > > >
> > > > > Von: Christian Ehrhardt
> > > > > [mailto:christian.ehrhardt at canonical.com]
> > > > > Gesendet: Montag, 4. April 2016 17:35
> > > > > An: Felix Brucker <FBrucker at xantaro.net>
> > > > > Cc: Daniele Di Proietto <diproiettod at vmware.com>; Loftus, Ciara
> > > > > <ciara.loftus at intel.com>; discuss at openvswitch.org
> > > > > Betreff: Re: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> > > > > queues does not work
> > > > >
> > > > > Hi Felix,
> > > > > here you already do the right thing:
> > > > >
> > > > > <interface type='vhostuser'> [...]
> > > > > <driver queues='2'/>
> > > > >
> > > > > Given you have the recent libvirt and qemu versions that
> > > > > translates to the right qemu parameters as you have seen in my
> > > > > initial
> > posts.
> > > > >
> > > > > You can then log into the guest and check with "ethtool -l" if
> > > > > the guest really "sees" its multiple queues (also shown in my
> > > > > first mail that this fails for me)
> > > > >
> > > > > Kind Regards,
> > > > > Christian
> > > > _______________________________________________
> > > > discuss mailing list
> > > > discuss at openvswitch.org
> > > > http://openvswitch.org/mailman/listinfo/discuss
More information about the discuss
mailing list