[ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does not work
Felix Brucker
FBrucker at xantaro.net
Tue Apr 5 11:07:00 UTC 2016
Hi Ciara,
yes i do see them, although i dont need to enter the ethtool command as the ubuntu image or qemu does this automatically it seems (ethtool -l eth0 always shows combined 8 even after boot)
for me they look like this:
Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: VHOST_CONFIG: set queue enable: 1 to qp idx: 0
Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: ovs|00142|dpdk(vhost_thread2)|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/usr/local/var/run/openvswitch/vhost-user-0' 0 changed to 'enabled'
Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: VHOST_CONFIG: set queue enable: 1 to qp idx: 1
[...]
I do see the other queues as well but they dont transport packets
Best regards
Felix
-----Ursprüngliche Nachricht-----
Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
Gesendet: Dienstag, 5. April 2016 12:54
An: Felix Brucker <FBrucker at xantaro.net>
Cc: discuss at openvswitch.org
Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does not work
>
> Hi Ciara,
>
> ok so it should work!
> for me the traffic also reaches the vm, but i only see two queues utilized.
>
> Configs used:
>
> Ovs:
>
> ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=30030
> //corresponds to cores 4+5 and their siblings, so 4 pmd threads
> ovs-vsctl set Interface vhost-user-0 options:n_rxq=8 //8
> queues for each vhost-user interface
> ovs-vsctl set Interface vhost-user-1 options:n_rxq=8
>
> libvirt:
>
> <vcpu placement='static' cpuset='0-3,12-15'>8</vcpu> //cores 0-3
> and their siblings, so 8 vcpu
> <cputune>
> <vcpupin vcpu='0' cpuset='0'/>
> <vcpupin vcpu='1' cpuset='1'/>
> <vcpupin vcpu='2' cpuset='2'/>
> <vcpupin vcpu='3' cpuset='3'/>
> <vcpupin vcpu='4' cpuset='12'/>
> <vcpupin vcpu='5' cpuset='13'/>
> <vcpupin vcpu='6' cpuset='14'/>
> <vcpupin vcpu='7' cpuset='15'/>
> </cputune>
> [...]
> <driver queues='8'/>
> //used on both virtio nics
>
>
> Output of ovs-appctl dpif-netdev/pmd-rxq-show:
>
> pmd thread numa_id 0 core_id 16:
> port: vhost-user-0 queue-id: 2 6
> port: vhost-user-1 queue-id: 0 4
> pmd thread numa_id 0 core_id 17:
> port: vhost-user-0 queue-id: 3 7
> port: vhost-user-1 queue-id: 1 5
> pmd thread numa_id 0 core_id 4:
> port: vhost-user-0 queue-id: 0 4
> port: dpdk0 queue-id: 0
> port: vhost-user-1 queue-id: 2 6
> pmd thread numa_id 0 core_id 5:
> port: vhost-user-0 queue-id: 1 5
> port: dpdk1 queue-id: 0
> port: vhost-user-1 queue-id: 3 7
>
>
> upon starting traffic and having cleared the stats:
> ovs-appctl dpif-netdev/pmd-stats-show
> main thread:
> emc hits:0
> megaflow hits:0
> miss:0
> lost:0
> polling cycles:953844 (100.00%)
> processing cycles:0 (0.00%)
> pmd thread numa_id 0 core_id 16:
> emc hits:11227924
> megaflow hits:1
> miss:231
> lost:0
> polling cycles:18074152077 (66.87%)
> processing cycles:8955323625 (33.13%)
> avg cycles per packet: 2407.29 (27029475702/11228156)
> avg processing cycles per packet: 797.58 (8955323625/11228156)
> pmd thread numa_id 0 core_id 17:
> emc hits:1774
> megaflow hits:37
> miss:32
> lost:0
> polling cycles:20839820676 (99.99%)
> processing cycles:1977540 (0.01%)
> avg cycles per packet: 11308626.27 (20841798216/1843)
> avg processing cycles per packet: 1073.00 (1977540/1843) pmd
> thread numa_id 0 core_id 4:
> emc hits:11541300
> megaflow hits:108
> miss:24
> lost:0
> polling cycles:503392599 (1.55%)
> processing cycles:32036341425 (98.45%)
> avg cycles per packet: 2819.38 (32539734024/11541432)
> avg processing cycles per packet: 2775.77
> (32036341425/11541432) pmd thread numa_id 0 core_id 5:
> emc hits:25307647
> megaflow hits:0
> miss:226
> lost:0
> polling cycles:2461529511 (7.57%)
> processing cycles:30065889423 (92.43%)
> avg cycles per packet: 1285.27 (32527418934/25307873)
> avg processing cycles per packet: 1188.01
> (30065889423/25307873)
>
> inside the vm there are only two out of the eight cores utilized.
> when manually setting the irq affinity inside the guest and disableing
> irqbalance like so:
>
> echo 0 > /proc/irq/45/smp_affinity_list //virtio0-input.0
> echo 0 > /proc/irq/46/smp_affinity_list //virtio0-output.0
> echo 1 > /proc/irq/47/smp_affinity_list //virtio0-input.1
> echo 1 > /proc/irq/48/smp_affinity_list //...
> echo 2 > /proc/irq/49/smp_affinity_list echo 2 >
> /proc/irq/50/smp_affinity_list echo 3 > /proc/irq/51/smp_affinity_list
> echo 3 > /proc/irq/52/smp_affinity_list echo 4 >
> /proc/irq/53/smp_affinity_list echo 4 > /proc/irq/54/smp_affinity_list
> echo 5 > /proc/irq/55/smp_affinity_list echo 5 >
> /proc/irq/56/smp_affinity_list echo 6 > /proc/irq/57/smp_affinity_list
> echo 6 > /proc/irq/58/smp_affinity_list echo 7 >
> /proc/irq/59/smp_affinity_list echo 7 > /proc/irq/60/smp_affinity_list
>
> echo 0 > /proc/irq/62/smp_affinity_list //virtio1-input.0
> echo 0 > /proc/irq/63/smp_affinity_list //...
> echo 1 > /proc/irq/64/smp_affinity_list echo 1 >
> /proc/irq/65/smp_affinity_list echo 2 > /proc/irq/66/smp_affinity_list
> echo 2 > /proc/irq/67/smp_affinity_list echo 3 >
> /proc/irq/68/smp_affinity_list echo 3 > /proc/irq/69/smp_affinity_list
> echo 4 > /proc/irq/70/smp_affinity_list echo 4 >
> /proc/irq/71/smp_affinity_list echo 5 > /proc/irq/72/smp_affinity_list
> echo 5 > /proc/irq/73/smp_affinity_list echo 6 >
> /proc/irq/74/smp_affinity_list echo 6 > /proc/irq/75/smp_affinity_list
> echo 7 > /proc/irq/76/smp_affinity_list echo 7 >
> /proc/irq/77/smp_affinity_list
>
> im getting the same result, only two cores used inside the vm.
> Inside the vm: ifconfig does not return any dropped packets, so it
> seems the queues are not used.
>
> Cat /proc/interrupts shows only
> virtio0-input.0
> virtio0-output.0
> virtio0-output.1
> virtio1-output.0
> virtio1-input.1
Hi Felix,
That's strange. I can see:
virtio0-input.0
virtio0-output.0
virtio0-input.1
virtio0-output.1
virtio0-input.2
virtio0-output.2
virtio0-input.3
virtio0-output.3
virtio0-input.4
virtio0-output.4
virtio0-input.5
virtio0-output.5
virtio0-input.6
virtio0-output.6
virtio0-input.7
virtio0-output.7
Do you get the following logs or similar when you set " ethtool -L eth0 combined 8" in the VM?:
2016-04-05T03:40:54Z|00010|dpdk(vhost_thread2)|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/usr/local/var/run/openvswitch/dpdkvhostuser0' 0 changed to 'enabled'
2016-04-05T03:40:54Z|00011|dpdk(vhost_thread2)|INFO|State of queue 2 ( tx_qid 1 ) of vhost device '/usr/local/var/run/openvswitch/dpdkvhostuser0' 0 changed to 'enabled'
....
2016-04-05T03:40:54Z|00017|dpdk(vhost_thread2)|INFO|State of queue 14 ( tx_qid 7 ) of vhost device '/usr/local/var/run/openvswitch/dpdkvhostuser0' 0 changed to 'enabled'
Thanks,
Ciara
>
> generating interrupts, all other queues have generated zero or one
> interrupt since start.
>
>
> Best regards
> Felix
>
>
> -----Ursprüngliche Nachricht-----
> Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
> Gesendet: Dienstag, 5. April 2016 12:14
> An: Felix Brucker <FBrucker at xantaro.net>
> Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues
> does not work
>
> Hi Felix,
>
> It should work with as many pmd threads. I tested 8 queues with 1,2,4
> and 8 pmd threads and each time traffic was able to reach the VM.
>
> For example with 1 PMD thread
> # sudo ./utilities/ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-
> mask=1 # sudo ./utilities/ovs-appctl dpif-netdev/pmd-rxq-show pmd
> thread numa_id 0 core_id 0:
> port: dpdkvhostuser0 queue-id: 0 1 2 3 4 5 6 7
> port: dpdk0 queue-id: 0
>
> You will get better performance if you assign a core to each queue though.
>
> Thanks,
> Ciara
>
> > -----Original Message-----
> > From: Felix Brucker [mailto:FBrucker at xantaro.net]
> > Sent: Tuesday, April 05, 2016 11:06 AM
> > To: Loftus, Ciara <ciara.loftus at intel.com>
> > Subject: AW: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> > queues does not work
> >
> > Hi Ciara,
> >
> > thanks, one question though:
> >
> > you wrote:
> >
> > If you want a pmd to service each of the 8 queues, you can set the
> > number of PMD threads via:
> > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=FF This will
> > set up
> > 8 pmd threads, on cores 0-7.
> >
> > Does this mean i have to use as many host pmd threads as i want to
> > use queues for the vhost-user?
> > So using 2 host pmd threads and 8 vhost-user queues does not work?
> >
> > Best regards
> > Felix
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
> > Gesendet: Dienstag, 5. April 2016 12:00
> > An: Felix Brucker <FBrucker at xantaro.net>; Christian Ehrhardt
> > <christian.ehrhardt at canonical.com>
> > Cc: discuss at openvswitch.org
> > Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> > queues does not work
> >
> > >
> > > Hi,
> > >
> > > with the branch-2.5 ovs and n-dpdk-rxqs option i was able to get
> > > four queues with four pmd host threads, which currently is my
> > > limit for vm queues in terms of usable cores and the command itself.
> > >
> > > To my understanding, with ovs post 2.5 (latest git master) i
> > > should be able to use two n-dpdk-rxqs queues for the host and use
> > > ovs-vsctl set Interface
> > > vhost-user-0 options:n_rxq=8 to get eight queues inside the vm, is
> > > this correct?
> >
> > Hi Felix,
> >
> > Hopefully my explanation below will help clear things up.
> >
> > Post-2.5 the 'n-dpdk-rxqs' option is not available to use any more.
> > Here is a snippet from the commit message that removes this option
> > (commit id
> > a14b8947fd13d4c587addbffd24eedc7bb48ee2b)
> >
> > "dpif-netdev: Allow different numbers of rx queues for different ports.
> >
> > Currently, all of the PMD netdevs can only have the same number of
> > rx queues, which is specified in other_config:n-dpdk-rxqs.
> >
> > Fix that by introducing of new option for PMD interfaces: 'n_rxq',
> > which specifies the maximum number of rx queues to be created for
> > this
> interface.
> >
> > Example:
> > ovs-vsctl set Interface dpdk0 options:n_rxq=8
> >
> > Old 'other_config:n-dpdk-rxqs' deleted."
> >
> > In your case, now on latest master, if you want 8 queues assigned to
> > vhost-
> > user-0 in the guest you need to do the following:
> >
> > On the host:
> > 1. ovs-vsctl set Interface vhost-user-0 options:n_rxq=8 2. QEMU:
> > -chardev
> > socket,id=char0,path=/path/to/vhost-user-0 -netdev type=vhost-
> > user,id=mynet1,chardev=char0,vhostforce,queues=8 -device
> > virtio-net-
> >
> pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=off,mq=on,vectors=1
> > 8
> > Or if you're using libvirt I think the equivalent would be: <driver
> > queues='8'/>
> >
> > On the VM:
> > 3. Check queues available:
> > [root at localhost ~]# ethtool -l eth0
> > Channel parameters for eth0:
> > Pre-set maximums:
> > RX: 0
> > TX: 0
> > Other: 0
> > Combined: 8
> > Current hardware settings:
> > RX: 0
> > TX: 0
> > Other: 0
> > Combined: 1
> > # Enable 8 queues
> > [root at localhost ~]# ethtool -l eth0
> > Channel parameters for eth0:
> > Pre-set maximums:
> > RX: 0
> > TX: 0
> > Other: 0
> > Combined: 8
> > Current hardware settings:
> > RX: 0
> > TX: 0
> > Other: 0
> > Combined: 8
> >
> > At this point your vhost-user-0 interface (eth0) on the guest can
> > use
> > 8 queues.
> >
> > If you want a pmd to service each of the 8 queues, you can set the
> > number of PMD threads via:
> > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=FF This will
> > set up
> > 8 pmd threads, on cores 0-7.
> >
> > If you have other interfaces that you want to increase the number of
> > rxqs for, you may do so like so:
> > ovs-vsctl set Interface <iface> options:n_rxq=X
> >
> > Thanks,
> > Ciara
> >
> > >
> > > If so, im experiencing a problem were only two queues out of the
> > > eight are used for traffic.
> > >
> > > Best regards
> > > Felix
> > >
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: discuss [mailto:discuss-bounces at openvswitch.org] Im Auftrag
> > > von Felix Brucker
> > > Gesendet: Dienstag, 5. April 2016 09:48
> > > An: Loftus, Ciara <ciara.loftus at intel.com>; Christian Ehrhardt
> > > <christian.ehrhardt at canonical.com>
> > > Cc: Daniele Di Proietto <diproiettod at vmware.com>;
> > > discuss at openvswitch.org
> > > Betreff: [MASSMAIL] Re: [ovs-discuss] [OVS-DPDK] vhost-user with
> > > multiple queues does not work
> > >
> > > Hi Ciara,
> > >
> > > thanks that clarified it, i got confused by > Also this does NOT
> > > set the multiqueues the guest shall get i read the Install md from
> > > here http://openvswitch.org/support/dist-docs/INSTALL.DPDK.md.txt
> > > i thought this is related to the download on the same site
> > > (http://openvswitch.org/releases/openvswitch-2.5.0.tar.gz), but it
> > > seems not to.
> > > With the n-dpdk-rxqs=2 option i was able to get 2 queues inside
> > > the vm and working communication, too.
> > > After testing i will try to get the latest (post 2.5) version of
> > > ovs to get a more fine grained control over the queues.
> > > Thanks all!
> > >
> > > Best regards
> > > Felix
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
> > > Gesendet: Montag, 4. April 2016 18:11
> > > An: Felix Brucker <FBrucker at xantaro.net>; Christian Ehrhardt
> > > <christian.ehrhardt at canonical.com>
> > > Cc: Daniele Di Proietto <diproiettod at vmware.com>;
> > > discuss at openvswitch.org
> > > Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> > > queues does not work
> > >
> > > > yes that part works, but for communication to work between the
> > > > guest and host OVS has to use 2 queues as well, which currently
> > > > does
> not work.
> > > > So how does one set multiple queues for vhostuser in OVS 2.5.0
> > > > or
> > below?
> > > > Im not talking about libvirt or qemu regarding the above
> > > > question, but
> > OVS.
> > >
> > > Hi Felix,
> > >
> > > As we've mentioned before, you need to use the following command:
> > >
> > > ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2
> > >
> > > ... to assign two rx queues to the vhost-user ports in OVS.
> > >
> > > This is clearly stated in INSTALL.DPDK.md on the 2.5 branch. I
> > > suspect you were previously looking at the latest INSTALL guide
> > > which pointed you to use the n_rxq option, which is not available
> > > on the
> older branch-2.5.
> > >
> > > Essentially, if your bridge has two vhost-user ports eg.
> > > vhost-user-0 and vhost-user-1, the effect of ' ovs-vsctl set
> > > Open_vSwitch
> .
> > > other_config:n- dpdk-rxqs=2'
> > > is the same as
> > > ovs-vsctl set Interface vhost-user-0 options:n_rxq=2 ovs-vsctl set
> > > Interface
> > > vhost-user-1 options:n_rxq=2
> > >
> > > On branch-2.5, you need to use the former command.
> > >
> > > Thanks,
> > > Ciara
> > >
> > > >
> > > > Grüße
> > > > Felix
> > > >
> > > > Von: Christian Ehrhardt
> > > > [mailto:christian.ehrhardt at canonical.com]
> > > > Gesendet: Montag, 4. April 2016 17:35
> > > > An: Felix Brucker <FBrucker at xantaro.net>
> > > > Cc: Daniele Di Proietto <diproiettod at vmware.com>; Loftus, Ciara
> > > > <ciara.loftus at intel.com>; discuss at openvswitch.org
> > > > Betreff: Re: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> > > > queues does not work
> > > >
> > > > Hi Felix,
> > > > here you already do the right thing:
> > > >
> > > > <interface type='vhostuser'> [...]
> > > > <driver queues='2'/>
> > > >
> > > > Given you have the recent libvirt and qemu versions that
> > > > translates to the right qemu parameters as you have seen in my
> > > > initial
> posts.
> > > >
> > > > You can then log into the guest and check with "ethtool -l" if
> > > > the guest really "sees" its multiple queues (also shown in my
> > > > first mail that this fails for me)
> > > >
> > > > Kind Regards,
> > > > Christian
> > > _______________________________________________
> > > discuss mailing list
> > > discuss at openvswitch.org
> > > http://openvswitch.org/mailman/listinfo/discuss
More information about the discuss
mailing list