[ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does not work
Felix Brucker
FBrucker at xantaro.net
Wed Apr 6 12:24:47 UTC 2016
Hi,
i noticed my dpdk devices are configured to have only one queue (default), so i tried with 2,4,8 queues for the dpdk devices themselves.
when configuring more than one queue on the dpdk device the vm has four queues transferring traffic instead of only two, however if configuring 4 queues on the dpdk devices the vm does not get 8 queues working, it stays at four (that is virtio0->queue0+1 and virtio1->queue0+1).
My current setup hast wo pmd threads which serve two nics with 4 queues each. The dpdk devices are attached to an ovs dpdk enabled bridge each, which has one vhost-user interface each, configured with eight queues, like so:
ovs-appctl dpif-netdev/pmd-rxq-show
pmd thread numa_id 0 core_id 4:
port: dpdk0 queue-id: 0 2
port: vhost-user-1 queue-id: 0 2 4 6
port: vhost-user-0 queue-id: 0 2 4 6
port: dpdk1 queue-id: 0 2
pmd thread numa_id 0 core_id 5:
port: dpdk0 queue-id: 1 3
port: vhost-user-1 queue-id: 1 3 5 7
port: vhost-user-0 queue-id: 1 3 5 7
port: dpdk1 queue-id: 1 3
ovs-vsctl show
88b09698-f11b-4f4c-ab2c-45b455d6a2d1
Bridge "br0"
Port "br0"
Interface "br0"
type: internal
Port "dpdk0"
Interface "dpdk0"
type: dpdk
options: {n_rxq="4"}
Port "vhost-user-0"
Interface "vhost-user-0"
type: dpdkvhostuser
options: {n_rxq="8"}
Bridge "br1"
Port "vhost-user-1"
Interface "vhost-user-1"
type: dpdkvhostuser
options: {n_rxq="8"}
Port "dpdk1"
Interface "dpdk1"
type: dpdk
options: {n_rxq="4"}
Port "br1"
Interface "br1"
type: internal
Maybe you or someone else on the list knows why that is?
Best regards
Felix
-----Ursprüngliche Nachricht-----
Von: Felix Brucker
Gesendet: Dienstag, 5. April 2016 13:07
An: 'Loftus, Ciara' <ciara.loftus at intel.com>
Cc: discuss at openvswitch.org
Betreff: AW: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does not work
Hi Ciara,
yes i do see them, although i dont need to enter the ethtool command as the ubuntu image or qemu does this automatically it seems (ethtool -l eth0 always shows combined 8 even after boot) for me they look like this:
Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: VHOST_CONFIG: set queue enable: 1 to qp idx: 0 Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: ovs|00142|dpdk(vhost_thread2)|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/usr/local/var/run/openvswitch/vhost-user-0' 0 changed to 'enabled'
Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: VHOST_CONFIG: set queue enable: 1 to qp idx: 1 [...]
I do see the other queues as well but they dont transport packets
Best regards
Felix
-----Ursprüngliche Nachricht-----
Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
Gesendet: Dienstag, 5. April 2016 12:54
An: Felix Brucker <FBrucker at xantaro.net>
Cc: discuss at openvswitch.org
Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does not work
>
> Hi Ciara,
>
> ok so it should work!
> for me the traffic also reaches the vm, but i only see two queues utilized.
>
> Configs used:
>
> Ovs:
>
> ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=30030
> //corresponds to cores 4+5 and their siblings, so 4 pmd threads
> ovs-vsctl set Interface vhost-user-0 options:n_rxq=8 //8
> queues for each vhost-user interface
> ovs-vsctl set Interface vhost-user-1 options:n_rxq=8
>
> libvirt:
>
> <vcpu placement='static' cpuset='0-3,12-15'>8</vcpu> //cores 0-3
> and their siblings, so 8 vcpu
> <cputune>
> <vcpupin vcpu='0' cpuset='0'/>
> <vcpupin vcpu='1' cpuset='1'/>
> <vcpupin vcpu='2' cpuset='2'/>
> <vcpupin vcpu='3' cpuset='3'/>
> <vcpupin vcpu='4' cpuset='12'/>
> <vcpupin vcpu='5' cpuset='13'/>
> <vcpupin vcpu='6' cpuset='14'/>
> <vcpupin vcpu='7' cpuset='15'/>
> </cputune>
> [...]
> <driver queues='8'/>
> //used on both virtio nics
>
>
> Output of ovs-appctl dpif-netdev/pmd-rxq-show:
>
> pmd thread numa_id 0 core_id 16:
> port: vhost-user-0 queue-id: 2 6
> port: vhost-user-1 queue-id: 0 4
> pmd thread numa_id 0 core_id 17:
> port: vhost-user-0 queue-id: 3 7
> port: vhost-user-1 queue-id: 1 5
> pmd thread numa_id 0 core_id 4:
> port: vhost-user-0 queue-id: 0 4
> port: dpdk0 queue-id: 0
> port: vhost-user-1 queue-id: 2 6
> pmd thread numa_id 0 core_id 5:
> port: vhost-user-0 queue-id: 1 5
> port: dpdk1 queue-id: 0
> port: vhost-user-1 queue-id: 3 7
>
>
> upon starting traffic and having cleared the stats:
> ovs-appctl dpif-netdev/pmd-stats-show
> main thread:
> emc hits:0
> megaflow hits:0
> miss:0
> lost:0
> polling cycles:953844 (100.00%)
> processing cycles:0 (0.00%)
> pmd thread numa_id 0 core_id 16:
> emc hits:11227924
> megaflow hits:1
> miss:231
> lost:0
> polling cycles:18074152077 (66.87%)
> processing cycles:8955323625 (33.13%)
> avg cycles per packet: 2407.29 (27029475702/11228156)
> avg processing cycles per packet: 797.58 (8955323625/11228156)
> pmd thread numa_id 0 core_id 17:
> emc hits:1774
> megaflow hits:37
> miss:32
> lost:0
> polling cycles:20839820676 (99.99%)
> processing cycles:1977540 (0.01%)
> avg cycles per packet: 11308626.27 (20841798216/1843)
> avg processing cycles per packet: 1073.00 (1977540/1843) pmd
> thread numa_id 0 core_id 4:
> emc hits:11541300
> megaflow hits:108
> miss:24
> lost:0
> polling cycles:503392599 (1.55%)
> processing cycles:32036341425 (98.45%)
> avg cycles per packet: 2819.38 (32539734024/11541432)
> avg processing cycles per packet: 2775.77
> (32036341425/11541432) pmd thread numa_id 0 core_id 5:
> emc hits:25307647
> megaflow hits:0
> miss:226
> lost:0
> polling cycles:2461529511 (7.57%)
> processing cycles:30065889423 (92.43%)
> avg cycles per packet: 1285.27 (32527418934/25307873)
> avg processing cycles per packet: 1188.01
> (30065889423/25307873)
>
> inside the vm there are only two out of the eight cores utilized.
> when manually setting the irq affinity inside the guest and disableing
> irqbalance like so:
>
> echo 0 > /proc/irq/45/smp_affinity_list //virtio0-input.0
> echo 0 > /proc/irq/46/smp_affinity_list //virtio0-output.0
> echo 1 > /proc/irq/47/smp_affinity_list //virtio0-input.1
> echo 1 > /proc/irq/48/smp_affinity_list //...
> echo 2 > /proc/irq/49/smp_affinity_list echo 2 >
> /proc/irq/50/smp_affinity_list echo 3 > /proc/irq/51/smp_affinity_list
> echo 3 > /proc/irq/52/smp_affinity_list echo 4 >
> /proc/irq/53/smp_affinity_list echo 4 > /proc/irq/54/smp_affinity_list
> echo 5 > /proc/irq/55/smp_affinity_list echo 5 >
> /proc/irq/56/smp_affinity_list echo 6 > /proc/irq/57/smp_affinity_list
> echo 6 > /proc/irq/58/smp_affinity_list echo 7 >
> /proc/irq/59/smp_affinity_list echo 7 > /proc/irq/60/smp_affinity_list
>
> echo 0 > /proc/irq/62/smp_affinity_list //virtio1-input.0
> echo 0 > /proc/irq/63/smp_affinity_list //...
> echo 1 > /proc/irq/64/smp_affinity_list echo 1 >
> /proc/irq/65/smp_affinity_list echo 2 > /proc/irq/66/smp_affinity_list
> echo 2 > /proc/irq/67/smp_affinity_list echo 3 >
> /proc/irq/68/smp_affinity_list echo 3 > /proc/irq/69/smp_affinity_list
> echo 4 > /proc/irq/70/smp_affinity_list echo 4 >
> /proc/irq/71/smp_affinity_list echo 5 > /proc/irq/72/smp_affinity_list
> echo 5 > /proc/irq/73/smp_affinity_list echo 6 >
> /proc/irq/74/smp_affinity_list echo 6 > /proc/irq/75/smp_affinity_list
> echo 7 > /proc/irq/76/smp_affinity_list echo 7 >
> /proc/irq/77/smp_affinity_list
>
> im getting the same result, only two cores used inside the vm.
> Inside the vm: ifconfig does not return any dropped packets, so it
> seems the queues are not used.
>
> Cat /proc/interrupts shows only
> virtio0-input.0
> virtio0-output.0
> virtio0-output.1
> virtio1-output.0
> virtio1-input.1
Hi Felix,
That's strange. I can see:
virtio0-input.0
virtio0-output.0
virtio0-input.1
virtio0-output.1
virtio0-input.2
virtio0-output.2
virtio0-input.3
virtio0-output.3
virtio0-input.4
virtio0-output.4
virtio0-input.5
virtio0-output.5
virtio0-input.6
virtio0-output.6
virtio0-input.7
virtio0-output.7
Do you get the following logs or similar when you set " ethtool -L eth0 combined 8" in the VM?:
2016-04-05T03:40:54Z|00010|dpdk(vhost_thread2)|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/usr/local/var/run/openvswitch/dpdkvhostuser0' 0 changed to 'enabled'
2016-04-05T03:40:54Z|00011|dpdk(vhost_thread2)|INFO|State of queue 2 ( tx_qid 1 ) of vhost device '/usr/local/var/run/openvswitch/dpdkvhostuser0' 0 changed to 'enabled'
....
2016-04-05T03:40:54Z|00017|dpdk(vhost_thread2)|INFO|State of queue 14 ( tx_qid 7 ) of vhost device '/usr/local/var/run/openvswitch/dpdkvhostuser0' 0 changed to 'enabled'
Thanks,
Ciara
>
> generating interrupts, all other queues have generated zero or one
> interrupt since start.
>
>
> Best regards
> Felix
>
>
> -----Ursprüngliche Nachricht-----
> Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
> Gesendet: Dienstag, 5. April 2016 12:14
> An: Felix Brucker <FBrucker at xantaro.net>
> Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues
> does not work
>
> Hi Felix,
>
> It should work with as many pmd threads. I tested 8 queues with 1,2,4
> and 8 pmd threads and each time traffic was able to reach the VM.
>
> For example with 1 PMD thread
> # sudo ./utilities/ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-
> mask=1 # sudo ./utilities/ovs-appctl dpif-netdev/pmd-rxq-show pmd
> thread numa_id 0 core_id 0:
> port: dpdkvhostuser0 queue-id: 0 1 2 3 4 5 6 7
> port: dpdk0 queue-id: 0
>
> You will get better performance if you assign a core to each queue though.
>
> Thanks,
> Ciara
>
> > -----Original Message-----
> > From: Felix Brucker [mailto:FBrucker at xantaro.net]
> > Sent: Tuesday, April 05, 2016 11:06 AM
> > To: Loftus, Ciara <ciara.loftus at intel.com>
> > Subject: AW: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> > queues does not work
> >
> > Hi Ciara,
> >
> > thanks, one question though:
> >
> > you wrote:
> >
> > If you want a pmd to service each of the 8 queues, you can set the
> > number of PMD threads via:
> > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=FF This will
> > set up
> > 8 pmd threads, on cores 0-7.
> >
> > Does this mean i have to use as many host pmd threads as i want to
> > use queues for the vhost-user?
> > So using 2 host pmd threads and 8 vhost-user queues does not work?
> >
> > Best regards
> > Felix
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
> > Gesendet: Dienstag, 5. April 2016 12:00
> > An: Felix Brucker <FBrucker at xantaro.net>; Christian Ehrhardt
> > <christian.ehrhardt at canonical.com>
> > Cc: discuss at openvswitch.org
> > Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> > queues does not work
> >
> > >
> > > Hi,
> > >
> > > with the branch-2.5 ovs and n-dpdk-rxqs option i was able to get
> > > four queues with four pmd host threads, which currently is my
> > > limit for vm queues in terms of usable cores and the command itself.
> > >
> > > To my understanding, with ovs post 2.5 (latest git master) i
> > > should be able to use two n-dpdk-rxqs queues for the host and use
> > > ovs-vsctl set Interface
> > > vhost-user-0 options:n_rxq=8 to get eight queues inside the vm, is
> > > this correct?
> >
> > Hi Felix,
> >
> > Hopefully my explanation below will help clear things up.
> >
> > Post-2.5 the 'n-dpdk-rxqs' option is not available to use any more.
> > Here is a snippet from the commit message that removes this option
> > (commit id
> > a14b8947fd13d4c587addbffd24eedc7bb48ee2b)
> >
> > "dpif-netdev: Allow different numbers of rx queues for different ports.
> >
> > Currently, all of the PMD netdevs can only have the same number of
> > rx queues, which is specified in other_config:n-dpdk-rxqs.
> >
> > Fix that by introducing of new option for PMD interfaces: 'n_rxq',
> > which specifies the maximum number of rx queues to be created for
> > this
> interface.
> >
> > Example:
> > ovs-vsctl set Interface dpdk0 options:n_rxq=8
> >
> > Old 'other_config:n-dpdk-rxqs' deleted."
> >
> > In your case, now on latest master, if you want 8 queues assigned to
> > vhost-
> > user-0 in the guest you need to do the following:
> >
> > On the host:
> > 1. ovs-vsctl set Interface vhost-user-0 options:n_rxq=8 2. QEMU:
> > -chardev
> > socket,id=char0,path=/path/to/vhost-user-0 -netdev type=vhost-
> > user,id=mynet1,chardev=char0,vhostforce,queues=8 -device
> > virtio-net-
> >
> pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=off,mq=on,vectors=1
> > 8
> > Or if you're using libvirt I think the equivalent would be: <driver
> > queues='8'/>
> >
> > On the VM:
> > 3. Check queues available:
> > [root at localhost ~]# ethtool -l eth0
> > Channel parameters for eth0:
> > Pre-set maximums:
> > RX: 0
> > TX: 0
> > Other: 0
> > Combined: 8
> > Current hardware settings:
> > RX: 0
> > TX: 0
> > Other: 0
> > Combined: 1
> > # Enable 8 queues
> > [root at localhost ~]# ethtool -l eth0
> > Channel parameters for eth0:
> > Pre-set maximums:
> > RX: 0
> > TX: 0
> > Other: 0
> > Combined: 8
> > Current hardware settings:
> > RX: 0
> > TX: 0
> > Other: 0
> > Combined: 8
> >
> > At this point your vhost-user-0 interface (eth0) on the guest can
> > use
> > 8 queues.
> >
> > If you want a pmd to service each of the 8 queues, you can set the
> > number of PMD threads via:
> > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=FF This will
> > set up
> > 8 pmd threads, on cores 0-7.
> >
> > If you have other interfaces that you want to increase the number of
> > rxqs for, you may do so like so:
> > ovs-vsctl set Interface <iface> options:n_rxq=X
> >
> > Thanks,
> > Ciara
> >
> > >
> > > If so, im experiencing a problem were only two queues out of the
> > > eight are used for traffic.
> > >
> > > Best regards
> > > Felix
> > >
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: discuss [mailto:discuss-bounces at openvswitch.org] Im Auftrag
> > > von Felix Brucker
> > > Gesendet: Dienstag, 5. April 2016 09:48
> > > An: Loftus, Ciara <ciara.loftus at intel.com>; Christian Ehrhardt
> > > <christian.ehrhardt at canonical.com>
> > > Cc: Daniele Di Proietto <diproiettod at vmware.com>;
> > > discuss at openvswitch.org
> > > Betreff: [MASSMAIL] Re: [ovs-discuss] [OVS-DPDK] vhost-user with
> > > multiple queues does not work
> > >
> > > Hi Ciara,
> > >
> > > thanks that clarified it, i got confused by > Also this does NOT
> > > set the multiqueues the guest shall get i read the Install md from
> > > here http://openvswitch.org/support/dist-docs/INSTALL.DPDK.md.txt
> > > i thought this is related to the download on the same site
> > > (http://openvswitch.org/releases/openvswitch-2.5.0.tar.gz), but it
> > > seems not to.
> > > With the n-dpdk-rxqs=2 option i was able to get 2 queues inside
> > > the vm and working communication, too.
> > > After testing i will try to get the latest (post 2.5) version of
> > > ovs to get a more fine grained control over the queues.
> > > Thanks all!
> > >
> > > Best regards
> > > Felix
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
> > > Gesendet: Montag, 4. April 2016 18:11
> > > An: Felix Brucker <FBrucker at xantaro.net>; Christian Ehrhardt
> > > <christian.ehrhardt at canonical.com>
> > > Cc: Daniele Di Proietto <diproiettod at vmware.com>;
> > > discuss at openvswitch.org
> > > Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> > > queues does not work
> > >
> > > > yes that part works, but for communication to work between the
> > > > guest and host OVS has to use 2 queues as well, which currently
> > > > does
> not work.
> > > > So how does one set multiple queues for vhostuser in OVS 2.5.0
> > > > or
> > below?
> > > > Im not talking about libvirt or qemu regarding the above
> > > > question, but
> > OVS.
> > >
> > > Hi Felix,
> > >
> > > As we've mentioned before, you need to use the following command:
> > >
> > > ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2
> > >
> > > ... to assign two rx queues to the vhost-user ports in OVS.
> > >
> > > This is clearly stated in INSTALL.DPDK.md on the 2.5 branch. I
> > > suspect you were previously looking at the latest INSTALL guide
> > > which pointed you to use the n_rxq option, which is not available
> > > on the
> older branch-2.5.
> > >
> > > Essentially, if your bridge has two vhost-user ports eg.
> > > vhost-user-0 and vhost-user-1, the effect of ' ovs-vsctl set
> > > Open_vSwitch
> .
> > > other_config:n- dpdk-rxqs=2'
> > > is the same as
> > > ovs-vsctl set Interface vhost-user-0 options:n_rxq=2 ovs-vsctl set
> > > Interface
> > > vhost-user-1 options:n_rxq=2
> > >
> > > On branch-2.5, you need to use the former command.
> > >
> > > Thanks,
> > > Ciara
> > >
> > > >
> > > > Grüße
> > > > Felix
> > > >
> > > > Von: Christian Ehrhardt
> > > > [mailto:christian.ehrhardt at canonical.com]
> > > > Gesendet: Montag, 4. April 2016 17:35
> > > > An: Felix Brucker <FBrucker at xantaro.net>
> > > > Cc: Daniele Di Proietto <diproiettod at vmware.com>; Loftus, Ciara
> > > > <ciara.loftus at intel.com>; discuss at openvswitch.org
> > > > Betreff: Re: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> > > > queues does not work
> > > >
> > > > Hi Felix,
> > > > here you already do the right thing:
> > > >
> > > > <interface type='vhostuser'> [...]
> > > > <driver queues='2'/>
> > > >
> > > > Given you have the recent libvirt and qemu versions that
> > > > translates to the right qemu parameters as you have seen in my
> > > > initial
> posts.
> > > >
> > > > You can then log into the guest and check with "ethtool -l" if
> > > > the guest really "sees" its multiple queues (also shown in my
> > > > first mail that this fails for me)
> > > >
> > > > Kind Regards,
> > > > Christian
> > > _______________________________________________
> > > discuss mailing list
> > > discuss at openvswitch.org
> > > http://openvswitch.org/mailman/listinfo/discuss
More information about the discuss
mailing list