[ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does not work
Felix Brucker
FBrucker at xantaro.net
Tue Apr 5 10:39:43 UTC 2016
Hi Ciara,
ok so it should work!
for me the traffic also reaches the vm, but i only see two queues utilized.
Configs used:
Ovs:
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=30030 //corresponds to cores 4+5 and their siblings, so 4 pmd threads
ovs-vsctl set Interface vhost-user-0 options:n_rxq=8 //8 queues for each vhost-user interface
ovs-vsctl set Interface vhost-user-1 options:n_rxq=8
libvirt:
<vcpu placement='static' cpuset='0-3,12-15'>8</vcpu> //cores 0-3 and their siblings, so 8 vcpu
<cputune>
<vcpupin vcpu='0' cpuset='0'/>
<vcpupin vcpu='1' cpuset='1'/>
<vcpupin vcpu='2' cpuset='2'/>
<vcpupin vcpu='3' cpuset='3'/>
<vcpupin vcpu='4' cpuset='12'/>
<vcpupin vcpu='5' cpuset='13'/>
<vcpupin vcpu='6' cpuset='14'/>
<vcpupin vcpu='7' cpuset='15'/>
</cputune>
[...]
<driver queues='8'/> //used on both virtio nics
Output of ovs-appctl dpif-netdev/pmd-rxq-show:
pmd thread numa_id 0 core_id 16:
port: vhost-user-0 queue-id: 2 6
port: vhost-user-1 queue-id: 0 4
pmd thread numa_id 0 core_id 17:
port: vhost-user-0 queue-id: 3 7
port: vhost-user-1 queue-id: 1 5
pmd thread numa_id 0 core_id 4:
port: vhost-user-0 queue-id: 0 4
port: dpdk0 queue-id: 0
port: vhost-user-1 queue-id: 2 6
pmd thread numa_id 0 core_id 5:
port: vhost-user-0 queue-id: 1 5
port: dpdk1 queue-id: 0
port: vhost-user-1 queue-id: 3 7
upon starting traffic and having cleared the stats:
ovs-appctl dpif-netdev/pmd-stats-show
main thread:
emc hits:0
megaflow hits:0
miss:0
lost:0
polling cycles:953844 (100.00%)
processing cycles:0 (0.00%)
pmd thread numa_id 0 core_id 16:
emc hits:11227924
megaflow hits:1
miss:231
lost:0
polling cycles:18074152077 (66.87%)
processing cycles:8955323625 (33.13%)
avg cycles per packet: 2407.29 (27029475702/11228156)
avg processing cycles per packet: 797.58 (8955323625/11228156)
pmd thread numa_id 0 core_id 17:
emc hits:1774
megaflow hits:37
miss:32
lost:0
polling cycles:20839820676 (99.99%)
processing cycles:1977540 (0.01%)
avg cycles per packet: 11308626.27 (20841798216/1843)
avg processing cycles per packet: 1073.00 (1977540/1843)
pmd thread numa_id 0 core_id 4:
emc hits:11541300
megaflow hits:108
miss:24
lost:0
polling cycles:503392599 (1.55%)
processing cycles:32036341425 (98.45%)
avg cycles per packet: 2819.38 (32539734024/11541432)
avg processing cycles per packet: 2775.77 (32036341425/11541432)
pmd thread numa_id 0 core_id 5:
emc hits:25307647
megaflow hits:0
miss:226
lost:0
polling cycles:2461529511 (7.57%)
processing cycles:30065889423 (92.43%)
avg cycles per packet: 1285.27 (32527418934/25307873)
avg processing cycles per packet: 1188.01 (30065889423/25307873)
inside the vm there are only two out of the eight cores utilized.
when manually setting the irq affinity inside the guest and disableing irqbalance like so:
echo 0 > /proc/irq/45/smp_affinity_list //virtio0-input.0
echo 0 > /proc/irq/46/smp_affinity_list //virtio0-output.0
echo 1 > /proc/irq/47/smp_affinity_list //virtio0-input.1
echo 1 > /proc/irq/48/smp_affinity_list //...
echo 2 > /proc/irq/49/smp_affinity_list
echo 2 > /proc/irq/50/smp_affinity_list
echo 3 > /proc/irq/51/smp_affinity_list
echo 3 > /proc/irq/52/smp_affinity_list
echo 4 > /proc/irq/53/smp_affinity_list
echo 4 > /proc/irq/54/smp_affinity_list
echo 5 > /proc/irq/55/smp_affinity_list
echo 5 > /proc/irq/56/smp_affinity_list
echo 6 > /proc/irq/57/smp_affinity_list
echo 6 > /proc/irq/58/smp_affinity_list
echo 7 > /proc/irq/59/smp_affinity_list
echo 7 > /proc/irq/60/smp_affinity_list
echo 0 > /proc/irq/62/smp_affinity_list //virtio1-input.0
echo 0 > /proc/irq/63/smp_affinity_list //...
echo 1 > /proc/irq/64/smp_affinity_list
echo 1 > /proc/irq/65/smp_affinity_list
echo 2 > /proc/irq/66/smp_affinity_list
echo 2 > /proc/irq/67/smp_affinity_list
echo 3 > /proc/irq/68/smp_affinity_list
echo 3 > /proc/irq/69/smp_affinity_list
echo 4 > /proc/irq/70/smp_affinity_list
echo 4 > /proc/irq/71/smp_affinity_list
echo 5 > /proc/irq/72/smp_affinity_list
echo 5 > /proc/irq/73/smp_affinity_list
echo 6 > /proc/irq/74/smp_affinity_list
echo 6 > /proc/irq/75/smp_affinity_list
echo 7 > /proc/irq/76/smp_affinity_list
echo 7 > /proc/irq/77/smp_affinity_list
im getting the same result, only two cores used inside the vm.
Inside the vm: ifconfig does not return any dropped packets, so it seems the queues are not used.
Cat /proc/interrupts shows only
virtio0-input.0
virtio0-output.0
virtio0-output.1
virtio1-output.0
virtio1-input.1
generating interrupts, all other queues have generated zero or one interrupt since start.
Best regards
Felix
-----Ursprüngliche Nachricht-----
Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
Gesendet: Dienstag, 5. April 2016 12:14
An: Felix Brucker <FBrucker at xantaro.net>
Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does not work
Hi Felix,
It should work with as many pmd threads. I tested 8 queues with 1,2,4 and 8 pmd threads and each time traffic was able to reach the VM.
For example with 1 PMD thread
# sudo ./utilities/ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=1 # sudo ./utilities/ovs-appctl dpif-netdev/pmd-rxq-show pmd thread numa_id 0 core_id 0:
port: dpdkvhostuser0 queue-id: 0 1 2 3 4 5 6 7
port: dpdk0 queue-id: 0
You will get better performance if you assign a core to each queue though.
Thanks,
Ciara
> -----Original Message-----
> From: Felix Brucker [mailto:FBrucker at xantaro.net]
> Sent: Tuesday, April 05, 2016 11:06 AM
> To: Loftus, Ciara <ciara.loftus at intel.com>
> Subject: AW: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues
> does not work
>
> Hi Ciara,
>
> thanks, one question though:
>
> you wrote:
>
> If you want a pmd to service each of the 8 queues, you can set the
> number of PMD threads via:
> ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=FF This will
> set up
> 8 pmd threads, on cores 0-7.
>
> Does this mean i have to use as many host pmd threads as i want to use
> queues for the vhost-user?
> So using 2 host pmd threads and 8 vhost-user queues does not work?
>
> Best regards
> Felix
>
>
> -----Ursprüngliche Nachricht-----
> Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
> Gesendet: Dienstag, 5. April 2016 12:00
> An: Felix Brucker <FBrucker at xantaro.net>; Christian Ehrhardt
> <christian.ehrhardt at canonical.com>
> Cc: discuss at openvswitch.org
> Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues
> does not work
>
> >
> > Hi,
> >
> > with the branch-2.5 ovs and n-dpdk-rxqs option i was able to get
> > four queues with four pmd host threads, which currently is my limit
> > for vm queues in terms of usable cores and the command itself.
> >
> > To my understanding, with ovs post 2.5 (latest git master) i should
> > be able to use two n-dpdk-rxqs queues for the host and use ovs-vsctl
> > set Interface
> > vhost-user-0 options:n_rxq=8 to get eight queues inside the vm, is
> > this correct?
>
> Hi Felix,
>
> Hopefully my explanation below will help clear things up.
>
> Post-2.5 the 'n-dpdk-rxqs' option is not available to use any more.
> Here is a snippet from the commit message that removes this option
> (commit id
> a14b8947fd13d4c587addbffd24eedc7bb48ee2b)
>
> "dpif-netdev: Allow different numbers of rx queues for different ports.
>
> Currently, all of the PMD netdevs can only have the same number of rx
> queues, which is specified in other_config:n-dpdk-rxqs.
>
> Fix that by introducing of new option for PMD interfaces: 'n_rxq',
> which specifies the maximum number of rx queues to be created for this interface.
>
> Example:
> ovs-vsctl set Interface dpdk0 options:n_rxq=8
>
> Old 'other_config:n-dpdk-rxqs' deleted."
>
> In your case, now on latest master, if you want 8 queues assigned to
> vhost-
> user-0 in the guest you need to do the following:
>
> On the host:
> 1. ovs-vsctl set Interface vhost-user-0 options:n_rxq=8 2. QEMU:
> -chardev
> socket,id=char0,path=/path/to/vhost-user-0 -netdev type=vhost-
> user,id=mynet1,chardev=char0,vhostforce,queues=8 -device virtio-net-
> pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=off,mq=on,vectors=1
> 8
> Or if you're using libvirt I think the equivalent would be: <driver
> queues='8'/>
>
> On the VM:
> 3. Check queues available:
> [root at localhost ~]# ethtool -l eth0
> Channel parameters for eth0:
> Pre-set maximums:
> RX: 0
> TX: 0
> Other: 0
> Combined: 8
> Current hardware settings:
> RX: 0
> TX: 0
> Other: 0
> Combined: 1
> # Enable 8 queues
> [root at localhost ~]# ethtool -l eth0
> Channel parameters for eth0:
> Pre-set maximums:
> RX: 0
> TX: 0
> Other: 0
> Combined: 8
> Current hardware settings:
> RX: 0
> TX: 0
> Other: 0
> Combined: 8
>
> At this point your vhost-user-0 interface (eth0) on the guest can use
> 8 queues.
>
> If you want a pmd to service each of the 8 queues, you can set the
> number of PMD threads via:
> ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=FF This will
> set up
> 8 pmd threads, on cores 0-7.
>
> If you have other interfaces that you want to increase the number of
> rxqs for, you may do so like so:
> ovs-vsctl set Interface <iface> options:n_rxq=X
>
> Thanks,
> Ciara
>
> >
> > If so, im experiencing a problem were only two queues out of the
> > eight are used for traffic.
> >
> > Best regards
> > Felix
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: discuss [mailto:discuss-bounces at openvswitch.org] Im Auftrag von
> > Felix Brucker
> > Gesendet: Dienstag, 5. April 2016 09:48
> > An: Loftus, Ciara <ciara.loftus at intel.com>; Christian Ehrhardt
> > <christian.ehrhardt at canonical.com>
> > Cc: Daniele Di Proietto <diproiettod at vmware.com>;
> > discuss at openvswitch.org
> > Betreff: [MASSMAIL] Re: [ovs-discuss] [OVS-DPDK] vhost-user with
> > multiple queues does not work
> >
> > Hi Ciara,
> >
> > thanks that clarified it, i got confused by > Also this does NOT set
> > the multiqueues the guest shall get i read the Install md from here
> > http://openvswitch.org/support/dist-docs/INSTALL.DPDK.md.txt
> > i thought this is related to the download on the same site
> > (http://openvswitch.org/releases/openvswitch-2.5.0.tar.gz), but it
> > seems not to.
> > With the n-dpdk-rxqs=2 option i was able to get 2 queues inside the
> > vm and working communication, too.
> > After testing i will try to get the latest (post 2.5) version of ovs
> > to get a more fine grained control over the queues.
> > Thanks all!
> >
> > Best regards
> > Felix
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Loftus, Ciara [mailto:ciara.loftus at intel.com]
> > Gesendet: Montag, 4. April 2016 18:11
> > An: Felix Brucker <FBrucker at xantaro.net>; Christian Ehrhardt
> > <christian.ehrhardt at canonical.com>
> > Cc: Daniele Di Proietto <diproiettod at vmware.com>;
> > discuss at openvswitch.org
> > Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> > queues does not work
> >
> > > yes that part works, but for communication to work between the
> > > guest and host OVS has to use 2 queues as well, which currently does not work.
> > > So how does one set multiple queues for vhostuser in OVS 2.5.0 or
> below?
> > > Im not talking about libvirt or qemu regarding the above question,
> > > but
> OVS.
> >
> > Hi Felix,
> >
> > As we've mentioned before, you need to use the following command:
> >
> > ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2
> >
> > ... to assign two rx queues to the vhost-user ports in OVS.
> >
> > This is clearly stated in INSTALL.DPDK.md on the 2.5 branch. I
> > suspect you were previously looking at the latest INSTALL guide
> > which pointed you to use the n_rxq option, which is not available on the older branch-2.5.
> >
> > Essentially, if your bridge has two vhost-user ports eg.
> > vhost-user-0 and vhost-user-1, the effect of ' ovs-vsctl set Open_vSwitch .
> > other_config:n- dpdk-rxqs=2'
> > is the same as
> > ovs-vsctl set Interface vhost-user-0 options:n_rxq=2 ovs-vsctl set
> > Interface
> > vhost-user-1 options:n_rxq=2
> >
> > On branch-2.5, you need to use the former command.
> >
> > Thanks,
> > Ciara
> >
> > >
> > > Grüße
> > > Felix
> > >
> > > Von: Christian Ehrhardt [mailto:christian.ehrhardt at canonical.com]
> > > Gesendet: Montag, 4. April 2016 17:35
> > > An: Felix Brucker <FBrucker at xantaro.net>
> > > Cc: Daniele Di Proietto <diproiettod at vmware.com>; Loftus, Ciara
> > > <ciara.loftus at intel.com>; discuss at openvswitch.org
> > > Betreff: Re: [ovs-discuss] [OVS-DPDK] vhost-user with multiple
> > > queues does not work
> > >
> > > Hi Felix,
> > > here you already do the right thing:
> > >
> > > <interface type='vhostuser'>
> > > [...]
> > > <driver queues='2'/>
> > >
> > > Given you have the recent libvirt and qemu versions that
> > > translates to the right qemu parameters as you have seen in my initial posts.
> > >
> > > You can then log into the guest and check with "ethtool -l" if the
> > > guest really "sees" its multiple queues (also shown in my first
> > > mail that this fails for me)
> > >
> > > Kind Regards,
> > > Christian
> > _______________________________________________
> > discuss mailing list
> > discuss at openvswitch.org
> > http://openvswitch.org/mailman/listinfo/discuss
More information about the discuss
mailing list