[ovs-dev] Performance Issue when using netdev-dpdk and multiple PMD threads
Mauricio Vásquez
mauricio.vasquezbernal at studenti.polito.it
Thu Jan 7 19:01:04 UTC 2016
Hi to All,
I performed some testing using physical NICs and an external traffic
generator. The issue was not present in that case.
Could it be due to that when using physical NICs the data rate is slower
than when the traffic is generated locally?
What else should I check?
Thank you,
On 31 December 2015 at 09:20, Mauricio Vásquez <
mauricio.vasquezbernal at studenti.polito.it> wrote:
> I updated to the last git version and then applied your patch. The issue
> is still present.
>
> The ovs log still shows the message
> "2015-12-31T14:18:04Z|00001|dpif_netdev(pmd95)|INFO|Core 5 processing port
> 'dpdkr2'
> 2015-12-31T14:18:04Z|00001|dpif_netdev(pmd96)|INFO|Core 4 processing port
> 'dpdkr4'
> 2015-12-31T14:18:04Z|00001|dpif_netdev(pmd97)|INFO|Core 3 processing port
> 'dpdkr3'
> 2015-12-31T14:18:04Z|00001|dpif_netdev(pmd98)|INFO|Core 2 processing port
> 'dpdkr1'
> 2015-12-31T14:18:26Z|00002|ofproto_dpif_upcall(pmd97)|WARN|Dropped
> 16960873 log messages in last 60 seconds (most recently, 0 seconds ago) due
> to excessive rate
> 2015-12-31T14:18:26Z|00003|ofproto_dpif_upcall(pmd97)|WARN|upcall_cb
> failure: ukey installation fails"
>
> Thank you,
>
>
> On 31 December 2015 at 06:33, Ilya Maximets <i.maximets at samsung.com>
> wrote:
>
>> So, possibly, there is more than one bug here.
>> Please try my new patch "[PATCH RFC] dpif-netdev: Rework of rx queue
>> management."
>> http://openvswitch.org/pipermail/dev/2015-December/063920.html
>>
>> May be it will help.
>>
>> Best regards, Ilya Maximets.
>>
>> On 30.12.2015 18:25, Mauricio Vásquez wrote:
>> > Hello Ilya,
>> >
>> > I applied the patch but I still getting a low throughput and the
>> message "ofproto_dpif_upcall(pmd101)|WARN|upcall_cb failure: ukey
>> installation fails" in the ovs log.
>> >
>> >
>> >
>> > On 30 December 2015 at 09:59, Ilya Maximets <i.maximets at samsung.com
>> <mailto:i.maximets at samsung.com>> wrote:
>> >
>> > As I see, this is exactly the same bug as fixed in
>> > commit e4e74c3a2b ("dpif-netdev: Purge all ukeys when reconfigure
>> pmd.")
>> > but reproduced while only reconfiguring of pmd threads without
>> restarting.
>> >
>> >
>> > Try this patch as a workaround:
>> >
>> > diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
>> > index fe2cd4b..0008d30 100644
>> > --- a/lib/dpif-netdev.c
>> > +++ b/lib/dpif-netdev.c
>> > @@ -1027,11 +1027,9 @@ dp_netdev_reload_pmd__(struct
>> dp_netdev_pmd_thread *pmd)
>> > static void
>> > dp_netdev_reload_pmds(struct dp_netdev *dp)
>> > {
>> > - struct dp_netdev_pmd_thread *pmd;
>> > -
>> > - CMAP_FOR_EACH (pmd, node, &dp->poll_threads) {
>> > - dp_netdev_reload_pmd__(pmd);
>> > - }
>> > + dp_netdev_destroy_all_pmds(dp);
>> > + dp_netdev_set_nonpmd(dp);
>> > + dp_netdev_reset_pmd_threads(dp);
>> > }
>> >
>> > static uint32_t
>> > ---
>> >
>> > I'll try to prepare proper fix later.
>> >
>> > Best regards, Ilya Maximets.
>> >
>> > On 30.12.2015 17:49, Mauricio Vásquez wrote:
>> > > I have no idea, ovs was running for a long time when I took that
>> data.
>> > > I restarted everything and now the main thread shows:
>> > >
>> > > main thread:
>> > > emc hits:1316
>> > > megaflow hits:0
>> > > miss:681
>> > > lost:1348
>> > > polling cycles:7226622 (19.41%)
>> > > processing cycles:30002635 (80.59%)
>> > > avg cycles per packet: 18642.59 (37229257/1997)
>> > > avg processing cycles per packet: 15023.85 (30002635/1997)
>> > >
>> > > Other threads continue to show a high number of miss packets,
>> throughput is still 700 kpps.
>> > >
>> > >
>> > >
>> > >
>> > > On 30 December 2015 at 09:36, Ilya Maximets <
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>>> wrote:
>> > >
>> > > On 30.12.2015 17:32, Mauricio Vásquez wrote:
>> > > > I just checked and the traffic is generated after
>> everything is already set up, ports and flows.
>> > >
>> > > And what is this 50K packets in that case?
>> > >
>> > > main thread:
>> > > emc hits:20341
>> > > megaflow hits:0
>> > > miss:10193
>> > > lost:20372
>> > >
>> > > >
>> > > > On 30 December 2015 at 08:50, Ilya Maximets <
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>>>> wrote:
>> > > >
>> > > > The transmission starts before the addition of dpdkr4
>> to ovs?
>> > > >
>> > > > On 30.12.2015 16:31, Mauricio Vásquez wrote:
>> > > > > Dear Ilya,
>> > > > >
>> > > > > ovs-appctl dpif-netdev/pmd-stats-show ->
>> http://pastebin.com/k1nnMfQZ
>> > > > > ovs-appctl coverage/show ->
>> http://pastebin.com/617CYR4n
>> > > > > ovs-appctl dpctl/show -> http://pastebin.com/JFCT8tgS
>> > > > > ovs-log -> http://pastebin.com/sJkaF20M
>> > > > >
>> > > > >
>> > > > > Thank you very much.
>> > > > >
>> > > > > On 30 December 2015 at 08:05, Ilya Maximets <
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>>>>> wrote:
>> > > > >
>> > > > > On 30.12.2015 15:51, Mauricio Vásquez wrote:
>> > > > > > Hello Ilya,
>> > > > > >
>> > > > > > The dpdkr ports involved have just one TX
>> queue, so it should not be the reason in this case.
>> > > > > >
>> > > > >
>> > > > > Please, provide output of:
>> > > > > ovs-appctl dpif-netdev/pmd-stats-show
>> > > > > ovs-appctl coverage/show
>> > > > > ovs-appctl dpctl/show
>> > > > > and log of ovs-vswitchd.
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>
>
More information about the dev
mailing list