[ovs-dev] 64Byte packet performance regression on 2.9 from 2.7

Ilya Maximets i.maximets at samsung.com
Tue Jul 10 08:41:55 UTC 2018


Hi.
Your test results looks very strange.

1.  Why "bi-Direction" results for testpmd are lower than "1-Direction" ?
2.  Could you, please, configure EMC probability to 1 for 2.8 and 2.9 cases.
    This could make tests more stable.
3.  Why you're using "n_rxq_desc=256" for physical NICs? If you're testing
    a zero packet drop rates, this could be a serious issue.
    Why not OVS default 2048 ?
4.  I hardly believe that testpmd could not achieve 10G line rate (14 Mpps)
    in this simple scenario. What CPU you're using? Is it low-power ARM core?
5.  "Flame Graph"s are not much useful to check 10% performance difference.
6.  Looks like second PMD is idle in Flame Graph for 2.9.
7.  "-n 7"... Hmm, you have some really interesting system.
8.  Is there some reason to fix mbuf/mempool sizes to particular values?
9.  How much time your test works for one case?
10. What is the NUMA/threads topology of your system? (To be sure that your
    cpu cores are not siblings/located on different NUMA)

Best regards, Ilya Maximets.

On 09.07.2018 20:36, Jay Ding wrote:
> Hi Ilya,
>
> Here is the test result for performance of OVS-2.7, 2.8, and 2.9. We do see the performance drops by 10% from OVS-2.7 to OVS-2.9. The performance of DPDK only (testpmd) is same for those versions. Please refer to the table for the versions of DPDK we tested. 
>
> The setup is Ixia <->PF0(OVS)PF1<->Ixia. Only one rxq is used. The test was running on Intel 82599ES 10-G.
>
> I attached the profiles for each version for your reference. We also recorded the profiles with O0. I can send them over if you need them. 
>
>   	-g -O2' 1-Direction  	  	-g -O2' bi-Direction 	  	-g -O0' 1-Direction 	  	-g -O0' bi-Direction 	 
> Intel 82599ES 10-Gigabit SFI/SFP+ 	  	  	  	  	  	  	  	 
> rxq=1 , pmd-rxq-affinity=0:1/0:3 	Mpps 	drop 	Mpps 	drop 	Mpps 	drop 	Mpps 	drop
> DPDK-16.11.7/OVS-2.7  	8.56 	  	16.66 	  	2.01 	  	4.02 	 
> DPDK-17.05.2/OVS-2.8 	8.38 	2% 	16.46 	1% 	1.96 	2% 	3.92 	2%
> DPDK-17.11.3/OVS-2.9 	7.73 	10% 	15.18 	9% 	1.78 	11% 	3.56 	11%
> Testpmd --rxq=1 --txq=1 	  	  	  	  	  	  	  	 
> DPDK-16.11.7 	12.05 	  	9.08 	  	  	  	  	 
> DPDK-17.05.2 	12.05 	  	9.08 	  	  	  	  	 
> DPDK-17.11.3 	12.05 	  	9.08 	  	  	  	  	 
>
>
> OVS setup (only list the steps with parameters):
> ./utilities/ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0xfffe
> ./utilities/ovs-vsctl add-port br0 dpdk128 -- set Interface dpdk128 type=dpdk options:dpdk-devargs=0000:05:00.0 options:n_rxq_desc=256 options:n_txq_desc=1024 ofport_request=1
> ./utilities/ovs-vsctl add-port br0 dpdk129 -- set Interface dpdk129 type=dpdk options:dpdk-devargs=0000:05:00.1 options:n_rxq_desc=256 options:n_txq_desc=1024 ofport_request=2
> ./utilities/ovs-vsctl set Interface dpdk128 options:n_rxq=1 other_config:pmd-rxq-affinity="0:1"
> ./utilities/ovs-vsctl set Interface dpdk129 options:n_rxq=1 other_config:pmd-rxq-affinity="0:3"
>
> Testpmd setup:
> build/app/testpmd -c 0xfe -n 7 -w 0000:05:00.0 -w 0000:05:00.1 -mbuf-size=4096 -- --total-num-mbufs=409600 -i --nb-cores=6 --rxq=1 --txq=1 --rxd=1024 --txd=1024 --port-topology=paired
>
> Please let us know if you need more information.
>
> Thanks,
>
> Jay
>
> On Tue, Jul 3, 2018 at 6:44 AM, Shahaji Bhosle <shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>> wrote:
>
>     Thanks Nitin.
>     Hi Ilya,
>     Looks like regression was in 2.8.x itself as per Nitin's email. We will update our results as well. Thanks, Shahaji
>
>     On Tue, Jul 3, 2018 at 12:29 AM, Nitin Katiyar <nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>> wrote:
>
>         Hi,
>         I had tested 2.8.1/2 earlier which uses 17.05.01 or 17.05.02 and found around 10% drop for udp traffic. OVS 2.7.4 gave the similar result as OVS 2.6.2 (DPDK 16.11.4). I was using Intel Niantic 82599 for testing.
>
>         Regards,
>         Nitin
>
>         -----Original Message-----
>         From: Ilya Maximets [mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>]
>         Sent: Monday, July 02, 2018 10:25 PM
>         To: Shahaji Bhosle <shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>
>         Cc: Jan Scheurich <jan.scheurich at ericsson.com <mailto:jan.scheurich at ericsson.com>>; Jay Ding <jay.ding at broadcom.com <mailto:jay.ding at broadcom.com>>; Kevin Traynor <ktraynor at redhat.com <mailto:ktraynor at redhat.com>>; Manasa Mudireddy <manasa.mudireddy at broadcom.com <mailto:manasa.mudireddy at broadcom.com>>; Nitin Katiyar <nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>; Randy Schacher <stuart.schacher at broadcom.com <mailto:stuart.schacher at broadcom.com>>; Stokes, Ian <ian.stokes at intel.com <mailto:ian.stokes at intel.com>>; ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>
>         Subject: Re: [ovs-dev] 64Byte packet performance regression on 2.9 from 2.7
>
>         Sure, you need to collect perf records for the same binary, i.e. built with the same compiler options (and on the same machine), to make them useful.
>
>         Unfortunately, I have no setup to test your case right now.
>         Data for 2.8 could help bisecting the issue.
>
>         On 02.07.2018 18:04, Shahaji Bhosle wrote:
>         > Hi Ilya,
>         > Thanks for the reply.
>         > For performance traffic testing we are running with -O2. You are right about the perf report, when were running with perf record we had set "-g -O0". Do you need us to run with just "-g -O2" and give you the profile, or any other optimization setting.
>         > Do you have a test setup for running 64B packets, and see the difference between 2.7 and 2.9? On our side we are trying to get 2.8 to work so we can give you an intermediate data point. Please let us know what we can do to help you debug this.
>         > Thanks, Shahaji
>         >
>         >
>         > On Mon, Jul 2, 2018 at 10:55 AM, Ilya Maximets <i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>> wrote:
>         >
>         >     Hi.
>         >     Sorry for late response.
>         >
>         >     Looking at your perf data, I see functions like "dp_packet_batch_size"
>         >     consuming ~0.5 - 0.7 % of time. Are you building with all compiler
>         >     optimizations disabled? Otherwise where should be no such symbols in
>         >     perf report. They should be completely inlined.
>         >
>         >     Best regards, Ilya Maximets.
>         >
>         >     On 27.06.2018 04:48, Shahaji Bhosle wrote:
>         >     > Hi Ilya,
>         >     > Just wanted to check if you found anything interesting. Or anything we can try. Thanks, Shahaji
>         >     >
>         >     > On Wed, Jun 20, 2018 at 9:01 AM, Shahaji Bhosle <shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>>> wrote:
>         >     >
>         >     >     Thanks Ilya, 
>         >     >      Sorry for the confusion with the number, we used to get some different numbers on both ports so were recording it per port. You have to compare it with the two port number....
>         >     >
>         >     >               CPU mask        Mpps
>         >     >     17.11 testpmd     6 queue 0xfe    21.5 + 21.5
>         >     >     OvS 2.9+DPDK17.11 6 queue 0xfe    15.5 + 15.5
>         >     >     16.11 testpmd     6 queue 0xfe    21.5 + 21.5
>         >     >     OvS 2.7+DPDK16.11 6 queue 0xfe    17.4+17.4
>         >     >
>         >     >
>         >     >     Thanks, Shahaji
>         >     >
>         >     >     On Wed, Jun 20, 2018 at 8:34 AM, Ilya Maximets <i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>>> wrote:
>         >     >
>         >     >         Ok, I'll look at the data later.
>         >     >
>         >     >         But your testpmd results are much lower than OVS results. 21.5Mpps for testpmd
>         >     >         versus 33.8Mpps for OVS. OVS should work slower than testpmd, because it performs
>         >     >         a lot of parsing and processing while testpmd does not.
>         >     >         You probably tested testpmd in deifferent environment or allocated less amount
>         >     >         of resources for PMD htreads. Could you please recheck?
>         >     >
>         >     >         What is your OVS configuration (pmd-cpu-mask, n_rxqs etc.)?
>         >     >         And what is your testpmd command-line?
>         >     >
>         >     >         On 20.06.2018 14:54, Shahaji Bhosle wrote:
>         >     >         > Thanks Ilya,
>         >     >         > Attaching the two perf reports...We did run testpmd on its own, there were no red flags there. In some of the cases like flowgen 17.11 performs much better than 16.11, but for the macswap case, the numbers are below. Let me know if you cannot see the attached perf reports. I can just cut and paste them in the email if attachment does not work. Sorry I am not sure I can post these on any outside servers. Let me know
>         >     >         > Thanks, Shahaji
>         >     >         >
>         >     >         > *DPDK on Maia (macswap)*      *Rings*         *Mpps*  *Cycles/Packet*
>         >     >         > 17.11 testpmd                 6 queue         21.5 + 21.5     60
>         >     >         >                               1 queue         10.4+10.4       14
>         >     >         > 16.11 testpmd                 6 queue         21.5 + 21.5     60
>         >     >         >                               1 queue         10.4+10.4       14
>         >     >         >
>         >     >         >
>         >     >         > On Wed, Jun 20, 2018 at 4:52 AM, Ilya Maximets <i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>>>> wrote:
>         >     >         >
>         >     >         >     Looking at your perf stats I see following:
>         >     >         >
>         >     >         >     OVS 2.7:
>         >     >         >
>         >     >         >       ??.??% - dp_netdev_process_rxq_port
>         >     >         >       |-- 93.36% - dp_netdev_input
>         >     >         >       |-- ??.??% - netdev_rxq_recv
>         >     >         >
>         >     >         >     OVS 2.9:
>         >     >         >
>         >     >         >       99.69% - dp_netdev_process_rxq_port
>         >     >         >       |-- 79.45% - dp_netdev_input
>         >     >         >       |-- 11.26% - dp_netdev_pmd_flush_output_packets
>         >     >         >       |-- ??.??% - netdev_rxq_recv
>         >     >         >
>         >     >         >     Could you please fill the missed (??.??) values?
>         >     >         >     This data I got from the picture attached to the previous mail, but pictures
>         >     >         >     are still not allowed in mail-list (i.e. stripped). It'll be good if you can
>         >     >         >     upload your raw data to some external resource and post the link here.
>         >     >         >
>         >     >         >     Anyway, from the data I have, I can see that total sum of time spent in
>         >     >         >     "dp_netdev_input" and "dp_netdev_pmd_flush_output_packets" for 2.9 is 90.71%,
>         >     >         >     which is less then 93.36% spent for 2.7. This means that processing + sending
>         >     >         >     become even faster or remains with the approximately same performance.
>         >     >         >     We definitely need all the missed values to be sure, but it seems that the
>         >     >         >     "netdev_rxq_recv()" could be the issue.
>         >     >         >
>         >     >         >     To check if DPDK itself causes the performance regression, I'd ask you
>         >     >         >     to check pure PHY-PHY test with testpmd app from DPDK 16.11 and DPDK 17.11.
>         >     >         >     Maybe it's the performance issue with bnxt driver that you're using.
>         >     >         >     There was too many changes in that driver:
>         >     >         >
>         >     >         >       30 files changed, 17189 insertions(+), 3358 deletions(-)
>         >     >         >
>         >     >         >     Best regards, Ilya Maximets.
>         >     >         >
>         >     >         >     On 20.06.2018 01:18, Shahaji Bhosle wrote:
>         >     >         >     > Hi Ilya,
>         >     >         >     > This issue is a release blocker for us, just wanted to check check if you need more details from us? Anything to expedite or root cause the problem we can help 
>         >     >         >     > Please let us know 
>         >     >         >     > Thanks Shahaji 
>         >     >         >     >
>         >     >         >     > On Mon, Jun 18, 2018 at 10:20 AM Shahaji Bhosle <shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>>
>         <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>>>>> wrote:
>         >     >         >     >
>         >     >         >     >     Thanks Ilya, I will look at the commit, but not sure now how to tell how much real work is being done, I would have liked polling cycles to be treated as before and not towards packet processing. That does explain, as long as there are packets on the wire we are always 100%, basically cannot tell how efficiently the CPUs are being used.
>         >     >         >     >     Thanks, Shahaji
>         >     >         >     >
>         >     >         >     >     On Mon, Jun 18, 2018 at 10:07 AM, Ilya Maximets <i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>
>         <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>>>>> wrote:
>         >     >         >     >
>         >     >         >     >         Thanks for the data.
>         >     >         >     >
>         >     >         >     >         I have to note additionally that the meaning of "processing cycles"
>         >     >         >     >         significantly changed since the following commit:
>         >     >         >     >
>         >     >         >     >             commit a2ac666d5265c01661e189caac321d962f54649f
>         >     >         >     >             Author: Ciara Loftus <ciara.loftus at intel.com <mailto:ciara.loftus at intel.com> <mailto:ciara.loftus at intel.com <mailto:ciara.loftus at intel.com>> <mailto:ciara.loftus at intel.com <mailto:ciara.loftus at intel.com> <mailto:ciara.loftus at intel.com <mailto:ciara.loftus at intel.com>>> <mailto:ciara.loftus at intel.com <mailto:ciara.loftus at intel.com> <mailto:ciara.loftus at intel.com <mailto:ciara.loftus at intel.com>> <mailto:ciara.loftus at intel.com <mailto:ciara.loftus at intel.com> <mailto:ciara.loftus at intel.com <mailto:ciara.loftus at intel.com>>>> <mailto:ciara.loftus at intel.com <mailto:ciara.loftus at intel.com> <mailto:ciara.loftus at intel.com <mailto:ciara.loftus at intel.com>> <mailto:ciara.loftus at intel.com <mailto:ciara.loftus at intel.com> <mailto:ciara.loftus at intel.com <mailto:ciara.loftus at intel.com>>> <mailto:ciara.loftus at intel.com <mailto:ciara.loftus at intel.com> <mailto:ciara.loftus at intel.com <mailto:ciara.loftus at intel.com>> <mailto:ciara.loftus at intel.com
>         <mailto:ciara.loftus at intel.com> <mailto:ciara.loftus at intel.com <mailto:ciara.loftus at intel.com>>>>>>
>         >     >         >     >             Date:   Mon Feb 20 12:53:00 2017 +0000
>         >     >         >     >
>         >     >         >     >                 dpif-netdev: Change definitions of 'idle' & 'processing' cycles
>         >     >         >     >
>         >     >         >     >                 Instead of counting all polling cycles as processing cycles, only count
>         >     >         >     >                 the cycles where packets were received from the polling.
>         >     >         >     >
>         >     >         >     >         This could explain the difference in "PMD Processing Cycles" column,
>         >     >         >     >         because successful "POLLING" cycles are now included into "PROCESSING".
>         >     >         >     >
>         >     >         >     >         Best regards, Ilya Maximets.
>         >     >         >     >
>         >     >         >     >         On 18.06.2018 16:31, Shahaji Bhosle wrote:
>         >     >         >     >         > Hi Ilya,
>         >     >         >     >         > Thanks for the quick reply, 
>         >     >         >     >         > Please find the numbers for our PHY-PHY test, please note that with OVS 2.9.1 + DPDK 17.11 even a 10% of the below numbers will make the OVS 2.9+DPDK17.11 processing cycles to hit 100%, but 2.7 will on our setup never goes above 75% for processing cycles. I am also attaching the perf report between the two code bases and I think the  "11.26%--dp_netdev_pmd_flush_output_packets" is causing us to take the performance hit. Out testing is also SRIOV and CPUs are ARM A72 cores. We are happy to run more tests, it is not easy for use to move back to OVS 2.8, but could happy to try more experiments if it helps us narrow down further. Please note we have also tried increasing the tx-flush-interval and it helps a little but still not significant enough. Let us know.
>         >     >         >     >         >
>         >     >         >     >         > Thanks, Shahaji 
>         >     >         >     >         >
>         >     >         >     >         >
>         >     >         >     >         > *Setup:*
>         >     >         >     >         > IXIA<----SFP28--->Port 0 {(PF0)==[OVS+DPDK]==(PF1)} Port 1<-----SFP28---->IXIA
>         >     >         >     >         >
>         >     >         >     >         > release/version       config  Test    direction       MPPS    Ixia Line rate (%)      PMD Processing Cycles (%)
>         >     >         >     >         > OVS 2.9 + DPDK 17.11  OVS on Maia (PF0--PF1)  No drop port 1 to 2     31.3    85      99.9
>         >     >         >     >         >                                                       port 2 to 1     31.3    85      99.9
>         >     >         >     >         >                                                       bi      15.5 + 15.5     42      99.9
>         >     >         >     >         >                                               
>         >     >         >     >         >                                               
>         >     >         >     >         > OVS 2.7 + DPDK 16.11  OVS on Maia (PF0--PF1)  No drop port 1 to 2     33.8    90      71
>         >     >         >     >         >                                                       port 2 to 1     32.7    88      70
>         >     >         >     >         >                                                       bi      17.4+17.4       47      74
>         >     >         >     >         >
>         >     >         >     >         >
>         >     >         >     >         >
>         >     >         >     >         >
>         >     >         >     >         >
>         >     >         >     >         >
>         >     >         >     >         >
>         >     >         >     >         > On Mon, Jun 18, 2018 at 4:25 AM, Nitin Katiyar <nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>> <mailto:nitin.katiyar at ericsson.com
>         <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>>>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>
>         <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>
>         >     <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>>>>>> wrote:
>         >     >         >     >         >
>         >     >         >     >         >     Hi,
>         >     >         >     >         >     We also experienced degradation from OVS2.6/2.7 to OVS2.8.2(with DPDK17.05.02). The drop is more for 64 bytes packet size (~8-10%) even with higher number of flows. I tried OVS 2.8 with DPDK17.11 and it improved for higher packet sizes but 64 bytes size is still the concern.
>         >     >         >     >         >
>         >     >         >     >         >     Regards,
>         >     >         >     >         >     Nitin
>         >     >         >     >         >
>         >     >         >     >         >     -----Original Message-----
>         >     >         >     >         >     From: Ilya Maximets [mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>
>         <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>>>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>
>         <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>
>         >     <mailto:i.maximets at samsung.com <mailto:i.maximets at samsung.com>>>>>>]
>         >     >         >     >         >     Sent: Monday, June 18, 2018 1:32 PM
>         >     >         >     >         >     To: ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>>> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>>>> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>>> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>>
>         <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>>>>> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>>> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>>>> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>>> <mailto:ovs-dev at openvswitch.org
>         <mailto:ovs-dev at openvswitch.org>
>         >     <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>>>>>>; shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com
>         <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>>>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com
>         <mailto:shahaji.bhosle at broadcom.com>>>>
>         >     >         <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>>>>>
>         >     >         >     >         >     Subject: Re: [ovs-dev] 64Byte packet performance regression on 2.9 from 2.7
>         >     >         >     >         >
>         >     >         >     >         >     CC: Shahaji Bhosle
>         >     >         >     >         >
>         >     >         >     >         >     Sorry, missed you in CC list.
>         >     >         >     >         >
>         >     >         >     >         >     Best regards, Ilya Maximets.
>         >     >         >     >         >
>         >     >         >     >         >     On 15.06.2018 10:44, Ilya Maximets wrote:
>         >     >         >     >         >     >> Hi,
>         >     >         >     >         >     >> I just upgraded from OvS 2.7 + DPDK 16.11 to OvS2.9 + DPDK 17.11 and
>         >     >         >     >         >     >> running into performance issue with 64 Byte packet rate. One
>         >     >         >     >         >     >> interesting thing that I notice that even at very light load from
>         >     >         >     >         >     >> IXIA the processing cycles on all the PMD threads run close to 100%
>         >     >         >     >         >     >> of the cpu cycle on 2.9 OvS, but on OvS 2.7 even under full load the
>         >     >         >     >         >     >> processing cycles remain at 75% of the cpu cycles.
>         >     >         >     >         >     >>
>         >     >         >     >         >     >> Attaching the FlameGraphs of both the versions, the only thing that
>         >     >         >     >         >     >> pops out to me is the new way invoking netdev_send() is on 2.9 is
>         >     >         >     >         >     >> being invoked via  dp_netdev_pmd_flush_output_packets() which seems
>         >     >         >     >         >     >> to be adding another ~11% to the whole rx to tx path.
>         >     >         >     >         >     >>
>         >     >         >     >         >     >> I also did try the tx-flush-interval to 50 and more it does seem to
>         >     >         >     >         >     >> help, but not significant enough to match the 2.7 performance.
>         >     >         >     >         >     >>
>         >     >         >     >         >     >>
>         >     >         >     >         >     >> Any help or ideas would be really great. Thanks, Shahaji
>         >     >         >     >         >     >
>         >     >         >     >         >     > Hello, Shahaji.
>         >     >         >     >         >     > Could you, please, describe your testing scenario in more details?
>         >     >         >     >         >     > Also, mail-list filters attachments, so they are not available. You
>         >     >         >     >         >     > need to publish them somewhere else or write in text format inside the letter.
>         >     >         >     >         >     >
>         >     >         >     >         >     > About the performance itself: Some performance degradation because of
>         >     >         >     >         >     > output batching is expected for tests with low number of flows or
>         >     >         >     >         >     > simple PHY-PHY tests. It was mainly targeted for cases with relatively
>         >     >         >     >         >     > large number of flows, for amortizing of vhost-user penalties
>         >     >         >     >         >     > (PHY-VM-PHY, VM-VM cases), OVS bonding cases.
>         >     >         >     >         >     >
>         >     >         >     >         >     > If your test involves vhost-user ports, then you should also consider
>         >     >         >     >         >     > vhost-user performance regression in stable DPDK 17.11 because of
>         >     >         >     >         >     > fixes for CVE-2018-1059. Related bug:
>         >     >         >     >         >     >       https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48> <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48>> <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48> <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48>>> <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48> <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48>> <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48> <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48>>>> <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48> <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48>>
>         <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48> <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48>>> <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48> <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48>> <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48> <https://dpdk.org/tracker/show_bug.cgi?id=48 <https://dpdk.org/tracker/show_bug.cgi?id=48>>>>>
>         >     >         >     >         >     >
>         >     >         >     >         >     > It'll be good if you'll be able to test OVS 2.8 + DPDK 17.05. There
>         >     >         >     >         >     > was too many changes since 2.7. It'll be hard to track down the root cause.
>         >     >         >     >         >     >
>         >     >         >     >         >     > Best regards, Ilya Maximets.
>         >     >         >     >         >     >
>         >     >         >     >         >
>         >     >         >     >         >
>         >     >         >     >
>         >     >         >     >
>         >     >         >
>         >     >         >
>         >     >
>         >     >
>         >     >
>         >
>         >
>
>
>



More information about the dev mailing list