[ovs-dev] 64Byte packet performance regression on 2.9 from 2.7

Jay Ding jay.ding at broadcom.com
Mon Jul 9 17:36:10 UTC 2018


Hi Ilya,

Here is the test result for performance of OVS-2.7, 2.8, and 2.9. We do see
the performance drops by 10% from OVS-2.7 to OVS-2.9. The performance of
DPDK only (testpmd) is same for those versions. Please refer to the table
for the versions of DPDK we tested.

The setup is Ixia <->PF0(OVS)PF1<->Ixia. Only one rxq is used. The test was
running on Intel 82599ES 10-G.

I attached the profiles for each version for your reference. We also
recorded the profiles with O0. I can send them over if you need them.

  -g -O2' 1-Direction    -g -O2' bi-Direction   -g -O0' 1-Direction   -g
-O0' bi-Direction
Intel 82599ES 10-Gigabit SFI/SFP+
rxq=1 , pmd-rxq-affinity=0:1/0:3 Mpps drop Mpps drop Mpps drop Mpps drop
DPDK-16.11.7/OVS-2.7  8.56   16.66   2.01   4.02
DPDK-17.05.2/OVS-2.8 8.38 2% 16.46 1% 1.96 2% 3.92 2%
DPDK-17.11.3/OVS-2.9 7.73 10% 15.18 9% 1.78 11% 3.56 11%
Testpmd --rxq=1 --txq=1
DPDK-16.11.7 12.05   9.08
DPDK-17.05.2 12.05   9.08
DPDK-17.11.3 12.05   9.08
OVS setup (only list the steps with parameters):
./utilities/ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0xfffe
./utilities/ovs-vsctl add-port br0 dpdk128 -- set Interface dpdk128
type=dpdk options:dpdk-devargs=0000:05:00.0 options:n_rxq_desc=256
options:n_txq_desc=1024 ofport_request=1
./utilities/ovs-vsctl add-port br0 dpdk129 -- set Interface dpdk129
type=dpdk options:dpdk-devargs=0000:05:00.1 options:n_rxq_desc=256
options:n_txq_desc=1024 ofport_request=2
./utilities/ovs-vsctl set Interface dpdk128 options:n_rxq=1
other_config:pmd-rxq-affinity="0:1"
./utilities/ovs-vsctl set Interface dpdk129 options:n_rxq=1
other_config:pmd-rxq-affinity="0:3"

Testpmd setup:
build/app/testpmd -c 0xfe -n 7 -w 0000:05:00.0 -w 0000:05:00.1
-mbuf-size=4096 -- --total-num-mbufs=409600 -i --nb-cores=6 --rxq=1 --txq=1
--rxd=1024 --txd=1024 --port-topology=paired

Please let us know if you need more information.

Thanks,

Jay

On Tue, Jul 3, 2018 at 6:44 AM, Shahaji Bhosle <shahaji.bhosle at broadcom.com>
wrote:

> Thanks Nitin.
> Hi Ilya,
> Looks like regression was in 2.8.x itself as per Nitin's email. We will
> update our results as well. Thanks, Shahaji
>
> On Tue, Jul 3, 2018 at 12:29 AM, Nitin Katiyar <nitin.katiyar at ericsson.com
> > wrote:
>
>> Hi,
>> I had tested 2.8.1/2 earlier which uses 17.05.01 or 17.05.02 and found
>> around 10% drop for udp traffic. OVS 2.7.4 gave the similar result as OVS
>> 2.6.2 (DPDK 16.11.4). I was using Intel Niantic 82599 for testing.
>>
>> Regards,
>> Nitin
>>
>> -----Original Message-----
>> From: Ilya Maximets [mailto:i.maximets at samsung.com]
>> Sent: Monday, July 02, 2018 10:25 PM
>> To: Shahaji Bhosle <shahaji.bhosle at broadcom.com>
>> Cc: Jan Scheurich <jan.scheurich at ericsson.com>; Jay Ding <
>> jay.ding at broadcom.com>; Kevin Traynor <ktraynor at redhat.com>; Manasa
>> Mudireddy <manasa.mudireddy at broadcom.com>; Nitin Katiyar <
>> nitin.katiyar at ericsson.com>; Randy Schacher <stuart.schacher at broadcom.com
>> >; Stokes, Ian <ian.stokes at intel.com>; ovs-dev at openvswitch.org
>> Subject: Re: [ovs-dev] 64Byte packet performance regression on 2.9 from
>> 2.7
>>
>> Sure, you need to collect perf records for the same binary, i.e. built
>> with the same compiler options (and on the same machine), to make them
>> useful.
>>
>> Unfortunately, I have no setup to test your case right now.
>> Data for 2.8 could help bisecting the issue.
>>
>> On 02.07.2018 18:04, Shahaji Bhosle wrote:
>> > Hi Ilya,
>> > Thanks for the reply.
>> > For performance traffic testing we are running with -O2. You are right
>> about the perf report, when were running with perf record we had set "-g
>> -O0". Do you need us to run with just "-g -O2" and give you the profile, or
>> any other optimization setting.
>> > Do you have a test setup for running 64B packets, and see the
>> difference between 2.7 and 2.9? On our side we are trying to get 2.8 to
>> work so we can give you an intermediate data point. Please let us know what
>> we can do to help you debug this.
>> > Thanks, Shahaji
>> >
>> >
>> > On Mon, Jul 2, 2018 at 10:55 AM, Ilya Maximets <i.maximets at samsung.com
>> <mailto:i.maximets at samsung.com>> wrote:
>> >
>> >     Hi.
>> >     Sorry for late response.
>> >
>> >     Looking at your perf data, I see functions like
>> "dp_packet_batch_size"
>> >     consuming ~0.5 - 0.7 % of time. Are you building with all compiler
>> >     optimizations disabled? Otherwise where should be no such symbols in
>> >     perf report. They should be completely inlined.
>> >
>> >     Best regards, Ilya Maximets.
>> >
>> >     On 27.06.2018 04:48, Shahaji Bhosle wrote:
>> >     > Hi Ilya,
>> >     > Just wanted to check if you found anything interesting. Or
>> anything we can try. Thanks, Shahaji
>> >     >
>> >     > On Wed, Jun 20, 2018 at 9:01 AM, Shahaji Bhosle <
>> shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:
>> shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>> wrote:
>> >     >
>> >     >     Thanks Ilya,
>> >     >      Sorry for the confusion with the number, we used to get some
>> different numbers on both ports so were recording it per port. You have to
>> compare it with the two port number....
>> >     >
>> >     >               CPU mask        Mpps
>> >     >     17.11 testpmd     6 queue 0xfe    21.5 + 21.5
>> >     >     OvS 2.9+DPDK17.11 6 queue 0xfe    15.5 + 15.5
>> >     >     16.11 testpmd     6 queue 0xfe    21.5 + 21.5
>> >     >     OvS 2.7+DPDK16.11 6 queue 0xfe    17.4+17.4
>> >     >
>> >     >
>> >     >     Thanks, Shahaji
>> >     >
>> >     >     On Wed, Jun 20, 2018 at 8:34 AM, Ilya Maximets <
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>>> wrote:
>> >     >
>> >     >         Ok, I'll look at the data later.
>> >     >
>> >     >         But your testpmd results are much lower than OVS results.
>> 21.5Mpps for testpmd
>> >     >         versus 33.8Mpps for OVS. OVS should work slower than
>> testpmd, because it performs
>> >     >         a lot of parsing and processing while testpmd does not.
>> >     >         You probably tested testpmd in deifferent environment or
>> allocated less amount
>> >     >         of resources for PMD htreads. Could you please recheck?
>> >     >
>> >     >         What is your OVS configuration (pmd-cpu-mask, n_rxqs
>> etc.)?
>> >     >         And what is your testpmd command-line?
>> >     >
>> >     >         On 20.06.2018 14:54, Shahaji Bhosle wrote:
>> >     >         > Thanks Ilya,
>> >     >         > Attaching the two perf reports...We did run testpmd on
>> its own, there were no red flags there. In some of the cases like flowgen
>> 17.11 performs much better than 16.11, but for the macswap case, the
>> numbers are below. Let me know if you cannot see the attached perf reports.
>> I can just cut and paste them in the email if attachment does not work.
>> Sorry I am not sure I can post these on any outside servers. Let me know
>> >     >         > Thanks, Shahaji
>> >     >         >
>> >     >         > *DPDK on Maia (macswap)*      *Rings*         *Mpps*
>> *Cycles/Packet*
>> >     >         > 17.11 testpmd                 6 queue         21.5 +
>> 21.5     60
>> >     >         >                               1 queue
>>  10.4+10.4       14
>> >     >         > 16.11 testpmd                 6 queue         21.5 +
>> 21.5     60
>> >     >         >                               1 queue
>>  10.4+10.4       14
>> >     >         >
>> >     >         >
>> >     >         > On Wed, Jun 20, 2018 at 4:52 AM, Ilya Maximets <
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>>>> wrote:
>> >     >         >
>> >     >         >     Looking at your perf stats I see following:
>> >     >         >
>> >     >         >     OVS 2.7:
>> >     >         >
>> >     >         >       ??.??% - dp_netdev_process_rxq_port
>> >     >         >       |-- 93.36% - dp_netdev_input
>> >     >         >       |-- ??.??% - netdev_rxq_recv
>> >     >         >
>> >     >         >     OVS 2.9:
>> >     >         >
>> >     >         >       99.69% - dp_netdev_process_rxq_port
>> >     >         >       |-- 79.45% - dp_netdev_input
>> >     >         >       |-- 11.26% - dp_netdev_pmd_flush_output_packets
>> >     >         >       |-- ??.??% - netdev_rxq_recv
>> >     >         >
>> >     >         >     Could you please fill the missed (??.??) values?
>> >     >         >     This data I got from the picture attached to the
>> previous mail, but pictures
>> >     >         >     are still not allowed in mail-list (i.e. stripped).
>> It'll be good if you can
>> >     >         >     upload your raw data to some external resource and
>> post the link here.
>> >     >         >
>> >     >         >     Anyway, from the data I have, I can see that total
>> sum of time spent in
>> >     >         >     "dp_netdev_input" and "dp_netdev_pmd_flush_output_packets"
>> for 2.9 is 90.71%,
>> >     >         >     which is less then 93.36% spent for 2.7. This means
>> that processing + sending
>> >     >         >     become even faster or remains with the
>> approximately same performance.
>> >     >         >     We definitely need all the missed values to be
>> sure, but it seems that the
>> >     >         >     "netdev_rxq_recv()" could be the issue.
>> >     >         >
>> >     >         >     To check if DPDK itself causes the performance
>> regression, I'd ask you
>> >     >         >     to check pure PHY-PHY test with testpmd app from
>> DPDK 16.11 and DPDK 17.11.
>> >     >         >     Maybe it's the performance issue with bnxt driver
>> that you're using.
>> >     >         >     There was too many changes in that driver:
>> >     >         >
>> >     >         >       30 files changed, 17189 insertions(+), 3358
>> deletions(-)
>> >     >         >
>> >     >         >     Best regards, Ilya Maximets.
>> >     >         >
>> >     >         >     On 20.06.2018 01:18, Shahaji Bhosle wrote:
>> >     >         >     > Hi Ilya,
>> >     >         >     > This issue is a release blocker for us, just
>> wanted to check check if you need more details from us? Anything to
>> expedite or root cause the problem we can help
>> >     >         >     > Please let us know
>> >     >         >     > Thanks Shahaji
>> >     >         >     >
>> >     >         >     > On Mon, Jun 18, 2018 at 10:20 AM Shahaji Bhosle <
>> shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:
>> shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>
>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>
>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>>
>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>
>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>
>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>
>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>>>>
>> wrote:
>> >     >         >     >
>> >     >         >     >     Thanks Ilya, I will look at the commit, but
>> not sure now how to tell how much real work is being done, I would have
>> liked polling cycles to be treated as before and not towards packet
>> processing. That does explain, as long as there are packets on the wire we
>> are always 100%, basically cannot tell how efficiently the CPUs are being
>> used.
>> >     >         >     >     Thanks, Shahaji
>> >     >         >     >
>> >     >         >     >     On Mon, Jun 18, 2018 at 10:07 AM, Ilya
>> Maximets <i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>>>>> wrote:
>> >     >         >     >
>> >     >         >     >         Thanks for the data.
>> >     >         >     >
>> >     >         >     >         I have to note additionally that the
>> meaning of "processing cycles"
>> >     >         >     >         significantly changed since the following
>> commit:
>> >     >         >     >
>> >     >         >     >             commit a2ac666d5265c01661e189caac321d
>> 962f54649f
>> >     >         >     >             Author: Ciara Loftus <
>> ciara.loftus at intel.com <mailto:ciara.loftus at intel.com> <mailto:
>> ciara.loftus at intel.com <mailto:ciara.loftus at intel.com>> <mailto:
>> ciara.loftus at intel.com <mailto:ciara.loftus at intel.com> <mailto:
>> ciara.loftus at intel.com <mailto:ciara.loftus at intel.com>>> <mailto:
>> ciara.loftus at intel.com <mailto:ciara.loftus at intel.com> <mailto:
>> ciara.loftus at intel.com <mailto:ciara.loftus at intel.com>> <mailto:
>> ciara.loftus at intel.com <mailto:ciara.loftus at intel.com> <mailto:
>> ciara.loftus at intel.com <mailto:ciara.loftus at intel.com>>>>>
>> >     >         >     >             Date:   Mon Feb 20 12:53:00 2017 +0000
>> >     >         >     >
>> >     >         >     >                 dpif-netdev: Change definitions
>> of 'idle' & 'processing' cycles
>> >     >         >     >
>> >     >         >     >                 Instead of counting all polling
>> cycles as processing cycles, only count
>> >     >         >     >                 the cycles where packets were
>> received from the polling.
>> >     >         >     >
>> >     >         >     >         This could explain the difference in "PMD
>> Processing Cycles" column,
>> >     >         >     >         because successful "POLLING" cycles are
>> now included into "PROCESSING".
>> >     >         >     >
>> >     >         >     >         Best regards, Ilya Maximets.
>> >     >         >     >
>> >     >         >     >         On 18.06.2018 16:31, Shahaji Bhosle wrote:
>> >     >         >     >         > Hi Ilya,
>> >     >         >     >         > Thanks for the quick reply,
>> >     >         >     >         > Please find the numbers for our PHY-PHY
>> test, please note that with OVS 2.9.1 + DPDK 17.11 even a 10% of the below
>> numbers will make the OVS 2.9+DPDK17.11 processing cycles to hit 100%, but
>> 2.7 will on our setup never goes above 75% for processing cycles. I am also
>> attaching the perf report between the two code bases and I think the
>> "11.26%--dp_netdev_pmd_flush_output_packets" is causing us to take the
>> performance hit. Out testing is also SRIOV and CPUs are ARM A72 cores. We
>> are happy to run more tests, it is not easy for use to move back to OVS
>> 2.8, but could happy to try more experiments if it helps us narrow down
>> further. Please note we have also tried increasing the tx-flush-interval
>> and it helps a little but still not significant enough. Let us know.
>> >     >         >     >         >
>> >     >         >     >         > Thanks, Shahaji
>> >     >         >     >         >
>> >     >         >     >         >
>> >     >         >     >         > *Setup:*
>> >     >         >     >         > IXIA<----SFP28--->Port 0
>> {(PF0)==[OVS+DPDK]==(PF1)} Port 1<-----SFP28---->IXIA
>> >     >         >     >         >
>> >     >         >     >         > release/version       config  Test
>> direction       MPPS    Ixia Line rate (%)      PMD Processing Cycles (%)
>> >     >         >     >         > OVS 2.9 + DPDK 17.11  OVS on Maia
>> (PF0--PF1)  No drop port 1 to 2     31.3    85      99.9
>> >     >         >     >         >
>>                port 2 to 1     31.3    85      99.9
>> >     >         >     >         >
>>                bi      15.5 + 15.5     42      99.9
>> >     >         >     >         >
>>
>> >     >         >     >         >
>>
>> >     >         >     >         > OVS 2.7 + DPDK 16.11  OVS on Maia
>> (PF0--PF1)  No drop port 1 to 2     33.8    90      71
>> >     >         >     >         >
>>                port 2 to 1     32.7    88      70
>> >     >         >     >         >
>>                bi      17.4+17.4       47      74
>> >     >         >     >         >
>> >     >         >     >         >
>> >     >         >     >         >
>> >     >         >     >         >
>> >     >         >     >         >
>> >     >         >     >         >
>> >     >         >     >         >
>> >     >         >     >         > On Mon, Jun 18, 2018 at 4:25 AM, Nitin
>> Katiyar <nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>
>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>
>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>
>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>>
>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>
>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>
>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>
>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>>>
>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>
>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>
>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>
>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>>>
>> <mailto:nitin.katiyar at ericsson.com <mailto:nitin.katiyar at ericsson.com>
>> <mailto:nitin.katiyar at ericsson.com
>> >     <mailto:nitin.katiyar at ericsson.com>> <mailto:nitin.katiyar at ericsson
>> .com <mailto:nitin.katiyar at ericsson.com> <mailto:nitin.katiyar at ericsson
>> .com <mailto:nitin.katiyar at ericsson.com>>>>>> wrote:
>> >     >         >     >         >
>> >     >         >     >         >     Hi,
>> >     >         >     >         >     We also experienced degradation
>> from OVS2.6/2.7 to OVS2.8.2(with DPDK17.05.02). The drop is more for 64
>> bytes packet size (~8-10%) even with higher number of flows. I tried OVS
>> 2.8 with DPDK17.11 and it improved for higher packet sizes but 64 bytes
>> size is still the concern.
>> >     >         >     >         >
>> >     >         >     >         >     Regards,
>> >     >         >     >         >     Nitin
>> >     >         >     >         >
>> >     >         >     >         >     -----Original Message-----
>> >     >         >     >         >     From: Ilya Maximets [mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>>>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com>> <mailto:
>> i.maximets at samsung.com <mailto:i.maximets at samsung.com> <mailto:
>> i.maximets at samsung.com
>> >     <mailto:i.maximets at samsung.com>>>>>]
>> >     >         >     >         >     Sent: Monday, June 18, 2018 1:32 PM
>> >     >         >     >         >     To: ovs-dev at openvswitch.org
>> <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:
>> ovs-dev at openvswitch.org>> <mailto:ovs-dev at openvswitch.org <mailto:
>> ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:
>> ovs-dev at openvswitch.org>>> <mailto:ovs-dev at openvswitch.org <mailto:
>> ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:
>> ovs-dev at openvswitch.org>> <mailto:ovs-dev at openvswitch.org <mailto:
>> ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:
>> ovs-dev at openvswitch.org>>>> <mailto:ovs-dev at openvswitch.org <mailto:
>> ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:
>> ovs-dev at openvswitch.org>> <mailto:ovs-dev at openvswitch.org <mailto:
>> ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:
>> ovs-dev at openvswitch.org>>> <mailto:ovs-dev at openvswitch.org <mailto:
>> ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org <mailto:
>> ovs-dev at openvswitch.org>> <mailto:ovs-dev at openvswitch.org
>> >     <mailto:ovs-dev at openvswitch.org> <mailto:ovs-dev at openvswitch.org
>> <mailto:ovs-dev at openvswitch.org>>>>>; shahaji.bhosle at broadcom.com
>> <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com
>> <mailto:shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com
>> <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com
>> <mailto:shahaji.bhosle at broadcom.com>>> <mailto:shahaji.bhosle at broadco
>> m.com <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadco
>> m.com <mailto:shahaji.bhosle at broadcom.com>> <mailto:
>> shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com> <mailto:
>> shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>>>
>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>
>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>>
>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com>
>> <mailto:shahaji.bhosle at broadcom.com <mailto:shahaji.bhosle at broadcom.com
>> >>>
>> >     >         <mailto:shahaji.bhosle at broadcom.com <mailto:
>> shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com <mailto:
>> shahaji.bhosle at broadcom.com>> <mailto:shahaji.bhosle at broadcom.com
>> <mailto:shahaji.bhosle at broadcom.com> <mailto:shahaji.bhosle at broadcom.com
>> <mailto:shahaji.bhosle at broadcom.com>>>>>
>> >     >         >     >         >     Subject: Re: [ovs-dev] 64Byte
>> packet performance regression on 2.9 from 2.7
>> >     >         >     >         >
>> >     >         >     >         >     CC: Shahaji Bhosle
>> >     >         >     >         >
>> >     >         >     >         >     Sorry, missed you in CC list.
>> >     >         >     >         >
>> >     >         >     >         >     Best regards, Ilya Maximets.
>> >     >         >     >         >
>> >     >         >     >         >     On 15.06.2018 10:44, Ilya Maximets
>> wrote:
>> >     >         >     >         >     >> Hi,
>> >     >         >     >         >     >> I just upgraded from OvS 2.7 +
>> DPDK 16.11 to OvS2.9 + DPDK 17.11 and
>> >     >         >     >         >     >> running into performance issue
>> with 64 Byte packet rate. One
>> >     >         >     >         >     >> interesting thing that I notice
>> that even at very light load from
>> >     >         >     >         >     >> IXIA the processing cycles on
>> all the PMD threads run close to 100%
>> >     >         >     >         >     >> of the cpu cycle on 2.9 OvS, but
>> on OvS 2.7 even under full load the
>> >     >         >     >         >     >> processing cycles remain at 75%
>> of the cpu cycles.
>> >     >         >     >         >     >>
>> >     >         >     >         >     >> Attaching the FlameGraphs of
>> both the versions, the only thing that
>> >     >         >     >         >     >> pops out to me is the new way
>> invoking netdev_send() is on 2.9 is
>> >     >         >     >         >     >> being invoked via
>> dp_netdev_pmd_flush_output_packets() which seems
>> >     >         >     >         >     >> to be adding another ~11% to the
>> whole rx to tx path.
>> >     >         >     >         >     >>
>> >     >         >     >         >     >> I also did try the
>> tx-flush-interval to 50 and more it does seem to
>> >     >         >     >         >     >> help, but not significant enough
>> to match the 2.7 performance.
>> >     >         >     >         >     >>
>> >     >         >     >         >     >>
>> >     >         >     >         >     >> Any help or ideas would be
>> really great. Thanks, Shahaji
>> >     >         >     >         >     >
>> >     >         >     >         >     > Hello, Shahaji.
>> >     >         >     >         >     > Could you, please, describe your
>> testing scenario in more details?
>> >     >         >     >         >     > Also, mail-list filters
>> attachments, so they are not available. You
>> >     >         >     >         >     > need to publish them somewhere
>> else or write in text format inside the letter.
>> >     >         >     >         >     >
>> >     >         >     >         >     > About the performance itself:
>> Some performance degradation because of
>> >     >         >     >         >     > output batching is expected for
>> tests with low number of flows or
>> >     >         >     >         >     > simple PHY-PHY tests. It was
>> mainly targeted for cases with relatively
>> >     >         >     >         >     > large number of flows, for
>> amortizing of vhost-user penalties
>> >     >         >     >         >     > (PHY-VM-PHY, VM-VM cases), OVS
>> bonding cases.
>> >     >         >     >         >     >
>> >     >         >     >         >     > If your test involves vhost-user
>> ports, then you should also consider
>> >     >         >     >         >     > vhost-user performance regression
>> in stable DPDK 17.11 because of
>> >     >         >     >         >     > fixes for CVE-2018-1059. Related
>> bug:
>> >     >         >     >         >     >
>> https://dpdk.org/tracker/show_bug.cgi?id=48 <
>> https://dpdk.org/tracker/show_bug.cgi?id=48> <
>> https://dpdk.org/tracker/show_bug.cgi?id=48 <
>> https://dpdk.org/tracker/show_bug.cgi?id=48>> <
>> https://dpdk.org/tracker/show_bug.cgi?id=48 <
>> https://dpdk.org/tracker/show_bug.cgi?id=48> <
>> https://dpdk.org/tracker/show_bug.cgi?id=48 <
>> https://dpdk.org/tracker/show_bug.cgi?id=48>>> <
>> https://dpdk.org/tracker/show_bug.cgi?id=48 <
>> https://dpdk.org/tracker/show_bug.cgi?id=48> <
>> https://dpdk.org/tracker/show_bug.cgi?id=48 <
>> https://dpdk.org/tracker/show_bug.cgi?id=48>> <
>> https://dpdk.org/tracker/show_bug.cgi?id=48 <
>> https://dpdk.org/tracker/show_bug.cgi?id=48> <
>> https://dpdk.org/tracker/show_bug.cgi?id=48 <
>> https://dpdk.org/tracker/show_bug.cgi?id=48>>>>
>> >     >         >     >         >     >
>> >     >         >     >         >     > It'll be good if you'll be able
>> to test OVS 2.8 + DPDK 17.05. There
>> >     >         >     >         >     > was too many changes since 2.7.
>> It'll be hard to track down the root cause.
>> >     >         >     >         >     >
>> >     >         >     >         >     > Best regards, Ilya Maximets.
>> >     >         >     >         >     >
>> >     >         >     >         >
>> >     >         >     >         >
>> >     >         >     >
>> >     >         >     >
>> >     >         >
>> >     >         >
>> >     >
>> >     >
>> >     >
>> >
>> >
>>
>
>


More information about the dev mailing list