[ovs-discuss] DPDK Performance problem
Traynor, Kevin
kevin.traynor at intel.com
Thu Jan 21 14:14:59 UTC 2016
> -----Original Message-----
> From: Ali Volkan Atli [mailto:Volkan.Atli at argela.com.tr]
> Sent: Monday, January 18, 2016 6:59 AM
> To: Traynor, Kevin; discuss at openvswitch.org
> Subject: RE: DPDK Performance problem
>
> Hi Kevin
>
> I isolated master and logical core and assigned the master cpu to pmd thread,
> then the performance increased from 500M to 550M, but I think it is still
> low. In fact, when I try to use 'fwd' option on DPDK's testpmd tools, the
> performance is ~3.000 M and the RX-errors is increasing rapidly. Actually I
> try to activate DPDK logs (sudo ovs-appctl vlog/set netdev:syslog:dbg) to
> debug the problem but syslog has nothing for netdev, am I missing something?
>
> Do you have an idea of the problem's origin? What is your suggestion? How
> should I continue to solve the problem?
Hi Volkan. I think I misread the numbers in the last mail, if I'm understanding
right then this is very low. The rx errors that are being reported could be from
corrupt packets or the software polling them can't keep up because something else
is using the core. The fact that DPDK, and OVS with DPDK give different performance
make me suspect it's the latter. I'd probably re-confirm there's nothing else using
the core and re-post the DPDK only test on the DPDK mailing list.
>
> Thanks in advance..
>
> - Volkan
>
> testpmd> show port stats all
>
> ######################## NIC statistics for port 0
> ########################
> RX-packets: 0 RX-missed: 0 RX-bytes: 0
> RX-errors: 0
> RX-nombuf: 0
> TX-packets: 75478729 TX-errors: 0 TX-bytes: 4830629960
>
> ############################################################################
>
> ######################## NIC statistics for port 1
> ########################
> RX-packets: 161580095 RX-missed: 0 RX-bytes: 10341131520
> RX-errors: 83276733
> RX-nombuf: 0
> TX-packets: 0 TX-errors: 0 TX-bytes: 0
>
> ############################################################################
>
> NAPATECH monitoring output
> *─────────────────────── monitoring (v. 2.7.4.3-ca9b9)
> ────────────────────────┐
> │ 0 0 SFP+ 10G Full 41 0.00M 9776.42M 9018 45.40 C
> │
> │ 1 0 Passive DAC 10G Full 62 3018.02M 0.00M 9018 N/A
> │
>
> In DPDK+OvS;
>
> argela at argela-HP-Z800-Workstation:~/ovs_dpdk$ sudo ovs-vsctl get Interface
> dpdk0 statistics
> {rx_bytes=0, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_packets=0,
> tx_bytes=22709654592, tx_dropped=0, tx_errors=0, tx_packets=354838349}
> argela at argela-HP-Z800-Workstation:~/ovs_dpdk$ sudo ovs-vsctl get Interface
> dpdk1 statistics
> {rx_bytes=23912587200, rx_crc_err=0, rx_dropped=0, rx_errors=6253280134,
> rx_packets=373634175, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
>
> argela at argela-HP-Z800-Workstation:~/ovs_dpdk$ sudo ovs-ofctl dump-ports br0
> OFPST_PORT reply (xid=0x2): 3 ports
> port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
> tx pkts=77, bytes=4928, drop=0, errs=0, coll=0
> port 1: rx pkts=0, bytes=0, drop=0, errs=0, frame=?, over=?, crc=0
> tx pkts=320903181, bytes=20537803840, drop=0, errs=0, coll=?
> port 2: rx pkts=335778207, bytes=21489805248, drop=0, errs=5620368004,
> frame=?, over=?, crc=0
> tx pkts=0, bytes=0, drop=0, errs=0, coll=?
>
>
> *─────────────────────── monitoring (v. 2.7.4.3-ca9b9)
> ────────────────────────┐
> │ 0 0 SFP+ 10G Full 3 0.00M 9776.36M 9018 45.60 C
> │
> │ 1 0 Passive DAC 10G Full 3 553.15M 0.00M 9018 N/A
> │
>
>
> ________________________________________
> From: Traynor, Kevin [kevin.traynor at intel.com]
> Sent: Tuesday, January 12, 2016 5:16 PM
> To: Ali Volkan Atli; discuss at openvswitch.org
> Subject: RE: DPDK Performance problem
>
> > -----Original Message-----
> > From: discuss [mailto:discuss-bounces at openvswitch.org] On Behalf Of Ali
> > Volkan Atli
> > Sent: Monday, January 11, 2016 1:31 PM
> > To: discuss at openvswitch.org
> > Subject: [ovs-discuss] DPDK Performance problem
> >
> > Hi
> >
> > I've been trying to measure the throughput of Open vSwitch with DPDK for
> > physical ports. My test setup is shown below:
> >
> > +-----------------------------+
> > | 82599ES 10-Gigabit SFI/SFP+ |
> > +-----------------------------+
> > | p0 | | p1 |
> > +----+ +----+
> > ^ ^
> > | |
> > v v
> > +----+ +----+
> > | p0 | | p1 |
> > +-----------------------------+
> > | NAPATECH Adapter - 2 port |
> > +-----------------------------+
> >
> > I manually created the ovs bridge, dpdk ports and a flow entry as follows:
> >
> > ===============================
> > $ sudo ovs-vsctl show
> > [sudo] password for argela:
> > dfa41660-4e24-4f4e-87de-a21398f51246
> > Bridge "br0"
> > Port "br0"
> > Interface "br0"
> > type: internal
> > Port "dpdk1"
> > Interface "dpdk1"
> > type: dpdk
> > Port "dpdk0"
> > Interface "dpdk0"
> > type: dpdk
> >
> > $ sudo ovs-ofctl dump-flows br0
> > [sudo] password for argela:
> > NXST_FLOW reply (xid=0x4):
> > cookie=0x0, duration=3659.413s, table=0, n_packets=5061744149,
> > n_bytes=3111037716380, idle_age=582, in_port=1 actions=output:2
> > ===============================
> >
> > Then, I isolated cores from Linux kernel according to the output of
> > dpdk/tools/cpu_layout.py.
> >
> > ===============================
> > $ ./cpu_layout.py
> >
> > cores = [0, 1, 2, 8, 9, 10]
> > sockets = [0, 1]
> >
> > Socket 0 Socket 1
> > -------- --------
> > Core 0 [0, 12] [6, 18]
> > Core 1 [1, 13] [7, 19]
> > Core 2 [2, 14] [8, 20]
> > Core 8 [3, 15] [9, 21]
> > Core 9 [4, 16] [10, 22]
> > Core 10 [5, 17] [11, 23]
> >
> > $ sudo vi /etc/default/grub
> > GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on
> > vfio_iommu_type1.allow_unsafe_interrupts=1 default_hugepagesz=1G
> > hugepagesz=1G hugepages=5 isolcpus=12,13,14,15,16,17,18,19,20,21,22,23"
> >
> > $ sudo ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=FFF000
> >
> > $ sudo top -p `pidof ovs-vswitchd` -H -d1
> > 3098 root 20 0 4250504 65688 2672 R 99,9 0,4 26:18.44 pmd42
> > 14
> > 3101 root 20 0 4250504 65688 2672 R 99,9 0,4 26:18.45 pmd45
> > 17
> > 3096 root 20 0 4250504 65688 2672 R 99,8 0,4 26:18.83 pmd46
> > 12
> > 3097 root 20 0 4250504 65688 2672 R 99,8 0,4 26:18.77 pmd41
> > 13
> > 3099 root 20 0 4250504 65688 2672 R 98,8 0,4 26:18.43 pmd43
> > 15
> > 3100 root 20 0 4250504 65688 2672 R 98,8 0,4 26:18.42 pmd44
> > 16
> > 2652 root 20 0 4250504 65688 2672 S 0,0 0,4 0:03.15 ovs-
> > vswitchd 1
> > 2653 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.01
> > dpdk_watchdog3 1
> > 2654 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.05
> > vhost_thread2 1
> > 2655 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.73 urcu1
> > 1
> > 2683 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00
> handler27
> > 1
> > 2684 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00
> handler26
> > 1
> > 2685 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00
> handler25
> > 1
> > 2686 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00
> handler24
> > 1
> > 2689 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00
> handler23
> > 1
> > 2690 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00
> handler17
> > 1
> > 2691 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00
> handler18
> > 1
> > 2692 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00
> handler19
> > 1
> > 2693 root 20 0 4250504 65688 2672 S 0,0 0,4 0:01.82
> > revalidator20 1
> > 2694 root 20 0 4250504 65688 2672 S 0,0 0,4 0:01.51
> > revalidator21 1
> > 2695 root 20 0 4250504 65688 2672 S 0,0 0,4 0:01.51
> > revalidator16 1
> > 2696 root 20 0 4250504 65688 2672 S 0,0 0,4 0:01.51
> > revalidator22 1
> > ===============================
> >
> > When I start the traffic by using a pcap file for NAPATECH port0 with 10G
> > rate, OvS+DPDK can reach the line rate. However, when I start sending 64
> byte
> > UDP packets at 10G rate, the throughtput is ~550M. And also, OvS+DPDK only
> > uses one core and the others are IDLE even if I set 12 cores.
>
> It only uses one core, as one core is polling from 1 queue on p0 and sending
> to p1.
> If you have multiple flows you could use more cores by increasing the rxq's
> for the
> dpdk ports.
>
> Could the difference be due to packet sizes? You're getting ~8 Mpps for 64
> byte pkts,
> that seems low but considering you're using a single 'logical' core and
> there's a
> current performance drop on master that could be the right number for your
> processor
> clock speed.
>
> I'd recommend looking through the perf tuning section here
> https://github.com/openvswitch/ovs/blob/master/INSTALL.DPDK.md#performance-
> tuning
>
> >
> > ===============================
> > $ sudo ovs-appctl dpif-netdev/pmd-stats-show
> > pmd thread numa_id 0 core_id 13:
> > emc hits:0
> > megaflow hits:0
> > miss:0
> > lost:0
> > polling cycles:2608878847988 (100.00%)
> > processing cycles:0 (0.00%)
> > pmd thread numa_id 0 core_id 14:
> > emc hits:0
> > megaflow hits:0
> > miss:0
> > lost:0
> > pmd thread numa_id 0 core_id 16:
> > emc hits:0
> > megaflow hits:0
> > miss:0
> > lost:0
> > main thread:
> > emc hits:0
> > megaflow hits:0
> > miss:0
> > lost:0
> > polling cycles:76924028 (100.00%)
> > processing cycles:0 (0.00%)
> > pmd thread numa_id 0 core_id 12:
> > emc hits:1963602000
> > megaflow hits:44356
> > miss:9
> > lost:0
> > polling cycles:1007444696020 (28.15%)
> > processing cycles:2570868572784 (71.85%)
> > avg cycles per packet: 1822.28 (3578313268804/1963646365)
> > avg processing cycles per packet: 1309.23 (2570868572784/1963646365)
> > pmd thread numa_id 0 core_id 15:
> > emc hits:0
> > megaflow hits:0
> > miss:0
> > lost:0
> > pmd thread numa_id 0 core_id 17:
> > emc hits:0
> > megaflow hits:0
> > miss:0
> > lost:0
> > ===============================
> >
> > Could you please explain why OvS+DPDK did not reach the line rate (10G) for
> > 64 byte UDP traffic?
> >
> > Thanks in advance.
> > - Volkan
> > _______________________________________________
> > discuss mailing list
> > discuss at openvswitch.org
> > http://openvswitch.org/mailman/listinfo/discuss
More information about the discuss
mailing list