[ovs-discuss] DPDK Performance problem
Ali Volkan Atli
Volkan.Atli at argela.com.tr
Mon Jan 18 06:58:46 UTC 2016
Hi Kevin
I isolated master and logical core and assigned the master cpu to pmd thread, then the performance increased from 500M to 550M, but I think it is still low. In fact, when I try to use 'fwd' option on DPDK's testpmd tools, the performance is ~3.000 M and the RX-errors is increasing rapidly. Actually I try to activate DPDK logs (sudo ovs-appctl vlog/set netdev:syslog:dbg) to debug the problem but syslog has nothing for netdev, am I missing something?
Do you have an idea of the problem's origin? What is your suggestion? How should I continue to solve the problem?
Thanks in advance..
- Volkan
testpmd> show port stats all
######################## NIC statistics for port 0 ########################
RX-packets: 0 RX-missed: 0 RX-bytes: 0
RX-errors: 0
RX-nombuf: 0
TX-packets: 75478729 TX-errors: 0 TX-bytes: 4830629960
############################################################################
######################## NIC statistics for port 1 ########################
RX-packets: 161580095 RX-missed: 0 RX-bytes: 10341131520
RX-errors: 83276733
RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
############################################################################
NAPATECH monitoring output
*─────────────────────── monitoring (v. 2.7.4.3-ca9b9) ────────────────────────┐
│ 0 0 SFP+ 10G Full 41 0.00M 9776.42M 9018 45.40 C │
│ 1 0 Passive DAC 10G Full 62 3018.02M 0.00M 9018 N/A │
In DPDK+OvS;
argela at argela-HP-Z800-Workstation:~/ovs_dpdk$ sudo ovs-vsctl get Interface dpdk0 statistics
{rx_bytes=0, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_packets=0, tx_bytes=22709654592, tx_dropped=0, tx_errors=0, tx_packets=354838349}
argela at argela-HP-Z800-Workstation:~/ovs_dpdk$ sudo ovs-vsctl get Interface dpdk1 statistics
{rx_bytes=23912587200, rx_crc_err=0, rx_dropped=0, rx_errors=6253280134, rx_packets=373634175, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
argela at argela-HP-Z800-Workstation:~/ovs_dpdk$ sudo ovs-ofctl dump-ports br0
OFPST_PORT reply (xid=0x2): 3 ports
port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
tx pkts=77, bytes=4928, drop=0, errs=0, coll=0
port 1: rx pkts=0, bytes=0, drop=0, errs=0, frame=?, over=?, crc=0
tx pkts=320903181, bytes=20537803840, drop=0, errs=0, coll=?
port 2: rx pkts=335778207, bytes=21489805248, drop=0, errs=5620368004, frame=?, over=?, crc=0
tx pkts=0, bytes=0, drop=0, errs=0, coll=?
*─────────────────────── monitoring (v. 2.7.4.3-ca9b9) ────────────────────────┐
│ 0 0 SFP+ 10G Full 3 0.00M 9776.36M 9018 45.60 C │
│ 1 0 Passive DAC 10G Full 3 553.15M 0.00M 9018 N/A │
________________________________________
From: Traynor, Kevin [kevin.traynor at intel.com]
Sent: Tuesday, January 12, 2016 5:16 PM
To: Ali Volkan Atli; discuss at openvswitch.org
Subject: RE: DPDK Performance problem
> -----Original Message-----
> From: discuss [mailto:discuss-bounces at openvswitch.org] On Behalf Of Ali
> Volkan Atli
> Sent: Monday, January 11, 2016 1:31 PM
> To: discuss at openvswitch.org
> Subject: [ovs-discuss] DPDK Performance problem
>
> Hi
>
> I've been trying to measure the throughput of Open vSwitch with DPDK for
> physical ports. My test setup is shown below:
>
> +-----------------------------+
> | 82599ES 10-Gigabit SFI/SFP+ |
> +-----------------------------+
> | p0 | | p1 |
> +----+ +----+
> ^ ^
> | |
> v v
> +----+ +----+
> | p0 | | p1 |
> +-----------------------------+
> | NAPATECH Adapter - 2 port |
> +-----------------------------+
>
> I manually created the ovs bridge, dpdk ports and a flow entry as follows:
>
> ===============================
> $ sudo ovs-vsctl show
> [sudo] password for argela:
> dfa41660-4e24-4f4e-87de-a21398f51246
> Bridge "br0"
> Port "br0"
> Interface "br0"
> type: internal
> Port "dpdk1"
> Interface "dpdk1"
> type: dpdk
> Port "dpdk0"
> Interface "dpdk0"
> type: dpdk
>
> $ sudo ovs-ofctl dump-flows br0
> [sudo] password for argela:
> NXST_FLOW reply (xid=0x4):
> cookie=0x0, duration=3659.413s, table=0, n_packets=5061744149,
> n_bytes=3111037716380, idle_age=582, in_port=1 actions=output:2
> ===============================
>
> Then, I isolated cores from Linux kernel according to the output of
> dpdk/tools/cpu_layout.py.
>
> ===============================
> $ ./cpu_layout.py
>
> cores = [0, 1, 2, 8, 9, 10]
> sockets = [0, 1]
>
> Socket 0 Socket 1
> -------- --------
> Core 0 [0, 12] [6, 18]
> Core 1 [1, 13] [7, 19]
> Core 2 [2, 14] [8, 20]
> Core 8 [3, 15] [9, 21]
> Core 9 [4, 16] [10, 22]
> Core 10 [5, 17] [11, 23]
>
> $ sudo vi /etc/default/grub
> GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on
> vfio_iommu_type1.allow_unsafe_interrupts=1 default_hugepagesz=1G
> hugepagesz=1G hugepages=5 isolcpus=12,13,14,15,16,17,18,19,20,21,22,23"
>
> $ sudo ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=FFF000
>
> $ sudo top -p `pidof ovs-vswitchd` -H -d1
> 3098 root 20 0 4250504 65688 2672 R 99,9 0,4 26:18.44 pmd42
> 14
> 3101 root 20 0 4250504 65688 2672 R 99,9 0,4 26:18.45 pmd45
> 17
> 3096 root 20 0 4250504 65688 2672 R 99,8 0,4 26:18.83 pmd46
> 12
> 3097 root 20 0 4250504 65688 2672 R 99,8 0,4 26:18.77 pmd41
> 13
> 3099 root 20 0 4250504 65688 2672 R 98,8 0,4 26:18.43 pmd43
> 15
> 3100 root 20 0 4250504 65688 2672 R 98,8 0,4 26:18.42 pmd44
> 16
> 2652 root 20 0 4250504 65688 2672 S 0,0 0,4 0:03.15 ovs-
> vswitchd 1
> 2653 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.01
> dpdk_watchdog3 1
> 2654 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.05
> vhost_thread2 1
> 2655 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.73 urcu1
> 1
> 2683 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler27
> 1
> 2684 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler26
> 1
> 2685 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler25
> 1
> 2686 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler24
> 1
> 2689 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler23
> 1
> 2690 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler17
> 1
> 2691 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler18
> 1
> 2692 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler19
> 1
> 2693 root 20 0 4250504 65688 2672 S 0,0 0,4 0:01.82
> revalidator20 1
> 2694 root 20 0 4250504 65688 2672 S 0,0 0,4 0:01.51
> revalidator21 1
> 2695 root 20 0 4250504 65688 2672 S 0,0 0,4 0:01.51
> revalidator16 1
> 2696 root 20 0 4250504 65688 2672 S 0,0 0,4 0:01.51
> revalidator22 1
> ===============================
>
> When I start the traffic by using a pcap file for NAPATECH port0 with 10G
> rate, OvS+DPDK can reach the line rate. However, when I start sending 64 byte
> UDP packets at 10G rate, the throughtput is ~550M. And also, OvS+DPDK only
> uses one core and the others are IDLE even if I set 12 cores.
It only uses one core, as one core is polling from 1 queue on p0 and sending to p1.
If you have multiple flows you could use more cores by increasing the rxq's for the
dpdk ports.
Could the difference be due to packet sizes? You're getting ~8 Mpps for 64 byte pkts,
that seems low but considering you're using a single 'logical' core and there's a
current performance drop on master that could be the right number for your processor
clock speed.
I'd recommend looking through the perf tuning section here
https://github.com/openvswitch/ovs/blob/master/INSTALL.DPDK.md#performance-tuning
>
> ===============================
> $ sudo ovs-appctl dpif-netdev/pmd-stats-show
> pmd thread numa_id 0 core_id 13:
> emc hits:0
> megaflow hits:0
> miss:0
> lost:0
> polling cycles:2608878847988 (100.00%)
> processing cycles:0 (0.00%)
> pmd thread numa_id 0 core_id 14:
> emc hits:0
> megaflow hits:0
> miss:0
> lost:0
> pmd thread numa_id 0 core_id 16:
> emc hits:0
> megaflow hits:0
> miss:0
> lost:0
> main thread:
> emc hits:0
> megaflow hits:0
> miss:0
> lost:0
> polling cycles:76924028 (100.00%)
> processing cycles:0 (0.00%)
> pmd thread numa_id 0 core_id 12:
> emc hits:1963602000
> megaflow hits:44356
> miss:9
> lost:0
> polling cycles:1007444696020 (28.15%)
> processing cycles:2570868572784 (71.85%)
> avg cycles per packet: 1822.28 (3578313268804/1963646365)
> avg processing cycles per packet: 1309.23 (2570868572784/1963646365)
> pmd thread numa_id 0 core_id 15:
> emc hits:0
> megaflow hits:0
> miss:0
> lost:0
> pmd thread numa_id 0 core_id 17:
> emc hits:0
> megaflow hits:0
> miss:0
> lost:0
> ===============================
>
> Could you please explain why OvS+DPDK did not reach the line rate (10G) for
> 64 byte UDP traffic?
>
> Thanks in advance.
> - Volkan
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> http://openvswitch.org/mailman/listinfo/discuss
More information about the discuss
mailing list