[ovs-discuss] DPDK Performance problem
Traynor, Kevin
kevin.traynor at intel.com
Tue Jan 12 15:16:10 UTC 2016
> -----Original Message-----
> From: discuss [mailto:discuss-bounces at openvswitch.org] On Behalf Of Ali
> Volkan Atli
> Sent: Monday, January 11, 2016 1:31 PM
> To: discuss at openvswitch.org
> Subject: [ovs-discuss] DPDK Performance problem
>
> Hi
>
> I've been trying to measure the throughput of Open vSwitch with DPDK for
> physical ports. My test setup is shown below:
>
> +-----------------------------+
> | 82599ES 10-Gigabit SFI/SFP+ |
> +-----------------------------+
> | p0 | | p1 |
> +----+ +----+
> ^ ^
> | |
> v v
> +----+ +----+
> | p0 | | p1 |
> +-----------------------------+
> | NAPATECH Adapter - 2 port |
> +-----------------------------+
>
> I manually created the ovs bridge, dpdk ports and a flow entry as follows:
>
> ===============================
> $ sudo ovs-vsctl show
> [sudo] password for argela:
> dfa41660-4e24-4f4e-87de-a21398f51246
> Bridge "br0"
> Port "br0"
> Interface "br0"
> type: internal
> Port "dpdk1"
> Interface "dpdk1"
> type: dpdk
> Port "dpdk0"
> Interface "dpdk0"
> type: dpdk
>
> $ sudo ovs-ofctl dump-flows br0
> [sudo] password for argela:
> NXST_FLOW reply (xid=0x4):
> cookie=0x0, duration=3659.413s, table=0, n_packets=5061744149,
> n_bytes=3111037716380, idle_age=582, in_port=1 actions=output:2
> ===============================
>
> Then, I isolated cores from Linux kernel according to the output of
> dpdk/tools/cpu_layout.py.
>
> ===============================
> $ ./cpu_layout.py
>
> cores = [0, 1, 2, 8, 9, 10]
> sockets = [0, 1]
>
> Socket 0 Socket 1
> -------- --------
> Core 0 [0, 12] [6, 18]
> Core 1 [1, 13] [7, 19]
> Core 2 [2, 14] [8, 20]
> Core 8 [3, 15] [9, 21]
> Core 9 [4, 16] [10, 22]
> Core 10 [5, 17] [11, 23]
>
> $ sudo vi /etc/default/grub
> GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on
> vfio_iommu_type1.allow_unsafe_interrupts=1 default_hugepagesz=1G
> hugepagesz=1G hugepages=5 isolcpus=12,13,14,15,16,17,18,19,20,21,22,23"
>
> $ sudo ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=FFF000
>
> $ sudo top -p `pidof ovs-vswitchd` -H -d1
> 3098 root 20 0 4250504 65688 2672 R 99,9 0,4 26:18.44 pmd42
> 14
> 3101 root 20 0 4250504 65688 2672 R 99,9 0,4 26:18.45 pmd45
> 17
> 3096 root 20 0 4250504 65688 2672 R 99,8 0,4 26:18.83 pmd46
> 12
> 3097 root 20 0 4250504 65688 2672 R 99,8 0,4 26:18.77 pmd41
> 13
> 3099 root 20 0 4250504 65688 2672 R 98,8 0,4 26:18.43 pmd43
> 15
> 3100 root 20 0 4250504 65688 2672 R 98,8 0,4 26:18.42 pmd44
> 16
> 2652 root 20 0 4250504 65688 2672 S 0,0 0,4 0:03.15 ovs-
> vswitchd 1
> 2653 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.01
> dpdk_watchdog3 1
> 2654 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.05
> vhost_thread2 1
> 2655 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.73 urcu1
> 1
> 2683 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler27
> 1
> 2684 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler26
> 1
> 2685 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler25
> 1
> 2686 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler24
> 1
> 2689 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler23
> 1
> 2690 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler17
> 1
> 2691 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler18
> 1
> 2692 root 20 0 4250504 65688 2672 S 0,0 0,4 0:00.00 handler19
> 1
> 2693 root 20 0 4250504 65688 2672 S 0,0 0,4 0:01.82
> revalidator20 1
> 2694 root 20 0 4250504 65688 2672 S 0,0 0,4 0:01.51
> revalidator21 1
> 2695 root 20 0 4250504 65688 2672 S 0,0 0,4 0:01.51
> revalidator16 1
> 2696 root 20 0 4250504 65688 2672 S 0,0 0,4 0:01.51
> revalidator22 1
> ===============================
>
> When I start the traffic by using a pcap file for NAPATECH port0 with 10G
> rate, OvS+DPDK can reach the line rate. However, when I start sending 64 byte
> UDP packets at 10G rate, the throughtput is ~550M. And also, OvS+DPDK only
> uses one core and the others are IDLE even if I set 12 cores.
It only uses one core, as one core is polling from 1 queue on p0 and sending to p1.
If you have multiple flows you could use more cores by increasing the rxq's for the
dpdk ports.
Could the difference be due to packet sizes? You're getting ~8 Mpps for 64 byte pkts,
that seems low but considering you're using a single 'logical' core and there's a
current performance drop on master that could be the right number for your processor
clock speed.
I'd recommend looking through the perf tuning section here
https://github.com/openvswitch/ovs/blob/master/INSTALL.DPDK.md#performance-tuning
>
> ===============================
> $ sudo ovs-appctl dpif-netdev/pmd-stats-show
> pmd thread numa_id 0 core_id 13:
> emc hits:0
> megaflow hits:0
> miss:0
> lost:0
> polling cycles:2608878847988 (100.00%)
> processing cycles:0 (0.00%)
> pmd thread numa_id 0 core_id 14:
> emc hits:0
> megaflow hits:0
> miss:0
> lost:0
> pmd thread numa_id 0 core_id 16:
> emc hits:0
> megaflow hits:0
> miss:0
> lost:0
> main thread:
> emc hits:0
> megaflow hits:0
> miss:0
> lost:0
> polling cycles:76924028 (100.00%)
> processing cycles:0 (0.00%)
> pmd thread numa_id 0 core_id 12:
> emc hits:1963602000
> megaflow hits:44356
> miss:9
> lost:0
> polling cycles:1007444696020 (28.15%)
> processing cycles:2570868572784 (71.85%)
> avg cycles per packet: 1822.28 (3578313268804/1963646365)
> avg processing cycles per packet: 1309.23 (2570868572784/1963646365)
> pmd thread numa_id 0 core_id 15:
> emc hits:0
> megaflow hits:0
> miss:0
> lost:0
> pmd thread numa_id 0 core_id 17:
> emc hits:0
> megaflow hits:0
> miss:0
> lost:0
> ===============================
>
> Could you please explain why OvS+DPDK did not reach the line rate (10G) for
> 64 byte UDP traffic?
>
> Thanks in advance.
> - Volkan
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> http://openvswitch.org/mailman/listinfo/discuss
More information about the discuss
mailing list