[ovs-discuss] ovs-dpdk performance

Sat Oct 30 05:07:07 UTC 2021

Folks,

I have configured ovs-dpdk to replace sriov deployment for bonding
support. everything good but somehow as soon as i start hitting
200kpps rate i start seeing packet drop.

I have configured CPU isolation as per documentation to assign a
dedicated pmd thread. I have assigned 8 dedicated PMD threads but
still performance is very poor.

I created an 8vCPU vm on openstack using dpdk and running some
workload using an in-house application, during the 200kpps packet rate
I noticed my all PMD cpu showing high CPU processing cycles.

 $ ovs-vswitchd -V
ovs-vswitchd (Open vSwitch) 2.13.3
DPDK 19.11.7

In the following output what does these queue-id:0 to 8 and why only
the first queue is in use but not others, they are always zero. What
does this mean?

ovs-appctl dpif-netdev/pmd-rxq-show
pmd thread numa_id 0 core_id 2:
  isolated : false
  port: vhu1c3bf17a-01    queue-id:  0 (enabled)   pmd usage:  0 %
  port: vhu1c3bf17a-01    queue-id:  1 (enabled)   pmd usage:  0 %
  port: vhu6b7daba9-1a    queue-id:  2 (disabled)  pmd usage:  0 %
  port: vhu6b7daba9-1a    queue-id:  3 (disabled)  pmd usage:  0 %
pmd thread numa_id 1 core_id 3:
  isolated : false
pmd thread numa_id 0 core_id 22:
  isolated : false
  port: vhu1c3bf17a-01    queue-id:  3 (enabled)   pmd usage:  0 %
  port: vhu1c3bf17a-01    queue-id:  6 (enabled)   pmd usage:  0 %
  port: vhu6b7daba9-1a    queue-id:  0 (enabled)   pmd usage: 54 %
  port: vhu6b7daba9-1a    queue-id:  5 (disabled)  pmd usage:  0 %
pmd thread numa_id 1 core_id 23:
  isolated : false
  port: dpdk1             queue-id:  0 (enabled)   pmd usage:  3 %
pmd thread numa_id 0 core_id 26:
  isolated : false
  port: vhu1c3bf17a-01    queue-id:  2 (enabled)   pmd usage:  0 %
  port: vhu1c3bf17a-01    queue-id:  7 (enabled)   pmd usage:  0 %
  port: vhu6b7daba9-1a    queue-id:  1 (disabled)  pmd usage:  0 %
  port: vhu6b7daba9-1a    queue-id:  4 (disabled)  pmd usage:  0 %
pmd thread numa_id 1 core_id 27:
  isolated : false
pmd thread numa_id 0 core_id 46:
  isolated : false
  port: dpdk0             queue-id:  0 (enabled)   pmd usage:  27 %
  port: vhu1c3bf17a-01    queue-id:  4 (enabled)   pmd usage:  0 %
  port: vhu1c3bf17a-01    queue-id:  5 (enabled)   pmd usage:  0 %
  port: vhu6b7daba9-1a    queue-id:  6 (disabled)  pmd usage:  0 %
  port: vhu6b7daba9-1a    queue-id:  7 (disabled)  pmd usage:  0 %
pmd thread numa_id 1 core_id 47:
  isolated : false

$ ovs-appctl dpif-netdev/pmd-stats-clear && sleep 10 && ovs-appctl
dpif-netdev/pmd-stats-show | grep "processing cycles:"
  processing cycles: 1697952 (0.01%)
  processing cycles: 12726856558 (74.96%)
  processing cycles: 4259431602 (19.40%)
  processing cycles: 512666 (0.00%)
  processing cycles: 6324848608 (37.81%)

Does processing cycles mean my PMD is under stress? but i am only
hitting 200kpps rate?

This is my dpdk0 and dpdk1 port statistics

sudo ovs-vsctl get Interface dpdk0 statistics
{flow_director_filter_add_errors=153605,
flow_director_filter_remove_errors=30829, mac_local_errors=0,
mac_remote_errors=0, ovs_rx_qos_drops=0, ovs_tx_failure_drops=0,
ovs_tx_invalid_hwol_drops=0, ovs_tx_mtu_exceeded_drops=0,
ovs_tx_qos_drops=0, rx_128_to_255_packets=64338613,
rx_1_to_64_packets=367, rx_256_to_511_packets=116298,
rx_512_to_1023_packets=31264, rx_65_to_127_packets=6990079,
rx_broadcast_packets=0, rx_bytes=12124930385, rx_crc_errors=0,
rx_dropped=0, rx_errors=12, rx_fcoe_crc_errors=0, rx_fcoe_dropped=12,
rx_fcoe_mbuf_allocation_errors=0, rx_fragment_errors=367,
rx_illegal_byte_errors=0, rx_jabber_errors=0, rx_length_errors=0,
rx_mac_short_packet_dropped=128, rx_management_dropped=35741,
rx_management_packets=31264, rx_mbuf_allocation_errors=0,
rx_missed_errors=0, rx_oversize_errors=0, rx_packets=71512362,
rx_priority0_dropped=0, rx_priority0_mbuf_allocation_errors=1096,
rx_priority1_dropped=0, rx_priority1_mbuf_allocation_errors=0,
rx_priority2_dropped=0, rx_priority2_mbuf_allocation_errors=0,
rx_priority3_dropped=0, rx_priority3_mbuf_allocation_errors=0,
rx_priority4_dropped=0, rx_priority4_mbuf_allocation_errors=0,
rx_priority5_dropped=0, rx_priority5_mbuf_allocation_errors=0,
rx_priority6_dropped=0, rx_priority6_mbuf_allocation_errors=0,
rx_priority7_dropped=0, rx_priority7_mbuf_allocation_errors=0,
rx_undersize_errors=6990079, tx_128_to_255_packets=64273778,
tx_1_to_64_packets=128, tx_256_to_511_packets=43670294,
tx_512_to_1023_packets=153605, tx_65_to_127_packets=881272,
tx_broadcast_packets=10, tx_bytes=25935295292, tx_dropped=0,
tx_errors=0, tx_management_packets=0, tx_multicast_packets=153,
tx_packets=109009906}

sudo ovs-vsctl get Interface dpdk1 statistics
{flow_director_filter_add_errors=126793,
flow_director_filter_remove_errors=37969, mac_local_errors=0,
mac_remote_errors=0, ovs_rx_qos_drops=0, ovs_tx_failure_drops=0,
ovs_tx_invalid_hwol_drops=0, ovs_tx_mtu_exceeded_drops=0,
ovs_tx_qos_drops=0, rx_128_to_255_packets=64435459,
rx_1_to_64_packets=107843, rx_256_to_511_packets=230,
rx_512_to_1023_packets=13, rx_65_to_127_packets=7049788,
rx_broadcast_packets=199058, rx_bytes=12024342488, rx_crc_errors=0,
rx_dropped=0, rx_errors=11, rx_fcoe_crc_errors=0, rx_fcoe_dropped=11,
rx_fcoe_mbuf_allocation_errors=0, rx_fragment_errors=107843,
rx_illegal_byte_errors=0, rx_jabber_errors=0, rx_length_errors=0,
rx_mac_short_packet_dropped=1906, rx_management_dropped=0,
rx_management_packets=13, rx_mbuf_allocation_errors=0,
rx_missed_errors=0, rx_oversize_errors=0, rx_packets=71593333,
rx_priority0_dropped=0, rx_priority0_mbuf_allocation_errors=1131,
rx_priority1_dropped=0, rx_priority1_mbuf_allocation_errors=0,
rx_priority2_dropped=0, rx_priority2_mbuf_allocation_errors=0,
rx_priority3_dropped=0, rx_priority3_mbuf_allocation_errors=0,
rx_priority4_dropped=0, rx_priority4_mbuf_allocation_errors=0,
rx_priority5_dropped=0, rx_priority5_mbuf_allocation_errors=0,
rx_priority6_dropped=0, rx_priority6_mbuf_allocation_errors=0,
rx_priority7_dropped=0, rx_priority7_mbuf_allocation_errors=0,
rx_undersize_errors=7049788, tx_128_to_255_packets=102664472,
tx_1_to_64_packets=1906, tx_256_to_511_packets=68008814,
tx_512_to_1023_packets=126793, tx_65_to_127_packets=1412435,
tx_broadcast_packets=1464, tx_bytes=40693963125, tx_dropped=0,
tx_errors=0, tx_management_packets=199058, tx_multicast_packets=146,
tx_packets=172252389}