[ovs-discuss] OVS DPDK performance with SELECT group

Rami Neiman russel777 at gmail.com
Tue Nov 26 15:41:45 UTC 2019


Hello,

I am using OVS DPDK 2.9.2 with TRex traffic generator to simply forward the
received traffic back to the traffic generator (i.e. ingress0->egeress0,
egress0->ingress0) over 2 port 10G NIC.

The OVS throughput with this setup matches the traffic generator (all
packets sent by TG are received). And we are getting around 2.5Mpps of
traffic forwarded fine (we can probably go even higher, so that’s not a
limit).

Our next goal is to have the TG traffic also mirrored over additional 2 10G
ports to a monitoring device and we use SELECT group to achieve load
balancing of mirrored traffic. When we add the group as follows:



ovs-ofctl -O OpenFlow13 add-group br0
group_id=5,type=select,bucket=output:mirror0,bucket=output:mirror1



ovs-ofctl -O OpenFlow13 add-flow br0 "table=5,
metadata=0,in_port=egress0,actions=group:5,output:ingress0"

ovs-ofctl -O OpenFlow13 add-flow br0 "table=5,
metadata=0,in_port=ingress0,actions=group:5,output:egress0"



mirror0 and mirror1 being our mirror ports. The mirroring works as
expected, however the OVS throughput drops to less than 500 Kpps (as
reported by the traffic generator).



The ingress0 and egress0 (i.e. ports that receive traffic) show packets
being dropped in large numbers. Adding more pmd cores and distributing Rx
queues among them has no effect. Changing the hash fields of the SELECT
group has no effect either.

My question is: is there a way to give more cores/memory or otherwise
influence the hash calculation and SELECT group action to make it more
performant? less than 500Kpps seems like a very low number.



Just in case, here’s the output of the most important statistics commands:



ovs-vsctl --column statistics list interface egress0

statistics          : {flow_director_filter_add_errors=0,
flow_director_filter_remove_errors=0, mac_local_errors=17,
mac_remote_errors=1, "rx_128_to_255_packets"=3936120,
"rx_1_to_64_packets"=14561687, "rx_256_to_511_packets"=1624884,
"rx_512_to_1023_packets"=2180436, "rx_65_to_127_packets"=21519189,
rx_broadcast_packets=17, rx_bytes=23487692367, rx_crc_errors=0,
rx_dropped=23759559, rx_errors=0, rx_fcoe_crc_errors=0, rx_fcoe_dropped=0,
rx_fcoe_mbuf_allocation_errors=0, rx_fragment_errors=0,
rx_illegal_byte_errors=0, rx_jabber_errors=0, rx_length_errors=0,
rx_mac_short_packet_dropped=0, rx_management_dropped=0,
rx_management_packets=0, rx_mbuf_allocation_errors=0,
rx_missed_errors=23759559, rx_oversize_errors=0, rx_packets=39363905,
"rx_priority0_dropped"=23759559, "rx_priority0_mbuf_allocation_errors"=0,
"rx_priority1_dropped"=0, "rx_priority1_mbuf_allocation_errors"=0,
"rx_priority2_dropped"=0, "rx_priority2_mbuf_allocation_errors"=0,
"rx_priority3_dropped"=0, "rx_priority3_mbuf_allocation_errors"=0,
"rx_priority4_dropped"=0, "rx_priority4_mbuf_allocation_errors"=0,
"rx_priority5_dropped"=0, "rx_priority5_mbuf_allocation_errors"=0,
"rx_priority6_dropped"=0, "rx_priority6_mbuf_allocation_errors"=0,
"rx_priority7_dropped"=0, "rx_priority7_mbuf_allocation_errors"=0,
rx_undersize_errors=0, "tx_128_to_255_packets"=1549647,
"tx_1_to_64_packets"=10995089, "tx_256_to_511_packets"=7309468,
"tx_512_to_1023_packets"=739062, "tx_65_to_127_packets"=7837579,
tx_broadcast_packets=6, tx_bytes=28481732482, tx_dropped=0, tx_errors=0,
tx_management_packets=0, tx_multicast_packets=0, tx_packets=43936201}



ovs-vsctl --column statistics list interface ingress0

statistics          : {flow_director_filter_add_errors=0,
flow_director_filter_remove_errors=0, mac_local_errors=37,
mac_remote_errors=1, "rx_128_to_255_packets"=2778420,
"rx_1_to_64_packets"=18198197, "rx_256_to_511_packets"=13168041,
"rx_512_to_1023_packets"=886524, "rx_65_to_127_packets"=14853438,
rx_broadcast_packets=17, rx_bytes=28481734408, rx_crc_errors=0,
rx_dropped=22718779, rx_errors=0, rx_fcoe_crc_errors=0, rx_fcoe_dropped=0,
rx_fcoe_mbuf_allocation_errors=0, rx_fragment_errors=0,
rx_illegal_byte_errors=0, rx_jabber_errors=0, rx_length_errors=0,
rx_mac_short_packet_dropped=0, rx_management_dropped=0,
rx_management_packets=0, rx_mbuf_allocation_errors=0,
rx_missed_errors=22718779, rx_oversize_errors=0, rx_packets=43936225,
"rx_priority0_dropped"=22718779, "rx_priority0_mbuf_allocation_errors"=0,
"rx_priority1_dropped"=0, "rx_priority1_mbuf_allocation_errors"=0,
"rx_priority2_dropped"=0, "rx_priority2_mbuf_allocation_errors"=0,
"rx_priority3_dropped"=0, "rx_priority3_mbuf_allocation_errors"=0,
"rx_priority4_dropped"=0, "rx_priority4_mbuf_allocation_errors"=0,
"rx_priority5_dropped"=0, "rx_priority5_mbuf_allocation_errors"=0,
"rx_priority6_dropped"=0, "rx_priority6_mbuf_allocation_errors"=0,
"rx_priority7_dropped"=0, "rx_priority7_mbuf_allocation_errors"=0,
rx_undersize_errors=0, "tx_128_to_255_packets"=1793095,
"tx_1_to_64_packets"=7027091, "tx_256_to_511_packets"=783763,
"tx_512_to_1023_packets"=1133960, "tx_65_to_127_packets"=14219400,
tx_broadcast_packets=6, tx_bytes=23487691707, tx_dropped=0, tx_errors=0,
tx_management_packets=0, tx_multicast_packets=0, tx_packets=39363894}





ovs-appctl dpif-netdev/pmd-rxq-show

pmd thread numa_id 0 core_id 2:

        isolated : true

        port: egress0           queue-id:  0    pmd usage:  0 %

        port: ingress0          queue-id:  0    pmd usage:  0 %

        port: mirror0           queue-id:  0    pmd usage:  0 %

pmd thread numa_id 0 core_id 3:

        isolated : true

        port: egress0           queue-id:  1    pmd usage:  0 %

        port: ingress0          queue-id:  1    pmd usage:  0 %

        port: mirror1           queue-id:  0    pmd usage:  0 %

pmd thread numa_id 0 core_id 4:

        isolated : true

        port: egress0           queue-id:  2    pmd usage:  0 %

        port: ingress0          queue-id:  2    pmd usage:  0 %

pmd thread numa_id 0 core_id 5:

        isolated : true

        port: egress0           queue-id:  3    pmd usage:  0 %

        port: ingress0          queue-id:  3    pmd usage:  0 %







ovs-appctl dpif-netdev/pmd-stats-show

pmd thread numa_id 0 core_id 2:

        packets received: 21323462

        packet recirculations: 0

        avg. datapath passes per packet: 1.00

        emc hits: 5119195

        megaflow hits: 8461953

        avg. subtable lookups per megaflow hit: 1.01

        miss with success upcall: 2286723

        miss with failed upcall: 5455591

        avg. packets per output batch: 2.81

        idle cycles: 18540475978691 (98.75%)

        processing cycles: 235616197026 (1.25%)

        avg cycles per packet: 880536.76 (18776092175717/21323462)

        avg processing cycles per packet: 11049.62 (235616197026/21323462)

pmd thread numa_id 0 core_id 3:

        packets received: 20654639

        packet recirculations: 0

        avg. datapath passes per packet: 1.00

        emc hits: 4449782

        megaflow hits: 7736708

        avg. subtable lookups per megaflow hit: 1.00

        miss with success upcall: 2567728

        miss with failed upcall: 5900421

        avg. packets per output batch: 3.00

        idle cycles: 18531334403507 (98.69%)

        processing cycles: 245349593515 (1.31%)

        avg cycles per packet: 909078.29 (18776683997022/20654639)

        avg processing cycles per packet: 11878.67 (245349593515/20654639)

pmd thread numa_id 0 core_id 4:

        packets received: 20430879

        packet recirculations: 0

        avg. datapath passes per packet: 1.00

        emc hits: 4361365

        megaflow hits: 8641516

        avg. subtable lookups per megaflow hit: 1.00

        miss with success upcall: 2175208

        miss with failed upcall: 5252790

        avg. packets per output batch: 2.86

        idle cycles: 18547971632247 (98.79%)

        processing cycles: 228120403283 (1.21%)

        avg cycles per packet: 919005.59 (18776092035530/20430879)

        avg processing cycles per packet: 11165.47 (228120403283/20430879)

pmd thread numa_id 0 core_id 5:

        packets received: 20891150

        packet recirculations: 0

        avg. datapath passes per packet: 1.00

        emc hits: 4679834

        megaflow hits: 8247466

        avg. subtable lookups per megaflow hit: 1.00

        miss with success upcall: 2350916

        miss with failed upcall: 5612934

        avg. packets per output batch: 2.92

        idle cycles: 18540271879537 (98.74%)

        processing cycles: 235820084225 (1.26%)

        avg cycles per packet: 898758.18 (18776091963762/20891150)

        avg processing cycles per packet: 11288.04 (235820084225/20891150)

main thread:

        packets received: 0

        packet recirculations: 0

        avg. datapath passes per packet: 0.00

        emc hits: 0

        megaflow hits: 0

        avg. subtable lookups per megaflow hit: 0.00

        miss with success upcall: 0

        miss with failed upcall: 0

        avg. packets per output batch: 0.00





Thank you in advance for your time

Rami Neiman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20191126/40a41ae1/attachment-0001.html>


More information about the discuss mailing list