[ovs-dev] [PATCHv18] netdev-afxdp: add new netdev type for AF_XDP.

William Tu u9012063 at gmail.com
Fri Aug 23 16:08:17 UTC 2019


On Wed, Aug 21, 2019 at 2:31 AM Eelco Chaudron <echaudro at redhat.com> wrote:
>
>
>
> >>> William, Eelco, which HW NIC you're using? Which kernel driver?
> >>
> >> I’m using the below on the latest bpf-next driver:
> >>
> >> 01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
> >> SFI/SFP+ Network Connection (rev 01)
> >> 01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
> >> SFI/SFP+ Network Connection (rev 01)
> >
> > Thanks for information.
> > I found one suspicious place inside the ixgbe driver that could break
> > the completion queue ring and prepared a patch:
> >     https://patchwork.ozlabs.org/patch/1150244/
> >
> > It'll be good if you can test it.
>
> Hi Ilya, I was doping some testing of my own, and also concluded it was
> in the drivers' completion ring. I noticed after sending 512 packets the
> drivers TX counters kept increasing, which looks related to your fix.
>
> Will try it out, and sent results to the upstream mailing list…
>
> Thanks,
>
> Eelco

Hi,

I'm comparing the performance of netdev-afxdp.c on current master and
the DPDK's AF_XDP implementation in OVS dpdk-latest branch.
I'm using ixgbe and doing physical port to physical port forwarding, sending
64 byte packets, with OpenFlow rule:
  ovs-ofctl  add-flow br0  "in_port=eth2, actions=output:eth3"

In short
A. OVS's netdev-afxdp: 6.1Mpps
B. OVS-DPDK  AF_XDP pmd: 8Mpps
So I start to think about how to optimize lib/netdev-afxdp.c. Any comments are
welcomed! Below is the analysis:

A. OVS netdev-afxdp Physical to physical 6.1Mpps
# pstree -p 702
ovs-vswitchd(702)-+-{ct_clean1) S 1 7(706)
                  |-{handler4}(712)
                  |-{ipf_clean2}(707)
                  |-{pmd6}(790)
                  |-{pmd7}(791)
                  |-{pmd8}(792)
                  |-{pmd9}(793)
                |-{revalidator5}(713)
                  `-{urcu3}(708)

# ovs-appctl  dpif-netdev/pmd-stats-show
pmd thread numa_id 0 core_id 6:
  packets received: 92290351
  packet recirculations: 0
  avg. datapath passes per packet: 1.00
  emc hits: 92290319
  smc hits: 0
  megaflow hits: 31
  avg. subtable lookups per megaflow hit: 1.00
  miss with success upcall: 1
  miss with failed upcall: 0
  avg. packets per output batch: 31.88
  idle cycles: 20835727677 (34.86%)           --> pretty high!?
  processing cycles: 38932097052 (65.14%)
  avg cycles per packet: 647.61 (59767824729/92290351)
  avg processing cycles per packet: 421.84 (38932097052/92290351)

# ./perf record -t 790 sleep 10
  13.80%  pmd6 ovs-vswitchd        [.] miniflow_extract
  13.58%  pmd6 ovs-vswitchd        [.] __netdev_afxdp_batch_send
   9.64%  pmd6   ovs-vswitchd        [.] dp_netdev_input__
   9.07%  pmd6   ovs-vswitchd        [.] dp_packet_init__
   8.91%  pmd6   ovs-vswitchd        [.] netdev_afxdp_rxq_recv
   7.40%  pmd6   ovs-vswitchd        [.] miniflow_hash_5tuple
   5.32%  pmd6   libc-2.23.so        [.] __memcpy_avx_unaligned
   4.60%  pmd6   [kernel.vmlinux]    [k] do_syscall_64
   3.72%  pmd6   ovs-vswitchd        [.] dp_packet_use_afxdp    -->
maybe optimize?
   2.74%  pmd6   libpthread-2.23.so  [.] __pthread_enable_asynccancel
   2.43%  pmd6   [kernel.vmlinux]    [k] fput_many
   2.18%  pmd6   libc-2.23.so        [.] __memcmp_sse4_1
   2.06%  pmd6   [kernel.vmlinux]    [k] entry_SYSCALL_64
   1.79%  pmd6   [kernel.vmlinux]    [k] syscall_return_via_sysret
   1.71%  pmd6   ovs-vswitchd        [.] dp_execute_cb
   1.03%  pmd6   ovs-vswitchd        [.] non_atomic_ullong_add
   0.86%  pmd6   ovs-vswitchd        [.]dp_netdev_pmd_flush_output_on_port

B. OVS-DPDK afxdp using dpdk-latest 8Mpps
ovs-vswitchd(19462)-+-{ct_clean3}(19470)
                    |-{dpdk_watchdog1}(19468)
                    |-{eal-intr-thread}(19466)
                    |-{handler16}(19501)
                    |-{handler17}(19505)
                    |-{handler18}(19506)
                    |-{handler19}(19507)
                    |-{handler20}(19508)
                    |-{handler22}(19502)
                    |-{handler24}(19504)
                    |-{handler26}(19503)
                    |-{ipf_clean4}(19471)
                    |-{pmd27}(19536)
                    |-{revalidator21}(19509)
                    |-{revalidator23}(19511)
                    |-{revalidator25}(19510)
                    |-{rte_mp_handle}(19467)
                    `-{urcu2}(19469)

# ovs-appctl  dpif-netdev/pmd-stats-show
pmd thread numa_id 0 core_id 11:
  packets received: 1813689117
  packet recirculations: 0
  avg. datapath passes per packet: 1.00
  emc hits: 1813689053
  smc hits: 0
  megaflow hits: 63
  avg. subtable lookups per megaflow hit: 1.00
  miss with success upcall: 1
  miss with failed upcall: 0
  avg. packets per output batch: 31.85
  idle cycles: 13848892341 (2.50%)
  processing cycles: 541064826249 (97.50%)
  avg cycles per packet: 305.96 (554913718590/1813689117)
  avg processing cycles per packet: 298.32 (541064826249/1813689117)

#  ./perf record -t 19536 sleep 10
  24.84%  pmd27 ovs-vswitchd        [.] eth_af_xdp_rx
  16.27%  pmd27 ovs-vswitchd        [.] eth_af_xdp_tx
  13.20%  pmd27 ovs-vswitchd        [.] dp_netdev_input__
  12.54%  pmd27 ovs-vswitchd        [.] pull_umem_cq
  10.85%  pmd27 ovs-vswitchd        [.] miniflow_extract
   5.67%  pmd27   ovs-vswitchd        [.] miniflow_hash_5tuple
   3.41%  pmd27   libc-2.23.so        [.] __memcmp_sse4_1
   2.14%  pmd27   ovs-vswitchd        [.] netdev_dpdk_rxq_recv
   2.13%  pmd27   ovs-vswitchd        [.] dp_execute_cb
   1.50%  pmd27   ovs-vswitchd        [.] non_atomic_ullong_add
   1.49%  pmd27   ovs-vswitchd        [.] dp_netdev_pmd_flush_output_on_port
   1.05%  pmd27   ovs-vswitchd        [.] netdev_dpdk_filter_packet_len
   0.79%  pmd27   ovs-vswitchd        [.] pmd_perf_end_iteration
   0.74%  pmd27   ovs-vswitchd        [.] dp_netdev_process_rxq_port
   0.47%  pmd27   ovs-vswitchd        [.] memcmp at plt
   0.42%  pmd27   ovs-vswitchd        [.] netdev_dpdk_eth_send


More information about the dev mailing list