[ovs-dev] [PATCHv13] netdev-afxdp: add new netdev type for AF_XDP.

Fri Jun 21 14:56:28 UTC 2019

On 19.06.2019 22:51, William Tu wrote:
> The patch introduces experimental AF_XDP support for OVS netdev.
> AF_XDP, the Address Family of the eXpress Data Path, is a new Linux socket
> type built upon the eBPF and XDP technology.  It is aims to have comparable
> performance to DPDK but cooperate better with existing kernel's networking
> stack.  An AF_XDP socket receives and sends packets from an eBPF/XDP program
> attached to the netdev, by-passing a couple of Linux kernel's subsystems
> As a result, AF_XDP socket shows much better performance than AF_PACKET
> For more details about AF_XDP, please see linux kernel's
> Documentation/networking/af_xdp.rst. Note that by default, this feature is
> not compiled in.
> 
> Signed-off-by: William Tu <u9012063 at gmail.com>
> ---

Hi!
This is about "conntrack - IP fragmentation expiry" tests I mentioned in a
previous mail:
    https://mail.openvswitch.org/pipermail/ovs-dev/2019-June/359971.html

There is a major bug related to a memory pools management. The issue is that
we *must not* free memory pool until there are packets from it in use by
any other code. For example, packets could be delayed for the future processing
like it happens in case of IP fragments re-assembly. We fixed same issue for
DPDK around a year ago. In practice, we must postpone actual freeing of
umem->buffer, umem_pool and the xpacket_pool until all packets freed i.e.
umemp->index != umemp->size.

You may use 'dpdk_mp_sweep' as a reference.

Here is a crash log I have with IPv4 fragmentation test:

# make -j8 check-afxdp TESTSUITEFLAGS='-v 52'
52. system-traffic.at:2398: testing conntrack - IPv4 fragmentation expiry

=================================================================
==17056==ERROR: AddressSanitizer: heap-use-after-free on address 0x7f55ab131f34 at pc 0x0000005a150f bp 0x7ffea9a44e80 sp 0x7ffea9a44e70
READ of size 1 at 0x7f55ab131f34 thread T0
    #0 0x5a150e in dp_packet_delete lib/dp-packet.h:191
    #1 0x5a150e in ipf_destroy lib/ipf.c:1332
    #2 0x8701bd in conntrack_destroy lib/conntrack.c:393
    #3 0x55bdb4 in dp_netdev_free lib/dpif-netdev.c:1642
    #4 0x55c7a7 in dp_netdev_unref lib/dpif-netdev.c:1678
    #5 0x55c83e in dp_netdev_unref lib/dpif-netdev.c:1673
    #6 0x55c83e in dpif_netdev_close lib/dpif-netdev.c:1689
    #7 0x576e25 in dpif_uninit lib/dpif.c:1683
    #8 0x576f62 in dpif_close lib/dpif.c:453
    #9 0x470a3e in close_dpif_backer ofproto/ofproto-dpif.c:684
    #10 0x47c7ef in destruct ofproto/ofproto-dpif.c:1658
    #11 0x45b519 in ofproto_destroy ofproto/ofproto.c:1665
    #12 0x414065 in bridge_destroy vswitchd/bridge.c:3319
    #13 0x426aa1 in bridge_exit vswitchd/bridge.c:509
    #14 0x409b97 in main vswitchd/ovs-vswitchd.c:143
    #15 0x7f55b44a7812 in __libc_start_main (/lib64/libc.so.6+0x23812)
    #16 0x40c46d in _start (/root/git/ovs/vswitchd/ovs-vswitchd+0x40c46d)

0x7f55ab131f34 is located 7988 bytes inside of 4653056-byte region [0x7f55ab130000,0x7f55ab5a0000)
freed by thread T0 here:
    #0 0x7f55b5e943a0 in free (/lib64/libasan.so.5+0xef3a0)
    #1 0x89fd3e in xpacket_pool_cleanup lib/xdpsock.c:168
    #2 0x7ea714 in xsk_destroy lib/netdev-afxdp.c:298
    #3 0x7ea714 in xsk_destroy_all lib/netdev-afxdp.c:315
    #4 0x7ee5d9 in netdev_afxdp_destruct lib/netdev-afxdp.c:836
    #5 0x5e8d80 in netdev_unref lib/netdev.c:577
    #6 0x4418ea in ofport_destroy__ ofproto/ofproto.c:2539
    #7 0x45b687 in ofproto_destroy ofproto/ofproto.c:1658
    #8 0x414065 in bridge_destroy vswitchd/bridge.c:3319
    #9 0x426aa1 in bridge_exit vswitchd/bridge.c:509
    #10 0x409b97 in main vswitchd/ovs-vswitchd.c:143
    #11 0x7f55b44a7812 in __libc_start_main (/lib64/libc.so.6+0x23812)

previously allocated by thread T0 here:
    #0 0x7f55b5e95580 in posix_memalign (/lib64/libasan.so.5+0xf0580)
    #1 0x780c82 in xmalloc_size_align lib/util.c:229
    #2 0x89fcb4 in xpacket_pool_init lib/xdpsock.c:156
    #3 0x7eb2da in xsk_configure_umem lib/netdev-afxdp.c:107
    #4 0x7eb2da in xsk_configure lib/netdev-afxdp.c:222
    #5 0x7eb2da in xsk_configure_all lib/netdev-afxdp.c:260
    #6 0x7eb2da in netdev_afxdp_reconfigure lib/netdev-afxdp.c:449
    #7 0x559835 in port_reconfigure lib/dpif-netdev.c:4330
    #8 0x559835 in reconfigure_datapath lib/dpif-netdev.c:4838
    #9 0x55b1cc in do_add_port lib/dpif-netdev.c:1842
    #10 0x55b683 in dpif_netdev_port_add lib/dpif-netdev.c:1868
    #11 0x5746c2 in dpif_port_add lib/dpif.c:577
    #12 0x4743b6 in port_add ofproto/ofproto-dpif.c:3713
    #13 0x44eac5 in ofproto_port_add ofproto/ofproto.c:2013
    #14 0x41536d in iface_do_create vswitchd/bridge.c:1811
    #15 0x41536d in iface_create vswitchd/bridge.c:1849
    #16 0x41536d in bridge_add_ports__ vswitchd/bridge.c:937
    #17 0x41c1a4 in bridge_add_ports vswitchd/bridge.c:953
    #18 0x41c1a4 in bridge_reconfigure vswitchd/bridge.c:667
    #19 0x4274ee in bridge_run vswitchd/bridge.c:3044
    #20 0x409a0c in main vswitchd/ovs-vswitchd.c:127
    #21 0x7f55b44a7812 in __libc_start_main (/lib64/libc.so.6+0x23812)

SUMMARY: AddressSanitizer: heap-use-after-free lib/dp-packet.h:191 in dp_packet_delete
Shadow bytes around the buggy address:
  0x0feb3561e390: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0feb3561e3a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0feb3561e3b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0feb3561e3c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0feb3561e3d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0feb3561e3e0: fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd
  0x0feb3561e3f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0feb3561e400: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0feb3561e410: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0feb3561e420: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0feb3561e430: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==17056==ABORTING

As you can see above, 'ipf_destroy' tries to free delayed dp-packet while
xpacket_pool already destroyed.

Best regards, Ilya Maximets.