[ovs-dev] AddressSanitizer: heap-use-after-free in rcu thread after ofproto destroy.

Ilya Maximets i.maximets at ovn.org
Fri Feb 19 11:17:22 UTC 2021


Caught this only once in GHA about a week ago while running tests
with AddressSanitizer, can't reproduce.  Might be similar to some
other ofproto issues seen recently.

The problem is that free_meter_id() uses 'baker' pointer from the
'ofproto', but destruction of the 'baker' is not even postponed as
dealloc of 'ofproto'.   And while 'beaker' has a refcount, postponing
of free_meter_id() doesn't increase a refcount.

Best regards, Ilya Maximets.


1070. ofproto-dpif.at:2091: testing ofproto-dpif - controller action without megaflows ...
==6622==ERROR: AddressSanitizer: heap-use-after-free on address 0x614000000e50 at pc 0x000000534dc7 bp 0x7f83f76fd990 sp 0x7f83f76fd988
READ of size 8 at 0x614000000e50 thread T4 (urcu3)
    #0 0x534dc6 in free_meter_id /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto-dpif.c:6622:37
    #1 0x727fa0 in ovsrcu_call_postponed /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/ovs-rcu.c:348:13
    #2 0x7283f1 in ovsrcu_postpone_thread /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/ovs-rcu.c:364:14
    #3 0x72cafc in ovsthread_wrapper /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/ovs-thread.c:383:12
    #4 0x7f83fcafe6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #5 0x7f83fc07d71e in clone (/lib/x86_64-linux-gnu/libc.so.6+0x12171e)

0x614000000e50 is located 16 bytes inside of 400-byte region [0x614000000e40,0x614000000fd0)
freed by thread T0 here:
    #0 0x49638d in free (/home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/vswitchd/ovs-vswitchd+0x49638d)
    #1 0x5176d8 in destruct /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto-dpif.c:1803:5
    #2 0x4f0610 in ofproto_destroy /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto.c:1730:5
    #3 0x4c7164 in bridge_destroy /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/bridge.c:3606:9
    #4 0x4c6eca in bridge_exit /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/bridge.c:553:9
    #5 0x4e0fba in main /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/ovs-vswitchd.c:143:5
    #6 0x7f83fbf7dbf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)

previously allocated by thread T0 here:
    #0 0x49660d in malloc (/home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/vswitchd/ovs-vswitchd+0x49660d)
    #1 0x79f0e6 in xmalloc /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/util.c:138:15
    #2 0x5281fd in open_dpif_backer /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto-dpif.c:774:14
    #3 0x516cbb in construct /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto-dpif.c:1615:13
    #4 0x4ec3c0 in ofproto_create /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto.c:549:13
    #5 0x4c7f27 in bridge_reconfigure /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/bridge.c:882:21
    #6 0x4c7455 in bridge_run /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/bridge.c:3331:9
    #7 0x4e0ed1 in main /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/ovs-vswitchd.c:127:9
    #8 0x7f83fbf7dbf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)

Thread T4 (urcu3) created by T2 (ct_clean1) here:
    #0 0x480d9a in pthread_create (/home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/vswitchd/ovs-vswitchd+0x480d9a)
    #1 0x72c717 in ovs_thread_create /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/ovs-thread.c:447:13
    #2 0x72793a in ovsrcu_quiesced /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/ovs-rcu.c:123:13
    #3 0x78d083 in time_poll /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/timeval.c:321:17
    #4 0x757cc7 in poll_block /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/poll-loop.c:364:14
    #5 0x88cbbc in clean_thread_main /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/conntrack.c:1583:9
    #6 0x72cafc in ovsthread_wrapper /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/ovs-thread.c:383:12
    #7 0x7f83fcafe6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)

Thread T2 (ct_clean1) created by T0 here:
    #0 0x480d9a in pthread_create (/home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/vswitchd/ovs-vswitchd+0x480d9a)
    #1 0x72c717 in ovs_thread_create /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/ovs-thread.c:447:13
    #2 0x88c9dc in conntrack_init /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/conntrack.c:316:24
    #3 0x5d0565 in create_dp_netdev /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/dpif-netdev.c:1794:21
    #4 0x5ca96e in dpif_netdev_open /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/dpif-netdev.c:1850:26
    #5 0x5f20e1 in do_open /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/dpif.c:347:13
    #6 0x5f2498 in dpif_create_and_open /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/dpif.c:415:13
    #7 0x528217 in open_dpif_backer /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto-dpif.c:776:13
    #8 0x516cbb in construct /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto-dpif.c:1615:13
    #9 0x4ec3c0 in ofproto_create /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto.c:549:13
    #10 0x4c7f27 in bridge_reconfigure /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/bridge.c:882:21
    #11 0x4c7455 in bridge_run /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/bridge.c:3331:9
    #12 0x4e0ed1 in main /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/ovs-vswitchd.c:127:9
    #13 0x7f83fbf7dbf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)

SUMMARY: AddressSanitizer: heap-use-after-free /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto-dpif.c:6622:37 in free_meter_id
Shadow bytes around the buggy address:
  0x0c287fff8170: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
  0x0c287fff8180: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c287fff8190: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c287fff81a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c287fff81b0: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
=>0x0c287fff81c0: fa fa fa fa fa fa fa fa fd fd[fd]fd fd fd fd fd
  0x0c287fff81d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c287fff81e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c287fff81f0: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
  0x0c287fff8200: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c287fff8210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==6622==ABORTING


More information about the dev mailing list