[ovs-dev] AddressSanitizer: heap-use-after-free in rcu thread after ofproto destroy.
Ilya Maximets
i.maximets at ovn.org
Fri Feb 19 11:17:22 UTC 2021
Caught this only once in GHA about a week ago while running tests
with AddressSanitizer, can't reproduce. Might be similar to some
other ofproto issues seen recently.
The problem is that free_meter_id() uses 'baker' pointer from the
'ofproto', but destruction of the 'baker' is not even postponed as
dealloc of 'ofproto'. And while 'beaker' has a refcount, postponing
of free_meter_id() doesn't increase a refcount.
Best regards, Ilya Maximets.
1070. ofproto-dpif.at:2091: testing ofproto-dpif - controller action without megaflows ...
==6622==ERROR: AddressSanitizer: heap-use-after-free on address 0x614000000e50 at pc 0x000000534dc7 bp 0x7f83f76fd990 sp 0x7f83f76fd988
READ of size 8 at 0x614000000e50 thread T4 (urcu3)
#0 0x534dc6 in free_meter_id /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto-dpif.c:6622:37
#1 0x727fa0 in ovsrcu_call_postponed /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/ovs-rcu.c:348:13
#2 0x7283f1 in ovsrcu_postpone_thread /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/ovs-rcu.c:364:14
#3 0x72cafc in ovsthread_wrapper /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/ovs-thread.c:383:12
#4 0x7f83fcafe6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
#5 0x7f83fc07d71e in clone (/lib/x86_64-linux-gnu/libc.so.6+0x12171e)
0x614000000e50 is located 16 bytes inside of 400-byte region [0x614000000e40,0x614000000fd0)
freed by thread T0 here:
#0 0x49638d in free (/home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/vswitchd/ovs-vswitchd+0x49638d)
#1 0x5176d8 in destruct /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto-dpif.c:1803:5
#2 0x4f0610 in ofproto_destroy /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto.c:1730:5
#3 0x4c7164 in bridge_destroy /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/bridge.c:3606:9
#4 0x4c6eca in bridge_exit /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/bridge.c:553:9
#5 0x4e0fba in main /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/ovs-vswitchd.c:143:5
#6 0x7f83fbf7dbf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)
previously allocated by thread T0 here:
#0 0x49660d in malloc (/home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/vswitchd/ovs-vswitchd+0x49660d)
#1 0x79f0e6 in xmalloc /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/util.c:138:15
#2 0x5281fd in open_dpif_backer /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto-dpif.c:774:14
#3 0x516cbb in construct /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto-dpif.c:1615:13
#4 0x4ec3c0 in ofproto_create /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto.c:549:13
#5 0x4c7f27 in bridge_reconfigure /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/bridge.c:882:21
#6 0x4c7455 in bridge_run /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/bridge.c:3331:9
#7 0x4e0ed1 in main /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/ovs-vswitchd.c:127:9
#8 0x7f83fbf7dbf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)
Thread T4 (urcu3) created by T2 (ct_clean1) here:
#0 0x480d9a in pthread_create (/home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/vswitchd/ovs-vswitchd+0x480d9a)
#1 0x72c717 in ovs_thread_create /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/ovs-thread.c:447:13
#2 0x72793a in ovsrcu_quiesced /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/ovs-rcu.c:123:13
#3 0x78d083 in time_poll /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/timeval.c:321:17
#4 0x757cc7 in poll_block /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/poll-loop.c:364:14
#5 0x88cbbc in clean_thread_main /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/conntrack.c:1583:9
#6 0x72cafc in ovsthread_wrapper /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/ovs-thread.c:383:12
#7 0x7f83fcafe6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
Thread T2 (ct_clean1) created by T0 here:
#0 0x480d9a in pthread_create (/home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/vswitchd/ovs-vswitchd+0x480d9a)
#1 0x72c717 in ovs_thread_create /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/ovs-thread.c:447:13
#2 0x88c9dc in conntrack_init /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/conntrack.c:316:24
#3 0x5d0565 in create_dp_netdev /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/dpif-netdev.c:1794:21
#4 0x5ca96e in dpif_netdev_open /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/dpif-netdev.c:1850:26
#5 0x5f20e1 in do_open /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/dpif.c:347:13
#6 0x5f2498 in dpif_create_and_open /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../lib/dpif.c:415:13
#7 0x528217 in open_dpif_backer /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto-dpif.c:776:13
#8 0x516cbb in construct /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto-dpif.c:1615:13
#9 0x4ec3c0 in ofproto_create /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto.c:549:13
#10 0x4c7f27 in bridge_reconfigure /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/bridge.c:882:21
#11 0x4c7455 in bridge_run /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/bridge.c:3331:9
#12 0x4e0ed1 in main /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../vswitchd/ovs-vswitchd.c:127:9
#13 0x7f83fbf7dbf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)
SUMMARY: AddressSanitizer: heap-use-after-free /home/runner/work/ovs/ovs/openvswitch-2.15.90/_build/sub/../../ofproto/ofproto-dpif.c:6622:37 in free_meter_id
Shadow bytes around the buggy address:
0x0c287fff8170: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
0x0c287fff8180: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c287fff8190: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c287fff81a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c287fff81b0: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
=>0x0c287fff81c0: fa fa fa fa fa fa fa fa fd fd[fd]fd fd fd fd fd
0x0c287fff81d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c287fff81e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c287fff81f0: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
0x0c287fff8200: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
0x0c287fff8210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==6622==ABORTING
More information about the dev
mailing list