[ovs-discuss] DPDK-OvS exception during stress test with ONOS

Ali Volkan Atli Volkan.Atli at argela.com.tr
Wed Oct 12 08:51:11 UTC 2016


We developed a custom application on top of ONOS and it pushes many flow-mods and packet-outs into OvS and also we have added meter codes into OvS. Therefore I think you cannot reproduce the problem, also I'm not sure the problem coming from OvS or our codes. I'm investigating and if you have any suggestion to fix the problem, that would be appreciated.

- Volkan
________________________________________
From: Daniele Di Proietto [diproiettod at ovn.org]
Sent: Wednesday, October 12, 2016 2:33 AM
To: Ali Volkan Atli
Cc: discuss at openvswitch.org
Subject: Re: [ovs-discuss] DPDK-OvS exception during stress test with ONOS

Could you share more details about the setup, please?

Which version of Open vSwitch are you using? (the exact commit would be helpful)
What's the controller?
What's the distro?

Thanks,

Daniele

2016-10-11 1:44 GMT-07:00 Ali Volkan Atli <Volkan.Atli at argela.com.tr<mailto:Volkan.Atli at argela.com.tr>>:
Hi OvS

I'm getting the same exception each time (dp_packet_delete) while 3 iperf3 traffic running between 6 qemu hosts. Also, when I start ovs with valgrind (--leak-check=yes), I cannot see the problem because of slowness. Any suggestion would be appreciated. Thanks in advance.

- Volkan

2016-10-08T13:46:47Z|01582|connmgr|INFO|Dropped 1994 log messages in last 6 seconds (most recently, 1 seconds ago) due to excessive rate
2016-10-08T13:46:47Z|01583|connmgr|INFO|e20<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: sending OFPMMFC_INVALID_METER error reply to OFPT_FLOW_MOD message
2016-10-08T13:46:48Z|01584|connmgr|INFO|e20<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: 9 flow_mods 10 s ago (9 adds)
2016-10-08T13:46:51Z|01585|connmgr|INFO|Dropped 103375 log messages in last 12 seconds (most recently, 0 seconds ago) due to excessive rate
2016-10-08T13:46:51Z|01586|connmgr|INFO|e20<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: dropping packet-in due to queue overflow
2016-10-08T13:46:53Z|01587|connmgr|INFO|Dropped 1591 log messages in last 6 seconds (most recently, 1 seconds ago) due to excessive rate
2016-10-08T13:46:53Z|01588|connmgr|INFO|e20<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: sending OFPMMFC_INVALID_METER error reply to OFPT_FLOW_MOD message
2016-10-08T13:46:59Z|01589|connmgr|INFO|Dropped 1448 log messages in last 6 seconds (most recently, 1 seconds ago) due to excessive rate
2016-10-08T13:46:59Z|01590|connmgr|INFO|e20<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: sending OFPMMFC_INVALID_METER error reply to OFPT_FLOW_MOD message
2016-10-08T13:47:00Z|01591|connmgr|INFO|c02<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: 1 flow_mods 29 s ago (1 deletes)
2016-10-08T13:47:00Z|01592|connmgr|INFO|a21<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: 1 flow_mods 24 s ago (1 deletes)
2016-10-08T13:47:00Z|01593|connmgr|INFO|a01<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: 1 flow_mods 24 s ago (1 deletes)
2016-10-08T13:47:03Z|01594|connmgr|INFO|Dropped 105195 log messages in last 12 seconds (most recently, 0 seconds ago) due to excessive rate
2016-10-08T13:47:03Z|01595|connmgr|INFO|e20<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: dropping packet-in due to queue overflow
2016-10-08T13:47:05Z|01596|connmgr|INFO|Dropped 1301 log messages in last 6 seconds (most recently, 1 seconds ago) due to excessive rate
2016-10-08T13:47:05Z|01597|connmgr|INFO|e20<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: sending OFPMMFC_INVALID_METER error reply to OFPT_FLOW_MOD message
2016-10-08T13:47:11Z|01598|connmgr|INFO|e00<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: 13318 flow_mods in the 33 s starting 34 s ago (13316 adds, 2 deletes)
2016-10-08T13:47:11Z|01599|connmgr|INFO|a20<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: 13334 flow_mods in the last 32 s (13330 adds, 4 deletes)
2016-10-08T13:47:11Z|01600|connmgr|INFO|a00<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: 13334 flow_mods in the last 32 s (13330 adds, 4 deletes)
2016-10-08T13:47:11Z|01601|connmgr|INFO|c01<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: 13335 flow_mods in the last 59 s (13330 adds, 5 deletes)
2016-10-08T13:47:11Z|01602|connmgr|INFO|e21<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: 2 flow_mods in the 25 s starting 59 s ago (2 deletes)
2016-10-08T13:47:11Z|01603|connmgr|INFO|Dropped 6345 log messages in last 6 seconds (most recently, 1 seconds ago) due to excessive rate
2016-10-08T13:47:11Z|01604|connmgr|INFO|e00<->tcp:10.10.10.200:6653<http://10.10.10.200:6653>: sending OFPMMFC_BAD_BURST error reply to OFPT_METER_MOD message
*** Error in `/usr/local/sbin/ovs-vswitchd': double free or corruption (out): 0x00000000216a8f90 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x77725)[0x7ffff6f48725]
/lib/x86_64-linux-gnu/libc.so.6(+0x7ff4a)[0x7ffff6f50f4a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7ffff6f54abc]
/usr/local/sbin/ovs-vswitchd[0x59f1c6]
/usr/local/sbin/ovs-vswitchd[0x5a3783]
/usr/local/sbin/ovs-vswitchd[0x5ca5e3]
/usr/local/sbin/ovs-vswitchd[0x59ce26]
/usr/local/sbin/ovs-vswitchd[0x58c0f4]
/usr/local/sbin/ovs-vswitchd[0x5912f9]
/usr/local/sbin/ovs-vswitchd[0x40da7d]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ffff6ef1830]
/usr/local/sbin/ovs-vswitchd[0x40eab9]
======= Memory map: ========
00400000-007cc000 r-xp 00000000 08:02 14813834                           /usr/local/sbin/ovs-vswitchd
009cc000-009cd000 r--p 003cc000 08:02 14813834                           /usr/local/sbin/ovs-vswitchd
009cd000-00a07000 rw-p 003cd000 08:02 14813834                           /usr/local/sbin/ovs-vswitchd
00a07000-21a19000 rw-p 00000000 00:00 0                                  [heap]
7fff5c000000-7fff5c021000 rw-p 00000000 00:00 0
7fff5c021000-7fff60000000 ---p 00000000 00:00 0
7fff64000000-7fff6c000000 rw-s 00000000 00:34 17314572                   /mnt/huge_2mb/qemu_back_mem._objects_mem.xGh4Vu (deleted)
7fff6c000000-7fffa9a00000 rw-s 00000000 00:26 16715628                   /dev/hugepages/rtemap_1
7fffab000000-7fffb3000000 rw-s 00000000 00:34 17305550                   /mnt/huge_2mb/qemu_back_mem._objects_mem.hVwHyv (deleted)
7fffb3000000-7fffbb000000 rw-s 00000000 00:34 17279527                   /mnt/huge_2mb/qemu_back_mem._objects_mem.cUDgVs (deleted)
7fffbb000000-7fffc3000000 rw-s 00000000 00:34 17265061                   /mnt/huge_2mb/qemu_back_mem._objects_mem.Bp5DMd (deleted)
7fffc3000000-7fffcb000000 rw-s 00000000 00:34 17240200                   /mnt/huge_2mb/qemu_back_mem._objects_mem.1WndGC (deleted)
7fffcb000000-7fffd3000000 rw-s 00000000 00:34 17198478                   /mnt/huge_2mb/qemu_back_mem._objects_mem.9Hfncw (deleted)
7fffd3000000-7fffd5600000 rw-s 00000000 00:26 16715627                   /dev/hugepages/rtemap_0
7fffd8000000-7fffd8022000 rw-p 00000000 00:00 0
7fffd8022000-7fffdc000000 ---p 00000000 00:00 0
7fffdc000000-7fffdc463000 rw-p 00000000 00:00 0
7fffdc463000-7fffe0000000 ---p 00000000 00:00 0
7fffe0000000-7fffe0021000 rw-p 00000000 00:00 0
7fffe0021000-7fffe4000000 ---p 00000000 00:00 0
7fffe63ea000-7fffe6400000 r-xp 00000000 08:02 5509547                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fffe6400000-7fffe65ff000 ---p 00016000 08:02 5509547                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fffe65ff000-7fffe6600000 rw-p 00015000 08:02 5509547                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fffe6600000-7fffe6800000 rw-s 00000000 00:34 17314572                   /mnt/huge_2mb/qemu_back_mem._objects_mem.xGh4Vu (deleted)
7fffe6800000-7fffe6a00000 rw-s 00000000 00:34 17305550                   /mnt/huge_2mb/qemu_back_mem._objects_mem.hVwHyv (deleted)
7fffe6a00000-7fffe6c00000 rw-s 00000000 00:34 17279527                   /mnt/huge_2mb/qemu_back_mem._objects_mem.cUDgVs (deleted)
7fffe6c00000-7fffe6e00000 rw-s 00000000 00:34 17265061                   /mnt/huge_2mb/qemu_back_mem._objects_mem.Bp5DMd (deleted)
7fffe6ffe000-7fffe6fff000 ---p 00000000 00:00 0
7fffe6fff000-7fffe77ff000 rw-p 00000000 00:00 0
7fffe77ff000-7fffe7800000 ---p 00000000 00:00 0
7fffe7800000-7fffe8000000 rw-p 00000000 00:00 0
7fffe8000000-7fffe8021000 rw-p 00000000 00:00 0
7fffe8021000-7fffec000000 ---p 00000000 00:00 0
7fffec000000-7fffec021000 rw-p 00000000 00:00 0
7fffec021000-7ffff0000000 ---p 00000000 00:00 0
7ffff0000000-7ffff0021000 rw-p 00000000 00:00 0
7ffff0021000-7ffff4000000 ---p 00000000 00:00 0
7ffff4200000-7ffff4400000 rw-s 00000000 00:34 17240200                   /mnt/huge_2mb/qemu_back_mem._objects_mem.1WndGC (deleted)
7ffff4400000-7ffff4600000 rw-s 00000000 00:34 17198478                   /mnt/huge_2mb/qemu_back_mem._objects_mem.9Hfncw (deleted)
7ffff46cc000-7ffff46cd000 ---p 00000000 00:00 0
7ffff46cd000-7ffff4ecd000 rw-p 00000000 00:00 0
7ffff4ecd000-7ffff4ece000 ---p 00000000 00:00 0
7ffff4ece000-7ffff56ce000 rw-p 00000000 00:00 0
7ffff56ce000-7ffff56cf000 ---p 00000000 00:00 0
7ffff56cf000-7ffff5ecf000 rw-p 00000000 00:00 0
7ffff5ecf000-7ffff5ed0000 ---p 00000000 00:00 0
7ffff5ed0000-7ffff66d0000 rw-p 00000000 00:00 0
7ffff66d0000-7ffff66d1000 ---p 00000000 00:00 0
7ffff66d1000-7ffff6ed1000 rw-p 00000000 00:00 0
7ffff6ed1000-7ffff7091000 r-xp 00000000 08:02 5509509                    /lib/x86_64-linux-gnu/libc-2.23.so<http://libc-2.23.so>
7ffff7091000-7ffff7290000 ---p 001c0000 08:02 5509509                    /lib/x86_64-linux-gnu/libc-2.23.so<http://libc-2.23.so>
7ffff7290000-7ffff7294000 r--p 001bf000 08:02 5509509                    /lib/x86_64-linux-gnu/libc-2.23.so<http://libc-2.23.so>
7ffff7294000-7ffff7296000 rw-p 001c3000 08:02 5509509                    /lib/x86_64-linux-gnu/libc-2.23.so<http://libc-2.23.so>
7ffff7296000-7ffff729a000 rw-p 00000000 00:00 0
7ffff729a000-7ffff72a4000 r-xp 00000000 08:02 14688949                   /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7ffff72a4000-7ffff74a3000 ---p 0000a000 08:02 14688949                   /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7ffff74a3000-7ffff74a4000 r--p 00009000 08:02 14688949                   /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7ffff74a4000-7ffff74a5000 rw-p 0000a000 08:02 14688949                   /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7ffff74a5000-7ffff75ad000 r-xp 00000000 08:02 5509579                    /lib/x86_64-linux-gnu/libm-2.23.so<http://libm-2.23.so>
7ffff75ad000-7ffff77ac000 ---p 00108000 08:02 5509579                    /lib/x86_64-linux-gnu/libm-2.23.so<http://libm-2.23.so>
7ffff77ac000-7ffff77ad000 r--p 00107000 08:02 5509579                    /lib/x86_64-linux-gnu/libm-2.23.so<http://libm-2.23.so>
7ffff77ad000-7ffff77ae000 rw-p 00108000 08:02 5509579                    /lib/x86_64-linux-gnu/libm-2.23.so<http://libm-2.23.so>
7ffff77ae000-7ffff77b5000 r-xp 00000000 08:02 5509663                    /lib/x86_64-linux-gnu/librt-2.23.so<http://librt-2.23.so>
7ffff77b5000-7ffff79b4000 ---p 00007000 08:02 5509663                    /lib/x86_64-linux-gnu/librt-2.23.so<http://librt-2.23.so>
7ffff79b4000-7ffff79b5000 r--p 00006000 08:02 5509663                    /lib/x86_64-linux-gnu/librt-2.23.so<http://librt-2.23.so>
7ffff79b5000-7ffff79b6000 rw-p 00007000 08:02 5509663                    /lib/x86_64-linux-gnu/librt-2.23.so<http://librt-2.23.so>
7ffff79b6000-7ffff79ce000 r-xp 00000000 08:02 5509655                    /lib/x86_64-linux-gnu/libpthread-2.23.so<http://libpthread-2.23.so>
7ffff79ce000-7ffff7bcd000 ---p 00018000 08:02 5509655                    /lib/x86_64-linux-gnu/libpthread-2.23.so<http://libpthread-2.23.so>
7ffff7bcd000-7ffff7bce000 r--p 00017000 08:02 5509655                    /lib/x86_64-linux-gnu/libpthread-2.23.so<http://libpthread-2.23.so>
7ffff7bce000-7ffff7bcf000 rw-p 00018000 08:02 5509655                    /lib/x86_64-linux-gnu/libpthread-2.23.so<http://libpthread-2.23.so>
7ffff7bcf000-7ffff7bd3000 rw-p 00000000 00:00 0
7ffff7bd3000-7ffff7bd6000 r-xp 00000000 08:02 5509533                    /lib/x86_64-linux-gnu/libdl-2.23.so<http://libdl-2.23.so>
7ffff7bd6000-7ffff7dd5000 ---p 00003000 08:02 5509533                    /lib/x86_64-linux-gnu/libdl-2.23.so<http://libdl-2.23.so>
7ffff7dd5000-7ffff7dd6000 r--p 00002000 08:02 5509533                    /lib/x86_64-linux-gnu/libdl-2.23.so<http://libdl-2.23.so>
7ffff7dd6000-7ffff7dd7000 rw-p 00003000 08:02 5509533                    /lib/x86_64-linux-gnu/libdl-2.23.so<http://libdl-2.23.so>
7ffff7dd7000-7ffff7dfd000 r-xp 00000000 08:02 5509481                    /lib/x86_64-linux-gnu/ld-2.23.so<http://ld-2.23.so>
7ffff7f63000-7ffff7f9e000 rw-s 00000000 00:13 703                        /run/.rte_hugepage_info
7ffff7f9e000-7ffff7fd7000 rw-s 00000000 00:13 702                        /run/.rte_config
7ffff7fd7000-7ffff7fdc000 rw-p 00000000 00:00 0
7ffff7ff5000-7ffff7ff8000 rw-p 00000000 00:00 0
7ffff7ff8000-7ffff7ffa000 r--p 00000000 00:00 0                          [vvar]
7ffff7ffa000-7ffff7ffc000 r-xp 00000000 00:00 0                          [vdso]
7ffff7ffc000-7ffff7ffd000 r--p 00025000 08:02 5509481                    /lib/x86_64-linux-gnu/ld-2.23.so<http://ld-2.23.so>
7ffff7ffd000-7ffff7ffe000 rw-p 00026000 08:02 5509481                    /lib/x86_64-linux-gnu/ld-2.23.so<http://ld-2.23.so>
7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0
7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Thread 1 "ovs-vswitchd" received signal SIGABRT, Aborted.
0x00007ffff6f06418 in __GI_raise (sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

(gdb) i threads
  Id   Target Id         Frame
* 1    Thread 0x7ffff7fd7b00 (LWP 7259) "ovs-vswitchd" 0x00007ffff6f06418 in __GI_raise (sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
  2    Thread 0x7ffff6ed0700 (LWP 7263) "eal-intr-thread" 0x00007ffff6fd8153 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
  3    Thread 0x7ffff66cf700 (LWP 7264) "dpdk_watchdog1" 0x00007ffff6f9c8dd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
  4    Thread 0x7ffff5ece700 (LWP 7265) "vhost_thread3" 0x00007ffff6fcdd13 in select () at ../sysdeps/unix/syscall-template.S:84
  5    Thread 0x7ffff56cd700 (LWP 7266) "urcu2" 0x00007ffff6fcbe8d in poll () at ../sysdeps/unix/syscall-template.S:84
  36   Thread 0x7fffe77fe700 (LWP 7460) "pmd34" 0x000000000048af27 in rte_vhost_dequeue_burst ()
  127  Thread 0x7ffff4ecc700 (LWP 8261) "handler125" 0x00007ffff6fcbe8d in poll () at ../sysdeps/unix/syscall-template.S:84
  128  Thread 0x7fffe7fff700 (LWP 8262) "revalidator126" 0x00007ffff6fcbe8d in poll () at ../sysdeps/unix/syscall-template.S:84
(gdb) where
#0  0x00007ffff6f06418 in __GI_raise (sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007ffff6f0801a in __GI_abort () at abort.c:89
#2  0x00007ffff6f4872a in __libc_message (do_abort=do_abort at entry=2, fmt=fmt at entry=0x7ffff70616b0 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007ffff6f50f4a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>, str=0x7ffff70617c0 "double free or corruption (out)", action=3) at malloc.c:5007
#4  _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3868
#5  0x00007ffff6f54abc in __GI___libc_free (mem=<optimized out>) at malloc.c:2969
#6  0x000000000059f1c6 in dp_packet_delete (b=0x211fc2a0) at ./lib/dp-packet.h:165
#7  handle_packet_out (ofconn=ofconn at entry=0x2178f140, oh=oh at entry=0x219a5db0) at ofproto/ofproto.c:3496
#8  0x00000000005a3783 in handle_openflow__ (msg=0x21291040, ofconn=0x2178f140) at ofproto/ofproto.c:7344
#9  handle_openflow (ofconn=0x2178f140, ofp_msg=0x21291040) at ofproto/ofproto.c:7516
#10 0x00000000005ca5e3 in ofconn_run (handle_openflow=0x5a2f70 <handle_openflow>, ofconn=0x2178f140) at ofproto/connmgr.c:1378
#11 connmgr_run (mgr=0x2181cf50, handle_openflow=handle_openflow at entry=0x5a2f70 <handle_openflow>) at ofproto/connmgr.c:320
#12 0x000000000059ce26 in ofproto_run (p=0x21832330) at ofproto/ofproto.c:1820
#13 0x000000000058c0f4 in bridge_run__ () at vswitchd/bridge.c:2880
#14 0x00000000005912f9 in bridge_run () at vswitchd/bridge.c:2937
#15 0x000000000040da7d in main (argc=3, argv=0x7fffffffe518) at vswitchd/ovs-vswitchd.c:112
_______________________________________________
discuss mailing list
discuss at openvswitch.org<mailto:discuss at openvswitch.org>
http://openvswitch.org/mailman/listinfo/discuss




More information about the discuss mailing list