[ovs-discuss] OpenVSwitch 1.7.0, hard system crashes

Polehn, MikeX A mikex.a.polehn at intel.com
Wed Sep 19 16:20:57 UTC 2012


During some bidirectional TCP data tests, with low power (1.4 Ghz) 8 core sandy bridge processors, open vswitch consistently crashed the system. Same setup of tests on the Linux bridge on same SUT were done with no issues.

Attached is a detailed MS-Word test report (Crystal_Forest OVS_8_Part_VMs_Netperf_V0.2_9_17_12.docx) with most with setup details. At the end is the test log which shows the sequence to setup  and recovery after crashes.

Also restarting the open-vswitch without reinitializing conf.db didn't work (all the VMs were off since the system had crashed) .
Attached is several conf.db  files (conf.db.old, conf.db2.old, conf.db3.old, conf.db4.old)  after crashes (see log in report )

Also attached is a log of doing the test set on 9/14/12  (OVS_mult_VM_net_test_9_14_12.txt), which crashed during a test, since I was busy with other things restart tests on 0/17/12.

General info requested (not in report) :

[root at cfCOS ~]# cat /proc/version
Linux version 3.2.2_mp (root at CF1cos.crystal) (gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC) ) #2 SMP Wed Feb 15 07:56:15 PST 2012

[root at cfCOS openvswitch-1.7.0]# ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 1.7.0
Compiled Sep  5 2012 15:39:36
OpenFlow versions 0x1:0x1


System log for restarting openvswitch without resetting conf.db (there was no system crash info in message log). The buffer errors at the end might be of interest, since this may be a different error.  I think this was the same as first attempt to restart openvswitch after the crash, but I was looking for kernel crash data and only looked  at this info later on.


Sep 17 08:13:17 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br0 tap_A0
Sep 17 08:13:17 cfCOS kernel: device tap_A0 entered promiscuous mode
Sep 17 08:13:17 cfCOS ovs-vswitchd: 00016|bridge|INFO|bridge br0: added interface tap_A0 on port 2
Sep 17 08:13:17 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br1 tap_A1
Sep 17 08:13:17 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_A1: couldn't determine device driver; ignoring...
Sep 17 08:13:17 cfCOS ovs-vswitchd: 00017|bridge|INFO|bridge br1: added interface tap_A1 on port 2
Sep 17 08:13:17 cfCOS kernel: device tap_A1 entered promiscuous mode
Sep 17 08:13:17 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_A0: couldn't determine device driver; ignoring...
Sep 17 08:13:18 cfCOS avahi-daemon[2567]: Registering new address record for fe80::38f4:84ff:fe3f:887b on tap_A1.*.
Sep 17 08:13:19 cfCOS avahi-daemon[2567]: Registering new address record for fe80::9f:f5ff:fe7e:eb6c on tap_A0.*.
Sep 17 08:15:03 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br0 tap_B0
Sep 17 08:15:03 cfCOS ovs-vswitchd: 00018|bridge|INFO|bridge br0: added interface tap_B0 on port 3
Sep 17 08:15:03 cfCOS kernel: device tap_B0 entered promiscuous mode
Sep 17 08:15:03 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_B0: couldn't determine device driver; ignoring...
Sep 17 08:15:03 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br1 tap_B1
Sep 17 08:15:03 cfCOS ovs-vswitchd: 00019|bridge|INFO|bridge br1: added interface tap_B1 on port 3
Sep 17 08:15:03 cfCOS kernel: device tap_B1 entered promiscuous mode
Sep 17 08:15:04 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_B1: couldn't determine device driver; ignoring...
Sep 17 08:15:05 cfCOS avahi-daemon[2567]: Registering new address record for fe80::5cfc:65ff:fef7:67ab on tap_B1.*.
Sep 17 08:15:05 cfCOS avahi-daemon[2567]: Registering new address record for fe80::6404:79ff:feb4:b2a1 on tap_B0.*.
Sep 17 08:17:20 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br0 tap_C0
Sep 17 08:17:20 cfCOS ovs-vswitchd: 00020|bridge|INFO|bridge br0: added interface tap_C0 on port 4
Sep 17 08:17:20 cfCOS kernel: device tap_C0 entered promiscuous mode
Sep 17 08:17:20 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_C0: couldn't determine device driver; ignoring...
Sep 17 08:17:20 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br1 tap_C1
Sep 17 08:17:20 cfCOS ovs-vswitchd: 00021|bridge|INFO|bridge br1: added interface tap_C1 on port 4
Sep 17 08:17:20 cfCOS kernel: device tap_C1 entered promiscuous mode
Sep 17 08:17:20 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_C1: couldn't determine device driver; ignoring...
Sep 17 08:17:21 cfCOS avahi-daemon[2567]: Registering new address record for fe80::f024:8dff:fec1:8e8f on tap_C0.*.
Sep 17 08:17:21 cfCOS avahi-daemon[2567]: Registering new address record for fe80::1cd1:fcff:fea9:b9f6 on tap_C1.*.
Sep 17 08:19:00 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br0 tap_D0
Sep 17 08:19:00 cfCOS ovs-vswitchd: 00022|bridge|INFO|bridge br0: added interface tap_D0 on port 5
Sep 17 08:19:00 cfCOS kernel: device tap_D0 entered promiscuous mode
Sep 17 08:19:00 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_D0: couldn't determine device driver; ignoring...
Sep 17 08:19:00 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br1 tap_D1
Sep 17 08:19:00 cfCOS ovs-vswitchd: 00023|bridge|INFO|bridge br1: added interface tap_D1 on port 5
Sep 17 08:19:00 cfCOS kernel: device tap_D1 entered promiscuous mode
Sep 17 08:19:00 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_D1: couldn't determine device driver; ignoring...
Sep 17 08:19:01 cfCOS avahi-daemon[2567]: Registering new address record for fe80::14d2:97ff:fe98:8fde on tap_D0.*.
Sep 17 08:19:02 cfCOS avahi-daemon[2567]: Registering new address record for fe80::fd:3dff:fe41:bfa4 on tap_D1.*.
Sep 17 08:20:30 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br0 tap_E0
Sep 17 08:20:30 cfCOS ovs-vswitchd: 00024|bridge|INFO|bridge br0: added interface tap_E0 on port 6
Sep 17 08:20:30 cfCOS kernel: device tap_E0 entered promiscuous mode

Sep 17 08:20:30 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_E0: couldn't determine device driver; ignoring...
Sep 17 08:20:30 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br1 tap_E1
Sep 17 08:20:30 cfCOS ovs-vswitchd: 00025|bridge|INFO|bridge br1: added interface tap_E1 on port 6
Sep 17 08:20:30 cfCOS kernel: device tap_E1 entered promiscuous mode
Sep 17 08:20:30 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_E1: couldn't determine device driver; ignoring...
Sep 17 08:20:31 cfCOS avahi-daemon[2567]: Registering new address record for fe80::90db:1eff:fe06:8ff4 on tap_E0.*.
Sep 17 08:20:32 cfCOS avahi-daemon[2567]: Registering new address record for fe80::80cc:97ff:fe89:83ca on tap_E1.*.
Sep 17 08:22:15 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br0 tap_G0
Sep 17 08:22:15 cfCOS ovs-vswitchd: 00026|bridge|INFO|bridge br0: added interface tap_G0 on port 7
Sep 17 08:22:15 cfCOS kernel: device tap_G0 entered promiscuous mode
Sep 17 08:22:15 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_G0: couldn't determine device driver; ignoring...
Sep 17 08:22:15 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br1 tap_G1
Sep 17 08:22:15 cfCOS ovs-vswitchd: 00027|bridge|INFO|bridge br1: added interface tap_G1 on port 7
Sep 17 08:22:15 cfCOS kernel: device tap_G1 entered promiscuous mode
Sep 17 08:22:15 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_G1: couldn't determine device driver; ignoring...
Sep 17 08:22:16 cfCOS avahi-daemon[2567]: Registering new address record for fe80::ceb:2bff:fef4:b784 on tap_G0.*.
Sep 17 08:22:16 cfCOS avahi-daemon[2567]: Registering new address record for fe80::d0a7:7fff:feeb:f59d on tap_G1.*.
Sep 17 08:24:32 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br0 tap_F0
Sep 17 08:24:32 cfCOS ovs-vswitchd: 00028|bridge|INFO|bridge br0: added interface tap_F0 on port 8
Sep 17 08:24:32 cfCOS kernel: device tap_F0 entered promiscuous mode
Sep 17 08:24:32 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_F0: couldn't determine device driver; ignoring...
Sep 17 08:24:32 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br1 tap_F1
Sep 17 08:24:32 cfCOS kernel: device tap_F1 entered promiscuous mode
Sep 17 08:24:32 cfCOS ovs-vswitchd: 00029|bridge|INFO|bridge br1: added interface tap_F1 on port 8
Sep 17 08:24:32 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_F1: couldn't determine device driver; ignoring...
Sep 17 08:24:33 cfCOS avahi-daemon[2567]: Registering new address record for fe80::60b2:93ff:fe02:cbca on tap_F0.*.
Sep 17 08:24:33 cfCOS avahi-daemon[2567]: Registering new address record for fe80::d86c:13ff:febe:f8e3 on tap_F1.*.
Sep 17 08:25:58 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br0 tap_H0
Sep 17 08:25:58 cfCOS ovs-vswitchd: 00030|bridge|INFO|bridge br0: added interface tap_H0 on port 9
Sep 17 08:25:58 cfCOS kernel: device tap_H0 entered promiscuous mode
Sep 17 08:25:58 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_H0: couldn't determine device driver; ignoring...
Sep 17 08:25:58 cfCOS ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port br1 tap_H1
Sep 17 08:25:58 cfCOS ovs-vswitchd: 00031|bridge|INFO|bridge br1: added interface tap_H1 on port 9
Sep 17 08:25:58 cfCOS kernel: device tap_H1 entered promiscuous mode
Sep 17 08:25:58 cfCOS NetworkManager[2555]: <warn> /sys/devices/virtual/net/tap_H1: couldn't determine device driver; ignoring...
Sep 17 08:25:59 cfCOS avahi-daemon[2567]: Registering new address record for fe80::5860:24ff:feb9:a19b on tap_H1.*.
Sep 17 08:26:00 cfCOS avahi-daemon[2567]: Registering new address record for fe80::3425:1fff:fead:810d on tap_H0.*.
Sep 17 08:34:05 cfCOS ovs-vswitchd: 00032|dpif|WARN|system at br0: recv failed (No buffer space available)
Sep 17 08:34:29 cfCOS ovs-vswitchd: 00033|dpif|WARN|system at br0: recv failed (No buffer space available)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20120919/cf00d692/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Crystal_Forest OVS_8_Part_VMs_Netperf_V0.2_9_17_12.docx
Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document
Size: 143017 bytes
Desc: Crystal_Forest OVS_8_Part_VMs_Netperf_V0.2_9_17_12.docx
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20120919/cf00d692/attachment.docx>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: OVS_mult_VM_net_test_9_14_12.txt
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20120919/cf00d692/attachment-0002.txt>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: conf.db.old
Type: application/octet-stream
Size: 70711 bytes
Desc: conf.db.old
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20120919/cf00d692/attachment-0008.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: conf.db.old2
Type: application/octet-stream
Size: 27528 bytes
Desc: conf.db.old2
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20120919/cf00d692/attachment-0009.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: conf.db.old3
Type: application/octet-stream
Size: 27528 bytes
Desc: conf.db.old3
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20120919/cf00d692/attachment-0010.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: conf.db.old4
Type: application/octet-stream
Size: 27528 bytes
Desc: conf.db.old4
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20120919/cf00d692/attachment-0011.obj>


More information about the discuss mailing list