[ovs-discuss] adding MPTCP code breaks OVS kmod datapath creation

Brandon Heller brandonh at stanford.edu
Fri Jan 20 09:50:19 UTC 2012


Hi,

I'm trying to run a single system that has OVS as well as the draft Linux
implementation of Multi-Path TCP.  Briefly, MPTCP uses multiple subflows to
take advantage of multiple interfaces (such as WiFi + WiFi + WiMax) or
multiple paths (within a Fat Tree in an ECMP-based data center).  The
project is fairly young, and more details are available at
http://mptcp.info.ucl.ac.be/.  I did not write any of this code.

On an Ubuntu 3.0.0-15-generic kernel on an Ubuntu 11.10 system w/OVS
{1.2.2/1.3.0}, I have no issues w/OVS.  When using 3.0.0-15-mptcp, which
has about 11K lines of MPTCP-specific changes, plus a few small config
changes, I cannot instantiate a datapath:

brandonh at nfcm7:~$ sudo ovs-dpctl add-dp dp0
ovs-dpctl: add_dp (Message too long)

Verbose console output is attached, along with dmesg showing the call
traces that get triggered. The netlink messages are fairly descriptive:

{snip - see attached}
Jan 19 11:45:08|00007|netlink_socket|DBG|nl_sock_send__ (Success):
nl(len:36, type=16(control), flags=5[REQUEST][ACK], seq=4f186731,
pid=5610(5610:0)),genl(cmd=3,version=1)
Jan 19 11:45:08|00008|netlink_socket|DBG|nl_sock_recv__ (Success):
nl(len:504, type=16(control), flags=0, seq=4f186731,
pid=5610(5610:0)),genl(cmd=1,version=2)
Jan 19 11:45:08|00009|netlink_socket|DBG|nl_sock_send__ (Success):
nl(len:32, type=27(odp_datapath), flags=d[REQUEST][ACK][ECHO],
seq=4f186732, pid=5610(5610:0)),genl(cmd=1,version=1)
Jan 19 11:45:08|00010|netlink_socket|DBG|nl_sock_recv__ (Success):
nl(len:52, type=2(error), flags=0, seq=4f186732, pid=5610(5610:0))
error(-90(Message too long), in-reply-to(nl(len:32, type=27(odp_datapath),
flags=d[REQUEST][ACK][ECHO], seq=4f186732, pid=5610(5610:0))))
Jan 19 11:45:08|00011|netlink_socket|DBG|received NAK error=90 (Message too
long)
ovs-dpctl: add_dp (Message too long)

I don't understand the call traces; a bunch of these are printed:

{snip - see attached}
[ 2867.679401] WARNING: at
/home/brandonh/openvswitch.orig/datapath/linux/datapath.c:220
dp_ifinfo_notify+0x287/0x2b0 [openvswitch_mod]()
[ 2867.679404] Hardware name: PDSBM
[ 2867.679405] Modules linked in: openvswitch_mod i915 psmouse serio_raw
drm_kms_helper drm i2c_algo_bit video lp parport floppy e1000e [last
unloaded: openvswitch_mod]
[ 2867.679419] Pid: 10808, comm: ovs-dpctl Tainted: G        W
3.0.0-15-mptcp #31
[ 2867.679422] Call Trace:
[ 2867.679427]  [<ffffffff8105e83f>] warn_slowpath_common+0x7f/0xc0
[ 2867.679431]  [<ffffffff8105e89a>] warn_slowpath_null+0x1a/0x20
[ 2867.679436]  [<ffffffffa014b127>] dp_ifinfo_notify+0x287/0x2b0
[openvswitch_mod]
[ 2867.679442]  [<ffffffffa014c9a6>] dp_detach_port+0x36/0xb0
[openvswitch_mod]
[ 2867.679447]  [<ffffffffa014cf13>] odp_dp_cmd_new+0x2a3/0x2d0
[openvswitch_mod]
[ 2867.679452]  [<ffffffff815060a5>] genl_rcv_msg+0x1d5/0x250
[ 2867.679456]  [<ffffffff81505ed0>] ? genl_rcv+0x40/0x40
[ 2867.679459]  [<ffffffff81505969>] netlink_rcv_skb+0xb9/0xe0
[ 2867.679463]  [<ffffffff81505eb5>] genl_rcv+0x25/0x40
[ 2867.679466]  [<ffffffff81505253>] netlink_unicast+0x2b3/0x300
[ 2867.679470]  [<ffffffff814d0e77>] ? memcpy_fromiovec+0x67/0xb0
[ 2867.679474]  [<ffffffff8150556a>] netlink_sendmsg+0x2ca/0x360
[ 2867.679478]  [<ffffffff814c266e>] sock_sendmsg+0x10e/0x130
[ 2867.679482]  [<ffffffff8115503c>] ? __kmalloc+0x12c/0x190
[ 2867.679485]  [<ffffffff814c9f4b>] ? sk_prot_alloc+0xfb/0x210
[ 2867.679490]  [<ffffffff81285bb6>] ? security_sk_alloc+0x16/0x20
[ 2867.679493]  [<ffffffff810465a3>] ? __wake_up+0x53/0x70
[ 2867.679497]  [<ffffffff814c5ebd>] sys_sendto+0x13d/0x190
[ 2867.679501]  [<ffffffff814c5b3b>] ? sys_connect+0xeb/0x110
[ 2867.679505]  [<ffffffff81166d61>] ? fd_install+0x61/0x80
[ 2867.679508]  [<ffffffff814c294a>] ? sock_map_fd+0x2a/0x40
[ 2867.679512]  [<ffffffff81600202>] system_call_fastpath+0x16/0x1b
[ 2867.679515] ---[ end trace e9610238c9c5115a ]---
[ 2867.679519] device dp0 left promiscuous mode


I binary-searched the source, and the addition of MPTCP-specific code
enabled by CONFIG_MPTCP/CONFIG_MPTCP_DEBUG, and not the .config
differences, causes the error.  The error is the same with OVS 1.2.1,
1.2.2, and 1.3.0.  I have some code that's dependent on openflowd, so
hopefully can stick w/1.2.2.

Replicating this on Ubuntu 11.10 is pretty easy: install the mptcp
kernel+headers via apt (
http://mptcp.info.ucl.ac.be/pmwiki.php?n=Users.HowToInstallMPTCP), reboot,
build the kmod, insmod, and run the command above.

Let me know what else to do to debug this, or any other output to send.

Thanks!
Brandon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20120120/e2eec826/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mptcp_ovs_dmesg
Type: application/octet-stream
Size: 64395 bytes
Desc: not available
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20120120/e2eec826/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: console_error_output
Type: application/octet-stream
Size: 1971 bytes
Desc: not available
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20120120/e2eec826/attachment-0005.obj>


More information about the discuss mailing list