[ovs-discuss] intermitting ARP problems on DP interface

[Andreas Schultz]

Hi all,

Upon further investigation it turns out the the dpif_linux_open() sequence
is broken somewhere. The flow of netlink messages simply makes no sence at
all.

first we get an create attempt, which is expected to fail since the DP already exists:
Oct 19 11:07:52|00014|netlink_socket|DBG|nl_sock_recv__ (Success): nl(len:60, type=2(error), flags=0, seq=4e9ef0dc, pid=25535(25535:0)) error(-17(File exists), in-reply-to(nl(len:40, type=33(ovs_datapath), flags=d[REQUEST][ACK][ECHO], seq=4e9ef0dc, pid=25535(25535:0))))
Oct 19 11:07:52|00015|netlink_socket|DBG|received NAK error=17 (File exists)

now it tries OVS_DP_CMD_GET:
Oct 19 11:07:52|00016|netlink_socket|DBG|nl_sock_transact_multiple__ (Success): nl(len:32, type=33(ovs_datapath), flags=d[REQUEST][ACK][ECHO], seq=4e9ef0dd, pid=25535(25535:0)),genl(cmd=3,version=1)
Oct 19 11:07:52|00017|netlink_socket|DBG|nl_sock_recv__ (Success): nl(len:84, type=33(ovs_datapath), flags=0, seq=4e9ef0dd, pid=25535(25535:0)),genl(cmd=1,version=1)

and succeeds. So far so good...

Next step is to flush any old flows:
Oct 19 11:07:52|00018|netlink_socket|DBG|nl_sock_transact_multiple__ (Success): nl(len:24, type=35(ovs_flow), flags=5[REQUEST][ACK], seq=4e9ef0de, pid=25535(25535:0)),genl(cmd=2,version=1)

send that to the kernel...

and the kernel give us a netlink error report for the OVS_DP_CMD_GET that was already ACKed OK:
Oct 19 11:07:52|00019|netlink_socket|DBG|nl_sock_recv__ (Success): nl(len:36, type=2(error), flags=0, seq=4e9ef0dd, pid=25535(25535:0)) error(0, in-reply-to(nl(len:32, type=33(ovs_datapath), flags=d[REQUEST][ACK][ECHO], seq=4e9ef0dd, pid=25535(25535:0))))
Oct 19 11:07:52|00020|netlink_socket|DBG|ignoring unexpected seq 0x4e9ef0dd

I have verified above sequence with strace and the decoded netlink messages where indeed send to and received from the kernel. So it is not a buffering issue in the controller.
The problem occurs at least with openvswitch version 1.2.2 and git-HEAD on kernel 2.6.32 and 2.6.38.

Regards
Andreas

----- Original Message -----
> Hi,
> 
> We have encountered a strange behavior. After restarting the
> controller process and even after removing and reinserting the
> datapath module, ARP packets are forwarded from dp0 to the real port
> intermittently.
> tcpdump confirms that the ARP request is seen on dp0, but not on the
> physical port (eth1).
> At the same time the following sequence on log messages appear:
> 
> Oct 18 12:22:54|00344|netlink_socket|DBG|nl_sock_recv__ (Success):
> nl(len:24, type=30(ovs_packet), flags=0, seq=0,
> pid=0(0:0)),genl(cmd=1,version=1)
> Oct 18 12:22:54|00345|dpif|DBG|system at dp0: miss upcall:
> in_port(0),eth(src=00:23:20:fc:70:43,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=172.28.0.1,tip=172.28.0.19,op=1,sha=00:23:20:fc:70:43,tha=00:00:00:00:00:00)
> 00:23:20:fc:70:43 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length
> 42: Request who-has 172.28.0.19 tell 172.28.0.1, length 28
> Oct 18 12:22:54|00346|ofproto_dpif|WARN|bridge dp0: received packet
> on unknown port 65534
> Oct 18 12:22:54|00347|netlink_socket|DBG|nl_sock_transact_multiple__
> (Success): nl(len:140, type=30(ovs_packet), flags=1[REQUEST],
> seq=4e9d5835, pid=75500397(2925:18)),genl(cmd=3,version=1)
> Oct 18 12:22:54|00348|netlink_socket|DBG|nl_sock_transact_multiple__
> (Success): nl(len:92, type=29(ovs_flow), flags=5[REQUEST][ACK],
> seq=4e9d5836, pid=75500397(2925:18)),genl(cmd=1,version=1)
> Oct 18 12:22:54|00349|netlink_socket|DBG|nl_sock_recv__ (Success):
> nl(len:36, type=2(error), flags=0, seq=4e9d5836,
> pid=75500397(2925:18)) error(0, in-reply-to(nl(len:92,
> type=29(ovs_flow), flags=5[REQUEST][ACK], seq=4e9d5836,
> pid=75500397(2925:18))))
> 
> There clearly is something strange going on. How can the dp receive
> something on an unknown port?
> 
> Any hints?
> 
> Regards
> Andreas
> --
> --
> Dipl. Inform.
> Andreas Schultz
> 
> 
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> http://openvswitch.org/mailman/listinfo/discuss
> 

-- 
-- 
Dipl. Inform.
Andreas Schultz