[ovs-discuss] not seeing return packets from IN_PORT

Wed Jan 11 18:29:50 UTC 2012

If TCP really isn't replying, then you might be able to find out why
by watching the counters that "netstat -s" prints.  Perhaps there is
something invalid about the packets from the TCP stack's point of
view, or something odd about routes from the IP stack's point of view,
or maybe the checksum adjustment code in the kernel module is borked
(not for the first time).

iptables could be swallowing the packet I guess.  "iptables -F" could
be helpful in that case.

On Tue, Jan 10, 2012 at 09:00:28PM -0500, Aaron Rosen wrote:
> Hey Ben,
> 
> Sorry for some reason this packet seem to be showing up now.. I'm not quite
> sure what the deal was before :/
> 
>  1f:29:32:92:4d (oui Unknown) > 00:1b:21:6a:85:88 (oui Unknown), ethertype
> IPv4 (0x0800), length 74: 10.0.0.10.36985 > 10.0.0.13.5004: S
>  00:1f:29:32:92:4d (oui Unknown) > 00:1f:29:32:92:4d (oui Unknown),
> ethertype IPv4 (0x0800), length 74: 10.0.0.10.36985 > 10.0.0.10.9877: S
> 
> 
> At this point though I'm curious why the the host 10.0.0.10 doesn't reply
> back to it's self with an syn-ack for the syn packet it receive?  (The
> SYN-ACK isn't coming across the loop back either) .
> 
> Thanks again for your help,
> 
> Aaron
> 
> pg46 ~ # netstat -an | grep 9877
> tcp        0      0 0.0.0.0:9877            0.0.0.0:*               LISTEN
> 
> 
> pg46 ~ # ifconfig dp0
> dp0       Link encap:Ethernet  HWaddr 00:1f:29:32:92:4d
>           inet addr:10.0.0.10  Bcast:10.0.0.255  Mask:255.255.255.0
> 
> 
> 
> 
> 
> On Tue, Jan 10, 2012 at 7:11 PM, Ben Pfaff <blp at nicira.com> wrote:
> 
> > As an experiment, you could start to add "printk"s to the kernel
> > module.  That's what I'd do next while debugging the problem.
> >
> > On Tue, Jan 10, 2012 at 05:10:38PM -0500, Aaron Rosen wrote:
> > > I'm not sure if this helps eliminate anything but I just tried this in
> > > mininet  and the same thing occurs there if dl_src/nw_src ==
> > > dl_dst/nw_dst  .
> > >
> > > Thanks,
> > >
> > > Aaron
> > >
> > >
> > > #packet that goes in
> > > 14:02:47.309736 06:3b:31:b8:d5:a2 (oui Unknown) > ca:05:25:be:fd:70
> > > (oui Unknown), ethertype IPv4 (0x0800), length 74: 10.0.0.10.39458 >
> > > 10.0.0.11.5004: Flags [S], seq 3794257069, win 5840, options [mss
> > > 1460,sackOK,TS val 47077922 ecr 0,nop,wscale 9], length 0
> > >
> > >
> > > #flow_mod
> > >   cookie=0, duration_sec=50s, duration_nsec=950000000s, table_id=1,
> > > priority=65535, n_packets=1, n_bytes=74,
> > >
> > idle_timeout=100,hard_timeout=0,tcp,in_port=2,dl_vlan=0xffff,dl_vlan_pcp=0x00,dl_src=06:3b:31:b8:d5:a2,dl_dst=ca:05:25:be:fd:70,nw_src=10.0.0.10,nw_dst=10.0.0.11,nw_tos=0x00,tp_src=39458,tp_dst=5004,actions=mod_dl_src:06:3b:31:b8:d5:a2,mod_dl_dst:06:3b:31:b8:d5:a2,mod_nw_src:10.0.0.10,mod_nw_dst:10.0.0.10,mod_tp_dst:9877,IN_PORT
> > >
> > >
> > > #packet is not returned.
> > >
> > >
> > > On Tue, Jan 10, 2012 at 3:02 PM, Aaron Rosen <arosen at clemson.edu> wrote:
> > > > Here are lines (154-158) from here http://codepad.org/APdmOTGN (more
> > debug)
> > > >
> > > > Thanks,
> > > >
> > > > Aaron
> > > >
> > > > Jan 10 14:46:07 localhost ovs-vswitchd: 04385|dpif|DBG|system at dp0:
> > miss
> > > > upcall:
> > > > Jan 10 14:46:07 localhost ovs-vswitchd:
> > > >
> > in_port(0),eth(src=00:1f:29:32:92:4d,dst=00:1b:21:6a:85:88),eth_type(0x0800),ipv4(src=10.0.0.10,dst=10.0.0.13,proto=6,tos=0,ttl=64,frag=no),tcp(src=38710,dst=5004)
> > > > Jan 10 14:46:07 localhost ovs-vswitchd: 00:1f:29:32:92:4d >
> > > > 00:1b:21:6a:85:88, ethertype IPv4 (0x0800), length 74: 10.0.0.10.38710
> > >
> > > > 10.0.0.13.5004: S 3410511224:3410511224(0) win 14600 <mss
> > > > 1460,sackOK,timestamp 9255759 0,nop,wscale 6>
> > > > Jan 10 14:46:07 localhost ovs-vswitchd: 04386|dpif|DBG|system at dp0:
> > execute
> > > >
> > set(eth(src=00:1f:29:32:92:4d,dst=00:1f:29:32:92:4d)),set(ipv4(src=10.0.0.10,dst=10.0.0.10,proto=6,tos=0,ttl=64,frag=no)),set(tcp(src=38710,dst=9877)),0
> > > > on packet 00:1f:29:32:92:4d > 00:1b:21:6a:85:88, ethertype IPv4
> > (0x0800),
> > > > length 74: 10.0.0.10.38710 > 10.0.0.13.5004: S
> > 3410511224:3410511224(0) win
> > > > 14600 <mss 1460,sackOK,timestamp 9255759 0,nop,wscale 6>
> > > > Jan 10 14:46:07 localhost ovs-vswitchd: 04387|dpif|DBG|system at dp0:
> > > > put[create][modify]
> > > >
> > in_port(0),eth(src=00:1f:29:32:92:4d,dst=00:1b:21:6a:85:88),eth_type(0x0800),ipv4(src=10.0.0.10,dst=10.0.0.13,proto=6,tos=0,ttl=64,frag=no),tcp(src=38710,dst=5004),
> > > >
> > actions:set(eth(src=00:1f:29:32:92:4d,dst=00:1f:29:32:92:4d)),set(ipv4(src=10.0.0.10,dst=10.0.0.10,proto=6,tos=0,ttl=64,frag=no)),set(tcp(src=38710,dst=9877)),0
> > > >
> > > >
> > > >
> > > > On Tue, Jan 10, 2012 at 1:39 PM, Ben Pfaff <blp at nicira.com> wrote:
> > > >>
> > > >> I'd expect such packets to show up in tcpdump in any case.
> > > >>
> > > >> On Tue, Jan 10, 2012 at 01:35:48PM -0500, Aaron Rosen wrote:
> > > >> > I'd be curious what the expected behavior should be if linux sees a
> > > >> > packet
> > > >> > arriving on an interface matching it's dl_src/src_ip. (Since this
> > should
> > > >> > have probably gone through lo ).
> > > >> >
> > > >> > I also enabled log_martians and it's not saying anything about this.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Aaron
> > > >> >
> > > >> > On Tue, Jan 10, 2012 at 1:32 PM, Aaron Rosen <arosen at clemson.edu>
> > wrote:
> > > >> >
> > > >> > > These packets are the normal SYN packets to initial a TCP
> > connection.
> > > >> > >
> > > >> > > The weird thing though is if I use this flow_mod it works fine:
> > > >> > > ?(using a
> > > >> > > fake ip/mac as the response).
> > > >> > >
> > > >> > > ?cookie=0x0, duration=6.622s, table=0, n_packets=93776,
> > > >> > > n_bytes=6191149,
> > > >> > >
> > > >> > >
> > idle_timeout=100,priority=65535,tcp,in_port=65534,vlan_tci=0x0000,dl_src=00:1f:29:32:92:4d,dl_dst=00:1b:21:6a:85:88,nw_src=10.0.0.10,nw_dst=10.0.0.13,nw_tos=0,tp_src=53641,tp_dst=5004
> > > >> > >
> > > >> > >
> > actions=mod_dl_src:66:f3:43:38:f4:a2,mod_dl_dst:00:1f:29:32:92:4d,mod_nw_src:10.0.0.100,mod_nw_dst:10.0.0.10,mod_tp_dst:9877,IN_PORT
> > > >> > >
> > > >> > >
> > > >> > > Then on return packets for that I have this: (so the client is
> > tricked
> > > >> > > in
> > > >> > > to connecting to 10.0.0.100 which is really it's self. Though I
> > don't
> > > >> > > want
> > > >> > > to have to rely on having an extra IP to do this.
> > > >> > >
> > > >> > > ?cookie=0x0, duration=6.622s, table=0, n_packets=143378,
> > > >> > > n_bytes=158529948,
> > > >> > >
> > > >> > >
> > idle_timeout=100,tcp,in_port=65534,dl_src=00:1f:29:32:92:4d,nw_src=10.0.0.10,nw_dst=10.0.0.100,tp_src=9877,tp_dst=53641
> > > >> > >
> > > >> > >
> > actions=mod_dl_src:00:1b:21:6a:85:88,mod_dl_dst:00:1f:29:32:92:4d,mod_nw_dst:10.0.0.10,mod_nw_src:10.0.0.13,mod_tp_src:5004,IN_PORT
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Tue, Jan 10, 2012 at 1:21 PM, Ben Pfaff <blp at nicira.com>
> > wrote:
> > > >> > >
> > > >> > >> The datapath actions look like what I'd expect to see. ?I'd
> > expect
> > > >> > >> the
> > > >> > >> modified packets to show up in tcpdump. ?I see that there are
> > only
> > > >> > >> two
> > > >> > >> such packets over an 18-second period. ?Can you send them
> > faster? ?It
> > > >> > >> looks like the second packet didn't get caught by the kernel flow
> > > >> > >> ("used:never") so both packets were actually processed in
> > userspace;
> > > >> > >> perhaps there is a bug in the userspace processing, which is
> > > >> > >> currently
> > > >> > >> in transition.
> > > >> > >>
> > > >> > >> On Tue, Jan 10, 2012 at 01:12:59PM -0500, Aaron Rosen wrote:
> > > >> > >> > pg46 openvswitch # ovs-ofctl dump-flows dp0
> > > >> > >> > NXST_FLOW reply (xid=0x4):
> > > >> > >> > # Removed other flows here for paste...
> > > >> > >> > ?cookie=0x0, duration=18.467s, table=0, n_packets=2,
> > n_bytes=148,
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > idle_timeout=100,priority=65535,tcp,in_port=65534,vlan_tci=0x0000,dl_src=00:1f:29:32:92:4d,dl_dst=00:1b:21:6a:85:88,nw_src=10.0.0.10,nw_dst=10.0.0.13,nw_tos=0,tp_src=58209,tp_dst=5004
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > actions=mod_dl_src:00:1f:29:32:92:4d,mod_dl_dst:00:1f:29:32:92:4d,mod_nw_src:10.0.0.10,mod_nw_dst:10.0.0.10,mod_tp_dst:9877,IN_PORT
> > > >> > >> > *
> > > >> > >> > *
> > > >> > >> > pg46 openvswitch # ovs-dpctl dump-flows dp0
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > in_port(0),eth(src=00:1f:29:32:92:4d,dst=00:1b:21:6a:85:88),eth_type(0x0800),ipv4(src=10.0.0.10,dst=10.0.0.13,proto=6,tos=0,ttl=64,frag=no),tcp(src=58209,dst=5004),
> > > >> > >> > packets:0, bytes:0, used:never,
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > actions:set(eth(src=00:1f:29:32:92:4d,dst=00:1f:29:32:92:4d)),set(ipv4(src=10.0.0.10,dst=10.0.0.10,proto=6,tos=0,ttl=64,frag=no)),set(tcp(src=58209,dst=9877)),0
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > pg46 openvswitch # ovs-appctl -t ovs-vswitchd ?ofproto/trace
> > dp0
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > 'in_port(0),eth(src=00:1f:29:32:92:4d,dst=00:1b:21:6a:85:88),eth_type(0x0800),ipv4(src=10.0.0.10,dst=10.0.0.13,proto=6,tos=0,ttl=64,frag=no),tcp(src=58209,dst=5004)'
> > > >> > >> > Flow: priority0:tunnel0:in_portfffe:tci(0)
> > > >> > >> > mac00:1f:29:32:92:4d->00:1b:21:6a:85:88 type0800 proto6 tos0
> > ttl64
> > > >> > >> > ip10.0.0.10->10.0.0.13 port58209->5004
> > > >> > >> > Rule: table=0 cookie=0
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > priority=65535,tcp,in_port=65534,vlan_tci=0x0000,dl_src=00:1f:29:32:92:4d,dl_dst=00:1b:21:6a:85:88,nw_src=10.0.0.10,nw_dst=10.0.0.13,nw_tos=0,tp_src=58209,tp_dst=5004
> > > >> > >> > OpenFlow
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > actions=mod_dl_src:00:1f:29:32:92:4d,mod_dl_dst:00:1f:29:32:92:4d,mod_nw_src:10.0.0.10,mod_nw_dst:10.0.0.10,mod_tp_dst:9877,IN_PORT
> > > >> > >> >
> > > >> > >> > Final flow: priority0:tunnel0:in_portfffe:tci(0)
> > > >> > >> > mac00:1f:29:32:92:4d->00:1f:29:32:92:4d type0800 proto6 tos0
> > ttl64
> > > >> > >> > ip10.0.0.10->10.0.0.10 port58209->9877
> > > >> > >> > Datapath actions:
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > set(eth(src=00:1f:29:32:92:4d,dst=00:1f:29:32:92:4d)),set(ipv4(src=10.0.0.10,dst=10.0.0.10,proto=6,tos=0,ttl=64,frag=no)),set(tcp(src=58209,dst=9877)),0
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > On Tue, Jan 10, 2012 at 12:28 PM, Ben Pfaff <blp at nicira.com>
> > wrote:
> > > >> > >> > > On Tue, Jan 10, 2012 at 12:15:39PM -0500, Aaron Rosen wrote:
> > > >> > >> > >> Hello,
> > > >> > >> > >>
> > > >> > >> > >> I have a quick question about what could be happening to
> > these
> > > >> > >> packets.
> > > >> > >> > >>
> > > >> > >> > >> I start a tcp connection 10.0.0.10 > 10.0.0.13:
> > > >> > >> > >>
> > > >> > >> > >> 12:02:55.961344 00:1f:29:32:92:4d (oui Unknown) >
> > > >> > >> > >> 00:1b:21:6a:85:88
> > > >> > >> > >> (oui Unknown), ethertype IPv4 (0x0800), length 74:
> > > >> > >> > >> 10.0.0.10.50490 >
> > > >> > >> > >> 10.0.0.13.5004: S
> > > >> > >> > >>
> > > >> > >> > >> Then the following flow entry is installed in order to
> > handle
> > > >> > >> > >> this
> > > >> > >> flow:
> > > >> > >> > >>
> > > >> > >> > >> ?cookie=0x0, duration=4.55s, table=0, n_packets=1,
> > n_bytes=74,
> > > >> > >> > >>
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > idle_timeout=10,priority=65535,tcp,in_port=65534,vlan_tci=0x0000,dl_src=00:1f:29:32:92:4d,dl_dst=00:1b:21:6a:85:88,nw_src=10.0.0.10,nw_dst=10.0.0.13,nw_tos=0,tp_src=50490,tp_dst=5004
> > > >> > >> > >>
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > actions=mod_dl_src:00:1f:29:32:92:4d,mod_dl_dst:00:1f:29:32:92:4d,mod_nw_src:10.0.0.10,mod_nw_dst:10.0.0.10,mod_tp_dst:9877,IN_PORT
> > > >> > >> > >>
> > > >> > >> > >>
> > > >> > >> > >> Though, I never see these packets come back in on the
> > interface
> > > >> > >> > >> that
> > > >> > >> > >> it went out on.
> > > >> > >> > >
> > > >> > >> > > What does "ovs-appctl ofproto/trace" or "ovs-dpctl
> > dump-flows"
> > > >> > >> > > show
> > > >> > >> > > happening to the packets?
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > --
> > > >> > >> > Aaron O. Rosen
> > > >> > >> > Masters Student - Network Communication
> > > >> > >> > 306B Fluor Daniel
> > > >> > >>
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > Aaron O. Rosen
> > > >> > > Masters Student - Network Communication
> > > >> > > 306B Fluor Daniel
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Aaron O. Rosen
> > > >> > Masters Student - Network Communication
> > > >> > 306B Fluor Daniel
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Aaron O. Rosen
> > > > Masters Student - Network Communication
> > > > 306B Fluor Daniel
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Aaron O. Rosen
> > > Masters Student - Network Communication
> > > 306B Fluor Daniel
> >
> 
> 
> 
> -- 
> Aaron O. Rosen
> Masters Student - Network Communication
> 306B Fluor Daniel