[ovs-dev] Conntrack with SCTP: +est is never reached.

Tue Mar 24 19:56:41 UTC 2020

The problem ended up being an OVN bug in the ordering of NAT operations in
flows. These particular packets are SNAT'ed and DNAT'ed. OVN is committing
them in an opposite order than it is handling subsequent packets in
established state. In other words, subsequent packets are being sent to CT
to do an SNAT, where they do not match the CT session, and then in table
14, ct matches this as invalid, because the packet did not match in the
previous CT call. It's not an SCTP specific problem, it happens in TCP as
well. For more info:

https://bugzilla.redhat.com/show_bug.cgi?id=1718372#c18
https://bugzilla.redhat.com/show_bug.cgi?id=1815217#c12

Tim Rozet
Red Hat CTO Networking Team

On Sun, Mar 22, 2020 at 11:47 AM Aaron Conole <aconole at redhat.com> wrote:

> Ben Pfaff <blp at ovn.org> writes:
>
> > If I'm interpreting this correctly, then it sounds like OVS does
> > work with SCTP.  If so, and if OVN does not, then it seems
> > likely that OVN is not using OVS conntrack correctly in some way
> > for SCTP.  (Or possibly that Tim and Aaron are using different
> > versions of OVS or the kernel or the OVS kernel module, such
> > that Tim is seeing an OVS kernel or conntrack bug that Aaron is
> > not.)
>
> I tried with a couple different versions of OvS and kernel, so I
> think it's probably not that.  I'm hoping Tim can confirm on
> Monday for us.   I also hacked up my kernel to print out the state
> flags to make sure I wasn't missing anything, and they're
> displaying as expected (from an IRC conversation):  14:50
> <aconole> I see the state flags as 0x22 and 0x2a
> 14:50 <aconole> with the initials as 0x21
> 14:52 <mleitner> hm
> 14:53 <aconole> so, 0x22 = OVS_CS_F_TRACKED | OVS_CS_F_ESTABLISHED
> 14:53 <aconole> 0x2a = OVS_CS_F_TRACKED | OVS_CS_F_ESTABLISHED |
>                 OVS_CS_F_REPLY_DIR
> 14:54 <aconole> 0x21 = OVS_CS_F_TRACKED | OVS_CS_F_NEW
>
> First two packets (INIT and INIT_ACK) are 0x21, and after that it's
> 0x22/0x2a (so it looks like even the COOKIE packets are flagged as +est
> state packets).  I hope I didn't miss anything.
>
> > I haven't looked at any details of OVS or OVN, so the above is just idle
> > speculation.
> >
> > On Fri, Mar 20, 2020 at 03:22:10PM -0400, Aaron Conole wrote:
> >> Aaron Conole <aconole at redhat.com> writes:
> >>
> >> > Aaron Conole <aconole at redhat.com> writes:
> >> >
> >> >> Tim Rozet <trozet at redhat.com> writes:
> >> >>
> >> >>> I filed https://bugzilla.redhat.com/show_bug.cgi?id=1815217 to
> track this issue.
> >> >>
> >> >> Thanks!
> >> >
> >> > I tested with the following setup (no modifications to kernel or ovs):
> >>
> >> Final update below
> >>
> >> > # using kernel 5.6.0-rc6+, ovs master built using make rpm-fedora and
> installed
> >> >
> >> > ip netns add left
> >> > ip netns add right
> >> > ip link add center-left type veth peer name left0
> >> > ip link add center-right type veth peer name right0
> >> > ip link set center-left up
> >> > ip link set center-right up
> >> > ip link set left0 netns left
> >> > ip link set right0 netns right
> >> > ip netns exec left ip addr add 172.31.110.1/30 dev left0
> >> > ip netns exec right ip addr add 172.31.110.2/30 dev right0
> >> > ip netns exec left ip link set left0 up
> >> > ip netns exec right ip link set right0 up
> >> >
> >> > # just to ignore any possible selinux issues...
> >> > setenforce Permissive
> >> > systemctl start openvswitch
> >> >
> >> > systemctl start openvswitch
> >> > ovs-vsctl add-br br0 -- set Bridge br0 fail-mode=secure
> >> > ovs-vsctl add-port br0 center-left
> >> > ovs-vsctl add-port br0 center-right
> >> > ovs-ofctl add-flow br0 table=0,arp,action=NORMAL
> >> >
> >> > ovs-ofctl add-flow br0 'table=0,sctp,actions=ct(table=1)'
> >> > ovs-ofctl add-flow br0 \
> >> >
>  'table=1,sctp,in_port=center-left,ct_state=+trk+new,actions=ct(commit),center-right'
> >>
> >>
> >> > ovs-ofctl add-flow br0 \
> >> >
>  'table=1,sctp,in_port=center-right,ct_state=+rpl+trk,actions=center-left'
> >> > ovs-ofctl add-flow br0 \
> >>
> >> ^^ This flow isn't actually needed.
> >>
> >> Additionally, I tested with the rhel8 kernel, and FDP from rhel8
> >> (openvswitch2.12-2.12.0-22), and still was able to get an SCTP
> >> connection.
> >>
> >> >
>  'table=1,sctp,in_port=center-left,ct_state=+trk+est,actions=center-right'
> >> > ovs-ofctl add-flow br0 \
> >> >
>  'table=1,sctp,in_port=center-right,ct_state=+trk+est,actions=center-left'
> >> >
> >> > # ensure arp is following action normal
> >> > ip netns exec left arping 172.31.110.2 -I left0
> >> >
> >> > # in one terminal
> >> > ip netns exec right ncat --listen --sctp -vv
> >> >
> >> > # in another terminal
> >> > ip netns exec left ncat --sctp 172.31.110.2 31337
> >> >
> >> > Result:
> >> >
> >> > [root at wsfd-netdev92 ~]# ip netns exec right ncat --listen --sctp -vv
> >> > Ncat: Version 7.70 ( https://nmap.org/ncat )
> >> > Ncat: Listening on :::31337
> >> > Ncat: Listening on 0.0.0.0:31337
> >> > Ncat: Connection from 172.31.110.1.
> >> > Ncat: Connection from 172.31.110.1:34461.
> >> > asdf
> >> > fdsa
> >> > fasdfsadf
> >> > asdfasdfasdfda
> >> >
> >> >
> >> > Seems I have bidirectional communications... It looks like you need
> the
> >> > +rpl flow to match replies (which is what I would expect).
> >> >
> >> > Looking at the dpctl flows, I see the following display
> (periodically):
> >> >
> recirc_id(0x2b),in_port(3),ct_state(-new+rpl+trk),eth(),eth_type(0x0800),ipv4(proto=132,frag=no),
> >> > packets:1, bytes:98, used:4.310s, actions:2
> >> >
> recirc_id(0x2c),in_port(2),ct_state(-new+est-rpl+trk),eth(),eth_type(0x0800),ipv4(proto=132,frag=no),
> >> > packets:1, bytes:98, used:4.314s, actions:3
> >> >
> >> > And from dump-conntrack:
> >> >
> sctp,orig=(src=172.31.110.1,dst=172.31.110.2,sport=34461,dport=31337),reply=(src=172.31.110.2,dst=172.31.110.1,sport=31337,dport=34461),protoinfo=(state=ESTABLISHED,vtag_orig=2708668805,vtag_reply=1149194430)
> >> >
> >> > Does it help?
> >> >
> >> >>> Tim Rozet
> >> >>> Red Hat CTO Networking Team
> >> >>>
> >> >>> On Thu, Mar 19, 2020 at 3:11 PM Ben Pfaff <blp at ovn.org> wrote:
> >> >>>
> >> >>>  On Thu, Mar 19, 2020 at 10:27:52AM -0400, Mark Michelson wrote:
> >> >>>  > I've recently been working on adding support for SCTP load
> balancers in
> >> >>>  > OVN[1]. In a recent test run by Tim Rozet, he ran into an issue
> with my
> >> >>>  > patch[2].
> >> >>>
> >> >>>  Do we have any idea whether OVS conntrack works for SCTP in
> general?
> >> >>>
> >> >>>  Aaron, you're the only person I can quickly find who has committed
> >> >>>  anything related to sctp and conntrack, with commit 93346d889271
> >> >>>  ("conntrack: add display support for sctp").  Did you test
> conntrack
> >> >>>  with sctp or did you have any reports of success or failure with
> it?
> >>
>
>