[ovs-discuss] ovs-conntrack kernel panic

Thu Mar 26 15:26:40 UTC 2015

Hi Joe,

I have tried your latest commit and it seems to work well.
The original kernel panic reported has been fixed.

I managed to get to the bottom of the TCP issue.
Tracing it though I found that some packets were being rejected in the
nf_conntrack_ipv4 module due to bad checksums in the transport layer.
This, in turn, meant that no nf_conn entry was created and there was
potentially a null pointer exception in ovs-conntrack.
I noticed that this error occurred even when replayed traffic checksums
were validated in wireshark.

Removing checksum validation from nf_conntrack did the trick .....
'echo  0 > /proc/sys/net/netfilter/nf_conntrack_checksum'

Thanks for your time,
John

On Wed, Mar 25, 2015 at 11:56 PM, Joe Stringer <joestringer at nicira.com>
wrote:

> I updated the branch at
> https://github.com/justinpettit/ovs/commits/conntrack.
>
> This includes various bugfixes, IPv6, connlabel, initial work on
> IP(v4) fragments and ALGs - mostly in userspace. It also makes use of
> Jarno's recent xlate context work to save the OF pipeline when
> recirculating.
>
> There are several new tests in tests/kmod-traffic.at; The python
> script for running those tests requires pyftpdlib.
>
> On 24 March 2015 at 23:04, Joe Stringer <joestringer at nicira.com> wrote:
> > FYI I'm planning to update this branch this week - hopefully tomorrow,
> > so you might have better luck with that version.
> >
> > Cheers,
> > Joe
> >
> > On 24 March 2015 at 20:29, Joe Stringer <joestringer at nicira.com> wrote:
> >> Hmm. You might be able to get more information about how the
> >> datapath(incl. conntrack) is identifying the flow by turning up debug
> >> logging in vswitchd (look for upcall and flow_put):
> >>
> >> ovs-appctl vlog/set dpif:dbg
> >>
> >> The conntrack tools can also be useful, although you'll need to filter
> >> the output to only the relevant traffic:
> >>
> >> conntrack -L
> >>
> >> I see you're also using zones; removing those would reduce the problem
> space.
> >>
> >> If it's still not clear, there's always the option to log some
> >> messages in ovs_ct_execute() and trace the execution.
> >>
> >> On 24 March 2015 at 09:03, John Hurley <john.hurley at netronome.com>
> wrote:
> >>> Hi Joe,
> >>>
> >>>
> >>> Following on from my earlier message, I have tested a few more
> scenarios.
> >>> It seems that Conntrack works fine for UDP and ICMP packet flows.
> >>> Earlier I had been testing with TCP traffic.
> >>> This is still not working.
> >>> Any ideas on why TCP would be failing on the conntrack commit?
> >>>
> >>> I have also noticed that if I replay UDP packets shorter than 64 bytes
> into
> >>> OVS they are rejected by conntract and can cause the kernel panic
> issue.
> >>> The NULL check on the nf_ct_is_confirmed() you mentioned earlier is
> required
> >>> to prevent this from happening.
> >>>
> >>> John
> >>>
> >>>
> >>> On Tue, Mar 24, 2015 at 9:56 AM, John Hurley <
> john.hurley at netronome.com>
> >>> wrote:
> >>>>
> >>>> Hi Joe,
> >>>>
> >>>> Thanks for the response.
> >>>> I have ensured the nf_conntrack_ipv4 module is loaded but still
> cannot get
> >>>> the commit to work.
> >>>>
> >>>> I am running Ubuntu 14.04
> >>>> Kernel version: 3.16.0-30-generic
> >>>>
> >>>> John
> >>>>
> >>>> On Mon, Mar 23, 2015 at 6:27 PM, Joe Stringer <joestringer at nicira.com
> >
> >>>> wrote:
> >>>>>
> >>>>> Hi John,
> >>>>>
> >>>>> I strongly suspect in this case, conntrack has been unable to
> identify
> >>>>> the connection. There's two parts to this: Perhaps you need to load
> >>>>> nf_conntrack_ipv4? if that module is not loaded, then conntrack
> cannot
> >>>>> identify any IP traffic. Separately, to address the crash, the line
> >>>>> with the nf_ct_is_confirmed() call likely needs to check if ct is
> >>>>> non-NULL first.
> >>>>>
> >>>>> One useful datapoint is what kernel version/distro are you running?
> >>>>> I've been testing on the latest net-next, although I should expect it
> >>>>> work on anything back to linux-3.10. Perhaps earlier depending on the
> >>>>> particular features required, YMMV.
> >>>>>
> >>>>> Apologies, there's a few bugs like this which I have found in local
> >>>>> testing but I have not pushed the changes yet. This is in part
> because
> >>>>> the ongoing development is against the linux tree, so I haven't
> >>>>> backported the current development code to build inside the OVS tree.
> >>>>>
> >>>>> Without the commit, the nf_conntrack tables will not be populated.
> >>>>>
> >>>>> On 23 March 2015 at 09:49, John Hurley <john.hurley at netronome.com>
> wrote:
> >>>>> > Hi,
> >>>>> > I am interested in using Conntrack within OVS and have installed
> the
> >>>>> > latest
> >>>>> > version from the banch
> >>>>> > (https://github.com/justinpettit/ovs/tree/conntrack).
> >>>>> >
> >>>>> > When I run an example mentioned in the tests:
> >>>>> >
> >>>>> > ovs-ofctl del-flows br0
> >>>>> > ovs-ofctl add-flow br0 \
> >>>>> >     "in_port=1,conn_state=-trk,tcp,action=ct(commit,zone=9),2"
> >>>>> > ovs-ofctl add-flow br0 \
> >>>>> >     "in_port=2,conn_state=-trk,tcp,action=ct(recirc,zone=9)"
> >>>>> > ovs-ofctl add-flow br0
> "in_port=2,conn_state=+trk+est-new,tcp,action=1"
> >>>>> > ovs-ofctl add-flow br0
> >>>>> > "in_port=2,conn_state=+trk-est+new,tcp,action=drop"
> >>>>> > ovs-ofctl add-flow br0 priority=10,action=normal
> >>>>> >
> >>>>> > I am seeing a kernel panic for the 'commit' action.
> >>>>> > I have traced this to the nf_ct_is_confirmed(ct) call on the
> >>>>> > nf_conntrack
> >>>>> > module.
> >>>>> > If I edit the code to avoid this check I avoid the panic but am
> getting
> >>>>> > an
> >>>>> > error response from calls such as 'nf_ct_get(skb, &ctinfo)'.
> >>>>> >
> >>>>> > It appears that ovs-conntrack is not populating the nf_conntrack
> tables
> >>>>> > for
> >>>>> > me.
> >>>>> > Is there any information on installing/running the ovs-conntrack
> branch
> >>>>> > specifically or is there anything else that should be done on top
> of
> >>>>> > the
> >>>>> > standard OVS installation.
> >>>>> > Doing an lsmod I can see both openvswitch and nf_conntrack modules
> >>>>> > running.
> >>>>> >
> >>>>> > Thanks,
> >>>>> >
> >>>>> > John
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > _______________________________________________
> >>>>> > discuss mailing list
> >>>>> > discuss at openvswitch.org
> >>>>> > http://openvswitch.org/mailman/listinfo/discuss
> >>>>> >
> >>>>
> >>>>
> >>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20150326/7fcef5bd/attachment-0002.html>