[ovs-dev] ovs-vswitch kernel panic randomly started after 400+ days uptime

Sun Jan 8 14:30:17 UTC 2017

Hi Pravin,

That's great, thank you! I hope that it gets picked up and backported to
3.13 in a new version. Do you know the likelihood of them incorporating it
or the timeline that these things generally take?

Two final questions:

   1. What do you think triggered this after so long? The bug has always
   been there and we are running a computing cluster with 10 nodes all running
   the exact same software configuration and not one of them experienced this
   for 406 days. Combined that's over 4,000 days of continuous processing.
   Curious as to what could have happened to start this and to essentially
   make it unsolvable by rebooting everything/rolling services etc. Do packets
   get "stuck" - what would generate this traffic? Is it malicious activity?

   2. Ultimately the fix we implemented was to disconnect a specific
   network interface that was plugged to a switch but that was only after we
   decided to test one node with this network card offline and seeing it
   stable for days while the others crashed. We tried over a dozen different
   troubleshooting techniques before arriving at that conclusion and each one
   took half a dozen hours because they had to be implemented across 10
   servers and then we had to randomly wait for one to maybe crash, or
   not...there was no way to test/trigger this that we knew of. So, just
   wanted to get your take on what you would have done/tried to troubleshoot
   had you been faced with this issue - removing the fact that you are so
   intimately close with the code that you would have known it was a bug very
   quickly :)

Thanks,
Uri

On Sun, Jan 8, 2017 at 9:16 AM, Pravin Shelar <pshelar at ovn.org> wrote:

>
>
> On Sun, Jan 8, 2017 at 12:18 AM, Uri Foox <uri at zoey.com> wrote:
>
>> Hi Pravin,
>>
>> Thanks. Does this mean it is a confirmed bug?
>>
>>
> Yes, there is atleast one bug in ovs vport gre implementation.
>
>
>> How would I be able to get the patch and install it into our environment?
>>
>> Easiest way would be compile OVS 2.5 or 2.6. But I have posted patch on
> netdev mailing list for upstream kernel you can use that for 3.13 kernel.
>
> https://patchwork.ozlabs.org/patch/712373/
>
> Thanks,
> Pravin.
>
>
> Thanks,
>> Uri
>>
>>
>> On Sat, Jan 7, 2017 at 1:01 PM, Pravin Shelar <pshelar at ovn.org> wrote:
>>
>>> Thanks for all investigation.
>>>
>>> On Sat, Jan 7, 2017 at 12:57 AM, Joe Stringer <joe at ovn.org> wrote:
>>> >
>>> >
>>> > On 5 January 2017 at 19:24, Uri Foox <uri at zoey.com> wrote:
>>> >>
>>> >> Hey Joe,
>>> >>
>>> >> Thank you so much for responding! After 10 days of trying to figure
>>> this
>>> >> out I'm at a loss.
>>> >>
>>> >> root at node-8:~# modinfo openvswitch
>>> >> filename:
>>> >> /lib/modules/3.13.0-106-generic/kernel/net/openvswitch/openvswitch.ko
>>> >> license:        GPL
>>> >> description:    Open vSwitch switching datapath
>>> >> srcversion:     94294A72258BA583D666607
>>> >> depends:        libcrc32c,vxlan,gre
>>> >> intree:         Y
>>> >
>>> >
>>> > ^ intree - that is, the version that comes with this kernel.
>>> >
>>> >>
>>> >> vermagic:       3.13.0-106-generic SMP mod_unload modversions
>>> >>
>>> >>
>>> >> Everything you've mentioned is what I've understood so far including
>>> the
>>> >> line of code that's triggered. That is what led me to upgrade the
>>> kernel to
>>> >> 3.13.0-106 because it claims that the CHECKSUM problems are fixed
>>> which I
>>> >> thought this might be related, guess not.
>>> >
>>> >
>>> > I forgot to actually look through those before, but the call chain
>>> looks a
>>> > bit different there so I thought it may be a different issue
>>> altogether.
>>> >
>>> >>
>>> >> You're saying that skb_headlen is too short for the ethernet header.
>>> Do
>>> >> you know what would cause this? This hardware configuration has been
>>> running
>>> >> for 400+ days of uptime with no errors or problems and this suddenly
>>> started
>>> >> to happen and no matter how many time we reboot things it doesn't go
>>> away.
>>> >> I assume given your interpretation we should try to restart the
>>> switches
>>> >> connected to the servers. Is there any way to log what packet is
>>> causing
>>> >> this issue? Perhaps that would provide more insight?
>>> >
>>> >
>>> > One thing is that it depends on the packets and how they arrive. I'm
>>> not too
>>> > familiar with this code, but I could imagine a situation where the
>>> IP+GRE
>>> > packet gets fragmented, causing a single inner frame to be split across
>>> > muliple GRE packets. Then, when Linux receives the two separate
>>> packets,
>>> > there would be some point in the stack responsible for stitching these
>>> > packets back together; but it may not put them into a single contiguous
>>> > buffer. If this is subsequently decapped for local delivery of the
>>> inner
>>> > frame, then perhaps there is less than an ethernet header's worth of
>>> packet
>>> > in the first of these buffers. It seems unlikely that packets would be
>>> > deliberately fragmented like this, but if anyone had access to your
>>> > underlying network then they could throw any kind of packet they want
>>> to
>>> > your server.
>>> >
>>> > There may be another, more likely, explanation - CC Pravin in case he
>>> has
>>> > any ideas.
>>> >
>>> >>
>>> >> As far as 4.4/newer kernel - I wish. I tried to go that far up but
>>> Ubuntu
>>> >> wouldn't even boot. The best I could do is 3.13.0-106. I'll try to
>>> report it
>>> >> over there as well.
>>> >
>>> >
>>> > That's too bad.
>>> >
>>> > FWIW, I see a check for pskb_may_pull() in the outer gre_rcv function,
>>> which
>>> > would check on the whole GRE packet.. this is then passed to
>>> gre_cisco_rcv()
>>> > which does the decap and calls through to the OVS gre_rcv() function.
>>> At a
>>> > glance, following the OVS' gre_rcv() I didn't see another
>>> psukb_may_pull()
>>> > check for the inner packet. By the time it gets to ovs_flow_extract(),
>>> > there's an expectation that this call was made but I'm really not sure
>>> who
>>> > was supposed to make that check. Also, it should be ETH_HLEN, which is
>>> 14,
>>> > not 12..
>>> >
>>> Right. OVS do expect the-header already in skb linear data. It is done
>>> in iptunnel_pull_header() for tunnel packets. This function is called
>>> for all packets received in GRE module.
>>>
>>> http://lxr.free-electrons.com/source/net/ipv4/ip_tunnel_core
>>> .c?v=3.13#L96
>>>
>>> But the skb eth-header is only pulled for GRE-TAP packets not for
>>> IP-GRE. The change in network could have introduced these IP-GRE
>>> packets that caused the crash.
>>>
>>> This bug does not exist in out of tree kernel module that come with
>>> OVS 2.5 and newer. So upgrading OVS kernel module to 2.5 should solve
>>> the problem.
>>>
>>> I will sent out a patch for older OVS kernel module.
>>>
>>> > Outer gre_rcv():
>>> > http://lxr.free-electrons.com/source/net/ipv4/gre_demux.c?v=3.13#L270
>>> >
>>> > Inner gre_rcv():
>>> > http://lxr.free-electrons.com/source/net/openvswitch/vport-g
>>> re.c?v=3.13#L92
>>>
>>
>>
>>
>> --
>> Uri Foox | Zoey | Founder
>> http://www.zoey.com
>>
>
>

-- 
Uri Foox | Zoey | Founder
http://www.zoey.com