[ovs-discuss] VMs lost network connections randomly

[Gurucharan Shetty]

On Wed, Aug 21, 2013 at 6:23 PM, Jian Qiu <swordqiu at gmail.com> wrote:

> Thanks Gurucharan!
>
> It sounds interesting. We'd like to try this approach. But could you
> please point out where I could find OVS kernel-userspace compatibility
> information? Is there any more detailed guide to upgrade ovs kernel without
> reboot?
>
The man page of ovs-ctl has information about 'force-reload-kmod'. If you
are using dhcp on OVS bridges, make sure to restart it after a
'force-reload-kmod'. As an example, now that you are on 1.4, you would be
installing the userspace and kernel module packages of 1.5. Then do a
'/etc/init.d/openvswitch[-switch] force-reload-kmod'. Restart your dhcp
service (if you are using it on a OVS bridge, else your lease will
eventually timeout). Then jump to 1.6, 1.7 and then to 1.9. I think you can
jump directly from 1.4->1.7. But test it before trying on a production
setup.




>
> Many thanks,
>
> Jian
>
>
>
> On Thu, Aug 22, 2013 at 5:04 AM, Gurucharan Shetty <shettyg at nicira.com>wrote:
>
>> On Mon, Aug 19, 2013 at 4:45 PM, Jian Qiu <swordqiu at gmail.com> wrote:
>>
>>> Jesse,
>>>
>>> Thanks for your quick response.
>>>
>>> Which version would you recommend upgrade to? As you know, this is our
>>> production environment, stability is the first priority.
>>>
>>> May I upgrade the userspace programs only? We might not be able to
>>> upgrade kernel module as replacing kernel module may require a server
>>> reboot and we couldn't stop the running service on it.
>>>
>> If you upgrade OVS from one released version to the immediate next one, I
>> don't think you will need a reboot. A "force-reload-kmod" should be good
>> enough. You can continue doing this till you reach the last released
>> branch. ( I think you can skip a few released branches in between that have
>> the userspace - kernel module compatibility).
>>
>>
>>
>>> Many thanks!
>>>
>>> Jian Qiu
>>>
>>>
>>>
>>> On Tue, Aug 20, 2013 at 12:29 AM, Jesse Gross <jesse at nicira.com> wrote:
>>>
>>>> On Mon, Aug 19, 2013 at 6:39 AM, Jian Qiu <swordqiu at gmail.com> wrote:
>>>> > Hi,
>>>> >
>>>> > We deployed openvswitch 1.4.6 on CentOS 6.4 in our production
>>>> envioronment.
>>>> > OVS were used to bridge KVM VMs.
>>>> >
>>>> > Recently, we found some VMs lost network connections randomly. When
>>>> the
>>>> > event happened, we login into the VMs through VNC console and found
>>>> the ARP
>>>> > tables were full of incomplete entries. It seemed that the ARP packets
>>>> > in/out of the VMs were being dropped.
>>>> >
>>>> > We checked the ovs-vswitchd message logs when the network
>>>> disconnection
>>>> > happened, we found most of the messages complained high CPU usage of
>>>> > ovs-switchd and "dropped log messages". (Samples are attached at the
>>>> end of
>>>> > email)
>>>> >
>>>> > I'm writing the email for your kind help.
>>>> >
>>>> > 1. Did the "dropped log messages" cause the VMs losing network
>>>> connections?
>>>>
>>>> No, it just means that the log messages have been rate limited.
>>>>
>>>> > 2. Is there any quick fix to this issue except replacing the OVS
>>>> kernel
>>>> > module as we couldn't reboot the hypervisor?
>>>>
>>>> Not a quick fix but newer versions of OVS will likely improve your
>>>> performance.
>>>>
>>>
>>>
>>> _______________________________________________
>>> discuss mailing list
>>> discuss at openvswitch.org
>>> http://openvswitch.org/mailman/listinfo/discuss
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20130822/d8492622/attachment.html>