[ovs-discuss] openvswitch udev event flood

Flavio Leitner fbl at redhat.com
Mon Sep 8 18:59:24 UTC 2014


On Mon, Sep 08, 2014 at 09:34:03AM -0700, Gurucharan Shetty wrote:
> I saw something similar today. In my case, 'force-reload-kmod' would
> trigger the udev event loop. Adding HOTPLUG=no in
> /etc/sysconfig/network-scripts/ifcfg-* solved the problem for me. I do
> not know the exact sequence of events that makes udev act that way.
> 
> Here is one bug report where the end result is the same but the
> trigger is different.
> https://bugzilla.redhat.com/show_bug.cgi?id=855107
 

There might be some scenarios where the hotplug event can trigger
a slave device which depends on the master which depends on something
that causes a loop.  The HOTPLUG=no on ports/slaves usually helps to
break such loops.

I don't recall anything apart of OpenStack monitoring for openvswitch
module to load in the system.  Even if you load openvswitch, it should
not create any device by default.  However, if you restart the service
with 'stale' devices in the DB, the hotplug event will be fired. Maybe
that is confusing the agent.

My suggestion to understand this better  would be to change
/etc/rc.d/init.d/openvswitch to capture additional info about
which pid is the parent, etc. For example, inserting those
3 'echo' lines below.

#!/bin/bash
#

echo "Parent: $(cat /proc/$PPID/cmdline) " >> /tmp/ovs.loop
echo "pid: $PPID, command line $*" >> /tmp/ovs.loop
echo "pstree: $(pstree -a -l -n)" >> /tmp/ovs.loop

# openvswitch
#
# chkconfig: 2345 09 91
# description: Manage Open vSwitch kernel modules and user-space
# daemons


fbl

> On Sun, Aug 24, 2014 at 12:16 PM, Ben Pfaff <blp at nicira.com> wrote:
> > No.
> >
> > I suggest you find out what program is invoking ovs-vsctl and go from
> > there.
> >
> > On Sun, Aug 24, 2014 at 12:00:29PM -0700, Krishna Sagiraju wrote:
> >> Thanks for the response, Ben.
> >>
> >> Actually I did try disabling all OpenStack Icehouse services - nova
> >> and neutron. But, I still see the messages and events. Restarting
> >> /etc/init.d/openvswitch service or executing "ovs-ctl
> >> force-reload-kmod" almost always stops the spew, but that could be
> >> for a totally different reason. I did say, "almost always" because,
> >> sometimes force-reload-kmod doesn't help either unless I reboot the
> >> box.  And even after a reboot, I can reproduce the problem
> >> consistently by just executing "ovs-ctl force-reload-kmod".
> >>
> >> Do you have any insights on what other services could be a suspect?
> >>
> >> --Krishna/.
> >>
> >> On 8/24/14, 11:39 AM, Ben Pfaff wrote:
> >> >On Sun, Aug 24, 2014 at 06:02:06AM -0700, Krishna Sagiraju wrote:
> >> >>I am running into a problem with openvswith and would greatly
> >> >>appreciate help/suggestions from experts.
> >> >>
> >> >>I installed openvswitch-1.11.0_8ce28d-1.el6ost.x86_64 on CentOS 6.5
> >> >>running Icehouse and as soon as the openvswitch kernel module is
> >> >>loaded, I get a flood of the following messages in
> >> >>/var/log/messages:
> >> >>
> >> >>Aug 22 15:23:55 host ovs-vsctl: ovs|00001|vsctl|INFO|Called as
> >> >>ovs-vsctl -t 10 -- --if-exists del-br br-eth1
> >> >>Aug 22 15:23:55 host kernel: device br-eth1 left promiscuous mode
> >> >>Aug 22 15:23:55 host ovs-vsctl: ovs|00001|vsctl|INFO|Called as
> >> >>ovs-vsctl -t 10 -- --may-exist add-br br-eth1
> >> >>Aug 22 15:23:55 host kernel: device br-eth1 entered promiscuous mode
> >> >>Aug 22 15:23:56 host ovs-vsctl: ovs|00001|vsctl|INFO|Called as
> >> >>ovs-vsctl -t 10 -- --if-exists del-br br-eth1
> >> >>Aug 22 15:23:56 host kernel: device br-eth1 left promiscuous mode
> >> >>Aug 22 15:23:56 host ovs-vsctl: ovs|00001|vsctl|INFO|Called as
> >> >>ovs-vsctl -t 10 -- --may-exist add-br br-eth1
> >> >>Aug 22 15:23:56 host kernel: device br-eth1 entered promiscuous mode
> >> >>Aug 22 15:23:56 host ovs-vsctl: ovs|00001|vsctl|INFO|Called as
> >> >>ovs-vsctl -t 10 -- --if-exists del-br br-eth1
> >> >>Aug 22 15:23:56 host kernel: device br-eth1 left promiscuous mode
> >> >>Aug 22 15:23:56 host ovs-vsctl: ovs|00001|vsctl|INFO|Called as
> >> >>ovs-vsctl -t 10 -- --may-exist add-br br-eth1
> >> >>
> >> >>I traced those messages to udev trying to toggle the interface up
> >> >>and down rapidly. Here's a sample of the events received by UDEV:
> >> >>
> >> >>[root at host ~]# udevadm monitor
> >> >>monitor will print the received events for:
> >> >>UDEV - the event which udev sends out after rule processing
> >> >>KERNEL - the kernel uevent
> >> >>
> >> >>KERNEL[1408843389.801627] remove
> >> >>/devices/virtual/net/br-eth1/queues/rx-0 (queues)
> >> >>KERNEL[1408843389.801667] remove
> >> >>/devices/virtual/net/br-eth1/queues/tx-0 (queues)
> >> >>KERNEL[1408843389.801742] remove   /devices/virtual/net/br-eth1 (net)
> >> >>UDEV  [1408843389.820986] remove   /devices/virtual/net/br-eth1 (net)
> >> >>KERNEL[1408843389.870369] add      /devices/virtual/net/br-eth1 (net)
> >> >>KERNEL[1408843389.870391] add
> >> >>/devices/virtual/net/br-eth1/queues/rx-0 (queues)
> >> >>KERNEL[1408843389.870401] add
> >> >>/devices/virtual/net/br-eth1/queues/tx-0 (queues)
> >> >>UDEV  [1408843389.940706] add      /devices/virtual/net/br-eth1 (net)
> >> >>UDEV  [1408843389.940944] add
> >> >>/devices/virtual/net/br-eth1/queues/tx-0 (queues)
> >> >>UDEV  [1408843389.940959] add
> >> >>/devices/virtual/net/br-eth1/queues/rx-0 (queues)
> >> >>UDEV  [1408843389.941067] remove
> >> >>/devices/virtual/net/br-eth1/queues/tx-0 (queues)
> >> >>UDEV  [1408843389.941082] remove
> >> >>/devices/virtual/net/br-eth1/queues/rx-0 (queues)
> >> >>KERNEL[1408843390.088915] remove
> >> >>/devices/virtual/net/br-eth1/queues/rx-0 (queues)
> >> >>KERNEL[1408843390.088941] remove
> >> >>/devices/virtual/net/br-eth1/queues/tx-0 (queues)
> >> >>KERNEL[1408843390.089002] remove   /devices/virtual/net/br-eth1 (net)
> >> >>UDEV  [1408843390.105143] remove   /devices/virtual/net/br-eth1 (net)
> >> >>
> >> >>
> >> >>I would appreciate any advice and/or any workarounds to alleviate
> >> >>the issue. Once again, thanks for the help in advance
> >> >The problem is that you have some software creating and deleting devices
> >> >rapidly.  Open vSwitch isn't doing that on its own, so you will have to
> >> >figure out what is.
> >>
> > _______________________________________________
> > discuss mailing list
> > discuss at openvswitch.org
> > http://openvswitch.org/mailman/listinfo/discuss
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> http://openvswitch.org/mailman/listinfo/discuss
> 



More information about the discuss mailing list