[ovs-discuss] Huge number of netlink file descriptors open

[Roman Bolshakov]

Hi Paul,

I see exactly the same behavior of OVS 2.7.0. Have it been fixed in 2.7.1+?

I have more than 13k netlink sockets open, the number is slowly growing
nearly 1200 descriptors/hour.
As of now ovsdb has ~100 bridges, ~700 ports, ~700 interfaces (1 per port).

The fix (included in 2.7.1+) could be related to the issue:
https://github.com/openvswitch/ovs/commit/27af7a0d0daa92bd699270bcc5b2931f4a0247bf

Thanks,
Roman

пт, 3 мар. 2017 г. в 20:28, Paul Banks <paul at compose.io>:

> Hi all,
>
> We use OVS extensively and have a fair amount of experience operating and
> debugging it.
>
> Recently we've come up against an issue we've not seen before.
>
> I should say we are running an older build of OVS due to the fact that
> it's worked for years and is somewhat disruptive to upgrade:
>
> $ovs-vsctl --version
> ovs-vsctl (Open vSwitch) 2.3.0
> Compiled Oct 29 2014 18:25:11
> DB Schema 7.6.0
>
> $uname -a
> Linux REDACTED 3.19.0-59-generic #65~14.04.1-Ubuntu SMP Tue Apr 19
> 18:57:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>
> On one or two hosts (out of hundreds) we are seeing errors in
> ovs-vswitchd.log like:
>
> 2017-03-03T16:51:48.021Z|26523023|dpif|WARN|system at ovs-system: failed to
> add veth548eth1 as port: Too many open files
>
> We have ulimit set to 65k file descriptors and indeed they are all in use
> and almost all are netlink sockets to the kernel:
>
> $sudo lsof -p $(cat /var/run/openvswitch/ovs-vswitchd.pid) | grep netlink
> | wc -l
> 65446
>
> $sudo lsof -p $(cat /var/run/openvswitch/ovs-vswitchd.pid) | wc -l
> 65536
>
> $cat /proc/$(cat /var/run/openvswitch/ovs-vswitchd.pid)/limits | grep open
> Max open files            65535                65535                files
>
> I understand that the switch uses 3 descriptors per bridge and 1 per port,
> but we have only 2 bridges one with about 5 ports the other with 300
>
> $ovs-vsctl show | grep -c Port
> 305
>
> I've ensured there aren't any ports/interfaces in OVS that are no longer
> in existence on the host.
>
> We've seen this once before on this host and remedied it by restarting OVS
> but that is obviously disruptive to our production workloads so would like
> to understand what is happening.
>
> I checked a few other hosts in our fleet and I've found a mixture - a few
> that have been up for years have ~10-40k descriptors, many have more like
> 600. There seems to be no correlation between actual number of ovs ports
> and the number of descriptors.
>
> If anyone has any suggestions for where to look or has seen this before,
> please let us know. I've found very little online or on this list that
> seems directly relevant.
>
> Thanks
>
> Paul
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20171017/05abd046/attachment-0001.html>