[ovs-discuss] ovs-vswitch Huge number of netlink file descriptors open

[quan_hping at heetian.com]

Hi,All,
The  file descriptors are "netlink".
 $sudo lsof -p $(pidof ovs-vswitchd)
ovs-vswit 2197 root *297u  netlink                         0t0   42309398 GENERIC
ovs-vswit 2197 root *298u  netlink                         0t0   42309399 GENERIC
ovs-vswit 2197 root *299u     sock                0,7      0t0   42309400 protocol: NETLINK
ovs-vswit 2197 root *300u  netlink                         0t0   42309401 GENERIC
ovs-vswit 2197 root *301u  netlink                         0t0   42309402 GENERIC
...More...
ovs-vswit 2197 root *351u  netlink                         0t0   39326247 GENERIC
ovs-vswit 2197 root *352u  netlink                         0t0   39121841 GENERIC
ovs-vswit 2197 root *354w      REG              253,0 10086253 1342248977 /var/log/openvswitch/ovs-vswitchd.log
$sudo reboot
When I restart the machine ,the file descriptor will increasing.Some time more then 12K record.

Recenty log about openvswitch.
ovsdb-server.log
2017-10-21T04:01:02.559Z|01308|jsonrpc|WARN|Dropped 7 log messages in last 17 seconds (most recently, 10 seconds ago) due to excessive rate
2017-10-21T04:01:02.559Z|01309|jsonrpc|WARN|unix: receive error: Connection reset by peer
2017-10-21T04:01:02.560Z|01310|reconnect|WARN|unix: connection dropped (Connection reset by peer)
2017-10-21T04:01:03.282Z|01311|reconnect|WARN|unix: connection dropped (Connection reset by peer)
2017-10-21T04:01:10.075Z|01312|jsonrpc|WARN|Dropped 1 log messages in last 7 seconds (most recently, 7 seconds ago) due to excessive rate
2017-10-21T04:01:10.075Z|01313|jsonrpc|WARN|unix: send error: Broken pipe
2017-10-21T04:01:10.076Z|01314|reconnect|WARN|unix: connection dropped (Broken pipe)
2017-10-21T04:01:14.907Z|01315|reconnect|WARN|unix: connection dropped (Connection reset by peer)
2017-10-21T04:01:20.134Z|01316|reconnect|WARN|unix: connection dropped (Broken pipe)
2017-10-21T04:01:27.499Z|01317|jsonrpc|WARN|Dropped 2 log messages in last 13 seconds (most recently, 8 seconds ago) due to excessive rate
2017-10-21T04:01:27.499Z|01318|jsonrpc|WARN|unix: receive error: Connection reset by peer
2017-10-21T04:01:27.499Z|01319|reconnect|WARN|unix: connection dropped (Connection reset by peer)
2017-10-21T04:01:33.758Z|01320|jsonrpc|WARN|unix: receive error: Connection reset by peer
2017-10-21T04:01:33.758Z|01321|reconnect|WARN|unix: connection dropped (Connection reset by peer)
2017-10-21T04:01:34.489Z|01322|reconnect|WARN|unix: connection dropped (Connection reset by peer)
2017-10-21T04:01:39.198Z|01323|reconnect|WARN|unix: connection dropped (Connection reset by peer)
2017-10-21T04:01:41.569Z|01324|reconnect|WARN|unix: connection dropped (Broken pipe)
2017-10-21T04:01:41.913Z|01325|reconnect|WARN|unix: connection dropped (Broken pipe)
2017-10-21T04:01:42.365Z|01326|reconnect|WARN|unix: connection dropped (Connection reset by peer)

ovs-vswitchd.log
2017-10-21T03:56:10.511Z|422666|netlink_socket|ERR|connect(0): Argument list too long
2017-10-21T03:56:10.513Z|422667|netlink_socket|ERR|connect(0): Argument list too long
2017-10-21T03:56:10.516Z|422668|netlink_socket|ERR|connect(0): Argument list too long
2017-10-21T03:56:10.518Z|422669|netlink_socket|ERR|connect(0): Argument list too long
2017-10-21T03:56:10.520Z|422670|netlink_socket|ERR|connect(0): Argument list too long
2017-10-21T03:56:10.523Z|422671|netlink_socket|ERR|connect(0): Argument list too long
2017-10-21T03:56:10.525Z|422672|netlink_socket|ERR|connect(0): Argument list too long



合天网安实验室-您身边的信息安全实验室 | quan_hping at heetian.com
 
From: Ben Pfaff
Date: 2017-10-21 01:41
To: quan_hping at heetian.com
CC: ovs-discuss
Subject: Re: [ovs-discuss] ovs-vswitch Huge number of netlink file descriptors open
On Fri, Oct 20, 2017 at 06:56:26PM +0800, quan_hping at heetian.com wrote:
> Hi All,
> I searched a mail list like my problem.
> https://mail.openvswitch.org/pipermail/ovs-discuss/2017-March/043817.html 
> 
> $sudo lsof -p $(pidof ovs-vswitchd) | wc -l
> 115536
> $cat /proc/$(cat /var/run/openvswitch/ovs-vswitchd.pid)/limits | grep open
> Max open files            999999                999999                files
> $ovs-vsctl show | grep -c Port
> 905
> If anyone has any suggestions for how to solve it?Thanks.
 
Can you find out what kinds of file descriptors are open?  Are they
sockets (and what kind), etc.?
 
This could be a file descriptor leak of some kind.  If it is, then
restarting OVS would fix it; if it is not, then restarting OVS will not
help, or at least not for long.  Can you figure out whether restarting
reduces the number of file descriptors, and by how much?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20171021/b9126357/attachment.html>