<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt">After doing some reading I'm wondering if instead of getting a core dump (which I already collected when the process got to around 700MB) it would be better to run OVS through Valgrind
and share the log file.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt"><br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt">What do you think? Any specific OVS or valgrind flags I should use?</div>
<br>
</div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Fernando Casas Schössow <casasfernando@outlook.com><br>
<b>Sent:</b> Saturday, March 2, 2019 10:59 AM<br>
<b>To:</b> Ben Pfaff<br>
<b>Cc:</b> ovs-discuss@openvswitch.org; ovs-dev@openvswitch.org<br>
<b>Subject:</b> Re: [ovs-discuss] ovs-vswitchd memory consumption behavior</font>
<div> </div>
</div>
<div><br>
Sorry for the dup but the mail client didn't add ovs-dev, so resending.
<div><br>
<div>On sáb, mar 2, 2019 at 10:53 AM, Fernando Casas Schössow <casasfernando@outlook.com> wrote:<br>
<blockquote type="cite">
<div>So I rebooted the server on Monday night, after around 5 days memory usage keeps growing continuously. Already over 500MB. You can clearly see the behavior in the attached graphs from Munin (rss size + meminfo for OVS process). Memory usage growths almost
linearly in time.</div>
<div><br>
</div>
<div>Also OVS log shows the memory usage increasing over time:</div>
<div><br>
</div>
<div>
<div>2019-02-25T21:48:37+01:00 vmsvr01 ovs-vswitchd: ovs|00035|memory|INFO|12164 kB peak resident set size after 10.0 seconds</div>
<div>2019-02-25T21:48:37+01:00 vmsvr01 ovs-vswitchd: ovs|00036|memory|INFO|handlers:5 ports:7 revalidators:3 rules:261 udpif keys:36</div>
<div>2019-02-25T22:52:21+01:00 vmsvr01 ovs-vswitchd: ovs|00087|memory|INFO|peak resident set size grew 52% in last 3823.5 seconds, from 12164 kB to 18496 kB</div>
<div>2019-02-25T22:52:21+01:00 vmsvr01 ovs-vswitchd: ovs|00088|memory|INFO|handlers:5 ports:20 revalidators:3 rules:517 udpif keys:68</div>
<div>2019-02-26T00:38:57+01:00 vmsvr01 ovs-vswitchd: ovs|00128|memory|INFO|peak resident set size grew 51% in last 6395.8 seconds, from 18496 kB to 28000 kB</div>
<div>2019-02-26T00:38:57+01:00 vmsvr01 ovs-vswitchd: ovs|00129|memory|INFO|handlers:5 ports:20 revalidators:3 rules:517 udpif keys:28</div>
<div>2019-02-26T04:22:36+01:00 vmsvr01 ovs-vswitchd: ovs|00188|memory|INFO|peak resident set size grew 51% in last 13419.8 seconds, from 28000 kB to 42256 kB</div>
<div>2019-02-26T04:22:36+01:00 vmsvr01 ovs-vswitchd: ovs|00189|memory|INFO|handlers:5 ports:20 revalidators:3 rules:517 udpif keys:137</div>
<div>2019-02-26T09:19:01+01:00 vmsvr01 ovs-vswitchd: ovs|00344|memory|INFO|peak resident set size grew 51% in last 17784.3 seconds, from 42256 kB to 63640 kB</div>
<div>2019-02-26T09:19:01+01:00 vmsvr01 ovs-vswitchd: ovs|00345|memory|INFO|handlers:5 ports:21 revalidators:3 rules:517 udpif keys:123</div>
<div>2019-02-26T16:19:24+01:00 vmsvr01 ovs-vswitchd: ovs|01027|memory|INFO|peak resident set size grew 50% in last 25223.3 seconds, from 63640 kB to 95584 kB</div>
<div>2019-02-26T16:19:24+01:00 vmsvr01 ovs-vswitchd: ovs|2019-02-25T21:48:37+01:00 vmsvr01 ovs-vswitchd: ovs|00035|memory|INFO|12164 kB peak resident set size after 10.0 seconds</div>
<div>2019-02-25T21:48:37+01:00 vmsvr01 ovs-vswitchd: ovs|00036|memory|INFO|handlers:5 ports:7 revalidators:3 rules:261 udpif keys:36</div>
<div>2019-02-25T22:52:21+01:00 vmsvr01 ovs-vswitchd: ovs|00087|memory|INFO|peak resident set size grew 52% in last 3823.5 seconds, from 12164 kB to 18496 kB</div>
<div>2019-02-25T22:52:21+01:00 vmsvr01 ovs-vswitchd: ovs|00088|memory|INFO|handlers:5 ports:20 revalidators:3 rules:517 udpif keys:68</div>
<div>2019-02-26T00:38:57+01:00 vmsvr01 ovs-vswitchd: ovs|00128|memory|INFO|peak resident set size grew 51% in last 6395.8 seconds, from 18496 kB to 28000 kB</div>
<div>2019-02-26T00:38:57+01:00 vmsvr01 ovs-vswitchd: ovs|00129|memory|INFO|handlers:5 ports:20 revalidators:3 rules:517 udpif keys:28</div>
<div>2019-02-26T04:22:36+01:00 vmsvr01 ovs-vswitchd: ovs|00188|memory|INFO|peak resident set size grew 51% in last 13419.8 seconds, from 28000 kB to 42256 kB</div>
<div>2019-02-26T04:22:36+01:00 vmsvr01 ovs-vswitchd: ovs|00189|memory|INFO|handlers:5 ports:20 revalidators:3 rules:517 udpif keys:137</div>
<div>2019-02-26T09:19:01+01:00 vmsvr01 ovs-vswitchd: ovs|00344|memory|INFO|peak resident set size grew 51% in last 17784.3 seconds, from 42256 kB to 63640 kB</div>
<div>2019-02-26T09:19:01+01:00 vmsvr01 ovs-vswitchd: ovs|00345|memory|INFO|handlers:5 ports:21 revalidators:3 rules:517 udpif keys:123</div>
<div>2019-02-26T16:19:24+01:00 vmsvr01 ovs-vswitchd: ovs|01027|memory|INFO|peak resident set size grew 50% in last 25223.3 seconds, from 63640 kB to 95584 kB</div>
<div>2019-02-26T16:19:24+01:00 vmsvr01 ovs-vswitchd: ovs|01028|memory|INFO|handlers:5 ports:21 revalidators:3 rules:517 udpif keys:75</div>
<div>2019-02-27T01:08:23+01:00 vmsvr01 ovs-vswitchd: ovs|01096|memory|INFO|peak resident set size grew 50% in last 31739.4 seconds, from 95584 kB to 143632 kB</div>
<div>2019-02-27T01:08:23+01:00 vmsvr01 ovs-vswitchd: ovs|01097|memory|INFO|handlers:5 ports:20 revalidators:3 rules:517 udpif keys:44</div>
<div>2019-02-27T18:49:17+01:00 vmsvr01 ovs-vswitchd: ovs|01933|memory|INFO|peak resident set size grew 50% in last 63654.0 seconds, from 143632 kB to 215704 kB</div>
<div>2019-02-27T18:49:17+01:00 vmsvr01 ovs-vswitchd: ovs|01934|memory|INFO|handlers:5 ports:20 revalidators:3 rules:517 udpif keys:21</div>
<div>2019-02-28T17:26:30+01:00 vmsvr01 ovs-vswitchd: ovs|03182|memory|INFO|peak resident set size grew 50% in last 81432.6 seconds, from 215704 kB to 323680 kB</div>
<div>2019-02-28T17:26:30+01:00 vmsvr01 ovs-vswitchd: ovs|03183|memory|INFO|handlers:5 ports:20 revalidators:3 rules:517 udpif keys:62</div>
<div>2019-03-02T04:33:27+01:00 vmsvr01 ovs-vswitchd: ovs|04201|memory|INFO|peak resident set size grew 50% in last 126417.0 seconds, from 323680 kB to 485776 kB</div>
<div>2019-03-02T04:33:27+01:00 vmsvr01 ovs-vswitchd: ovs|04202|memory|INFO|handlers:5 ports:20 revalidators:3 rules:517 udpif keys:2801028|memory|INFO|handlers:5 ports:21 revalidators:3 rules:517 udpif keys:75</div>
<div>2019-02-27T01:08:23+01:00 vmsvr01 ovs-vswitchd: ovs|01096|memory|INFO|peak resident set size grew 50% in last 31739.4 seconds, from 95584 kB to 143632 kB</div>
<div>2019-02-27T01:08:23+01:00 vmsvr01 ovs-vswitchd: ovs|01097|memory|INFO|handlers:5 ports:20 revalidators:3 rules:517 udpif keys:44</div>
<div>2019-02-27T18:49:17+01:00 vmsvr01 ovs-vswitchd: ovs|01933|memory|INFO|peak resident set size grew 50% in last 63654.0 seconds, from 143632 kB to 215704 kB</div>
<div>2019-02-27T18:49:17+01:00 vmsvr01 ovs-vswitchd: ovs|01934|memory|INFO|handlers:5 ports:20 revalidators:3 rules:517 udpif keys:21</div>
<div>2019-02-28T17:26:30+01:00 vmsvr01 ovs-vswitchd: ovs|03182|memory|INFO|peak resident set size grew 50% in last 81432.6 seconds, from 215704 kB to 323680 kB</div>
<div>2019-02-28T17:26:30+01:00 vmsvr01 ovs-vswitchd: ovs|03183|memory|INFO|handlers:5 ports:20 revalidators:3 rules:517 udpif keys:62</div>
<div>2019-03-02T04:33:27+01:00 vmsvr01 ovs-vswitchd: ovs|04201|memory|INFO|peak resident set size grew 50% in last 126417.0 seconds, from 323680 kB to 485776 kB</div>
<div>2019-03-02T04:33:27+01:00 vmsvr01 ovs-vswitchd: ovs|04202|memory|INFO|handlers:5 ports:20 revalidators:3 rules:517 udpif keys:28</div>
</div>
<div><br>
</div>
<div>I can collect a dump of the process (using gcore) and then try to run your scripts to see if they help to get some information out of the dump but as I mentioned my debugging skills are very limited and from what I read in the other thread the scripts
may not help without modifications.</div>
<div>For this reason I'm adding OVS dev mailing list to the thread in case any of the devs want to have a look at the dump and debug this issue.</div>
<div>If anyone wants access to the dump, reply to me and I will upload it to a server so it can be downloaded.</div>
<div><br>
</div>
<div>BTW, should I open an issue in github about this?</div>
<div><br>
</div>
<div>Thanks!</div>
<br>
On dom, feb 17, 2019 at 2:28 AM, Fernando Casas Schössow <casasfernando@outlook.com> wrote:<br>
<blockquote type="cite">OK, let's do this.
<div>Up to the part of getting the process core dump, I'm familiar with (gcore PID).</div>
<div>The analysis part of the dump, I will have to learn it...but I hope your scripts will help at least in part.</div>
<div><br>
</div>
<div>I restarted OVS last night so it will take a couple of weeks to get to 1.5GB or so, then I will collect the dump and start the analysis.</div>
<div><br>
</div>
<div>Thanks in advance for helping with this Ben.</div>
<div><br>
<br>
</div>
</blockquote>
<br>
<br>
</blockquote>
<br>
<br>
</div>
</div>
</div>
</body>
</html>