<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi, </div><div><br></div><div>thanks for your help! </div><div><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 5, 2019 at 7:26 PM Ben Pfaff <<a href="mailto:blp@ovn.org">blp@ovn.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">You're talking about the email where you dumped out a repeating sequence<br>
from some blocks? That might be the root of the problem, if you can<br>
provide some more context. I didn't see from the message where you<br>
found the sequence (was it just at the beginning of each of the 4 MB<br>
blocks you reported separately, or somewhere else), how many copies of<br>
it, or if you were able to figure out how long each of the blocks was.<br>
If you can provide that information I might be able to learn some<br>
things.<br></blockquote><div><br></div><div>Yes, those were beginnings of 0x4000000 size blocks reported by the script.</div><div>I also checked 0x8000000 blocks reported and the content is the same.</div><div>Examples of how those blocks end:</div><div> - <a href="https://pastebin.com/D9M6T2BA">https://pastebin.com/D9M6T2BA</a></div><div> - <a href="https://pastebin.com/gNT7XEGn">https://pastebin.com/gNT7XEGn</a></div><div> - <a href="https://pastebin.com/fqy4XDbN">https://pastebin.com/fqy4XDbN</a></div><div><br></div><div>So basically contents of the blocks are sequences of:</div><div><br></div><div><div><b><i><font face="monospace, monospace">00000020: 0000 0000 0000 0000 6500 0000 0000 0000 ........e.......</font></i></b></div><div><b><i><font face="monospace, monospace">00000030: 0000 0000 0000 4014 0000 0000 0000 0000 ......@.........</font></i></b></div><div><b><i><font face="monospace, monospace">00000040: 0000 0000 0000 0000 fa16 3e2b c5d5 0000 ..........>+....</font></i></b></div><div><b><i><font face="monospace, monospace">00000050: 0000 0022 0000 0000 0000 0000 0000 4014 ..."..........@.</font></i></b></div><div><b><i><font face="monospace, monospace">00000060: 0000 0000 0000 0000 0000 0000 ffff ffff ................</font></i></b></div><div><b><i><font face="monospace, monospace">00000070: ffff ffff ffff 0000 0000 0fff 0000 0000 ................</font></i></b></div></div><div><br></div><div>following each other and sometimes separated by sequences like this:</div><div><br></div><div><b><i><font face="monospace, monospace">00001040: 6861 6e64 6c65 7232 3537 0000 0000 0000 handler257......</font></i></b><br></div><div><b><i><br></i></b></div><div>I ran the scripts against several core dumps of several compute nodes with the issue and</div><div>the picture is pretty much the same: 0x4000000 blocks and less 0x8000000 blocks.</div><div>I checked the core dump from a compute node where OVS memory consumption was ok:</div><div>no such block sizes reported.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
On Tue, Mar 05, 2019 at 09:07:55AM +0400, Oleg Bondarev wrote:<br>
> Hi Ben,<br>
> <br>
> I didn't have a chance to debug the scripts yet, but just in case you<br>
> missed my last email with examples of repeatable blocks<br>
> and sequences - do you think we still need to analyze further, will the<br>
> scripts tell more about the heap?<br>
> <br>
> Thanks,<br>
> Oleg<br>
> <br>
> On Thu, Feb 28, 2019 at 10:14 PM Ben Pfaff <<a href="mailto:blp@ovn.org" target="_blank">blp@ovn.org</a>> wrote:<br>
> <br>
> > On Tue, Feb 26, 2019 at 01:41:45PM +0400, Oleg Bondarev wrote:<br>
> > > Hi,<br>
> > ><br>
> > > thanks for the scripts, so here's the output for a 24G core dump:<br>
> > > <a href="https://pastebin.com/hWa3R9Fx" rel="noreferrer" target="_blank">https://pastebin.com/hWa3R9Fx</a><br>
> > > there's 271 entries of 4MB - does it seem something we should take a<br>
> > closer<br>
> > > look at?<br>
> ><br>
> > I think that this output really just indicates that the script failed.<br>
> > It analyzed a lot of regions but didn't output anything useful. If it<br>
> > had worked properly, it would have told us a lot about data blocks that<br>
> > had been allocated and freed.<br>
> ><br>
> > The next step would have to be to debug the script. It definitely<br>
> > worked for me before, because I have fixed at least 3 or 4 bugs based on<br>
> > it, but it also definitely is a quick hack and not something that I can<br>
> > stand behind. I'm not sure how to debug it at a distance. It has a<br>
> > large comment that describes what it's trying to do. Maybe that would<br>
> > help you, if you want to try to debug it yourself. I guess it's also<br>
> > possible that glibc has changed its malloc implementation; if so, then<br>
> > it would probably be necessary to start over and build a new script.<br>
> ><br>
</blockquote></div></div></div></div></div></div></div>