<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 5, 2019 at 9:00 PM Ben Pfaff <<a href="mailto:blp@ovn.org">blp@ovn.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">OK. That gives me something to investigate. I'll see what I can find<br>
out.<br>
<br>
You're running 64-bit kernel and userspace, x86-64?<br></blockquote><div><br></div><div>Yes.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
On Tue, Mar 05, 2019 at 08:42:14PM +0400, Oleg Bondarev wrote:<br>
> On Tue, Mar 5, 2019 at 7:26 PM Ben Pfaff <<a href="mailto:blp@ovn.org" target="_blank">blp@ovn.org</a>> wrote:<br>
> <br>
> > You're talking about the email where you dumped out a repeating sequence<br>
> > from some blocks? That might be the root of the problem, if you can<br>
> > provide some more context. I didn't see from the message where you<br>
> > found the sequence (was it just at the beginning of each of the 4 MB<br>
> > blocks you reported separately, or somewhere else), how many copies of<br>
> > it, or if you were able to figure out how long each of the blocks was.<br>
> > If you can provide that information I might be able to learn some<br>
> > things.<br>
> ><br>
> <br>
> Yes, those were beginnings of 0x4000000 size blocks reported by the script.<br>
> I also checked 0x8000000 blocks reported and the content is the same.<br>
> Examples of how those blocks end:<br>
> - <a href="https://pastebin.com/D9M6T2BA" rel="noreferrer" target="_blank">https://pastebin.com/D9M6T2BA</a><br>
> - <a href="https://pastebin.com/gNT7XEGn" rel="noreferrer" target="_blank">https://pastebin.com/gNT7XEGn</a><br>
> - <a href="https://pastebin.com/fqy4XDbN" rel="noreferrer" target="_blank">https://pastebin.com/fqy4XDbN</a><br>
> <br>
> So basically contents of the blocks are sequences of:<br>
> <br>
> *00000020: 0000 0000 0000 0000 6500 0000 0000 0000 ........e.......*<br>
> *00000030: 0000 0000 0000 4014 0000 0000 0000 0000 ......@.........*<br>
> *00000040: 0000 0000 0000 0000 fa16 3e2b c5d5 0000 ..........>+....*<br>
> *00000050: 0000 0022 0000 0000 0000 0000 0000 4014 ..."..........@.*<br>
> *00000060: 0000 0000 0000 0000 0000 0000 ffff ffff ................*<br>
> *00000070: ffff ffff ffff 0000 0000 0fff 0000 0000 ................*<br>
> <br>
> following each other and sometimes separated by sequences like this:<br>
> <br>
> *00001040: 6861 6e64 6c65 7232 3537 0000 0000 0000 handler257......*<br>
> <br>
> I ran the scripts against several core dumps of several compute nodes with<br>
> the issue and<br>
> the picture is pretty much the same: 0x4000000 blocks and less 0x8000000<br>
> blocks.<br>
> I checked the core dump from a compute node where OVS memory consumption<br>
> was ok:<br>
> no such block sizes reported.<br>
> <br>
> <br>
> ><br>
> > On Tue, Mar 05, 2019 at 09:07:55AM +0400, Oleg Bondarev wrote:<br>
> > > Hi Ben,<br>
> > ><br>
> > > I didn't have a chance to debug the scripts yet, but just in case you<br>
> > > missed my last email with examples of repeatable blocks<br>
> > > and sequences - do you think we still need to analyze further, will the<br>
> > > scripts tell more about the heap?<br>
> > ><br>
> > > Thanks,<br>
> > > Oleg<br>
> > ><br>
> > > On Thu, Feb 28, 2019 at 10:14 PM Ben Pfaff <<a href="mailto:blp@ovn.org" target="_blank">blp@ovn.org</a>> wrote:<br>
> > ><br>
> > > > On Tue, Feb 26, 2019 at 01:41:45PM +0400, Oleg Bondarev wrote:<br>
> > > > > Hi,<br>
> > > > ><br>
> > > > > thanks for the scripts, so here's the output for a 24G core dump:<br>
> > > > > <a href="https://pastebin.com/hWa3R9Fx" rel="noreferrer" target="_blank">https://pastebin.com/hWa3R9Fx</a><br>
> > > > > there's 271 entries of 4MB - does it seem something we should take a<br>
> > > > closer<br>
> > > > > look at?<br>
> > > ><br>
> > > > I think that this output really just indicates that the script failed.<br>
> > > > It analyzed a lot of regions but didn't output anything useful. If it<br>
> > > > had worked properly, it would have told us a lot about data blocks that<br>
> > > > had been allocated and freed.<br>
> > > ><br>
> > > > The next step would have to be to debug the script. It definitely<br>
> > > > worked for me before, because I have fixed at least 3 or 4 bugs based<br>
> > on<br>
> > > > it, but it also definitely is a quick hack and not something that I can<br>
> > > > stand behind. I'm not sure how to debug it at a distance. It has a<br>
> > > > large comment that describes what it's trying to do. Maybe that would<br>
> > > > help you, if you want to try to debug it yourself. I guess it's also<br>
> > > > possible that glibc has changed its malloc implementation; if so, then<br>
> > > > it would probably be necessary to start over and build a new script.<br>
> > > ><br>
> ><br>
</blockquote></div></div>