OutOfMemory exception - caching problem?

Poster Content
nk4um User
Posts: 90
May 10, 2013 10:16

Hello,

as I have to know what's going on I looked at a heap dump after an OutOfMemoryException with the fabulous MAT and found this:

14 instances of "org.netkernel.http.transport.HTTPRequestSpace2Wrapper", 
loaded by "org.netkernel.layer0.util.SpaceClassLoader @ 0xe0b6d5b0" 
occupy 227.470.768 (76,28%) bytes. 

Biggest instances:

•org.netkernel.http.transport.HTTPRequestSpace2Wrapper @ 0xf0aba640 - 18.957.928 (6,36%) bytes. 
•org.netkernel.http.transport.HTTPRequestSpace2Wrapper @ 0xe5442ea8 - 18.955.136 (6,36%) bytes.
[12 more]

These objects have exactly the same size as my input XML file and they are accumulated by this object:

class org.netkernel.lang.groovy.endpoint.GroovyCompiler @ 0xe0ba0df8
class java.util.HashMap @ 0xe052d0e8
class java.util.HashMap$Entry[] @ 0xe055edb0
class java.util.HashMap$Entry @ 0xe055ed40

Inside GroovyCompiler.java I assume it's this HashMap which stores the objects:

private Map<List<ISpace>,GroovyClassLoader> mCLCache=new HashMap();

Some lines later there is this code with a comment:

synchronized(mCLCache)
            {   if(mCLCache.size()>200)
                {   mCLCache.clear();           //Safety limit to ensure this can't leak but still gives typical optimal reuse of classloaders              
                }
                mCLCache.put(l,gcl);
            }

This cache gets cleared if more than 200 objects are inside, but there is no limit to the overall memory size of the cache. If my assumptions are correct, the heap space is eaten up by this cache. Could the management of this cache be done in a more robust way, which takes the memory size of the stored objects into account or use the representation cache or provide a NO_CACHE flag?

Thanks, Stefan

Like · Post Reply
nk4um Administrator
Posts: 607
January 12, 2013 11:22

Hi Stefan, something unknown to us is happening to cause this out of memory. The standard way to track these problems down is to run the scenario to close to where and out of memory will occur and then take a heap dump. Analysis of this heap dump will show what is consuming heap and why.

Amendum: I've reread the whole thread and I see we've tried this path before. It may be worth doing it again with the latest groovy code and see what's happening now.

Cheers, Tony

Like · Post Reply
nk4um User
Posts: 90
January 12, 2013 11:13

After an update to lang-groovy 1.13.1 as described in the News i tested my module again but the out of memory exception persists.

Stefan

Like · Post Reply
nk4um User
Posts: 90
September 3, 2012 14:25

As it is very important for me to know why the Groovy script is not working as expected, i tried NKEE as Peter was using together with the Grovvy script implementation. And it worked for 50+ times without a memory problem. The status graph looks pretty much like Peters:

status graph screenshot

So something is different between my NKEE 5.1.1 and NKSE 5.1.1. NKEE originates from a download from May without any apposite updates or changes. NKSE has the latest updates. JRE, memory size, max headroom and the other configurations as shown in the configuration panel are the same.

What are the consequences: avoid scripting runtimes and just use Java to avoid possible memory issues with some NK installations?

Thanks, Stefan

Stefan

Like · Post Reply
nk4um User
Posts: 90
August 23, 2012 13:04

Hello Peter,

I have tried your suggestion, changed the Groovy script to a Java class and things worked without memory problems.

Nevertheless here is the visualizer screenshot of the Groovy script:

And here is the complete visualizer trace.

Maybe this is important: Win 7 with jdk1.7.0_05.

NK Cache Configuration: Heap Headroom: 170; Max Size: 6000

Thanks, Stefan

Like · Post Reply
nk4um Moderator
Posts: 901
August 22, 2012 14:38

Hi Stefan,

I've run your code and unfortunately I cannot replicate your problem. I uploaded a 24Mb XML file 50 times. Here's the heap trace...

...

However I'm not that surprised since with the visualizer I am seeing that groovy script is compiling and caching - which is what I would expect

...

So the next step is to try to understand how your compilation of the script is different to mine. Can you please run the visualizer and send me a copy of the trace for http://localhost:8080/aqua-intern/template (shown above for my system).

We'll sort this out one way or another! Hang in there...

P.

Like · Post Reply
nk4um Moderator
Posts: 901
August 21, 2012 16:17

Thanks Stefan - I'll take a look and try to reproduce.

In the mean time - you will avoid the problem entirely if you switch the implementation of active:templateConfig from groovy to a regular Java accessor.

Will talk with Mr B about this and see if we can find you a better solution.

P.

Like · Post Reply
nk4um User
Posts: 90
August 19, 2012 11:38

Hello Peter,

I tried your suggestion but no change. I created a new module just with this one endpoint, this is the module.xml:

<moduleversion="2.0">
  <meta>
    <identity>
      <uri>urn:stefan:test&lt;/uri&gt;
      <version>1.0.0&lt;/version&gt;
    &lt;/identity&gt;
    <info>
      <name>test_aqua&lt;/name&gt;
      <description>Created with New Module Wizard&lt;/description&gt;
      <icon>res:/test_aqua/pub/icon.png&lt;/icon&gt;
    &lt;/info&gt;
  &lt;/meta&gt;
  <system>
    <dynamic /&gt;
  &lt;/system&gt;
  <rootspacename="test_aqua - Development Space" uri="urn:stefan:test">
    <fileset>
      <regex>res:/etc/system/SimpleDynamicImportHook.xml&lt;/regex&gt;
    &lt;/fileset&gt;
    <fileset>
      < !--Icon-->
      <regex>res:/test_aqua/pub/(.*)&lt;/regex&gt;
      <rewrite>res:/resources/pub/$1&lt;/rewrite&gt;
    &lt;/fileset&gt;
    <fileset>
      <regex>res:/test_aqua/(.*)&lt;/regex&gt;
      <rewrite>res:/resources/web/$1&lt;/rewrite&gt;
    &lt;/fileset&gt;
    <accessor>
      <grammar>res:/aqua-intern/template&lt;/grammar&gt;
      <class>resources.endpoints.TemplateConfiguration&lt;/class&gt;
    &lt;/accessor&gt;
    <mapper>
      <config>
        <endpoint>
          <id>urn:stefan:aqua:Template-configuration&lt;/id&gt;
          <name>Template-configuration&lt;/name&gt;
          <description>Create the SQL import configuration for a template&lt;/description&gt;
          <grammar>
            <active>
              <identifier>active:templateConfig&lt;/identifier&gt;
              <argumentname="file" desc="The XML file with the template decription" min="0" max="1" /&gt;
              <argumentname="template" desc="The name of the template" min="0" max="1" /&gt;
            &lt;/active&gt;
          &lt;/grammar&gt;
          <request>
            <identifier>active:groovy&lt;/identifier&gt;
            <argumentname="operator">res:/resources/endpoints/template-configuration.gy&lt;/argument&gt;
            <argumentname="template">arg:template&lt;/argument&gt;
            <argumentname="file">arg:file&lt;/argument&gt;
          &lt;/request&gt;
        &lt;/endpoint&gt;
      &lt;/config&gt;
      <space>
        <fileset>
          <regex>res:/resources/endpoints/.*&lt;/regex&gt;
        &lt;/fileset&gt;
        <import>
          <uri>urn:org:netkernel:lang:groovy&lt;/uri&gt;
        &lt;/import&gt;
        <import>
          <uri>urn:org:netkernel:ext:layer1&lt;/uri&gt;
          <private /&gt;
        &lt;/import&gt;
      &lt;/space&gt;
    &lt;/mapper&gt;
    <import>
      <uri>urn:org:netkernel:ext:layer1&lt;/uri&gt;
      <private /&gt;
    &lt;/import&gt;
  &lt;/rootspace&gt;
  <rootspacename="test_aqua - Documentation Space" public="true" uri="urn:stefan:test:doc">
    <fileset>
      <regex>res:/etc/system/(Books|Docs).xml&lt;/regex&gt;
    &lt;/fileset&gt;
    <fileset>
      <regex>res:/resources/doc/.*&lt;/regex&gt;
    &lt;/fileset&gt;
  &lt;/rootspace&gt;
  <rootspacename="test_aqua - Unit Test Space" public="true" uri="urn:stefan:test:test">
    <fileset>
      <regex>res:/etc/system/Tests.xml&lt;/regex&gt;
    &lt;/fileset&gt;
    <fileset>
      <regex>res:/resources/test/.*&lt;/regex&gt;
    &lt;/fileset&gt;
    <endpoint>
      < !-- stop any configuration from module under test from polluting address space -->
      <prototype>Limiter&lt;/prototype&gt;
      <grammar>res:/etc/
        <regextype="anything" /&gt;
      &lt;/grammar&gt;
    &lt;/endpoint&gt;
    <import>
      < !-- import the main space which we are testing -->
      <uri>urn:stefan:test&lt;/uri&gt;
    &lt;/import&gt;
    <import>
      <uri>urn:org:netkernel:ext:layer1&lt;/uri&gt;
      <private /&gt;
    &lt;/import&gt;
  &lt;/rootspace&gt;
&lt;/module&gt;

Here is a screenshot of the NK status graph:

NK status graph

If you have time to take a look, the zipped module with the code is here. The xml file contains business data so I can't post it here. But any XML file of similar size will do if it contains the template element that is included in the zip file. The value of the attribute name has then to be put in the second input field of the form at http://localhost:8080/test_aqua/template-configuration.html. The first field takes the XML file.

Thanks for your patience, Stefan

Like · Post Reply
nk4um Moderator
Posts: 901
August 17, 2012 08:45

Hi Stefan,

Thanks - the heap shows the problem pretty clearly. The GroovyCompiler has an internal cache, this is because Groovy holds internally the dynamically generated classes it creates when it compiles a script. This is blowing up and using all the memory...

Somehow your HTTP superstack is getting mixed into the Transreption of your groovy script. Probably through a missing import in the mapper space where you map active:templateConfig to active:groovy ??

Essentially the scope of the HTTP request is being held on to indefinitely. Unfortunately this also means that the transient upload data is also referenced.

I think if you turn on the visualizer and take a look at the resolution of the Groovy Transreption you'll find why this top scope is getting mixed in - actually now I think about it - I bet its because you're missing a local import of layer1 transreptors and so they're getting resolved up through the HTTP fulcrum superstack. Make sure to import layer1 in your mapper space too!

Let me know if this helps...

P.

PS For those that want to know why scope comes into it... its slightly subtle - but since the GroovyCompiler is a pseudo-singleton, it needs to use the spacial context as part of its key to the compiled class in its local cache map - since the identity of a script does not have to be unique globally.

Like · Post Reply
nk4um User
Posts: 90
August 16, 2012 21:49

Hello Peter,

I followed the documentation to do the upload process and looked through the apposite module how it's done there but didn't get a clue.

Here is the linkt to the heapDump.

Thanks for your support!

Stefan

Like · Post Reply
nk4um Moderator
Posts: 901
August 16, 2012 11:03

Hi Stefan,

Can you make the heap dump available (gziped as I assume its a pretty big file). I'll take a look.

The file upload is just dereferenced when the POST request is completed so its not clear why you're seeing references still. I know for sure that the guys at DeltaXML have a very similar solution and it handles very large XML file uploads - so I don't think there is anything fundamental here. But lets see if we can figure it out from the heap analysis first.

Cheers,

Peter

Like · Post Reply
nk4um User
Posts: 90
August 15, 2012 20:04

Hello,

as the problem stays with me i made a heapDump and and tried to read it. I looked at the class resources.endpoints.TemplateConfiguration (which is called by the browser to handle the input, please see my first post) and found references to 11 objects of type ByteArray which are each 18MB large. The are referenced by org.netkernel.http.util.MultipartRequest$Part and are the uploaded files aqua.xml. Is it correct to assume that the uploaded file lies multiple time on the heap and can't get garbage collected because there are still references?

How would i construct an upload process to avoid this problem?

Thanks, Stefan

BTW: In the 'Explore' menu i tried the 'Search' feature, but with my browser (Firefox 14) nothing happens after entering the search string and pressing 'Enter'. No button around to click either.

Like · Post Reply
nk4um User
Posts: 90
August 12, 2012 18:46

Hello Tony,

thank your for the quick response. I tried different combinations or your suggestions, but nothing helped.

I set the NO_CACHE in the code above:

INKFRequest req = context.createRequest("active:templateConfig");
      req.setHeader(INKFRequest.HEADER_NO_CACHE, true);

and tried the following things:

Cache items: 6000 Heap Headroom: 170MB

Cache items: 600 Heap Headroom: 170MB

Cache items: 300 Heap Headroom: 170MB

After 6-54 calls I got OutOfMemoryException.

Then I changed the jvmsettings.cnf, deleted the -Xmx and -Xms options to let java set the amount of memory. The machine has 4GB RAM and java took around 670MB for the heap, but no luck either.

Next thing I'll try is to use a tool like visualgc to see what's going on. But it seems it has nothing to with the NK cache configuration.

Stefan

Like · Post Reply
nk4um Administrator
Posts: 607
August 8, 2012 17:29

Hi Stefan,

I suspect the problem lies with the default cache configuration. One of the limitations that the cache has, java in general actually, that it can't determine how big each item it is caching is. And because all of you representations are very large you're seeing problems with the default configuration that assumes lots of smaller/medium sized ones. Also note that an in-memory DOM of a 18MB XML file will be many times larger, maybe 5x - 10x so larger heap may be a good way to go or consider a custom SAX/StaX streaming pipeline.

There are several things you can do however. 1) make the heap headroom much much bigger 2) limit the size of the cache (observe how big it gets before running out of memory) 3) inhibit caching within the application where you don't want it by using the "no-cache" header.

Cheers, Tony

Like · Post Reply
nk4um User
Posts: 90
August 8, 2012 17:13OutOfMemory exception - caching problem?

Hello,

I'm trying to process a rather large XML file with about 18MB through a simple web application but get into some memory trouble.

This is the html form that prompts for the user input:

<div>
<form method="POST" enctype="multipart/form-data" action="/aqua/template"> 
<p>
Select the XML file: <br />
<input type="file" name="aqua-xml" size="80"/>
</p>
<p>
Name of the template:<br />
<input type="template" name="template" size="80"/><br />
</p>
<input type="submit" value="Start"/>
</form>
</div>

This is the Java code that is called for getting the user inputs:

[...]
publicclass TemplateConfiguration extends StandardAccessorImpl {
   
  @Override
  publicvoid onSource(INKFRequestContext context) throws Exception {
      [...]
  INKFRequest req = context.createRequest("active:templateConfig");
  //req.setHeader(INKFRequest.HEADER_FORGET_DEPENDENCIES, true);//req.setHeader(INKFRequest.HEADER_NO_CACHE, true);
  req.addArgument("template", template);
  req.addArgument("file", "httpRequest:/upload/aqua-xml");
  Object result = context.issueRequest(req);
  INKFResponse response = context.createResponseFrom(result);
  //response.setExpiry(INKFResponse.EXPIRY_ALWAYS);
}

And this is the code of the active:templateConfig endpoint, which is implemented as a Groovy script:

[...]
file = context.getThisRequest().getArgumentValue("file");
Document doc = (Document)context.source(file, Document.class);
DOMXDA xda = new DOMXDA(doc, false);
StringBuilder sb = new StringBuilder();

[.. some XML and string  processing...]

response = context.createResponseFrom(sb.toString());
response.setMimeType("text/plain");
//response.setExpiry(INKFResponse.EXPIRY_ALWAYS);

The result is always ok, but after a unspecific number of calls - varying from 3-15 so far - a java.lang.OutOfMemoryError exception is thrown, sometimes combined with a GC overhead limit exceeded.

There is one call after the other, no concurrent calls occur. The memory for the JVM is 512MB which is sufficient for the processing of the XML file.

The comments in the above code show my tryouts to change the cache behaviour, but no luck at all. I also changed the Heap Headroom in the Netkernel configuration but no result.

Does this look like a Netkernel caching problem or like a Java garbage collector and memory problem? Anybody with some experience on this topic who can give me a hint?

Thanks in advance,

Stefan

Like · Post Reply