Rails/JRuby WAR Heap Space Issue

I am using JRuby / Rails / Glassfish Gem to run an app on Tomcat.

When someone submits a large file as an uploaded file I see the following in the log:

Processing ScholarshipsController#create (for 174.120.167.34 at 2009-12-21 10:03:13) [POST] Parameters: {"authenticity_token"=>"9c26734a0f31...bf3a9d644e9001abb931", "scholarship"=>{...data removed for privacy...}

Dec 21, 2009 10:03:15 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet default threw exception java.lang.OutOfMemoryError: Java heap space        at org.jruby.util.Pack.unpack(Pack.java:924)        at org.jruby.RubyString.unpack(RubyString.java:6806)        at org.jruby.RubyString$i_method_1_0$RUBYINVOKER$unpack.call (org/ jruby/RubyString$i_method_1_0$RUBYINVOKER$unpack.gen)        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall (CachingCallSite.java:310)        at org.jruby.runtime.callsite.CachingCallSite.call (CachingCallSite.java:149)        at ruby.jit.ruby.home.mfromin.jvm.apache_minus_tomcat_minus_6_dot_0_dot_14.domains.scholarships_dot_sam_dot_org.SamScholar.WEB_minus_INF.gems.gems.activerecord_minus_jdbc_minus_adapter_minus_0_dot_9_dot_2.lib.jdbc_adapter.jdbc_mysql.quote17997250_17974850.__file__ (jdbc_mysql.rb:82)        at ruby.jit.ruby.home.mfromin.jvm.apache_minus_tomcat_minus_6_dot_0_dot_14.domains.scholarships_dot_sam_dot_org.SamScholar.WEB_minus_INF.gems.gems.activerecord_minus_jdbc_minus_adapter_minus_0_dot_9_dot_2.lib.jdbc_adapter.jdbc_mysql.quote17997250_17974850.__file__ (jdbc_mysql.rb)        at org.jruby.ast.executable.AbstractScript.__file__ (AbstractScript.java:51)        at org.jruby.internal.runtime.methods.JittedMethod.call (JittedMethod.java:187)        at org.jruby.runtime.callsite.CachingCallSite.call (CachingCallSite.java:187)        at ruby.jit.ruby.home.mfromin.jvm.apache_minus_tomcat_minus_6_dot_0_dot_14.domains.scholarships_dot_sam_dot_org.SamScholar.WEB_minus_INF.gems.gems.activerecord_minus_2_dot_2_dot_2.lib.active_record.base.attributes_with_quotes1213394_2293223.block_0$RUBY $__block__(base.rb:2817)        at ruby.jit.ruby.home.mfromin.jvm.apache_minus_tomcat_minus_6_dot_0_dot_14.domains.scholarships_dot_sam_dot_org.SamScholar.WEB_minus_INF.gems.gems.activerecord_minus_2_dot_2_dot_2.lib.active_record.base.attributes_with_quotes1213394_2293223BlockCallback $block_0$RUBY$__block__xx1.call(Unknown Source)        at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java: 105)        at org.jruby.runtime.Block.yield(Block.java:194)        at org.jruby.RubyArray.eachCommon(RubyArray.java:1635)        at org.jruby.RubyArray.each(RubyArray.java:1642)        at org.jruby.RubyArray$i_method_0_0$RUBYFRAMEDINVOKER$each.call (org/ jruby/RubyArray$i_method_0_0$RUBYFRAMEDINVOKER$each.gen)        at org.jruby.runtime.callsite.CachingCallSite.callBlock (CachingCallSite.java:116)        at org.jruby.runtime.callsite.CachingCallSite.callIter (CachingCallSite.java:133)        at ruby.jit.ruby.home.mfromin.jvm.apache_minus_tomcat_minus_6_dot_0_dot_14.domains.scholarships_dot_sam_dot_org.SamScholar.WEB_minus_INF.gems.gems.activerecord_minus_2_dot_2_dot_2.lib.active_record.base.attributes_with_quotes1213394_2293223.__file__ (base.rb:2808)        at ruby.jit.ruby.home.mfromin.jvm.apache_minus_tomcat_minus_6_dot_0_dot_14.domains.scholarships_dot_sam_dot_org.SamScholar.WEB_minus_INF.gems.gems.activerecord_minus_2_dot_2_dot_2.lib.active_record.base.attributes_with_quotes1213394_2293223.__file__ (base.rb)        at org.jruby.internal.runtime.methods.JittedMethod.call (JittedMethod.java:102)        at org.jruby.internal.runtime.methods.DefaultMethod.call (DefaultMethod.java:147)        at org.jruby.internal.runtime.methods.AliasMethod.call (AliasMethod.java:76)        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall (CachingCallSite.java:280)        at org.jruby.runtime.callsite.CachingCallSite.call (CachingCallSite.java:69)        at ruby.jit.ruby.home.mfromin.jvm.apache_minus_tomcat_minus_6_dot_0_dot_14.domains.scholarships_dot_sam_dot_org.SamScholar.WEB_minus_INF.gems.gems.activerecord_minus_jdbc_minus_adapter_minus_0_dot_9_dot_2.lib.active_record.connection_adapters.jdbc_adapter.attributes_with_quotes24825033_2293223.__file__ (jdbc_adapter.rb:67)        at ruby.jit.ruby.home.mfromin.jvm.apache_minus_tomcat_minus_6_dot_0_dot_14.domains.scholarships_dot_sam_dot_org.SamScholar.WEB_minus_INF.gems.gems.activerecord_minus_jdbc_minus_adapter_minus_0_dot_9_dot_2.lib.active_record.connection_adapters.jdbc_adapter.attributes_with_quotes24825033_2293223.__file__ (jdbc_adapter.rb)        at org.jruby.ast.executable.AbstractScript.__file__ (AbstractScript.java:43)        at org.jruby.internal.runtime.methods.JittedMethod.call (JittedMethod.java:119)        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall (CachingCallSite.java:290)        at org.jruby.runtime.callsite.CachingCallSite.call (CachingCallSite.java:109) Dec 21, 2009 10:07:42 AM org.apache.catalina.core.ApplicationContext log

Is this issue caused on the upload itself or when the file is attempting to be persisted to MySQL? Is it caused when the file is uploaded but not yet persisted?

What is the best way to address this? Can I limit the size of the file uploaded?

I currently have 64MB heap space - I can update this but don't want to spend more $$ with my host if there is a better solution.

Sorry, right off the bat that makes zero sense.

Are you using Tomcat or the Glassfish gem to run your app? It can't be both.

I am using JRuby / Rails / Glassfish Gem to run an app on Tomcat.

OK, never mind the previous comment -- looking at the stack trace it's obviously Tomcat.

When someone submits a large file as an uploaded file I see the following in the log:

Processing ScholarshipsController#create (for 174.120.167.34 at 2009-12-21 10:03:13) [POST] Parameters: {"authenticity_token"=>"9c26734a0f31...bf3a9d644e9001abb931", "scholarship"=>{...data removed for privacy...}

Dec 21, 2009 10:03:15 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet default threw exception java.lang.OutOfMemoryError: Java heap space

Is this issue caused on the upload itself or when the file is attempting to be persisted to MySQL? Is it caused when the file is uploaded but not yet persisted?

Are you really saving a file to MySQL? Not usually recommended.

What is the best way to address this? Can I limit the size of the file uploaded?

Probably -- what are you using to handle the upload now?

I currently have 64MB heap space -

Regardless of anything else, that seems awfully low. What are the startup params for Tomcat?

Have you tried uploading to a local instance with more memory?

I am using both Tomcat and the Glassfish GEM (not Glassfish server). The Glassfish gem replaces Mongrel in the stack.

As for the file - I am aware of the pros/cons of storing the file in the DB and for this specific app storing the PDF in the DB is indeed what I want to do. The DB will not ever exceed 500 records and a single person should never have more than one file so its easy enough to overwrite what is in the db.

As for the heap - what I am really trying to determine is whether or not the issue is really heap size or something in MySQL such as max_packet_size.

On my local test server I have never had heap issues (but I have more than 64 MB) but I did get max_packet_size errors before increasing that on MySQL. From the hosting side I just have less control over the server and fewer things I can see myself.

I am using both Tomcat and the Glassfish GEM (not Glassfish server). The Glassfish gem replaces Mongrel in the stack.

I use all three in various circumstances, and I can't begin to imagine how you think the GF gem is being used in conjunction with Tomcat. But whatever.

As for the heap - what I am really trying to determine is whether or not the issue is really heap size or something in MySQL such as max_packet_size.

Your error log shows an OOM - I'd say it's pretty obvious. But if you want more detail, set up the JVM to heap dump on OOM and look at the dump with a profiler.

Or set up your local test system with 64Mb and recreate it there with more logging.