Massive memory eating

rubyonrails-talk
1

Hi All,

I have a small ruby on rails site, using passenger and apache. The site is moderately successful. It allows people to upload photos. It seems like when people upload photos, the memory is never returned, or very late, so on a busy day I can start to use 1 GB of swap in 30 minutes.

I use the attachment_fu to handle the uploads.

Any ideas on how to use less memory and give the memory back quicker ?

Trausti

2

Try using Enterprise Ruby (from the same people as Passenger). It can reduce your memory footprint quite a bit. However, in the case you are describing, it could just be that your app needs to be tuned. If you are doing processing on the images, you might want to put those to a background thread like delayed_job, although that process could also suck all the ram, so you might need to use monit on it to periodically restart it.

However, tuning aside, if you are finding your volume is requiring more RAM, you might need to bump your hardware to support it.

3

http://www.therailsway.com/2009/4/23/uploading-files

4

I am using enterprise ruby. Thought perhaps that was the problem.

Trausti

5

Do I just add this module in my apache, and don't have to do anything in particular to my ruby stuff ?

Trausti

6

One thing I could think of without seeing the code is maybe the files are being opened while being saved, but you are not closing the files?

Chris

7

Good question. But I don't do anything like open or close. I just save the picture on upload.

Here is the add code that is when you press submit on the form

  def add     if (params[:photo])       @photo = Photo.new(params[:photo])       if @photo.save         flash[:notice] = "Picture saved"         redirect_to :controller=>'photos',:action=>'index'       end     else       @photo = Photo.new     end   end

Trausti

8

Now I just installed mod_porter and all. But when I use passenger-memory-stats

I can see that my app uses 194 MB memory, I upload 1 picture and bang it uses 285 MB and 2nd picture it uses 316 MB.

This is just ridiculous for an app that only does one thing, upload pictures and display thumbnails.

Any ideas ?

Trausti

9

By the looks of it (every spawned Rails process takes about 40-ish MB), you’re wildly spawning extra Rails processes for some reason. We are using attachment_fu (and paperclip for that matter) intensively in a number of our applications and have no problems with high memory usage (except the initial RMagick hit). There must be something either in your server processes (RMagick, ImageMagick, other gem, …) that’s miscompiled or outdated or you have something around the image uploading that kills your server (threads spawning). The simple fact of the matter is that something is keeping the Passenger rails process busy and it spawns new ones to cope with new incoming requests.

Best regards

Peter De Berdt

10

Hi Peter, here is my gemlist

actionmailer (2.3.5, 2.3.4, 2.3.3) actionpack (2.3.5, 2.3.4, 2.3.3) activerecord (2.3.5, 2.3.4, 2.3.3) activeresource (2.3.5, 2.3.4, 2.3.3) activesupport (2.3.5, 2.3.4, 2.3.3) cgi_multipart_eof_fix (2.5.0) daemons (1.0.10) fastthread (1.0.7) gem_plugin (0.2.3) mongrel (1.1.5) mysql (2.8.1) passenger (2.2.7) rack (1.0.1, 1.0.0) rails (2.3.5, 2.3.3) rake (0.8.7) rmagick (2.12.1) rubyzip (0.9.1)

and these are my plugins

attachment_fu (1-2 months old) easy-fckeditor 0.8.1 (not used anywhere) modporter-plugin (installed today) will_paginate 2.3.6

Thats all, I just updated passenger to 2.2.7 but

just by uploading one picture, private memory grows from 10.5 MB to 81.5 (with picture size 1.5 MB) and with next picture 118 MB

I never ever use find(:all). This is beyond ridiculous. There has to be something I am doing wrong.

Best regards,

Trausti

11

What do your passenger-memory-stats show if I may ask (before and after uploading)?

Best regards

Peter De Berdt

12

Before ----- Passenger processes ----- PID VMSize Private Name

13

Pfew, that's one Rails instance just eating more and more memory indeed. Have you been able to boil it down to RMagick itself. Otherwise I would suggest leaving thumbnailing out for a second and just saving the uploaded file to see if it is in fact the thumbnailing that eats your memory. We can work our way down from there.

14

It consumes less memory if I don't create thumbnails.

Trausti

15

OK, let’s go one step further then and boil it down to RMagick (and not ImageMagick): install the mini_magick gem and change the thumbnail processor to minimagick explicitly and look at memory consumption. Since Minimagick uses the command line to process images, you should see no significant memory usage increase in your Rails instance. Also keep an eye on the total memory usage on your server and whether the imagemagick process stays around after the file has been processed (mogrify), it shouldn’t.

Best regards

Peter De Berdt

16

Does that mean I need to change any more code than is in my model ?

I did set GC.start inside Photo.save in the controller. The situation does not go out of hand any more, but the app is using quite a lot of memory none the less.

Trausti

17

RMagick is quite memory hungry. RMagick 2 does have automatic garbage collection, but it seems you might be running into this gotcha:

http://rubyforge.org/forum/forum.php?thread_id=1374&forum_id=1618

Best regards

Peter De Berdt

18

One more note on this. Since you seem to be just thumbnailing, you could as well use another processor if memory is valuable. If you need to do extra processing like applying filters and constructing an image out of several other ones, then you’re best off with RMagick.

There’s nothing more to it than making sure the processor (minimagick, imagescience, gd2, core image on macos x servers) is installed correctly and explicitly defining it in your model.

Otherwise attachment_fu will see if it can use one of them in this order: @@default_processors = %w(ImageScience Rmagick MiniMagick Gd2 CoreImage).

Best regards

Peter De Berdt