Querying a large data set

Hi all,

I hope someone might have some sage words about the right approach here. I have a database application that works OK locally but is reporting an error on my host provider. The error comes up when I try to bring back a fairly large dataset (22500 rows, 20 columns). In the production log I see this error:

"ActionView::TemplateError (failed to allocate memory) on line #52"

From some initial googling it looks like I am running out of resource on my server. Is there a setting that would allocate more resource? I am running on a Dreamhost server using mod_rails (or should it be Passenger?).

Actually, I suppose what I should be doing is rewriting my query so that MySQL does the heavy lifting and my application just gets back a smaller resultset. Given that, is there a recommended approach to writing my app so that it is only having to handle small tranches of data at any time?

Any guidance much appreciated.

macfizz.

Hi all,

I hope someone might have some sage words about the right approach here. I have a database application that works OK locally but is reporting an error on my host provider. The error comes up when I try to bring back a fairly large dataset (22500 rows, 20 columns). In the production log I see this error:

"ActionView::TemplateError (failed to allocate memory) on line #52"

From some initial googling it looks like I am running out of resource on my server. Is there a setting that would allocate more resource? I am running on a Dreamhost server using mod_rails (or should it be Passenger?).

Rails doesn't throttle your resources. If there is a limit it will be imposed by your hosting provider (or just the hard limit on what is available in that environment)

Actually, I suppose what I should be doing is rewriting my query so that MySQL does the heavy lifting and my application just gets back a smaller resultset. Given that, is there a recommended approach to writing my app so that it is only having to handle small tranches of data at any time?

I think it will vary according to what you are trying to do.

Fred

Can you break your dataset up into pages? If so, look into will_paginate plugin. Easy to use and easy to set up.

Can you break your dataset up into pages? If so, look into will_paginate plugin. Easy to use and easy to set up.