rails 3.1 assets:precompile deployment

Hi all,

I'm just deploying a few rails apps after upgrading to rails 3.1. I
have implemented the asset pipeline and everything works ok.

I'm deploying using Capistrano with the pre fabricated deploy/assets
recipe.

The problem I encounter is that the rake task takes like forever to
precompile all my assets. I have to admit that there are a lot of assets
that need to be precompiled +300 (images / js / css ) , becouse they
include things like jquery ui, tiny mce, etc .

I didn't think this might cause a problem, because when I run the
assets:precompile rake task on my dev machine, all assets get
precompiles within a few seconds, while doing this during deployment, it
take almost more then 10minutes and the server load get really high!

I haven't done any custom changes in the way I implemented the asset
pipeline whatsoever.
Hoping that someone can clarify if there are differences between
precompiling in dev <-> prod, because I definitely need to optimize
this.

Thank you in advance

I experienced the same thing, although in my case it takes so long to precompile on my server I don’;t know if it ever finishes, waited 15 minutes one time and it still had not finished.

My solution was to precompile locally which takes a few seconds, then copy the public/assets directory to my server. My cap task is…

desc "deploy the precompiled assets"
task :deploy_assets, :except => { :no_release => true } do
   run_locally("rake assets:clean && rake precompile")
   upload("public/assets", "#{release_path}/public/assets", :via => :scp, :recursive => true)
end

Note I don’t use the supplied assets:precompile due to a bug in it which has not yet been released but the task has see the rails bug lists for that.

I run into the same problem.

add this in Capfile:

load ‘deploy/assets’

it will automatically do the deploy_assets task ( assets:precompile )

it will create a link under public directory

assets -> $application_path/production/shared/assets/

for this to work you must use:

gem ‘rails’

gem ‘sprockets’, :git => ‘git://github.com/sstephenson/sprockets.git

and in environments/production.rb (otherwise it won’t work )

#config.action_dispatch.x_sendfile_header = “X-Sendfile” # Use ‘X-Accel-Redirect’ for nginx

config.action_dispatch.x_sendfile_header = “X-Accel-Redirect”

config.assets.compile = true

config.assets.digest = true

Hope this helps.

– Aldo Nievas
SATIO Internet Solutions

www.satio.com.ar
aldo@satio.com.ar

M: 54 11 3487 4900

@jim: precompiling them locally seems indeed the best way at the moment.
But I'm still curious what the reason is that precompiling on server
takes that long. I'm setting up a test so I can investigate it a bit
further. But thanks for the tip, it's a nice workaround.

@aldo: I also use the default capistrano recipe, so that can't be the
actual problem. Can I ask why you include the sprockets gem through
source in your Gem file? sprockets should already be loaded through
rails, or is there a bugfix in the sprockets head that causes this
problem?

@aldo: I also use the default capistrano recipe, so that can’t be the
actual problem. Can I ask why you include the sprockets gem through

source in your Gem file? sprockets should already be loaded through
rails, or is there a bugfix in the sprockets head that causes this
problem?

Yes there was a bug in the sprockets head so I should include gem through source as recommended in Rails issues list in github.

– Aldo Nievas
SATIO Internet Solutions

www.satio.com.ar
aldo@satio.com.ar

M: 54 11 3487 4900

@aldo, thanks for the info. I'll look into it. But it seems that I don't
find any problems atm.

Meanwhile, I've been playing a bit around to see what happens during the
precompilation task. (I used the task from rails head for testing, not
the buggy one included in rails 3.1).

I have noticed the 2 loops:

      config.assets.precompile.each do |path|
        env.each_logical_path do |logical_path|
            ........
        end
      end

if first starts to loop over all files that need to be precompiled. My
setting was a bit different. I had 4 extra files that needed to be
precompiled. So that meant, 4 extra loops.

In the inner loop, it starts to loop over all the assets available.

First I thought this was the error. But it seemed like it only matches
files that are ok by the result of the first loop. Not that big of a
deal, the penalty for looping multiple times over the same asset list is
pretty small.

I started to calculte the time it took to run the task localy:

real 0m31.001s
user 0m28.357s
sys 0m2.330s

So I started to tinker a bit with the config.assets.precompile. I don't
use the application.css/js so they can be removed. So came up with this:

config.assets.precompile -= [/application.(css|js)$/]
config.assets.precompile += [/^(juggler|frontend).(css|js)$/]

Using this config setting, I was able to almost remove a whole second of
the precompilation time :stuck_out_tongue:

real 0m30.606s
user 0m28.409s
sys 0m2.324s

What I did noticed, was that precompiling my biggest js manifes took a
lot af time. So I created a custom js compressor based on the jsmin gem
and used that for precompilation:

config.assets.js_compressor = JsminTransformer.new

The result was tremendous:

real 0m22.895s
user 0m20.884s
sys 0m2.008s

It takes almost 8 seconds less to precompile using jsmin instead of
uglifier!

Since I was impressed, I retook a look at the 2 loops we encountered
earlier. It seems like env.each_logical_path always calls up the same
assets. But why do all the checks over again (see the sprockets base
class)? Either way you put it, it's a IO penalty, when you check for
files.

So I changed the rake task a bit. I call each_logical_path once and
store the result (an Enumerable object) in a variable that I'll loop:

asset_list = env.each_logical_path

config.assets.precompile.each do |path|
    asset_list.each do |logical_path|
        ....
     end
end

So I ran the altered precompile task again and got this result:

real 0m18.404s
user 0m17.211s
sys 0m1.158s

Again, almost 4 seconds off the clock!

As a final test, I wanted to see if limiting the asset list while the
script ran had some performance gain. So I tried to remove the assets
that where already precompiled:

asset_list = env.each_logical_path.to_a
remove_list = []
config.assets.precompile.each do |path|
    asset_list -= remove_list
    remove_list = []

    asset_list.each do |logical_path|
        ....

        if asset = env.find_asset(logical_path)
      ...

            remove_list << logical_path
        end
     end
end

The end result was a bit slim, but still:

real 0m17.772s
user 0m16.548s
sys 0m1.189s

Now that I altered the deploy recipe to use my altered precompile task,
everything seems to be in order, except the high cpu usage during the
rake task. But this is due to I/O I guess while copying the files
around.
At this point, I think I'll use my custom recipe, and precompile the
localy like Jim suggested, so that I can avoid the the high load on the
production machine.

At this point, I'm still curious of other findings that improve the
precompilation of assets. So any info is welcome.

I'm actually getting this same slowdown issue but both on my
production and local machines. The asset precompilation task takes
about 10 minutes to complete, with the same setup using jammit and
manually generating the hashes for the assets (using asset_hash) took
much less than a minute.
There's definitely something wrong somewhere but I haven't been able
to figure out what exactly is causing the slowdown.
Any hints on how I could debug this issue?

I experience the same.
Running rake assets:precompile on my local machine takes about a minute.
Running it on my production machine (which is MUCH faster), takes
forever. Over 20 minutes.
Is there a specific Sprockets commit I can point to in my gemfile to
solve this issue? Otherwise I need to locally generate the assets and
then upload them to my server.

This issue seems to have been fixed with 3.1.1, at least for me I was able to deploy using capistrano’s deploy/assets additions, it worked well, and the assets were compiled on the server pretty quickly. Whereas with 3.1.0 it never completed the compile on the server.

It must be linked to Capistrano.
I did a fresh checkout on the same server. No issues at all.
When I run 'rake assets:precompile' from a capistrano deployed release,
it takes 20 minutes. This is on the same server, using bundler.

@Jim I am also using Rails 3.1.1, so that can't be it.

Or it is caused by the directory structure used by capistrano. Or the
way bundler is used in capistrano is causing a difference in a gem
version.

I am not sure where to look anymore.

*FINALLY!*

Okay, so when setting, config.assets.compress = true, this uses a
JavaScript (V8) runtime, right?
Somewhere in the compressor, your whole Rails app is being loaded
(multiple times!). In my case, this include features/**/*.rb
These Ruby files have some references to the constant Capybara. Because
I don't have the Capybara gem available on my server (it's in the 'test'
group in Gemfile), Rails will be doing its magic constant missing
trickery, which is crazy expensive.
This brought the total time to run the 'rake assets:precompile' task to
20 minutes! I could reproduce it locally by removing the Capybara gem

I really hope Google will index this well, and nobody has to spend
another full day tracing this bug.
Now let's fix this!

By the way, here is some output from the process sampler:

1 rb_gc_mark (in ruby) + 55 [0x100039d47]
    1457 Thread_33491: SamplerThread
      1457 thread_start (in libsystem_c.dylib) + 13 [0x7fff8bdf8b75]
        1457 _pthread_start (in libsystem_c.dylib) + 335
[0x7fff8bdf58bf]
          1457 _ZN2v88internalL11ThreadEntryEPv (in v8.bundle) + 85
[0x102dd4d75]
            1457 v8::internal::SamplerThread::Run() (in v8.bundle) + 55
[0x102dd55c7]
              1457
v8::internal::RuntimeProfilerRateLimiter::SuspendIfNecessary() (in
v8.bundle) + 120 [0x102dedbc8]
                1457 semaphore_wait_trap (in libsystem_kernel.dylib) +
10 [0x7fff8d2f96b6]

Total number in stack (recursive counted multiple, when >=5):
        27300 rb_gc_mark (in ruby) + 190 [0x100039dce]
        12110 st_foreach (in ruby) + 98 [0x1000878a2]
        11346 mark_entry (in ruby) + 12 [0x100039eec]
        5576 gc_mark_children (in ruby) + 124 [0x10003959c]
        5312 gc_mark_children (in ruby) + 1778 [0x100039c12]
        5181 gc_mark_children (in ruby) + 1524 [0x100039b14]
        4558 gc_mark_children (in ruby) + 1769 [0x100039c09]
        4017 rb_call (in ruby) + 188 [0x1000237ac]
        3100 rb_call0 (in ruby) + 3267 [0x1000233c3]
        2648 eval_call (in ruby) + 347 [0x10002d95b]
        2186 gc_mark_children (in ruby) + 1386 [0x100039a8a]
        1461 rb_eval (in ruby) + 525 [0x10001e95d]
        1257 rb_yield_0 (in ruby) + 1135 [0x10002174f]
        1014 eval_iter (in ruby) + 1029 [0x10002e1c5]
        1010 eval_fcall (in ruby) + 330 [0x10002d69a]
        926 rb_call0 (in ruby) + 3436 [0x10002346c]
        775 gc_mark_children (in ruby) + 1796 [0x100039c24]
        748 mark_keyvalue (in ruby) + 25 [0x100039ec9]
        496 blk_mark (in ruby) + 50 [0x100017622]
        495 gc_mark_children (in ruby) + 1993 [0x100039ce9]
        480 rb_ary_each (in ruby) + 46 [0x1000015ce]
        430 rb_eval (in ruby) + 1464 [0x10001ed08]
        381 gc_mark_children (in ruby) + 1539 [0x100039b23]
        357 blk_mark (in ruby) + 76 [0x10001763c]
        357 rb_eval (in ruby) + 3331 [0x10001f453]
        335 gc_mark_children (in ruby) + 0 [0x100039520]
        294 rb_gc_mark (in ruby) + 0 [0x100039d10]
        293 eval_call (in ruby) + 461 [0x10002d9cd]
        274 eval_rescue (in ruby) + 311 [0x100024b87]
        273 rb_eval (in ruby) + 853 [0x10001eaa5]
        272 proc_invoke (in ruby) + 1181 [0x1000224fd]
        270 gc_mark_children (in ruby) + 1515 [0x100039b0b]
        264 block_pass (in ruby) + 868 [0x100029494]
        254 eval_call (in ruby) + 114 [0x10002d872]
        253 blk_mark (in ruby) + 97 [0x100017651]
        174 eval_while (in ruby) + 268 [0x1000207fc]
        174 garbage_collect (in ruby) + 79 [0x10003b00f]
        174 garbage_collect_0 (in ruby) + 200 [0x10003a138]
        174 rb_str_resize (in ruby) + 94 [0x10008e00e]
        174 ruby_xrealloc (in ruby) + 215 [0x10003b9c7]
        174 strio_extend (in stringio.bundle) + 60 [0x1001f190c]
stringio.c:757
        174 strio_putc (in stringio.bundle) + 184 [0x1001f2cb8]
stringio.c:1135
        166 rb_eval (in ruby) + 1901 [0x10001eebd]
        165 eval_slit (in ruby) + 81 [0x100024cb1]
        165 rb_eval (in ruby) + 1987 [0x10001ef13]
        143 rb_eval (in ruby) + 3144 [0x10001f398]
        141 rb_eval (in ruby) + 2205 [0x10001efed]
        133 st_foreach (in ruby) + 0 [0x100087840]
        132 rb_eval (in ruby) + 3073 [0x10001f351]
        117 eval_attrasgn (in ruby) + 397 [0x10002dc4d]
        99 eval_ensure (in ruby) + 231 [0x1000204a7]
        53 gc_mark_children (in ruby) + 642 [0x1000397a2]
        49 rb_mark_generic_ivar (in ruby) + 57 [0x100099639]
        37 rb_fast_mark_table_contains (in ruby) + 0
[0x100036eb0]
        31 mark_keyvalue (in ruby) + 17 [0x100039ec1]
        28 gc_mark_children (in ruby) + 331 [0x10003966b]
        27 gc_mark_children (in ruby) + 1556 [0x100039b34]
        24 rb_gc_mark (in ruby) + 105 [0x100039d79]
        23 mark_source_filename (in ruby) + 0 [0x100037000]
        22 rb_fast_mark_table_add (in ruby) + 0 [0x100036e90]
        14 blk_mark (in ruby) + 41 [0x100017619]
        13 gc_mark_children (in ruby) + 1473 [0x100039ae1]
        12 gc_mark_children (in ruby) + 1486 [0x100039aee]
        11 rb_fast_mark_table_add_filename (in ruby) + 0
[0x100036ef0]
        10 blk_mark (in ruby) + 121 [0x100017669]
        10 rb_gc_mark (in ruby) + 118 [0x100039d86]
        8 rb_load (in ruby) + 1476 [0x100030184]
        7 mark_entry (in ruby) + 0 [0x100039ee0]
        7 rb_f_load (in ruby) + 53 [0x1000307f5]
        5 blk_mark (in ruby) + 0 [0x1000175f0]
        5 gc_mark_children (in ruby) + 463 [0x1000396ef]

Sort by top of stack, same collapsed (when >= 5):
        semaphore_wait_trap (in libsystem_kernel.dylib) 1457
        gc_mark_children (in ruby) 399
        rb_gc_mark (in ruby) 386
        rb_fast_mark_table_heap_contains (in ruby) 224
        st_foreach (in ruby) 222
        garbage_collect_0 (in ruby) 78
        rb_fast_mark_table_contains (in ruby) 37
        mark_source_filename (in ruby) 24
        rb_fast_mark_table_add (in ruby) 22
        rb_fast_mark_table_heap_remove (in ruby) 16
        rb_fast_mark_table_add_filename (in ruby) 11
        mark_entry (in ruby) 7
        blk_mark (in ruby) 5
        rb_fast_mark_table_contains_filename (in ruby) 5

good job, I suggest you file this as a bug in the rails bug list so it can be fixed

Ah, great to know what's causing the problem.

Were you suggesting that adding Capybara to the gemfile should reduce
the precompile time? I tried it and it still took 10minutes.