Concurrency problem with memcached fragments mongrel cluster

I have the following question: What happens when you memcache fragments
and you are running with a mongrel cluster. In the case when we check
read_fragment() in our action/controller and by the time Rails rechecks
to see if the fragment exists in the view, the fragment could have
possibly expired in memcache, or could have gotten deleted if you are
running some expiration sweeper of some sort. This means that none of
the instance variables or SQL was run since read_fragment() returned
true saying there is a fragment. Does this mean view rendering will
crash because it would try to create the cache block? How does one fix
this window if this is a problem? How do multiple mongrels synchronize
around this shared fragment?

This could very well happen, but if your memcache ram size is large enough and you've told it not to expire, probably won't.

But...memcache is *not* permanent storage... they make that very clear. So if you're relying on it being there long term, well... don't.


Yes this could happen as well, but if your cache size is large enough, it’ll happen probably about one in a million times. Can’t think of a way around this.


My understanding is that memcached uses LRU, so calling read_fragment() will push the fragment to the top of he used list. This should give it a good chance of surviving until you use it in the view. However, you should check out extended_fragment_cache [1]. Though I have never used it, I believe when calling read_fragment(), it saves the cached fragment in an in-process cache. This would in theory cut down on the number of cache queries, and possibly guarantee that the fragment exists (in the in-process cache) when you hit the view. I found this extension through Tyler Kovaks’ slides titled “Efficient Memcached Usage” [2] (check out slide 8/11).