# Can we keep validation in memcached ? **URL:** \ **Category:** rubyonrails-talk\ **Created:** [November 2, 2013, 8:16pm UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067 "2013-11-02T20:16:20Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Fahim\_Patel](https://avatars.discourse-cdn.com/v4/letter/f/c89c15/32.png) [@Fahim\_Patel](https://discuss.rubyonrails.org/u/Fahim_Patel)\ **Post date:** [November 2, 2013, 8:16pm UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/1 "2013-11-02T20:16:20Z")
Hi All , Can we validate request input data using memcache ? Actually I am facing scenario in which user uploading CSV file which can have hundreds of rows and to validate each row using ActiveRecord taking long time. I have below question:- 1. So can we cache our validation rule in memcache ? 2. If it is possible to store validation rules in memcache but can I able to perform validation check ? ---
**Author:** ![Peter\_Hickman1](https://avatars.discourse-cdn.com/v4/letter/p/8e7dd6/32.png) [@Peter\_Hickman1](https://discuss.rubyonrails.org/u/Peter_Hickman1)\ **Post date:** [November 2, 2013, 9:08pm UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/2 "2013-11-02T21:08:32Z")
Memcached is only useful for storing the responses and unless the validation rules are incredibly complicated using memcached is not going to speed anything up. Simply not the right tool for the job, the real question is why are your validation rules so slow. ---
**Author:** ![Fahim\_Patel](https://avatars.discourse-cdn.com/v4/letter/f/c89c15/32.png) [@Fahim\_Patel](https://discuss.rubyonrails.org/u/Fahim_Patel)\ **Post date:** [November 2, 2013, 10:08pm UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/3 "2013-11-02T22:08:36Z")
Thanks Peter to reply. Actually my table consist more than 15 columns and each column have lots of validation like presence validation,format validation. Major issue is user will upload CSV file which can consist hundred’s of rows.Through active record validating each row consuming lots of time. That’s why I am thinking to move validation rule to cache level. It will fast process. Is this can be possible to validate data on cache level ? ---
**Author:** ![Fahim\_Patel](https://avatars.discourse-cdn.com/v4/letter/f/c89c15/32.png) [@Fahim\_Patel](https://discuss.rubyonrails.org/u/Fahim_Patel)\ **Post date:** [November 2, 2013, 10:09pm UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/4 "2013-11-02T22:09:49Z")
Thanks Peter to reply. Actually my table consist more than 15 columns and each column have lots of validation like presence validation,format validation. Major issue is user will upload CSV file which can consist hundred’s of rows.Through active record validating each row consuming lots of time. That’s why I am thinking to move validation rule to cache level. It will fast process. Is this can be possible to validate data on cache level ? ---
**Author:** ![Josh\_Jordan](https://avatars.discourse-cdn.com/v4/letter/j/c37758/32.png) [@Josh\_Jordan](https://discuss.rubyonrails.org/u/Josh_Jordan)\ **Post date:** [November 3, 2013, 6:38pm UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/5 "2013-11-03T18:38:53Z")
Why do you think memcached is only useful for storing response data? That is simply not true. Memcache has been around longer than Rails, and is useful as a general purpose, fast, volatile data store. You can use it for anything you like, and it applies well in any case where there is deterministic output given a certain set of inputs. ---
**Author:** ![Fahim\_Patel](https://avatars.discourse-cdn.com/v4/letter/f/c89c15/32.png) [@Fahim\_Patel](https://discuss.rubyonrails.org/u/Fahim_Patel)\ **Post date:** [November 3, 2013, 6:56pm UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/6 "2013-11-03T18:56:08Z")
Thanks for reply.@josh - So you are saying that we can validate request on memcache level also. Please correct me If I am wrong ? ---
**Author:** ![Sur](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Sur](https://discuss.rubyonrails.org/u/Sur)\ **Post date:** [November 4, 2013, 8:42am UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/7 "2013-11-04T08:42:47Z")
@Fahim In my opinion memcached is not a suitable tool for the purpose you’ve mentioned. In fact it’d be waste of resources in such a case. I would go for processing that thing in background using any of the available gem for processing the background job. Showing an interim screen to the user mentioning that your uploaded file is being processed and process it in background with the help of ActiveRecord + DelayedJob. ---
**Author:** ![Tamara\_Temple](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@Tamara\_Temple](https://discuss.rubyonrails.org/u/Tamara_Temple)\ **Post date:** [November 4, 2013, 9:46am UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/8 "2013-11-04T09:46:55Z")
> @Fahim > > In my opinion memcached is not a suitable tool for the purpose you've mentioned. In fact it'd be waste of resources in such a case. I would go for processing that thing in background using any of the available gem for processing the background job. Showing an interim screen to the user mentioning that your uploaded file is being processed and process it in background with the help of ActiveRecord + DelayedJob. I tend to agree. ActiveRecord does not do well for mass insert/updates, and better to offload this task and not keep the Rails stack busy with it. I also wouldn’t hold up the user with a wait screen. I’d put a little indicator in an area that shows the processing is being done but does not restrict further use of the site, except for that new data. When the background processing is finished, the indicator can be removed. ---
**Author:** ![Fahim\_Patel](https://avatars.discourse-cdn.com/v4/letter/f/c89c15/32.png) [@Fahim\_Patel](https://discuss.rubyonrails.org/u/Fahim_Patel)\ **Post date:** [November 4, 2013, 10:32am UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/9 "2013-11-04T10:32:51Z")
@sur and @tam thanks for reply… @sur - Currently I am using rescue …I am using worker’s but still I am thinking to use memcache for fast performance…What you say can I achieve validation on cache level. @tam - I even showing processing image till process done… Friends …can i achieve fast performance through putting validation on cache level …or I have to assume that this is max performance code… > @Fahim > > In my opinion memcached is not a suitable tool for the purpose you’ve mentioned. In fact it’d be waste of resources in such a case. I would go for processing that thing in background using any of the available gem for processing the background job. Showing an interim screen to the user mentioning that your uploaded file is being processed and process it in background with the help of ActiveRecord + DelayedJob. I tend to agree. ActiveRecord does not do well for mass insert/updates, and better to offload this task and not keep the Rails stack busy with it. I also wouldn’t hold up the user with a wait screen. I’d put a little indicator in an area that shows the processing is being done but does not restrict further use of the site, except for that new data. When the background processing is finished, the indicator can be removed. ---
**Author:** ![Tamara\_Temple](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@Tamara\_Temple](https://discuss.rubyonrails.org/u/Tamara_Temple)\ **Post date:** [November 4, 2013, 10:45am UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/10 "2013-11-04T10:45:46Z")
> @sur and @tam thanks for reply.... > > @sur - Currently I am using rescue ....I am using worker's but still I am thinking to use memcache for fast performance....What you say can I achieve validation on cache level. > > @tam - I even showing processing image till process done.... > > Friends ....can i achieve fast performance through putting validation on cache level .....or I have to assume that this is max performance code…. Here’s what I don’t get: what could you possibly be caching here? If what’s happening is a user is uploading a CSV file that is going to inserted into table, what is going to be cached? ---
**Author:** ![Fahim\_Patel](https://avatars.discourse-cdn.com/v4/letter/f/c89c15/32.png) [@Fahim\_Patel](https://discuss.rubyonrails.org/u/Fahim_Patel)\ **Post date:** [November 4, 2013, 12:02pm UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/11 "2013-11-04T12:02:47Z")
Thanks @tamouse for reply. Actually my table consist more than 15 columns and each column have lots of validation like presence validation,format validation. Major issue is user will upload CSV file which can consist hundred’s of rows.Through active record validating each row consuming lots of time. That’s why I am thinking to move validation rule to cache level. It will fast process. Hope you understood situation … - show quoted text - > @sur and @tam thanks for reply… > > @sur - Currently I am using rescue …I am using worker’s but still I am thinking to use memcache for fast performance…What you say can I achieve validation on cache level. > > @tam - I even showing processing image till process done… > Friends …can i achieve fast performance through putting validation on cache level …or I have to assume that this is max performance code…. Here’s what I don’t get: what could you possibly be caching here? If what’s happening is a user is uploading a CSV file that is going to inserted into table, what is going to be cached? - show quoted text - > To view this discussion on the web visit [https://groups.google.com/d/msgid/rubyonrails-talk/7902ebce-ee88-42f7-bd2c-cbe9b7374d18%40googlegroups.com](https://groups.google.com/d/msgid/rubyonrails-talk/7902ebce-ee88-42f7-bd2c-cbe9b7374d18%40googlegroups.com). - show quoted text - ---
**Author:** ![Sur](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Sur](https://discuss.rubyonrails.org/u/Sur)\ **Post date:** [November 4, 2013, 12:14pm UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/12 "2013-11-04T12:14:54Z")
> Thanks @tamouse for reply. Actually my table consist more than 15 columns and each column have lots of validation like presence validation,format validation. Major issue is user will upload CSV file which can consist hundred's of rows.Through active record validating each row consuming lots of time. > > That's why I am thinking to move validation rule to cache level. It will fast process. If I understood you correctly, Validation rule resides inside a Ruby class i.e. a class inheriting ActiveRecord::Base, which is already there in memory to even be functional. It doesn't make any other sense to cache this class (or rules inside the class). Further, Rails automatically caches all the classes in production environment. Do you correctly understand this ? In short, your say "I am thinking to move validation rule to cache level" itself is incorrect because the validation is already in memory and is already cached, you can not speed it up further by saying you want to cache it. And for processing millions of rows with 15 columns each row, by any chance it surely is going to be a slow procedure even with memcached (unless you want to put a super computer in place for this task)... and the practical solution is to move this to background job. or, I didn't understand you at all. regards, Sur crimson9.com ---
**Author:** ![Fahim\_Patel](https://avatars.discourse-cdn.com/v4/letter/f/c89c15/32.png) [@Fahim\_Patel](https://discuss.rubyonrails.org/u/Fahim_Patel)\ **Post date:** [November 4, 2013, 12:28pm UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/13 "2013-11-04T12:28:50Z")
Thanks @sur for reply… Yes you understood correctly my issue, and I even understood your solution… let me cross check your solution and will update here. If I understood you correctly, Validation rule resides inside a Ruby class i.e. a class inheriting ActiveRecord::Base, which is already there in memory to even be functional. It doesn’t make any other sense to cache this class (or rules inside the class). Further, Rails automatically caches all the classes in production environment. Do you correctly understand this ? In short, your say “I am thinking to move validation rule to cache level” itself is incorrect because the validation is already in memory and is already cached, you can not speed it up further by saying you want to cache it. And for processing millions of rows with 15 columns each row, by any chance it surely is going to be a slow procedure even with memcached (unless you want to put a super computer in place for this task)… and the practical solution is to move this to background job. or, I didn’t understand you at all. regards, Sur [crimson9.com](http://crimson9.com/) ---
**Author:** ![Tamara\_Temple](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@Tamara\_Temple](https://discuss.rubyonrails.org/u/Tamara_Temple)\ **Post date:** [November 4, 2013, 10:48pm UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/14 "2013-11-04T22:48:23Z")
> Thanks @tamouse for reply. Actually my table consist more than 15 columns and each column have lots of validation like presence validation,format validation. Major issue is user will upload CSV file which can consist hundred's of rows.Through active record validating each row consuming lots of time. > > That's why I am thinking to move validation rule to cache level. It will fast process. How do you move a validation rule to memcache? ---
**Author:** ![Fahim\_Patel](https://avatars.discourse-cdn.com/v4/letter/f/c89c15/32.png) [@Fahim\_Patel](https://discuss.rubyonrails.org/u/Fahim_Patel)\ **Post date:** [November 5, 2013, 5:15am UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/15 "2013-11-05T05:15:31Z")
@tam - I am even not sure how can experiment it. Collecting data and studying in progress.@sur - I am still cross checking… If you provide any reference link or data to proof that validation already get cached. Friends and @sur please clear me on below question- From rails app, when user insert data then each time data hitted on database for validation or validation is present in cache ? If validation get cached then we don’t need move validation on cache. ---
**Author:** ![Dheeraj\_Kumar](https://avatars.discourse-cdn.com/v4/letter/d/439d5e/32.png) [@Dheeraj\_Kumar](https://discuss.rubyonrails.org/u/Dheeraj_Kumar)\ **Post date:** [November 5, 2013, 5:23am UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/16 "2013-11-05T05:23:55Z")
This is a great use case for a background job. Have you tried Resque? If I have a user uploading thousands of rows, all of which need to be validated and processed/saved, I’d do it like this: 1. Let the user upload the file, save it somewhere. Trigger a Resque job. 2. Read the file one or more rows at a time, make a ActiveRecord or ActiveModel object out of them and run the validators. Collate the errors and include the row number. 3. If all the rows were valid, continue processing or saving. 4. If not, send an email to the uploader and/or show in his dashboard the failure message, along with a list of all errors with row numbers. I’ve used this method in our app, and is working out great for our users. ---
**Author:** ![Fahim\_Patel](https://avatars.discourse-cdn.com/v4/letter/f/c89c15/32.png) [@Fahim\_Patel](https://discuss.rubyonrails.org/u/Fahim_Patel)\ **Post date:** [November 5, 2013, 5:54am UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/17 "2013-11-05T05:54:51Z")
@dheeraj - thanks for reply.I am using resque & redis. You exactly got use case . can you please give answer for below point:- 1. From rails app, when user insert data then each time data hitted on database for validation or validation is present in cache ? ---
**Author:** ![Dheeraj\_Kumar](https://avatars.discourse-cdn.com/v4/letter/d/439d5e/32.png) [@Dheeraj\_Kumar](https://discuss.rubyonrails.org/u/Dheeraj_Kumar)\ **Post date:** [November 5, 2013, 6:04am UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/18 "2013-11-05T06:04:18Z")
First validate the data, and then insert it to the database. Till then, it’s read from the file and still in-memory. I’m not bringing the cache into the picture at all. ---
**Author:** ![Sur](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Sur](https://discuss.rubyonrails.org/u/Sur)\ **Post date:** [November 5, 2013, 6:07am UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/19 "2013-11-05T06:07:42Z")
Well, this is a classic use case when we jump too early on Rails not understanding the Ruby (or programming in general) to at least a decent level… let alone the meta-programming stuff. One should get a good hold of Ruby at least to a level where s/he understands how the memory mapping is working behind the scenes… otherwise you can end up writing the piles of junk which is eating up all your server’s memory. ---
**Author:** ![Jordon\_Bedwell](https://avatars.discourse-cdn.com/v4/letter/j/e480ec/32.png) [@Jordon\_Bedwell](https://discuss.rubyonrails.org/u/Jordon_Bedwell)\ **Post date:** [November 5, 2013, 6:20am UTC](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067/20 "2013-11-05T06:20:28Z")
Preach on about this bullshit when Ruby isn't primarily used for web applications and when Ruby allows my system to reclaim memory, until then lets leave the symbol argument bullshit out of this. Even if they are constants they can lead to trouble if you just blindly state they are better without explaining the dangers in a web application, or people will repeat history, history people like me thought was already common knowledge. [Next page](https://discuss.rubyonrails.org/t/can-we-keep-validation-in-memcached/68067.md?page=2)