Hi,
I am currently reimplementing how my project’s data collection. The project itself uses rails, and we collect a lot of data from APIs and then store them to the database. We are dealing with ten of thousands sets of data per couple minutes being processed. The challenges here is that there are a many associations with unique constraints on each model to represent each set of data.
To by pass the unique constraints, I have been using find_or_initialize_by the unique indices and then update but this result in many database calls. I have attempted to use upsert gem, but there are problems with hstore and array data and it also does not support associations and callbacks, which I end up dealing with lot of hash and array to rebuild the associations. I have also attempted to use manually written psql query to insert HABTM associations in a single database commit for a batch of associations.
All these attempts from above seems to result in code that is not maintainable, although some approaches might result in less database calls. I want gather your thoughts from the Rails community about how to maximize performance using the Rails way to deal with large amount of data upsert.
Thanks,
