I can not use find_each to write huge csv because find_in_batch ignores order scope.

rubyonrails-core
1

This is a feature request.

I can not use ActiveRecord::Batches#find_each to write huge csv because find_in_batch ignores order scope.

find_in_batch’s note say:

# NOTE: It's not possible to set the order. That is automatically set to
# ascending on the primary key ("id ASC") to make the batch ordering
# work.

OK. But I want to use this method to output csv too, because I want to cut memory space.

I have an idea to solve this:

  • find_each and find_by_batch method receive :use_order_scope option. How about this?
2

Hi,

find_each depends on sequential ids to query for batches. It does this to get batches:


2.0.0-p353 :024 > Catalog.find_each { }

Catalog Load (4.3ms)  SELECT `catalogs`.* FROM `catalogs` ORDER BY `catalogs`.`id` ASC LIMIT 1000

Catalog Load (2.6ms)  SELECT `catalogs`.* FROM `catalogs` WHERE (`catalogs`.`id` > 2088) ORDER BY `catalogs`.`id` ASC LIMIT 1000

Catalog Load (2.4ms)  SELECT `catalogs`.* FROM `catalogs` WHERE (`catalogs`.`id` > 3088) ORDER BY `catalogs`.`id` ASC LIMIT 1000

If you’re ordering on something else there’s no way to pick out subsequent batches as the ids are not ordered.

You could use OFFSET and LIMIT but it’s going to be hard on the database.

Therefore the only safe and fast option is supported.