I have a rails 6.0 app running on Heroku. I use Heroku scheduler to schedule a Sidekiq job that sends a daily email to ~ 20,000 users. The email is custom for each user though there are common components that could be shared/cached for efficiency. I have two Sidekiq workers with -c 3.
I’ve tried several strategies already and none of them feel right, my Google-Fu fails me and so I thought I’d ask here. How would you approach this?
Two strategies that I’ve tried:
Single job for everything
- scheduler runs a rake tasks that schedules OneBigJob
- OneBigJob iterates through 20,000 users, generates an email for each and sends it.
Lots of little jobs
- scheduler runs a rake tasks that schedules MainJob
- MainJob iterates through 20,000 users, generates a LittleJob for each user
- LittleJob generates an email and sends it
Strategy 1 has some efficiencies because some of the content is shared but it is running up against memory limits in my workers.
I am new-ish to ActiveJob & Sidekiq and wonder if there are any downsides to scheduling 20,000 jobs.
How would you approach this?