Scheduled job execution (cron-like)

rubyonrails-talk
1

As someone who is new to Ruby and Rails I am interested in feedback on my thoughts. Thanks!

Requirements:

  • Allow users to configure jobs that get executed with a custom priority and at a custom interval.
    • Example:
      • User A has a job to be executed daily at 01:00 (AM) with medium priority. In other words it does not matter whether the execution starts at 01:00:00 or 01:05:35
      • User B has a job that should run every minute with high priority but without overlap. So if the job that was started at 14:35:15 finishes at 14:36:36, the next instance should be started right away
  • If the Rails server is temporarily down and the exact point in time to start a job is missed, that job must be started as soon as Rails is up again. Example: A job is supposed to start at 01:00:00, but the server is down due to a VM/Docker restart. So when Rails is up again at 01:00:35 the job must be started asap.
  • The actual job logic is different between jobs, but they are all “defined” upfront. So the user can choose one job type and have it run with custom parameters.
  • Active-active clustering will likely be needed at some point

Hope that makes sense!

In the past and with another technology I had implemented this kind of logic with a single scheduling thread and multiple execution threads (sometimes one per priority, sometimes one per job, depending on the exact requirements).

The scheduling thread ran every minute (sufficiently granular here) and for each job determined the next execution date/time. That time stamp was written to a database. If the current time was equal/greater then this next execution time, the job would be started and the timestamp for the next execution be updated. There were a few more subtleties, but I think they are not important here.

Questions:

  • Having looked at ActiveJob and Sidekiq, my current feeling is that they are not a perfect fit for me. But obviously I never worked with either and the somewhat introductory documentation I have gone through may have created a wrong impression. What is your take here?
  • I am currently leaning towards replicating the aforementioned database-based logic and then trigger ActiveJobs using the built-in, RAM-based backend from Rails. A restart of the job would be handled with my custom logic. Does that make sense to you?
  • Are there other directions you would recommend that I explore?

Again, thanks a lot for your time!

2

I would suggest you don’t reinvent the wheel. Since you mentioned Sidekiq, did you check sidekiq-cron?

At work we use the cron jobs framework provided by the infrastructure software (Nomad) and I added a few classes to configure the cron job with a YAML file like this:

---

myjob:
  cron: 0 * * * *
  command: bundle exec rails my_ns:my_task
  ...

If you’re using postgres, you can try good_job, I’ve tried it before and I love it.

1 Like
3

Thanks!

For now I want to avoid Sidekiq. It requires Redis and as a separate server that comes with considerable complexity, esp. for deployment and operations.

At the end of the day all I need is to run a certain piece of code once every x seconds.

My assumption was that I can relatively easily do something like this within(!) my Rails application:

# This is pseudo code

# File: `config/initializers/scheduler.rb`
# Purpose: Start separate thread that gets executed every 10 seconds
# and runs until the application stops
#
# (No idea if this is the right place, but so far I was not able to
# find more information for my use-case. Of course that does not 
# mean it does no exist, and I am happy to be guided into
# the right direction.)
Thread.new(MyTask, { interval => 10, unit => :seconds } )

# File: `lib/my_task.rb`
# Purpose: Check if actual job needs to be enqueud
class MyTask
  def initialize
     # Enqueue ActiveJob to default implementation,
    # not using Sidekiq or other framework
  end
end
4

Would whenver work for you? It’s basically just some syntactical goodness that generates crontabs to run stuff in your app but I’ve used it a few times and it works great. If that’s not the right fit for you needs check out the “Scheduling” section of the Ruby Toolbox.

5

Thanks for the hint. Unfortunately whenever does not fit my needs.

For the time being I have decided to use puma-rufus-scheduler (GitHub - javierav/puma-rufus-scheduler: Puma plugin to run Rufus Scheduler). It seems to be a perfect fit for my requirements.

6

I’m having trouble implementing a job scheduling feature where users can set custom priorities and intervals. I need User A to run a daily job at 01:00 AM with medium priority, while User B needs a high-priority job that runs every minute without overlaps. I want missed jobs to automatically trigger if the Rails server goes down and then restarts.

7

I am not sure if this is a premium feature or not, but I believe you can schedule Sidekiq jobs to run at a certain datetime. I may be mis-attributing from back when I used to use DelayedJob. I know I have used one of these to do just this. Benefit of this approach is that it would survive a server failure, as long as your job queue is in something durable, unlike a memory-only configured Redis. You could also make the end of one scheduled job set up the next one, too, so the loop would be gated by success of the last job. That may not be a feature, depending on your design.

Walter