Active Jobs, Sidekiq, Redis - GalacticPlastic/ironhack GitHub Wiki

Quicklinks:

Local Development

Certain delayed endpoints and services may not work without a Sidekiq server (see Active Jobs).

A local Redis instance must be running for the Sidekiq server to run.

Run Redis.

Run in foreground (dedicated terminal / command line):

$ redis-server
# oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
# Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=93261, just started
# Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 5.0.4 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 93261
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

# Server initialized
* DB loaded from disk: 0.000 seconds
* Ready to accept connections

Or run in background:

redis-server --daemonize yes
# oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
# Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=93493, just started
# Configuration loaded

Check if the background process started or not:

$ ps aux | grep redis-server
computername          1502   0.0  0.0  4338472    680   ??  S    12:51PM   0:15.21 /usr/local/opt/redis/bin/redis-server 127.0.0.1:6379  
computername         93569   0.0  0.0  4268056    816 s002  S+   11:12AM   0:00.00 grep redis-server
computername         93261   0.0  0.0  4311848   2136 s002  S    11:11AM   0:00.05 redis-server *:6379

Run Sidekiq queuing server.

To run Sidekiq, you will need to open a terminal, navigate to your application's directory, and start the Sidekiq process exactly as you would start a web server for the application itself. The application will then add jobs to Redis for the Sidekiq process to find and process.

$ cd path/to/your/app
$ bundle exec sidekiq
INFO: Booting Sidekiq 5.1.3 with redis options {:url=>"redis://localhost:6379/12", :id=>"Sidekiq-server-PID-97510"}

         m,
         `$b
    .ss,  $$:         .,d$
    `$$P,d$P'    .,md$P"'
     ,$$$$$bmmd$$$P^'
   .d$$$$$$$$$$P'
   $$^' `"^$$$'       ____  _     _      _    _
   $:     ,$$:       / ___|(_) __| | ___| | _(_) __ _
   `b     :$$        \___ \| |/ _` |/ _ \ |/ / |/ _` |
          $$:         ___) | | (_| |  __/   <| | (_| |
          $$         |____/|_|\__,_|\___|_|\_\_|\__, |
        .d$$                                       |_|

INFO: Running in ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-darwin17]
INFO: See LICENSE and the LGPL-3.0 for licensing details.
INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
INFO: Starting processing, hit Ctrl-C to stop

When the command executes, you will see the above message that sidekiq has started. This window must remain open and running for sidekiq to continue to function. If you prefer, you can run sidekiq as a daemon with the -d or --daemon flag when starting the process.

Or run in background:

bundle exec sidekiq -e 'development' &

Check if the background process started or not:

$ ps aux | grep sidekiq
computername         98976   0.0  0.0  4268056    808 s002  S+   11:24AM   0:00.00 grep sidekiq

Alternately, you can configure Sidekiq to run inline for local development by adding the following to config / environments / development.rb:

# Run Sidekiq tasks synchronously so that Sidekiq is not required in Development
require 'sidekiq/testing/inline'

In some cases, however, it could be desirable to have an application run with Sidekiq processing jobs in the background to better match a live, production environment.

Seed database with dummy data.

RAILS_ENV=development bundle exec rails db:seed

Run Rails server on localhost:3000.

RAILS_ENV=development bundle exec rails s

Open a browser and go to localhost:3000.

Rails Active Jobs

Active Job is a framework for declaring jobs and making them run on a variety of queuing backends. These jobs can be everything from regularly scheduled clean-ups, to billing charges, to mailings. Anything that can be chopped up into small units of work and run in parallel.

Active Job can be configured to work with Sidekiq.

Active Job Setup

The Active Job adapter must be set to :sidekiq or it will simply use the default :async. This can be done in config/application.rb:

class Application < Rails::Application
  # ...
  config.active_job.queue_adapter = :sidekiq
end

We can use the generator to create a new job.

rails generate job Example

This above command will create app/jobs/example_job.rb

class ExampleJob < ActiveJob::Base
  # Set the Queue as Default
  queue_as :default

  def perform(*args)
    # Perform Job
  end
end

Usage

Jobs can be added to the job queue from anywhere. We can add a job to the queue by:

ExampleJob.perform_later args

At this point, Sidekiq will run the job for us. If the job for some reason fails, Sidekiq will retry as normal.

Customizing Error Handling

Active Job does not support the full richness of Sidekiq's retry feature out of the box. Instead, it has a simple abstraction for encoding retries upon encountering specific exceptions.

class ExampleJob < ActiveJob::Base
  rescue_from(ErrorLoadingSite) do
    retry_job wait: 5.minutes, queue: :low_priority 
  end 

  def perform(*args)
    # Perform Job
  end
end

The default AJ retry scheme is 3 retries, 5 seconds apart. Once this is done (after 15-30 seconds), AJ will kick the job back to Sidekiq, where Sidekiq's retries with exponential backoff will take over.

As of Sidekiq 6.0.1 you can use sidekiq_options with your Rails 6.0.1+ ActiveJobs and configure the standard Sidekiq retry mechanism.

class ExampleJob < ActiveJob::Base
  sidekiq_options retry: 5

  def perform(*args)
    # Perform Job
  end
end

Action Mailer

Action Mailer now comes with a method named #deliver_later which will send emails asynchronously (your emails send in a background job). As long as Active Job is setup to use Sidekiq we can use #deliver_later. Unlike Sidekiq, using Active Job will serialize any activerecord instance with Global ID. Later the instance will be deserialized.

Mailers are queued in the queue mailers. Remember to start sidekiq processing that queue:

bundle exec sidekiq -q default -q mailers

To send a basic message to the Job Queue we can use:

UserMailer.welcome_email(@user).deliver_later

If you would like to bypass the job queue and perform the job synchronously you can use:

UserMailer.welcome_email(@user).deliver_now

With Sidekiq we had the option to send emails with a set delay. We can do this through Active Job as well.

Old syntax for delayed message in Sidekiq:

UserMailer.delay_for(1.hour).welcome_email(@user.id)
UserMailer.delay_until(5.days.from_now).welcome_email(@user.id)

New syntax to send delayed message through Active Job:

UserMailer.welcome_email(@user).deliver_later(wait: 1.hour)
UserMailer.welcome_email(@user).deliver_later(wait_until: 5.days.from_now)

Using Global ID

Rails's Global ID feature allows serializing full ActiveRecord objects as an argument to #perform, so that

def perform(user_id)
  user = User.find(user_id)
  user.send_welcome_email!
end

can be replaced with:

def perform(user)
  user.send_welcome_email!
end

Unfortunately this means that if the User record is deleted after the job is enqueued but before the perform method is called, exception handling is different. With regular Sidekiq, you could handle this with

def perform(user_id)
  user = User.find_by(id: user_id)

  if user
    user.send_welcome_email!
  else
    # handle a deleted user record
  end
end

With ActiveJob, the perform(user) will instead raise for a missing record exception as part of deserializing the User instance. You can work around this with:

class MyJob < ActiveJob::Base
  rescue_from ActiveJob::DeserializationError do |exception|
    # handle a deleted user record
  end

  # ...
end

Job ID

ActiveJob has its own Job ID which means nothing to Sidekiq. You can get Sidekiq's JID by using provider_job_id:

job = SomeJob.perform_later
jid = job.provider_job_id

Queue Prefixes

Active Job allows you to configure a queue prefix. Don't use environment-specific prefixes. Each environment should use a separate Redis database altogether, otherwise all of your environments will share the same retry and scheduled sets and chaos will likely ensue.

Intro to Sidekiq

Sidekiq is a framework for background job processing. It allows you to scale your application by performing work in the background. This requires 3 parts:

Client - The Sidekiq client runs in any Ruby process (typically a puma or passenger process) and allows you to create jobs for processing later. There are two ways to create a job in your application code:
```
MyWorker.perform_async(1, 2, 3)
Sidekiq::Client.push('class' => MyWorker, 'args' => [1, 2, 3])  # Lower-level generic API
```
These 2 methods are equivalent and create a Hash which represents the job, serializes that Hash to a JSON string and pushes that String into a queue in Redis. This means the arguments to your worker must be simple JSON datatypes (numbers, strings, boolean, array, hash). Complex Ruby objects (e.g. Date, Time, ActiveRecord models) will not serialize properly.
Redis (see below).
Server - Each Sidekiq server process pulls jobs from the queue in Redis and processes them. Like your web processes, Sidekiq boots Rails so your jobs and workers have the full Rails API, including Active Record, available for use. The server will instantiate the worker and call perform with the given arguments. Everything else is up to your code.

Intro to Redis

Redis provides data storage for Sidekiq. It holds all the job data along with runtime and historical data to power Sidekiq's Web UI.

See Using Redis for info about connecting to Redis.

What is Redis?

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes with radius queries and streams. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.

You can run atomic operations on these types, like appending to a string; incrementing the value in a hash; pushing an element to a list; computing set intersection, union and difference; or getting the member with highest ranking in a sorted set.

In order to achieve its outstanding performance, Redis works with an in-memory dataset. Depending on your use case, you can persist it either by dumping the dataset to disk every once in a while, or by appending each command to a log. Persistence can be optionally disabled, if you just need a feature-rich, networked, in-memory cache.

Redis also supports trivial-to-setup master-slave asynchronous replication, with very fast non-blocking first synchronization, auto-reconnection with partial resynchronization on net split.

Starting Redis

The simplest way to start the Redis server is just executing the redis-server binary without any argument.

$ redis-server
[28550] 01 Aug 19:29:28 # Warning: no config file specified, using the default config. In order to specify a config file use 'redis-server /path/to/redis.conf'
[28550] 01 Aug 19:29:28 * Server started, Redis version 2.2.12
[28550] 01 Aug 19:29:28 * The server is now ready to accept connections on port 6379
... more logs ...

In the above example Redis was started without any explicit configuration file, so all the parameters will use the internal default. This is perfectly fine if you are starting Redis just to play a bit with it or for development, but for production environments you should use a configuration file.

In order to start Redis with a configuration file use the full path of the configuration file as first argument, like in the following example: redis-server /etc/redis.conf. You should use the redis.conf file included in the root directory of the Redis source code distribution as a template to write your configuration file.

Check if Redis is Working

External programs talk to Redis using a TCP socket and a Redis specific protocol. This protocol is implemented in the Redis client libraries for the different programming languages. However to make hacking with Redis simpler Redis provides a command line utility that can be used to send commands to Redis. This program is called redis-cli.

The first thing to do in order to check if Redis is working properly is sending a PING command using redis-cli:

$ redis-cli ping
PONG

Running redis-cli followed by a command name and its arguments will send this command to the Redis instance running on localhost at port 6379. You can change the host and port used by redis-cli, just try the --help option to check the usage information.

Configuration

Sidekiq files:

config / application.rb

module ApplicationName
  class Application < Rails::Application
    # Initialize configuration defaults for originally generated Rails version.
    config.load_defaults 5.1
    config.active_job.queue_adapter = :sidekiq
    config.i18n.available_locales = [:en]

config / deploy.rb

require 'mina_sidekiq/tasks'

desc "Deploys the current version to the server."
task deploy: :remote_environment do

  deploy do
    invoke :'git:clone'
    invoke :'cp_secrets:shared'
    invoke :'deploy:link_shared_paths'
    invoke :'bundle:install'
    invoke :'rails:db_create'
    invoke :'rails:db_migrate'
    invoke :'rails:assets_precompile'
    invoke :'deploy:cleanup'

    on :launch do
      in_path(fetch(:current_path)) do
        invoke :'sidekiq:quiet'
        invoke :'sidekiq:restart'
        invoke :'whenever:update'
        invoke :'sockets:clear'
        invoke :'puma:phased_restart'
        command %{mkdir -p tmp/}
        command %{touch tmp/restart.txt}
        invoke :'puma:hard_restart'
      end
    end
  end

  # you can use `run :local` to run tasks on local machine before of after the deploy scripts
  # run(:local){ say 'done' }
end

config / routes.rb

require 'sidekiq/web'

Rails.application.routes.draw do
  authenticated :user, lambda { |u| u.admin? } do
    mount Sidekiq::Web => 'admin/sidekiq'
    mount RailsEmailPreview::Engine, at: 'admin/emails'
  end

config / schedule.rb

# Restart sidekiq server if not running
every 10.minutes do
  runner 'Monitor::SidekiqProcessSet.run'
end

config / initializers / rollbar.rb

# Enable delayed reporting (using Sidekiq)
config.use_sidekiq
# You can supply custom Sidekiq options:
config.use_sidekiq 'queue' => 'default'

config / initializers / sidekiq.rb

Sidekiq.configure_server do |config|
  config.redis = { url: 'redis://localhost:6379/12' }
end

Sidekiq.configure_client do |config|
  config.redis = { url: 'redis://localhost:6379/12' }
end