Monitoring - rselk/sidekiq GitHub Wiki
Sidekiq comes with a Sinatra application that can display the current state of a Sidekiq installation.
Rails 3 and 4
Add sinatra
(and sprockets
if you are on Rails 3.0) to your Gemfile
# if you require 'sinatra' you get the DSL extended to Object
gem 'sinatra', '>= 1.3.0', :require => nil
Add the following to your config/routes.rb
:
require 'sidekiq/web'
mount Sidekiq::Web => '/sidekiq'
You can now generate a link to the Sidekiq web UI with the sidekiq_web
url/path helpers:
<%= link_to 'Monitoring', sidekiq_web_path %>
Security
In a production application you'll likely want to protect access to this information. You can use the constraints feature of routing (in the config/routes.rb file) to accomplish this:
Devise
Allow any authenticated User
# config/routes.rb
authenticate :user do
mount Sidekiq::Web => '/sidekiq'
end
Same as above but also ensures that User#admin?
returns true
# config/routes.rb
authenticate :user, lambda { |u| u.admin? } do
mount Sidekiq::Web => '/sidekiq'
end
Authlogic
# lib/admin_constraint.rb
class AdminConstraint
def matches?(request)
return false unless request.cookies['user_credentials'].present?
user = User.find_by_persistence_token(request.cookies['user_credentials'].split(':')[0])
user && user.admin?
end
end
# config/routes.rb
require "admin_constraint"
mount Sidekiq::Web => '/sidekiq', :constraints => AdminConstraint.new
Restful Authentication or Sorcery
Checks a User
model instance that responds to admin?
# lib/admin_constraint.rb
class AdminConstraint
def matches?(request)
return false unless request.session[:user_id]
user = User.find request.session[:user_id]
user && user.admin?
end
end
# config/routes.rb
require 'sidekiq/web'
require 'admin_constraint'
mount Sidekiq::Web => '/sidekiq', :constraints => AdminConstraint.new
Custom External Authentication
class AuthConstraint
def self.admin?(request)
return false unless (cookie = request.cookies['auth'])
Rails.cache.fetch(cookie['user'], :expires_in => 1.minute) do
auth_data = JSON.parse(Base64.decode64(cookie['data']))
response = HTTParty.post(Auth.validate_url, :query => auth_data)
response.code == 200 && JSON.parse(response.body)['roles'].to_a.include?('Admin')
end
end
end
# config/routes.rb
constraints lambda {|request| AuthConstraint.admin?(request) } do
mount Sidekiq::Web => '/admin/sidekiq'
end
Standalone
Here's an example config.ru
for booting Sidekiq::Web in your choice of Rack server:
require 'sidekiq'
Sidekiq.configure_client do |config|
config.redis = { :size => 1 }
end
require 'sidekiq/web'
run Sidekiq::Web
You can mount sidekiq to existing Rack (Sinatra) application as well:
require 'your_app'
require 'sidekiq/web'
run Rack::URLMap.new('/' => Sinatra::Application, '/sidekiq' => Sidekiq::Web)
Or if you want to use Sidekiq::Web with Rakefiles:
task :monitor do
# optional: Process.daemon (and take care of Process.pid to kill process later on)
require 'sidekiq/web'
app = Sidekiq::Web
app.set :environment, :production
app.set :bind, '0.0.0.0'
app.set :port, 9494
app.run!
end
If you do everything right, you should see this in your browser:
Standalone with GitHub OAuth
This configuration will allow access only to members of your GitHub organization. Start by creating a new OAuth application. You will also need to create a secure session cookie:
openssl rand -base64 48
require 'sinatra_auth_github'
module Sidekiq
class Web
use Rack::Session::Cookie, :secret => ENV['RACK_SESSION_COOKIE']
set :github_options, {
:scopes => "user",
:client_id => ENV['GITHUB_KEY'],
:secret => ENV['GITHUB_SECRET']
}
register Sinatra::Auth::Github
before do
authenticate!
github_organization_authenticate!(ENV['GITHUB_ORG'])
end
get '/logout' do
logout!
end
end
end
Standalone with Basic Auth
# this code goes in your config.ru
require 'sidekiq'
Sidekiq.configure_client do |config|
config.redis = { :size => 1 }
end
require 'sidekiq/web'
map '/sidekiq' do
use Rack::Auth::Basic, "Protected Area" do |username, password|
username == 'sidekiq' && password == 'sidekiq'
end
run Sidekiq::Web
end
Nagios
Below is a collection of nagios checks that includes check_sidekiq_queue script, which validates that a given queue depth is within a particular range. It's a simple shell script that uses redis-cli command line tool, and does not have any dependency on ruby.
https://github.com/wanelo/nagios-checks
Scout
The Sidekiq Monitor plugin for Scout, a hosted server monitoring service, reports key metrics like enqueued, processed, and failed jobs. Triggers can be configured to alert when the metrics reach specified thresholds. A Scout account is required to use the plugin.
Monitoring Queue Backlog
At The Clymb, we use a simple HTTP endpoint with Pingdom to check the size of our Sidekiq 'default' queue backlog. Put this in config/routes.rb
:
require 'sidekiq/api'
match "queue-status" => proc { [200, {"Content-Type" => "text/plain"}, [Sidekiq::Queue.new.size < 100 ? "OK" : "UHOH" ]] }
Now when you hit http://example.com/queue-status, the body of the response will be either 'OK' or 'UHOH'. We have a Pingdom check every minute which fires off an email if the response == 'UHOH'.
Monitoring Queue Latency
Using a custom end-point
If you throw a lot of jobs into the queue, you can get false positives when monitoring the queue backlog. Instead, monitor the queue latency. Queue latency is the difference between when the oldest job was pushed onto the queue versus the current time. This code will check that jobs don't spend more than 30 seconds enqueued. Put this in config/routes.rb
:
require 'sidekiq/api'
match "queue-latency" => proc { [200, {"Content-Type" => "text/plain"}, [Sidekiq::Queue.new.latency < 30 ? "OK" : "UHOH" ]] }
Now when you hit http://example.com/queue-latency, the body of the response will be either 'OK' or 'UHOH'.
Using the built-in dashboard
Sidekiq provides a JSON formatted dashboard at http://example.com/sidekiq/dashboard/stats. You get this :
{
"sidekiq": {
"processed": "12345",
"failed": "56",
"busy": "25",
"enqueued": "178",
"scheduled": "0",
"retries": "0",
"default_latency": "12",
},
"redis": {
"connected_clients": "120",
"uptime_in_days": "35",
"used_memory_human": "602.31M",
"used_memory_peak_human": "1.01G"
}
}