Server Monitoring with Monit - OpenDemocracyManitoba/winnipegelection GitHub Wiki

Researched by Jeffrey Fulton.

Monit

Easy, proactive monitoring of processes, programs, files, directories, filesystems and hosts.

Official Monit Website.

Types of Monitoring Available

• Site is live.
• Specific pages with specific content (similar to Capybara testing).
• Processes. i.e. Web server, Database (expect query result).
• Restart processes in the event of failure; over n cycles.
• System resources. i.e. CPU, memory, storage. Set thresholds for alert.
• Rotate log files if too large.
• Security: Test file/directory attributes for tampering?

Available Alerts

• Email: Configure alert only if n cycles have persisted.
• Web portal on our server.
• Manually check logs.

Installation and Configuration

1. Install Monit (Ubuntu):

• sudo aptitude install monit

2. Configure:

• Add configurations to /etc/monit/monitrc
• Full documentation: http://mmonit.com/monit/documentation/monit.html
• Example configurations in /etc/monit/monitrc (commented) and at http://mmonit.com/monit/ (follow presentation by pressing left arrow)

4. Configure Gmail by adding the following to /etc/monit/monitrc:
   set mailserver smtp.gmail.com port 587
username [[email protected]] password [user_password]
using tlsv1
with timeout 30 seconds
5. Enable Google Apps external access:

• Visit https://accounts.google.com/DisplayUnlockCaptcha
• Log in as user from above configuration.
• Click 'Continue' button

7. Reload Monit:

• sudo monit reload

Example Monit Config File

This is an example Monit configuration with the following enabled:

• "Site is live" monitoring.
• HD monitoring. Warning at 80% full.
• Memory monitoring. Warning at 80% full.
• CPU monitoring. Warning at 90% load.
• Process monitoring for Nginx and Postgres
• Gmail Servers for Email Alerts

The example /etc/monit/monitrc:

# Monit config file with comments removed for clarity

# Default values from /etc/monit/monitrc:    
set daemon 120			# check services at 2 minute intervals

set logfile /var/log/monit.log
set idfile /var/lib/monit/id
set statefile /var/lib/monit/state

set eventqueue
        basedir /var/lib/monit/events # set the base directory where events will be stored
        slots 100                     # optionally limit the queue size

set httpd port 2812 and
        allow admin:monit      # require user 'admin' with password 'monit'


# Added by Jeffrey:

# Use gmail servers to send alert emails
set mailserver smtp.gmail.com port 587 
        username [[email protected]] password [password] 
  using tlsv1 
  with timeout 30 seconds 
 
# Emails addresses to send alerts to:
set alert [email protected]
set alert [email protected] 
 
# "Site is live" monitoring 
check host 107.170.153.40 with address 107.170.153.40 
        if failed 
                port 80 protocol http 
        then alert 
 
# HD monitoring. Warning at 80% full. 
check filesystem rootfs with path /dev/vda 
        if space usage > 80% then alert 
 
# Memory monitoring. Warning at 80% full. 
# CPU monitoring. Warning at 90% load. 
check system $HOST 
        if memory usage > 80% then alert 
        if cpu usage (user) > 90% then alert 
        if cpu usage (system) > 90% then alert 
        if cpu usage (wait) > 90% then alert

# Restart nginx if process dies or goes crazy spawning child processes
check process nginx with pidfile /var/run/nginx.pid
        start program = "/etc/init.d/nginx start"
        stop program = "/etc/init.d/nginx stop"
        if children > 250 then restart
        if 5 restarts within 5 cycles then timeout

## 
# Restart postgresql if process dies or is unreachable.
# note: Pidfile needs to be configured because it includes postgresql version
# number and can change.
##
check process postgresql with pidfile <%= postgresql_pid %>
        start program = "/etc/init.d/postgresql start"
        stop program = "/etc/init.d/postgresql stop"
        if failed host localhost port 5432 protocol pgsql then restart
        if 5 restarts within 5 cycles then timeout

# Separate configurations into separate files in etc/monit/conf.d/
include /etc/monit/conf.d/*