Nagios Slack Notifications - KeegMitch/Operations-Engineering-group-c GitHub Wiki

Issues page here

Set up the Slack plugin for Nagios

  1. Install the Perl modules libwww-perl libcrypt-ssleay-perlby adding to install.pp in the nagios module. Apply using the puppet agent command.
 package { "libwww-perl":
                ensure => present,
        }

 package { "libcrypt-ssleay-perl":
                ensure => present,
        }
  • I've double checked that they're installed using dpkg -l

image

  1. Generate a token from Slack
  • On the Operations Engineering 2 workspace on the Slack app, there is an App menu, click Add apps and you should see Nagios there

image

image

  • Click on the Nagios app which should direct you to the Slack app directory, click Add to Slack, then Add Nagios Intergration From there this will give you Setup instructions and Intergration settings with the token that's generated. Don't forget to save the settings!
  1. Execute the following commands:

I used the command inside this directory for reference: /etc/puppetlabs/code/modules/nagios/files

sudo wget https://raw.github.com/tinyspeck/services-examples/master/nagios.pl
sudo cp nagios.pl /usr/lib/nagios/plugins/slack_nagios.pl
sudo chmod 755 /usr/lib/nagios/plugins/slack_nagios.pl

Edit the slack_nagios.pl file to set the values of $opt domain and $opt token

sudo vim /usr/lib/nagios/plugins/slack_nagios.pl

edit these lines:

my $opt domain: "operationseng-rck5174.slack.com";
my $opt token: ""; # enter token you've generated in previous step
  1. Test to make sure it works, make sure you test in the /usr/lib/nagios/plugins/ directory:

cd /usr/lib/nagios/plugins/

./slack_nagios.pl -field slack_channel=#group-c -field HOSTALIAS="Test Host" -field HOSTSTATE="UP" -field HOSTOUTPUT="Host is UP" -field NOTIFICATIONTYPE="RECOVERY"

or alternatively:

/usr/lib/nagios/plugins/slack_nagios.pl -field slack_channel=#group-c -field HOSTALIAS="Test Host #2" -field HOSTSTATE="UP" -field HOSTOUTPUT="Host is UP" -field NOTIFICATIONTYPE="RECOVERY"

You should be able to get a notification from Slack in the chosen channel: image

  1. Manually Create a file /etc/nagios-plugins/config/slack.cfg with the following contents (change the channel name to your channel):

sudo vim /etc/nagios-plugins/config/slack.cfg then:

define command {
command_name notify-service-by-slack
command_line /usr/lib/nagios/plugins/slack_nagios.pl -field slack_channel=#group-c -field \
HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" \
-field SERVICESTATE="$SERVICESTATE$" -field SERVICEOUTPUT="$SERVICEOUTPUT$" \
-field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
1
}

define command {
command_name notify-host-by-slack
command_line /usr/lib/nagios/plugins/slack_nagios.pl -field slack_channel=#group-c -field \
HOSTALIAS="$HOSTNAME$" -field HOSTSTATE="$HOSTSTATE$" \
-field HOSTOUTPUT="$HOSTOUTPUT$" \
-field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
}

Configure a Nagios contact inside Nagios module

  1. Inside the Nagios module, Add the following code to config.pp:
# Slack notification for nagios
nagios_contact { "slack":
target => "/etc/nagios3/conf.d/ppt_contacts.cfg",
alias => "Slack",
service_notification_period => "24x7",
host_notification_period => "24x7",
service_notification_options => "w,u,c,r",
host_notification_options => "d,r",
service_notification_commands => "notify-service-by-slack",
host_notification_commands => "notify-host-by-slack",
email => "root@localhost",
mode => "0444",
}

nagios_contactgroup { "slackgroup":
target => "/etc/nagios3/conf.d/ppt_contactgroups.cfg",
alias => "Slack channel",
members => "slack",
mode => "0444",
}
  1. Add this line to all the nagios hosts and the root disk check services (change it from admins to slackgroup):

contact_groups => 'slackgroup'

Should look like this:

nagios_host { "db-c":
    target                => "/etc/nagios3/conf.d/ppt_hosts.cfg",
    alias                 => "db",
    check_period          => "24x7",
    max_check_attempts    => 3,
    check_command         => "check-host-alive",
    notification_interval => 30,
    notification_period   => "24x7",
    notification_options  => "d,u,r",
    mode                  => "0444",
    contact_groups        => 'slackgroup',
 }

and for services (do this for all the services if you want them notified)

nagios_service {"root_disk_check":
  service_description => "Root Disk Space",
  hostgroup_name => "Remote-Disks",
  target => "/etc/nagios3/conf.d/ppt_services.cfg",
  check_command => "check_nrpe!check_sda1",
  max_check_attempts => 3,
  retry_check_interval => 1,
  normal_check_interval => 5,
  check_period => "24x7",
  notification_interval => 30,
  notification_period => "24x7",
  notification_options => "w,u,c",
  contact_groups => "slackgroup",
  mode => "0444",
}
  1. Apply the puppet agent once you done it and restart nagios

image

Testing alerts

If successfully configured you should see the nagios notification on slack

image