Ticket ID #253 ‐ Implement Slack Alert Notifications for Nagios Monitoring - GriffinKat/group-a GitHub Wiki

Implement Slack Alert Notifications for Nagios Monitoring

  1. Enable Nagios notifications in the Slack Workspace.

  2. Install Slack

  3. Setup Slack Plugin for Nagios

    • Create a module called 'nagios_slack_alert' on the puppet master(mgmt-a server) in the /etc/puppetlabs/code/modules directory.

    • Create a file structure as shown below.

      image

    • Now, create three files in the manifests directory called init.pp, install.pp and config.pp. Write the following lines of code in the init.pp, install.pp and config.pp files respectively.

         class nagios_slack_alert {
     include nagios_slack_alert::install
     include nagios_slack_alert::config
   }

         class nagios_slack_alert::install {
       exec {'apt_update':
         command     => '/usr/bin/apt-get update',
         refreshonly => true,
         before      => Package['libwww-perl', 'libcrypt-ssleay-perl'],
       }

       package { ['libwww-perl', 'libcrypt-ssleay-perl']:
       ensure => installed,
       provider => 'apt',
      }
   }

      
   class nagios_slack_alert::config {
     file {'/usr/lib/nagios/plugins/nagios_slack.pl':
       ensure => present,
       source => 'puppet:///modules/nagios_slack_alert/nagios_slack.pl',
       owner => 'nagios',
       group => 'nagios',
       mode => '0755',
       require => Class['nagios_slack_alert::install']
     }

     file { '/etc/nagios-plugins/config/slack.cfg':
       ensure => present,
       content => template('nagios_slack_alert/slack.cfg'),
       owner => 'nagios',
       group => 'nagios',
       mode => '0755'
     }

     nagios_contact { 'slack':
      target => '/etc/nagios4/conf.d/ppt_contacts.cfg',
      alias => 'Slack',
      service_notification_period => '24x7',
      host_notification_period => '24x7',
      service_notification_options => 'w,u,c,r',
      host_notification_options => 'd,r',
      service_notification_commands => 'notify-service-by-slack',
      host_notification_commands => 'notify-host-by-slack',
      email => 'root@localhost',
     }

     nagios_contactgroup { 'slackgroup':
      target => '/etc/nagios4/conf.d/ppt_contactgroups.cfg',
      alias => 'Slack channel',
      members => 'slack',
     }
   }

  4. Download the plugin file from https://raw.github.com/tinyspeck/services-examples/master/nagios.pl. and place the file in the files directory of the module. I named the file nagios_slack.pl. Edit the file to set the values of $opt domain and $opt token.

    image

  5. Now, we will create a file called slack.cfg. We will use a template for this to dynamically generate the file content. Create a file called slack.cfg in the templates directory of the module using the command sudo nano slack.cfg. Paste the following lines of code in the file and save the file.

       define command {
     command_name notify-service-by-slack
     command_line /usr/lib/nagios/plugins/nagios_slack.pl -field slack_channel=#group-a -field \
     HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" \
     -field SERVICESTATE="$SERVICESTATE$" -field SERVICEOUTPUT="$SERVICEOUTPUT$" \
     -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
   }

   define command {
     command_name notify-host-by-slack
     command_line /usr/lib/nagios/plugins/nagios_slack.pl -field slack_channel=#group-a -field \
     HOSTALIAS="$HOSTNAME$" -field HOSTSTATE="$HOSTSTATE$" \
     -field HOSTOUTPUT="$HOSTOUTPUT$" \
     -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
   }

  6. Configure a Nagios contact

    Now we have notification commands ready to use. To use them, we need to configure a contact. We will create a special Slack contact to receive these notifications. The code for this is provided earlier in the config.pp file. After all the steps, the config.pp file should look like this.

    image

  7. Now we will edit the config.pp file in the nagios module to include the clients (db-a, mgmt-a, apps-a and backup-a) into the contact groups.

    image

    Also, we will include the slackgroup for the nagios_service check-disk-space.

    image

  8. Apply the module by using the command sudo /opt/puppetlabs/bin/puppet agent --test and restart nagios4.service.

    image

  9. Verification and Testing

    • Manual Testing- Type the following command on the mgmt-a server command line. /usr/lib/nagios/plugins/nagios_slack.pl -field slack_channel="#group-a" -field HOSTALIAS="Test Host" -field HOSTSTATE="UP" -field HOSTOUTPUT="Mayank Testing Slack Nagios Alert" -field NOTIFICATIONTYPE="WARNING"

    image

    Below is the screenshot of the notification.

    image

    • Automatic Notifications- For this, we will edit the nrpe.cfg file in the nrpe moudle on the mgmt-a server and change the check_dev_root command to give critical message when the disk space is 90 percent free.

      image

      Now apply the module in the host servers using the command-

      sudo /opt/puppetlabs/puppet/bin/puppet agent --server=mgmt-a --no-daemonize --verbose --onetime

      Below is the screenshot of the notification received on the slack group-a channel.

      image