Ticket #256: Implement Slack Alert Notifications for Nagios Monitoring - SupaHotBall/OE2-Group-D GitHub Wiki

Task

Slack Integration Setup Using Puppet Module

• Install required Perl modules (libwww-perl, libcrypt-ssleay-perl, liblwp-protocolhttps-perl)
• Deploy Nagios Slack plugin to /usr/lib/nagios/plugins
• Configure Slack API token and channel in plugin script

Puppet Configuration

• Create slack.cfg with service/host notification commands
• Define Nagios contact and contactgroup resources for Slack
• Apply configuration to all monitored services requiring alerts within your nagios config manifest

Configure alert notifications in Nagios manifests by

• Adding the 'slackgroup' contact group to all relevant service and host definitions
• Setting appropriate notification intervals and thresholds for each service
• Applying changes through Puppet to ensure consistent configuration across all monitored nodes"

Testing & Validation

• Trigger test alerts by modifying disk threshold checks
• Verify Slack channel receives notifications
• Adjust notification options to balance urgency and noise

Acceptance Criteria

• Slack notifications received for critical/warning events
• No false alerts during testing period
• Puppet-managed configuration deployed to all servers
• Documentation includes alert response procedures

---

Steps Taken

Create slack_plugin in modules

• Go to cd /etc/puppletlabs/code/modules/

• Then do this code:

sudo mkdir -p slack_plugin/{manifests,files,templates}

Install the Perl modules libwww-perl libcrypt-ssleay-perl liblwp-protocolhttps-perl

• Go to cd ../manifests and sudo nano init.pp

class slack_plugin {

  # Ensure required Perl packages
  package { ['libwww-perl', 'libcrypt-ssleay-perl', 'liblwp-protocol-https-perl']:
    ensure => installed,
  }

Download the plugin file from https://raw.github.com/tinyspeck/services-examples/master/nagios.pl

• Go to cd ../files and then sudo wget https://raw.github.com/tinyspeck/services-examples/master/nagios.pl
• and the sudo nano nagios.pl
• you have to find your $opt_domain and $opt_token in this website : https://api.slack.com/apps
• Click Create New App
• Choose "From scratch"
• App Name: NagiosAlerts
• Pick our workspace OE2
• Then in the left sidebar, click Incoming Webhooks
• Turn the switch ON (Activate)
• Scroll down and click Add New Webhook to Workspace
• Choose group-d channel
• Click Allow

• Place 'hook.slack.com' after $opt_domain =
• Place 'services/T06LE2NV39R/B06KDUGQK8F/aSecretWebhookTokenHere' after $opt_token =
• Replace the POST()

use JSON;

my $json_payload = encode_json({
  text => "⚠️ Nagios Alert: $event{NOTIFICATIONTYPE}\nHost: $event{HOSTALIAS}\nState: $event{HOSTSTATE}\nOutput: $event{HOSTOUTPUT}"
});

my $req = HTTP::Request->new(
  'POST',
  "https://${opt_domain}/${opt_token}"
);
$req->header('Content-Type' => 'application/json');
$req->content($json_payload);

Copy the file to /usr/lib/nagios/plugins and set its permissions to 0755.

• Go to cd ../manifests/ and then sudo nano init.pp

# Copy the nagios.pl file
  file { '/usr/lib/nagios/plugins/nagios.pl':
    ensure  => file,
    mode    => '0755',
    owner   => 'root',
    group   => 'root',
    source  => 'puppet:///modules/slack_plugin/nagios.pl',
  }

  # Create the slack.cfg file from template
  file { '/etc/nagios-plugins/config/slack.cfg':
    ensure  => file,
    mode    => '0644',
    owner   => 'root',
    group   => 'root',
    content => template('slack_plugin/slack.cfg.erb'),
  }

Then do sudo cp /etc/puppetlabs/code/modules/slack_plugin/files/nagios.pl /usr/lib/nagios/plugins/nagios.pl and sudo chmod 755 /usr/lib/nagios/plugins/nagios.pl

Create a file /etc/nagios-plugins/config/slack.cfg with the following contents

• Go to cd ../templates/ and sudo nano slack.cfg.erb

define command {
  command_name notify-service-by-slack
  command_line /usr/lib/nagios/plugins/nagios.pl -field slack_channel=#group-d -field \
HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" \
-field SERVICESTATE="$SERVICESTATE$" -field SERVICEOUTPUT="$SERVICEOUTPUT$" \
-field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
}

define command {
  command_name notify-host-by-slack
  command_line /usr/lib/nagios/plugins/nagios.pl -field slack_channel=#group-d -field \
HOSTALIAS="$HOSTNAME$" -field HOSTSTATE="$HOSTSTATE$" \
-field HOSTOUTPUT="$HOSTOUTPUT$" \
-field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
}

Configure a Nagios contact

• Go back to cd ../manifests/ and sudo nano init.pp

# Define the Slack contact for Nagios
  nagios_contact { 'slack':
    target                      => '/etc/nagios4/conf.d/ppt_contacts.cfg',
    alias                       => 'Slack',
    service_notification_period => '24x7',
    host_notification_period    => '24x7',
    service_notification_options => 'w,u,c,r',
    host_notification_options   => 'd,r',
    service_notification_commands => 'notify-service-by-slack',
    host_notification_commands => 'notify-host-by-slack',
    email                       => 'root@localhost',
  }

  # Define the Slack contact group
  nagios_contactgroup { 'slackgroup':
    target  => '/etc/nagios4/conf.d/ppt_contactgroups.cfg',
    alias   => 'Slack channel',
    members => 'slack',
  }

Apply the Puppet module

sudo puppet apply /etc/puppetlabs/code/modules/slack_plugin/manifests/init.pp

---

Challenges

---

External Resources

---

Ticket Reference

https://rt.dataraster.com/Ticket/Display.html?id=256&results=0f1be34fd1a43932f827af2b032dfbe9

---