Ticket #349: System Hardening & Patch Management Implementation - SupaHotBall/OE2-Group-D GitHub Wiki

Task

📌 Early this week, our systems underwent changes and experienced some external attacks.

Today, we must conduct system-wide updates on all assigned servers using Puppet, and apply basic system hardening techniques. Your team manager is able to provide some guide, and you are also encouraged to research and seek clarification as needed.

📌 Implementation Tasks

• Update all system packages across all assigned servers using Puppet.
• Ensure servers are up to date with the latest security patches.
• Configure automatic updates via Puppet manifests.

Harden SSH settings using Puppet by:

• Disabling root login
• Set a LoginGraceTime and maximum authentication attempts.

Configure the fail2ban service to stop brute force password attacks (against SSH on port 22):

• Install Fail2Ban on each assigned server
• For guide, refer to https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-ubuntu-20-04
• Copy and customize the configuration:
• Edit jail.local to monitor the SSH service and configure banning thresholds (e.g., maxretry = 4). Specify a ban time of 30 minutes, a find time of 15 minutes and set the maximum number of incorrect passwords attempts to 4. You may need to restart the SSH and fail2ban services after applying your configuration
• Ensure configuration is applied via Puppet and the service is started/enabled.

Regularly Monitor SSH Logs:

• Monitor /var/log/auth.log for suspicious SSH login behavior.
• Configure Nagios to track this log and alert based on repeated failed logins.
• Document your monitoring rules and test them by simulating failed login attempts.

Automate the Configuration Using Puppet

• Use one or more Puppet manifests to apply all above configurations.
• Ensure Puppet runs without error and changes apply across all four assigned servers.
• Ensure servers are up to date with the latest security patches.

📌 Validation Procedures Verify that:

• System packages are up-to-date (apt list --upgradable returns no packages)
• /etc/ssh/sshd_config reflects the hardened settings
• Root login is successfully disabled
• Confirm Puppet reports show successful application with no errors
• Simulate failed SSH login attempts and verify Fail2Ban bans IPs.
• Confirm fail2ban-client status shows SSH jail active and banning.
• Review /var/log/auth.log to confirm monitoring logic works.
• Ensure Nagios alerts are triggered for repeated failed SSH attempts.

📌 Acceptance Criteria

• ✔ All assigned servers are updated and configured via Puppet
• ✔ SSH root login is disabled, and hardened configs are in place
• ✔ Fail2Ban is installed, configured, and active via Puppet
• ✔ SSH log monitoring via Nagios is operational
• ✔ Puppet manifests are idempotent and well-documented
• ✔ Provide evidence:
• Screenshots of commands and statuses (Fail2Ban, logs, Nagios alerts)
• Puppet manifest code
• Explanation of the steps taken and their importance

📄 Documentation & Deliverables:

• Clear documentation with evidence of before and after logs
• If you create any script, provide a clear description of how the script works
• Well-commented Puppet manifests (*.pp files)
• Screenshots showing before/after system state
• Screenshots of SSH brute-force simulation and response
• Brief report explaining each hardening step and validation outcome

---

Steps Taken

Step 1: Create system_hardening Module

• File: /etc/puppetlabs/code/modules/system_hardening/

sudo mkdir -p /etc/puppetlabs/code/modules/system_hardening/{manifests,files,templates}

Create the following structure:

system_hardening/
├── manifests/
│   ├── init.pp
│   ├── update.pp
│   ├── autoupdate.pp
│   ├── ssh_hardening.pp
│   ├── fail2ban.pp
│   └── log_monitor.pp
├── templates/
└── files/

Step 2: System Update Script via Puppet

• File: manifests/update.pp

class system_hardening::update {
  exec { 'system-update':
    command => '/usr/bin/apt update && /usr/bin/apt -y upgrade',
    path    => ['/usr/bin', '/bin'],
    unless  => '/usr/bin/apt list --upgradable | grep -v "Listing..." | grep -q .',
  }
}

Updates all installed packages on the system unless there’s nothing left to upgrade.

Step 3: Enable Automatic Security Updates

• File: manifests/autoupdate.pp

class system_hardening::autoupdate {
  package { 'unattended-upgrades':
    ensure => installed,
  }

  file { '/etc/apt/apt.conf.d/20auto-upgrades':
    content => "APT::Periodic::Update-Package-Lists \"1\";\nAPT::Periodic::Unattended-Upgrade \"1\";\n",
    mode    => '0644',
  }

  service { 'unattended-upgrades':
    ensure => running,
    enable => true,
  }
}

Makes the system auto-install security patches every day.

Step 4: Harden SSH Configuration

• File: manifests/ssh_hardening.pp

class system_hardening::ssh_hardening {
  file_line { 'Disable root login':
    path  => '/etc/ssh/sshd_config',
    line  => 'PermitRootLogin no',
    match => '^PermitRootLogin',
  }

  file_line { 'Set LoginGraceTime':
    path  => '/etc/ssh/sshd_config',
    line  => 'LoginGraceTime 30',
    match => '^LoginGraceTime',
  }

  file_line { 'Set MaxAuthTries':
    path  => '/etc/ssh/sshd_config',
    line  => 'MaxAuthTries 3',
    match => '^MaxAuthTries',
  }

  service { 'ssh':
    ensure => running,
    enable => true,
    subscribe => File['/etc/ssh/sshd_config'],
  }
}

Makes SSH safer by:

• Disabling root login
• Setting login timeout
• Limiting incorrect password attempts

Step 5: Fail2Ban Installation and Activation

• File: manifests/fail2ban.pp

class system_hardening::fail2ban {
  package { 'fail2ban':
    ensure => installed,
  }

  file { '/etc/fail2ban/jail.local':
    content => "
[sshd]
enabled = true
port = ssh
logpath = /var/log/auth.log
maxretry = 4
findtime = 900
bantime = 1800
",
    mode => '0644',
  }

  service { 'fail2ban':
    ensure => running,
    enable => true,
  }
}

Protects SSH from brute-force attacks:

• Bans IPs with 4 failed attempts
• Ban lasts 30 minutes
• Checks log every 15 minutes

Step 6: Create Custom NRPE Script

• File: manifests/log_monitor.pp

class system_hardening::log_monitor {
  file { '/usr/lib/nagios/plugins/check_auth_logs.sh':
    ensure => file,
    mode   => '0755',
    owner  => 'root',
    group  => 'root',
    source => 'puppet:///modules/system_hardening/check_auth_logs.sh',
  }

  service { 'nagios-nrpe-server':
    ensure => running,
    enable => true,
    subscribe => File['/usr/lib/nagios/plugins/check_auth_logs.sh'],
  }
}

Script in check_auth_logs.sh

• File: files/check_auth_logs.sh

#!/bin/bash
count=$(grep "Failed password" /var/log/auth.log | grep "$(date --date='5 min ago' '+%b %e %H')" | wc -l)

if [ "$count" -ge 5 ]; then
  echo "CRITICAL: $count failed SSH login attempts in the last 5 minutes."
  exit 2
elif [ "$count" -ge 1 ]; then
  echo "WARNING: $count failed SSH login attempts in the last 5 minutes."
  exit 1
else
  echo "OK: No recent failed SSH logins."
  exit 0
fi

Make it executable:

sudo chmod +x /usr/lib/nagios/plugins/check_auth_logs.sh

Step 7: site.pp Configuration

• File: /etc/puppetlabs/code/environments/production/manifests/site.pp

include system_hardening::update
include system_hardening::autoupdate
include system_hardening::ssh_hardening
include system_hardening::fail2ban
include system_hardening::log_monitor

Repeat with the same includes for all nodes (backup-d, db-d, mgmt-d).

Step 8: Nagios Service Configuration

• File: /etc/nagios4/conf.d/ppt_services.cfg

define service {
  use                 generic-service
  hostgroup_name      hardened-servers
  service_description SSH Log Failure Monitoring
  check_command       check_nrpe!check_auth_logs
  max_check_attempts  3
  retry_interval      1
  check_interval      5
  notification_interval 30
  notification_period  24x7
  notification_options w,u,c,r
  contact_groups      slackgroup
}

Expected Output:

• In Nagios4 Web UI:
• Status: OK if no issues
• Status: WARNING or CRITICAL if login attempts detected

In Slack:

• You will receive alerts from notify-service-by-slack when login attempts exceed threshold

Manual NRPE test from mgmt:

/usr/lib/nagios/plugins/check_nrpe -H 10.2.0.X -c check_auth_logs

Should return:

OK: No recent failed SSH logins.

---

Challenges

N/A

---

External Resources

N/A

---

Ticket Reference

https://rt.dataraster.com/Ticket/Display.html?id=349&results=83682c3850cef8b99f6e1f769d0e56ee

---