Harden SSH & Configure the fail2ban service to stop brute force password attacks

We were the unfortunate victims of an attack to our precious servers, because of this we need to update our security settings

Hardening SSH means strengthening its security settings to reduce vulnerabilities

fail2ban is a tool that scans log files and bans IPs showing malicious behavior — like too many failed SSH login attempts

We will use Puppet to do this because it saves time, reduces errors and to ensure consistency

Step 1: Create a puppet module

    
cd sudo mkdir /etc/puppetlabs/code/modules/fail2ban  
  
sudo mkdir files  
  
sudo mkdir manifests  
  
cd files  
  
sudo touch jail.local  
  
cd ..
  
cd manifests  
  
sudo touch init.pp  
  

Step 2: Create files to install fail2ban and harden ssh using jail.local

In the manifests directory add to 'init.pp'

class fail2ban {
  package { 'fail2ban':
    ensure => installed,
  }

  file { '/etc/fail2ban/jail.local':
    ensure  => file,
    source  => 'puppet:///modules/fail2ban/jail.local',
    owner   => 'root',
    group   => 'root',
    mode    => '0644',
    notify  => Service['fail2ban'],
  }

  service { 'fail2ban':
    ensure => running,
    enable => true,
  }
}

In the files folder add to 'jail.local'

[sshd]
enabled   = true
port      = ssh
filter    = sshd
logpath   = /var/log/auth.log
maxretry  = 4
bantime   = 1800
findtime  = 900

Apply the module to the nodes by adding 'fail2ban' to each node in 'Site.pp'

/etc/puppetlabs/code/environments/production/manifests/site.pp

Step 3: Test the configuration on the puppet nodes

In each of the servers run the command

sudo puppet agent --test OR

sudo /opt/puppetlabs/bin/puppet agent --test

Database:

Backup:

Apps:

Management:

Step 4: Check if fail2ban is running using this command

sudo systemctl status fail2ban

Database:

Management:

Apps:

Backup:

Simulate failed SSH login attempts and verify Fail2Ban bans IPs

Confirm fail2ban-client status shows SSH jail active and banning

Command:

sudo fail2ban-client status sshd

Mgmt:

Backup-a:

Apps-a:

Database:

Update all system packages across all assigned servers using Puppet

We want to update our system packages automatically as it makes our servers more secure. Updates can include patches for new security vulnerabilities

Create a puppet module on the management server in the puppetlabs directory

sudo mkdir /etc/puppetlabs/code/modules/system_update

sudo mkdir /etc/puppetlabs/code/modules/system_update/manifests

In the manifests directory create an init.pp file

sudo nano /etc/puppetlabs/code/modules/system_update/manifests/init.pp

Then add this code to install 'Unattended Updates' which automatically updates packages

class system_update {

  # Step 0: Unhold important packages if held (e.g. initramfs-tools)
  ['initramfs-tools', 'initramfs-tools-core', 'initramfs-tools-bin'].each |String $pkg| {
    exec { "unhold_${pkg}":
      command => "/usr/bin/apt-mark unhold ${pkg}",
      path    => ['/usr/bin', '/usr/sbin'],
      onlyif  => "/usr/bin/apt-mark showhold | /bin/grep -q '^${pkg}'",
    }
  }

  # Step 1: Update APT cache
  exec { 'apt_update':
    command => '/usr/bin/apt-get update',
    path    => ['/usr/bin', '/usr/sbin'],
  }

  # Step 2: Upgrade all safe packages
  exec { 'apt_upgrade':
    command => '/usr/bin/apt-get upgrade -y',
    path    => ['/usr/bin', '/usr/sbin'],
    require => Exec['apt_update'],
  }

  # Step 3: Full upgrade including kernel, initramfs, etc.
  exec { 'apt_dist_upgrade':
    command => '/usr/bin/apt-get dist-upgrade -y',
    path    => ['/usr/bin', '/usr/sbin'],
    require => Exec['apt_upgrade'],
  }

  # Step 4: Explicitly force-upgrade critical packages if not already upgraded
  ['initramfs-tools', 'initramfs-tools-core', 'initramfs-tools-bin'].each |String $pkg| {
    exec { "force_upgrade_${pkg}":
      command => "/usr/bin/apt-get install -y --only-upgrade ${pkg}",
      path    => ['/usr/bin', '/usr/sbin'],
      require => Exec['apt_dist_upgrade'],
    }
  }

  # Step 5: Optional reboot if required
  exec { 'check_reboot':
    command => '/sbin/shutdown -r +1 "Reboot scheduled by Puppet due to system update."',
    onlyif  => '/usr/bin/test -f /var/run/reboot-required',
    path    => ['/usr/bin', '/usr/sbin', '/sbin'],
  }

}

Command	Purpose
apt-get update	Refreshes the APT package index, including security repositories like jammy-security.
apt-get upgrade -y	Installs the latest versions of all installed packages that don’t require dependency changes; includes many security updates.
apt-get dist-upgrade -y	Installs updates that require installing/removing packages, such as kernel updates, initramfs-tools, or security patches with new dependencies.
**Reboot if required**	If /var/run/reboot-required exists, a reboot is needed to fully apply updates (e.g., kernel/initramfs upgrades).

Tell Puppet to apply the system_update class when it configures each puppet node

Add 'system_update' to the end of each node

/etc/puppetlabs/code/environment/production/manifests/site.pp

Check that the module works using this command

sudo puppet agent --test

OR

sudo /opt/puppetlabs/bin/puppet agent --test

Database:

Backup:

Apps:

Management:

Verify that the servers are all up to date with the command

apt list --upgradable

Management:

Apps:

Database:

Backup:

🛡️ Harden SSH Settings Using Puppet

Enhance system security by configuring SSH with stricter settings via Puppet

🔒 Disable Root Login

⏱️ Set a LoginGraceTime and maximum authentication attempts

Create an SSH Hardening module using Puppet

sudo mkdir -p/etc/puppetlabs/code/environments/production/modules/ssh_hardening/manifests

sudo nano/etc/puppetlabs/code/environments/production/modules/ssh_hardening/manifests/init.pp

class ssh_hardening {

  # Disable root login
  file_line { 'Disable root login':
    path  => '/etc/ssh/sshd_config',
    line  => 'PermitRootLogin no',
    match => '^PermitRootLogin',
  }

  # Set login grace time to 30 seconds
  file_line { 'Set LoginGraceTime':
    path  => '/etc/ssh/sshd_config',
    line  => 'LoginGraceTime 30',
    match => '^LoginGraceTime',
  }

  # Set maximum authentication attempts to 3
  file_line { 'Set MaxAuthTries':
    path  => '/etc/ssh/sshd_config',
    line  => 'MaxAuthTries 3',
    match => '^MaxAuthTries',
  }

  # Ensure the SSH service is running and restarts on config changes
  service { 'ssh':
    ensure    => running,
    enable    => true,
    subscribe => File_line['Disable root login'],
  }

}

Install the 'puppetlabs-stdlib' module

🔐 It lets you tweak specific lines in SSH config files using file_line without rewriting the whole file

sudo puppet module install puppetlabs-stdlib

Add ssh_hardening to the site.pp file so it runs on the nodes

Apply the module to the nodes

sudo /opt/puppetlabs/bin/puppet agent --test

Management:

Backup:

Database:

Apps:

Check that SSH has indeed been hardened by viewing the sshd_config file

grep -Ei 'PermitRootLogin|LoginGraceTime|MaxAuthTries' /etc/ssh/sshd_config

Management:

Backup:

Database:

Apps:

Ticket Reference:

https://rt.dataraster.com/Ticket/Display.html?id=351

SSH Log Monitoring via Nagios (Puppet-Based)

This task has been completed and is documented here:

https://github.com/GriffinKat/group-a/wiki/Ticket-ID%23346:-System-Hardening-&-Patch-Management-Implementation

🎯 Troubleshooting Automatic Package Updates

Some files were not updating on backup and db

In the configuration file of Unattended Upgrades I uncommented some lines to allow it to not just update security packages

/etc/apt/apt.conf.d/50unattended-upgrades

Uncommenting this line

Also this line

I edited the init.pp file

class system_update {

  # Step 1: Update APT cache
  exec { 'apt_update':
    command => '/usr/bin/apt-get update',
    path    => ['/usr/bin', '/usr/sbin'],
  }

  # Step 2: Force upgrade of apt-utils
  exec { 'upgrade_apt-utils':
    command => '/usr/bin/apt-get install -y apt-utils',
    path    => ['/usr/bin', '/usr/sbin'],
    require => Exec['apt_update'],
  }

  # Step 3: Force upgrade of apt
  exec { 'upgrade_apt':
    command => '/usr/bin/apt-get install -y apt',
    path    => ['/usr/bin', '/usr/sbin'],
    require => Exec['apt_update'],
  }

  # Step 4: Force upgrade of libapt-pkg6.0
  exec { 'upgrade_libapt-pkg6.0':
    command => '/usr/bin/apt-get install -y libapt-pkg6.0',
    path    => ['/usr/bin', '/usr/sbin'],
    require => Exec['apt_update'],
  }

  # Optional reboot if required
  exec { 'check_reboot':
    command => '/sbin/shutdown -r +1 "Reboot scheduled by Puppet due to core package upgrade."',
    onlyif  => '/usr/bin/test -f /var/run/reboot-required',
    path    => ['/usr/bin', '/usr/sbin', '/sbin'],
  }

}

After this I found that there was one update that hadn't applied to the backup app so I edited the init.pp file again to force it to update these packages

class system_update {

  # Step 1: Update APT cache
  exec { 'apt_update':
    command => '/usr/bin/apt-get update',
    path    => ['/usr/bin', '/usr/sbin'],
  }

  # Step 2: Force upgrade of core APT-related packages
  $packages_to_upgrade = [
    'apt',
    'apt-utils',
    'apt-transport-https',
    'libapt-pkg6.0',
  ]

  $packages_to_upgrade.each |String $pkg| {
    exec { "upgrade_${pkg}":
      command => "/usr/bin/apt-get install -y ${pkg}",
      path    => ['/usr/bin', '/usr/sbin'],
      require => Exec['apt_update'],
    }
  }

  # Optional: reboot if required
  exec { 'check_reboot':
    command => '/sbin/shutdown -r +1 "Reboot scheduled by Puppet due to core package upgrade."',
    onlyif  => '/usr/bin/test -f /var/run/reboot-required',
    path    => ['/usr/bin', '/usr/sbin', '/sbin'],
  }

}

Finally this is the code that worked

class system_update {

  # Step 0: Unhold important packages if held (e.g. initramfs-tools)
  ['initramfs-tools', 'initramfs-tools-core', 'initramfs-tools-bin'].each |String $pkg| {
    exec { "unhold_${pkg}":
      command => "/usr/bin/apt-mark unhold ${pkg}",
      path    => ['/usr/bin', '/usr/sbin'],
      onlyif  => "/usr/bin/apt-mark showhold | /bin/grep -q '^${pkg}'",
    }
  }

  # Step 1: Update APT cache
  exec { 'apt_update':
    command => '/usr/bin/apt-get update',
    path    => ['/usr/bin', '/usr/sbin'],
  }

  # Step 2: Upgrade all safe packages
  exec { 'apt_upgrade':
    command => '/usr/bin/apt-get upgrade -y',
    path    => ['/usr/bin', '/usr/sbin'],
    require => Exec['apt_update'],
  }

  # Step 3: Full upgrade including kernel, initramfs, etc.
  exec { 'apt_dist_upgrade':
    command => '/usr/bin/apt-get dist-upgrade -y',
    path    => ['/usr/bin', '/usr/sbin'],
    require => Exec['apt_upgrade'],
  }

  # Step 4: Explicitly force-upgrade critical packages if not already upgraded
  ['initramfs-tools', 'initramfs-tools-core', 'initramfs-tools-bin'].each |String $pkg| {
    exec { "force_upgrade_${pkg}":
      command => "/usr/bin/apt-get install -y --only-upgrade ${pkg}",
      path    => ['/usr/bin', '/usr/sbin'],
      require => Exec['apt_dist_upgrade'],
    }
  }

  # Step 5: Optional reboot if required
  exec { 'check_reboot':
    command => '/sbin/shutdown -r +1 "Reboot scheduled by Puppet due to system update."',
    onlyif  => '/usr/bin/test -f /var/run/reboot-required',
    path    => ['/usr/bin', '/usr/sbin', '/sbin'],
  }

}