Ticket #933: Perform System‐Wide Updates and Basic System Hardening - KeegMitch/Operations-Engineering-group-c GitHub Wiki

Ticket: #933

Subtickets: #935 , #936, #937

Keep SSH Updated

  • Inside the mgmt server, go inside this directory to create a new puppet module: /etc/puppetlabs/code/modules

  • Inside that directory, do the following commands in order:

sudo mkdir update_ssh
cd update_ssh
sudo mkdir files
sudo mkdir manifests

cd manifests 
sudo touch init.pp

image

  • Inside the init.pp we are going to do update and upgrade everything for security
class update_ssh {

  # Ensure grep is installed
  package { 'grep':
    ensure => installed,
  }

  # Ensure the apt package list is up to date
  exec { 'apt_update':
    command   => '/usr/bin/apt-get update',
    path      => ['/usr/local/sbin', '/usr/sbin', '/sbin', '/usr/local/bin', '/usr/bin', '/bin'],
    unless    => 'test $(find /var/lib/apt/periodic/update-success-stamp -mtime -1 -print)',
    logoutput => true,
    require   => Package['grep'],
  }

  # Ensure the openssh-server package is updated
  package { 'openssh-server':
    ensure  => latest,
    require => Exec['apt_update'],
  }

  # Upgrade all other packages on the system
  exec { 'apt_upgrade':
    command     => '/usr/bin/apt-get upgrade ssh -y',
    path        => ['/usr/local/sbin', '/usr/sbin', '/sbin', '/usr/local/bin', '/usr/bin', '/bin'],
    unless      => '/usr/bin/apt-get upgrade --just-print | /bin/grep "0 upgraded"',
    logoutput   => true,
    require     => Exec['apt_update'],
  }

  # Log a message indicating the update is complete
  notify { 'SSH update and system upgrade complete':
    require => Exec['apt_upgrade'],
  }
}
  • Go to site.pp and apply the ssh_disable_root_login module to all of the server nodes, by adding include ssh_disable_root_login:

sudo vim /etc/puppetlabs/code/environments/production/manifests/site.pp or our alias site_pp

image

  • Apply the puppet agent to all of the servers: sudo /opt/puppetlabs/puppet/bin/puppet agent --test or our alias test_puppet_agent

Screenshot is when tested in mgmt, but output should be the same on all the servers image

Disable Root Login

  • Before doing anything, make a backup of the sshd_config file: sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak

  • Create another puppet module called ssh_disable_root_login inside /etc/puppetlabs/code/modules

sudo mkdir ssh_disable_root_login
cd ssh_disable_root_login
sudo mkdir files
sudo mkdir manifests

cd manifests 
sudo vim init.pp
  • Add the following code ( be careful, if you have the wrong code it could end up deleting all the other settings!):
class ssh_disable_root_login (
  Boolean $disable_root_login = true,
) {
  package { 'nano':
    ensure => installed,
  }

  service { 'ssh':
    ensure => running,
    enable => true,
    require => Package['openssh-server'],
  }

  $permit_root_login = if $disable_root_login {
    'no'
  } else {
    'yes'
  }

  augeas { 'disable_root_login':
    context => '/files/etc/ssh/sshd_config',
    changes => [
      "set PermitRootLogin ${permit_root_login}",
    ],
    onlyif  => "match PermitRootLogin[. = '${permit_root_login}'] size == 0",
    require => Package['openssh-server'],
    notify  => Service['ssh'],
  }
}
  • Go to site.pp and apply the ssh_disable_root_login module to all of the server nodes, by adding include ssh_disable_root_login:

sudo vim /etc/puppetlabs/code/environments/production/manifests/site.pp or our alias site_pp

  • Apply the puppet agent to all of the servers: sudo /opt/puppetlabs/puppet/bin/puppet agent --test or our alias test_puppet_agent

image

Enable fail2ban

Create the puppet module folders and files

image

config.pp

class fail2ban::config {
  file { '/etc/fail2ban/jail.local':
    ensure  => 'file',
    content => template('fail2ban/jail.local.erb'),
    require => Package['fail2ban'],
  }
}

install.pp

class fail2ban::install {
  package { 'fail2ban':
    ensure => 'present',
  }
}

service.pp

class fail2ban::service {
  service { 'fail2ban':
    ensure    => 'running',
    enable    => true,
    require   => [Package['fail2ban'], File['/etc/fail2ban/jail.local']],
  }
}

init.pp

class fail2ban {
        include fail2ban::install
        include fail2ban::config
        include fail2ban::service
}

templates/jail.local.erb

[DEFAULT]
bantime  = 600
findtime  = 600
maxretry = 5
ignoreip = 127.0.0.1/8

[sshd]
enabled  = true
port     = ssh
filter   = sshd
logpath  = /var/log/auth.log
maxretry = 5
bantime  = 360

image

Disable Password Authentication and Enable Public Key Authentication

  • Create a backup of sshd_config if you somehow haven't already
  • Create a puppet module called ssh_disable_password_auth inside /etc/puppetlabs/code/modules
class ssh_disable_password_auth (
  Boolean $disable_password_auth = true,
) {
  package { 'nano':
    ensure => installed,
  }

  service { 'ssh':
    ensure => running,
    enable => true,
    require => Package['openssh-server'],
  }

  $password_auth = if $disable_password_auth {
    'no'
  } else {
    'yes'
  }

  augeas { 'disable_password_auth':
    context => '/files/etc/ssh/sshd_config',
    changes => [
      "set PasswordAuthentication ${password_auth}",
    ],
    onlyif  => "match PasswordAuthentication[. = '${password_auth}'] size == 0",
    require => Package['openssh-server'],
    notify  => Service['ssh'],
  }
  
  # enable PubkeyAuthentication
   augeas { 'enable_pubkey_auth':
    context => '/files/etc/ssh/sshd_config',
    changes => [
      'set PubkeyAuthentication yes',
    ],
    onlyif  => "match PubkeyAuthentication[. = 'yes'] size == 0",
    require => Package['openssh-server'],
    notify  => Service['ssh'],
  }
}

Go to site.pp and apply the ssh_disable_password_auth module to all of the server nodes (try just mgmt first to test!), by adding include ssh_disable_password_auth: sudo vim /etc/puppetlabs/code/environments/production/manifests/site.pp or our alias site_pp

  • Important: Troubleshoot the SSH keys BEFORE applying the agent, or you will risk getting locked out of your servers!

Setup all the SSH keys to each other including the gateway server

Setup SSH Keys page here

Table of all the keys we set up:

From To SSH Key Name
mgmt backup ~/.ssh/id_rsa
mgmt storage ~/.ssh/id_rsa_offsite
mgmt app ~/.ssh/id_rsa_mgmt_app
mgmt db ~/.ssh/id_rsa_mgmt_db
db backup ~/.ssh/id_rsa_db_1
db storage ~/.ssh/id_rsa_db_storage
db mgmt ~/.ssh/id_rsa_db_mgmt
db app ~/.ssh/id_rsa_db_app
app backup ~/.ssh/id_rsa_app_backup
app storage ~/.ssh/id_rsa_app_storage
app mgmt ~/.ssh/id_rsa_app_mgmt
app db ~/.ssh/id_rsa_app_db
backup storage ~/.ssh/id_rsa_backup_storage
backup app ~/.ssh/id_rsa_backup_app
backup mgmt ~/.ssh/id_rsa_backup_mgmt
backup db ~/.ssh/id_rsa_backup_db
gateway mgmt ~/.ssh/id_rsa_gateway_mgmt
gateway db ~/.ssh/id_rsa_gateway_db
gateway app ~/.ssh/id_rsa_gateway_app
gateway backup ~/.ssh/id_rsa_gateway_backup
gateway storage ~/.ssh/id_rsa_gateway_offsite

To add your ssh key from the gateway server, do the following:

# from gateway to mgmt server
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa_gateway_mgmt
chmod 600 ~/.ssh/id_rsa_gateway_mgmt && chmod 600 ~/.ssh/id_rsa_gateway_mgmt.pub
ssh-copy-id -i ~/.ssh/id_rsa_gateway_mgmt.pub [email protected]
ssh -i ~/.ssh/id_rsa_gateway_mgmt [email protected]
logout

# from gateway to db server
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa_gateway_db
chmod 600 ~/.ssh/id_rsa_gateway_db && chmod 600 ~/.ssh/id_rsa_gateway_db.pub
ssh-copy-id -i ~/.ssh/id_rsa_gateway_db.pub [email protected]
ssh -i ~/.ssh/id_rsa_gateway_db [email protected]
logout

# from gateway to backup server
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa_gateway_backup
chmod 600 ~/.ssh/id_rsa_gateway_backup && chmod 600 ~/.ssh/id_rsa_gateway_backup.pub
ssh-copy-id -i ~/.ssh/id_rsa_gateway_backup.pub [email protected]
ssh -i ~/.ssh/id_rsa_gateway_backup [email protected]
logout

# from gateway to app server
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa_gateway_app
chmod 600 ~/.ssh/id_rsa_gateway_app && chmod 600 ~/.ssh/id_rsa_gateway_app.pub
ssh-copy-id -i ~/.ssh/id_rsa_gateway_app.pub [email protected]
ssh -i ~/.ssh/id_rsa_gateway_app [email protected]
logout

# from gateway to offsite/storage server
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa_gateway_offsite
chmod 600 ~/.ssh/id_rsa_gateway_offsite && chmod 600 ~/.ssh/id_rsa_gateway_offsite.pub
ssh-copy-id -i ~/.ssh/id_rsa_gateway_offsite.pub [email protected]
ssh -i ~/.ssh/id_rsa_gateway_offsite [email protected]
logout

Note: This applies for each member of the team as well, check out the authorized_keys file inside the server you are setting up the gateway ssh to, it should come up with your username and "@fthictedge01" at the end of the key.

Regularly Monitor SSH Logs using Nagios

  • Nagios SSH log script configured here

Limit SSH Access

  • Create a puppet module inside the same directory as the previous ones, and add the following:

image

class ssh_group_c {
  file { '/etc/ssh/sshd_config':
    ensure  => file,
    content => template('ssh_group_c/sshd_config.erb'),
    notify  => Service['ssh'],
  }
}
  • Contents from sshd_config.erb

The contents: just copy and paste the /etc/ssh/sshd_config, then add the allow users

image

  • Allow only your group username, it will deny any other user from accessing the system image

  • Include the module inside all the servers in site.pp

  • Apply the puppet agent to all the servers