Building Linux Security Monitoring Capabilities with Splunks' Universal Forwarder - cfloquetprojects/homelab GitHub Wiki

Introduction

• UNIX and Linux operating systems are often regarded as "built secure" and therefore somehow impervious to malware and malicious activity. Unfortunately, this couldn't be further from the case. We as security investigators must take extra steps to gain visibility into those hard to reach places and difficult to configure systems because that is where our adversaries most often hide.

• Today we will be installing the Splunk Universal Forwarder onto a Red Hat Enterprise Linux (RHEL) 7 host in order to collect important security logs which can often be used to detect malicious activity, or trace/track a threat actor after a breach has occurred.

• Unlike many other tutorials you will find online, we will be installing the forwarding agent under a dedicated non-root splunk service account. This is a very important best practice from a security perspective, and all things considered granularly adding specific read access to the files/directories which need to be monitored isn't all that difficult when compared to the risk of delegating root or even sudoer access.

  💡 As of v9, the Linux installation package for Splunk UF automatically installs a "least privileged" user. This has been a long time coming, with the many aforementioned security benefits of having a non-root user which is allocated only the exact privileges needed to run/forward system/authentication logs.

• The three primary log sources we will be monitoring in order to track actions and logins on each *nix host with an Universal Forwarder agent are as follows:

  • /home/*/.bash_history will monitor the bash history for all authenticated users on the system.

  • /root/bash_history will monitor the bash history for specifically the root user.

  • /var/log/wtmp combined with the utmpdump utility provides a useful history of successful logins and logouts (when combined with either wtmp or utmp log files, this log is also useful for detecting log tampering.

    💡 The reason utmpdump is useful for detecting log tampering is because it does essentially no parsing on its down, only rendering the raw data for the user to do what they want with. This makes it more impervious to corruption and renders tampered log entries in null format, rather than removing them entirely.

Diagram of Deployment/Forwarding Architecture:

• Shown below is diagram which outlines at a very high level how we can deploy apps (configurations) to our universal forwarder clients which then dictate what data or logs are forwarded to our intermediate forwarding tier and finally the cloud:

  💡 Please note that while this diagram depicts some of the other common Linux distributions we can deploy a universal forwarder agent onto, for the purposes of this tutorial I will only be covering the commands to install/configure the agent on Fedora based distributions (RHEL, AL2).

Resources

• Splunk Docs: Install a *nix Universal Forwarder
• Patrick Bareiss: Monitoring Linux Servers with Splunk
• Sandfly Security: Using Linux utmpdump for Forensics and Detecting Log File Tampering
• Hurricane Labs: Deploying the Splunk Universal Forwarder on Linux
• Bruce Nikkel: Practical Linux Forensics (2021)
• Red Hat Docs: An Introduction to Linux User Account Monitoring
• Splunkbase: Splunk Add-on for Unix and Linux
• Splunk Docs: Secure Linux Universal Forwarders with a Least Privileged User
• Splunk: Splunk Validated Architectures
• Graffletopia: Splunk Documentation Stencil Icons

Pre-Flight Check:

• Ensure you have a properly networked, updated, and patched RHEL 7 system. If you're unsure how to obtain licenses for RHEL based systems I created a Red Hat Developer Account which comes with 15 free licenses, more than enough for our purposes in an educational context.
• Go create a splunk account in order to download the latest version of the Universal Forwarder.

Preparing RHEL 7 Host for Splunk Installation:

• Just to re-enforce what was outlined above, let's make sure we have the latest packages available to us installed, while also grabbing some useful utilities we will be using later on like wget and tree.

$ sudo yum -y update
$ sudo yum -y upgrade
$ sudo yum -y install wget net-tools tree

• Now that we're fully updated & patched, let's create a dedicated installation directory for the forwarder using mkdir under /opt/splunkforwarder, and assign full permissions to a new dedicated splunk user with the useradd command and chown commands, respectively.

$ sudo mkdir -p /opt/splunkforwarder
$ sudo useradd splunk
$ sudo chown -R splunk:splunk /opt/splunkforwarder

• Now that we have our dedicated working installation directory, along with a dedicated non-sudoer splunk user with full permissions to the aforementioned direcotry,

$ sudo su - splunk
$ ls -l /opt
total 0
drwxr-xr-x. 2 splunk splunk 6 Oct 23 22:31 splunkforwarder

• Now that we've confirmed our installation directory has the correct permissions, let's navigate to it as the splunk user and use the wget utility to fetch Splunks Universal Forwarder (check latest version of UF here as of this writing it is v9.0.1):

$ cd /opt/splunkforwarder
$ wget -O splunkforwarder.tgz "https://download.splunk.com/products/universalforwarder/releases/9.0.1/linux/splunkforwarder-9.0.1-82c987350fde-Linux-ppc64le.tgz"

• Using the tar utility we can extract/install the contents of the tarball file that we just fetched from Splunk:

$ tar -xvzf /opt/splunkforwarder/splunkforwarder.tgz -C /opt/

Generating Strong Administrator Credentials for Splunk Universal Forwarder:

• Let's create a strong randomized credential pair for our forwarder using the openssl utility built into Linux. Once we do so we won't be prompted to enter one during actual installation in the next step.

  💡 Splunk automatically encrypts and removes the data present within the initial user-seed.conf file, and if you're curious for more information about how to secure administrator credentials for Splunk they have a very useful article on this exact topic on their docs page.

seedpw=$(openssl rand -base64 20)
seed=$'[user_info]\nUSERNAME=splufadmin\nPASSWORD='$seedpw
echo "$seed" >> /opt/splunkforwarder/etc/system/local/user-seed.conf

Obtaining and Installing Required Apps:

• While we are on our deployment server, let's quickly also create an app that can be used to test the monitoring of our *nix based hosts as well

$ pwd
/opt/splunk/etc/deployment-apps
$ mkdir -p nix_server_inputs/local

• For the purposes of this initial test we are just going to monitor the /var/log/messages file:

$ pwd
/opt/splunk/etc/deployment-apps/nix_server_inputs/local
$ vi inputs.conf
[monitor:///var/log]
disabled = 0
index = unix
whitelist=(messages)

Configure Linux Host and UF for Log Forwarding:

• Since we have already retrieved and installed the Splunk Universal Forwarder using a guide online (or even the article I wrote for this exact process) we can now move onto assigning the requisite (read) permissions to allow our dedicated splunk user to read critical system logs and files before sending them off.

• Within the context of our test RHEL 7 instance that we will be monitoring using the previously created nix_server_inputs app, we will just need to grant the splunk user read access to our /var/log/messages file as well as our /var/log/secure file using the setfacl command.

# setfacl -m g:splunk:r /var/log/messages
# setfacl -m g:splunk:r /var/log/secure

• Now that we have the proper permissions to read the files we will be monitoring, the last change we will need to make to the forwarder before starting it and setting our admin account credentials is defining the deployment server that will be administrating that host within the /opt/splunkforwarder/etc/system/local/deploymentclient.conf file.

$ pwd
/opt/splunkforwarder/etc/system/local/
$ vi deploymentclient.conf
[deployment-client]
[target-broker:deploymentServer]
targetUri= <ip of deployment server>:8089

• Start the universal forwarder on the *nix-based host by issuing the following command:

$ /opt/splunk/bin/splunk start

• After a short while, we should see that deployment client appear within the Forwarder Management panel on our dedicated Splunk Deployment Server host.