Install & Configure Splunk Universal Forwarder v9.0.0.1 as Non Root User on CentOS 7 - cfloquetprojects/homelab GitHub Wiki

• Monitoring Unix/Linux devices using the Splunk Universal Forwarder agent can be extremely rewarding in terms of yielding a wealth of information about a particular host using relatively lightweight and simple configurations.

• We will be installing Splunk Universal Forwarder v9.0.0.1 onto a Centos 7 host, make sure this host is built in advance and that you have access to an account with at minimum sudoer privileges.

  • This is mainly to create the non-sudoer splunk service account, which will be created and granted only the minimum level of permissions required to do it's job as a forwarder.

• The two primary log sources we will be monitoring in order to track actions and logins on each host with an Universal Forwarder agent are as follows:

/var/log/messages contains all system messages, including the those which are logged during system startup such as mail, cron, daemon, kern, auth, etc.

/var/log/secure primarily contains logs relating to authentication and authorization privileges. The Secure Shell Daemon (sshd) logs all important remote authentication messages here, including unsuccessful login which is useful in tracking brute force attempts.

• Splunkbase: Splunk Add-On for Unix and Linux
• Splunk Docs: Install a *nix Universal Forwarder
• Hurricane Labs: Deploying the Splunk Universal Forwarder on Linux
• Patrick Bareiss: Monitoring Linux Server with Splunk

• Ensure you have a properly networked, updated, and patched Linux hosted system (Fedora or Debian based)
• Go create a splunk account in order to download the latest version of the Universal Forwarder.

• Let's install any & all updates, as well as rpm, net-tools as well as wget which we will use to fetch the Universal Forwarder.

sudo yum -y update
sudo yum -y upgrade
sudo yum -y install wget net-tools tree

• Using the mkdir command we can now create the folder we will be storing the configuration files for our universal forwarder in, under /opt/splunkforwarder:

sudo mkdir -p /opt/splunkforwarder

• Next let's add a splunk user to manage/configure Splunk's universal forwarder RPM (updated for v9.0.0.1 as of 8/12/22), we will also be using the su command in order to assume that user for the remainder of the installation process.

sudo useradd splunk
sudo setfacl -m u:splunk:rwx /opt/splunkforwarder/
sudo su - splunk

• Initial wget operation to fetch Splunks Universal Forwarder (check latest version of UF here as of this writing it is 9.0.0.1):

cd /opt/splunkforwarder
wget -O splunkforwarder.tgz "https://download.splunk.com/products/universalforwarder/releases/9.0.1/linux/splunkforwarder-9.0.1-82c987350fde-Linux-ppc64le.tgz"

• Now that we have created the dedicated splunk user and assigned the proper permissions to the installation directory, we can effectively install the Splunk Universal Forwarder using the command below:

tar -xvzf splunkforwarder.tgz -C /opt/

• Let's create a strong randomized credential pair for our forwarder, as a matter of best practice. Once we do so we won't be prompted to enter one during actual installation in the next step.

  💡 It's even more important to do this if you are using deployment servers to manage the universal forwarder agents on a number of different clients, this effectively mitigates the risk of a single credential for all forwarders being compromised leading to agent manipulation etc.

seedpw=$(openssl rand -base64 20)
seed=$'[user_info]\nUSERNAME=splufadmin\nPASSWORD='$seedpw
echo "$seed" >> /opt/splunkforwarder/etc/system/local/user-seed.conf

• The next step is to configure the Universal Forwarder to run on boot as our dedicated splunk user:

sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk

• Now we are able to run the Splunk install, which involves accepting terms, and setting admin credentials (hint: use the space-bar to quickly scroll through terms & conditions):

$ /opt/splunkforwarder/bin/splunk start --accept-license
<...>
Do you agree with this license? [y/n]: y
Please enter an administrator username: forwarder
Password must contain at least:
   * 8 total printable ASCII character(s).
Please enter a new password: **************

• Our next step is to grant the splunk user read access to our /var/log/messages file as well as our /var/log/secure file using the setfacl command.
• We need to make sure the splunk user has access to the logs it will be ingesting:

  💡 If you are installing this forwarder onto our other Zeek host the folder for logs is /usr/local/zeek/logs/current/:

sudo setfacl -m g:splunk:r /var/log/messages
sudo setfacl -m g:splunk:r /var/log/secure

• Now let's walk back the file ACL we created earlier, since the splunk user performed the installation:

sudo setfacl -x u:splunk /opt/splunkforwarder/

• We also need to add Splunk to our $PATH variable, which we can do within ~./bashrc:

$ sudo vi ~/.bashrc export 
SPLUNK_HOME="/opt/splunkforwarder" 
export PATH=$PATH:$SPLUNK_HOME/bin 
$ exec bash

• Reboot your host using sudo reboot and confirm splunk is running using the right id (splunk):

ps -ef | grep splunk

• Just because the Splunk Universal Forwarder agent is technically installed and running doesn't mean it is logging anything, we need to give it properly formatted instructions on exactly what we'd like to monitor and where we'd like to forward that data.

💡 These configurations can be made locally, but they can also be completed in a more centralized manner through the use of a Deployment Server. Read more about deployment server architecture here

• For the purposes of this initial test we are just going to monitor the /var/log/messages file:

$ pwd
/opt/splunk/etc/deployment-apps/nix_server_inputs/local
$ vi inputs.conf
[monitor:///var/log]
disabled = 0
index = unix
whitelist=(messages)

• Start the configuration by adding your local Splunk log indexer as the forward server:

  💣 Be sure to use the credentials you defined earlier during the forwarder installation process, as the admin:changeme are defaults:

sudo su splunk
cd /opt/splunkforwarder/bin
./splunk ftr

• Ensure that the outputs.conf file found at /opt/splunkforwarder/etc/system/local/outputs.conf` is accurate, similar to the following:

[tcpout]
defaultGroup = splunk_hf

[tcpout:splunk_hf]
server = ip_of_splunk_server:9997

• We can also set the max throughput limit is set to zero, found within /opt/splunkforwarder/etc/system/default/limits.conf:

maxKBps = 0

• Now lets configure the folders we want to monitor/forward using inputs.conf

💡 I've created a much more robust inputs.conf file for Zeek network security monitoring which is uploaded here that forwards not only dns.log but also other important log files such as ssh.log and also http.log

• We have the option of using the built in command, or manually defining it using vim, both are shown below:

Splunk Built-In Command:
./splunk add monitor /path/to/your/logs

Manually Configure using Vim:

$ vi $SPLUNK_HOME/etc/system/local/inputs.conf
## example folder entry
[monitor:///var/log] 

• Lastly an optional solution to storage problems we could see down the road is simply a log rotation scheme:

• Edit /etc/logrotate.conf and add the following (this will rotate when the size hits 5G, the new log file it creates will be owned by splunk/splunk with 740 permissions, and we’ll keep up to 20 files)

$ chmod 740 /etc/logrotate.conf
$ sudo vim /etc/logrotate.conf
/var/log/udp514.log {
    size 1g 
    create 740 splunk splunk
    rotate 10
}

• Now let's restart our Universal Forwarder:

/opt/splunk/bin/splunk restart

• Use the splunk command to tell if our splunk forwarder is active and sending logs with the following:

$ ./splunk list forward-server
Active forwards:
         ip_of_your_splunk_server:9997
Configured but inactive forwards:
         None