Install Splunk Enterprise v9.0.0 on RHEL 7 as Non Root User - cfloquetprojects/homelab GitHub Wiki

Introduction

  • On June 16th, 2022 Splunk disclosed a nasty vulnerability (CVE-2022-32158) affecting all existing Splunk Enterprise Deployment Servers (prior to the most recently version v9.0.0) which could be used to execute arbitrary code on any & all clients connecting to vulnerable assets, fetching a CVSS Meta Temp Score of 8.3 by VulnDB.

  • The security advisory by Splunk states that the only mitigation/fix is to upgrade Splunk Enterprise deployment servers to v9.0.0 or higher immediately.

  • In light of this disappointing news, I have decided to update my article on installing Splunk Enterprise to exclusively reflect the configs for v9.0.0, since prior versions will fall out of use soon as more vulnerabilities are disclosed. Later labs will cover the use of custom APIs, integrations, and apps that we will implement to achieve more value from these tools.

  • We will be performing the initial installation steps for Splunk as a dedicated splunk user, rather than running through the commands using the root account. Installing any third party program onto a Linux machine as a root user should always be considered very carefully from a security perspective, if not outright disallowed.

Resources:

Pre-Flight Check

💡 We will be registering our host for updates, so ensure you have created a valid Red Hat Developer Account, which is not only free, but includes up to 16 different licenses for personal use, which is more than enough for us to get by in our lab environment.

  • Now that we have our Red Hat Developer Account created, we can link it to our host for updates, and attach the account to complete the linkage using the commands below:
# subscription-manager register
*enter developer account credentials*
# subscription-manager attach --auto
# subscription-manager status
  • Now that we have our host registered, we can update our system using yum -y update, and while it's updating make sure you read on about creating a Splunk.com account for downloading the free enterprise installer.

💡 I've also included an install command for utilities we will be using later in this guide, these are explicitly required for the installation of Splunk, but helpful nevertheless.

# yum -y update
# yum -y upgrade
# yum install -y zip unzip wget tree
  • While we are still root user, it's important that we create our designated splunk user, and use the setfacl to set our Splunk installation directory access controls (/opt/splunk):
# useradd splunk
# setfacl -m u:splunk:rwx /opt
# getfacl /opt

💡 It'd also be a good idea to create your own Splunk.com account using their Get Started page, this will allow us to obtain links to the latest free version of Splunk Enterprise.

  • As of this writing, the current version of Splunk Enterprise is v9.0.0, however you should use your newly created Splunk.com account to check out their free download for Splunk Enterprise, or follow the command-line installation I will walk through below.

Installing Python v3.8 using Subscription Manager:

  • We need to be root to use subscription-manager to enable the correct repo before installing Python v3.8
# subscription-manager repos --enable=rhel-server-rhscl-7-rpms
# yum install -y rh-python38.x86_64
  • Now that we have installed Python v3.8, we need to enable it within /etc/profile.d/rh-python38.sh for it to work permanently:
# vi /etc/profile.d/rh-python38.sh
# cat /etc/profile.d/rh-python38.sh
#!/bin/bash
source scl_source enable rh-python38
# reboot

Downloading Splunk and Installing Requisite Packages:

  • We will be using tar to install Splunk from a .tgz file that you can either use a tool like wget to fetch from a web-hosted source, or download and scp separately to the host.

💣: Ensure that you have followed the directions shown above that included creating the dedicated splunk user, as well as setting the proper access controls for the /opt directory using setfacl, or else the installation will fail!

  • Our distribution of RHEL 7 does not come with wget installed, and to follow best security practices I have chosen to separately copy the requisite files from a workstation, but I will include directions for getting the .tgz file in both ways.

Manual Copying from External Windows Workstation:

  • If you haven't already, download the latest version of Splunk from their free download page for Splunk Enterprise:

  • Once you have both the .tgz as well as the associated md5 hash file for verifying the integrity of the installation, you can run the following command from a PS terminal on your windows workstation:

💡: I have chosen to download and store both the .tgz as well as the .md5 file in the same SplunkDL folder, before using the Compress-Archive function within powershell to zip that folder before sending it to our splhf01 host.

PS> Compress-Archive -Path .\SplunkDL\ -DestinationPath .\SplunkDLComp.zip
PS> scp -r SplunkDLComp.zip user@splunkHWF01:

Using wget to fetch .tgz from Splunk Website:

  • We can use wget to pull down a specific version of Splunk defined within the URL, ensure this is the latest version for Splunk Enterprise by checking the download page here, as of this writing the version is v9.0.0, so the command for that version is as follows:
# sudo su - splunk
$ wget -O splunk-9.0.0-x86_64.tgz 'https://download.splunk.com/products/splunk/releases/9.0.0/linux/splunk-9.0.0-6818ac46f2ec-Linux-x86_64.tgz'

💡 It's a good idea to download the MD5 checksum from Splunk's website as well for the specific version you downloaded, and check it against the MD5 of the .tgz file that you retrieved using wget, this can be done by viewing the MD5 hash of the .tgz using md5sum, as shown below: $ md5sum splunk-9.0.0-x86_64.tgz

  • Now that we have Splunk downloaded, we need to extract it into the proper directory, which our splunk user should already have the requisite permissions for if we have followed the directions correctly up to this point.
$ whoami
splunk
$ tar -xvzf splunk-9.0.0-x86_64.tgz -C /opt

Begin Installation of Splunk Enterprise

  • Navigate to the installation directory (/opt/splunk) and execute the following commands as splunk to complete the installation. This will require you to define the login credentials for the local splunk application administrator, which should be stored in a secure location after the initial installation.
$ cd /opt/splunk/bin
$ ./splunk ftr --no-prompt --accept-license

Generating Splunk Enterprise Admin Credential Set

  • Since we opted not to manually create a credential set, we should do that now using the openssl utility built into our newly installed Splunk Enterprise cmd feature.
  • Generating the Splunk Enterprise admin credential set using user-seed.conf is currently the most secure way of creating privileged credentials. Other methods often introduce security risks in the form of accessing command-line history, or even process output.

    💡 The method shown below first creates a random complex 16 character password, before automatically generating the user-seed.conf file using that credential and storing it as well as a backup to be saved externally for future reference, since the user-seed.conf file will be removed upon restart.

$ complexPW=$(/opt/splunk/bin/splunk cmd openssl rand -base64 12)
$ echo -e "[user_info]\nUSERNAME = spladmin\nPASSWORD = $complexPW" | tee /opt/splunk/etc/system/local/user-seed.conf /opt/splunk/etc/system/local/user-seed.conf.backup > /dev/null

Completing Installation of Splunk Enterprise v9.0

  • Before we go any further, let's go back and walk back our permissions for the entire /opt directory, which is overly permissive now that we already have permissions just for /opt/splunk since that user extracted the file there.
$ sudo -i
# setfacl -x u:splunk /opt
# getfacl /opt
  • Our last action as the root user is to enable Splunk to start on boot under our splunk user:
# /opt/splunk/bin/splunk enable boot-start -user splunk
  • Now that we have enabled boot start and corrected the permissions for our installation directory, let's confirm a working install of Splunk Enterprise v9.0 by issuing the following commands within $SPLUNK_HOME/bin/:
$ ./splunk start
$ ./splunk status

Hardening Splunk Enterprise v9.0 Instance

  • We will be using a static server.conf file to disable our splunkd service from listening on port 8089/tcp, since this is only necessary for deployment servers. We will do this by essentially overriding this default setting and configuring Splunk to just bind to our local loopback interface.

    💣 Keep in mind if you are configuring this setting on a deployment server you will break critical functionality, this should only be done on heavy/universal forwarders.

[httpServer]
disableDefaultPort = true

[httpServerListener:127.0.0.1:8089]
ssl = true
  • Now that we have our splunkd listening issue resolved, we can start and enable firewalld on our server and create some firewall rules to allow for web access to our Splunk application.

    💡 If you are configuring a deployment server here, make sure you enable either 8089/tcp or a different port dedicated for fielding splunkd communications from clients.

$ sudo firewall-cmd --set-default-zone=public
$ sudo firewall-cmd --add-service=https --permanent

Lastly, we can add a rule that redirects normal HTTPS traffic to the necessary 8000 socket used for Splunk web, which cleans things up a bit for the user on the client side of things.

$ sudo firewall-cmd --zone=public --add-forward- port=443:proto=tcp:toport=8000 --permanent
$ sudo firewall-cmd --reload

At this point after a reboot you should be all set to run and manage Splunk enterprise from your non-root splunk account, and access the web dashboard from https://yourSplunkServerDN.domain.com:8000

💡 In future labs, we will investigate using our established two-tier PKI infrastructure in order to better secure the communications between the different pieces of our Splunk infrastructure, namely Universal/Heavy Forwarders and Deployment Servers.