Adding ADCS SSL Certificates to Splunk Enterprise - cfloquetprojects/homelab GitHub Wiki

In previous labs we have installed both our Root CA as well as our Subordinate CA onto standalone and domain-joined (enterprise) hosts, respectively.

We have also spent time installing and configuring Splunk to run on a RHEL 7 instance as a non-root user, the guide that I made for which is linked here.

I will be making a guide for setting up SAML authentication for easier authentication and user tracking shortly, however within the context of this lab we will only be discussing the process of configuring SSL certificates for Splunk ES using our existing two-tier ADCS infrastructure.

As I mentioned in the introduction, I've covered the requisite steps up until this point in guides that I've outlined in the correct order below:

Deploying an Offline Root CA on Windows Server 2019

Installing Subordinate Issuing Certificate Authority on Windows Server 2019

Configuring Auto-Enrollment for ADCS

Employing ADCS to Secure Web Server Communciations

Additionally, you will need to complete the installation of Splunk ES on a RHEL based system prior to starting this lab, a guide for which can be found here.

It's also good practice to make sure all of your systems are properly patched and updated prior to moving onto the next steps in this lab.

When we navigate to our Splunk instance at https://<splunk-ip>:8000 we are greeted with a nasty warning about an untrustworthy or insecure web page, since it is technically using a self-signed certificate rather than a CA-issued one.

#1

We will change this in the following steps by first enabling SSL based connections to our Splunk ES instance, followed be changing the self-signed certificate to a more trustworthy CA signed certificate that we will use as an additional authentication measure for the web server.

Using SSH, or whatever remote command line interface technology you prefer for RHEL server management, generate a new private key for the Splunk instance with the following command:

[splunk@splhf01 ~]$ openssl genrsa -aes256 -out spl_private.key 2048

Set whatever password you'd like for your private key, but it will only be temporary, because Splunk requires that this key not be encrypted with a password. The commands shown below will first remove the password that was required to be set during initial creation, and then displays the contents of the .key file for verification purposes.

💡 If you're concerned about the implications of this requirement, Splunk has written an article about this practice here that you can refer to.

[splunk@splhf01 ~]$ pwd
/opt/splunk/bin/
[splunk@splhf01 ~]$ ./splunk cmd openssl rsa -in spl_private.key -out spl_private.key
[splunk@splhf01 ~]$ ./splunk cmd openssl rsa -in mySplunkWebPrivateKey.key -text

Next up we need to generate our .csr to send to our Subordinate CA for approval, most modern browsers require several fields to be present within the request that we submit, and it's often easier to store these in a static *.cnf file rather than write a long one-liner with many flags/options appended.

While this is not optional, it's often easier for tracking and generating new CSRs in the future if we create a dedicated directory to store the various files associated with generating a CSR that will be used with Splunk.

$ mkdir ssl
$ cd ssl
$ vi req.cnf
<..>
[ req ]
default_bits       = 2048
distinguished_name = req_distinguished_name
req_extensions     = req_ext
prompt = no

[ req_distinguished_name ]
countryName                = <country abbrev>
stateOrProvinceName        = <state abbrev>
localityName               = <city>
organizationName           = <orgname>
commonName                 = <FQDN of Splunk server>

[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1   = <FQDN of Splunk server>

Now we are able to generate the CSR using the .cnf file we just wrote with the following using openssl:

$ openssl req -out splunk_cert.csr -key spl_private.key -config san.cnf

We have now generated our certificate request that we will send to our issuing CA for approval, after which we can use that certificate for our Splunk web server. Display the contents of the .csr that we will be sending to our Subordinate CA via web request using the command shown below:

[splunk@splhf01 ~]$ cat splunk.csr
-----BEGIN CERTIFICATE REQUEST-----
MIICuDCCAaACAQAwczELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAk1BMQ8wDQYDVQQH
<..>

As mentioned, copy the certificate displayed from that command and open up the Microsoft Active Directory Certificate Services (ADCS) to request a new certificate using that .csr file.

While not required for this lab, if you have properly followed the steps from the other articles leading up to this point, your connection to your ADCS web server should be over HTTPS, shown below.

#2

Once the certificate has downloaded, transfer it back to your Splunk instance in whichever way meets the needs of you/your organization. I have chosen to use scp, but there are also GUI-based tools like WinSCP at your disposal should you choose.

PS> scp certnew.cer cfloquet@splhf01:

Now that we have the .cer file back on our requesting host, we can again use openssl to convert the cert to the .pem format, as required by Splunk, we also need to create a directory to store both our private key as well as our .pem file for the Splunk web server, I named mine ssl and stored it under /opt/splunk/etc/auth/ssl.

[splunk@splhf01 ~]$ openssl x509 -in certnew.cer -inform DER -out splunk.pem -outform PEM
[splunk@splhf01 ~]$ mkdir -p /opt/splunk/etc/auth/ssl

Move both the .pem as well as the .key file to the ssl directory at /opt/splunk/etc/auth/ssl:

[splunk@splhf01 ssl]$ pwd
/opt/splunk/etc/auth/ssl
[splunk@splhf01 ssl]$ mv ~/splunk.pem .
[splunk@splhf01 ssl]$ mv ~/spl_private.key .

Now that we have our certificate set, we need to inform the splunk web.conf file that we'd like to use SSL and specify the certificate we will be using for it.

Create the web.conf file under /opt/splunk/etc/system/local/ and add the following configuration to it:

[splunk@splhf01 ~]$ cat /opt/splunk/etc/system/local/web.conf
[settings]
enableSplunkWebSSL = true
privKeyPath = /opt/splunk/etc/auth/ssl/spl_private.key
serverCert = /opt/splunk/etc/auth/ssl/splunk.pem

Now all that's left to do is restart our splunk instance and check that our configuration applied successfully:

[splunk@splhf01 ~]$ /opt/splunk/bin/splunk restart