Centralized Logging Server on RHEL 8 using Rsyslog - cfloquetprojects/homelab GitHub Wiki

Introduction

A staple of many enterprises today is the use of Red Hat Enterprise Linux (RHEL), with the latest iteration of this OS being RHEL 8, which we will be working with today.

I've already created a Red Hat Developer Account, which is not only free, but includes up to 16 different licenses for personal use, which is more than enough for us to get by in our lab environment.

Next we will be configuring Rsyslog as our method of sending and ingesting logs from other nodes on our network. This is a highly configurable, "rocket" fast system for log processing which will allow us to parse logs of nearly all sources and formats.

Pre-Flight Check

As a matter of principle, unfortunately RHEL does require us to authenticate to our developer account before being able to fetch supported packages or updates out of the box.

This can be easily done with the following commands, leveraging subscription-manager as root enter the following commands:

# subscription-manager register
*enter developer account credentials*
# subscription-manager attach --auto
# subscription-manager status

Setting up our .conf files:

We will be using a .conf file on both our client and server, which will define which logs we would like to forward, how they will be forwarded, and how they will be stored on the centralized logging server.

Let's start by adding a .conf file to our client within /etc/rsyslog.d/, in our case we will be using a file I found and updated that I have uploaded for use here

Next we can add a shorter but more specific/granular .conf file to /etc/rsyslog.d/ on our centralized logging server, a good template for which I've also uploaded for public use here

That's it! As always, we can expand/change these .conf files to better fit our needs in the future, but for right now let's press on to permit these logs to be sent/ingested using the semanage utility...

Configure Firewall and SELinux to Allow Log Traffic:

We will need to configure both our local firewall, as well as the SELinux package included within RHEL (if you're distro doesn't include the semanage command I've included the command below to install it) to allow for traffic logs to be accepted from external systems.

First let's check that the semanage utility is installed, and if it's not, we can add it with the following:

~]# yum -y install policycoreutils-python

An important step in the process of setting up/configuring a syslog client/server relationship is simply confirming a working connection between the two distinct parties by temporarily disabling SELinux:

~]# getenforce 
1
~]# setenforce 0
~]# systemctl restart rsyslog

You should now see logs appearing within /var/log/remote-syslog/, which means that your log flow is currently working, congratulations! Now using setenforce re-enable SELinux, and we can proceed creating permissive rules which will allow SELinux to accept our log traffic.

Let's look at what ports SELinux allows by default for hosts supporting syslogd:

~]# semanage port -l | grep syslogd
syslogd_port_t                 tcp      6514, 601
syslogd_port_t                 udp      514, 6514, 601

We can see that 514/udp is allowed, but we will need to allow 514 for both TCP and UDP before continuing.

Rather than using the -a flag for adding a rule, we will use the -m flag to modify an existing rule on both the server and the client hosts:

~]# semanage port -m -t syslogd_port_t -p tcp 514

Now we will authorize the /etc/rsyslog.d/ directory in SELinux, which will allow it to read our .conf file:

~]# semanage fcontext -a -t syslog_conf_t "/etc/rsyslog.d/"
~]# restorecon -R -v /etc/rsyslog.d/
~]# semanage fcontext -a -t etc_t "/etc/rsyslog.d"
~]# restorecon -v /etc/rsyslog.d

All that is left to do at this point is creating granular firewall rules on the centralized logging server to allow the inbound traffic:

~]# firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address="123.123.123.0/24" port port="514" protocol="tcp" accept'
~]# firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address="123.123.123.0/24" port port="514" protocol="tcp" accept'