[!NOTE] Ticket #38: Centralised Logging with Puppet

If an issue occurs on a server, each machine holds its own system logs. Centralised logging allows us to avoid SSHing into each machine by moving it to a single server as they log.

Part 1: Module Structure and Manifests

Task 1: Verify prerequisites and create the module structure.

On mgmt-a, what is the current rsyslog version?

group-a@mgmt-a:~$ rsyslogd -v | head -2
rsyslogd  8.2112.0 (aka 2021.12) compiled with:
        PLATFORM:                               x86_64-pc-linux-gnu

 
version: 8.2112.0 

Is rsyslog already listening on any ports? Run sudo ss -ulnp | grep rsyslog and paste the output. (Expected: nothing — it does not yet listen for remote logs.)

group-a@mgmt-a:~$ sudo ss -ulnp | grep rsyslog

No output showing it's not running

Task 2: Write init.pp - the module entry point

sudo nano $MODPATH/logging/manifests/init.pp

# modules/logging/manifests/init.pp
# Manages centralised rsyslog configuration.
# mgmt-x acts as the log server; all other nodes are clients.
class logging {
contain logging::config
contain logging::service
# config must be applied before the service starts/restarts
Class['logging::config'] -> Class['logging::service']
# any config change triggers a service restart
Class['logging::config'] ~> Class['logging::service']
}

[!NOTE] rsyslog is pre-installed on Ubuntu 22.04 and 24.04, but adding the ensure => present for rsyslong is good practice in case it is uninstalled.

Task 3: Write config.pp

sudo nano $MODPATH/logging/manifests/config.pp

# modules/logging/manifests/config.pp
# Deploys the rsyslog drop-in configuration file.
# Uses an ERB template to configure the node as either a server or client
# based on whether it is mgmt-x (the Puppet/log server).
class logging::config (String $log_server = 'mgmt-x',) {
# Determine this node's role from its hostname fact
$is_server = ($facts['networking']['hostname'] == $log_server)
# On the server only: ensure the remote log directory exists
# Clients do not need this directory
if $is_server {
file { '/var/log/remote':
ensure => directory,
owner => 'syslog',
group => 'adm',
mode => '0750',
}
}
# Deploy the rsyslog drop-in config via ERB template
# This file is placed in rsyslog.d/ so it does not overwrite
# the main rsyslog.conf (which contains Ubuntu's own defaults)
file { '/etc/rsyslog.d/90-puppet-managed.conf':
ensure => file,
owner => 'root',
group => 'root',
mode => '0644',
content => template('logging/rsyslog.conf.erb'),
}
# On the server only: open port 514 in the firewall
# The 'unless' guard makes this idempotent
if $is_server {
exec { 'allow_syslog_tcp':
command => '/usr/sbin/ufw allow proto tcp from any to any port 514',
unless => '/usr/sbin/ufw status | grep -q "514/tcp"',
path => ['/usr/sbin', '/usr/bin', '/bin'],
}
}
}

Task 4: Write service.pp

sudo nano $MODPATH/logging/manifests/service.pp

# modules/logging/manifests/service.pp
# Ensures the rsyslog service is running and enabled at boot.
# The ~> arrow in init.pp handles restart-on-config-change.
class logging::service {
service { 'rsyslog':
ensure => running,
enable => true,
}

Part 2: The ERB Template

Task 5: Write the rsyslog ERB template

A different output dependent if the log is from a client (eg. db-a, app-a) or log server (mgmt-a)

`sudo nano $MODPATH/logging/templates/rsyslog.conf.erb``

1. Which lines will appear in 90-puppet-managed.conf on mgmt-x that will NOT appear on db-x?

group-a@mgmt-a:~$ cat /etc/hosts 127.0.0.1 localhost 10.2.0.4 mgmt-a.oe2.org.nz mgmt-a 10.2.0.5 app-a.oe2.org.nz app-a 10.2.0.6 backup-a.oe2.org.nz backup-a 10.2.0.7 db-a.oe2.org.nz db-a

The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts 127.0.1.1 mgmt-a group-a@mgmt-a:~$

2. What does the target line in the client block resolve to? The template uses @log_server, which comes from the $log_server class parameter in config.pp. What is its default value, and how does /etc/hosts (configured in Lab 2.2) make this short hostname resolvable without requiring a DNS server?

because we have this set it doesn't require dns:

10.2.0.4 mgmt-a.oe2.org.nz mgmt.-a

3. Why is TCP preferred over UDP for log forwarding in a monitoring context? What happens to UDP log messages if the server is temporarily unavailable?

TCP checks if the message is delivered while UDP just sends the message and isn't worried if it's revived. If the server is unavailable the message will be lost

Part 3: Apply the Module and Verify

Task 6: Add the module to site.pp

Task 7: Apply to mgmt-a first, then all agents

1. Configuration for mgmt-a

2. Refreshing certificates for client servers.

db-a:

app-a:

backup-a:

3. Send a test log from the client server

4. Logs should forward to mgmt-a

Task 8: Verify live log forwarding between nodes

Part 4: Interpreting Aggregated Logs

Task 9: Read and interpret real log entries from multiple nodes

---

1. On db-x, manually delete the rsyslog drop-in config:

2. Run the Puppet agent on db-a:

3. Verify forwarding is restored:

---

1. When you deleted the config file and ran Puppet, what did the agent output show?
Did it recreate the file and restart rsyslog?

group-a@db-a:~$ sudo puppet agent --no-daemonize --verbose --onetime
Info: Refreshing CA certificate
Info: CA certificate is unmodified, using existing CA certificate
Info: Refreshing CRL
Info: CRL is unmodified, using existing CRL
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: Requesting catalog from mgmt-a.oe2.org.nz:8140 (10.2.0.4)
Notice: Catalog compiled by mgmt-a.oe2.org.nz
Info: Caching catalog for db-a.oe2.org.nz
Info: Applying configuration version '1773873205'
Notice: /Stage[main]/Logging::Config/File[/etc/rsyslog.d/90-puppet-managed.conf]                            /ensure: defined content as '{sha256}eff9cdfc92d26580b7496a069cf3f2d89dbd729a852                            88eec9fd76c884a364ae4' (corrective)
Info: Class[Logging::Config]: Scheduling refresh of Class[Logging::Service]
Info: Class[Logging::Service]: Scheduling refresh of Service[rsyslog]
Notice: /Stage[main]/Logging::Service/Service[rsyslog]: Triggered 'refresh' from                             1 event
Notice: Applied catalog in 0.85 seconds

2. In a real production environment with 50 servers, a sysadmin might delete the forwarding
config on one machine to “temporarily stop log noise”. Without Puppet, how
would you discover this had happened? With Puppet, what happens automatically?

3. How does having all logs centralised on mgmt-x change the way you would investigate
an incident compared to SSHing into each node separately?

1. Module tree — output of find $MODPATH/logging -type f.

group-a@mgmt-a:~$ find $MODPATH/logging -type f
/opt/puppetlabs/puppet/modules/logging/templates/rsyslog.conf.erb
/opt/puppetlabs/puppet/modules/logging/manifests/config.pp
/opt/puppetlabs/puppet/modules/logging/manifests/init.pp
/opt/puppetlabs/puppet/modules/logging/manifests/service.pp

2. All manifests — contents of init.pp, config.pp, service.pp.

3. ERB template — contents of templates/rsyslog.conf.erb.

4. Rendered configs — cat /etc/rsyslog.d/90-puppet-managed.conf from mgmt-a
and from db-a side by side, showing the server vs client difference.