Log - ether42/bootable-usb GitHub Wiki

LXC

Create a storage volume for the logs:

lvcreate -L 4GiB -n logs lxc
mkfs.ext3 -L logs /dev/lxc/logs

Attach it to the LXC:

lxc.mount.entry = /dev/lxc/logs var/log/receive ext3 defaults,create=dir 0 2

Setup

On the rootfs:

apt-get install --no-install-recommends rsyslog logrotate

Configure the clients to send their logs to the log server (but do not forward the logs from the log server to itself or it would create a loop), /etc/rsyslog.d/forward.conf:

*.* @log.administrative.lorn.space

All rules could be commented from the default rsyslog.conf.

Even if UDP (@ syntax, TCP being @@) is unreliable, it is best to use it for logs as any problem (be it on the network or on the log server) may block proper execution of any syslogging program on all the clients.

Configure the log server to listen for incoming logs: /etc/rsyslog.d/receive.conf:

module(load="imudp")
input(type="imudp" port="514" device="administrative")
template(name="DynFile" type="string" string="/var/log/receive/%HOSTNAME%/%PROGRAMNAME%.log")
*.info ?DynFile

/etc/logrotate.d/receive:

/var/log/receive/*/*.log
{
  rotate 7
  size 10M
  daily
  missingok
  notifempty
  compress
  postrotate
    invoke-rc.d rsyslog rotate > /dev/null
  endscript
}

The receive directory is specified to ensure the logrotate rules won't clash with pre-existing ones (for example /var/log/apt/history.log defined in /etc/logrotate.d/apt would also match /var/log/*/*.log and logrotate would refuse to execute).

The logrotate configuration may be tested via (remove the debug option to execute it):

logrotate -df /etc/logrotate.d/receive

Optionally, systemd-journald's may be told to not write to disk, /etc/systemd/journald.conf (the default auto won't create /var/log/journal, use none to disable the journal completely but some early logs may be missed):

[Journal]
Storage=volatile

Finally, remove any local logs from the rootfs and the clients as they are not needed anymore, take some space and only {b,w,u}tmp should be present in overlays (as it's not possible to forward those).

Resolving error

There is a weird problem with the resolvconf package. Debian's libc should normally watch /etc/resolv.conf for changes.

Rsyslog, starting early at boot, won't be able to resolve the remote log server and will wait 30 seconds before retrying. However, even after /etc/resolv.conf has been updated by resolvconf, rsyslog will still encounter resolving failures which will go immediately away with a simple touch /etc/resolv.conf. The problem doesn't happen when resolvconf isn't installed and /etc/resolv.conf is a simple file.

The easiest way to debug rsyslog is to add to its environment some debugging variables, /etc/systemd/system/rsyslog.service.d/override.conf:

[Service]
Environment=RSYSLOG_DEBUGLOG=/tmp/debug
Environment=RSYSLOG_DEBUG=Debug