title: Systemd logging nightmare? link: https://onemoretech.wordpress.com/2014/07/10/systemd-logging-nightmare/ author: phil2nc description: post_id: 7992 created: 2014/07/10 11:37:47 created_gmt: 2014/07/10 15:37:47 comment_status: closed post_name: systemd-logging-nightmare status: publish post_type: post

Systemd logging nightmare?

Not really, as long as you don't depend on the defaults in Fedora. Something broke in my Fedora 20 workstation at home and while checking things with journalctl I discovered it held messages from as far back as last Fall. After a few dead ends I finally found the explaination: Fedora's shipping default configuration for journald, the logger for systemd, had persistence enabled but didn't set any explicit limits on how much space the logs could take up (the built-in default is supposed to be 10% of disk, but the space being used in my case was way over that). With a little more searching I finally decided to set the limit to 100M, as a resonable compromise between the oft-recommended 50M (e.g. in this section of the Arch Wiki) and... infinity. The setting is found in /etc/systemd/journald.conf as the value for the SystemMaxUse parameter. Fedora ships with this parameter commented out, giving journald an unlimited amount of space to fill with logs (until the volume gets filled up). Here's a snippet from my modified config:

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=login
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
SystemMaxUse=100M

To make this change on your systems just uncomment SystemMaxUse and make the value 100M. Delete the old logs (if you don't need them for debugging). Finally, restart journald using "systemctl restart systemd-journald". You can find an informative and entertaining rant about systemd logging here that unfortunately doesn't mention journald.conf. Ironically, given that the lead developer is a Red Hat employee, documentation on systemd from the Fedora Project is sparse (chapters 6 and 18 of the new System Administrators Guide for RHEL 7 do provide some good information on systemd but lacks detail on journald). There are a wealth of links to helpful information over on the systemd information page at FreeDesktop.org. The FAQs are especially essential reading. As released with Red Hat Enterprise Linux 7, journald logging is not in persistent mode by default so the advice here will only apply if you've deliberately enabled persistence. In general I've found the Arch Wiki section on systemd and journald to be the most comprehensive treatment in one place, and so highly recommend it. Note: I'm not going to join in the seemingly endless dispute over the wisdom of implementing systemd. The major Linux vendors (including the until recently contrarian Ubuntu) have signed on and there doesn't seem to be any chance anyone will be turning back (although Red Hat's more conservative implementation of journald in RHEL 7 is an interesting development). In other words, the replacement of the old System V init scripts and text-based logging with systemd and journald are a fait accompli that Linux system administrators need to accept and work with.

Copyright 2004-2019 Phil Lembo