1. Introduction
   • What is journalctl?
2. Basic Commands
   • Getting Help
   • Viewing All Logs
   • Viewing Logs in Reverse Order (Newest First)
   • Viewing the Last N Log Entries
   • Following Logs in Real Time
3. Filtering Logs
   • Viewing Logs for a Specific Service
   • Viewing Only Error-Level Logs
   • Viewing Only Kernel Messages
   • Filtering by Time: Today's Logs
   • Filtering by Specific Time Range
4. Boot Troubleshooting

• Viewing Logs from Current Boot
• Viewing Logs from Previous Boot
• Listing All Boot Sessions

19. Maintenance & Disk Usage

• Checking Log Disk Usage
• Cleaning Logs by Age
• Limiting Logs by Size

23. Combining Filters

• Examples of Combined Filters

25. Common Troubleshooting Scenarios

• Scenario 1: Service Won't Start
• Scenario 2: System Just Crashed
• Scenario 3: Something Happened at a Specific Time
• Scenario 4: Monitoring a Service in Real Time
• Scenario 5: Disk Space is Low

31. Additional Tips

• Saving Logs to a File
• Searching Within Logs
• Exporting Logs in Different Formats

35. Advanced Filtering Options

• Filtering by User
• Filtering by Executable or Process
• Filtering by Process ID (PID)
• Filtering by Systemd Unit Type
• Filtering by Log Field

41. Understanding Log Entry Structure

• What Information is in Each Log Entry?

43. Using grep with journalctl

• Basic grep Usage
• Advanced grep Patterns

46. Remote Log Viewing

• Viewing Logs from Remote Systems

48. Understanding Systemd Unit States from Logs

• Common Service States

50. More Troubleshooting Scenarios

• Scenario 6: Service Keeps Crashing
• Scenario 7: Slow System Performance
• Scenario 8: Network Connectivity Issues
• Scenario 9: Permission Denied Errors
• Scenario 10: Finding When a Service Last Started

56. Performance Tips for Large Logs

• Speeding Up Searches

58. Configuring Journal Logging

• Understanding journald.conf

60. Interpreting Common Error Messages

• Understanding Common Log Patterns

62. Additional Useful Commands

• Viewing Logs with No Pager
• Viewing Logs with Cursor Support
• Counting Log Entries

66. Summary

This guide will help you learn how to use journalctl, a powerful tool for viewing and troubleshooting system logs on Linux systems that use systemd (which includes most modern Linux distributions like Fedora, Arch Linux, Debian, Ubuntu, and others).

journalctl is the command-line tool used to read logs collected by systemd-journald, the systemd journal service. It provides a centralized way to view and analyze system logs.

What logs does it show?

• Kernel messages: Low-level system messages from the Linux kernel
• System services: Logs from services like nginx (web server), sshd (SSH server), libvirtd (virtualization), and many others
• User applications: Logs from applications running on your system
• Boot/shutdown events: Information about when your system started, what happened during boot, and shutdown events

Why is this useful?

• Troubleshooting: When something goes wrong, logs tell you what happened
• Monitoring: Keep an eye on system health and service status
• Debugging: Understand why services failed or applications crashed
• Security: Check for suspicious activity or unauthorized access attempts

What is systemd-journald?

• systemd-journald is a system service that collects and stores log messages
• It runs in the background and automatically captures logs from all over your system
• It stores logs in a binary format for efficient storage and fast searching
• It's part of the systemd suite (the system and service manager used by most modern Linux distributions)

The first command to learn is how to get help:

journalctl --help

What this does:

• Displays a comprehensive list of all available journalctl options and commands
• Shows syntax and usage examples
• This is your reference guide when you need to remember a command

When to use it:

• When you forget a command syntax
• When you want to explore available options
• When you need to understand what a flag does

To see all logs from oldest to newest:

journalctl

What this does:

• Shows all log entries collected by systemd-journald
• Displays logs in chronological order (oldest first)
• Uses a pager (usually less) so you can scroll through logs

What you'll see:

• Each log entry shows:
• Timestamp: When the event occurred
• Hostname: Your computer's name
• Service/Component: What generated the log (e.g., systemd, kernel, a service name)
• Message: The actual log message

Example output:

-- Logs begin at Mon 2025-01-15 10:00:00 EST, end at Mon 2025-01-15 15:30:00 EST. --
Jan 15 10:00:01 mycomputer systemd[1]: Starting systemd-journald.service...
Jan 15 10:00:02 mycomputer kernel: Linux version 6.1.0...
Jan 15 10:00:03 mycomputer systemd[1]: Started systemd-journald.service.
...

Navigation:

• Space: Scroll down one page
• Page Down: Scroll down one page
• Page Up: Scroll up one page
• q: Quit and return to terminal
• /searchterm: Search for text (press Enter, then n for next match)

Note: This can show a LOT of logs. Use filtering (covered below) to narrow it down!

To see the most recent logs first:

journalctl -r

What this does:

• -r stands for "reverse"
• Shows logs in reverse chronological order (newest first)
• Most useful when you want to see what just happened

When to use it:

• When troubleshooting a recent problem
• When you want to see the latest system activity
• When checking if a service just started or stopped

Example use case: If your system just had an error, this command shows the most recent events first, making it easier to find what went wrong.

To see only the most recent log entries:

journalctl -n 20

What this does:

• -n stands for "number" or "lines"
• 20 is the number of log entries to show
• Shows only the last 20 log entries (you can change the number)

What you'll see: Only the most recent 20 log entries, making it much easier to read than viewing all logs.

Customizing the number: You can change 20 to any number:

• journalctl -n 50 - Shows last 50 entries
• journalctl -n 100 - Shows last 100 entries
• journalctl -n 10 - Shows last 10 entries

When to use it:

• Quick check of recent system activity
• Seeing what happened in the last few minutes
• Getting a snapshot of current system status

Example output:

Jan 15 15:30:15 mycomputer systemd[1]: Started NetworkManager.service.
Jan 15 15:30:16 mycomputer NetworkManager[1234]: <info>  [1736965816.1234] manager[0x...]: NetworkManager state is now CONNECTED_GLOBAL
Jan 15 15:30:17 mycomputer kernel: [12345.678] usb 1-2: new high-speed USB device...

To watch logs as they happen (like watching a live feed):

journalctl -f

What this does:

• -f stands for "follow" (similar to tail -f)
• Shows new log entries as they appear in real time
• Continuously updates the display with new logs

What you'll see:

• Logs appear on screen as they're generated
• The display updates automatically
• You'll see system activity happening live

When to use it:

• Monitoring a service that's starting up
• Watching for errors as they occur
• Debugging a problem that's happening right now
• Observing system behavior during a specific action

How to stop:

• Press Ctrl + C to stop following and return to the terminal

Example use case: Start a service, then run journalctl -f to see all the log messages it generates in real time.

Tip: You can combine this with filtering (see below) to follow logs for a specific service:

journalctl -f -u sshd

This follows logs only for the SSH service.

To see logs only for a particular service:

journalctl -u sshd

What this does:

• -u stands for "unit" (systemd services are called "units")
• sshd is the service name (SSH daemon)
• Shows only log entries from that specific service

What you'll see: Only log messages related to the SSH service, making it much easier to troubleshoot SSH issues.

Common service names:

• sshd - SSH server
• nginx - Nginx web server
• apache2 or httpd - Apache web server
• libvirtd - Virtualization daemon
• NetworkManager - Network management service
• systemd - System and service manager

Finding service names: If you're not sure of the exact service name:

systemctl list-units --type=service

This lists all available services.

Example output:

Jan 15 10:00:01 mycomputer sshd[1234]: Server listening on 0.0.0.0 port 22.
Jan 15 10:05:23 mycomputer sshd[5678]: Accepted publickey for user from 192.168.1.100
Jan 15 10:05:24 mycomputer sshd[5678]: pam_unix(sshd:session): session opened

When to use it:

• Troubleshooting a specific service that's not working
• Monitoring a service's activity
• Checking if a service started successfully
• Finding errors related to a particular service

To see only error messages and warnings:

journalctl -p err

What this does:

• -p stands for "priority" or "priority level"
• err means "error" level and above
• Shows only log entries at error priority or higher (errors, critical, alert, emergency)

Priority levels (from highest/most severe to lowest/least severe):

• 0 or emerg - Emergency: system is unusable (highest priority)
• 1 or alert - Action must be taken immediately
• 2 or crit - Critical conditions
• 3 or err - Error conditions
• 4 or warning - Warning conditions
• 5 or notice - Normal but significant conditions
• 6 or info - Informational messages
• 7 or debug - Debug-level messages (lowest priority)

Other useful priority filters:

• journalctl -p warning - Shows warnings and above (warnings, errors, critical, etc.)
• journalctl -p crit - Shows only critical errors and above
• journalctl -p info - Shows info level and above (most logs)

When to use it:

• Quickly finding system errors
• Troubleshooting problems (errors usually indicate what went wrong)
• Monitoring system health
• Identifying issues that need attention

Example output:

Jan 15 14:30:15 mycomputer kernel: [12345.678] ACPI Error: [\_SB_.PCI0.LPC0.EC0] Namespace lookup failure
Jan 15 14:35:22 mycomputer systemd[1]: Failed to start nginx.service: Unit nginx.service failed to load
Jan 15 14:40:10 mycomputer NetworkManager[1234]: <error> [1736965810.1234] device (eth0): activation failed

To see only messages from the Linux kernel:

journalctl -k

What this does:

• -k stands for "kernel"
• Shows only kernel messages (also called "dmesg" messages)
• These are low-level system messages from the Linux kernel

What are kernel messages?

• Hardware detection and initialization
• Driver loading and errors
• System-level errors and warnings
• Low-level system events

When to use it:

• Troubleshooting hardware issues
• Checking if drivers loaded correctly
• Finding kernel-level errors
• Debugging boot problems
• Monitoring hardware events

Example output:

[    0.000000] Linux version 6.1.0-arch1-1 (linux@archlinux) (gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39.0) #1 SMP PREEMPT_DYNAMIC
[    0.123456] Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=xxxx-xxxx rw
[    0.234567] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    1.345678] usb 1-1: new high-speed USB device number 2 using xhci_hcd

Alternative command: You can also use dmesg to view kernel messages, but journalctl -k provides better formatting and filtering options.

To see only logs from today:

journalctl --since today

What this does:

• --since specifies the start time
• today means "from the start of today until now"
• Shows only log entries from today

When to use it:

• Checking today's system activity
• Troubleshooting problems that happened today
• Reviewing what the system did today
• Getting a daily summary

Other time options:

• journalctl --since yesterday - Logs from yesterday
• journalctl --since "1 hour ago" - Logs from the last hour
• journalctl --since "30 minutes ago" - Logs from the last 30 minutes
• journalctl --since "2025-01-15" - Logs from a specific date

Example output: Shows only logs from today, making it easier to find recent events without scrolling through days of old logs.

To see logs between specific times:

journalctl --since "2025-09-25 09:00" --until "2025-09-25 12:00"

What this does:

• --since specifies the start time
• "2025-09-25 09:00" is the start time (September 25, 2025 at 9:00 AM)
• --until specifies the end time
• "2025-09-25 12:00" is the end time (September 25, 2025 at 12:00 PM)
• Shows only logs between these two times

Time format:

• Date format: YYYY-MM-DD (year-month-day)
• Time format: HH:MM (24-hour format) or HH:MM:SS (with seconds)
• You can also use: "2025-09-25 09:00:00" for more precision

When to use it:

• Investigating a problem that happened at a specific time
• Reviewing system activity during a particular period
• Correlating logs with a known event time
• Creating a timeline of events

Example use case: If you know your system crashed at 10:30 AM on September 25th, you can view logs from 9:00 AM to 12:00 PM to see what led up to the crash and what happened after.

Other time formats:

• "yesterday 14:00" - Yesterday at 2:00 PM
• "2 hours ago" - Two hours ago
• "2025-01-15" - All of January 15, 2025
• "09:00" - Today at 9:00 AM (if no date specified, assumes today)

To see only logs from the current boot session:

journalctl -b

What this does:

• -b stands for "boot"
• Shows only log entries from the current boot session
• Excludes logs from previous boots

What you'll see:

• Boot process messages
• Services starting up
• System initialization
• Everything that happened since you last turned on your computer

When to use it:

• Troubleshooting boot problems
• Checking if services started correctly on boot
• Reviewing what happened during system startup
• Getting a clean view of current session (no old logs)

Example output:

-- Logs begin at Mon 2025-01-15 10:00:00 EST (current boot). --
Jan 15 10:00:01 mycomputer kernel: Linux version 6.1.0...
Jan 15 10:00:02 mycomputer systemd[1]: Starting systemd-journald.service...
Jan 15 10:00:03 mycomputer systemd[1]: Started systemd-journald.service.
...

To see logs from the boot before the current one:

journalctl -b -1

What this does:

• -b stands for "boot"
• -1 means "previous boot" (one boot before current)
• Shows logs from the boot session before your current one

When to use it:

• Investigating a crash or problem from a previous session
• Checking what happened during the last boot
• Troubleshooting issues that occurred before you rebooted
• Reviewing system behavior from a previous session

Other boot numbers:

• journalctl -b -2 - Two boots ago
• journalctl -b -3 - Three boots ago
• journalctl -b 0 - Current boot (same as -b)
• journalctl -b 1 - First boot in the list (oldest available)

Example use case: If your system crashed and you had to reboot, use this command to see what happened in the boot session that crashed.

To see a list of all available boot sessions:

journalctl --list-boots

What this does:

• Lists all boot sessions that have logs available
• Shows each boot with an ID number and timestamp
• Helps you identify which boot session to examine

What you'll see:

IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
 -3 e2823260542d43079f404889c7a8933e Tue 2025-11-11 07:47:33 AST Tue 2025-11-11 10:47:37 AST
 -2 6e3f9edb96f7487aad8506269da4aa5f Wed 2025-11-12 08:37:31 AST Wed 2025-11-12 21:30:27 AST
 -1 ad8c48546fc148ab87554dccd7a4a26c Wed 2025-11-12 17:31:00 AST Wed 2025-11-12 23:44:47 AST
  0 a89c3d22884e408fb963184d2706fa89 Thu 2025-11-13 03:33:56 AST Thu 2025-11-13 15:04:01 AST

What each column means:

• IDX: Boot index number (use with -b flag, e.g., -b -1 for previous boot)
• BOOT ID: Boot UUID (unique identifier for this boot session)
• FIRST ENTRY: When this boot session started
• LAST ENTRY: When this boot session ended (or current time if it's the active boot)

When to use it:

• Finding a specific boot session to examine
• Seeing how many boot sessions are available
• Identifying when your system was last booted
• Selecting a boot session to investigate

Using the boot ID: Once you have the list, you can view logs from a specific boot:

journalctl -b -1

Or use the boot index number from the list.

Example workflow:

1. Run journalctl --list-boots to see available boots
2. Identify the boot session you want to examine (e.g., the one that crashed)
3. Use journalctl -b -1 (or the appropriate number) to view that boot's logs

To see how much disk space your logs are using:

journalctl --disk-usage

What this does:

• Shows the total disk space used by systemd journal logs
• Helps you understand if logs are taking up too much space
• Useful for system maintenance and cleanup planning

What you'll see:

Archived and active journals take up 1.2G in the file system.

When to use it:

• Before cleaning up logs (to see how much space you can free)
• Monitoring disk usage
• Troubleshooting disk space issues
• Planning log retention policies

Understanding the output:

• Shows total space used by all journal logs
• Includes both active (current) and archived (old) logs
• Helps you decide if cleanup is needed

To delete logs older than a specified time:

sudo journalctl --vacuum-time=2weeks

What this does:

• --vacuum-time removes logs older than the specified time
• 2weeks means "delete logs older than 2 weeks"
• Frees up disk space by removing old log entries

What you'll see:

Deleted archived journal /var/log/journal/abc123def456/[email protected] (8.0M).
Vacuuming done, freed 1.2G of archived journals from /var/log/journal/.

Time formats:

• 2weeks - Two weeks
• 30days - 30 days
• 1month - One month
• 3months - Three months
• 1year - One year

When to use it:

• Freeing up disk space
• Regular log maintenance
• Keeping only recent logs for troubleshooting
• Managing disk usage on systems with limited storage

Important notes:

• This permanently deletes old logs (they cannot be recovered)
• Make sure you don't need the old logs before deleting
• Consider your troubleshooting needs - keep logs long enough to investigate past issues
• You need sudo because this modifies system files

Example: If you only need logs from the last week for troubleshooting, you can delete everything older:

sudo journalctl --vacuum-time=1week

To limit logs to a specific total size:

sudo journalctl --vacuum-size=1G

What this does:

• --vacuum-size removes old logs until total size is below the specified limit
• 1G means "keep logs under 1 gigabyte total"
• Automatically deletes oldest logs first to stay under the limit

Size formats:

• 1G - 1 gigabyte
• 500M - 500 megabytes
• 2G - 2 gigabytes
• 100M - 100 megabytes

What you'll see:

Deleted archived journal /var/log/journal/abc123def456/[email protected] (8.0M).
Vacuuming done, freed 500M of archived journals from /var/log/journal/.

When to use it:

• Systems with limited disk space
• Setting a maximum log size limit
• Automatic log management
• Ensuring logs don't fill up your disk

How it works:

• Keeps the most recent logs
• Deletes oldest logs first
• Maintains total size below your specified limit
• Automatically manages itself as new logs are created

Example use case: On a system with limited storage, you might want to keep logs under 500MB:

sudo journalctl --vacuum-size=500M

This ensures logs never exceed 500MB, automatically cleaning up as needed.

Combining with automatic cleanup: You can set up automatic log rotation by creating a systemd timer or cron job to run this command periodically.

You can combine multiple filters to get very specific log views:

View errors from a specific service:

journalctl -u sshd -p err

Shows only error-level logs from the SSH service.

View recent kernel messages:

journalctl -k -n 50

Shows the last 50 kernel messages.

Follow logs for a service in real time:

journalctl -f -u nginx

Follows (real-time) logs from the Nginx web server.

View errors from today:

journalctl -p err --since today

Shows only error-level logs from today.

View logs from a specific service during a time range:

journalctl -u libvirtd --since "2025-01-15 10:00" --until "2025-01-15 12:00"

Shows libvirtd logs between 10:00 AM and 12:00 PM on January 15, 2025.

View current boot errors:

journalctl -b -p err

Shows error-level logs from the current boot session.

Problem: A service (e.g., nginx) won't start.

Solution:

journalctl -u nginx -p err

This shows error messages from the nginx service, which will tell you why it's failing.

Then check recent logs:

journalctl -u nginx -n 50

Shows the last 50 log entries from nginx to see what happened.

Problem: Your system crashed and you want to see what happened.

Solution:

journalctl -b -1 -p err

Shows errors from the previous boot (the one that crashed).

Or view all logs from the crashed boot:

journalctl -b -1

Problem: You know something happened at 2:30 PM today and want to see logs from that time.

Solution:

journalctl --since "14:00" --until "15:00"

Shows logs from 2:00 PM to 3:00 PM today (captures the 2:30 PM event).

Problem: You're starting a service and want to watch it initialize.

Solution:

journalctl -f -u servicename

Follows logs for that service in real time as it starts.

Problem: Your disk is full and you suspect logs might be taking up space.

Solution:

journalctl --disk-usage

Check how much space logs are using.

Then clean up old logs:

sudo journalctl --vacuum-time=2weeks

Or limit total size:

sudo journalctl --vacuum-size=1G

You can save logs to a file for later analysis:

journalctl -u sshd > sshd_logs.txt

What this does:

• Redirects output to a file instead of displaying on screen
• > saves the output to sshd_logs.txt
• You can then open the file with a text editor or share it with others

Appending to a file:

journalctl -u sshd >> sshd_logs.txt

The >> appends to the file instead of overwriting it.

While viewing logs, you can search:

1. Press / to enter search mode
2. Type your search term
3. Press Enter
4. Press n to go to next match
5. Press N (shift+n) to go to previous match

Example: View logs, press /, type "error", press Enter - this finds all lines containing "error".

You can export logs in different formats:

JSON format:

journalctl -u sshd -o json

Useful for parsing with scripts or tools that understand JSON.

Short format (compact):

journalctl -u sshd -o short

Shows logs in a compact, single-line format.

Verbose format (more details):

journalctl -u sshd -o verbose

Shows all available fields for each log entry, including metadata.

Other formats:

• -o export - Binary format for export/import
• -o json-pretty - JSON with readable formatting
• -o cat - Simple text format (no metadata)
• -o json-seq - JSON Sequence format

To see logs from a specific user:

journalctl _UID=1000

What this does:

• _UID filters by User ID
• 1000 is typically the first regular user on Linux systems
• Shows only log entries generated by that user

Finding your User ID:

id -u

This shows your current User ID.

Finding a user's ID:

id -u username

Replace username with the actual username.

When to use it:

• Troubleshooting user-specific issues
• Monitoring what a specific user is doing
• Security auditing (checking user activity)
• Debugging application issues for a particular user

Example:

journalctl _UID=1000 --since today

Shows all logs from user ID 1000 today.

To see logs from a specific program or process:

journalctl /usr/bin/sshd

What this does:

• Filters logs by the executable path
• Shows only logs from that specific program
• Useful when you know which program is causing issues

Finding executable paths:

which programname

Shows the full path to an executable.

Example:

which sshd
# Output: /usr/bin/sshd
journalctl /usr/bin/sshd

When to use it:

• Troubleshooting a specific application
• Monitoring a particular program's activity
• Finding all logs related to a specific executable

To see logs from a specific process:

journalctl _PID=1234

What this does:

• _PID filters by Process ID
• 1234 is the process ID (replace with actual PID)
• Shows only logs from that specific process

Finding a process ID:

ps aux | grep programname

Or:

pgrep programname

When to use it:

• Debugging a specific process instance
• Tracking a process that's having issues
• Monitoring a particular process's activity

Example: If you see a process with PID 5678 is causing problems:

journalctl _PID=5678

You can filter by different types of systemd units:

Services:

journalctl -u servicename.service

Sockets:

journalctl -u socketname.socket

Timers:

journalctl -u timername.timer

Mounts:

journalctl -u mountname.mount

What this does:

• Systemd has different unit types (services, sockets, timers, mounts, etc.)
• You can filter by the full unit name including the type
• This is more specific than just using -u servicename

When to use it:

• When you need to be specific about the unit type
• When multiple units have similar names
• When troubleshooting specific systemd unit types

You can filter by specific log fields:

By hostname:

journalctl _HOSTNAME=mycomputer

By systemd unit:

journalctl _SYSTEMD_UNIT=sshd.service

By command:

journalctl _COMM=sshd

Available fields:

• _PID - Process ID
• _UID - User ID
• _GID - Group ID
• _HOSTNAME - Computer name
• _MACHINE_ID - Machine identifier
• _SYSTEMD_UNIT - Systemd unit name
• _COMM - Command name
• _EXE - Executable path
• _CMDLINE - Command line
• _TRANSPORT - How the log was received

When to use it:

• Very specific filtering needs
• Advanced troubleshooting
• Creating precise log queries

Each log entry contains multiple fields of information:

Timestamp:

• When the event occurred
• Format: Jan 15 10:00:01 (month day time)
• Can be displayed in different formats

Hostname:

• The name of the computer that generated the log
• Useful when viewing logs from multiple machines

Service/Component:

• What generated the log (service name, kernel, application)
• Examples: systemd[1], kernel, sshd[1234]

Priority Level:

• The severity of the log entry
• From emergency (most severe) to debug (least severe)

Message:

• The actual log message describing what happened
• This is usually the most important part

Metadata (in verbose mode):

• Process ID (PID)
• User ID (UID)
• Executable path
• Command line
• And much more

Viewing all fields:

journalctl -o verbose -n 1

This shows one log entry with all available fields.

You can combine journalctl with grep for powerful text searching:

Search for specific text:

journalctl | grep "error"

What this does:

• | pipes journalctl output to grep
• grep "error" searches for lines containing "error"
• Shows only log entries that match

Case-insensitive search:

journalctl | grep -i "error"

The -i flag makes the search case-insensitive.

Search with context:

journalctl | grep -A 5 -B 5 "error"

• -A 5 shows 5 lines after the match
• -B 5 shows 5 lines before the match
• Useful for seeing what happened around an error

Inverse search (exclude matches):

journalctl | grep -v "info"

The -v flag shows lines that DON'T match (excludes "info" messages).

Search for multiple terms:

journalctl | grep -E "error|warning|critical"

The -E flag enables extended regular expressions, and | means "or".

Count matches:

journalctl | grep -c "error"

The -c flag counts how many lines match.

Show only matching text:

journalctl | grep -o "error:.*"

The -o flag shows only the matching part of each line.

You can view logs from other machines on your network:

journalctl -M remotemachine

What this does:

• -M specifies a machine/container (requires a machine name argument)
• remotemachine is the hostname or machine ID of the remote system
• Connects to that machine and shows its logs

Requirements:

• SSH access to the remote machine
• Proper SSH configuration
• systemd-journal-remote service (for some setups)
• The remote machine must be accessible via SSH

Note: The -M option requires you to specify a machine name. You cannot use -M without an argument.

When to use it:

• Managing multiple servers
• Troubleshooting remote systems
• Centralized log monitoring

Alternative method: You can also SSH into the remote machine and run journalctl there:

ssh user@remotemachine "journalctl -u sshd -n 50"

When troubleshooting services, you'll see various states in logs:

Starting:

systemd[1]: Starting nginx.service...

The service is beginning to start.

Started:

systemd[1]: Started nginx.service.

The service started successfully.

Stopping:

systemd[1]: Stopping nginx.service...

The service is being shut down.

Stopped:

systemd[1]: Stopped nginx.service.

The service has stopped.

Failed:

systemd[1]: Failed to start nginx.service: Unit nginx.service failed to load.

The service failed to start - this is an error that needs investigation.

Activating:

systemd[1]: Activating nginx.service...

The service is in the process of becoming active.

Active (running):

systemd[1]: nginx.service: Main process exited, code=exited, status=1/FAILURE

The service was running but the main process exited with an error.

Understanding exit codes:

• status=0 - Success
• status=1 or other non-zero - Failure
• Check the log message for details about why it failed

Problem: A service starts but then immediately crashes.

Solution:

journalctl -u servicename -f

Follow logs in real time to see what happens when it crashes.

Then check for patterns:

journalctl -u servicename --since "1 hour ago" | grep -i "error\|fail\|crash"

Look for error patterns in recent logs.

Problem: System is slow and you want to see what's causing it.

Solution:

journalctl -p warning --since "1 hour ago"

Check for warnings that might indicate performance issues.

Check kernel messages:

journalctl -k --since "1 hour ago" | grep -i "slow\|timeout\|error"

Look for hardware or driver issues.

Problem: Network connection is not working.

Solution:

journalctl -u NetworkManager -p err --since "30 minutes ago"

Check NetworkManager errors.

Check kernel network messages:

journalctl -k | grep -i "network\|ethernet\|wifi\|wlan"

Look for network-related kernel messages.

Problem: Getting permission errors and want to see what's causing them.

Solution:

journalctl | grep -i "permission denied\|access denied\|unauthorized"

Search for permission-related errors.

Check specific service:

journalctl -u servicename | grep -i "permission\|denied\|access"

Problem: Want to know when a service was last started.

Solution:

journalctl -u servicename | grep "Started"

Shows all "Started" messages, with the most recent at the bottom.

Or in reverse:

journalctl -u servicename -r | grep "Started" | head -1

Shows the most recent start (newest first, take first match).

Limit the time range: Instead of searching all logs, limit to a specific time:

journalctl --since "1 hour ago" | grep "searchterm"

Much faster than searching all logs.

Use specific filters first: Filter by service before using grep:

journalctl -u sshd | grep "error"

Faster than journalctl | grep "sshd.*error".

Limit number of results:

journalctl -n 1000 | grep "searchterm"

Only searches the last 1000 entries.

The journald service can be configured in /etc/systemd/journald.conf.

Note: This file may not exist by default on all systems. If it doesn't exist, systemd uses default settings. You can create the file or copy the default template from /usr/lib/systemd/journald.conf if you need to customize settings.

Important settings:

Storage:

Storage=persistent

• persistent - Logs stored on disk (survive reboots)
• volatile - Logs only in RAM (lost on reboot)
• auto - Uses /var/log/journal/ if it exists, otherwise volatile

Maximum file size:

SystemMaxFileSize=100M

Maximum size for individual journal files.

Maximum disk usage:

SystemMaxUse=1G

Maximum total disk space for all journal files.

Maximum retention time:

MaxRetentionSec=2week

How long to keep logs.

Viewing current configuration:

cat /etc/systemd/journald.conf

Note: If the file doesn't exist, systemd uses default values. You can check the default template with:

cat /usr/lib/systemd/journald.conf

Applying changes: After editing the configuration file:

sudo systemctl restart systemd-journald

Warning: Be careful when editing this file. Incorrect settings can cause logging issues.

"Failed to start" errors:

systemd[1]: Failed to start nginx.service: Unit nginx.service failed to load.

Usually means:

• Configuration file has errors
• Required dependencies are missing
• Permissions are incorrect
• Service file is malformed

"Main process exited" errors:

systemd[1]: nginx.service: Main process exited, code=exited, status=1/FAILURE

The service started but the main program crashed. Check the service's own logs for details.

"Connection refused" errors:

sshd[1234]: error: Could not bind to address 0.0.0.0:22: Address already in use

Another service is using the port, or the service is misconfigured.

"Permission denied" errors:

systemd[1]: nginx.service: Failed at step EXEC spawning /usr/bin/nginx: Permission denied

The executable doesn't have execute permissions, or SELinux/AppArmor is blocking it.

"No such file or directory" errors:

systemd[1]: nginx.service: Failed to execute command: No such file or directory

The executable path is wrong, or the program isn't installed.

To view logs without the pager (useful for scripting):

journalctl --no-pager

What this does:

• Outputs logs directly to terminal (no paging)
• Useful when piping to other commands
• Faster for automated scripts

Example:

journalctl --no-pager -u sshd | grep "error" > errors.txt

To resume viewing from where you left off:

journalctl --cursor="s=abc123def456;..."

What this does:

• --cursor specifies a position in the logs
• You can save the cursor from one session and resume later
• Useful for long log analysis sessions

Getting the cursor:

journalctl -n 1 --show-cursor

Shows the cursor for the last log entry.

To count how many log entries match your criteria:

journalctl -u sshd | wc -l

What this does:

• wc -l counts lines
• Shows total number of log entries for that service

Counting errors:

journalctl -p err | wc -l

Shows how many error-level log entries exist.

This comprehensive guide covered:

1. Basic Commands:

• Getting help with --help
• Viewing all logs
• Viewing logs in reverse order
• Viewing last N entries
• Following logs in real time

2. Filtering Logs:

• By service (-u)
• By priority level (-p)
• By kernel messages (-k)
• By time (--since, --until)

3. Advanced Filtering:

• By user (_UID)
• By executable path
• By process ID (_PID)
• By systemd unit type
• By log fields (_HOSTNAME, _SYSTEMD_UNIT, _COMM, etc.)

4. Boot Troubleshooting:

• Current boot logs (-b)
• Previous boot logs (-b -1)
• Listing all boots (--list-boots)

5. Maintenance:

• Checking disk usage (--disk-usage)
• Cleaning by age (--vacuum-time)
• Limiting by size (--vacuum-size)

6. Advanced Topics:

• Understanding log entry structure
• Using grep with journalctl for text searching
• Remote log viewing
• Understanding systemd unit states from logs
• Configuring journald.conf
• Interpreting common error messages
• Performance tips for large logs

7. Practical Applications:

• Combining filters for precise log viewing
• 10 common troubleshooting scenarios with solutions
• Saving and exporting logs
• Searching techniques
• Additional useful commands

Key Takeaways:

• journalctl is your primary tool for viewing system logs
• Use filters to narrow down what you're looking for
• Combine multiple filters for precise results
• Regular log maintenance prevents disk space issues
• Real-time following (-f) is great for monitoring
• Boot-specific logs (-b) help troubleshoot startup issues
• Understanding log structure helps interpret messages
• Advanced filtering options provide powerful troubleshooting capabilities
• grep integration adds powerful text search capabilities
• Understanding error message patterns speeds up troubleshooting

With these commands and techniques, you'll be able to effectively troubleshoot system issues, monitor services, maintain your Linux system's logs, and become proficient at log analysis!