The Kernel Collector monitors and collects kernel messages from the Linux kernel's ring buffer via /dev/kmsg. This collector provides real-time insights into kernel-level events, errors, warnings, and system behavior. It's particularly valuable for:

• System Diagnostics: Identifying hardware failures, driver issues, and kernel panics
• Security Monitoring: Detecting kernel-level security events and anomalies
• Performance Analysis: Understanding system bottlenecks and resource constraints
• Troubleshooting: Correlating application issues with kernel events

The collector can operate in both one-shot mode (collecting recent messages) and continuous mode (streaming new messages as they occur).

• Type: performance.MetricTypeKernel ("kernel")
• Registry: Auto-registered via init() function

• Primary: /dev/kmsg - Kernel message ring buffer interface
• Secondary: /proc/stat - Used to determine system boot time for timestamp calculations

CollectorCapabilities{
    SupportsOneShot:    true,   // Can collect recent messages
    SupportsContinuous: true,   // Can stream new messages
    RequiresRoot:       true,   // /dev/kmsg requires CAP_SYSLOG or root
    RequiresEBPF:       false,
    MinKernelVersion:   "3.5.0" // /dev/kmsg introduced in Linux 3.5
}

• Message Limit: Default 50 messages (configurable via WithMessageLimit())
• Buffer Size: 8KB for reading kernel messages
• Channel Buffer: 100 messages for continuous collection

• 0 - KERN_EMERG: System is unusable
• 1 - KERN_ALERT: Action must be taken immediately
• 2 - KERN_CRIT: Critical conditions
• 3 - KERN_ERR: Error conditions
• 4 - KERN_WARNING: Warning conditions
• 5 - KERN_NOTICE: Normal but significant condition
• 6 - KERN_INFO: Informational
• 7 - KERN_DEBUG: Debug-level messages

The collector returns []*performance.KernelMessage in one-shot mode or individual *performance.KernelMessage objects in continuous mode.

Source Code: pkg/performance/collectors/kernel.go

config := performance.CollectionConfig{
    HostProcPath: "/proc",
    HostDevPath:  "/dev",
}
collector, err := collectors.NewKernelCollector(logger, config)

collector, err := collectors.NewKernelCollector(
    logger, 
    config,
    collectors.WithMessageLimit(100), // Collect up to 100 messages
)

When running in containers, mount the host's /dev directory:

volumes:
  - name: dev
    hostPath:
      path: /dev
      type: Directory
volumeMounts:
  - name: dev
    mountPath: /host/dev
    readOnly: true

• Minimum Version: Linux 3.5.0 (when /dev/kmsg was introduced)
• Capabilities: Requires CAP_SYSLOG or root privileges
• File Access: Read access to /dev/kmsg and /proc/stat

1. Device Access: Must mount host's /dev directory
2. Privileges: Container needs appropriate capabilities or run as root
3. Security: Consider using CAP_SYSLOG instead of full root access
4. Namespace: Messages from host kernel, not container-specific

The collector parses the standard /dev/kmsg format:

<priority>,<sequence>,<timestamp>,<flags>;<message>

Example:

6,1234,5678901234,-;usb 1-1: new high-speed USB device number 2 using xhci_hcd

Problem: Cannot read /dev/kmsg

failed to open /dev/kmsg: permission denied

Solution:

• Ensure container has CAP_SYSLOG capability
• Or run with appropriate privileges
• Check SELinux/AppArmor policies

Problem: Kernel ring buffer overrun

Kernel ring buffer overrun, some messages lost

Solution:

• Increase kernel log buffer size (log_buf_len kernel parameter)
• Collect messages more frequently
• Filter less important messages at kernel level

Problem: Empty results in containers Solution:

• Verify /dev/kmsg is properly mounted from host
• Check if kernel logging is enabled
• Ensure messages aren't being consumed by other readers

{
  "Timestamp": "2024-01-15T10:30:45.123456Z",
  "Facility": 0,
  "Severity": 6,
  "SequenceNum": 12345,
  "Message": "usb 1-1: new high-speed USB device number 2 using xhci_hcd",
  "Subsystem": "usb",
  "Device": "1-1"
}

messages, _ := collector.Collect(ctx)
errors := filterBySeverity(messages, 3) // ERR and above

usbMessages := filterBySubsystem(messages, "usb")
networkMessages := filterBySubsystem(messages, "network")

• CPU: Minimal - only active when reading messages
• Memory:
  • Ring buffer: Up to messageLimit * average_message_size
  • Continuous mode: Additional channel buffer
• I/O: Direct kernel interface, no disk I/O

1. Message Limit: Adjust based on monitoring needs
2. Collection Frequency: Balance between real-time needs and overhead
3. Filtering: Consider kernel-level filtering with printk levels
4. Continuous Mode: Use for real-time monitoring, one-shot for periodic checks

• One-shot collection: ~1-5ms for 50 messages
• Message parsing: ~10μs per message
• Continuous mode overhead: <0.1% CPU

• Load Collector - System load averages
• CPU Collector - CPU usage statistics
• Memory Collector - Memory usage and pressure
• Disk Collector - Disk I/O and errors
• Network Collector - Network interface statistics

• Linux Kernel Documentation - /dev/kmsg
• Kernel Log Levels
• Syslog Protocol (RFC 3164)
• Linux Capabilities