Network Collector - antimetal/system-agent GitHub Wiki

Network Collector

Overview

The Network Collector monitors real-time network interface statistics for all network interfaces on the system. It provides critical metrics for understanding network performance, troubleshooting connectivity issues, and capacity planning. The collector tracks packet transmission and reception rates, error counts, and interface metadata such as link speed and operational state.

This collector is essential for:

  • Performance Monitoring: Track bandwidth utilization and packet rates
  • Error Detection: Monitor packet drops, errors, and collisions
  • Capacity Planning: Understand network usage patterns and bottlenecks
  • Troubleshooting: Identify network issues through error counters
  • SLA Monitoring: Ensure network performance meets requirements

Technical Details

Property Value
Metric Type MetricTypeNetwork
Collection Mode Continuous (interval-based)
Primary Data Source /proc/net/dev
Secondary Data Sources /sys/class/net/[interface]/*
Requires Root No
Requires eBPF No
Min Kernel Version 2.6.0
Return Type []performance.NetworkStats

Data Sources

  1. /proc/net/dev - Primary source for interface statistics

    • Cumulative counters since interface initialization
    • Updated by the kernel network stack in real-time
    • Contains receive and transmit statistics for all interfaces

  2. /sys/class/net/[interface]/* - Interface metadata

    • speed: Link speed in Mbps
    • duplex: Full or half duplex mode
    • operstate: Operational state (up/down/unknown)
    • carrier: Physical link detection status

Collected Metrics

Receive Statistics (RX)

Metric Field Description
Bytes Received RxBytes Total bytes received on the interface
Packets Received RxPackets Total packets received
Receive Errors RxErrors Total receive errors (CRC, frame, etc.)
Packets Dropped RxDropped Packets dropped due to buffer overflow
FIFO Errors RxFIFO FIFO buffer overrun errors
Frame Errors RxFrame Frame alignment errors
Compressed Packets RxCompressed Compressed packets received
Multicast Packets RxMulticast Multicast packets received

Transmit Statistics (TX)

Metric Field Description
Bytes Transmitted TxBytes Total bytes transmitted
Packets Transmitted TxPackets Total packets transmitted
Transmit Errors TxErrors Total transmit errors
Packets Dropped TxDropped Packets dropped on transmit
FIFO Errors TxFIFO FIFO buffer underrun errors
Collisions TxCollisions Ethernet collisions detected
Carrier Errors TxCarrier Carrier signal losses
Compressed Packets TxCompressed Compressed packets transmitted

Interface Metadata

Metric Field Description
Interface Name Interface Network interface name (eth0, lo, etc.)
Link Speed Speed Link speed in Mbps (0 if unavailable)
Duplex Mode Duplex "full", "half", or empty string
Operational State OperState "up", "down", "unknown", etc.
Link Detection LinkDetected Physical link present (true/false)

Data Structure

The collector returns []performance.NetworkStats, an array containing statistics for each network interface. Each NetworkStats struct contains all the metrics listed above.

Source Code: pkg/performance/collectors/network.go

Configuration

The Network Collector is configured through the performance.CollectionConfig:

config := performance.CollectionConfig{
    HostProcPath: "/proc",     // Path to proc filesystem
    HostSysPath:  "/sys",      // Path to sys filesystem
    Interval:     time.Minute, // Collection interval
}

Container Configuration

When running in containers, mount the host filesystems:

volumes:
  - name: proc
    hostPath:
      path: /proc
  - name: sys
    hostPath:
      path: /sys
volumeMounts:
  - name: proc
    mountPath: /host/proc
    readOnly: true
  - name: sys
    mountPath: /host/sys
    readOnly: true

Then configure:

config := performance.CollectionConfig{
    HostProcPath: "/host/proc",
    HostSysPath:  "/host/sys",
}

Platform Considerations

Linux Kernel Requirements

  • Minimum Version: 2.6.0 (for basic /proc/net/dev support)
  • Recommended: 3.x or later for full sysfs metadata support

Virtual Interfaces

Virtual interfaces (lo, docker0, veth*, etc.) may not provide all metadata:

  • speed file may not exist or return errors
  • duplex typically unavailable
  • carrier may always report 1

The collector gracefully handles missing metadata files without failing.

Container Considerations

  1. Network Namespace: The collector sees interfaces in its network namespace

    • Host network mode: Sees all host interfaces
    • Container network mode: Sees only container interfaces

  2. Permissions: Read access to /proc/net/dev and /sys/class/net/* required

Common Issues

Issue: Missing Interface Metadata

Symptoms: Speed, duplex, or carrier information missing

Causes:

  • Virtual interfaces don't expose hardware metadata
  • Interface is down
  • Insufficient permissions to read sysfs files

Resolution: This is expected behavior. The collector logs at verbosity level 2 and continues with available data.

Issue: High Error Counts

Symptoms: Increasing RxErrors, TxErrors, or dropped packets

Common Causes:

  • Network congestion
  • Hardware issues (bad cable, failing NIC)
  • Driver problems
  • Buffer size issues

Debugging Steps:

  1. Check dmesg for network driver errors
  2. Verify cable connections
  3. Check network switch/router logs
  4. Review interface configuration with ethtool

Issue: Counter Wraparound

Symptoms: Suddenly decreasing counter values

Cause: 32-bit counter overflow on 32-bit systems

Resolution: Most modern systems use 64-bit counters. For 32-bit systems, implement delta calculation with wraparound detection.

Examples

Sample Output

[
  {
    "Interface": "eth0",
    "RxBytes": 9876543210,
    "RxPackets": 654321,
    "RxErrors": 10,
    "RxDropped": 5,
    "RxFIFO": 0,
    "RxFrame": 2,
    "RxCompressed": 0,
    "RxMulticast": 100,
    "TxBytes": 8765432109,
    "TxPackets": 543210,
    "TxErrors": 20,
    "TxDropped": 10,
    "TxFIFO": 0,
    "TxCollisions": 5,
    "TxCarrier": 3,
    "TxCompressed": 0,
    "Speed": 1000,
    "Duplex": "full",
    "OperState": "up",
    "LinkDetected": true
  },
  {
    "Interface": "lo",
    "RxBytes": 1234567890,
    "RxPackets": 123456,
    "RxErrors": 0,
    "RxDropped": 0,
    "RxFIFO": 0,
    "RxFrame": 0,
    "RxCompressed": 0,
    "RxMulticast": 0,
    "TxBytes": 1234567890,
    "TxPackets": 123456,
    "TxErrors": 0,
    "TxDropped": 0,
    "TxFIFO": 0,
    "TxCollisions": 0,
    "TxCarrier": 0,
    "TxCompressed": 0,
    "Speed": 0,
    "Duplex": "",
    "OperState": "unknown",
    "LinkDetected": false
  }
]

Monitoring Best Practices

  1. Baseline Normal Traffic: Establish baseline metrics during normal operations
  2. Alert on Error Rates: Set thresholds for error and drop rates
  3. Monitor Utilization: Alert when interface utilization exceeds 80%
  4. Track Trends: Look for gradual increases in errors or drops
  5. Correlate Metrics: Compare network metrics with application performance

Performance Impact

The Network Collector has minimal performance impact:

  • CPU Usage: Negligible - simple file reads and parsing
  • Memory Usage: O(n) where n is the number of interfaces (typically < 100)
  • I/O Operations: 1 read of /proc/net/dev + up to 4 reads per interface from sysfs
  • Collection Time: < 10ms for typical systems

Optimization Considerations

  1. Collection Interval: Default 1 minute is suitable for most use cases
  2. Interface Filtering: Consider filtering out unused interfaces if many exist
  3. Metadata Collection: Can be disabled for virtual interfaces if not needed

Related Collectors

References