Temperature and Fan Control - Mellanox/mlxsw GitHub Wiki

Temperature monitoring and fan control is implemented using a standard kernel interface called hwmon.

This page describes how to monitor the hardware.

Table of Contents
  1. Using sysfs
  2. Using lm_sensors
  3. Critical And Emergency Alarms
  4. Periodical Alert Reporting Using sensord
    1. Install From Source
    2. Redirect Alarms
    3. systemd Configuration
    4. Start sensord Daemon
  5. Further Resources

Using sysfs

sysfs files may be accessed directly to monitor temperature and control fans.

Drivers which want to use the kernel sysfs interface, create a hwmon instance under the /sys/class/hwmon/ directory. To check if a hwmon instance is of the mlxsw driver, run:

$ cat /sys/class/hwmon/hwmon1/name 
mlxsw

To query the ASIC for current and highest temperature, run:

$ cat /sys/class/hwmon/hwmon1/temp1_input 
38000
$ cat /sys/class/hwmon/hwmon1/temp1_highest 
39000

Values are in milli-degrees Celsius.

To query the RPM of a fan, run:

$ cat /sys/class/hwmon/hwmon1/fan1_input
12335

Or to query all the fans in the system, run:

$ grep -H . /sys/class/hwmon/hwmon1/fan*_input
/sys/class/hwmon/hwmon1/fan1_input:13043
/sys/class/hwmon/hwmon1/fan2_input:10775
/sys/class/hwmon/hwmon1/fan3_input:12798
/sys/class/hwmon/hwmon1/fan4_input:10691
/sys/class/hwmon/hwmon1/fan5_input:12562
/sys/class/hwmon/hwmon1/fan6_input:10861
/sys/class/hwmon/hwmon1/fan7_input:12335
/sys/class/hwmon/hwmon1/fan8_input:10948

Regulating the speed of a fan is done by altering the PWM value. For example:

$ cat /sys/class/hwmon/hwmon1/pwm1 
153
$ echo "255" > /sys/class/hwmon/hwmon1/pwm1 
$ cat /sys/class/hwmon/hwmon1/fan1_input
21459
$ echo "100" > /sys/class/hwmon/hwmon1/pwm1
$ cat /sys/class/hwmon/hwmon1/fan1_input
8187

Range is 0-255.

Using lm_sensors

Instead of using sysfs it is possible to monitor the temperature and fans using the lm_sensors package:

$ sensors
...
mlxsw-pci-0300
Adapter: PCI adapter
fan1:        12679 RPM
fan2:        10691 RPM
fan3:        12798 RPM
fan4:        10691 RPM
fan5:        12562 RPM
fan6:        10861 RPM
fan7:        12448 RPM
fan8:        10775 RPM
temp1:        +38.0°C  (highest = +39.0°C)

For automatic fan regulation, fancontrol (part of lm_sensors) may be used. However, the user must first create /etc/fancontrol. In order to generate it, it is recommended to use pwmconfig. Run:

$ pwmconfig
... Will generate /etc/fancontrol ...

On systemd based systems, launch fancontrol by running:

$ systemctl start fancontrol

And enable it on boot by running:

$ systemctl enable fancontrol

Critical And Emergency Alarms

Starting from kernel 5.10 critical and emergency alarms are reported. In case that current temperature is higher than emergency threshold, EMERGENCY alarm will be reported in sensors utility:

$ sensors
...
front panel 025:  +90.0°C  (crit = +70.0°C, emerg = +80.0°C) ALARM(EMERGENCY)

In case that current temperature is higher than critical threshold, CRIT alarm will be reported in sensors utility:

$ sensors
...
front panel 025:  +75.0°C  (crit = +70.0°C, emerg = +80.0°C) ALARM(CRIT)

Periodical Alert Reporting Using sensord

Sensord is a daemon that can be used to periodically log sensor readings from hardware health-monitoring chips to syslog and to alert when a sensor alarm is signalled.

Install From Source

Install dependencies

$ dnf install libxml2-devel rrdtool-devel

Clone lm-sensors git

$ git clone https://github.com/lm-sensors/lm-sensors.git

Compile and install

$ cd lm-sensors && make PROG_EXTRA=sensord && make install PROG_EXTRA=sensord
Redirect Alarms

All messages from sensord daemon are logged to syslog under the program named 'sensord' and facility daemon, or whatever is specified on the command line.

Regular sensor readings are logged at the level info. Alarms are logged at the level alert.

To direct these messages in a useful manner, edit '/etc/syslog.conf' or '/etc/rsyslog.conf' file. For example, to direct alarms to console add the line below:

\*.=alert;                                               /dev/console

Restart rsyslog

$ systemctl restart rsyslog.service

systemd Configuration

Add an appropriate systemd service unit file

# /usr/lib/systemd/system/sensord.service
[Unit]
Description=Hardware Monitoring Data Logger
After=lm_sensors.service

[Service]
EnvironmentFile=-/etc/sysconfig/sensord
Type=forking
PIDFile=/var/run/sensord.pid
ExecStart=/usr/libexec/lm_sensors/sensord-service-wrapper

[Install]
WantedBy=multi-user.target

Example of ExecStart - /usr/libexec/lm_sensors/sensord-service-wrapper:

#!/usr/bin/sh

. /etc/sysconfig/sensord

ARGS=""
[ "$INTERVAL" = "" ] || ARGS=`echo "$ARGS -i $INTERVAL"`
[ "$LOG_INTERVAL" = "" ] || ARGS=`echo "$ARGS -l $LOG_INTERVAL"`
[ "$RRD_INTERVAL" = "" ] || ARGS=`echo "$ARGS -t $RRD_INTERVAL"`
[ "$RRD_LOGFILE" = "" ] || ARGS=`echo "$ARGS -r $RRD_LOGFILE"`
[ "$RRD_NO_AVG" = "1" ] && ARGS=`echo "$ARGS -T"`
[ "$LOAD_AVG" = "1" ] && ARGS=`echo "$ARGS -a"`

/usr/local/sbin/sensord -f daemon $ARGS

Example of EnvironmentFile - /etc/sysconfig/sensord:

# interval between scanning alarms
INTERVAL=10s

# interval between logging
LOG_INTERVAL=10s

Start sensord Daemon

To start the daemon, run:

$ systemctl start sensord.service

To start the daemon automatically after each boot, run:

$ systemctl enable sensord.service

For more details see sensord man page.

Further Resources

  1. man sensors
  2. man pwmconfig
  3. man fancontrol
  4. Fan speed control on Arch Linux Wiki