Hardware Messages Monitoring - Mellanox/mlxsw GitHub Wiki

The driver communicates with the device using Ethernet packets called Ethernet Management Datagrams (EMADs). As of Linux 4.8, it has become possible to monitor all messages sent between the driver and the device using a kernel tracepoint called devlink:devlink_hwmsg.

For debugging, EMADs can be recorded to a PCAP file and dissected using Wireshark.

This page describes how to record the packets and dissect them.

Table of Contents
  1. Record EMADs
  2. Dissect EMADs
    1. Windows Guide
    2. Linux Guide
  3. Filter EMADs

Record EMADs

Using devlink-hwmsg.py it is possible to hook on the devlink:devlink_hwmsg tracepoint and record events into a PCAP file.

Hardware message packets contain data in a custom TLV-based format. The tool bwz.py can be used to filter the data based on values inside individual TLV fields, and to slice the data (i.e., omit some TLV records or remove the TLV headers altogether).

The scripts may be found in the following repository in Debugging/EMADs folder:

switch$ git clone https://github.com/Mellanox/mlxsw.git

Connect to the switch and run:

switch$ ./devlink-hwmsg.py | ./bwz.py -s buf -w emad.pcap -t 1

This command will record the EMADs into emad.pcap with LINKTYPE=1 (i.e., Ethernet) in the PCAP header.

bwz depends on pcapy, a Python package for processing PCAP files. devlink-hwmsg depends on python3-perf package.

It is also possible to record EMADs using the emadump utility, as explained here.

Dissect EMADs

Windows Guide

EMAD Dissector Installation

This phase should be done once.

  1. Download the latest stable release of Wireshark
  2. Open Wireshark, click Help → About Wireshark
  3. Go to 'Folders' tab
  4. Copy the EMAD dissector to 'Personal Plugins' or 'Global Plugins' folder
  5. Reopen Wireshark or use Ctrl+Shift+L to reload Lua plugins
  6. Go to 'Plugins' tab
  7. Make sure that emad_dissector appears in the list

Load PCAP file and dissect the EMADs

  1. Open Wireshark, click File → Open
  2. Choose the PCAP file
View the payload of the EMADs

figure 1

Linux Guide

Wireshark Installation

Install the latest stable release of Wireshark:

$ sudo dnf install wireshark

Load EMAD dissector and PCAP file

  1. Download the EMAD dissector

  2. Instruct Wireshark to use the EMAD dissector and dissect emad.pcap

    $ wireshark -X lua_script:emad_dissector.lua -r emad.pcap

At this point a window similar to the one above should be presented.

Filter EMADs

Wireshark allows filtering packets by their fields. For example, it is possible to filter all the EMADs with a specific register ID: figure 2

NOTE: A full list of the available protocols and fields is available through the menu View → Internals → Supported Protocols: figure 3

More information appears in Wireshark documentation.