OpenBMC has a variety of interfaces and debug tools available. Some are industry tools and others are home grown. This wiki page is here to provide an overview of these tools and provide people pointers on where to look further.

• Accessing OpenBMC
  • ssh
  • Redfish
  • Web Interface
  • Host Console
• Updating OpenBMC
• Debugging
• Tools
  • obmcutil
  • journalctl
  • systemctl
  • busctl
  • redfishtool
  • ipmitool
  • dreport
  • gpio
  • i2c
  • vpd
• Patching
  • Using an SDK
  • Using devtool modify
  • Applying a Patch Using Overlay Filesystem
• Miscellaneous
  • Programming MAC Addresses
  • Enabling Application Debug
  • Core File Debug

Most OpenBMC systems have ssh enabled: ssh root@<bmc ip> The default password is 0penBmc (note the 0 is a zero)

Redfish API's are also available. See this doc for more information.

Most OpenBMC systems have a web interface available: https://<bmc ip>

The host console is mapped to port 2200 of the BMC so to get to the host console: ssh root@<bmc ip> -p 2200

There are a variety of ways to update an OpenBMC system. Some systems may only support a subset. Here are the advanced user options which we wont go into much detail up here:

• Physically remove BMC flash chip and dediprog it
• Use uboot to network copy in an image and write it or to setup a netboot
• Use bmc command line tools like pflash or dd

Here are the recommended ways to update your BMC flash image:

• Utilize the web ui: https://<bmc ip> and then Configuration->Firmware
• Utilize Redfish API via curl commands (See Redfish section above and link to doc). Look for "Firmware Update" section

Here are some common steps when debugging issues on OpenBMC

• dreport -d /tmp -v: Generate a debug tar with relevant debug information in /tmp/obmcdump_*.tar.xz
• systemctl list-units --failed: Look for any failed services files
• systemctl status <failed unit>: Look at details of failed units (if any)
• obmcutil listlogs: Look for any log entries
• obmcutil showlog <entry>: Look at details of entries from listlogs
• journalctl: Look through journal for fails
  • journalctl | grep state-manager: Look for key power on/off events to get a rough time frame for failures to then zero in on the larger journal for the issue

obmcutil is located in the phosphor-state-manager repo. It provides basic functionality such as:

• obmcutil state: Query the state of the system
• obmcutil chassison: Power on just the chassis, do not start the host
• obmcutil poweron : Power on the chassis and start the host
• obmcutil poweroff: Power off the system
• obmcutil recoveryoff: Turn off all automated BMC recovery (like auto reboot on boot failures)
• obmcutil listlogs: List all OpenBMC event logs
• obmcutil showlog <log>: Show log details (input comes from listlogs)
• obmcutil deletelogs: Delete all event logs
• obmcutil --help : List all supported commands

• BMC Ready: The BMC has started all services and is ready for system power on commands when obmcutil state shows the following for the BMC state:

CurrentBMCState     : xyz.openbmc_project.State.BMC.BMCState.Ready

• Chassis Power On: Turn power on to the chassis and verify it is on (via obmcutil state):

obmcutil chassison
...
CurrentPowerState   : xyz.openbmc_project.State.Chassis.PowerState.On

• Host Start: Start the host and verify it is running (via obmcutil state)

obmcutil poweron
...
CurrentHostState    : xyz.openbmc_project.State.Host.HostState.Running

Note that a obmcutil poweron will power on the chassis and start the host. The obmcutil chassison is really just a debug command for situations where the user only wants chassis power on.

• Other Host Progress Data: obmcutil state also provides some more detailed boot progress. They can be found in the following:
  • BootProgress : xyz.openbmc_project.State.Boot.Progress.ProgressStages.Unspecified
  • OperatingSystemState: xyz.openbmc_project.State.OperatingSystem.Status.OSStatus.Inactive

BootProgress and OperatingSystemState provide more details on what information is provided in these as the system boots.

• Rebooting the BMC: reboot

BMC software uses the systemd-journald for logging debug data. journalctl is used to access this data. There are a variety of parameters this tool can take, here are a few of the most useful for OpenBMC:

• journalctl: Dump all logs to the console in oldest to newest order
• journalctl -o json-pretty MESSAGE="<specific message>": Dump all meta data associated with a specific message
• journalctl -t phosphor-host-state-manager -t phosphor-chassis-state-manager -t phosphor-bmc-state-manager: Provide only the state manager log data

By default, most OpenBMC systems only enable Info and higher journal entries. If you would like to enable the debug as well, run the following:

sed -i 's/info/debug/' /lib/systemd/journald.conf.d/journald-maxlevel-policy.conf
systemctl restart systemd-journald.service

OpenBMC uses systemd to manage it's applications. systemctl is the command line interface into systemd. It provides the ability to check the status of the different applications and the ability to restart them when needed.

• systemctl list-units --failed: List all failed OpenBMC applications
• systemctl status <unit>: Input a unit (i.e. one from above failed command) to get status and relevant journal entries for failed service
• systemctl restart <unit>: Restart an application
• systemctl stop <unit>: Stop an application
• systemctl daemon-reload: Restart systemd to pick up any changes you've made locally to systemd files

The majority of OpenBMC specific systemd service and target files can be found in /lib/systemd/system/. Utilize the overlay filesystem (described below) if /lib is not a read-write filesystem and patching is needed. Be sure to run the systemctl daemon-reload command after changing any systemd service or target files.

Documentation on OpenBMC's use of systemd can be found here

OpenBMC uses D-Bus for all IPC. busctl is a command line tool for interacting with D-Bus.

• busctl tree xyz.openbmc_project.Settings: List all BMC settings
• busctl introspect xyz.openbmc_project.Settings /xyz/openbmc_project/control/host0/auto_reboot Dump information associated with auto_reboot setting
• busctl get-property xyz.openbmc_project.Settings /xyz/openbmc_project/control/host0/auto_reboot xyz.openbmc_project.Control.Boot.RebootPolicy AutoReboot: Read the AutoReboot setting

At times, a view of D-Bus transactions is needed to debug a thorny issue. In these cases, the capture command can be used to have busctl monitor and log every D-Bus transaction within your system.

• busctl capture > /tmp/dbus_capture.pcap: Start capturing a pcap file of all D-Bus transactions. Kill the process once testing is complete and copy off the /tmp/dbus_capture.pcap file to your PC.

This dbus_capture.pcap file can then be loaded into an application like wireshark. Complex queries can then be put into wireshark to zero in on the service(s) you are interested in. You may need to run a busctl -j --list to find the well known name of your service(s) of interest. Then you would put something like this into the query function of wireshark:

dbus.value.str == ":1.5" || dbus.value.str == ":1.432" || dbus.value.str == "xyz.openbmc_project.Hiomapd"

redfishtool.py is a command line tool that can be utilized to interact with OpenBMC's Redfish API's. It is run from your PC and requires python3 be installed.

https://github.com/DMTF/Redfishtool#example-usage has a variety of examples so no need to copy/paste them to here.

OpenBMC continues to support IPMI, although in general it encourages users to move to the more secure and feature rich Redfish API's. ipmitool is the tool to utilize when interacting with an OpenBMC system. Here are a few examples. Note the use of Cipher Suite 17 is required on OpenBMC systems:

• ipmitool -C 17 -I lanplus -U root -H -P chassis power status
• ipmitool -C 17 -I lanplus -U root -H -P chassis power on
• ipmitool -C 17 -I lanplus -U root -H -P sdr elist full

dreport is a OpenBMC command line tool with plugin architecture for collecting a debug tar files for a variety of configurable error conditions. When an error has occurred, it's always good practice to first generate a dreport and save it off in case the failure data wraps out of the system moves out of the error state.

• dreport -v: Generate a default debug tar in the default location
• dreport -d /tmp -v: Generate a default debug tar in the /tmp/ directory

OpenBMC utilizes a mix of the legacy sysfs GPIO subsystem and the new char based GPIO subsystem. The new char based GPIO subsystem has a set of tools called libgpiod.

• gpioinfo: List all GPIO's known to the char based implementation
• gpiofind <gpio name>: Return GPIO chip and line information for input GPIO
• gpioget <chip> <line>: Get the value of the input GPIO

In general, the sysfs interface is used to look at these GPIO's. /sys/class/gpio/ is where this occurs.

The sysfs interface is clunky - all GPIOs attached to the platform are mapped into a single number-space, where each gpiochip is assigned a base value and lines on that chip are contiguously numbered from base upwards. In general users of the sysfs interface will need to explicitly discover the base address associated with the gpiochip in question before deriving the absolute index of the GPIO they wish to access.

More documentation on the sysfs interface can be found in the kernel's administration guide

The kernel i2c-tools package is used within OpenBMC. It provides a suite of tools. See each --help for required parameters.

• i2cdetect: Detect i2c devices on input bus
• i2cdump: Dump a chunk of data from an i2c device
• i2cget: Get a specific chunk of data from an i2c device
• i2cset: Set a specific chunk of data to an i2c device
• i2ctransfer: Transfer a large chunk of data to an i2c device

Use your browser's network monitor to see the HTTP operations between the browser and the BMC. For example, bring the web application to the point of failure, start the network monitor (e.g., https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor), reproduce the error using the web application, and inspect the log entries.

See first two tutorials out in openbmc docs

In a bitbake environment, run the following: devtool modify <recipe> This will extract the source to your local bitbake directory. You can then update the source as needed and do the normal bitbake process to create an image with your changes in it.

If your filesystem is not writeable, then use an overlay to be able to copy in applications or libraries for testing on your system.

# patch needed in /usr
mkdir -p /tmp/persist/usr
mkdir -p /tmp/persist/work/usr
mount -t overlay -o lowerdir=/usr,upperdir=/tmp/persist/usr,workdir=/tmp/persist/work/usr overlay /usr

# patch needed in /lib
mkdir -p /tmp/persist/lib
mkdir -p /tmp/persist/work/lib
mount -t overlay -o lowerdir=/lib,upperdir=/tmp/persist/lib,workdir=/tmp/persist/work/lib overlay /lib

# patch needed in /etc
mkdir -p /tmp/persist/etc
mkdir -p /tmp/persist/work/etc
mount -t overlay -o lowerdir=/etc,upperdir=/tmp/persist/etc,workdir=/tmp/persist/work/etc overlay /etc

On some systems, the MAC address is located in VPD. A system specific service will usually read this MAC address out of VPD and place within the phosphor-inventory objects. phosphor-networkd will then read the MAC from phosphor-inventory and use it to program the networking device. /etc/systemd/network/ will contain the network device files with information on how they are configured and what their MAC addresses are.

• All applications debug logs

sed -i 's/info/debug/' /lib/systemd/journald.conf.d/journald-maxlevel-policy.conf
systemctl restart systemd-journald.service

• PLDM

sed -i 's/0/1/' /etc/default/pldm_verbosity
systemctl restart pldmd.service

• General Here are some common services and how to enable their debug

sed -i 's/mboxd/mboxd -vv/' /lib/systemd/system/mboxd.service
sed -i 's/btbridged/btbridged --vv/' /lib/systemd/system/org.openbmc.HostIpmi.service
sed -i 's/ipmid/ipmid -d 0xff/'  /lib/systemd/system/phosphor-ipmi-host.service
systemctl daemon-reload
systemctl restart mboxd.service
systemctl restart org.openbmc.HostIpmi.service
systemctl restart phosphor-ipmi-host.service

This section could possibly still need some more polishing. The basic idea is to take a core file from a BMC and then debug it using the SDK.

The latest SDK's can be found here. See Download and Install SDK.

One should be able to generate a core file by issuing a command such as kill -3 <process ID>.

1. Source the SDK.
2. Go to the $SDKTARGETSYSROOT directory.
3. Start up arm-openbmc-linux-gnueabi-gdb (or appropriate GNU debugger).
4. Set sysroot.
   • I think the $SDKTARGETSYSROOT somehow gets lost once you are in GDB, thus the hardcoded path in the example.
5. Point at the various source directories needed to debug the application.
   • All that directory stuff seems a bit verbose, but there is command completion with tabs. I suspect the git hash or version values can change from one SDK to the next.
6. Point at the executable file.
7. Point at and load the core file.
8. Start debugging.

The example commands below assume that you downloaded the SDK for p10bmc (IBM Rainier, Everest, etc.) into a src/openbmc/sdk/p10bmc directory under your home directory, and expanded it into a directory called latest (or a symlink to it called latest).

The example commands below assume that you created a src/openbmc/dumps directory under your home directory, where you have stored core dump files.

# Source SDK.
$ source ~/src/openbmc/sdk/p10bmc/latest/environment-setup-armv7ahf-vfpv4d16-openbmc-linux-gnueabi
# Setup PS1 to give a subtle reminder on the command line that this is using the SDK for system p10bmc.
$ export PS1="(p10bmcsdk)$PS1
(p10bmcsdk)$ 
# Go to the SDKTARGETSYSROOT directory.
(p10bmcsdk)$ cd $SDKTARGETSYSROOT
# Start up debugger.
(p10bmcsdk)$ arm-openbmc-linux-gnueabi-gdb
# Commands to show and setup various debugger variables.
show sysroot
set sysroot ~/src/openbmc/sdk/p10bmc/2021-08-11/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi
show sysroot

# Previous instructions had commands to set solib-search-path. Not needed if set sysroot.(?)
show solib-search-path # that set sysroot seems to make the set solib-search-path unnecessary.

# Point at the various source directories. Use tab for auto-completion
directory ./usr/src/debug/phosphor-fan/1.0+gitAUTOINC+5a2b501786-r1/git/
directory ./usr/src/debug/sdeventplus/0.1+gitAUTOINC+dce7381a71-r1/git/
directory ./usr/src/debug/sdbusplus/1.0+gitAUTOINC+270f242cc2-r1/git/
directory ./usr/src/debug/stdplus/0.1+gitAUTOINC+73a20c4a7b-r1/git/
directory ./usr/src/debug/systemd/1_248.3-r0/git/
directory ./usr/src/debug/fmt/7.1.3-r0/git/
directory ./usr/src/debug/glib-2.0/1_2.68.3-r0/glib-2.68.3/
directory ./usr/src/debug/glibc/2.33-r0/git/
directory ./usr/src/debug/libcap/2.50-r0/libcap-2.50/

# Point at executable file you need for the core dump
exec-file ./usr/bin/phosphor-fan-presence-tach

# Prior to loading the core dump file, the sharedlibrary list should be empty
info sharedlibrary

# Something with setting sysroot will generate a security complaint about safe paths when trying to load the core dumpe file.
# If you leave this out, and try to load the core file, it should output a message recommending what to run.
add-auto-load-safe-path ~/src/openbmc/sdk/p10bmc/2021-08-11/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/libstdc++.so.6.0.29-gdb.py

# Point at the core file to debug
core-file ~/src/openbmc/dumps/obmcdump_2_1628637100/core.phosphor-fan-pr.0.f5d2ee9e5fea4a3f911c6aeb28a61778.380.1628637097000000

# Check the list of loaded shared libraries
info sharedlibrary

# Get the backtrace
bt

See arm-openbmc-linux-gnueabi-gdb --help to see command line options.

See GDB User Manual.