Logging - Southampton-Maritime-Robotics/Delphin2_notes GitHub Wiki

What happens on Delphin2 is logged in several ways:

  • The actual ros relevant data is logged with rosbag
  • The cpu load and memory usage is logged with a small script that logs the output of linux system analysis tools
  • The shell output where ros is launched is logged via screen on delphin2
  • The shell output where ros is launched is also logged on the laptop of the operator, so in case of the vehicle not re-surfacing, a log is available for analysis

rosbag logging

how to start it

Currently, rosbag is started separately from the main launch file, in a bash script that controls the timing of recordings, however launching it in a launch script will work similarly, needing the same set of options. rosbag record --split --size=2000 -a -x "/octomap_full(.*)" -o ${logpath}${fileprefix}

--split --size=2000
  • at the moment this is just in as a precaution, since the actual rosbag file size is currently around 500 Mb (originally this was added because whole maps were recorded by accident, see below)
  • the current version of rosbag crashes after recording 2gb of data
  • generally it is good to not record too large file, so starting a new file after 2gb is a good idea, however, it seems that in the current version of rosbag some data is lost
-a -x "/octomap_full(.*)"
  • record all messages, except those starting with /octomap_full
  • the regex for removing the octomap topics was added since it lead to recording full octomap maps every time they were published over a topic, which inflated the recorded rosbags with several gigabites of maps
-o ${logpath}${fileprefix}
  • save to the location given by the variables logpath and fileprefix
  • since the option is -o (not -O), rosbag will automatically add the current date/time in the filename

cpu load logging

  • this repository contains the log-load.sh bash script for logging the cpu load
  • on delphin, an executable copy of this script lies in the ~/useful-scripts folder
  • at the start of the day, the folder is checked for old logs. Since new data is appended at the bottom, old logs are best renamed or deleted before starting a new test day
  • open a new screen (eg. screen -s cpulog), and run the shell script in there, to keep logging the entire day (sh log-load.sh)
  • the script doesn't generate any visible output, so it is completely normal that nothing happens
  • remember to re-start the logger, if delphin2 was re-booted
  • the final log files cpu.log and memory.log are copied over at the end of the day, scripts for plotting them are available

shell output logging on delphin2

  • screen -s roslog -L
  • on delphin2, everything is best run within a screen anyways, so it can be re-connected to after the vehicle re-surfaces
-s roslog

This names the screen, it can now be re-connected to via screen -x roslog

-L

Actual logging. the logfile will land in the folder where the screen was first opened.

shell output logging on the operator laptop

This logging also tries to catch the shell output on delphin2, however with a different purpose: Usually the shell logging on Delphin2 is just fine, but should delphin2 not return within the estimated time, it can be very useful to have a detailed look at the outputs from delphin that were captured before diving, e.g. to check if any odd error messages passed by unnoticed, or if the correct launch script was run. Since it logs the same shell that later connects to delphin2, it is not advisable to use screen -L here as well - depending on the operator familiarity with screen, it can get very confusing which screen is being disconnected at times. Instead, a separate logger is used:

  • script logfilename is run to start a logged shell, before connecting to delphin2 via SSH or MOSH in that shell. Just like any shell, at the end of work it can be closed via ctrl+d at the end of the work time. Logs can be read the same way as the screen -L logs.