LCR Troubleshooting - directedmachines/customer-support GitHub Wiki

Table of Contents

Overview

This page provides information on how to troubleshoot various issues with the robot and mitigation actions Please go to the specific pages, based on the class of the problem you are diagnosing Table Of Contents

How to diagnose your issue

  • Verify your concern - gather as much information as possible from the UI and the LCR

  • Perform preliminary checks - conduct a thorough visual inspection of the UI services, batteries, machinery, cabling, hardware, PCBs

  • Read the Guides

  • Isolate the root cause of your concern - specify what and how exactly the issue occurred

  • Use the instructions below to capture the logs and all data samples (cache/samples state JSON) and include in github Issue

Using the Web browser for diagnostics

The robot software runtime, called CAP (or Demeter Control Plane publicly), has a collection of web services (nano services) that can be configured, inspected through a web browser.

If you want to inspect the realtime data from a sensor or a motor controller from example:

  • Load Manual UI (assuming you can connect to the robot through WiFi or remotely)

  • touch Gear Icon

  • In the list of services, there will be an input box, where you can type the name of a service or scroll

  • type "log" to see links to all logs, and associate log data with an issue you report (kernel, system, process)

  • type "devices" to see links to video and serial device lists

  • Every service has state / configuration URI and a Data Stream URI.

  • Touch / Click the "stream (JSON)" button to see raw data for the current data sample for a given service. Example below of real time data from the AUX motor controller

  • logs

  • poses and all samples

  • devices

  • example of a per service stream and link

The link for the data stream can be seen in the "documentSelfLink" field in the JSON text data "documentSelfLink": "/data-streams/actuators/motor-controllers/aux1"

Failure to login to robot, through browser

  1. Ensure the operator credentials are the most recent version available for that robot and are for "MobileRobotRaspberryPiHost" not "Linux".
  2. Ensure you are in a new incognito tab and the username is correct for remote login.
  3. If CAP was started without a cell connection the most recent login credentials may be incorrect. Follow these steps to restore the runtime:
    1. SSH into robot
    2. execute the following commands, to delete the local runtime index and restart:
      1. sudo systemctl stop dcentralizedcaphost.service
      2. rm -rf /var/local/xenon/8000
      3. sudo systemctl restart dcentralizedcaphost.service

Accessing Logs

Always use the web UI to access logs and device lists. SSH is reserved for the software team.

Web Access

Use the service list and type "logs" in the input box

Secure Shell (SSH)

If UI is not available (CAP runtime is not running), please contact the software team. Instructions below are for software / supervisor team members.

SSH Instructions

Establish and SSH connection to the robot and run the following the

  • System Log: cat /var/log/syslog or tail -n 200 -f /var/log/syslog
  • CAP Runtime Log: cat /var/local/xenon/8000/MobileRobotRaspberryPiHost.8000.0.log. Pay attention to exit codes on the MobileRobotXXXX log
  • Kernel Log: dmesg | tail -n 50

Capturing service data stream JSON from command line

  • establish SSH connection
  • cd dCentralizedSystems/cap-provisioning/pi
  • login, supplying the same credentials you would supply to the HTTP UI login screen
    • scripts/httplogin.sh operator@lcr:<password>
  • perform a HTTP get using the cached security token
    • scripts/httpGet.sh <path>
    • example: scripts/httpGet.sh /data-streams/actuators/motor-controllers/default
    • expect to see JSON data

WiFi

Please review connectivity troubleshooting

  • Symptom - Phone / Tablet / PC cannot connect
    • "Forget" the WiFi SSID, turn of WiFi on the device (phone), re-enable WiFi, select robot WiFi and enter your password
    • If http://edge.directedmachines.com:8000/ is not responding, try http://192/168.50.10:8000. We need a one time DNS resolution and LTE is not ever connected, it will not work using the friendly name
    • If the above does not work, contact support
  • Symptom - WiFi is not found / broad cast
    • Robot might have powered down. Our telemetry would have notified us our support will walk you through powering up the robot CPU. Robot can power down under certain conditions (faulty sensors or electrical issues)

Internet Access

Please review connectivity troubleshooting

Autonomous Ground Task

AGT Troubleshooting

Manual Control

Manual Control Troubleshooting

⚠️ **GitHub.com Fallback** ⚠️