LCR Manual Control Troubleshooting - directedmachines/customer-support GitHub Wiki
Table of Contents
Overview
This page describes diagnosing issues during manual control.
Erratic Behavior
If control of the robot through Manual UI is erratic, stops and starts, the most common cause is poor WiFi connection between robot and controlling device (Phone).
- Turn off phone WiFi. Wait a few seconds, turn back on and connect to robot SSID
- Determine if erratic behavior stops if you are controlling from caster side vs axle side. WiFi antenna is mounted on the axle side, by the 3PT hitch
To confirm it is WiFi related, SSH into robot and display logs
tail -f /var/log/syslog
You might see the following logs:
Sep 10 07:54:09 LCR24ZS0-703442644098d7215 runLoop.sh[10990]: [1405][S][2022-09-10T14:54:09.673Z][11][8000][exceptionCaught][Listener channel exception: Connection reset by peer] Sep 10 07:54:09 LCR24ZS0-703442644098d7215 runLoop.sh[10990]: [1407][S][2022-09-10T14:54:09.677Z][102][8000][exceptionCaught][Listener channel exception: Connection reset by peer]
Connection resets indicate an unstable connection.
Further diagnosis requires the robot to be e-stopped.
- Press e-stop
- Open solar panel and tray cover
- Very carefully, check SMA connections between Modem (black box by right corner, axle side, inside tray) and extension cable to wifi antenna inside cable gland
Warnings
'High Log Count Detected'
A component of the software runtime is producing warning messages at a high rate. Please contact the software team (send email to support)
'AXLE_3PT_LIFT fault'
Cause: the pico tray 004260 stopped responding over USB An email alert should be sent as well. If you restart CAP it sends a second alert and then restarts and runs a script to reset the USB driver and autorecover the connection.
"Obstacle Sensors powered down (IDLE)"
Obstacle avoidance and the realsense camera sensors are in an idle state, not streaming images. When manual UI or AGT is active, the cameras and obstacle avoidance will normally be in an ACTIVE state. This false positive IDLE, can happen if the self monitoring task service has not started or is in stuck state. Contact a software developer (or send a support email).
"Obstacles close"
Obstacles within a given region or corrupted / frozen depth sensor images. See Camera Troubleshooting - Corrupted Depth Image
"Obstacle Avoidance Bias enabled"
See failure to apply bias warning in AGT
"Obstacles, Ignored, Caution!"
Obstacle avoidance set to "ignore" instead of "avoid". Note: only disable obstacle avoidance in rare occasions
"Drive Disabled"
Robot motion is software-disabled. This automatically happens after a reboot or triggering the E-stop. This warning in yellow is coupled with a red flashing "disabled". Enable motion by hitting the "Active" button under the drive control.
If the fault won't clear follow the e-stop troubleshooting instructions here: E-Stop Faults
"Another user is active: Linux armv8l:1659531773256"
The warning displays when multiple users are active (the listed OS and ID following the ':' depends on the user's computer). Ensure that multiple browser tabs, phones, computers, or remote SSH connections are not active. If the issue persists it could be due to USB instability, see Camera Troubleshooting
"LEFT|RIGHT|AXLE_AUX_MOTOR Disconnected"
For a LEFT or RIGHT MOTOR disconnect, review the Drivetrain Disconnected Motor Guide
If the problem is present on ALL motor controllers, this will result in a battery voltage not detected alert and may be indicative of a short
For an AXLE_AUX_MOTOR disconnect, review the Mowing Deck Disconnected Motor Guide
"LEFT|RIGHT|AXLE_AUX_MOTOR Controller Unenumerated"
The motor controllers communicate with USB to the CPU and USB enumeration failed.
Follow the Motor Controller Troubleshooting Tree or the Motor Controller Unenumerated guide
"LEFT|RIGHT|AUX_MOTOR disconnected (check panel switch, wiring, motor brushes)"
First, attempt a remote recovery and follow the Motor Controller Unenumerated guide
If the problem persists, have the panel switch, wiring and motor brushes inspected for poor connection / damage.
This problem has also been known to be caused by a MC that is not properly secured.
'Invalid db9 switch state'
Check that the db9 connector is fully seated in the AUX MC. If that does not fix it, double-check the order and routing of the USB cables, cables may be in wrong port.
'Disabled due to AUX Motor sample is late or marked UNSAFE'
Usually caused by a high current for an extended period of time. This is generally caused by high growth, or something stuck in the mowing deck. If there is not high growth, or if the current does not return to a normal value (under 100A with no growth, or under 200A with low growth), review the Stalled Motor Troubleshooting Guide
'AUX MC Sample Late'
The same conditions as the above warning my cause an 'AUX MC Sample Late' alert in manual UI. A reboot may be necessary if the current does not reduce or the warning persists in areas of low growth.
If the warning persists, use the Motor Controller Troubleshooting Tree
Frame short detected
PicoTray4260: Frame short detected
See details here: https://github.com/directedmachines/customer-support/wiki/LCR-Fleet-Alerts#frame-short-detected
A short that results in all MCs suddenly going away from the USB bus will result in both a MC disconnect error on all 3 MCs, as well as a battery voltage not detected alert.
Inertial (IMU) Sensor Fault
if GPS antenna has been moved from optimal position, for example, during robot moves or repairs, the GPS chip will loose contact with satellites and will need either power cycle or FW upgrade to force a reset
Self Monitoring Fault
- Check self-monitoring text(stream) and ensure it updates on refresh
- If stream not updating, stop the runtime, delete this file, restart the run time, and SYNCH self monitoring state
sudo systemctl stop dcentralizedcaphost.service
rm ~/dCentralizedSystems/cap-config/config/mgmt-self-monitoring-tasks-default.`hostname`.json
sudo systemctl restart dcentralizedcaphost.service
- Verify Manual UI is operational