LCR Electronics Troubleshooting - directedmachines/customer-support GitHub Wiki
Table of Contents
- Overview
- USB Port Topology
- Connectivity
- Speakers
- Charging State
- Telemetry Details Dashboard UI
- Solar Troubleshooting
- AC (shore power) Troubleshooting
- AC (Renogy 48V inverter) Troubleshooting
- Pi Troubleshooting
- Continuity and Shorts Troubleshooting
Overview
This page covers diagnosis of low current electrical and electronics components.
USB Port Topology
The USB topology is like a tree, and the ports further down the branches have longer addresses based on where the earlier hubs are plugged in.
Pi
| Location | USB Path Segment | Full Path | Device | Tape Colors |
|---|---|---|---|---|
| Pi Top Left | 1.3 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.3:1.0 |
Pico Tray | Pink - Red |
| Pi Bottom Left | 1.4 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4:1.0 |
USB Hub 1 | Pink - Brown |
| Pi Top Right | 1.1 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.1:1.0 |
Caster Camera | Pink - Blue |
| Pi Bottom Right | 1.2 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.2:1.0 |
Axle Camera | Pink - Green |
Pluggable 60W USB Hub
Hub 1
Assumes Hub 1 plugged into Pi Bottom Left (1.4)
| Location | USB Path Segment | Full Path | Device | Tape Colors |
|---|---|---|---|---|
| Port 1 | .3.3 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.3:1.0 |
Aux MC | No Tape |
| Port 2 | .3.2 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.2:1.0 |
Left MC | No Tape |
| Port 3 | .3.1 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.1:1.0 |
Right MC | No Tape |
| Port 4 | .3.4 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.4:1.0 |
Pico Brain | Brown - White |
| Port 5 | .2 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.2:1.0 |
USB Hub 2 | Brown - Yellow |
| Port 6 | .1 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.1:1.0 |
GPS | Brown - Blue |
| Port 7 | .4 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.4:1.0 |
Speakers | Brown - Green |
Hub 2
- This assumes Hub 2 is plugged into Port 5 of Hub 1 (
1.4.2) - Note that video devices show up as
/dev/v4l/rather than/dev/serial/
| Location | USB Path Segment | Full Path | Device | Tape Colors |
|---|---|---|---|---|
| Port 1 | .3.3 |
/dev/v4l/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.2.3.3:1.0 |
Left Side Camera | Yellow - Green |
| Port 2 | .3.2 |
/dev/v4l/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.2.3.2:1.0 |
Right Side Camera | Yellow - Red |
| Port 3 | .3.1 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.2.3.1:1.0 |
||
| Port 4 | .3.4 |
/dev/v4l/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.2.3.4:1.0 |
Dock Camera | Yellow - Blue |
| Port 5 | .2 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.2.2:1.0 |
||
| Port 6 | .1 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.2.1:1.0 |
||
| Port 7 | .4 |
/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.2.4:1.0 |
Connectivity
Connectivity Overview
The LCR is connected to the internet through LTE using a modem (pepwave/peplink) and sim card. The LCR also broadcasts a local wifi connection that allows operators to connect locally but does not provide connected devices access to the internet/LTE signal.
Operators can connect to the robot UI either through a VPN + the LTE connection OR through the local wifi connection broadcast by the LCR.
Wifi
No Wifi signal
No Wifi signal: Symptoms
If the WiFi signal from the robot is not broadcasting, the robot CPU (Raspberry PI) is likely not powered on. This could also be caused by a bad antenna/weak signal, or by an improper hotspot configuration.
No Wifi signal: Fixes
- Ensure the Pi is powered on.
- If not powered on but lights are on, the pi may have been intentionally shutdown. Unplug and replug the pi usb-c power cable to reboot.
- If no lights are on check for a wiring fault (loose wiring, short, battery connector disconnected, busbar connector disconnected).
- Ensure the wifi antenna is properly connected to the Pi SMA connector and insulated with electrical tape, and that the small end is properly connected to the pi board (LINK IMAGE HERE). Put your device very close to the pi and see if there is a weak signal.
- SSH into the robot and run
~/dCentralizedSystems/cap-provisioning/pi/scripts/install-hotspot.shwith <HOTSPOT_NAME> and <HOTSPOT_PASSWORD> as arguments to reinstall the wifi hotspot.
Wifi visible, can't connect
Wifi visible, can't connect: Symptoms
Wifi network is seen on phone, unable to connect when selecting the wifi network.
Wifi visible, can't connect: Fixes
- Ensure Wifi password is correct
- Ensure the wifi signal strength is sufficient. Move device near the pi and retry.
- SSH into the robot and run
~/dCentralizedSystems/cap-provisioning/pi/scripts/install-hotspot.shwith <HOTSPOT_NAME> and <HOTSPOT_PASSWORD> as arguments to reinstall the wifi hotspot.
Wifi connected, UI not Loading
Wifi connected, UI not Loading: Symptoms
You can connect to the wifi network but the UI does not show up when you try any of the following URLs
Wifi connected, UI not Loading: Fixes (Ensure Modem is healthy and PI is active)
- Ensure the url is correct (192.168.50.10:8000, 192.168.8.100:8000 or edge.directedmachines.com:8000). If the edge.directedmachines.com url is not working there may be LTE connectivity issues see: LCR Electronics Troubleshooting - No LTE connection
- Runtime may be syncing and rebuilding. You can wait ~2 minutes or SSH into the robot and tail the logs (
tail /var/log/syslog) to see if it is rebuilding. - Check robot status in Soracom. If online, the problem is either with the Pi4 or the modem to Pi4 connection. Attempt to power cycle the Raspberry Pi4. To do this, unplug the PI4 USB from the USB Station, then re-plug it in .
- The CAP service may be disabled. Run
topand see if a "java" service is running. If not, start the runtime withsudo systemctl start dcentralizedcaphost.service - When connecting to the wifi network it may warn for you to "use this network as is" if you don't select this you won't have access to the UI. Forget the network and reconnect.
Cellular Internet
Pepwave MAX BR1 Modem Resources
AGT UI / Satellite maps will not load
If the modem is not connected to a cell network:
- it can not publish telemetry ( a yellow warning icon will be shown in fleet management)
- AGT UI will not load satellite images in google map view and will be in offline mode
- The edge.directedmachines.com:8000 URL may not work either (use http://192.168.8.100:8000 instead)
Abrupt loss of connection during remote sessions
If the robot has solid connectivity when stationary, but connection is lost when active, there is likely a poor electrical connection
- check the power leads into the modem power connector. Make sure everything is secure and tight from modem to 12V bus bar
- check the SMA connectors (gold connectors on modem) connecting antennas to modem
If the robot is in an area with poor connectivity and is often losing connection, follow the Poor Connectivity Area guide below
No LTE Connection: Fixes
- Ensure the antennas are intact and properly connected to the modem, the ethernet cable connects the modem to the pi, and the modem is powered and status lights are green.
- Unplug and replug the ethernet cable to the pi, SSH into the robot
- Run
ping github.comfrom command line to test connectivity- If you can ping github.com, it means connectivity has been restored and you should restart the CAP runtime and reload the AGT UI.
- Ensure eth0 IP is 192.168.1.100 when running
ifconfig
- Run
- Directly connect laptop to modem using ethernet cable and test connectivity (ping github.com from command line).
- When connected by ethernet can log into the modem at 192.168.8.1 and check the settings are correct (follow instructions for #003737 - DM (A) Advanced Cellular Connectivity Kit)
The Pepwave modem has a WAN RSSI page that will display the signal strength during the past hour. The support team can help get that information:
Before taking any further actions contact a software team member on slack and ask for advice.
- Power cycle the modem (unplug and replug power cable) and wait 5 minutes for it to boot back up.
- Could be related to bad USB peripherals.
Poor Connectivity Area
If the robot is operating in an area with poor connectivity, it might fully lose connection with the fleet and require a modem reboot to bring it back online.
Rather than rebooting the modem in person, the modem can be scheduled to reboot daily by following these steps:
- Connect to the modem wifi and login to https://192.168.8.1/cgi-bin/MANGA/support.cgi
- Select the time interval for the reboot to occur
- To confirm the reboot is scheduled, return to the modem dashboard page. There should be a reboot scheduled notice at the bottom of the page
Rebooting and Resetting
Reboot
A modem reboot can be done by following these steps:
- Connect to the modem wifi and access the dashboard
- Go to the System page, select Reboot in the options on the left, and click Reboot. Ensure the proper Firmware is selected for the reboot
Reset
A full modem reboot can take upwards of 10 minutes to complete. If only a reset is necessary, it can be done by following these steps:
- Connect to the modem wifi and access the dashboard
- Go to the cell connection details page, and click the question mark in the top right to find the option to reset
Modem Continuity Test
The frame of the modem should be isolated from the LCR tray. To ensure isolation:
- Use a multimeter on continuity mode to test for continuity between the modem and the tray. Continuity should not be found
- If continuity is found, use velcro or electrical tape to ensure no part of the modem touches the metal tray
- Continue to isolate the modem and test with the multimeter until no continuity is found
Speakers
The speakers emit a tone when autonomy is about to start. If no sound is heard,
- check that both the USB tail is connected to the 7 port hub (its the power supply for the speakers) and the 3.5mm audio male connector is inserted all the way into the PI audio port
- the sound properties Java runtime file has proper permissions
The sound properties file is normally configured during auto provisioning but it can be made writable using these steps
- SSH into the robot
- change permissions using this command:
sudo chmod 777 /etc/java-8-openjdk/sound.properties - restart CAP runtime
Charging State
The telemetry details page on a robot provides metrics relating to solar and AC charging input. The wattSecs metric, reported by the self-monitoring-task-service, reports the current power draw on the robot.
- a negative watt-seconds number indicates power is going in the robot, it is charging
- a positive number means that energy is being used at a rate higher than input (charge) energy
- a number between -50 to -500 means we are solar charging
- a number between -700 to -5000 means we are AC charging (and potentially also solar charging
You can find this number in two ways, on the robot itself. You can also find charge rate and time to full, by clicking the battery icon, in the telemetry details page
Telemetry Details Dashboard UI
Please use the Fleet Management UI, data details page for the robot and follow the same instructions as below to search stats. You do not need access to a robot to check any of its metrics, you can do so through your web browser and our fleet management
- Touch the "bars" icon on the lower left corner to bring up the telemetry dashboard
- Type "wattSecs" in the "Inspect Telemetry" -> "Search stats" input box
Data Multiplexing Service Stream
You can view the metrics directly from the data multiplexing service
- Touch / Click the "gear" icon on lower left corner of screen in Manual UI
- Type "multi" in the service input box
- Select the "stream/text" button for the service entry
- Search for "wattSecs" text
Solar Troubleshooting
If you do not see a negative wattSecs number, robot is not charging from the sun. The following steps make sense if solar is expected, so during the day, cloudy or sunny, with the solar panel clean and not shaded
- Make sure panel is receiving solar radiation
- E-stop robot
- Open panel
- Verify panel + and - cables going to inside of high current electronics (HCE) tray are connected to the Solar Charge Controller (SCC)
- Use a Clamp current meter on the positive + cable from solar panel to determine if current is flowing
- Check connections from SCC to battery bus bar
- Check lights on solar charge controllers
- Alt SCC Genasun: Should be green or flashing green
- DM SCC board: Should be flashing green. If the "solar fault" LED is flashing red, press the small black "reset" button just below the DB9 connector and restart the robot from the UI (video here).
If no charging is reported, this could mean the pico tray is not reporting.
- Look at the pico-controllers/tray-4260 stream in the CAP UI
- If the stream is not updating, reboot the robot to restore the pico
AC (shore power) Troubleshooting
The robot will charge from AC if the robot onboard charger unit is connected through an extension cord to an AC 110V circuit (or 240V). If the wattSecs is not indicating a number of 700W or more of charging:
- Make sure AC extension cable is connect to robot side plug and to AC plug on wall / infrastructure
- Verify charging unit inside robot has Green blinking LED, or Yellow LED on. If robot has new batteries, unplug extension cord, wait 1 minute, plug back in, check wattSecs (sometimes charger stops charging, the first time, on new batteries)
- Verify AC circuit provides current - use a AC current clamp meter to measure current flowing from AC plug. Sometimes circuit breaker trips
AC (Renogy 48V inverter) Troubleshooting
- Make sure AC extension cable is connect to robot side plug and to AC plug on wall / infrastructure
- Make sure Renogy is powered on once it is plugged in (using remote switch)
- If circuit trips need to check the max current setting (Setting #28). It depends what the circuit breaker is rated for, (MaxBreakerCurrent*120V)/58V = Current Setting. 20A is good if we're limiting it to 10A from the outlet. 30A is better if the outlet can handle 15A.
Pi Troubleshooting
Pi LED Blink Patterns
Power LED (Red)
- The power LED should be solidly lit. A blinking power LED indicates possible Pi under-voltage. No light indicates the Pi is not receiving power.
Activity LED (Green)
- If the Pi has an SD card inserted, the activity LED should blink in an irregular pattern. A regular pattern of short and/or long blinks indicates a possible failed or corrupted SD card.
Pi Under-Voltage
- Can manifest with "Drive MC sample late", "AUX MC sample late", "MC Disconnected", or "E-Stop engaged" alerts
- Check kernal log for:
"Nov 6 15:56:54 LCR24ZS0 kernel: [975264.239616] Under-voltage detected! (0x00050005)","Nov 6 15:56:58 LCR24ZS0 kernel: [975268.409668] Voltage normalised (0x00000000)","Nov 6 15:59:47 LCR24ZS0 kernel: [975437.251767] usb 2-1: USB disconnect, device number 33",
- The 5V supply to the Pi is unstable. The issue seems to get worse the more USB peripherals are plugged in
- To fix, install the 004679 Tray Controller Board, replacing the currently installed tray board
- If Tray Brain is not available, recommended to unplug all non essential USB peripherals (analog speakers, unneeded cams, etc)
- If Tray Brain is already installed, try following the Pi to Tray Brain USB Failure guide below.
Pi to Tray Brain USB Failure
- Can manifest with similar symptoms to the Pi under-voltage symptoms listed above, or continuity symptoms listen in the Continuity and Shorts Troubleshooting wiki including the following:
- Under-voltage alerts in the Kernel log
- A sudden Pi reboot or restart
- Diagnose by wiggling the USB C to USB C cable connecting the Pi to the tray brain. If this causes Pi LEDs to flicker or shutoff, the USB cable needs to be replaced.
Continuity and Shorts Troubleshooting
Information on testing for continuity between various electronics and components can be found here