• Overview
• Logs
• SSH Access
• Realsense Cameras
• GPS Sensor
• Inertial Measurement Unit
• Pico Tray Brain
  • Current Sensor

This page helps with narrowing down sensor faults. It covers advanced topics that require knowledge of basic linux commands.

We want to avoid physically touching the robot during the sensor debugging except as a last resort so we know if the problem is something that we could recover remotely using software commands or if it requires physical intervention. These are the types of questions we're trying to answer in the debugging process:

1. What caused the issue to happen how how can it be avoided in the future?
2. Can it be recovered with a CAP restart?
3. Can it be recovered with a command line script? (In the future we could run this script automatically from CAP)
4. Can it be recovered with a Pi reboot?
5. Can it be recovered with a physical power cycle (unplug/replug, last resort)

All logs are available through the user interface:

• system log - contains all log messages and is persisted across boot cycles. Search for "system" in the input search box
• kernel log - kernel messages, relevant for USB debugging
• process log - CAP runtime log for our runtime

Troubleshooting may require a connection to the robot operating system through SSH (secure shell). Steps to establish an SSH connection

1. Connect a PC to the robot WiFi
2. Open a terminal on the PC side (Linux or Mac OSX, windows requires a PuTTY terminal)
3. In the terminal window
4. ping edge.directedmachines.com
5. if the ping is not successful, try ping with either 192.168.8.100 or 192.168.50.10 as the IP address
6. select the IP address that worked
7. ssh pi@<ipaddress>
8. you will be prompted for a password. The password is emailed to the credentials group, per robot, or, your PC will have a private key file that enables it to connect without password. If you cant find the password, ask the software team

• Serial devices can be listed by their USB port topology path (by-path) or by more human readable description (by-id)
• The list of usb serial devices (excluding cameras) can be found in the UI under:
  • http://<IP-ADDRESS>:8000/dashboard/devices/serial
• It can also be printed over ssh using ls -al /dev/serial/by-id and ls -al /dev/serial/by-path
• NOTE: if devices are shown by-path but not by-id then there is likely an issue with the kernel versions. See this thread as reference and an example of the device list below:

{
  "devs_by_id": {},
  "devs_by_path": {
    "/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.4:1.0": "../../ttyACM5",
    "/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.1:1.0": "../../ttyACM2",
    "/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.2:1.0": "../../ttyACM3",
    "/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.3:1.0": "../../ttyACM4",
    "/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.1:1.0": "../../ttyACM1",
    "/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.3:1.0": "../../ttyACM0"
  },
  "documentVersion": 0,
  "documentUpdateTimeMicros": 0,
  "documentExpirationTimeMicros": 0
}

Camera Sensor Troubleshooting

All robots have a GPS sensor positioned above the intersection of the robot midline and the main drive axles. If you see an error with the GPS sensor, consult the GPS Troubleshooting Tree

• Real-time kinematic positioning (RTK) is the application of surveying to correct for common errors in current satellite navigation (GNSS) systems
• NTRIP stands for "Networked Transport of RTCM via Internet Protocol".
• RTCM is a communication protocol for sending differential GPS (DGPS) to a GPS receiver from a secondary source
• Differential GPS uses GPS basestations with a known survey location to produce corrections that improve the accuracy of nearby (ideally <25 km) mobile GPS receivers. It can decrease from +/- ~3m accuracy to +/- ~3 cm accuracy.
• This accuracy is impacted by obstacles such as buildings/trees so even with GPS corrections the accuracy may be low

• The GPS accuracy is shown in mm under the "GPS" header in the top-right corner of the screen in the manual and AGT UIs. If the color is grey, there is no RTK lock, if the color is yellow, it is in RTK float, if green it is RTK Fix.
• To save data the GPS corrections are not applied when idle, when transistioning from idle to active (e.g., connecting to an LCR after a long period of inactivity), please allow a couple minutes for the GPS to lock into the fix.

• An inaccurate GPS reading can be resolved by forcing a firmware update using the following steps:
  • Connect to the LCR and perform a Runtime Restart + GPS Firmware Update
  • 
  • Connect to the LCR and confirm the GPS is achieving float/fix.

If the GPS never reaches float/fix, check the following:

• Ensure the robot is active by going to the manual UI or AGT screens
• Ensure the sensors/ntrip-corrections service is enabled and has valid entries for gatewayReference, username, and key.
• Ensure the corrections provider defined by gatewayReference is online and functional:
  • e.g. Check station is shown in green at https://www.myfloridagps.com/DMap/ for Florida LCRs
• Check the sensors/ublox-gps-serial/default/stats and ensure the gpsRtcm3Rx value is incrementing (increments each time we receive a correction from the NTRIP service)
• Ensure the modem firewall is configured to allow GPS related subnets
  • Login to the modem and go to Advanced -> Firewall -> Content Blocking
  • Check that IPs 40.121.5.206 and 69.44.86.36 are included in the Exempted Subnets
  • 
• If all is accurate and still no response from NTRIP, send an email to all three listed emails, do not email the support email directly. They may respond, but will take longer.

The Pico Brain (#004216) consists of a Rasperry Pi Pico micro controller + an Adafruit TDK InvenSense ICM-20948 9-DoF IMU.

The IMU must be calibrated when:

• No Calibration data exists (new sensor, new Pi image)
• Too far from previous calibration (Transported LCR, large worksite)
• Temperature is significantly different than temperature during stored calibrations.

Please refer to the Calibration Guide to calibrate the IMU. (ensure the instructions are up to date)

View calibration info when connected to an LCR at: http://<ROBOT-IP>:8000/actuators/pico-controllers/brain-4216

1. LCR Fleet Alerts - Autonomous task disabled cause large heading error switching to idle
2. LCR AGT Troubleshooting - gpsimu
3. LCR AGT Troubleshooting - large heading error switching to IDLE

The output of the Pico IMU can be found at the following link when connected to an LCR: http://<ROBOT-IP>:8000/data-streams/actuators/pico-controllers/brain-4216 Follow the steps in the following order for debugging:

1. Check if the data stream at the link above contains IMU data and if it updates on a refresh. Example data stream: {"timeMicros":1666371365451000,"flags":["ENABLED","FROZEN"],"scalars":{"pitchDeg":1.69,"temp":38.74,"headingDeg":308.0217624867827,"magHeadingDeg":308.0217624867827,"temp/IMU":30.5,"magHeadingOffDeg":-90.0,"magAccDeg":10.0,"rollDeg":0.43,"faultStatusFlag":0.0},"scalarMaps":{"volts":{"CHANNEL1":0.175,"CHANNEL2":0.587,"CHANNEL0":0.628,"CHANNEL3_ONE_THIRD_VSYS":0.583}},"vectors":{"magXYZ":{"x":0.09701124219518159,"y":-0.0924756391276567,"z":0.07585284622423812},"IMUAccelGyroAxisX":{"x":-1.0,"y":0.0,"z":0.0},"IMUAccelGyroAxisY":{"x":0.0,"y":0.0,"z":-1.0},"IMUAccelGyroAxisZ":{"x":0.0,"y":1.0,"z":0.0},"rawAngVel":{"x":0.976,"y":0.884,"z":-0.366},"IMUMagAxisX":{"x":-1.0,"y":0.0,"z":0.0},"IMUMagAxisZ":{"x":0.0,"y":1.0,"z":0.0},"IMUMagAxisY":{"x":0.0,"y":0.0,"z":1.0},"angVel":{"x":1.159,"y":0.5489999999999999,"z":-0.884},"accel":{"x":0.053,"y":1.08,"z":0.035},"rawMagXYZ":{"x":0.2643,"y":-0.5286,"z":0.0387}},"tags":{"activeIMUModel":"ICM20948","id":"LCR24ZS0-706798070153183e7"},"documentVersion":4654,"documentKind":"cap-ds","documentSelfLink":"/data-streams/actuators/pico-controllers/brain-4216","documentUpdateTimeMicros":1666371365451005,"documentUpdateAction":"PATCH","documentExpirationTimeMicros":0,"documentAuthPrincipalLink":"/core/authz/system-user"}
2. Check the robot logs to see if there are any warnings associated with the PicoControlService
3. If this does not work, and the IMU is connected but still not updating, try power cycling the USB Hub via the tray relay. Note, you will need to restart the robot after this to re-enumerate the Motor Controllers after this.

This is done by navigating to the settings menu and searching for the tray (4260) as seen below

Then in the relay control, turn the USB hub first off, then turn back on.

Finally, after restarting the robot, check the system log of the robot and verify the device is back, and verify it is reporting through the pico brain stream(text).

4. If the device is not updating ensure it shows on the list of serial devices actuators/pico-controllers/brain-4216. If it shows up skip the subsections below and continue at step 4.

4.i. Check if there are any kernel logs of the device dropping in/out by navigating to the stream(text) of the kernel log as pictured

The logs may look like this:

[Fri Oct 21 09:00:14 2022] usb 1-1.4.4: USB disconnect, device number 26
[Fri Oct 21 09:00:15 2022] usb 1-1.4.4: new full-speed USB device number 52 using xhci_hcd
[Fri Oct 21 09:00:15 2022] usb 1-1.4.4: New USB device found, idVendor=2e8a, idProduct=000a, bcdDevice= 1.00
[Fri Oct 21 09:00:15 2022] usb 1-1.4.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[Fri Oct 21 09:00:15 2022] usb 1-1.4.4: Product: Pico 004216
[Fri Oct 21 09:00:15 2022] usb 1-1.4.4: Manufacturer: Raspberry Pi
[Fri Oct 21 09:00:15 2022] usb 1-1.4.4: SerialNumber: DF6098C0D7614B27
[Fri Oct 21 09:00:15 2022] cdc_acm 1-1.4.4:1.0: ttyACM6: USB ACM device

4.ii. If it is not showing up visually inspect the USB connections on both sides to see that they are plugged in and that the end nearest the microcontroller is not plugged in upside down.
4.iii. Power cycle the pico by physically unplugging and replugging and restart the runtime.

5. If it shows up, restart the runtime and see if it connects. Look at the robot logs/data stream and see if the pico successfully connects on startup.
6. If it does not you can ssh into the robot and connect to the device directly to over serial and see if it responds to commands (contact software team to assist)
7. Reboot the robot, wait for the runtime to start, and check the robot logs/data stream and see if the pico successfully connects on startup.
8. If all else fails, the IMU will need replaced.

• The 004260 Pico Tray is the "brain" of the 004679 DM (A) Tray Brain and 003988 DM (A) SCC Alternative (AKA AltSCC).
  • See diagrams of 004679 and 003988

• This alert signals a short between the USB ground and the frame/chassis. This is a newer feature implemented in the Tray Controller 1.2 and 1.3.
• The short may be continuous (always shorted) or intermittent (shorted when the LCR is moving or external relays/sensors are turned on)
  • The email alert is only sent once per session, so will always send on restart for a continuous short
  • The frameShortDetectedCount stat under http://<IP_ADDRESS>:8000/actuators/pico-controllers/tray-4260/stats can help determine this. If the "latestValue" is equal to the "receivedReportsCount" stat "latestValue" then the short is continuous. If the numbers do not match, it is intermittent, or is a continuous short that started after the last CAP restart.:

{"name":"frameShortDetectedCount","accumulatedValue":0.0,"lastUpdateMicrosUtc":1718133062263002,"latestValue":308.0,"version":308},
{"name":"receivedReportsCount","accumulatedValue":0.0,"lastUpdateMicrosUtc":1718316281624001,"latestValue":25167.0,"version":25167}

• False positive alerts can be caused by:
  • Older versions of Pico Tray (AltSCC or Tray Controller 1.1)
  • The alerts can be overridden by setting a host-specific override to dCentralizedSystems/cap-config/config/actuators-pico-controllers-tray-4260.json with "disableTrayShortDetection": true,, or by connecting pin 19/GPIO14 of the green Pico tray PCB to 3.3V.
• False negative alerts can be caused by:
  • The mounting pad labeled "FRAME SHORT" in the linked image is not connected to a metal standoff or mounted to a nylon washer.
• Follow specific steps linked here to verify the frame short: https://dcentralized.sharepoint.com/:p:/s/dCentralizedSystems/Eej4GSVo2HtNpt14TuwlrocBGSeXskhhdiBYJR88X5HeBw?e=QhiTL9
• If not a false positive, follow steps in Electronics and Chassis Short Troubleshooting to diagnose the cause of the short.

LCRs will have one of two current sensors.

• The 50-S is the correct model number, and no action needs to be taken.
• If the LCR has a 100-S, the charge rate reported while charging will be half of the actual charge rate. To fix this, Alt + Left Click on the Battery in the LCR UI to access the Configure Components page and switch the Current Sensor Type to LEM_HO_100_S.

Out of range output detected (-72.79 amps). Inspect Pico Tray 004260 and sensor wiring.

• This alert is caused by
  • high-current charging (no action)
  • a faulty sensor/sensor wiring (check the ADC wiring and voltages in the following section)

Unexpected ADC reference voltage (measured: 0.89, expected: 1.65). Inspect Pico Tray 004260 and sensor wiring.

• The ADC reference voltage is outside of the acceptable range
• Check the data stream (stream (text)) for the actuators/pico-controllers/tray-4260 service (http://<IP_ADDRESS>:8000/data-streams/actuators/pico-controllers/tray-4260?jsonStyle=HTML)
  • Example below, CHANNEL0 is the Vref from the current sensor and should be ~1.65V. CHANNEL1 is the Vout from the current sensor and should be ~1.65V but higher/lower when large positive/negative currents are being sensed.

  "scalarMaps": {
    "volts": {
      "CHANNEL1": 1.687,
      "CHANNEL2": 0.757,
      "CHANNEL0": 1.685,
      "CHANNEL3_ONE_THIRD_VSYS": 1.646
    }

• If the CHANNEL 0 and 1 voltages above do not look correct, the blue current sensor (004317 - Sensor, Current LEM 50A) may
  • be broken
  • the plug may not be fully seated
  • the wiring between the current sensor plug and the pico may be disconnected
  • may require a power cycle
• If debugging remotely:
  • attempt to reboot the pi to power cycle the Pico Tray and current sensor (Pico Tray must be connected to the Pi USB port directly, if connected to USB Hub a hub power cycle must be done)
  • If this fails, schedule an in person visit to debug further.
• If debugging in person:
  • First try unplugging the Pico Tray from the Pi, then unplugging/replugging the current sensor. Restart the robot and check if the robot is now reporting the expected wattage
  • measure the output voltages from the current sensor with a multimeter to see if the plug is powered properly and outputting correct voltages.
    • OCD: Skip, not connected
    • Vref: Should be ~1.65V
    • Vout: Should be ~1.65V
    • GND: Should be 0V and have continuity with 12V GND on the Tray Brain
    • +Uc: Should be 3.3V
• If there appears to be physical damage, shorting, or the current sensor isn't reporting properly in person and appears physically undamaged, then we will need to replace the current sensor.