• Overview
• SSH Access
• Realsense Cameras
• GPS Sensor
• Inertial Measurement Unit
• Pico Tray Brain
  • Current Sensor

This page helps with narrowing down sensor faults. It covers advanced topics that require knowledge of basic linux commands.

We want to avoid physically touching the robot during the sensor debugging except as a last resort so we know if the problem is something that we could recover remotely using software commands or if it requires physical intervention. These are the types of questions we're trying to answer in the debugging process:

1. What caused the issue to happen how how can it be avoided in the future?
2. Can it be recovered with a CAP restart?
3. Can it be recovered with a command line script? (In the future we could run this script automatically from CAP)
4. Can it be recovered with a Pi reboot?
5. Can it be recovered with a physical power cycle (unplug/replug, last resort)

Troubleshooting may require a connection to the robot operating system through SSH (secure shell). Steps to establish an SSH connection

1. Connect a PC to the robot WiFi
2. Open a terminal on the PC side (Linux or Mac OSX, windows requires a PuTTY terminal)
3. In the terminal window
4. ping edge.directedmachines.com
5. if the ping is not successful, try ping with either 192.168.8.100 or 192.168.50.10 as the IP address
6. select the IP address that worked
7. ssh pi@<ipaddress>
8. you will be prompted for a password. The password will either be given to you by the software team, or, your PC will have a private key file that enables it to connect without password

• Serial devices can be listed by their USB port topology path (by-path) or by more human readable description (by-id)
• The list of usb serial devices (excluding cameras) can be found in the UI under:
  • http://<IP-ADDRESS>:8000/dashboard/devices/serial
• It can also be printed over ssh using ls -al /dev/serial/by-id and ls -al /dev/serial/by-path
• NOTE: if devices are shown by-path but not by-id then there is likely an issue with the kernel versions. See this thread as reference and an example of the device list below:

{
  "devs_by_id": {},
  "devs_by_path": {
    "/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.4:1.0": "../../ttyACM5",
    "/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.1:1.0": "../../ttyACM2",
    "/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.2:1.0": "../../ttyACM3",
    "/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.3:1.0": "../../ttyACM4",
    "/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.1:1.0": "../../ttyACM1",
    "/dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.3:1.0": "../../ttyACM0"
  },
  "documentVersion": 0,
  "documentUpdateTimeMicros": 0,
  "documentExpirationTimeMicros": 0
}

Camera Sensor Troubleshooting

All robots have a GPS sensor positioned above the intersection of the robot midline and the main drive axles.

• Real-time kinematic positioning (RTK) is the application of surveying to correct for common errors in current satellite navigation (GNSS) systems
• NTRIP stands for "Networked Transport of RTCM via Internet Protocol".
• RTCM is a communication protocol for sending differential GPS (DGPS) to a GPS receiver from a secondary source
• Differential GPS uses GPS basestations with a known survey location to produce corrections that improve the accuracy of nearby (ideally <25 km) mobile GPS receivers. It can decrease from +/- ~3m accuracy to +/- ~3 cm accuracy.
• This accuracy is impacted by obstacles such as buildings/trees so even with GPS corrections the accuracy may be low

• The GPS accuracy is shown in mm under the "GPS" header in the top-right corner of the screen in the manual and AGT UIs. If the color is grey, there is no RTK lock, if the color is yellow, it is in RTK float, if green it is RTK Fix.
• To save data the GPS corrections are not applied when idle, when transistioning from idle to active (e.g., connecting to an LCR after a long period of inactivity), please allow a couple minutes for the GPS to lock into the fix.

• An inaccurate GPS reading can be resolved by forcing a firmware update using the following steps:
  • SSH into the robot and stop the runtime: sudo systemctl stop dcentralizedcaphost.service
  • Run the script ~/dCentralizedSystems/cap-provisioning/pi/scripts/firmware/GPS/ZED-F9P/updateFirmwareZEDF9P.sh and ensure it exits successfully
  • Restart the runtime: sudo systemctl restart dcentralizedcaphost.service
  • Connect to the LCR and confirm the GPS is achieving float/fix.

If the GPS never reaches float/fix, check the following:

• Ensure the robot is active by going to the manual UI or AGT screens
• Ensure the sensors/ntrip-corrections service is enabled and has valid entries for gatewayReference, username, and key.
• Ensure the corrections provider defined by gatewayReference is online and functional:
  • e.g. Check station is shown in green at https://www.myfloridagps.com/DMap/ for Florida LCRs
• Check the sensors/ublox-gps-serial/default/stats and ensure the gpsRtcm3Rx value is incrementing (increments each time we receive a correction from the NTRIP service)
• Ensure the modem firewall is configured to allow GPS related subnets
  • Login to the modem and go to Advanced -> Firewall -> Content Blocking
  • Check that IPs 40.121.5.206 and 69.44.86.36 are included in the Exempted Subnets
  • 

The Pico Brain (#004216) consists of a Rasperry Pi Pico micro controller + an Adafruit TDK InvenSense ICM-20948 9-DoF IMU.

The IMU must be calibrated when:

• No Calibration data exists (new sensor, new Pi image)
• Too far from previous calibration (Transported LCR, large worksite)
• Temperature is significantly different than temperature during stored calibrations.

Please refer to the Calibration Guide to calibrate the IMU. (ensure the instructions are up to date)

View calibration info when connected to an LCR at: http://<ROBOT-IP>:8000/actuators/pico-controllers/brain-4216

1. LCR Fleet Alerts - Autonomous task disabled cause large heading error switching to idle
2. LCR AGT Troubleshooting - gpsimu
3. LCR AGT Troubleshooting - large heading error switching to IDLE

The output of the Pico IMU can be found at the following link when connected to an LCR: http://<ROBOT-IP>:8000/data-streams/actuators/pico-controllers/brain-4216 Follow the steps in the following order for debugging:

1. Check if the data stream at the link above contains IMU data and if it updates on a refresh. Example data stream: {"timeMicros":1666371365451000,"flags":["ENABLED","FROZEN"],"scalars":{"pitchDeg":1.69,"temp":38.74,"headingDeg":308.0217624867827,"magHeadingDeg":308.0217624867827,"temp/IMU":30.5,"magHeadingOffDeg":-90.0,"magAccDeg":10.0,"rollDeg":0.43,"faultStatusFlag":0.0},"scalarMaps":{"volts":{"CHANNEL1":0.175,"CHANNEL2":0.587,"CHANNEL0":0.628,"CHANNEL3_ONE_THIRD_VSYS":0.583}},"vectors":{"magXYZ":{"x":0.09701124219518159,"y":-0.0924756391276567,"z":0.07585284622423812},"IMUAccelGyroAxisX":{"x":-1.0,"y":0.0,"z":0.0},"IMUAccelGyroAxisY":{"x":0.0,"y":0.0,"z":-1.0},"IMUAccelGyroAxisZ":{"x":0.0,"y":1.0,"z":0.0},"rawAngVel":{"x":0.976,"y":0.884,"z":-0.366},"IMUMagAxisX":{"x":-1.0,"y":0.0,"z":0.0},"IMUMagAxisZ":{"x":0.0,"y":1.0,"z":0.0},"IMUMagAxisY":{"x":0.0,"y":0.0,"z":1.0},"angVel":{"x":1.159,"y":0.5489999999999999,"z":-0.884},"accel":{"x":0.053,"y":1.08,"z":0.035},"rawMagXYZ":{"x":0.2643,"y":-0.5286,"z":0.0387}},"tags":{"activeIMUModel":"ICM20948","id":"LCR24ZS0-706798070153183e7"},"documentVersion":4654,"documentKind":"cap-ds","documentSelfLink":"/data-streams/actuators/pico-controllers/brain-4216","documentUpdateTimeMicros":1666371365451005,"documentUpdateAction":"PATCH","documentExpirationTimeMicros":0,"documentAuthPrincipalLink":"/core/authz/system-user"}
2. Check the robot logs to see if there are any warnings associated with the PicoControlService
3. If the device is not updating ensure it shows on the list of serial devices (ssh into robot and run ls -al /dev/serial/by-id). If it shows up skip the subsections below and continue at step 4.

3.i. Check if there are any kernel logs of the device dropping in/out by running dmesg -T:

[Fri Oct 21 09:00:14 2022] usb 1-1.4.4: USB disconnect, device number 26
[Fri Oct 21 09:00:15 2022] usb 1-1.4.4: new full-speed USB device number 52 using xhci_hcd
[Fri Oct 21 09:00:15 2022] usb 1-1.4.4: New USB device found, idVendor=2e8a, idProduct=000a, bcdDevice= 1.00
[Fri Oct 21 09:00:15 2022] usb 1-1.4.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[Fri Oct 21 09:00:15 2022] usb 1-1.4.4: Product: Pico 004216
[Fri Oct 21 09:00:15 2022] usb 1-1.4.4: Manufacturer: Raspberry Pi
[Fri Oct 21 09:00:15 2022] usb 1-1.4.4: SerialNumber: DF6098C0D7614B27
[Fri Oct 21 09:00:15 2022] cdc_acm 1-1.4.4:1.0: ttyACM6: USB ACM device

3.ii. If it is not showing up visually inspect the USB connections on both sides to see that they are plugged in and that the end nearest the microcontroller is not plugged in upside down.
3.iii. Power cycle the pico by physically unplugging and replugging and restart the runtime.

4. If it shows up, restart the runtime and see if it connects. Look at the robot logs/data stream and see if the pico successfully connects on startup.
5. If it does not you can ssh into the robot and connect to the device directly to over serial and see if it responds to commands (contact software team to assist)
6. Reboot the robot, wait for the runtime to start, and check the robot logs/data stream and see if the pico successfully connects on startup.

• The 004260 Pico Tray is the "brain" of the 004679 DM (A) Tray Brain and 003988 DM (A) SCC Alternative (AKA AltSCC).
  • See diagrams of 004679 and 003988

tray-4260-current
Unexpected ADC reference voltage (measured: 0.89, expected: 1.65). Inspect Pico Tray 004260 and sensor wiring.
Click for telemetry details

• The ADC reference voltage is outside of the acceptable range
• Check the data stream (stream (text)) for the actuators/pico-controllers/tray-4260 service (http://<IP_ADDRESS>:8000/data-streams/actuators/pico-controllers/tray-4260?jsonStyle=HTML)
  • Example below, CHANNEL0 is the Vref from the current sensor and should be ~1.65V. CHANNEL1 is the Vout from the current sensor and should be ~1.65V but higher/lower when large positive/negative currents are being sensed.

  "scalarMaps": {
    "volts": {
      "CHANNEL1": 1.687,
      "CHANNEL2": 0.757,
      "CHANNEL0": 1.685,
      "CHANNEL3_ONE_THIRD_VSYS": 1.646
    }

• If the CHANNEL 0 and 1 voltages above do not look correct, the blue current sensor (004317 - Sensor, Current LEM 50A) may be broken, or the plug may not be fully seated, or the wiring between the current sensor plug and the pico may be disconnected.
  • Measure the output voltages from the current sensor with a multimeter to see if the plug is powered properly and outputting correct voltages.
    • OCD: Skip, not connected
    • Vref: Should be ~1.65V
    • Vout: Should be ~1.65V
    • GND: Should be 0V and have continuity with 12V GND on the Tray Brain
    • +Uc: Should be 3.3V

tray-4260-current
Out of range output detected (-72.79 amps). Inspect Pico Tray 004260 and sensor wiring.
Click for telemetry details

• This is likely caused by high-current charging, but may be caused by a faulty sensor as well.