• Overview
• Safety
• Batteries
• Motor controllers
  • Motor Controller Troubleshooting Tree
  • 300 Amp Motor Controllers
  • 300 Amp Motor Controller FW Update
  • 25A Cytron Motor controller

WARNING: This is an advanced guide for trained field technicians, do not attempt unless you have training or specific guidance from the Directed Machines support team ([email protected]).

• Make sure robot e-stop is active
• Disable drive motors using side rotary switch
• If tray cover is opened, you might need to disconnect batteries from tray
• Full shutdown Instructions

For instructions on how to access the battery compartment while in the field, follow this presentation

Here is a video showing the process of replacing a terminal on the LiFePO4 batteries: https://dcentralized.sharepoint.com/:v:/s/dCentralizedSystems/ESdk0eRiADVKn4fOQktm5pUBmMIzwwHOZeqqAxTNtCAxxg?e=wq7zfS Only execute this process if you have a screwdriver, 1/2" wrench, extra terminal, and a dry work environment.

• 300A Motor Controllers power Left/Right drive and AUX motors (e.g, mower, PTO).
• They are located inside the electronics tray and plugged into the USB hub
• Diagnose MC issues using the Motor Controller Troubleshooting Tree

• Detailed logging is available if add "options":["CONSOLE_LOGGING"], in the configuration file for the faulty motor controller(s) and restart CAP to enable verbose logging from the motor controller service.
• Logs only printed when the motor controller receives non-zero motor commands so the robot must be driving or with the drive switch off + motor commands sent to see logs.
• Remember to restore JSON file to original configuration
• If console logging is enabled and needs disabled, navigate to the corresponding MC UI and simply toggle the green CONSOLE_LOGGING button and restart the robot, when you navigate back the console logging button will be grey, showing that it's disabled.
• If console logging is stuck in an on state, first try to restore defaults in the MC UI and restart the robot. If console logging is still stuck ON, we want to SSH into the robot, and git checkout actuators-dcentralized-motor-controllers-default.(ROBOT-ID).json, then restart the robot. If this fails, SSH back into the robot, manually remove the file with rm actuators-dcentralized-motor-controllers-default.(ROBOT-ID).json and restart the robot.

The MC serial port driver can occasionally fail to initialize properly. This can manifest itself with a spamming of the following logs:

"[Trying with port name variation /dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.2:1.0-port0, failure opening name /dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.2:1.0: jssc.SerialPortException: Port name - /dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.2:1.0; Method name - openPort(); Exception type - Permission denied.]",

"[8000/actuators/dcentralized-motor-controllers/default][initializeMotorControllers][jssc.SerialPortException: Port name - /dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.3.2:1.0-port0; Method name - openPort(); Exception type - Port not found.",

This is usually resolved by rebooting the Pi runtime or following the Recovery Steps below. If this does not resolve the issue, the MC, USB cable, and USB hub should be inspected.

On occasion the MCs will be stuck in DFU mode (FW update mode) and runtime or leaveDFU.sh script will fail to recover them. Forcing a FW update will usually restore them to normal operation

1. SSH into robot
2. stop runtime - sudo systemctl stop dcentralizedcaphost.service
3. Run FW update script: ~/dCentralizedSystems/cap-provisioning/pi/scripts/firmware/DM_MC300A_32L/updateFirmwareDM300AMC.sh
   1. wait for all MCs to be updated
4. restart runtime - sudo systemctl start dcentralizedcaphost.service

The steps below pertain for MCs not enumerating on USB bus. If the kernel logs show the "STM Bootloader" logs, then controller is in DFU mode and different steps apply (see unenumerated controller section) To attempt recovery of MC, so its seen by USB bus:

1. Restart the CAP runtime
2. Reboot
3. Use the Pico Tray UI to power cycle the USB Hub Relay, then restart the runtime
4. Power cycling the logic side of the faulty motor controller. This can be done for a motor controller by unplugging and replugging the USB cable if in person, or running /cap-provisioning/pi/scripts/cycleHubPower.sh if cycling remotely and 60W 7-port HUB is installed.
5. Power cycling the USB hub by unplugging and replugging the barrel connector (ensure it is not powering the Pi before doing this)
6. Full power cycle.

If the motor controller is not found in CAP, it may be plugged into the incorrect port. SSH into the robot and list the ports using ls -al /dev/serial/by-id and compare with the "portName" or "altPortName" in the motor controller .json files.

• L/R Motors: actuators-dcentralized-motor-controllers-default.json
• AUX Motor: actuators-dcentralized-motor-controllers-aux1.json

If Manual UI is indicating that the LEFT or RIGHT or AUX or all of them are unenumerated, the USB hub is likely damaged or in a bad state (may be fixed by hub power cycle). SSH into the robot, display kernel messages and look for USB errors activate or enumerating ports. Please review sensor troubleshooting and SSH pages for more details on kernel message diagnosis.

May be indicated by the following alert:

• LEFT_MOTOR: device address opened previously but no longer exists, check for USB disconnects or chassis shorts

Follow the Electronics and Chassis Short Troubleshooting Wiki

The motor controller service will send alerts when the MC reports faults through FW and serial connection. Details on MC protocol and faults in the firmware wiki (internal access only). Major faults reported through email alerts outlined below

Controller is using maximum current. Can be caused by:

• AUX (mower) when the blades stall or when stopping and starting quickly (regen)
• Drive wheel controllers when robot stalls (wheels have traction but robot cannot move). Autonomy will stop on its own on stall.

If the LCR is idle and has an over-current fault it is likely due to:

1. LCR is undergoing maintenance but the drive motor switch is still on (Left and Right motors), or the 175A Anderson connector for the Aux motor is still connected (AUX) but mower is being rotated by hand (very dangerous). Contact customer or support engineer and reiterate the safe maintenance procedures.
2. LCR is being transported in a vehicle and rolling is causing motor regen. Motor disconnect switch can be turned off during transport to suppress these errors.
3. LCR was parked on a hill and is rolling down it. See example photo of telemetry showing steep pitch and motor regen (negative watts) leading to fault: current-when-idle-hill.png. Contact customer or support engineer and reiterate the safe operation/parking procedures.
4. A short or damage to the motor controller. Contact customer or support engineer to diagnose further. See the following email alert as an example and note it is idle ("I: 1" in the report):

RIGHT_MOTOR HW fault: R: M:0 T:0 P:0 V:0 T1:027.15C T2:026.90C VB:54.77V M+:06.97V M-:10.69V IF:-000.06A IR:-000.28A W:900 MF:1 FS:4194320(0) ES:1 I:1 300AMC Firmware Faults: MC_STATUS_OVER_CURRENT_ERROR: MC_STATUS_SYSTEM_SAFETY_FAULT: DRV8353 Faults: none.

Pay attention to the alert and look for temperatures close to 80C. If temperatures are close to 80C, stop using the robot, give it 10 minutes to cool off

If motor controller functionality is not working (i.e. 4th motor controller not able to read lift arm). First open the process log to see if the motor controller is enumerating (see image below)

Here all four motor controllers are enumerating. The next step is to look at the service log (see image below)

The log tells us that a motor controller is enumerating but is expected in a different port. Verify that motor controllers are plugged into the correct ports.

• May show up if the following is seen in kernel: "May 8 14:04:19 LCR24ZS0-756933705b2f75136 kernel: [796998.621627] usb 1-1.4.3.1: USB disconnect, device number 29", "May 8 14:04:19 LCR24ZS0-756933705b2f75136 kernel: [796998.896420] usb 1-1.4.3.1: new full-speed USB device number 30 using xhci_hcd", "May 8 14:04:19 LCR24ZS0-756933705b2f75136 kernel: [796999.049613] usb 1-1.4.3.1: New USB device found, idVendor=0483, idProduct=df11, bcdDevice=22.00", "May 8 14:04:19 LCR24ZS0-756933705b2f75136 kernel: [796999.049641] usb 1-1.4.3.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3", "May 8 14:04:19 LCR24ZS0-756933705b2f75136 kernel: [796999.049650] usb 1-1.4.3.1: Product: STM32 BOOTLOADER", "May 8 14:04:19 LCR24ZS0-756933705b2f75136 kernel: [796999.049657] usb 1-1.4.3.1: Manufacturer: STMicroelectronics", "May 8 14:04:19 LCR24ZS0-756933705b2f75136 kernel: [796999.049663] usb 1-1.4.3.1: SerialNumber: 2061339B4256"
• USB cable from MC to HUB not plugged in or damaged
• The motor controller might be stuck in Device Firmware Upgrade (DFU) mode
  • restart runtime, may successfully kick them out of dfu mode
  • Check directly if the motor controllers are in DFU mode, if restart did not help
    • SSH in the robot and execute the command dfu-util -l
  • If in DFU mode:
    • Update firmware
    • Full robot battery disconnect/power cycle
• USB hub is damaged due to ground loop. Look at kernel log messages, look for USB errors activating USB ports, and no USB peripherals are enumerating.

The following Fleet Alert email is also indicative of this problem:

• A DM (A) MC AUX Dongle (P/N 004500) is installed on AUX MCs to allow CAP to differentiate them from the L/R MCs.
• This is a safety measure to prevent USB cables from being wired improperly
  • Prevents sending high power to a L/R drivetrain motor when trying to power the AUX mower motor
• Legacy robots without the dongle still function, but the AUX MC will display "E" on the LED display

The 300A motor controllers have 7-segment LED displays that provide information about the MC state. The display depends on whether a DM (A) MC AUX Dongle is installed, and which Hub USB port the MC is connected to.

• A dongle should only be installed on the AUX MC (see previous section)
• If a dongle is installed, the MC will display "A" using it's internal firmware even when there is no communication with our CAP software on the Pi.
• When CAP software connects, if it expects it to be a L/R MC it will display an "E". This is a hard error and the MC will not be functional.

• If no dongle, the MCs do not display anything until they communicate with the Pi and then the software tells it what letter to display.
  • If pi software expects it to be a L/R MC it will display an "L/P" (P represents R for "right")
  • If pi software expects it to be an AUX MC it will display an "E" since it is an AUX MC and there is no dongle. This is a soft error and the MC will still be functional to support legacy robots without dongles.

Product Page: https://www.cytron.io/p-25amp-7v-58v-high-voltage-dc-motor-driver
Manual: MD25HV Datasheet.pdf

• Is the green/yellow PWR LED in the top right corner of the board on? (this is means the board is receiving power)
• When you press the rocker switch do the red LEDs labeled MA" and "MB" turn on?
• Check if "ERR" LED is flashing. If flashing, note which error state using the figure below. To recover from an error state the board must be power cycled. This can be done using the busbar connector on newer models or by following the high current power cycle on older models.
• Check the voltage across the MA and MB terminals using a multimeter, being careful not to short the leads.
• Are the linear actuator(s) on the 3PT hitch connected to the Molex connector(s) on the motor controller?

• Check if the pico tray light is flashing
• Check connections between pico tray and cytron motor controller

If the battery voltage isn't reported we need to check each MC stream and check their reported voltages. The AUX MC stream is found at actuators/motor-controllers/aux1, after opening the stream(text) json, the user will find the battery voltage as displayed in the picture

This image is also how the voltage will display in the drive MC stream. The drive MC stream is located at actuators/motor-controllers.

If these streams are not reporting the battery voltage, either not displaying any voltage or are otherwise displaying an unexpected readout, there is damage to the corresponding motor controller and it will need to be replaced.