LCR USB Side Camera Troubleshooting - directedmachines/customer-support GitHub Wiki

Table of Contents

Overview

The LCR uses optional USB Color cameras, mounted on the side, to aid with remote autonomy

Symptoms

  • Camera stream (color or depth) is not updating in side camera views of AGT
  • Camera stream in camera UI page not updating, no image shown

Please see below for more detailed descriptions and mitigation

Root Causes

  • USB Cable to camera and to Raspberry PI is loose or damaged
  • USB cable plugged into the wrong port of second 7 port USB HUB
  • Camera connector is corroded or damaged
  • Camera internal electronics are corroded or damaged
  • Camera frame touches the aluminum mount. Camera must be electrically isolated from frame

Note: A USB level fault will cause other USB peripherals, such as motor controllers and GPS, to also malfunction. If GPS is not reporting, the cause is likely a bad camera.

SSH

The operator is expected to have SSH access to the robot, so they can log in and use a terminal session to issues the commands listed in the following sections

Useful commands when sshing

  1. See all image files being created: ls -al /var/tmp/images
  2. List all running dm-capture processes: ps aux | grep dm-capture | grep -v grep
  3. View image capture log files. Logs are automatically deleted on CAP restart
    1. Print all logs cat /var/tmp/images/*.log
    2. Tail all logs tail -f /var/tmp/images/*.log
  4. list all cameras
    1. Legacy method: ls -al /sys/class/video4linux
    2. Updated dev method:
      1. By ID (warning not all devices shown): ls -al /dev/v4l/by-id
      2. By path: ls -al /dev/v4l/by-path
      3. By path, proper video streams only: ls -al /dev/v4l/by-path | grep index0 | grep usb
  5. List device info about a video device (e.g., video0): udevadm info /dev/video0
  6. Get info about a specific stream (-d4 is short for --device=/dev/video4) : v4l2-ctl -d4 --list-formats-ext
  7. List if pi camera is detected: vcgencmd get_camera

  • SSH into the robot using the robot IP address (either IP addressed used when connected through WIFI (usually 192.168.50.10), or static VPN IP)

  • issue the command listed before the text output, in the sections below. For example, type: "dmesg" + ENTER to get kernel log output

  • First run ls -al /dev/v4l/by-id to list video4linux (v4l) devices by id. The USB cameras have the same name so they're only shown once by-id. See video28 and 29 below. Since they end in "index0 and index1" they're the same camera.

ls -al /dev/v4l/by-id
total 0
drwxr-xr-x 2 root root 240 Aug  4 07:27 .
drwxr-xr-x 4 root root  80 Aug  4 07:27 ..
lrwxrwxrwx 1 root root  13 Aug  4 07:27 usb-HD_USB_Camera_HD_USB_Camera-video-index0 -> ../../video28
lrwxrwxrwx 1 root root  13 Aug  4 07:27 usb-HD_USB_Camera_HD_USB_Camera-video-index1 -> ../../video29
lrwxrwxrwx 1 root root  12 Aug  4 07:27 usb-Intel_R__RealSense_TM__Depth_Camera_455_Intel_R__RealSense_TM__Depth_Camera_455_208223062991-video-index0 -> ../../video4
lrwxrwxrwx 1 root root  12 Aug  4 07:27 usb-Intel_R__RealSense_TM__Depth_Camera_455_Intel_R__RealSense_TM__Depth_Camera_455_208223062991-video-index1 -> ../../video5
lrwxrwxrwx 1 root root  12 Aug  4 07:27 usb-Intel_R__RealSense_TM__Depth_Camera_455_Intel_R__RealSense_TM__Depth_Camera_455_208223062991-video-index2 -> ../../video2
lrwxrwxrwx 1 root root  12 Aug  4 07:27 usb-Intel_R__RealSense_TM__Depth_Camera_455_Intel_R__RealSense_TM__Depth_Camera_455_208223062991-video-index3 -> ../../video3
lrwxrwxrwx 1 root root  13 Aug  4 07:27 usb-Intel_R__RealSense_TM__Depth_Camera_455_Intel_R__RealSense_TM__Depth_Camera_455_208223063292-video-index0 -> ../../video11
lrwxrwxrwx 1 root root  13 Aug  4 07:27 usb-Intel_R__RealSense_TM__Depth_Camera_455_Intel_R__RealSense_TM__Depth_Camera_455_208223063292-video-index1 -> ../../video12
lrwxrwxrwx 1 root root  12 Aug  4 07:27 usb-Intel_R__RealSense_TM__Depth_Camera_455_Intel_R__RealSense_TM__Depth_Camera_455_208223063292-video-index2 -> ../../video8
lrwxrwxrwx 1 root root  12 Aug  4 07:27 usb-Intel_R__RealSense_TM__Depth_Camera_455_Intel_R__RealSense_TM__Depth_Camera_455_208223063292-video-index3 -> ../../video9
  • Second run ls -al /dev/v4l/by-path to list video devices by usb path. You can look for the video device number 28 and 29 above in the list below:
ls -al /dev/v4l/by-path
total 0
drwxr-xr-x 2 root root 480 Aug  4 07:27 .
drwxr-xr-x 4 root root  80 Aug  4 07:27 ..
lrwxrwxrwx 1 root root  13 Aug  4 07:27 platform-bcm2835-codec-video-index0 -> ../../video31
lrwxrwxrwx 1 root root  13 Aug  4 07:27 platform-bcm2835-isp-video-index0 -> ../../video23
lrwxrwxrwx 1 root root  13 Aug  4 07:27 platform-bcm2835-isp-video-index1 -> ../../video25
lrwxrwxrwx 1 root root  13 Aug  4 07:27 platform-bcm2835-isp-video-index2 -> ../../video26
lrwxrwxrwx 1 root root  13 Aug  4 07:27 platform-bcm2835-isp-video-index3 -> ../../video27
lrwxrwxrwx 1 root root  12 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1:1.0-video-index0 -> ../../video0
lrwxrwxrwx 1 root root  12 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1:1.0-video-index1 -> ../../video1
lrwxrwxrwx 1 root root  12 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1:1.0-video-index2 -> ../../video2
lrwxrwxrwx 1 root root  12 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1:1.0-video-index3 -> ../../video3
lrwxrwxrwx 1 root root  12 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1:1.3-video-index0 -> ../../video4
lrwxrwxrwx 1 root root  12 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1:1.3-video-index1 -> ../../video5
lrwxrwxrwx 1 root root  13 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.2.3.2:1.0-video-index0 -> ../../video17
lrwxrwxrwx 1 root root  13 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.2.3.2:1.0-video-index1 -> ../../video20
lrwxrwxrwx 1 root root  13 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.2.3.3:1.0-video-index0 -> ../../video28
lrwxrwxrwx 1 root root  13 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.2.3.3:1.0-video-index1 -> ../../video29
lrwxrwxrwx 1 root root  12 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:2:1.0-video-index0 -> ../../video6
lrwxrwxrwx 1 root root  12 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:2:1.0-video-index1 -> ../../video7
lrwxrwxrwx 1 root root  12 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:2:1.0-video-index2 -> ../../video8
lrwxrwxrwx 1 root root  12 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:2:1.0-video-index3 -> ../../video9
lrwxrwxrwx 1 root root  13 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:2:1.3-video-index0 -> ../../video11
lrwxrwxrwx 1 root root  13 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:2:1.3-video-index1 -> ../../video12
lrwxrwxrwx 1 root root  13 Aug  4 07:27 platform-feb10000.codec-video-index0 -> ../../video19
  • In this case we can see the cameras are right next to each other so the usb path should be different by 1 number. The devices we want are the ones that end in index0, so the side cameras are:
lrwxrwxrwx 1 root root  13 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.2.3.2:1.0-video-index0 -> ../../video17
lrwxrwxrwx 1 root root  13 Aug  4 07:27 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4.2.3.3:1.0-video-index0 -> ../../video28
  • Lastly check the configuration files for the USB side camers in dCentralizedSystem/cap-config/config
/home/pi/dCentralizedSystems/cap-config/config/sensors-generic-cameras-usbcam-left-color.LCR24ZS0-7369413914fe9514c.json
/home/pi/dCentralizedSystems/cap-config/config/sensors-generic-cameras-usbcam-right-color.LCR24ZS0-7369413914fe9514c.json

Contact a software team member if the cameras in the list above have the usb1/1-1/1-1.xx prefix and are not streaming images

Inspect Camera UI

First thing to check is to ensure that the camera(s) are enumerated and being picked up by the system.

Check Camera Enumeration

Enumerate the video devices using the command below and expect to find devices linked to /dev/video12 (right camera), /dev/video14(left camera). Note the other camera devices are likely the Realsense depth cameras. Devices on the 1-1 usb1 path are likely the USB color cameras.

To check video enumeration through the UI, search for "dashboard/devices/video".

image

Verify the cameras are listed. The side cameras prefix will start with usb1/1-1/1-1.xx. The side cameras should have a similar look to: "../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.2/1-1.4.2.3/1-1.4.2.3.3/1-1.4.2.3.3:1.0/video4linux/video28".

Cameras are not enumerating

Restore Defaults

  • Open the settings (click the cogwheel)
  • Search for `sensors/generic-cameras/usbcam-left-color' or 'sensors/generic-cameras/usbcam-right-color'

image

  • Click on the "state" button
  • Press RESTORE_DEFAULT. Then press the Patch button.

image

  • Perform a runtime restart by pressing the power icon in the bottom left, and select Restart
  • Once restarted, use the same search query as above, and click on the white button. (i.e. sensors/generic-cameras/usbcam-right-color)
  • Click Enable

From Manual UI (Cameras are Enumerating)

  • Once confirmed cameras are enumerated, continue with the following steps
  • touch gear icon
  • in service search input box, type usb
  • click in the main link for the usb left or right camera
  • expect to see streaming video. If camera disabled, enabled

Example UI URL: http://ipaddress:8000/sensors/generic-cameras/usbcam-left-color/ for the left camera and http://ipaddress:8000/sensors/generic-cameras/usbcam-right-color/ for the right camera.

Ensure Camera Paths are set properly

To verify camera paths, first navigate to the camera in which you are troubleshooting, in this example using the left usb color camera. Select the gear icon in the User Interface. In the search bar type in /sensors/generic-cameras/usbcam-left-color. Then click on the state(text) button. This will pull up information about the camera. Find the address for the current camera in the text file. The address displayed here is what the system is expecting the address to be.

image

You can then verify that the camera is being picked up in the device list by clicking the gear icon and searching 'devices/video' and clicking state(text). The addresses listed here give information for devices that are currently plugged in. You should be able to find a matching path:

image

If a matching file path is not found between the devices/video file and the expected path in the device's information file, then one of two things can be done.

The first option is to remove the host specific file, and perform a runtime restart to reset the expected path to its default. To do this, navigate to the specific camera, i.e. /sensors/generic-cameras/usbcam-left-color and click on the state button (UI page shown below). At the top of the page, click the button RESTORE_DEFAULTS then in the bottom left click PATCH. Perform a runtime restart and recheck expected camera path and detected camera paths.

image

If removing the host specific file did not work, adjust the port in which the device is plugged in. Refresh the devices page and see where the device is now plugged into. If this does not work, you can manually update the expected file path under the specific device. To do this you will need to SSH into the robot. Once SSH'd in, enter the following navigation: cd dCentralizedSystems/cap-config/config

Once there enter ls to see all files. You should see something like this: image

The files with ".LCR24ZS0-*****" are the host specific files. These are the files we will want to change, ensure you are updating the correct file though. If you are having issues with the left usb camera, ensure to update the left usb camera file. While SSH'd you can use Nano or any other terminal text editor. Navigate to the address line, and update it to the populated address from your devices/video list.

Once path is manually updated, perform a runtime restart on the robot. Then retrace the above steps to see if the path updated correctly.

Video Completion handler error

It appears the kernel USBVideo stack gets in a bad state and while the cameras are enumerated, frames can not be acquired. The kernel (dmesg) would have lots of these messages

[249806.659207] uvcvideo: Non-zero status (-75) in video completion handler.
[249806.659511] uvcvideo: Non-zero status (-75) in video completion handler.
[249806.659824] uvcvideo: Non-zero status (-75) in video completion handler.
[249806.660131] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.776957] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.777418] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.777791] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.778165] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.778540] uvcvideo: Non-zero status (-75) in video completion handler.

Please contact member of software team, then reboot, to recover the camera streams

Bad Cable

Using a laptop,

  • SSH into the robot using the robot IP address (either IP addressed used when connected through WIFI (usually 192.168.50.10), or static VPN IP)
  • issue the dmesg linux command
  • look for red USB related error messages

Example of camera USB level errors due to either bad cable or damaged camera connector

$ dmesg

...
[351788.919646] usb usb2-port1: attempt power cycle
[351793.609349] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351797.959404] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351797.959770] usb usb2-port1: unable to enumerate USB device
[351802.279471] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351806.819497] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351811.179569] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351811.179935] usb usb2-port1: attempt power cycle
[351815.859646] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351820.199712] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351820.200079] usb usb2-port1: unable to enumerate USB device
[351824.519759] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351829.059798] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351833.399844] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
...

Damaged camera (corroded electronics)

Another symptom of a faulty camera, manifests as UVC video kernel error messages

$ dmesg
[ 2698.590869] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2698.591144] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2705.820078] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2705.820357] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2712.788413] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2712.788699] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2719.616349] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2719.616633] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).

Camera must be unplugged and replaced. Camera case should be opened and electronics inspected for corrosion

Other kernel logs that might be related:

$dmesg
[52502.095129] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) 430 (8086:0b07)
[52505.324859] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[52505.325136] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[52516.052617] usb 2-1: USB disconnect, device number 69
[52516.189848] uvcvideo: Failed to set UVC probe control : -108 (exp. 48).
[52521.582673] xhci_hcd 0000:01:00.0: Timeout while waiting for setup device command
[52521.802877] usb 2-1: Device not responding to setup address.
[52522.022668] usb 2-1: device not accepting address 70, error -71
[52527.342670] xhci_hcd 0000:01:00.0: Timeout while waiting for setup device command
[52527.572838] usb 2-1: Device not responding to setup address.
[52527.792612] usb 2-1: device not accepting address 71, error -71
[52527.792993] usb usb2-port1: attempt power cycle
[52532.482634] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52536.862602] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52536.862829] usb usb2-port1: unable to enumerate USB device
[52537.762926] usb 2-1: Device not responding to setup address.
[52537.982914] usb 2-1: Device not responding to setup address.
[52538.202619] usb 2-1: device not accepting address 74, error -71
[52542.652938] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52546.992866] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52548.473020] uvcvideo: Failed to set UVC probe control : -110 (exp. 48).
[52549.823444] uvcvideo: Failed to set UVC probe control : -110 (exp. 48).
[52551.312939] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52553.503485] uvcvideo: Failed to set UVC probe control : -110 (exp. 48).
[52554.863384] uvcvideo: Failed to set UVC probe control : -110 (exp. 48).
[52555.842887] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52560.162904] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52560.572600] usb 2-2: USB disconnect, device number 3
[52561.013562] usb 2-2: Device not responding to setup address.
[52561.233526] usb 2-2: Device not responding to setup address.
[52561.452610] usb 2-2: device not accepting address 76, error -71
[52561.723626] usb 2-2: Device not responding to setup address.
[52561.943519] usb 2-2: Device not responding to setup address.
[52562.162598] usb 2-2: device not accepting address 77, error -71
[52562.286914] usb usb2-port2: attempt power cycle
[52566.962670] usb usb2-port2: Cannot enable. Maybe the USB cable is bad?
[52567.663489] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52571.302671] usb usb2-port2: Cannot enable. Maybe the USB cable is bad?
[52571.303858] usb usb2-port2: unable to enumerate USB device
[52572.863603] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52578.063526] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52583.263476] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52588.463405] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52593.663577] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52598.863520] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52604.063340] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52609.263559] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52614.463517] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52619.663380] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52624.863495] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52630.063347] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52635.263345] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52638.783522] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52640.463307] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52643.983390] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52645.663608] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52649.183540] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52650.863344] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52654.383418] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52656.063329] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52659.583372] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52661.263484] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52664.783420] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52666.463597] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52669.983411] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52671.663512] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52675.183478] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52676.863371] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52680.383441] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52682.063615] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52685.583626] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52687.263533] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52690.783558] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52692.463383] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52695.983566] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52697.663378] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52701.183499] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52702.863361] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52706.383585] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52708.063657] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52711.583495] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52713.263659] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52716.783417] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)

Camera must be unplugged and replaced. Camera case should be opened and electronics inspected for corrosion

Electronics and Chassis Short

May be indicated by the following alert:

  • usbcam-left-color: Capture process returned nonzero exit code: (3.000000) Error initializing the camera stream

Follow the Electronics and Chassis Short Troubleshooting Wiki

Reboot

A OS reboot can sometimes resolve kernel driver issues and camera FW. Execute the command below, then wait 2 minutes before attempting connection of VPN / SSH for OS to boot.

$ sudo reboot now