LCR Realsense Camera Troubleshooting - directedmachines/customer-support GitHub Wiki

Table of Contents

Overview

The LCR uses 2x Realsense D435 / D455 cameras for visual navigation. The cameras are sensitive to environmental and electrical conditions and are often the cause of many faults.

Symptoms

  • Camera stream (color or depth) is not updating in Manual UI or AGT
  • "Obstacle close" warning stays on even if they are no obstacles
  • Image is not displayed in Manual UI (blank screen)
  • "Another user is active" warning stays on in Manual UI and AGT / Pose estimation is not working as expected

Please see below for more detailed descriptions and mitigation

Root Causes

  • USB Cable to camera and to Raspberry PI is loose or damaged
  • USB Hub or port is damaged or not compatible with camera
  • Camera connector is corroded or damaged
  • Camera internal electronics are corroded or damaged
  • Camera frame touches the aluminum mount. Camera must be electrically isolated from frame

Note: A USB level fault will cause other USB peripherals, such as motor controllers and GPS, to also malfunction. If GPS is not reporting, the cause is likely a bad camera.

SSH

The operator is expected to have SSH access to the robot, so they can log in and use a terminal session to issues the commands listed in the following sections

  • SSH into the robot using the robot IP address (either IP addressed used when connected through WIFI (usually 192.168.8.100), or static VPN IP)
  • issue the command listed before the text output, in the sections below. For example, type: "dmesg" + ENTER to get kernel log output

Useful commands when sshing

  1. See all image files being created: ls -al /var/tmp/images
  2. List all running dm-capture processes: ps aux | grep dm-capture | grep -v grep
  3. View image capture log files. Logs are automatically deleted on CAP restart
    1. Print all logs cat /var/tmp/images/*.log
    2. Tail all logs tail -f /var/tmp/images/*.log
  4. list all cameras
    1. Legacy method: ls -al /sys/class/video4linux
    2. Updated dev method:
      1. By ID (warning not all devices shown): ls -al /dev/v4l/by-id
      2. By path: ls -al /dev/v4l/by-path
      3. By path, proper video streams only: ls -al /dev/v4l/by-path | grep index0 | grep usb
  5. List device info about a video device (e.g., video0): udevadm info /dev/video0
  6. Get info about a specific stream (-d4 is short for --device=/dev/video4) : v4l2-ctl -d4 --list-formats-ext
  7. RealSense SDK-Based Scripts
    1. Enumerate rs devices: ~/dCentralizedSystems/cap-provisioning/pi/scripts/firmware/REALSENSE/rs-enumerate-devices
    2. Reset rs devices: ~/dCentralizedSystems/cap-provisioning/pi/scripts/firmware/REALSENSE/dm-rs-reset
    3. Upgrade rs firmware: ~/dCentralizedSystems/cap-provisioning/pi/scripts/firmware/REALSENSE/updateFirmwareRealsense.sh
  8. List if pi camera is detected: vcgencmd get_camera

Persistent "Obstacles Close" Warning / Corrupted Depth Image

On occasion the camera FW fails to properly calibrate, or, the camera electronics are damaged (see sections below). To determine if the depth image is corrupted, use the depth image display in AGT, on a laptop (requires wide screen to display images), or, use the service input search, and load the casters-depth or front-depth images.

Corrupted depth image due to bad initialization

Shadow artifact on left, due to reflection of solar panels

You can sometimes recover the camera using one of the following steps

  • Stand 1m in front of the camera, shielding it from direct sunlight
  • Using the UI, select "restart" to restart the runtime, which will re-initialize the sensor streams
  • verify depth camera view by going to the front/casters depth image UI

Shadows Causing False Obstacles

The realsense cameras use stero visuals in order to create the depth image. When creating this depth image the cameras project a ground plane. When there is a solid shadow that projects out into the horizon, the cameras can mistake that for a chasm or indentation in the ground. This can then in turn cause the camera to believe that there are obstacles below the projected ground plane. In the images below you can see the solid shadow line projected from the fixed solar panels, and the depth image making an assumption that there is a chasm to avoid.

image

image

Image frozen or missing

Camera not enumerated

Manual UI

Cameras are not enumerating

Restore Defaults

  • Open the settings (click the cogwheel)
  • Search for sensors/realsense-cameras
  • Find the specific realsense camera that you need to enumerate

image

  • Click on the "state" button
  • Press RESTORE_DEFAULTS. Then press the Patch button.

image

  • Perform a runtime restart by pressing the power icon in the bottom left, and select Restart
  • Once restarted, use the same search query as above, and click on the white button. (i.e. sensors/realsense-cameras/casters-color)
  • Click Enable

SSH

  • issue the ls -la /sys/class/video4linux linux command to enumerate video devices. Expect 12 devices, 6 per realsense camera. Note: you may see additional devices if side USB cameras are installed (usually 2x per camera).

Missing Camera Device List

We use two realsense cameras, each produces 6 video devices, for a total of 12. In the device listing below, note that only one camera, in the USB device tree usb2/2-1/2-1, is present.

$ ls -la /sys/class/video4linux/
total 0
drwxr-xr-x  2 root root 0 Jul 25 09:07 .
drwxr-xr-x 65 root root 0 Feb 14  2019 ..
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video0 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video0
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video1 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video1
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video2 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video2
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video3 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video3
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video4 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.3/video4linux/video4
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video5 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.3/video4linux/video5

Camera Enumerated as USB2.0 mode (vs 3.0)

The device listing below has all video devices, for both axle and caster camera, but, one camera is enumerated as USB2.0. Properly enumerated cameras will be USB3.0 devices which appear under the 'usb2' sub path. If they enumerate in reduced function mode they enumerate as USB2.0 under the 'usb1' subpath - e.g., /usb1/1-1/1-1.2/1-1.2:1.3/video4linux

$ ls -la /sys/class/video4linux/
total 0
drwxr-xr-x  2 root root 0 Jul 25 09:07 .
drwxr-xr-x 65 root root 0 Feb 14  2019 ..
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video0 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video0
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video1 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video1
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video10 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.3/video4linux/video10
lrwxrwxrwx  1 root root 0 Jul 25 09:21 video11 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.3/video4linux/video11
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video2 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video2
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video3 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video3
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video4 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.3/video4linux/video4
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video5 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.3/video4linux/video5
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video6 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.0/video4linux/video6
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video7 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.0/video4linux/video7
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video8 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.0/video4linux/video8
lrwxrwxrwx  1 root root 0 Jul 25 09:07 video9 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.0/video4linux/video9

Contact a software team member if the cameras in the list above have the usb1/1-1/1-1.xx prefix and are not streaming images

Video Completion handler error

It appears the kernel USBVideo stack gets in a bad state and while the cameras are enumerated, frames can not be acquired. The kernel (dmesg) would have lots of these messages

[249806.659207] uvcvideo: Non-zero status (-75) in video completion handler.
[249806.659511] uvcvideo: Non-zero status (-75) in video completion handler.
[249806.659824] uvcvideo: Non-zero status (-75) in video completion handler.
[249806.660131] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.776957] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.777418] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.777791] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.778165] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.778540] uvcvideo: Non-zero status (-75) in video completion handler.

Please contact member of software team, then reboot, to recover the camera streams

Bad Cable

Using a laptop,

  • SSH into the robot using the robot IP address (either IP addressed used when connected through WIFI (usually 192.168.50.10), or static VPN IP)
  • issue the dmesg linux command
  • look for red USB related error messages

Example of camera USB level errors due to either bad cable or damaged camera connector

$ dmesg

...
[351788.919646] usb usb2-port1: attempt power cycle
[351793.609349] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351797.959404] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351797.959770] usb usb2-port1: unable to enumerate USB device
[351802.279471] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351806.819497] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351811.179569] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351811.179935] usb usb2-port1: attempt power cycle
[351815.859646] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351820.199712] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351820.200079] usb usb2-port1: unable to enumerate USB device
[351824.519759] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351829.059798] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351833.399844] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
...

Damaged camera (corroded electronics)

Another symptom of a faulty camera, manifests as UVC video kernel error messages

$ dmesg
[ 2698.590869] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2698.591144] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2705.820078] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2705.820357] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2712.788413] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2712.788699] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2719.616349] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2719.616633] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).

Camera must be unplugged and replaced. Camera case should be opened and electronics inspected for corrosion

"Another user is active" Stuck in Manual UI

A mis behaving camera can cause the entire runtime to issue faults and prevent motion. A common sign is that other USB peripherals will stop working. The kernel logs will have warnings similar to below, notice that they start with a USB device 2-1 or 2-2 disconnect, which is the USB path prefix of the cameras

$dmesg
[52502.095129] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) 430 (8086:0b07)
[52505.324859] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[52505.325136] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[52516.052617] usb 2-1: USB disconnect, device number 69
[52516.189848] uvcvideo: Failed to set UVC probe control : -108 (exp. 48).
[52521.582673] xhci_hcd 0000:01:00.0: Timeout while waiting for setup device command
[52521.802877] usb 2-1: Device not responding to setup address.
[52522.022668] usb 2-1: device not accepting address 70, error -71
[52527.342670] xhci_hcd 0000:01:00.0: Timeout while waiting for setup device command
[52527.572838] usb 2-1: Device not responding to setup address.
[52527.792612] usb 2-1: device not accepting address 71, error -71
[52527.792993] usb usb2-port1: attempt power cycle
[52532.482634] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52536.862602] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52536.862829] usb usb2-port1: unable to enumerate USB device
[52537.762926] usb 2-1: Device not responding to setup address.
[52537.982914] usb 2-1: Device not responding to setup address.
[52538.202619] usb 2-1: device not accepting address 74, error -71
[52542.652938] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52546.992866] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52548.473020] uvcvideo: Failed to set UVC probe control : -110 (exp. 48).
[52549.823444] uvcvideo: Failed to set UVC probe control : -110 (exp. 48).
[52551.312939] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52553.503485] uvcvideo: Failed to set UVC probe control : -110 (exp. 48).
[52554.863384] uvcvideo: Failed to set UVC probe control : -110 (exp. 48).
[52555.842887] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52560.162904] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52560.572600] usb 2-2: USB disconnect, device number 3
[52561.013562] usb 2-2: Device not responding to setup address.
[52561.233526] usb 2-2: Device not responding to setup address.
[52561.452610] usb 2-2: device not accepting address 76, error -71
[52561.723626] usb 2-2: Device not responding to setup address.
[52561.943519] usb 2-2: Device not responding to setup address.
[52562.162598] usb 2-2: device not accepting address 77, error -71
[52562.286914] usb usb2-port2: attempt power cycle
[52566.962670] usb usb2-port2: Cannot enable. Maybe the USB cable is bad?
[52567.663489] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52571.302671] usb usb2-port2: Cannot enable. Maybe the USB cable is bad?
[52571.303858] usb usb2-port2: unable to enumerate USB device
[52572.863603] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52578.063526] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52583.263476] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52588.463405] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52593.663577] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52598.863520] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52604.063340] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52609.263559] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52614.463517] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52619.663380] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52624.863495] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52630.063347] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52635.263345] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52638.783522] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52640.463307] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52643.983390] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52645.663608] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52649.183540] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52650.863344] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52654.383418] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52656.063329] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52659.583372] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52661.263484] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52664.783420] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52666.463597] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52669.983411] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52671.663512] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52675.183478] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52676.863371] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52680.383441] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52682.063615] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52685.583626] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52687.263533] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52690.783558] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52692.463383] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52695.983566] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52697.663378] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52701.183499] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52702.863361] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52706.383585] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52708.063657] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52711.583495] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52713.263659] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52716.783417] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)

Camera must be unplugged and replaced. Camera case should be opened and electronics inspected for corrosion

Frozen Images (sub case)

In some cases we have seen the cameras enumerating on startup loading the color/depth for the axle/caster sides then not reporting anything afterwards. In this case please make sure the cables, USB Hubs, and ports are up to the most current versions and topology. Test the camera using the realsense-viewer to makes sure the firmware is up to date and that the cameras work. If the cause cannot be determined you may need to replace the camera.

Relevant kernel logs:

$ dmesg
[432426.612732] usb 2-1: Device not responding to setup address.
[432426.832470] usb 2-1: device not accepting address 19, error -71
[432426.840712] usb usb2-port1: unable to enumerate USB device"

A camera unplug/plug can recover the cameras, if they are not damaged.

Camera Recovery

The FW faults, badly seated cables, or sometimes kernel driver issues can be resolved in one of two ways:

  1. reboot
  2. unplug/plug camera

Cable unplug/plug

Sometimes cable is not fully inserted into camera or Raspberry PI. Disconnect cable from both ends, plug back in, then using same linux kernel log command, you might see camera enumeration logs

$ dmesg
[19718.233791] usb 2-2: new SuperSpeed Gen 1 USB device number 9 using xhci_hcd
[19718.264707] usb 2-2: New USB device found, idVendor=8086, idProduct=0b5c, bcdDevice=50.d0
[19718.264731] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[19718.264750] usb 2-2: Product: Intel(R) RealSense(TM) Depth Camera 455 
[19718.264767] usb 2-2: Manufacturer: Intel(R) RealSense(TM) Depth Camera 455 
[19718.264785] usb 2-2: SerialNumber: 210523063619
[19718.269271] uvcvideo: Unknown video format 00000050-0000-0010-8000-00aa00389b71
[19718.269462] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 455  (8086:0b5c)
[19718.277120] input: Intel(R) RealSense(TM) Depth Ca as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-2/2-2:1.0/input/input5
[19718.278637] uvcvideo: Unknown video format 36315752-1a66-a242-9065-d01814a8ef8a
[19718.278674] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 455  (8086:0b5c)
[19718.290921] hid-generic 0003:8086:0B5C.0006: hiddev96,hidraw0: USB HID v1.10 Device [Intel(R) RealSense(TM) Depth Camera 455  Intel(R) RealSense(TM) Depth Camera 455 ] on usb-0000:01:00.0-2/input5

Reboot

A OS reboot can sometimes resolve kernel driver issues and camera FW. Execute the command below, then wait 2 minutes before attempting connection of VPN / SSH for OS to boot.

$ sudo reboot now

Electronics and Chassis Short

Sometimes causes the cameras to fail to enumerate. Follow the Electronics and Chassis Short Troubleshooting Wiki

Point Cloud Not Showing Structure When Bias Enable

  • After going through the preflight check list and not seeing structures on the right like shown in the picture below.

screenshot_2023-08-27_at_1 55 26_pm_720

  • To resolve this issue, search for the front depth camera in the UI and click on "sensors/realsense-cameras/front-depth". screenshot_2023-08-27_at_1 55 37_pm_720

  • Click on "Configure Sensor" screenshot_2023-08-27_at_1 55 51_pm_720

    *The correct configuration for the realsense camera is the "region left" is 0.02 and the "region width" is 0.98. The top and height values are automatic.