LCR Realsense Camera Troubleshooting - directedmachines/customer-support GitHub Wiki
Table of Contents
- Overview
- Symptoms
- Root Causes
- SSH
- Persistent "Obstacles Close" Warning / Corrupted Depth Image
- Image frozen or missing
- "Another user is active" Stuck in Manual UI
- Camera Recovery
- Cable unplug/plug
- Reboot
- Electronics and Chassis Short
- Point Cloud Not Showing Structure When Bias Enable
Overview
The LCR uses 2x Realsense D435 / D455 cameras for visual navigation. The cameras are sensitive to environmental and electrical conditions and are often the cause of many faults.
Symptoms
- Camera stream (color or depth) is not updating in Manual UI or AGT
- "Obstacle close" warning stays on even if they are no obstacles
- Image is not displayed in Manual UI (blank screen)
- "Another user is active" warning stays on in Manual UI and AGT / Pose estimation is not working as expected
Please see below for more detailed descriptions and mitigation
Root Causes
- USB Cable to camera and to Raspberry PI is loose or damaged
- USB Hub or port is damaged or not compatible with camera
- Camera connector is corroded or damaged
- Camera internal electronics are corroded or damaged
- Camera frame touches the aluminum mount. Camera must be electrically isolated from frame
Note: A USB level fault will cause other USB peripherals, such as motor controllers and GPS, to also malfunction. If GPS is not reporting, the cause is likely a bad camera.
SSH
The operator is expected to have SSH access to the robot, so they can log in and use a terminal session to issues the commands listed in the following sections
- SSH into the robot using the robot IP address (either IP addressed used when connected through WIFI (usually 192.168.8.100), or static VPN IP)
- issue the command listed before the text output, in the sections below. For example, type: "dmesg" + ENTER to get kernel log output
Useful commands when sshing
- See all image files being created:
ls -al /var/tmp/images
- List all running dm-capture processes:
ps aux | grep dm-capture | grep -v grep
- View image capture log files. Logs are automatically deleted on CAP restart
- Print all logs
cat /var/tmp/images/*.log
- Tail all logs
tail -f /var/tmp/images/*.log
- Print all logs
- list all cameras
- Legacy method:
ls -al /sys/class/video4linux
- Updated dev method:
- By ID (warning not all devices shown):
ls -al /dev/v4l/by-id
- By path:
ls -al /dev/v4l/by-path
- By path, proper video streams only:
ls -al /dev/v4l/by-path | grep index0 | grep usb
- By ID (warning not all devices shown):
- Legacy method:
- List device info about a video device (e.g., video0):
udevadm info /dev/video0
- Get info about a specific stream (
-d4
is short for--device=/dev/video4
) :v4l2-ctl -d4 --list-formats-ext
- RealSense SDK-Based Scripts
- Enumerate rs devices:
~/dCentralizedSystems/cap-provisioning/pi/scripts/firmware/REALSENSE/rs-enumerate-devices
- Reset rs devices:
~/dCentralizedSystems/cap-provisioning/pi/scripts/firmware/REALSENSE/dm-rs-reset
- Upgrade rs firmware:
~/dCentralizedSystems/cap-provisioning/pi/scripts/firmware/REALSENSE/updateFirmwareRealsense.sh
- Enumerate rs devices:
- List if pi camera is detected:
vcgencmd get_camera
Persistent "Obstacles Close" Warning / Corrupted Depth Image
On occasion the camera FW fails to properly calibrate, or, the camera electronics are damaged (see sections below). To determine if the depth image is corrupted, use the depth image display in AGT, on a laptop (requires wide screen to display images), or, use the service input search, and load the casters-depth or front-depth images.
Corrupted depth image due to bad initialization
Shadow artifact on left, due to reflection of solar panels
You can sometimes recover the camera using one of the following steps
- Stand 1m in front of the camera, shielding it from direct sunlight
- Using the UI, select "restart" to restart the runtime, which will re-initialize the sensor streams
- verify depth camera view by going to the front/casters depth image UI
Shadows Causing False Obstacles
The realsense cameras use stero visuals in order to create the depth image. When creating this depth image the cameras project a ground plane. When there is a solid shadow that projects out into the horizon, the cameras can mistake that for a chasm or indentation in the ground. This can then in turn cause the camera to believe that there are obstacles below the projected ground plane. In the images below you can see the solid shadow line projected from the fixed solar panels, and the depth image making an assumption that there is a chasm to avoid.
Image frozen or missing
Camera not enumerated
Manual UI
Cameras are not enumerating
Restore Defaults
- Open the settings (click the cogwheel)
- Search for
sensors/realsense-cameras
- Find the specific realsense camera that you need to enumerate
- Click on the "state" button
- Press
RESTORE_DEFAULTS
. Then press thePatch
button.
- Perform a runtime restart by pressing the power icon in the bottom left, and select
Restart
- Once restarted, use the same search query as above, and click on the white button. (i.e.
sensors/realsense-cameras/casters-color
) - Click
Enable
SSH
- issue the
ls -la /sys/class/video4linux
linux command to enumerate video devices. Expect 12 devices, 6 per realsense camera. Note: you may see additional devices if side USB cameras are installed (usually 2x per camera).
Missing Camera Device List
We use two realsense cameras, each produces 6 video devices, for a total of 12.
In the device listing below, note that only one camera, in the USB device tree usb2/2-1/2-1
, is present.
$ ls -la /sys/class/video4linux/
total 0
drwxr-xr-x 2 root root 0 Jul 25 09:07 .
drwxr-xr-x 65 root root 0 Feb 14 2019 ..
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video0 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video0
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video1 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video1
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video2 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video2
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video3 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video3
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video4 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.3/video4linux/video4
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video5 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.3/video4linux/video5
Camera Enumerated as USB2.0 mode (vs 3.0)
The device listing below has all video devices, for both axle and caster camera, but, one camera is enumerated as USB2.0.
Properly enumerated cameras will be USB3.0 devices which appear under the 'usb2' sub path. If they enumerate in reduced function mode they enumerate as USB2.0 under the 'usb1' subpath - e.g., /usb1/1-1/1-1.2/1-1.2:1.3/video4linux
$ ls -la /sys/class/video4linux/
total 0
drwxr-xr-x 2 root root 0 Jul 25 09:07 .
drwxr-xr-x 65 root root 0 Feb 14 2019 ..
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video0 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video0
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video1 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video1
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video10 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.3/video4linux/video10
lrwxrwxrwx 1 root root 0 Jul 25 09:21 video11 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.3/video4linux/video11
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video2 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video2
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video3 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/video4linux/video3
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video4 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.3/video4linux/video4
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video5 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.3/video4linux/video5
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video6 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.0/video4linux/video6
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video7 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.0/video4linux/video7
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video8 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.0/video4linux/video8
lrwxrwxrwx 1 root root 0 Jul 25 09:07 video9 -> ../../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.0/video4linux/video9
Contact a software team member if the cameras in the list above have the usb1/1-1/1-1.xx prefix and are not streaming images
Video Completion handler error
It appears the kernel USBVideo stack gets in a bad state and while the cameras are enumerated, frames can not be acquired. The kernel (dmesg) would have lots of these messages
[249806.659207] uvcvideo: Non-zero status (-75) in video completion handler.
[249806.659511] uvcvideo: Non-zero status (-75) in video completion handler.
[249806.659824] uvcvideo: Non-zero status (-75) in video completion handler.
[249806.660131] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.776957] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.777418] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.777791] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.778165] uvcvideo: Non-zero status (-75) in video completion handler.
[249813.778540] uvcvideo: Non-zero status (-75) in video completion handler.
Please contact member of software team, then reboot, to recover the camera streams
Bad Cable
Using a laptop,
- SSH into the robot using the robot IP address (either IP addressed used when connected through WIFI (usually 192.168.50.10), or static VPN IP)
- issue the
dmesg
linux command - look for red USB related error messages
Example of camera USB level errors due to either bad cable or damaged camera connector
$ dmesg
...
[351788.919646] usb usb2-port1: attempt power cycle
[351793.609349] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351797.959404] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351797.959770] usb usb2-port1: unable to enumerate USB device
[351802.279471] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351806.819497] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351811.179569] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351811.179935] usb usb2-port1: attempt power cycle
[351815.859646] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351820.199712] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351820.200079] usb usb2-port1: unable to enumerate USB device
[351824.519759] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351829.059798] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[351833.399844] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
...
Damaged camera (corroded electronics)
Another symptom of a faulty camera, manifests as UVC video kernel error messages
$ dmesg
[ 2698.590869] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2698.591144] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2705.820078] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2705.820357] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2712.788413] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2712.788699] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2719.616349] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[ 2719.616633] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
Camera must be unplugged and replaced. Camera case should be opened and electronics inspected for corrosion
"Another user is active" Stuck in Manual UI
A mis behaving camera can cause the entire runtime to issue faults and prevent motion. A common sign is that other USB peripherals will stop working. The kernel logs will have warnings similar to below, notice that they start with a USB device 2-1 or 2-2 disconnect, which is the USB path prefix of the cameras
$dmesg
[52502.095129] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) 430 (8086:0b07)
[52505.324859] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[52505.325136] uvcvideo: Failed to set UVC probe control : -32 (exp. 48).
[52516.052617] usb 2-1: USB disconnect, device number 69
[52516.189848] uvcvideo: Failed to set UVC probe control : -108 (exp. 48).
[52521.582673] xhci_hcd 0000:01:00.0: Timeout while waiting for setup device command
[52521.802877] usb 2-1: Device not responding to setup address.
[52522.022668] usb 2-1: device not accepting address 70, error -71
[52527.342670] xhci_hcd 0000:01:00.0: Timeout while waiting for setup device command
[52527.572838] usb 2-1: Device not responding to setup address.
[52527.792612] usb 2-1: device not accepting address 71, error -71
[52527.792993] usb usb2-port1: attempt power cycle
[52532.482634] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52536.862602] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52536.862829] usb usb2-port1: unable to enumerate USB device
[52537.762926] usb 2-1: Device not responding to setup address.
[52537.982914] usb 2-1: Device not responding to setup address.
[52538.202619] usb 2-1: device not accepting address 74, error -71
[52542.652938] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52546.992866] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52548.473020] uvcvideo: Failed to set UVC probe control : -110 (exp. 48).
[52549.823444] uvcvideo: Failed to set UVC probe control : -110 (exp. 48).
[52551.312939] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52553.503485] uvcvideo: Failed to set UVC probe control : -110 (exp. 48).
[52554.863384] uvcvideo: Failed to set UVC probe control : -110 (exp. 48).
[52555.842887] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52560.162904] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[52560.572600] usb 2-2: USB disconnect, device number 3
[52561.013562] usb 2-2: Device not responding to setup address.
[52561.233526] usb 2-2: Device not responding to setup address.
[52561.452610] usb 2-2: device not accepting address 76, error -71
[52561.723626] usb 2-2: Device not responding to setup address.
[52561.943519] usb 2-2: Device not responding to setup address.
[52562.162598] usb 2-2: device not accepting address 77, error -71
[52562.286914] usb usb2-port2: attempt power cycle
[52566.962670] usb usb2-port2: Cannot enable. Maybe the USB cable is bad?
[52567.663489] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52571.302671] usb usb2-port2: Cannot enable. Maybe the USB cable is bad?
[52571.303858] usb usb2-port2: unable to enumerate USB device
[52572.863603] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52578.063526] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52583.263476] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52588.463405] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52593.663577] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52598.863520] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52604.063340] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52609.263559] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52614.463517] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52619.663380] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52624.863495] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52630.063347] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52635.263345] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52638.783522] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52640.463307] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52643.983390] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52645.663608] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52649.183540] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52650.863344] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52654.383418] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52656.063329] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52659.583372] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52661.263484] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52664.783420] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52666.463597] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52669.983411] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52671.663512] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52675.183478] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52676.863371] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52680.383441] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52682.063615] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52685.583626] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52687.263533] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52690.783558] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52692.463383] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52695.983566] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52697.663378] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52701.183499] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52702.863361] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52706.383585] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52708.063657] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52711.583495] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
[52713.263659] hub 1-1:1.0: hub_ext_port_status failed (err = -110)
[52716.783417] hub 1-1.4:1.0: hub_ext_port_status failed (err = -110)
Camera must be unplugged and replaced. Camera case should be opened and electronics inspected for corrosion
Frozen Images (sub case)
In some cases we have seen the cameras enumerating on startup loading the color/depth for the axle/caster sides then not reporting anything afterwards. In this case please make sure the cables, USB Hubs, and ports are up to the most current versions and topology. Test the camera using the realsense-viewer to makes sure the firmware is up to date and that the cameras work. If the cause cannot be determined you may need to replace the camera.
Relevant kernel logs:
$ dmesg
[432426.612732] usb 2-1: Device not responding to setup address.
[432426.832470] usb 2-1: device not accepting address 19, error -71
[432426.840712] usb usb2-port1: unable to enumerate USB device"
A camera unplug/plug can recover the cameras, if they are not damaged.
Camera Recovery
The FW faults, badly seated cables, or sometimes kernel driver issues can be resolved in one of two ways:
- reboot
- unplug/plug camera
Cable unplug/plug
Sometimes cable is not fully inserted into camera or Raspberry PI. Disconnect cable from both ends, plug back in, then using same linux kernel log command, you might see camera enumeration logs
$ dmesg
[19718.233791] usb 2-2: new SuperSpeed Gen 1 USB device number 9 using xhci_hcd
[19718.264707] usb 2-2: New USB device found, idVendor=8086, idProduct=0b5c, bcdDevice=50.d0
[19718.264731] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[19718.264750] usb 2-2: Product: Intel(R) RealSense(TM) Depth Camera 455
[19718.264767] usb 2-2: Manufacturer: Intel(R) RealSense(TM) Depth Camera 455
[19718.264785] usb 2-2: SerialNumber: 210523063619
[19718.269271] uvcvideo: Unknown video format 00000050-0000-0010-8000-00aa00389b71
[19718.269462] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 455 (8086:0b5c)
[19718.277120] input: Intel(R) RealSense(TM) Depth Ca as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-2/2-2:1.0/input/input5
[19718.278637] uvcvideo: Unknown video format 36315752-1a66-a242-9065-d01814a8ef8a
[19718.278674] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 455 (8086:0b5c)
[19718.290921] hid-generic 0003:8086:0B5C.0006: hiddev96,hidraw0: USB HID v1.10 Device [Intel(R) RealSense(TM) Depth Camera 455 Intel(R) RealSense(TM) Depth Camera 455 ] on usb-0000:01:00.0-2/input5
Reboot
A OS reboot can sometimes resolve kernel driver issues and camera FW. Execute the command below, then wait 2 minutes before attempting connection of VPN / SSH for OS to boot.
$ sudo reboot now
Electronics and Chassis Short
Sometimes causes the cameras to fail to enumerate. Follow the Electronics and Chassis Short Troubleshooting Wiki
Point Cloud Not Showing Structure When Bias Enable
- After going through the preflight check list and not seeing structures on the right like shown in the picture below.
-
To resolve this issue, search for the front depth camera in the UI and click on "sensors/realsense-cameras/front-depth".
-
Click on "Configure Sensor"
*The correct configuration for the realsense camera is the "region left" is 0.02 and the "region width" is 0.98. The top and height values are automatic.