Camera Hardware

PS3Eye Webcamera

Originally the only controller tracking work was done with the PS3 Eye Camera. It has the advantage of being a super cheap 60fps web camera. However, the PS3 Eye Camera isn't a standard windows web camera and thus can't be used out of the box in Windows. PSVRTracker uses WinUSB Drivers + custom code to communicate with the PS3 Eye and get video frame data.

Standard Windows Webcameras

Since there are many other popular and affordable web cameras with higher resolution than the PS3 Eye it would be ideal to support these as well. A standard Windows web camera can be accessed using Windows Media Framework. A WindowsMediaFramework camera wrapper was added to PSVRTracker to support this use case. However you still need a way to filter out useful camera modes since not all resolutions and framerates are good for tracking. Therefore a camera filter JSON file is also needed for each camera that the tracking system wants to support. Some examples can be found in PSVRTracker here.

Stereo Web Cameras, Part 1

Both the PS3 Eye Camera and the Logitech C920 are "Monoscopic" (single lens) web cameras. Another class of web cameras useful for tracking are Stereo Cameras, which have two lenses side-by-side. Unlike monoscopic cameras, a stereo camera can be used to compute a depth map or "disparity map" for every single pixel from a single video frame (I go into more detail about stereo tracking later):

The PlayStation 4 has a stereo web camera that is used to track the PSVR headset, PSMove Controllers and the DualShock 4. While the PS4 Camera is actually just a USB3 camera, it has a non-standard plug that needs hand built adapter if you want to use it on the PC.

The PS4 camera also is not a standard windows web camera so it too needs a custom driver to initialize correctly. A WinUSB driver + some custom code is used to upload some firmware to the PS4 Camera, at which point it turns into a normal WMF supported webcamera.

Since the PS4 camera is a bit of a hassle on the PC I also looked into alternative stereo web cameras. Unfortunately most stereo web cameras are used for industrial or resource purposes and thus are very expensive. There was one relatively inexpensive one I found and tested with called the ELP 920P Stereo Camera. I supports the same resolution and 60fps frame rate as the PS4 Camera.

USB Bandwidth

Often the single biggest complication with optical tracking on the PC is USB bandwidth. Multiple USB ports on a PC share the same USB root hub. A USB2.0 root hub has a maximum bandwidth of 480 Mbit/s (60 MB/s) to share among all of it's connected ports, while a USB3.0 root hub has 5.0 Gbit/s (625 MByte/s). This seems like a lot until you realize that a PS3Eye webcamera running at 640x480@fps using YUV442 video encoding (2bytes per pixel) consumes 36MB/s. So on a USB2 root hub you can't even run two PS3Eye cameras at 60fps.

To make matters more confusing for users it's not really obvious how many USB root hubs they have, which ports are connected to which root hub, or how heavily utilized a given root hub is. There are tools for analyzing/inspecting this sort of thing (Windows device manager, USBlyzer, etc) but they aren't very user friendly. Most camera troubleshooting devolves into having users shuffle around what they have plugged in or just buying a dedicated USB3.0 expansion card.

Video Codecs

To help lessen USB bandwidth burden it's really important to pick the right video codec. It turns out that the PS3 Eye camera also supports a Bayer Filter option that cuts the bandwidth requirement in half (1 byte per pixel) at the cost of lower color resolution, but this seems to be fine for tracking purposes. You can also lower the frame rate of the PS3 Eye camera, but anything below about 40fps tends to not work as well for fast motion due to motion blur of the tracking LED image.

A lot of standard windows web cameras support the MJPG format, which is just JPG encoding each video frame on the fly. This helps tremendously with the USB bandwidth but the savings it can be quite variable since JPG encoding isn't fixed BytePerPixel size like YUYV or Bayer. You can see this manifest on an MJPG stream if you turn the exposure setting up. When the image is mostly dark MJPG can compress it well. As the exposure increases more of the background can be seen which means the JPG encoding can't compress it as well. If there isn't enough USB bandwidth overhead suddenly the video stream will start stuttering.

It's also worth calling out the insane USB bandwidth PS4 Camera uses. The PS4 Camera supports 3 image frame sizes:

320x200, 640x408, and 1280x808

Since it's stereo image (left and right image) you have to double the horizontal pixel width so:

640x200, 1280x408, and 2560x808

Except it turns out that when you use the 640x408 frame size the camera also includes the lower resolution 320x200 frames as well (2 high res frames and 2 low res frames), so the actual video frame width is 1748x408. The low res frames don't have the same image span as the higher res ones so the resulting raw frame looks a bit weird on the right side of the image:

images/StereoCamera/EyeLab.png

Finally if you select 1280x808 frame size you get the 640x408 and 320x200 frames (6 frames total) for a total video frame size of 3448x808. The PS4 camera only supports the YUV442 (2 bytes per pixel) so if you are running at 60fps you are transferring a whopping 334MB/s! Since a USB3 root hub can "only" handle 625 MByte/s max you are only getting 1 PS4 per root hub at that resolution and frame rate. This is another reason why I think using a camera like ELP 920P might be a better bet on the PC depending on the tracking setup you want.

You might also be wondering why the PS4 camera includes the lower resolution frames. I believe this is an optimization for finding tracking LEDs more cheaply. It's expensive to search an entire HD video frame for a LED so if you can find the LED in a lower resolution frame first and get a bounding box, then you can extract the LED contour information in a higher resolution frame.