PCL Feature Representation - AmyPhung/ml_person_detection GitHub Wiki

Previous: PCL Segmentation

Cluster Features

In order to train a classifier to distinguish between point clusters, the team chose to define a set of features to extract from each cluster and feed to the classifier. This approach to using lidar data with machine learning techniques is fairly standard, but the feature sets used by different classifier implementations differ greatly. Below are the feature sets explored by the team.

Feature Set 1: Volume, Density, Eigenvalues

The first feature set that the team chose to use came from a 2018 paper titled "Classifying 3D objects in LiDAR point clouds with a back‑propagation neural network". The researchers in the paper defined their feature set as the following:

  • Cluster bounding box volume
  • Custer density (points per bounding box volume)
  • Eigenvalues of three principal component vectors

Volume, in theory, is useful in distinguishing large objects from small objects. However, it becomes misleading when a section of an object is scanned. Additionally, scanning a car from the side vs from a forward angle provides point clouds with significantly different volumes, as shown in Fig. 1.

[191212_ml_car-scan-side.png]] ](/AmyPhung/ml_person_detection/wiki/[[191212_ml_car-scan-front.png) :-------------------------------:|:--------------------------------: Figure 1a: Car scanned from side | Figure 1b: Car scanned from front

Density is helpful in solving the first problem with volume, namely distinguishing between well-scanned objects and poorly scanned objects. This metric is robust enough that the team has explored using it as a filtering step, removing objects below a certain density as inclassifiable. As seen in Fig. 2a, vehicles that are further away from the scanner are much less dense and, therefore, difficult to make out.

[191212_ml_car-scan-sparse.png]] ](/AmyPhung/ml_person_detection/wiki/[[191212_ml_car-scan-dense.png) :---------------------------------:|:--------------------------------: Figure 2a: Sparse car at distance | Figure 2b: Dense car up close

Thresholding valid objects by density could run into issues with large objects, however. Figure 3 shows density-distance curves plotted for each of the three major classes. Signs have a very high density that drops off exponentially with distance, and pedestrians have a reasonable density drop-off. However, cars have relatively low densities across the board (with several outliers). This is likely due to having large 3-d volumes that, depending on how they are scanned, may be minimally covered, as can be seen in Figure 1b.

[191212_ml_density-xydist-car.png]] ](/AmyPhung/ml_person_detection/wiki/[[191212_ml_density-xydist-person.png) | 191212_ml_density-xydist-sign.png :---------------------------------:|:--------------------------------:|:-------------------- Figure 3a: Density vs. distance - car | Figure 3b: Density vs. distance - person | Figure 3c: Density vs distance - signs

Eigenvalues are another feature that shows promise at categorizing objects with different aspect ratios. The previously cited paper uses the magnitudes of the eigenvectors obtained by decomposing the covariance matrix of a cluster to capture the directions and ratios of spatial variance. This approach shows promise with the waymo dataset, as the classes being differentiated have significantly different overall shapes. Car scans have different aspect ratios than pedestrian scans (Fig 4).

[191212_ml_car-scan-eigen.png]] ](/AmyPhung/ml_person_detection/wiki/[[191212_ml_person-scan-eigen.png) :------------------------------------:|:--------------------------------: Figure 4a: Car proportions from above | Figure 4b: Person proportions from above

Eigenvalues have drawbacks, however. They do not contain coordinate system-related information, and therefore cannot be used to determine if something is tall and thin vs long and thin. Additionally, because scans only capture part of an object, the actual ratios of the scan may be significantly different than those of the object.

Feature Set 2: Intensities, Volume, Point-level Features

The second feature set explored by the team came from a paper written by a Velodyne research group titled "LIDAR-based 3D Object Perception". This team defined a different set of features, and were able to train a successful SVM on their problem. Their chosen features are as follows:

  • Volume
  • Intensity maximum, mean, & variance
  • Point-level features
    • Lalonde Features
    • Anguelov Features

Volume, as defined by the Velodyne team, is identical to the volume used in the first feature set.

Intensity is the actual return intensity of the laser light to the sensor as it reflects off of an object. This information is highly useful because it tells you something about the material of the object. The waymo dataset stands to gain significantly from this feature - signs should be very intense due to their purpose of being highly visible and reflective. Cars should be relatively intense, much more than people, who should have minimal intensity of reflection. The graphs in Figure 5 reflect this hypothesis to a certain extent - signs have a much wider variance in mean intensity than vehicles, which have a much wider variance than people. However, there is still a large concentration of mean variances that could be any of the three classes, meaning that this metric needs other supporting metrics to be meaningful.

[191212_ml_mean-intensity-car-sign.png]] ](/AmyPhung/ml_person_detection/wiki/[[191212_ml_mean-intensity-person-car.png) :------------------------------------:|:--------------------------------: Figure 5a: Compare car and sign mean intensity distributions | Figure 5b: Compare people and car mean intensity distributions

Point-level Features have not yet been implemented by this team on this dataset. The theory behind the specific features mentioned (pulled from prior art and named after the papers that defined them) is to capture a histogram representation of a specific metric as calculated at all points in a pointcloud. Colloquially speaking, this operation serves to capture "scatter-ness, flatt-ness, and linear-ness", as well as create a more specifically defined measurement of distribution in various locations throughout the pointcloud. This analysis certainly warrants more consideration, as comparing point density at points along the z-axis should easily distinguish street signs from people. However, this analysis is currently relegated to future art discussions.

Next: Data Visualization