Test Plan

The test plan can be found here.

Summary

Overall, most of the test resulted in the expected outcomes, validating our work and data. A few tests identified issues in accuracy that were mitigated as well as small bugs including off by one errors. These were fixed and retested until they passed the tests.

Test Results

Traffic Sign Module Test 1

A series of test images were fed to the CNN model. The model then returned an integer corresponding to the road sign it thought was detected. This integer was compared to the true value of the image. All test images were categorized correctly.

Traffic Sign Module Test 2

A series of test images were fed to the K-NN model. The model then returned an integer corresponding to the road sign it thought was detected. This integer was compared to the true value of the image. Over half of the images were categorized correctly, and all images were the correct type of sign (speed/stop/arrow). This was considered acceptable as this was an unsupervised model and thus less accurate.

Traffic Sign Module Test 3

A series of test images were fed to the GAN model. The model then returned an integer corresponding to the road sign it thought was detected. This integer was compared to the true value of the image. The images were found to be incorrectly categorized, but this was fixed when an off-by-one error was detected for the outputs.

Object Detection Module Test 1

A zip file containing the frames of a scene were fed to the CNN model. The model then returned an integer value corresponding to the action of the car it was tracking. The outputs were found to be correct on all four scene types [closer, further, stationary, anomaly].

Object Detection Module Test 2

A zip file containing the frames of a scene were fed to the HOG model. The model then returned an integer value corresponding to the action of the car it was tracking. The outputs were found to be correct on all four scene types [closer, further, stationary, anomaly].

Object Detection Module Test 3

A zip file containing the frames of a scene were fed to both the HOG and CNN model. The outputs of both models were compared against each other and the true results. These outputs were found to be correct for each scene type [closer, further, stationary, anomaly, scene change].

GPS Module Test 1

A series of polar coordinates and a known coordinate were fed into the Random Forest model. This data was known to be correct, or non-anomalous. The results of the model were checked to see if they also identified the coordinate set as non-anomalous, which they did.

GPS Module Test 2

A series of polar coordinates and a known coordinate were fed into the Random Forest model. This data was known to be incorrect, or anomalous. The results of the model were checked to see if they also identified the coordinate set as anomalous, which they did.

Module Failure Output Controller Test 1

The full demo was run with only the GPS and Object Detection models available to be loaded. The program was able to proceed without the Traffic Sign models and only supply outputs for the two modules provided, as intended.

Module Failure Output Controller Test 2

The full demo was run with only the GPS and Traffic Sign models available to be loaded. The program was able to proceed without the Object Detection models and only supply outputs for the two modules provided, as intended.

Module Failure Output Controller Test 3

The full demo was run with only the Traffic Sign and Object Detection models available to be loaded. The program was able to proceed without the GPS models and only supply outputs for the two modules provided, as intended.

Combined Output Controller Test 1

The models and modules were run individually, with their output compared to those of the full demo. The data fed into each model should return true, as there are no anomalies in the data. The results of both tests were the same, validating the demo.

Combined Output Controller Test 2

The models and modules were run individually, with their output compared to those of the full demo. The data fed into each model should return false, as there are detectable anomalies in the data. The results of both tests were the same, validating the demo.

Combined Output Controller Test 3

The models and modules were run individually, with their output compared to those of the full demo. The data fed into two of the models was true while one of the models was false, which should return one anomaly detected and two cleared models. The results were found to be as expected, validating the demo.

Combined Output Controller Test 4

The models and modules were run individually, with their output compared to those of the full demo. The data fed into one of the models was true while two of the models was false, which should return two anomaly detected and one cleared models. This result will throw a full warning to the system. The results were found to be as expected, validating the demo.