Home - PARC-projects/video-query-home GitHub Wiki
Welcome to the video-query-home wiki!
The Video Query app enables agile, user-in-the-loop mining of video datasets. Rather than requiring an upfront specification of actions and investment in labeling data for those actions, Video Query embraces early exploration by a user of evolving questions and ideas. The effort spent increases only as the user iterates to, focuses down on, and validates a particular line of inquiry. This app enables agile browsing through a large dataset, in order to discover and zoom in on relevant data in a dynamic and interactive way.
How it works: A user of the Video Query app begins by providing an example of a video clip of interest. For instance, the example clip could be a forward-looking dash cam of a vehicle stopping for pedestrians. The user receives back a small sample of possible matches, and after inspecting these candidates, the user validates or invalidates them and requests a refined search. The refined search returns a new sample of candidates that reflects the scoring in the previous round. After enough iterations are conducted to satisfy the user, the final video search criterion is used to retrieve all matches in the dataset of interest. Following up on the above instance, these matches could be all video clips where a vehicle stops for pedestrians at a marked crosswalk, or even more specifically at a marked crosswalk in clear weather.
The collaborative search approach is more agile than conventional supervised learning of video actions. It does not require an upfront specification of the actions of interest, nor a large upfront investment in labeling data. The collaborative search can also help a user discern what aspects of the video clips are important to match, since the user can experiment while giving feedback.
Software architecture: The front end is an Angular 6.x web application. It calls the RESTful Video Query API, which uses Video Query Algorithms to perform deep learning calculations. The API can also be used independently of the front end.
Getting started
Clone the following three Video Query repositories on appropriate servers:
Then follow the instructions below for setting up and running each server: