Web Client Example - PARC-projects/video-query-home GitHub Wiki

Here is an example of how to use the Video Query web application. It uses the DowntownBrooklynDrive_480p mp4 video in the Client video assets folder. This video is a 30 frames per second video.

1. Preliminary Steps
2. Example

Webinar for using the browser application (complements Example)
Webinar for setting up the software servers (complements Preliminary Steps)

• If you have not already reviewed the Wiki pages for Video Query Algorithms, doing so now will make it easier to understand the following steps.
• On an appropriate server enabled with GPU computing, put the sample video in a folder in the main TSN directory. In this example, we name the folder "stock-video/".
• Run build_wof_clips.py from the main TSN directory:

python tools/build_wof_clips.py stock-video/ stock-video-clips/ --num_worker 1 --new_width 340 --new_height 256 --fps 30 2>local/errors_stock_video.log 

  For help, run `python tools/build_wof_clips.py --help`

• Create features for the clips by running

bash scripts/calcSig_wOF_ensemble.sh stock-video-clips/ stock-video-clips_features/ 32

  32 is the number of CPU processes.  Reduce if your server has fewer CPU cores.

• Move the "stock-video-clips_features/" directory, with all contents, to "video-query-algorithms/data/features/".

• Refer to the API Wiki pages for setting up and running the API server.

• From "video-query-algorithms/src/" on the algorithms server, run

python load_db.py ../data/features/stock-video-clips_features/

The features and video clips for the sample video should now be in the database, as well as a record for the sample video.

python broker.py

• Refer to the Client Wiki pages for setting up and running the client web server.

• In a browser, go to the URL for the web application, e.g. http://localhost:4200/ for a development server, and log in.

• Click on "Search Sets" in the navigation bar at the top of the window.
• Then click on "Add New" near the top right corner.
• Choose the "DowntownBrooklynDrive_480p" video in Search For Videos and click on "Add"
• Give a name to the Search Set and click on Save
• In general, search sets can have multiple videos, but we are including only one to keep this example simple.

• Return to the browser tab for the web application.
• Select "New Query"
• Fill in the form as shown below. In this example, our intention is to find clips where the vehicle has stopped and has pedestrians walking in front. The development of this application was funded in part by the US Dept. of Transportation Federal Highway Administration, for whom safety when vehicles and pedestrians interact is an important part of their overall safety mission.

• Note that the video goes to the reference time once it is entered. A user can either enter different times or scroll through the video to investigate different possible reference clips. Remember to enter the time of the clip you choose, even if you have already scrolled to the clip.
• Target bootstrapping is still in the experimental stage. If you click to enable this feature, then the target is modified during each query revision to be an "average" of sorts of all validated matching clips. For this example, we are leaving this option unclicked.
• Once the New Query form is complete, click "Submit"

• Click on "Existing" in the top navigation bar, or click on "Existing Queries" at the "Home" page, to go to the list of existing queries. Find the name "sample query" as entered in the New Query form above, and click on it. If the broker is done computing candidate matches, then a screen like the following screen should come up, except without the circles filled in:

Round 1:

• Follow the "?" tooltips to provide feedback on the candidate matches. In this example, we judged all 6 candidates to be valid matches of our intention in choosing the reference clip. The outer circles are colored green for suggested matches and red for suggested near-matches. When the user selects "Valid" or "Invalid", the circles are filled with the appropriate color (green or red). Clicking on a circle brings the video to that clip, and enlarges the circle for easy reference.
• When done with your feedback, click on "Submit".

• The intention is to perform a small number of rounds, until most of the suggested matches (green circle) are valid and most of the near matches (red circle) are invalid.
• Below are screen captures of the subsequent two rounds performed by us. See if you agree with our selections for Valid and Invalid matches. There is some randomness because the software chooses a random subset of matches and near misses, in order to keep the number needing review to less than your "Matches to review" preference.

Round 2: Most suggested matches were judged to actually be Invalid for the intended purpose. There seems to be a separation of validated and invalidated matches, just not near the current estimated threshold.

Round 3: Now the query is working well. There are 11 "Valid" matches with only two matches in between that we felt were invalid. See what results you get. Since in this case there are more matches and near matches than the user-specified Max Matches to review, the algorithm randomly chooses among all matches and near matches to display no more than Max Matches. Thus, your results might be different from the screen shot below, and you may take more rounds to converge to a result with high accuracy relative to the intended criteria.

• Click Finalize when you feel the query has reached a reasonable level of accuracy. A csv file with results will be generated and an icon presented for downloading.

The Round 1-3 screens shown above are using chart option 1. Options 2 and 3 are also available and can be selected by clicking the appropriate number below the chart. Here are screen shots for Chart 2 and Chart 3 options:

Chart 2 is a histogram view:

Chart 3 is a tabular view - click on the tooltip (?) for more help:

• In this example, we want clips where a pedestrian is walking in front of the vehicle and the vehicle stays in motion. The application quickly converges on accurate scoring. See if you agree.

Round 1:

Round 2: