Algorithms Compute Video Features - PARC-projects/video-query-home GitHub Wiki

0. Set environment variables if not already set
1. Build RGB and warped optical flow clips
2. Calculate features for video clips
3. Load features and clips into the API database

One way to set these is to execute

bash set_environ.sh in Linux or
source set_environ.sh in MacOS,

using a set_environ.sh file in your local video-query-algorithms main directory. Here is an example of a set_environ.sh file used for development:

#!/usr/bin/env bash
export TSN_ROOT='/data/torres/temporal-segment-networks'
export TSN_ENVIRON='TSN'
export API_CLIENT_USERNAME='your_username'
export API_CLIENT_PASSWORD='your_password'
export BROKER_THREADING='True'
export COMPUTE_EPS=.000003
export RANDOM_SEED=73459912436

Run in the temporal segment networks directory within the conda TSN environment (or equivalent virtualenv):

cd TSN_ROOT

Run build_wof_clips.py:

python tools/build_wof_clips.py SRC_FOLDER OUT_FOLDER 
   --num_worker NUM_WORKER  
   --new_width 340
   --new_height 256 2>local/errors.log

• SRC_FOLDER
  • Folder with videos we will use to build clips.
  • e.g. Videos/
• OUT_FOLDER
  • The output folder that will be populated with the frames and warped optical flow images. There is a subdirectory for each video, holding the jpg frames and optical flow images for that video sorted into clips.
  • e.g. Video_clips/
  • Produces a subdirectory structure of the form
    OUT_FOLDER/{video name}/clip_{clip number}/{img, flow_x, or flow_y}_{frame number}.jpg
• NUM_WORKER
  • The number of CPU's to spread the work across. 16 seems to be a good number if there are 16 or more videos and 16 or more CPU cores available, and fewer GPUs. Higher is not always faster, as getting files on and off the GPUs can be rate limiting. There will not be more CPU processes than the number of videos, so NUM_WORKER greater than the number of videos has no effect.
  • default = 16
• errors.log can be substituted with a more specific name. The program will not overwrite an existing file.
• Frames per second: the default is 15 fps. Add --fps nn to the command line for different frame rates.

bash calcSig_wOF_ensemble.sh SRC_FOLDER_2 OUT_FOLDER_2 NUM_WORKER

• SRC_FOLDER_2
  • The folder with jpeg RGB and warped optical flow images. Normally this folder is the same as OUT_FOLDER from step 1. The folder is structured as SRC_FOLDER_2/{video names}/{clip folders}/*.jpg.
• OUT_FOLDER_2
  • The root folder for the computed features.
  • e.g. Video_features/
  • the subdirectory structure is
    OUT_FOLDER_2/{video name}/{CNN model name}/{feature type}_{feature name}_features.csv
• NUM_WORKER
  • The number of GPU processes. Can be greater than the number of GPUs as long as the jobs fit in the GPU memory.
  • NUM_WORKER <= min(number of clips per video, number of CPU cores) is the guideline.

The API needs to be running for this program to work.
load_db.py loads the "video_clips" and "features" tables. Normally src_dir is the same as OUT_FOLDER_2 from step 3.

python load_db.py --help

usage: load_db.py [-h] [--duration DURATION]
                  [--video_path_type {absolute,relative}]
                  [--base_url BASE_URL]
                  src_dir

Load video clip features into Video-Query database

positional arguments:
  src_dir               directory with video files

optional arguments:
  -h, --help            show this help message and exit
  --duration DURATION   clip duration, s, integer only, default=10
  --video_path_type {absolute,relative}
                        relative paths will have a parent specified in the
                        video query api
  --base_url BASE_URL   url for video query api, default="http://127.0.0.1:8000/"

For example, to load all the features for the stock video ("DowntownBrooklynDrive_480p") provided in the video-query-algorithms GitHub repository, execute (note: only works in python 3.6 and higher because of changes in os.scandir):

python load_db.py ../data/features/stock-video-clips_features --base_url "<api url>"