Open Face 2 integration - isir/greta GitHub Wiki

OpenFace 2, developed at the CMU MultiComp Lab, is a facial behavior analysis toolkit capable of real-time performance. The system is capable of performing a number of facial analysis tasks:

  • Facial Landmark Detection
  • Facial Landmark and head pose tracking (links to YouTube videos)
  • Facial Action Unit Recognition
  • Gaze tracking (image of it in action)
  • Facial Feature Extraction (aligned faces and HOG features)

It can forward all facial expressions, head pose, and gaze to any other software

image

It can either :

  • read the standard output file from the standard OpenFace2 program. Reading is dynamic, synchronous with OpenFace writing.
  • connect via ZeroMQ protocol on port 5000. Using the OpenFaceOfflineZeroMQ application (built with this branch of OpenFace 2)

image

In Greta use "OpenFace2 Output Stream Reader" module to listen to OpenFace inputs and connect them to the specific Greta modules handling AUs and headpose.

image

The module allows :

  1. OpenFace data Input selection

  2. Signal processing

  3. To forward any information to a debug application using the OSC protocol

  4. OpenFace data Input selection The UI allows dynamic and easy selection of the OpenFace 2 information. It can be used to use only a specific set of AUs.

image

  1. Signal processing A Facial action unit is composed of:
  • a continuous signal : how much is feature is activated
  • a discrete signal : is feature detected or not Raw signal :

image

Mask :

image

Hence the need to filter the signal processing filter.

Filtered signal (kernel size of 5, weight function with power 0.5) :

image

A dynamically sized kernel processing approach where the most recent signal value is the last index of the kernel. Each kernel weight is valued with the mathematical “pow” function which conveniently grows from 0 to 1 for x =[0-1]. So the most recent values have the most weight.

image 12 normalized kernel values for different power values

Demonstration videos :