9.9.2 Face Detection and Face Recognition - caligrafy/caligrafy-quill GitHub Wiki

Detecting human faces and recognizing faces and facial expressions have always been an area of interest for different applications such as games, utilities and even security. With the advancement of machine learning, the techniques of detection and recognition have become more accurate and precise than ever before.

However, machine learning remains a relatively complex field that could feel intimidating or inaccessible to many of us. Luckily, in the last couple of years, several organizations and open source communities have been developing tools and libraries that help abstract the complex mathematical algorithms in order to encourage developers to easily create learning models and train them using any programming languages.

Artificial Intelligence (AI) and Machine Learning in particular don't have to be difficult and we hope that the FaceDetect framework gives developers the means to include face detection and recognition in a seamless way in their applications.

The Caligrafy FaceDetect framework was built on top of a FaceAPI library developed by Vincent Muhler and relies on TensorFlow models. FaceDetect can be used directly on the Web without the need to directly manipulate TensorFlow models. It uses pre-trained models for face detection and recognition.

Learn more about the FaceAPI Library here

Detecting and Recognizing human faces with Caligrafy

In this video, we show how the Caligrafy Face Detect library can be used to detect and recognize human faces and use the info to create all sorts of applications.

Live Example

The Caligrafy FaceDetect framework is a javascript library that is agnostic to the framework. It can be used with bare-bone Javascript implementations or with any other frameworks.

The Caligrafy-Quill distribution has a tight integration with Vue and you will need to use Caligrafer to create a boiler-plate code template to get you started.

In order to do so, run the following command from the command line: .bin/caligrafer facedetect <app_name> or php caligrafer.php facedetect <app_name>

Once the scaffolding is completed, verify that a new folder <app_name> got created in the public folder containing all the necessary modules needed to implement face detection and recognition.

You can even immediately try some of the features that are provided out-of-the-box by running it in the browser: http://localhost/<caligrafy_project_root>/apps/<app_name>

The Page Route to the application is already created for all client-side apps in the web.php file.

    // ROUTING TO JAVASCRIPT APPS
    // Make sure you create a JS app or a Vue app in Caligrafer first
    Route::get('/apps/{appName}', 'ClientController'); // you can specify any path as long as it ends with appName

Learn more about how to create your own Page Routes here

FaceDetect comes with a structure of its own. Understanding that structure is key to understanding the different components of the framework and how they interact with each other. All the client applications that you create through Caligrafer or manually reside in the public folder. The <app_name> contains the following parts:

• scripts/main.js: The VueJS script file

• index.php: The markup HTML file that will reference the VueJS instance

• recognition folder: In order to recognize specific people, FaceDetect needs models to compare the detections to. Those need to be provided to your application in the recognition folder. These models are nothing more but a series of pictures of faces organized in sub-folders named after the face that they represent. As an example, you can find a subfolder called Flash that has 6 (PNG) pictures named by number of the superhero Flash. The same mechanism should be used to create more models.

• css folder: The css folder all the css files needed for your application.

FaceDetect is a client-side framework. All the logic happens on the browser side using Javascript. The FaceDetect framework relies on 3 main components:

• FaceDetector Class: This is the core class for using FaceDetect in your application. The core code is in detect.js that can be found in the public/js/services folder.

• neural network models: Every machine learning application relies on trained models to be able to do things such as object detection or recognition. FaceDetect is no exception to that rule and it needs these models to operate properly. These models are included in Caligrafy and can be found in public/resources/models.

• faceapi.js: The FaceAPI developed by Vincent Muhler is included in the package.

In order to illustrate the different features of FaceDetector, Caligrafy comes prepackaged with an application that uses some of its features. The documentation will rely on it for illustrating the different concepts.

Detection and Recognition are 2 different concepts. While both use machine learning and neural networks in particular, they achieve different things.

• Detection: Detection is about identifying a human face among all other "things" perceived through either an image or a video. So for example, a picture or a video can have people, objects, scenery etc... Face detection is when the system is capable of pointing the presence of a human face among all those other things.

• Recognition: Recognition is about identifying who the human face is among all other faces and things perceived. So for example, Face recognition is when the system is capable of pointing out "Flash" among all other superheroes in a picture or a video.

Understanding the distinction between detection and recognition is key to understanding the underlying logic of the FaceDetector class.

FaceDetector relies on an important principle: "First you detect then you do something with the detections". With that principle in mind, the framework focuses on providing an easy-to-code sandbox for you to do something with the detections. Each sandbox is an application of its own. So if in your application you intend to detect the age of a face or detect the facial expressions or count the number of people or recognize a face etc.. Each one of those is referred to as an application.

Your application index.php file is the user interface that will define the source of the face detections (image or video) and any other controls needed to make something useful with the detections. You will need to include the FaceDetect needed libraries into the markup with and without Vue.

• Without Vue

<!-- Initialization scripts -->

<script src="<?php echo APP_SERVICE_ROOT.'app.js'; ?>"></script>
<script src="<?php echo APP_SERVICE_ROOT.'face-api.min.js'; ?>"></script>
<script src="<?php echo APP_SERVICE_ROOT.'detect.js'; ?>"></script>
<script>loadEnvironment(`<?php echo $env; ?>`);</script>

<!-- Additional scripts go here -->
<script src="<?php echo scripts('bootstrap_jquery'); ?>"></script>
<script src="<?php echo scripts('bootstrap_script'); ?>"></script>

<!--[if lt IE 9] -->
<script src="<?php echo scripts('fallback_html5shiv'); ?>"></script>
<script src="<?php echo scripts('fallback_respond'); ?>"></script>
<!--<![endif]-->

• With Vue

<!-- Initialization scripts -->
<script src="https://unpkg.com/vue@3"></script>
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
<script src="<?php echo APP_SERVICE_ROOT.'app.js'; ?>"></script>
<script src="<?php echo APP_SERVICE_ROOT.'face-api.min.js'; ?>"></script>
<script src="<?php echo APP_SERVICE_ROOT.'detect.js'; ?>"></script>
<script>loadEnvironment(`<?php echo $env; ?>`);</script>
<script> 
   /* Loading the app client framework
    * Any environment variables to be passed on from the server can take place in this here
    */
   loadVue({
      scripts: ['main']
    });
</script>
        
<!-- Additional scripts go here -->
<script src="<?php echo scripts('bootstrap_jquery'); ?>"></script>
<script src="<?php echo scripts('bootstrap_script'); ?>"></script>
        
<!--[if lt IE 9] -->
<script src="<?php echo scripts('fallback_html5shiv'); ?>"></script>
<script src="<?php echo scripts('fallback_respond'); ?>"></script>
<!--<![endif]-->

FaceDetect is identified in the markup by the id detector. FaceDetect will be encapsulated within that block. If you are using Vue, the detector block lives within the Vue app id.

<!-- Vue app -->
<div id="app">
   <section id="detector">
   ....
   </section>
</div>

No matter what you want to do with FaceDetect, detection is the first step to it. It is therefore important to identify what the detection source is. Is it an image, a video or a live webcam?

The markup of your application needs to provide that source:

<!-- Beginning of the app -->
<div id="app">
   
   <section id="detector">

      <!-- media can be an image 
      <img id="detection" class="show" src="">
      -->

      <!-- media can be a video 
      <video id="detection" src="<video local url>" type="video/mp4" width="720" height="560" class="show" autoplay="autoplay" muted playsinline controls></video>
      -->

      <!-- media can be live webcam -->
      <video id="detection" width="720" height="560" class="show" autoplay="autoplay" muted playsinline></video>

   </section>

</div>

So far, only the source has been specified. In order to do something with it, it needs one or more UI triggers to activate it. FaceDetector provides you with a way to create these controls if you desire. In order to do that, you will need to add the controls placeholder to your markup.

<section class="controls">
   <div id="apps"></div>
</section>

FaceDetect provides you with a UI component to display welcome messages, status or instruction messages called Infobar. In order to use it, you will need to add the infobar placeholder to your markup.

<section id="infobar"></section>

A basic stylesheet is provided in the facedetect example app to illustrate how the media sources, controls and infobar can be styled if you decide to use them.

If you are using the Vue integration, you will need to indicate in the loadVue script described in the markup the name of the javascript main Vue file. In the example provided, we call it main.

If you are not using Vue, link a javascript file called main.js to the markup. We will refer to this file as main for convenience. It could have any name.

Whether you are using Vue or not, FaceDetect has not been written for VueJS. In fact we will integrate it in a VueJS structure but it is still used as an independent JS class. The code examples that we will show moving forward use VueJS but the same can be done using standard JS.

FaceDetector is an object that can be initialized by providing it the HTML ID of the media source. It is therefore important to make sure that the markup for either an image or a video has an id attribute.

Upon successful instantiation, all the models are loaded and the features of the class can be used.

// make sure to use the HTML id of the media source
var detector = new FaceDetector('detection');

var app = Vue.createApp({
  el: '#app',
  data () {
    return {
        detector: detector, /* important */
        env: env

    }
  },
  /* Method Definition  */
  methods: {
      
      
  },
  /* upon object load, the following will be executed */
  mounted () {
      
      
  }

});

// mount the app
app.mount('#app');

FaceDetector provides you with 4 out-of-the-box sandboxes that you can use to benefit from face detection and face recognitions.

Upon instantiating a sandbox several things may happen:

1. A button is created that allows triggering/starting the detection
2. A canvas is created overlapping the media source, allowing you to draw on top of the image or video
3. If you are using the infobar, it is instantiated with a message that you specify

The basic sandbox is a detection sandbox that will draw a box around the detected faces in the media source. In order to run the basic sandbox, the loadApp method is called on the instantiated FaceDetector object.

<!-- without Vue -->
detector.loadApp();

<!-- with Vue -->
this.detector.loadApp();

A more elaborate sandbox is one where you can configure what to draw on the detections. It offers several options that are additive when present.

<!-- Vue example -->
this.detector.loadApp({
    name: 'Name of the app',  // name that will appear on the button created
    method: this.detector.draw, // or detector.draw without Vue
    options: {
       welcome: "The message to display in the infobar",
       detection: true, // draws a box around the detected faces when set to true
       landmarks: true, // line draws the detect faces when set to true
       gender: true, // detects the gender - male or female - when set to true
       age: true, // estimates the age when set to true
       expression: true // detects happy, sad, angry or surprised when set to true
    },
    algorithm: faceapi.SsdMobilenetv1Options // different algorithms can be used for detection. The default algorithm is faceapi.TinyFaceDetectorOptions

});

The recognition sandbox is another one that recognizes the detections. If you would like to run recognitions, there are 2 steps that you need to do:

Step 1: Define the recognition models

The easiest way to define a recognition model is to create a recognition folder in your app main folder. The recognition folder will contain many pictures of people organized by person name. Each person to be recognized will have a folder in their name and the pictures inside must be PNGs and must be named with a number.

So for example, if you are detecting the Flash, in the public/<your app folder>/recognition/Flash, you will have a list of of PNG files such as 1.png, 2.png etc.

The same number of images need to be provided for all the models. The lack of doing that may result in unexpected behaviors.

Step 2: Run recognition

Running the recognition is as easy as loading a detection app.

<!-- Vue example -->
this.detector.loadApp({
    name: "Recognize",
    method: this.detector.recognize,
    models: {
         labels: ['Flash', 'Person X', etc...] // array of all the names of the people which are also the names of the folders in the structure
         sampleSize: 6 // number of pictures per person (this number must be the same for all)
    },
    options: {
        welcome: "The message that you want to display in the infobar",
        recognition: true // the recognition engine needs to be activate
    },
    algorithm: faceapi.SsdMobilenetv1Options // Optional. The detection algorithm that will be used
})

The 4th sandbox is a custom sandbox. For more flexibility, the FaceDetect framework provides you with the ability to invoke your custom methods at specific points of the face detections.

• Continuously, during face detection

If you want to do something with the detections or the recognitions continuously while a video or webcam are continuously detecting and/or recognizing faces, you need to invoke your own method from the loadApp and to set the custom setting to false.

this.detector.loadApp({
  name: "Custom continuous",
  method: this.continuousMethod,
  custom: false, // set to false if you want the method to be applied continuously at every interval of detection
  options: {
      welcome: "Open the console to see how it is continuously being called at every detection",
      detection: true
  }
});

The continuousMethod() needs to be defined as a your own method in Vue or in JS. It takes the object facedetector as an argument.

// Vue example
var detector = new FaceDetector('detection');

var app = Vue.createApp({
  el: '#app',
  data () {
    return {
        detector: detector,
        env: env

    }
  },
  /* Method Definition  */
  methods: {
      
      continuousMethod: function(facedetector) {
          // you can access detections by calling the property facedetector.app.detections. It returns an array of all faces detected every 100ms
          console.log(facedetector.app.detections);

         // you can access recognitions by calling the property facedetector.app.recognitions. It returns an array of the recognized faces every 100ms. recognitions is only available after the recognize method is called
         //console.log(facedetector.app.recognitions;
      }
      
  },
  ...

• Or, through a callback method that will allow you to manually control face detection and recognition

If you would like to have full control over the framework, you can load a custom app by setting the custom property to true and invoking a callback method in the method property.

this.detector.loadApp({
          name: "Custom callback",
          method: this.callbackMethod,
          custom: true, // set to true if you want the method to do something else before calling in FaceDetect features
          options: {
              welcome: "Open the console to see how it is executing its content and waiting for more to be done",
              detection: true
          }
          
});

When the callbackMethod() is invoked, only the app is initialized with the canvas initiated but no detection nor recognition is started. You can control all the detections, recognitions, and UI elements manually.

// Vue example
var detector = new FaceDetector('detection');

var app = Vue.createApp({
  el: '#app',
  data () {
    return {
        detector: detector,
        env: env

    }
  },
  /* Method Definition  */
  methods: {
      
      callbackMethod: function(facedetector) {
         
        //<---- do whatever you want here
          console.log('hello');
        
      }
      
  },
  ...

Both FaceDetect and FaceApi provide a set of properties and methods that can help you control the detections and the recognitions and manipulate them the way you would like.

   
   // PROPERTIES

   this.detector.app.detections // returns a stream of detections. You can iterate through them to act on them

   this.detector.app.recognitions // returns a stream of recognitions

   this.detector.app.canvas // returns the canva that overlays the video

   this.detector.app.options // exposes all the options of the app

   this.detector.media // exposes the media used (video, picture, cam stream)
               .media.width // gets the width
               .media.height // gets the height
   

   // METHODS

   this.detector.loadApp(app) // load another app

   (this.detector.detectFaces(app, facedetector))() // self invoking function to start face detection

   this.detector.detect(callback, recognize = false, fetchRate = 100) // starts a parallel stream that captures any detections or recognitions when available

   this.detector.prepareCanva(options = null) // returns a new canvas on top of the media source

   this.detector.draw(facedetector) // draws the detections on the canvas

   this.detector.loadRecognition({ labels: [], images: [], sampleSize: 100}) // load models to recognize by the recognition engine

   this.detector.recognize(facedetector): // runs the recognition engine and draws on canvas. Must make sure that detections is started before

   this.detector.fetchImage(canvas, media) // takes a canvas capture of the media and returns a blob data image (data url)

   this.detector.display(message, output) // displays a message in the infobar and gives it an ID as specified by the 'output' input

   this.detector.clearDisplay() // clears the infobar display

   // PROPERTIES OF EVERY DETECTION

      - detection.detection.box // the box around the face
      - detection.age
      - detection.expression
      - detection.gender
      - detection.landmarks
      - detection.descriptors

    // METHODS

      (new faceapi.draw.DrawFaceLandmarks(landmark, {settings})).draw(canva) // function that draws the face landmark. Landmarks are attributes of the detections like age, gender etc.
      
      draw landmark settings : drawLines (boolean), drawPoints (boolean), lineWidth (number), lineColor (rgba(x, x, x, x)), pointSize(number), pointColor(rgba(x, x, x, x))
 
      faceapi.draw.drawDetections(canva, detections)// Draws the detections on the canva
       
      (new faceapi.draw.DrawTextField(Array of strings to display, anchor)).draw(canva) 
      // Draws text on the canva around the detections
      // anchor: where the text should be placed at every detection. For example it could be: detection.detection.box.bottomLeft
      
      (new faceapi.draw.DrawBox(box, {settings})).draw(canva) // draws a box around the face 
   
      faceapi.matchDimensions(canva, {width: media width, height: media height}) // changes the dimensions of the canva to match the media

      faceapi.resizeResults(detections, {width: media width, height: media height}) // resizes the detections to fit within the video
      
      faceapi.detectAllFaces(media, algorithm) 
      /* could be appended with:
       *                                         .withFaceLandmarks()
       *                                        .withFaceDescriptors()
       *                                         .withAgeAndGender()
       *                                         .withFaceExpressions()
       */

       faceapi.detectSingleFace(image model, algorithm) 
       /* detects a single face
       *                                        .withFaceLandmarks()
       *                                        .withFaceDescriptor()
       *                                        .withAgeAndGender()
       *                                        .withFaceExpressions()
       */
       
       faceapi.LabeledFaceDescriptors(label, descriptors) // returns the face descriptors from a an image and associates them with a label
       
       faceapi.FaceMatcher(recognition model, threshold for matching)  // Compares the detections to the model and identifies if there is a match. The model could be created using the FaceDetect loadRecognitionModel(...)

       faceapi.createCanvasFromMedia(media) // creates a canva on top of the video or picture to manipulate it
       
       faceapi.fetchImage(url) // creates a model from the image to be use by the recognition engine