9.9.3 Machine Learning - caligrafy/caligrafy-quill GitHub Wiki

Machine Learning is a cornerstone of Artificial Intelligence. It is attempting to provide the machine with the ability to learn and anticipate outcomes based on previous successes and failures.

Machine Learning remains a relatively complex field that could feel intimidating to many of us. Luckily, in the last couple of years, several organizations and open source communities have been developing tools and libraries that help abstract the complex mathematical algorithms in order to encourage developers to easily create learning models and train them using any programming languages.

Caligrafy created a built-in framework that is implemented on top of several open source libraries. It uses the ML5.js library built on top of Google's Tensor Flow to give developers the means to implement detection and recognition based on pre-trained models or to implement simple neural networks to train models. More over, in order to create web user interfaces more easily to interact with the AI models, Caligrafy uses the P5 library.

In this section, we will explain and demonstrate how Caligrafy can be used to build compelling detection and recognition applications.

Creating and training an AI with Caligrafy

In this video, we use Caligrafy to build neural networks and computer vision algorithms to create an AI and train it to detect and recognize humans, objects and to predict outcomes.

The Caligrafy Machine Learning framework is a javascript library that is agnostic to the framework. It can be used with bare-bone Javascript implementations or with any other frameworks.

The Caligrafy-Quill distribution uses a bare bone Javascript implementation that is built on top of the ML5.js and P5.js. You need to use Caligrafer to create a boiler-plate code template to get your started.

In order to do so, run the following command from the command line: .bin/caligrafer ml <app_name> or php caligrafer.php ml <app_name>

Once the scaffolding is completed, verify that a new folder <app_name> got created in the public folder.

You can even immediately try some of the features that are provided out-of-the-box by running it in the browser:

• General Framework: http://localhost/<caligrafy_project_root>/ml/<app_name>
• Feature Extractor: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=featureextractor
• Pose Detector: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=posedetector
• Neural Network: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=neuralnetwork

The Page Route to the application is already created for all client-side apps in the web.php file.

    /* ML Routes
     *
     * If you don't specify an app, it will go to the face, body detector.
     * Specify an app as a URI Parameter ?app=<name of app>
     * 
     * There are 3 possible apps: featureextractor, neuralnetwork and posedetector
    */
    Route::get('/ml/{appName}', 'ClientController'); 

Learn more about how to create your own Page Routes here

Caligrafy Machine Learning comes with a structure of its own. Understanding that structure is key to understanding the different components of the framework and how they interact with each other. All the client applications that you create through Caligrafer or manually reside in the public folder. The <app_name> contains the following parts:

• scripts/main.js: The javascript files that use the Caligrafy ML Core library to show the 4 different implementations (General, Feature Extractor, Pose Detector and Neural Network)

• .php files: The markup HTML files that will reference each of the scripts. You need to make sure to load the right js file from the markup.

• resources folder: This folder is empty out-of-the-box. It will be the folder that will contain any models or data needed to load onto the application

• css folder: The css folder all the css files needed for your application.

Caligrafy ML Core is a client-side framework. All the logic takes place on the browser side using Javascript. The ML Core framework relies on 3 main components:

• MlCore Class: This is the core class for using the machine learning in your application. The core code is in mlcore.js that can be found in the public/js/services folder

• ML5.js: ML5 is a Google-backed open source project that aims to make machine learning approachable for a broad audience of artists, creative coders and students. It provides access to several machine learning algorithms and models directly in the browser building on top of TensorFlow.js without any additional dependencies. Learn more about ML5

  The Ml Core Class builds on top of the ML5 library. The Ml5 library therefore needs to be imported into the markup before loading ML Core as shown in the example files index.php and featureextrator.php that are in your <app_name> folder.

     <!-- Initialization scripts -->
  
     <!-- ml5 -->
     <script src="https://unpkg.com/[email protected]/dist/ml5.min.js"></script>

• P5.js: P5 is an Javascript open-source library for creative coding that focuses on making coding accessible and approachable to a larger audience of artists, designers, educators, beginners and anyone else. P5 provides a full set of drawing capabilities to be used in HTML. Learn more about P5

  The Ml Core Class builds on top of the P5 library. The P5 library therefore needs to be imported into the markup before loading ML Core as shown in the example files index.php and featureextrator.php that are in your <app_name> folder.

     <!-- Initialization scripts -->
  
     <!-- p5.js -->
     <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/p5.min.js"></script>
     <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/addons/p5.dom.min.js"></script>
  	
     <!-- ml5 -->
     <script src="https://unpkg.com/[email protected]/dist/ml5.min.js"></script>

ML Core relies fundamentally on the following concept: "There is a Brain that detects and learns and there is a UI that needs to be drawn to interface with it". There are therefore 2 main paradigms that play together to create that intelligence: A Brain and an Interface

The ML Core Brain relies on neural networks to achieve both detection and learning. Caligrafy come pre-packaged with trained networks that can be refined and improved. Or, it provides a framework for you to build and train your own neural network.

• Initialization

  Out of the box the brain has been pre-trained to detect human faces and body postures in videos, photos or webcams. ML Core uses Face Detection (using UNet model) and Posture Detection (using the PoseNet model) to achieve those detections.

     
      // Initializing ML Core for detections
  
      var myMl = new MlCore();
  
      myMl.detect((myMl) => {
       // do something upon detections
  
      });

• Methods

      
      /* 
       * Training Methods 
       */
      
      myMl.addImage(myMl, labelInputs) // adds sample image data for training the brain
  
      myMl.train(myMl, options = {}) // trains the model and upon results automatically starts classifying
      // options: { batchSize: 24, epochs: 32 }
  
      myMl.classify(myMl) // classifies the detections
  
      myMl.save(outputName, callback, myMl) // saves model
  
      myMl.load(fileOrpath, callback, myMl) // loads modal
  
  
      /*
       * Visualization/Drawing methods
       */
  
      myMl.drawFeature(myMl, Array features to draw) // draws particular feature of the face (nose, leftEye, rightEye, leftEar, rightEar, mouth, topLip, bottomLip etc.
  
      myMl.drawKeypoints(myMl) // draws the keypoints detected on the entire body
  
      myMl.drawSkeleton(myMl) // draws the lines for arms, shoulders, legs etc...

• Initialization

  Caligrafy relies on well established computer vision models to train the machine to recognize data and image patterns. In order to learn from images, MLCore uses Feature Extractor that relies on a highly pre-trained computer vision algorithm MobileNet.

     // Initializing a feature classifier
     var myMl = new MlCore({ brain: {type: 'featureextractor', options: { numLabels: 3, epochs: 20, batchSize: 0.4 }}}); 

• Methods

       
     myMl.addImage(myMl, labelInputs) // adds sample image data for training the brain
  
     myMl.train(myMl, options = {}) // trains the model and upon results automatically starts classifying
     // options: { batchSize: 24, epochs: 32 }
  
     myMl.classify(myMl) // classifies the detections
  
     myMl.save(outputName, callback, myMl) // saves model
  
     myMl.load(fileOrpath, callback, myMl) // loads modeal

• Initialization

  When we talk about learning, under the hood it means improving the ability of the machine to predict an outcome (output) in a particular context (input) by training it (manually or programmatically) to recognize patterns. The predictions of outcomes could be either qualitative or quantitative.

  
  

  A qualitative prediction for example could be the recognition of a human pose (crouching, standing, etc.). This is about labeling a combination of posture coordinates. ML Core achieves this type of learning through what is called Classification in the AI field.

     // Initializing a neural network for classification
     var myMl = new MlCore({ brain: { type: 'neuralnetwork', options: { inputs: 3, outputs: 1, debug:true, task: 'classification'}}}); 

  
  

  A quantitative prediction for example could be the recognition of a price based on the recognition of a historical trend. ML Core achieves this type of learning through what is called Regression in the AI field.

     // Initializing a neural network for regression
     var myMl = new MlCore({ brain: { type: 'neuralnetwork', options: { inputs: 3, outputs: 1, debug:true, task: 'regression'}}}); 

  
  

• Methods

     myMl.brain.addData(labelInputs) //adds sample data for training the brain
  
     myMl.brain.addImage(labelInputs) // adds sample image data for training the brain
  
     myMl.brain.train(options = {}) // trains the model and upon results automatically starts classifying
     // options: { batchSize: 24, epochs: 32 }
  
     myMl.brain.classify() // classifies the detections
      
     myMl.brain.normalizeData() // normalizes the data (to be used when continuous data and not discrete)
      
     myMl.brain.save(outputName, callback) // saves the model to specified file name
  
     myMl.brain.load(fileOrpath, callback) // loads the saved model

When the Brain makes sense of the information, it needs a way to communicate its findings and reciprocally, when it needs intervention from the human operator, it needs to be able to provide them with graphical input elements.

ML Core uses the P5 library to allow the creation of more elaborate visualizations and interfaces using simple language. You can learn more about the capabilities of the P5 library here

In order to use the P5 library, MLCore needs to hand the canvas over to P5

   function setup() {
          
       myMl.toP5(myMl); // hand off to P5

       /* P5 UI components for interfacing with the brain go here */

   }
   
   function draw() {
        
     /* P5 Canvas drawing for visualizing detections goes here. 
This function is executed only if the hand off to P5 is executed in setup */

   }

ML Core has 3 distinct code patterns that are optimized for different purposes.

1. Detect and Visualize

   In this mode, the ML Core code structure flows as such: initialize the brain, set it up, visualize the detections and optionally do custom drawings.

   A typical example of this flow is the General Framework example http://localhost/<caligrafy_project_root>/ml/<app_name>.

   
   

   View Code

   Live Example

   
   

2. Unattended Learning

   In this mode, the ML Core flow is designed to programmatically train the machine to predict an outcome. The ML Core structure flows as such: initialize the neural network, load or create the samples, add the samples to the network, optionally normalize the data, train the network, classify/predict.

   A typical example of this flow is the Neural Network example http://localhost/<caligrafy_project_root>/ml/<app_name>?app=neuralnetwork.

   
   

   View Code

   Live Example

   
   

3. Attended Learning

   In this mode, the ML Core flow is designed for the intervention of a human operator in the training of the machine. It therefore requires a user interface. The Ml Core structure flows as such: initialize the brain, visualize the detections, set up a P5 user interface that adds samples, trains and classifies/predicts, visualize the predictions on the user interface.

   Typical examples of this flow are the Feature Extractor and Pose Detector examples:

   • Feature Extractor: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=featureextractor.

     
     

     View Code

     Live Example

     
     

   • Pose Detector: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=posedetector.

     
     

     View Code

     Live Example