9.9.3 Machine Learning - caligrafy/caligrafy-quill Wiki

Machine Learning is a cornerstone of Artificial Intelligence. It is attempting to provide the machine with the ability to learn and anticipate outcomes based on previous successes and failures.

Machine Learning remains a relatively complex field that could feel intimidating to many of us. Luckily, in the last couple of years, several organizations and open source communities have been developing tools and libraries that help abstract the complex mathematical algorithms in order to encourage developers to easily create learning models and train them using any programming languages.

Caligrafy created a built-in framework that is implemented on top of several open source libraries. It uses the ML5.js library built on top of Google's Tensor Flow to give developers the means to implement detection and recognition based on pre-trained models or to implement simple neural networks to train models. More over, in order to create web user interfaces more easily to interact with the AI models, Caligrafy uses the P5 library.

In this section, we will explain and demonstrate how Caligrafy can be used to build compelling detection and recognition applications.

Initializing a Machine Learning application

Step 1: Scaffold an ml application using Caligrafer

The Caligrafy Machine Learning framework is a javascript library that is agnostic to the framework. It can be used with bare-bone Javascript implementations or with any other frameworks.

The Caligrafy-Quill distribution uses a bare bone Javascript implementation that is built on top of the ML5.js and P5.js. You need to use Caligrafer to create a boiler-plate code template to get your started.

In order to do so, run the following command from the command line: .bin/caligrafer ml <app_name> or php caligrafer.php ml <app_name>


Step 2: Verify the application

Once the scaffolding is completed, verify that a new folder <app_name> got created in the public folder.

You can even immediately try some of the features that are provided out-of-the-box by running it in the browser:

  • General Framework: http://localhost/<caligrafy_project_root>/ml/<app_name>
  • Feature Extractor: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=featureextractor
  • Pose Detector: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=posedetector
  • Neural Network: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=neuralnetwork

Step 3 (optional): Customize the Page Route

The Page Route to the application is already created for all client-side apps in the web.php file.

    /* ML Routes
     *
     * If you don't specify an up, it will go to the face, body detector.
     * Specify an app as a URI Parameter ?app=<name of app>
     * 
     * There are 3 possible apps: featureextractor, neuralnetwork and posedetector
    */
    Route::get('/ml/{appName}', 'ClientController'); 

Learn more about how to create your own Page Routes here


Understanding the application structure

Caligrafy Machine Learning comes with a structure of its own. Understanding that structure is key to understanding the different components of the framework and how they interact with each other. All the client applications that you create through Caligrafer or manually reside in the public folder. The <app_name> contains the following parts:

  • scripts/main.js: The javascript files that use the Caligrafy ML Core library to show the 4 different implementations (General, Feature Extractor, Pose Detector and Neural Network)

  • .php files: The markup HTML files that will reference each of the scripts. You need to make sure to load the right js file from the markup.

  • resources folder: This folder is empty out-of-the-box. It will be the folder that will contain any models or data needed to load onto the application

  • css folder: The css folder all the css files needed for your application.


Caligrafy ML Core

Caligrafy ML Core is a client-side framework. All the logic takes place on the browser side using Javascript. The ML Core framework relies on 3 main components:

  • MlCore Class: This is the core class for using the machine learning in your application. The core code is in mlcore.js that can be found in the public/js/services folder

  • ML5.js: ML5 is a Google-backed open source project that aims to make machine learning approachable for a broad audience of artists, creative coders and students. It provides access to several machine learning algorithms and models directly in the browser building on top of TensorFlow.js without any additional dependencies. Learn more about ML5

    The Ml Core Class builds on top of the ML5 library. The Ml5 library therefore needs to be imported into the markup before loading ML Core as shown in the example files index.php and featureextrator.php that are in your <app_name> folder.

   <!-- Initialization scripts -->
   
   <!-- ml5 -->
   <script src="https://unpkg.com/[email protected]/dist/ml5.min.js"></script>
  • P5.js: P5 is an Javascript open-source library for creative coding that focuses on making coding accessible and approachable to a larger audience of artists, designers, educators, beginners and anyone else. P5 provides a full set of drawing capabilities to be used in HTML. Learn more about P5

    The Ml Core Class builds on top of the P5 library. The P5 library therefore needs to be imported into the markup before loading ML Core as shown in the example files index.php and featureextrator.php that are in your <app_name> folder.

   <!-- Initialization scripts -->
   
   <!-- p5.js -->
   <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/p5.min.js"></script>
   <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/addons/p5.dom.min.js"></script>
		
   <!-- ml5 -->
   <script src="https://unpkg.com/[email protected]/dist/ml5.min.js"></script>

Understanding ML Core

ML Core relies fundamentally on the following concept: "There is a Brain that detects and learns and there is a UI that needs to be drawn to interface with it". There are therefore 2 main paradigms that play together to create that intelligence: A Brain and an Interface

A Brain that detects and learns

The ML Core Brain relies on neural networks to achieve both detection and learning.

  • Detections

    When we talk about detections, under the hood it means that the computer has been pre-trained to recognize patterns and features in videos, photos or other media.ML Core uses Feature Extraction (using MobileNet model) and Posture Detection (using the PoseNet model) to achieve those detections.

       
       // Initializing ML Core for detections
       var myMl = new MlCore({ brain: {type: 'featureextractor'}}); 
       
       //Or, simply
       var myMl = new MlCore();
    

  • Learning

    When we talk about learning, under the hood it means improving the ability of the machine to predict an outcome (output) in a particular context (input) by training it (manually or programmatically) to recognize patterns. The predictions of outcomes could be either qualitative or quantitative.

    A qualitative prediction for example could be the recognition of the facial expression of a person (happy, sad, angry etc.). This is about labeling a face based on the recognition of a certain pattern of images. ML Core achieves this type of learning through what is called Classification in the AI field.

     // Initializing a neural network for classification
     var myMl = new MlCore({ brain: { type: 'neuralnetwork', options: { inputs: 3, outputs: 1, debug:true, task: 'classification'}}}); 
    

    A quantitative prediction for example could be the recognition of a price based on the recognition of a historical trend. ML Core achieves this type of learning through what is called Regression in the AI field.

     // Initializing a neural network for regression
     var myMl = new MlCore({ brain: { type: 'neuralnetwork', options: { inputs: 3, outputs: 1, debug:true, task: 'regression'}}}); 
    

An Interface to interact with the Brain

When the Brain makes sense of the information, it needs a way to communicate its findings and reciprocally, when it needs intervention from the human operator, it needs to be able to provide them with graphical input elements.

  • Visualization

    ML Core provides a pre-packaged set of visualization capabilities that allow drawing human features and poses directly on top of the detection medium (video, image etc.)

       
       drawFeature(myMl, Array features to draw) // draws particular feature of the face (nose, leftEye, rightEye, leftEar, rightEar, mouth, topLip, bottomLip etc.
    
       drawKeypoints(myMl) // draws the keypoints detected on the entire body
    
       drawSkeleton(myMl) // draws the lines for arms, shoulders, legs etc...
    
  • User Interface using P5

    ML Core uses the P5 library to allow the creation of more elaborate visualizations and interfaces using simple language. You can learn more about the capabilities of the P5 library here

Code Patterns

ML Core has 3 distinct code patterns that are optimized for different purposes.

  1. Detect and Visualize

    In this mode, the ML Core code structure flows as such: initialize the brain, set it up, visualize the detections and optionally do custom drawings.

    A typical example of this flow is the General Framework example http://localhost/<caligrafy_project_root>/ml/<app_name>. View Code

  2. Unattended Learning

    In this mode, the ML Core flow is designed to programmatically train the machine to predict an outcome. The ML Core structure flows as such: initialize the neural network, load or create the samples, add the samples to the network, optionally normalize the data, train the network, classify/predict.

    A typical example of this flow is the Neural Network example http://localhost/<caligrafy_project_root>/ml/<app_name>?app=neuralnetwork. View Code

  3. Attended Learning

    In this mode, the ML Core flow is designed for the intervention of a human operator in the training of the machine. It therefore requires a user interface. The Ml Core structure flows as such: initialize the brain, visualize the detections, set up a P5 user interface that adds samples, trains and classifies/predicts, visualize the predictions on the user interface.

    Typical examples of this flow are the Feature Extractor and Pose Detector examples:

    • Feature Extractor: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=featureextractor. View Code

    • Pose Detector: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=posedetector. View Code

⚠️ **GitHub.com Fallback** ⚠️