5-2 15 Naga Venkata Satya Pranoop Mutha

5-2 23 Geovanni West

---

Part 1 includes these steps:
Run KeyFrameDetection, which:

• Takes a video
• Breaks it up into seperate frames
• Get key frames from the list of frames

public static void main(String args[]){
        String path = "input/food.mkv";
        Frames(path);
        MainFrames();
    }
public static void Frames(String path){
    video = new XuggleVideo(new File(path));
    int j=0;
    for (MBFImage mbfImage : video) {
        BufferedImage bufferedFrame = ImageUtilities.createBufferedImageForDisplay(mbfImage);
        j++;
        String name = "output/frames/new" + j + ".jpg";
        File outputFile = new File(name);

        try {
            ImageIO.write(bufferedFrame, "jpg", outputFile);
        } catch (IOException e) {
            e.printStackTrace();
        }
        MBFImage b = mbfImage.clone();
        imageList.add(b);
        timeStamp.add(video.getTimeStamp());
    }
}

public static void MainFrames(){
    for (int i=0; i<imageList.size() - 1; i++)
    {
        MBFImage image1 = imageList.get(i);
        MBFImage image2 = imageList.get(i+1);
        DoGSIFTEngine engine = new DoGSIFTEngine();
        LocalFeatureList<Keypoint> queryKeypoints = engine.findFeatures(image1.flatten());
        LocalFeatureList<Keypoint> targetKeypoints = engine.findFeatures(image2.flatten());
        RobustAffineTransformEstimator modelFitter = new RobustAffineTransformEstimator(5.0, 1500,
                new RANSAC.PercentageInliersStoppingCondition(0.5));
        LocalFeatureMatcher<Keypoint> matcher = new ConsistentLocalFeatureMatcher2d<Keypoint>(
                new FastBasicKeypointMatcher<Keypoint>(8), modelFitter);
        matcher.setModelFeatures(queryKeypoints);
        matcher.findMatches(targetKeypoints);
        double size = matcher.getMatches().size();
        mainPoints.add(size);
        System.out.println(size);
    }
    Double max = Collections.max(mainPoints);
    for(int i=0; i<mainPoints.size(); i++){
        if(((mainPoints.get(i))/max < 0.01) || i==0){
            Double name1 = mainPoints.get(i)/max;
            BufferedImage bufferedFrame = ImageUtilities.createBufferedImageForDisplay(imageList.get(i+1));
            String name = "output/mainframes/" + i + "_" + name1.toString() + ".jpg";
            File outputFile = new File(name);
            try {
                ImageIO.write(bufferedFrame, "jpg", outputFile);
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
}

The mainframe method outputs its calculated value of each frame. It then uses this info to save only frames that are drastically different based on its value.

Then run Image Annotation, which:

• Pass these key frames onto Clarifai
• Once response is received, display the images with their respective description

public class ImageAnnotation {
    public static void main(String[] args) throws IOException {
        final ClarifaiClient client = new ClarifaiBuilder("****************************************", "*****************************")
                .client(new OkHttpClient())
                .buildSync();
        client.getToken();
    File file = new File("output/mainframes");
    File[] files = file.listFiles();
    for (int i=0; i<files.length;i++){
        ClarifaiResponse response = client.getDefaultModels().generalModel().predict()
                .withInputs(
                        ClarifaiInput.forImage(ClarifaiImage.of(files[i]))
                )
                .executeSync();
        List<ClarifaiOutput<Concept>> predictions = (List<ClarifaiOutput<Concept>>) response.get();
        MBFImage image = ImageUtilities.readMBF(files[i]);
        int x = image.getWidth();
        int y = image.getHeight();

        System.out.println("*************" + files[i] + "***********");
        List<Concept> data = predictions.get(0).data();
        for (int j = 0; j < data.size(); j++) {
            System.out.println(data.get(j).name() + " - " + data.get(j).value());
            image.drawText(data.get(j).name(), (int)Math.floor(Math.random()*x), (int) Math.floor(Math.random()*y), HersheyFont.ASTROLOGY, 20, RGBColour.RED);
        }
        DisplayUtilities.displayName(image, "image" + i);
    }

}

}

Using the Clarifai API, it sends the main frames off to Clarifai which then responds which each images description/annotation. Based on each main frame image, display it and add its annotation randomly on the image.

Source Code for Part 1: https://github.com/GeoSnipes/Big-Data/tree/master/lab_assignments/Lab%202/part1/src/src/main/java

---

Objective:

The objective is to classify the images collected from videos or data set which is used above using the below classification algorithms and then compare and contrast their accuracy.

• Random Forest
• Decision Tree
• Naive Bayes

Features:

• Random Forest

This method is one of the most successful machine learning models for classification and regression. In order to reduce the overfitting limitation, it combines many decision trees. It also handles categorical fashion in a similar way the decision trees handle. It also supports multi class classification. It doesn't require feature scaling. It can also capture both linear and non-linear feature interactions.



• Decision Tree

Decision trees are widely used since they are easy to interpret, handle categorical features, extend to the multiclass classification setting, do not require feature scaling, and are able to capture non-linearities and feature interactions. Tree ensemble algorithms such as random forests and boosting are among the top performers for classification and regression tasks.



• Naive Bayes

Naive Bayes is a simple multiclass classification algorithm with the assumption of independence between every pair of features. Naive Bayes can be trained very efficiently. Within a single pass to the training data, it computes the conditional probability distribution of each feature given label, and then it applies Bayes’ theorem to compute the conditional probability distribution of label given an observation and use it for prediction.

---

Steps of Implementation:

• Decision Tree

The recursive tree construction is stopped at a node when one of the following conditions is met:

• The node depth is equal to the maxDepth training parameter.
• No split candidate leads to an information gain greater than minInfoGain.
• No split candidate produces child nodes which each have at least minInstancesPerNode training instances.

There are some problem specification parameters which are used to solve the data set. They are:

• algo: Type of decision tree
• numClasses: Number of Classes
• categoricalFeaturesInfo: specifies categorical features and number of categorical features each of the feature can take.

For example, Map(0 -> 2, 4 -> 10) specifies that feature 0 is binary (taking values 0 or 1) and that feature 4 has 10 categories (values {0, 1, ..., 9}). Note that feature indices are 0-based: features 0 and 4 are the 1st and 5th elements of an instance’s feature vector.

Code Snippet for Decision Tree:

Output for Decision Tree

---

---

• Random Forest
This model traines a set of decision trees separately in order to ensure a parallel training. This injects randomness into the training process so that each decision tree behaves different. This combining reduces the variance of the predictions which improves the performance on test data.

In training, it includes mainly two things

1. BootStrapping - subsampling original data set on each iteration to get different training set
2. Considering different random subsets of features to split on at each tree node

While coming to prediction, it aggregates all the decision trees and outputs the best decision which suits for the data set.

There are 2 main parameters which are very important in the random forest. They are numTrees and maxDepth. As the name indicates, numTrees is the count of decision trees and maxDepth denotes the maximum depth of all the decision trees taken into consideration.

Code Snippet for Random Forest:




Output for Random Forest

---

---

• Naive Bayes

This model is typically used for document classification. This just splits the data set into training and test data and then compare the training data set w.r.t. test data set. It takes an RDD of labelled point and an optional parameter lambda as input and we need to specify the model type parameter which is default taken as "multinomial".

Naive Bayes Code Snippet and Output:

Summary:

From the above table, we can summarize that Random Forest is the best classification method for image classification. It is obvious because, it selects the best decision tree. So, it has the best accuracy than the other two models. As expected, the naive bayes works good only for the text classification. So, it has the least accuracy than the rest of the models.

On the other way, though the Random Forest has the best accuracy among the models, when we see the value, it is very low i.e. 41% which is not at all a good value. So, we need to either select a data set which is more accurate in key frame detection or we need to change the split ratio for training and test data and observe the models which suits best for the data and where we get more accuracy value.

Screenshots for model changed to split ratio as 70-30 When we changed to 70-30 split ratio, the accuracy increased to almost double the value for random forest and it reached 98%





Source Code Link for Image Classification: https://github.com/GeoSnipes/Big-Data/tree/master/lab_assignments/Lab%202/part2/source