LAB4 - Hiresh12/UMKC GitHub Wiki

BIG DATA ANALYTICS AND APPLICATIONS – LAB 4

Hiresh Jakkala Bhaskar – 8

Anvesh Mandadi – 17

AIM :

  • To Develop a show and model using the dataset
  • I used images that has objects like table, door, kitchen, jacket
  • This dataset is used to train the model so that we can identify the indoor activities described in the image
  • This is help us in track the activities of children and pets in home
  • Model is trained using features from CNN and SIFT
  • To make the model to tell caption for the given test images
  • To compute the performance metrics of the model using BLEU.

Step Followed:

  • Pre-processing the dataset to generate lemma for the captions, splitting the dataset into train and test data.
  • Extract the features from the train images and captions
  • Train the model using CNN, SIFT and LSTM (features (CNN+SIFT)->relu->image embedding->dropout->LSTM)
  • Save the model
  • Generate captions for the test data and calculate the performance metrics of the model

Code Implementation:

  • Extracting features - CNN

  • SIFT Feature Extraction and store it in sift_feature.pkl file

  • Loading Captions and creating Vocabulary

  • Develop Model

  • Generate Captions for the test data

  • Computing performance of the model

Sample Test images:

Sample Captions:

Performance of the model (BLEU):