a. Description about the ICP

In this ICP we are given 3 tasks

1. Find errors in code
2. Use embedding layers
3. Apply code to a different dataset We also have a bonus question that asks us to plot our accuracy and loss

b. Objective

The goal of this ICP is to get familiar with embedding layers and use them for word embedding

c. Design/ implementation(include screenshots with explanations)

We are given an initial source code to run which gives an error when run

We need to find 3 errors in the code in order to make it run correctly

The changes we make to the code to fix the mistakes are

1. Input dim is not defined before being used so we use the input value of 2000 to correct it

2. We only have an input layer and output layer, we need a middle hidden layer to complete our neural network

3. We change the activation from sigmoid which is good for 2 outputs to softmax which will work with our 3 possible outputs

The changes to our code lead to a running program

Our next task is to add embedding layers to our network

To use the embedding layer we have to first prepare the data

We analyze the words in the sentences to find the longest one

Then add padding to the rest of the sentences to make them all the same length for processing

Use embedding layer by providing the input variables and flatten to process in the neural network

The resulting output did not seem to show any improvement on the model without embedding layer

Our accuracy on the testing set remained around 50%

As a bonus question we were also asked to plot our accuracy and loss on the training and testing data

We use matplotlib to assign our variables and plot the result

The massive difference in our training and testing samples implies a problem with our training data

Our final task for the ICP was to use the same process on newsgroup data set

We replace our csv file with the imported newsgroup data set and reassign our variables

We retain most of the original code but change our output from 3 to 20 for the new data set

Our result from running our code shows an accuracy of 5% on both our training and testing data set

Such a low accuracy rate implies there are problems in our training data set

The accuracy and loss plot show poor training and testing results

d. Video link

https://youtu.be/2zUB9WWFRwU

e. Conclusion

i. What you have learned from the ICP

There were quite a few unexpected results in this ICP, our initial results produced a very low accuracy of 50% and adding an embedding layer did not seem to show any improvement to our results. Our final task in the ICP produced a very low accuracy of 5% so it is worth exploring if there are things that can be done to the code to improve the results. I did however learn how to prepare and use embedding layer which may to useful when working with categorical variables in the future.

ii. What the challenge you have faced

The major problem in this ICP was the neural network was time and resource consuming, compiling the code on my computer put a lot of stress on the system so I had to move my code to Google Colab where each iteration took 40-45 min each time the execution took place.

Another challenge I faced was I was unsure if what I was doing in my neural network was correct since my results were producing very low accuracy results.