ICP 7 - AsharGit/Python-ICP GitHub Wiki

Welcome to the Python-ICP-7 wiki!

This ICP we use natural language processing

Video for this ICP: https://youtu.be/aEehtGmP2l0

The first task for this ICP is to replace the Naive Bayles classifier with SVM.

We can use the same code as NB and simply replace it with SVM to get our result and as shown below our accuracy improved.

We also modify tfidfVector to use bigram to get decreased accuracy score.

Finally the tfidfVector is modified again to use stop words which gives us our most accurate score yet

Code

Second task of this ICP is to use different types of processing on our sample text

We used Tokenization, POS tagging, Stemming, Lemmatization, Trigram and NER on text we scraped from Wikipedia and put into a text file.

The code is similar for all these classifications but the way they affect the text is different as shown below

Tokenization

Here we tokenize the stream of text into word and sentences and print using loops

Code

Output

Output Output

Part-of-speech tagging (POS)

Code

Output

Output

Stemming

Code

Output

Output

Lemmatization

Code

Output

Output

Trigram

Words are grouped in a set of 3 and printed using a loop

Code

Output

Output

Named-entity recognition (NER)

Code

Output

Output