# Notebook from persas/intro-to-nlp
Path: Thursdays brown bag meeting, NLP.ipynb

# **Thursdays Brown Bag Meeting**: Introduction to NLP

---

In this session we will:
1. Walkthrough the mathematical foundation of NLP-based Machine Learning
2. Understand better the fields of application, and the differences between NLP / NLU
3. A practical exercise on sentiment prediction

---_____no_output_____

import pandas as pd
from gensim.corpora import Dictionary
import spacy
from gensim.models import TfidfModel
# !python -m spacy download en_core_web_sm
nlp = spacy.load('en_core_web_sm')
import gensim.models as g
from sklearn.manifold import TSNE
import numpy as np
import matplotlib.pyplot as plt
pd.set_option('display.max_columns', 500)
_____no_output_____# ! pip install spicy
# ! pip install tqdm
from scipy.spatial.distance import cosine as cosine_distance
from tqdm.auto import tqdm
import io_____no_output_____# !pip install emoji
# !pip install sklearn
import emoji
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split_____no_output_____# ! mkdir ./doc2vec/
# ! aws s3 cp s3://clarity-data-science-team/controversies/pretrained_doc2vec/wikipedia/doc2vec.bin ./doc2vec
# ! aws s3 cp s3://clarity-data-science-team/controversies/pretrained_doc2vec/wikipedia/doc2vec.bin.syn0.npy ./doc2vec
# ! aws s3 cp s3://clarity-data-science-team/controversies/pretrained_doc2vec/wikipedia/doc2vec.bin.syn1neg.npy ./doc2vec
download: s3://clarity-data-science-team/controversies/pretrained_doc2vec/wikipedia/doc2vec.bin to ./doc2vec.bin
download: s3://clarity-data-science-team/controversies/pretrained_doc2vec/wikipedia/doc2vec.bin.syn0.npy to ./doc2vec.bin.syn0.npy
download: s3://clarity-data-science-team/controversies/pretrained_doc2vec/wikipedia/doc2vec.bin.syn1neg.npy to ./doc2vec.bin.syn1neg.npy
# aws s3 sync s3://clarity-data-science-team/controversies/pretrained_doc2vec/wikipedia/ ./doc2vec_____no_output_____! mkdir ./models
! aws s3 cp s3://clarity-data-science-team/controversies/pretrained_word2vec/wikipedia/wiki-news-300d-1M.vec ./models/
download: s3://clarity-data-science-team/controversies/pretrained_word2vec/wikipedia/wiki-news-300d-1M.vec to models/wiki-news-300d-1M.vec

## 1. Motivation and relevant use cases

### 1.1. Motivation and challenges

- Natural Language Processing or **NLP is "*a field of artificial intelligence that gives the machines the ability to read, understand and derive meaning from human languages*"**

- **Language carries information that humans can naturally unpack**. We take for granted the complexity inherent to language, resulting from elongated periods of evolution, most of the times bypassing languague amibiguities:

    - Example 1: *"John kissed his wife, and so did Sam"*

    - Example 2: The buffalo sentence [buffalo sentence](https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo)

![Buffalo](images/buffalo.PNG)

- From a conceptual perspective and under the assumption that we can disambiguate the language, **text is virtually the most complete source of information**. Where do we find important applications?
---

### 1.2. Relevant use cases

Several use cases have been identified across the industry for the use of natural language as a source of data for prediction purposes. Some [notable ones](https://medium.com/@ODSC/an-introduction-to-natural-language-processing-nlp-8e476d9f5f59):

- **Detection of spam email**: Probably the first succesful application of NLP in the industry
- **Recognition and prediction of diseases**: Using information such as health records or patient's own speech, conditions such as cardiovascular diseases or schizophrenia can be potentially diagnosed, with [specific services](https://aws.amazon.com/comprehend/medical/) around this topic (*)
- **Brand sentiment and customer satisfaction**: Social media data can be used for sentiment analysis, understanding the degree of customer satisfaction, brand perception and main improvement areas for businesses
- **Recommendation systems and personal assistants**: Amazon Alexa's and Google's Nest are two powerful examples.
- **Identification of fake news**: the NLP Group at MIT developed a new system to determine if a source is accurate or politically biased, detecting if a news source can be trusted or not

Or eveeeen...

- **Detection of controversies from raw text powering Clarity controversy scores!**
---
---

## 2. Mathematical foundation and feature generation

We know how to attempt the prediction of numbers with numbers (regression, classification). But, **how do we attempt the prediction of numbers with text?**

In this case the simplest answer is the right one: **convert text into numbers!**. Let's start with the simple exercise below:

![sentence](images/sentence.PNG)
_____no_output_____

sentence = nlp('The big cat is drinking milk')
sentence_____no_output_____

Pretrained English model from gensim lets us identify the elements in the sentence_____no_output_____

print('Our tokenized sentence:', [token for token in sentence])
print('\nSyntactic analysis:', [token.pos_ for token in sentence])Our tokenized sentence: [The, big, cat, is, drinking, milk]

Syntactic analysis: ['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'NOUN']

### 2.1. First approach: Bag of Word counts

We can extract features by simply counting the number of appearances of each word in our vocabulary in the studied sentence. _____no_output_____

# Your corpus (set of texts) will define your vocabulary
universe_of_sentences = ["The small cat will never drink milk, and it only eats its catnip",
                         'The dog only eats soft food, and it loves drinking cold water water',
                         'The big cat loves raw meat and drinks very little water in the day',
                         'The animals in the farm are fed all the same, including what they drink',
                         "A cow is pacing grass, which has enough water on it so it doesn't really drink during the day",
                         'The ducklings are being fed by the mom',

                         'The knight is battling in the battlefield, wearing its shinny armor',
                         'The castle is being sieged and the enemies will not stop until conquering it',
                         'The empire is being attacked by the barbarians',
                         'The blacksmith is forging axes for the incoming battle']

# Create the vocabulary object and obtain the bag of word representation
dct = Dictionary([x.split() for x in universe_of_sentences])
record = {dct[k]: 0 for k,v in dct.items()}
record.update({dct[k]: v for k,v in dct.doc2bow(universe_of_sentences[0].split())})
pd.DataFrame([record])
_____no_output_____

Conceptually, the above is sufficient to provide a mathematical representation of our sentence, where we have variables (each word) based on which we could try and predict something.

However, several issues arise:
- Variable values would increase for longer texts, so we could consider normalizing to the total length of the text
- Several words provide no real meaning in the context of a word count. For example, the word "the" is common across any type of sentence and we cannot expect it to provide valuable information. This is what we call **stop words**, which should be removed  as a part of the **tokenization** process
- Several words present similar forms, all with the same lemma. For simplicity and generalization, we should either apply [**lematization** / **stemming**](https://towardsdatascience.com/stemming-lemmatization-what-ba782b7c0bd8).

### 2.2. Second approach, TF-IDF scores

Around the concept of bag of words, we can build **more powerful mathematical artifacts**. These include among others [TF-IDF scores](https://en.wikipedia.org/wiki/Tf%E2%80%93idf). These scores provide a view of how relevant a word is in a document in relation to its relevancy across the corpus, increasing proportionally to the number of times a word appears in the document and offset by the number of documents in the corpus that contain the word._____no_output_____

stopwords = spacy.lang.en.stop_words.STOP_WORDS
print('Some example stopwords:', list(stopwords)[20:25])

# Modify the universe of sentences by removing stopwords and lemmatizing
universe_of_sentences_mod = [[token.lemma_.lower() for token in nlp(sentence)
                              if token not in stopwords]
                             for sentence in universe_of_sentences]

print('\n3rd sentence, transformed:', universe_of_sentences_mod[3])

# Train the TFIDF model over our corpus
TFIDF_model = TfidfModel([dct.doc2bow(sentence) for sentence in universe_of_sentences_mod])

# Get the representation for the first sentence in the universe
record = {dct[k]: 0 for k,v in dct.items()}
record.update({dct[k]:v for k,v in TFIDF_model[dct.doc2bow(universe_of_sentences_mod[0])]})
pd.DataFrame([record])

Some example stopwords: ['behind', 'alone', 'afterwards', 'also', 'by']

3rd sentence, transformed: ['the', 'animal', 'in', 'the', 'farm', 'be', 'feed', 'all', 'the', 'same', ',', 'include', 'what', '-pron-', 'drink']

While we now have a more powerful representation of the sentence, **we are still facing the main issue in bag of words approaches: we have eliminated the syntactical representation provided by the sentence.** In this context *"the door is behing the table"* and *"the table is behind the door"* would have **the same** representation.

### 2.3. Introduction to document embeddings

More advance approaches start making use of **language models** that learn **representation of the words / sentences in alternative feature spaces**, capturing relevant text features.

Some of the most relevant **word representations are**:
- [GloVe](https://nlp.stanford.edu/projects/glove/)
- [Word2Vec](https://towardsdatascience.com/introduction-to-word-embedding-and-word2vec-652d0c2060fa)

Famously, Word2Vec embeddings are depicted as presenting the same ("*king*" - *"queen"*) vector distance as the ("*man*" - *"woman"*) vector distance, which illustrates the ability of the algorithm of representing words in relevant syntactical spaces.

![sentence](images/king_queen.PNG)

As an extension of word representations we find **sentence representations**, such as Doc2Vec.

### 2.4. Third approach, document embeddings

Doc2Vec enables a vector representation in an n-dimensional space, where angular proximity between vectors is associated with syntactical similarity (refer [here](https://medium.com/wisio/a-gentle-introduction-to-doc2vec-db3e8c0cce5e) for a gentle introduction)
_____no_output_____

# Load pretrained wikipedia-trained Doc2Vec model
pd.options.display.max_colwidth = 100
Doc2Vec_model = g.Doc2Vec.load('doc2vec/doc2vec.bin')

# Get the vector representation of our sentences
embeddings = pd.DataFrame([Doc2Vec_model.infer_vector(sentence.split(), alpha=0.05, steps=1) for sentence in universe_of_sentences],
                          columns=['Doc2Vec_{}'.format(x) for x in range(300)])
embeddings['sentence'] = universe_of_sentences
embeddings = embeddings[['sentence']+list(embeddings.columns[:-1])]
embeddings.loc[0]
_____no_output_____

For each sentence we have a 300-dimensional vector... but do they really capture syntactical similarity? We can use the t-SNE embedding algorithm to embedd the vectors in a 2-dimensional space._____no_output_____

# Embedd in a 2-dimensional space
TSNE_model = TSNE(perplexity=1.5)
TSNE_embeddings = pd.DataFrame(TSNE_model.fit_transform(X=np.array(embeddings.drop('sentence', axis=1))))
TSNE_embeddings['sentence'] = universe_of_sentences

fig, axes = plt.subplots(1,1, figsize=(10, 10))
plt.scatter(TSNE_embeddings[0], TSNE_embeddings[1])
for i, row in TSNE_embeddings.iterrows(): axes.annotate(row['sentence'], xy=(row[0], row[1]), xycoords='data')
plt.title('T-SNE embedding of the Doc2Vec features')
plt.xlabel('X coordinate')
plt.ylabel('Y coordinate')
plt.grid()
_____no_output_____

We can see how the "epic-themed" sentences are grouped together. Same happens for the animals that "don't drink", and the same happens for the rest of the animals that have let's say, a regular diet.

### 2.5. The great advance: LSTM cells  and transformers

While Doc2Vec makes use of Neural Networks as part of its components (see [CBOW](https://www.kdnuggets.com/2018/04/implementing-deep-learning-methods-feature-engineering-text-data-cbow.html) and [Skip-gram](https://www.kdnuggets.com/2018/04/implementing-deep-learning-methods-feature-engineering-text-data-skip-gram.html) models), **most relevant advances in NLP come hand by hand with the development of advanced deep network architectures leveraging cell memory mechanisms such as the LSTM cells**.

#### CONTEXT IS IMPORTANT: RNN's

Text can be seen as time series data in the sense that all previous words are relevant to understand further contents, since they create context. While the RNN architecture (recurrent neural networks) already enable learning from patterns repeated over time (time-series data) by introducing *hidden states* as representation of **previous inputs** they suffer from specific issues relating to the computation of gradients (check out [this article](https://towardsdatascience.com/illustrated-guide-to-recurrent-neural-networks-79e5eb8049c9) for a very nice and easy introduction to RNN)

#### LSTM

[LSTM](https://medium.com/mlreview/understanding-lstm-and-its-diagrams-37e2f46f1714) (or *long short term memory*) is a network architecture that enables a model to capture both recent, syntatically-relevant information as well as previous, topic-relevant information in a balanced fashion. Below is depicted the architecture of an LSTM cell

![lstm cell](images/lstm_cell.PNG)

What is relevant from here is that each cell will have two outputs: the short-term memory (cell specific output) and the long-term memory (which is passed from cell to cell and altered by each cell with their own contribution).

This development has proven massive in order to increase model comprehension ability as part of a wave of advanced NLP models known as transformers.

#### TRANSFORMERS AND RELATED OPEN SOURCE PROJECTS

The biggest NLP players are currently in a race to develop pretrained model architectures able to generalize to the widest range of tasks (measured through comprehensive benchmarks such as [GLUE](https://gluebenchmark.com/)), with relevant advance's such as Google's [BERT](https://arxiv.org/abs/1906.08237) or [XLNet](https://arxiv.org/abs/1906.08237). This, along with the democratization of contents promoted by the open source NLP community (such as the [Hugging Face project](https://github.com/huggingface/transformers)) is making it affordable to adapt these models to pretty much any NLP task through widely supported API's such as PyTorch or TensorFlow.

### 3. Practical example, sentiment analysis for reviews
_____no_output_____- Word vectors to play with here: https://fasttext.cc/docs/en/english-vectors.html
- Dataset for trainign sentimental analysis here: https://www.kaggle.com/kazanova/sentiment140

_____no_output_____

# Pretrained models from: T. Mikolov, E. Grave, P. Bojanowski, C. Puhrsch, A. Joulin. Advances in Pre-Training Distributed Word Representations
def load_vectors(fname):
    fin = io.open(fname, 'r', encoding='utf-8', newline='\n', errors='ignore')
    n, d = map(int, fin.readline().split())
    data = {}
    for line in tqdm(fin):
        tokens = line.rstrip().split(' ')
        data[tokens[0]] = np.array([ float(x) for x in tokens[1:] ])
    return data_____no_output_____fasttext = load_vectors("./models/wiki-news-300d-1M.vec")_____no_output_____def analogy(a, b, c, fast_vectors):

    # Words to Vec
    emb_a, emb_b, emb_c = fast_vectors[a.lower()], fast_vectors[b.lower()], fast_vectors[c.lower()]

    words = fast_vectors.keys()
    max_cosine_sim = -999
    best_word = None

    # loop over the universe
    for w in tqdm(words):

        # skip the input words
        if w in [a.lower(), b.lower(), c.lower()] :
            continue

        # Cosine similarity = 1 - cosine distance
        cosine_sim = 1 - cosine_distance(emb_b-emb_a, fast_vectors[w]-emb_c)

        if cosine_sim > max_cosine_sim:
            max_cosine_sim = cosine_sim
            best_word = w
    return best_word_____no_output_____analogy_words = ['queen', 'king', 'woman']
print ('{} -> {} // {} -> {}'.format( *analogy_words, analogy(*analogy_words,fasttext)))_____no_output_____analogy_words = ['spain', 'spanish', 'italy']
print ('{} -> {} // {} -> {}'.format( *analogy_words, analogy(*analogy_words,fasttext)))_____no_output_____analogy_words = ['man', 'woman', 'boy']
print ('{} -> {} // {} -> {}'.format( *analogy_words, analogy(*analogy_words,fasttext)))_____no_output_____

### 3.1 Sentiment Analysis_____no_output_____First we load the emoji package, so we can identify the sentiment of a tweet by an emoji_____no_output_____

emojify = {
    0: ':face_with_symbols_on_mouth:',
    4: ':smiling_face_with_smiling_eyes:'
}_____no_output_____# !mkdir ./data
# !aws s3 cp s3://clarity-data-science-team/controversies/sampling_tweets.csv ./data/training.1600000.processed.noemoticon.csvdownload: s3://clarity-data-science-team/controversies/sampling_tweets.csv to data/training.1600000.processed.noemoticon.csv
# Load tweets dataset if downloaded from s3
train_df = pd.read_csv('./data/training.1600000.processed.noemoticon.csv', encoding = "ISO-8859-1", error_bad_lines=False)
_____no_output_____# Load tweets dataset if downloaded from kaggle
tweets = pd.read_csv('./data/training.1600000.processed.noemoticon.csv', encoding = "ISO-8859-1", error_bad_lines=False, header=None)[[0, 5]]
tweets.columns = ['target', 'tweet']

positive_t = tweets[tweets.target == 4]
negative_t = tweets[tweets.target == 0]

# This type of sample selection could bias the modeling samlpe
train_df = pd.concat([positive_t.head(3000), negative_t.head(3000)])
train_df = train_df.sample(train_df.shape[0], random_state=8)
train_df.sample(10)
_____no_output_____# We generate the universe of sentences from a random sample of the dataset:
universe_of_sentences = train_df.tweet.values_____no_output_____

## BoW_____no_output_____In this cell we create the vocabulary_____no_output_____

# Create the vocabulary object and obtain the bag of word representation
all_sentences = []
for sentence in tqdm(universe_of_sentences):
    w_sentence = [ token.text for token in nlp(sentence) ]
    all_sentences.append(w_sentence)
dct = Dictionary(all_sentences)_____no_output_____

Create bag of words (wordcount of each sentence)_____no_output_____

records = []
record = dict.fromkeys(list(dct.values()), 0)
for sentence in tqdm(universe_of_sentences):
    record_1 = record.copy()
    # We are deleting punctuation signs, but for this specific case it may be useful.
    record_1.update({dct[k]: v for k,v in dct.doc2bow([token.text for token in nlp(sentence)])})
    records.append(record_1)
bow_df = pd.DataFrame(records)
_____no_output_____

Creation of target labels_____no_output_____

bow_df['__target__'] = train_df.target.values_____no_output_____

Fit and score the classifier_____no_output_____

clf = RandomForestClassifier(max_depth=2, n_estimators=100, random_state=24)
X = bow_df[bow_df.columns[:-1]].values
y = bow_df.__target__.values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=8)
clf.fit(X_train, y_train)
print(f"Accuray of random forest classifier: {round(clf.score(X_test, y_test)*100, 2)}%")Accuray of random forest classifier: 64.47%

Play with the results_____no_output_____

to_predict = input("Introduce a tweet: ")
record_1 = record.copy()
record_1.update({dct[k]: v for k,v in dct.doc2bow([token.text for token in nlp(to_predict)])})
res = clf.predict(np.array(list(record_1.values())).reshape(1, -1))[0]
print(emoji.emojize(f"Your tweet was: {emojify[res]}"))Introduce a tweet: I'm super angry!! You are the worst!
Your tweet was: 🤬

### 3.2 TF-IDF_____no_output_____As we are modeling sentiment analysis for tweets we consider the lemmatization to heavy, so we are modeling on plain text. In a different context it would be useful to use lemmatization_____no_output_____

stopwords = spacy.lang.en.stop_words.STOP_WORDS
print('Some example stopwords:', list(stopwords)[20:25])

# Modify the universe of sentences by removing stopwords and lemmatizing
universe_of_sentences_mod = [[token.text for token in nlp(sentence)
                              if token not in stopwords]
                             for sentence in universe_of_sentences]

# Train the TFIDF model over our corpus
TFIDF_model = TfidfModel([dct.doc2bow(sentence) for sentence in universe_of_sentences_mod])

# Get the representation for the first sentence in the universe
record = {dct[k]: 0 for k,v in dct.items()}
record.update({dct[k]:v for k,v in TFIDF_model[dct.doc2bow(universe_of_sentences_mod[0])]})
pd.DataFrame([record])Some example stopwords: ['behind', 'alone', 'afterwards', 'also', 'by']
records = []
record = dict.fromkeys(list(dct.values()), 0)
for sentence in tqdm(universe_of_sentences_mod):
    record_1 = record.copy()
    record_1.update({dct[k]:v for k,v in TFIDF_model[dct.doc2bow(sentence)]})
    records.append(record_1)
tfidf_df = pd.DataFrame(records)_____no_output_____tfidf_df['__target__'] = train_df.target.values_____no_output_____clf_tf = RandomForestClassifier(max_depth=2, n_estimators=100, random_state=24)
X = tfidf_df[tfidf_df.columns[:-1]].values
y = tfidf_df.__target__.values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=8)
clf_tf.fit(X_train, y_train)
print(f"Accuray of random forest classifier: {round(clf_tf.score(X_test, y_test)*100, 2)}%")Accuray of random forest classifier: 67.07%
to_predict = input("Introduce a tweet: ")
to_predict = [token.text for token in nlp(to_predict) if token not in stopwords]
record_1 = record.copy()
record_1.update({dct[k]:v for k,v in TFIDF_model[dct.doc2bow(to_predict)]})
res = clf_tf.predict(np.array(list(record_1.values())).reshape(1, -1))[0]
print(emoji.emojize(f"Your tweet was: {emojify[res]}"))Introduce a tweet: I'm super happy!!! This is a wanderful day
Your tweet was: 😊

Doc2Vec Embeddings_____no_output_____

# Get the vector representation of our sentences
embeddings = pd.DataFrame([Doc2Vec_model.infer_vector([token.text for token in nlp(sentence) if token not in stopwords],
                                                      alpha=0.05, steps=1) for sentence in tqdm(universe_of_sentences)],
                          columns=['Doc2Vec_{}'.format(x) for x in range(300)])

embeddings['sentence'] = universe_of_sentences
embeddings = embeddings[['sentence']+list(embeddings.columns[:-1])]
embeddings.loc[0]
_____no_output_____doc_df = pd.concat([embeddings, tfidf_df[tfidf_df.columns[:-1]]], axis=1)_____no_output_____doc_df = doc_df[doc_df.columns[1:]]
# doc_df = embeddings[embeddings.columns[1:]]
doc_df['__target__'] = train_df.target.values_____no_output_____clf_doc = RandomForestClassifier(max_depth=2, n_estimators=100, random_state=24)
X = doc_df[doc_df.columns[:-1]].values
y = doc_df.__target__.values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=8)
clf_doc.fit(X_train, y_train)
print(f"Accuray of random forest classifier: {round(clf_doc.score(X_test, y_test)*100, 2)}%")Accuray of random forest classifier: 62.8%
to_predict = input("Introduce a tweet: ")
to_predict = [token.text for token in nlp(to_predict) if token not in stopwords]
tp = Doc2Vec_model.infer_vector(to_predict)

record_1 = record.copy()
record_1.update({dct[k]:v for k,v in TFIDF_model[dct.doc2bow(to_predict)]})
tp = np.array(list(tp) + list(record_1.values()))

res = clf_doc.predict(tp.reshape(1, -1))[0]

print(emoji.emojize(f"Your tweet was: {emojify[res]}"))Introduce a tweet: This is awful mate, I'm so sad
Your tweet was: 🤬