# Notebook from tanmay-kulkarni/amazon_food_reviews
Path: t_SNE_on_Amazon_reviews.ipynb

# Objective

To reduce the number of dimensions of the Amazon food reviews dataset so that we can visualize it using a scatterplot. The plot should ideally distinguish between positive and negative reviews.

I'm not building a classification model here. The goal is to apply t-SNE on various vector representation of the text data. I will use these models to vectorize text:

*   Bag of Words (Unigram and Bigram)
* Tfidf
* Doc2Vec
* Average W2V
* Tfidf weighted W2V

_____no_output_____As always, let's start by loading the files that we need. I'm working on Google Colab for this Exercise, since my laptop isn't powerful enough to handle the workload.

To avoid having to upload data from local disk everytime the environment is disconnected, I'll be using Google Drive to store all the data and pickled models._____no_output_____

from google.colab import drive
drive.mount('/gdrive')_____no_output_____

List files on the drive._____no_output_____

# # List all files present on google drive
# import os
# os.listdir('/gdrive/My Drive')_____no_output_____

Import needed libraries_____no_output_____

%matplotlib inline

import sqlite3
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sbn
import nltk
from sklearn.feature_extraction.text import TfidfTransformer, TfidfVectorizer, CountVectorizer
from sklearn.metrics import confusion_matrix, roc_curve, auc
from nltk.stem.porter import PorterStemmer_____no_output_____

The raw data is present in the form of a sqlite file. Let's retrieve it from google drive._____no_output_____

# load sqlite database
con = sqlite3.connect(r'/gdrive/My Drive/amazon/database.sqlite')_____no_output_____

Since the aim here is to just visualize the positive and negative reviews, I select only the reviews that aren't neutral. It's a fair assumption that the reviews with score = 3 are neutral. We'll work with reviews that have score either 1,2,4 or 5._____no_output_____

df = pd.read_sql_query("select * from reviews where score <> 3;", con)_____no_output_____

There are about half a million reviews in the original data file that are not neautral._____no_output_____

df.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 525814 entries, 0 to 525813
Data columns (total 10 columns):
Id                        525814 non-null int64
ProductId                 525814 non-null object
UserId                    525814 non-null object
ProfileName               525814 non-null object
HelpfulnessNumerator      525814 non-null int64
HelpfulnessDenominator    525814 non-null int64
Score                     525814 non-null int64
Time                      525814 non-null int64
Summary                   525814 non-null object
Text                      525814 non-null object
dtypes: int64(5), object(5)
memory usage: 40.1+ MB

For this exercise, we want score to be a categorical feature.
* Mark reviews with rating > 3 as positive
* Mark reviews with rating < 3 as negative_____no_output_____

m1 =  df['Score'] > 3
m2 =  df['Score'] < 3

df['Score'] = np.select([m1,m2], ['positive','negative'])_____no_output_____

This is an imbalanced dataset. The number of positice reviews is almost 6 times the number of negative reviews!_____no_output_____

df.Score.value_counts()_____no_output_____

Let's change the datatype of Score in our Pandas dataframe to 'Category'. Using Categories instead of the default 'Object' datatype leads to performance improvement._____no_output_____

df['Score']=df['Score'].astype('category')
df.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 525814 entries, 0 to 525813
Data columns (total 10 columns):
Id                        525814 non-null int64
ProductId                 525814 non-null object
UserId                    525814 non-null object
ProfileName               525814 non-null object
HelpfulnessNumerator      525814 non-null int64
HelpfulnessDenominator    525814 non-null int64
Score                     525814 non-null category
Time                      525814 non-null int64
Summary                   525814 non-null object
Text                      525814 non-null object
dtypes: category(1), int64(4), object(5)
memory usage: 36.6+ MB
df.head()_____no_output_____

Are there any duplicate rows in the dataset?
Inspecting the "Text" column, we clearly see there are duplicates._____no_output_____

df.duplicated('Text').value_counts()_____no_output_____

There are also a few anomalies in the data where the HelpfulnessNumerator is greater than the helpfulnessDenominator.

* HelpfulnessNumberator = Number of positive reviews
* HelpfulnessDenominator = Number of positive reviews + Number of negative reviews

Therefore, HelpfulnessDenominator can't be less than HelpfulnessNumerator.  We need to get rid of such erroneous records._____no_output_____

df[df['HelpfulnessNumerator'] > df['HelpfulnessDenominator']]_____no_output_____

Let's drop these data points using the drop() method given by pandas._____no_output_____

df.drop(df[df['HelpfulnessNumerator'] > df['HelpfulnessDenominator']].index.tolist(), axis=0, inplace=True)_____no_output_____

Verify whether the rows have been dropped._____no_output_____

df [df['HelpfulnessNumerator'] > df['HelpfulnessDenominator']]_____no_output_____df.info()<class 'pandas.core.frame.DataFrame'>
Int64Index: 525812 entries, 0 to 525813
Data columns (total 10 columns):
Id                        525812 non-null int64
ProductId                 525812 non-null object
UserId                    525812 non-null object
ProfileName               525812 non-null object
HelpfulnessNumerator      525812 non-null int64
HelpfulnessDenominator    525812 non-null int64
Score                     525812 non-null category
Time                      525812 non-null int64
Summary                   525812 non-null object
Text                      525812 non-null object
dtypes: category(1), int64(4), object(5)
memory usage: 40.6+ MB

Now, let's drop the reviews which have the same data for the attributes:
* UserId
* ProfileName
* Time
* Text

These are the reviews for same products, duplicated in the dataset because Amazon considers slight variations of the same product to be different products. i.e. A food item with red color would be different than the same item of green color. We need to drop these._____no_output_____

df.drop_duplicates(subset={"UserId","ProfileName","Time","Text"}, keep = 'first', inplace=True)_____no_output_____df.info()<class 'pandas.core.frame.DataFrame'>
Int64Index: 364171 entries, 0 to 525813
Data columns (total 10 columns):
Id                        364171 non-null int64
ProductId                 364171 non-null object
UserId                    364171 non-null object
ProfileName               364171 non-null object
HelpfulnessNumerator      364171 non-null int64
HelpfulnessDenominator    364171 non-null int64
Score                     364171 non-null category
Time                      364171 non-null int64
Summary                   364171 non-null object
Text                      364171 non-null object
dtypes: category(1), int64(4), object(5)
memory usage: 28.1+ MB

Dropping the duplicate significantly reduces the size of our dataframe._____no_output_____

df.shape_____no_output_____

### Text Cleaning_____no_output_____Data collected from the web often contains unrendered HTML tags in them. Let's see if our reviews have them. We are only interested in the 'Text' column, because this is the column that will help us decide if a review is positive or negative. _____no_output_____

import re

i=0

for review in df['Text'].values:
    if (len(re.findall('<.*>', review))):
            i+=1
            r=review_____no_output_____

How many reviews contain HTML tags?_____no_output_____

print('Number of reviews which contain HTML tags: {}'.format(i), end='\n\n------------------\n\n')
print('Sample review containing HTML tags: {}'.format(r))Number of reviews which contain HTML tags: 93153

------------------

Sample review containing HTML tags: Hoping there is no MSG in this, this tastes extremely good when i make it with toasted fish. I used to eat fried chicken and other chicken dishes when i was young and had no idea they tasted so good because of this spice mix. Now i know when i remember about the great aroma and lip smacking flavor this imparts to my fish.  This is also good in soups and gives you a kind of addiction that you will want to put this spice mix into everything you eat.
Would like to add this will make anyone eat any kind of meat or vegetable, in my case I prefer costly red salmon in a can but pink salmon is so cheaper and now with Five spice pink salmon toast feels as enjoyable as anything else i have eaten.

Let's import some text processing libraries we need. _____no_output_____

import re
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer
from nltk.stem.wordnet import WordNetLemmatizer_____no_output_____

Download the stopwords and wordnet if needed._____no_output_____

nltk.download('stopwords')_____no_output_____nltk.download('wordnet')_____no_output_____

Let's define a couple of handy functions to clean the data and the stemmer._____no_output_____

stop_words = set(stopwords.words('english'))
porter = PorterStemmer()
lemma = WordNetLemmatizer()

def clean_html(sentence, compiled_regex):
    cleaned_sentence = re.sub(compiled_regex, ' ', sentence)
    return cleaned_sentence

def clean_punctuation(sentence):
    cleaned_sentence = re.sub(r'[?|!|\'|"|#]', r'', sentence)
    cleaned_sentence = re.sub(r'[.|,|)|(|\|/]', r' ', cleaned_sentence)
    return cleaned_sentence_____no_output_____print(stop_words, end='\n\n-------------------------\n\n')
print('Stemmed form of "Goodness" is: {}'.format(porter.stem('Goodness')))
print('Lemmatized form of "Goodness" is: {}'.format(lemma.lemmatize('Goodness'))){'doesn', 'and', 'same', 'mustn', 'an', 'where', 'than', 'so', "wouldn't", 'we', 'who', 'am', 'there', 're', 'been', 'off', 'once', 'you', 'here', 'when', 'don', "you'd", 'them', "don't", 'what', 'did', 'doing', 'yourselves', 'haven', 'herself', 'while', 'hers', 'him', 'will', 'ma', 'didn', 'having', "didn't", 'are', "hasn't", "weren't", "mustn't", 'himself', 'was', 'these', "wasn't", 'some', 'theirs', 'me', 'yourself', 'can', 'of', 'into', 'from', 'for', "aren't", 'after', "isn't", 'through', 'most', "needn't", 'weren', 'have', 'not', 'no', 'both', 'because', 'wouldn', "shouldn't", 'down', 'yours', "she's", 'ain', 'my', 'by', 'each', 'wasn', 'your', 'being', 'up', 'during', 'other', 'they', 'then', 'ourselves', "you've", 'about', 'over', "should've", 'needn', 'do', "couldn't", 'won', 'out', 'to', 'those', 't', 'but', 'until', 'm', 'itself', 'under', 'shouldn', 'any', 'own', "hadn't", 'his', 'that', 'or', 'y', 'against', 'her', 'this', 'had', 'is', 'd', 'it', 'now', 'below', 'why', 'in', 'll', 'how', 'myself', 'the', 'whom', 'few', 'hasn', 'their', 'she', 'were', 'all', 'such', 'should', 'which', 'couldn', 'isn', 'a', 'nor', 'has', 'on', "mightn't", 'ours', 'at', 'just', 's', 'before', 'again', 'further', 'if', "you'll", "doesn't", 'shan', "that'll", 'with', 'between', 'above', 'its', 'aren', "shan't", "you're", 'does', 'too', "won't", 'themselves', 'hadn', 'mightn', 'very', 'he', "it's", 'our', 've', 'only', 'as', 'o', 'more', 'i', 'be', "haven't"}

-------------------------

Stemmed form of "Goodness" is: good
Lemmatized form of "Goodness" is: Goodness

Let's clean the reviews using the functions defined above._____no_output_____

#Code for implementing step-by-step the checks mentioned in the pre-processing phase
# this code takes a while to run as it needs to run on 500k sentences.
i=0
str1=' '
final_string=[]

all_positive_words=[] # store words from +ve reviews here
all_negative_words=[] # store words from -ve reviews here.

s=''

regex_html=re.compile('<.*?>')

for review in df['Text'].values:
    filtered_sentence=[]
    #print(sent);
    review=clean_html(review, regex_html) # remove HTMl tags
    for w in review.split():
        for cleaned_words in clean_punctuation(w).split():
            if((cleaned_words.isalpha()) & (len(cleaned_words)>2)):
                if(cleaned_words.lower() not in stop_words):
                    s = (porter.stem(cleaned_words.lower())).encode('utf8')
                    filtered_sentence.append(s)
                    if (df['Score'].values)[i] == 'positive':
                        all_positive_words.append(s) #list of all words used to describe positive reviews
                    if(df['Score'].values)[i] == 'negative':
                        all_negative_words.append(s) #list of all words used to describe negative reviews reviews
                else:
                    continue
            else:
                continue
    #print(filtered_sentence)
    str1 = b" ".join(filtered_sentence) #final string of cleaned words
    #print("***********************************************************************")

    final_string.append(str1)
    i+=1

df['cleaned_text']=final_string #adding a column of CleanedText which displays the data after pre-processing of the review _____no_output_____

The column "cleaned_text" contains the cleaned reviews._____no_output_____

df.head()_____no_output_____

Considering how long it takes for a modest computer to run the above code for cleaning data, it's a good idea to store it  on disk for future use. The entire dataframe along with the newly created column "cleaned_data" is stored._____no_output_____

# #Save final cleaned dataframe to the drive.
# conn = sqlite3.connect('/gdrive/My Drive/amazon/reviews_cleaned_final.sqlite')
# df.to_sql('Reviews', conn)
# conn.close()_____no_output_____

If the environment was disconnected, load the cleaned dataframe._____no_output_____

# conn = sqlite3.connect('/gdrive/My Drive/amazon/reviews_cleaned_final.sqlite')
# df = pd.read_sql('select * from Reviews;', conn, index_col='index')
# conn.close()
# df.head()_____no_output_____

### Bag of words

Let's vectorize the data using the simplest method first: BoW

There is class imbalance in our original dataset. The positive reviews are far more than the negative ones. Let's take 1500 positive and 1500 negative samples._____no_output_____

n = df['Score'] == 'negative'
p = df['Score'] == 'positive'
#df_n = df[df['Score']]
df_n = df[n][['cleaned_text','Score']][:1500]
df_p = df[p][['cleaned_text', 'Score']][:1500]

df_np = pd.concat([df_n, df_p])_____no_output_____

Initialize the CountVectorizer class which creates a Bag of Words representation. _____no_output_____

count_vec = CountVectorizer()
final_counts = count_vec.fit_transform(df_np['cleaned_text'].values)_____no_output_____

Type of the returned object._____no_output_____

print('The type of final_counts is {}'.format(type(final_counts)))
print('The shape of the matrix is {}'.format(final_counts.get_shape))The type of final_counts is <class 'scipy.sparse.csr.csr_matrix'>
The shape of the matrix is <bound method spmatrix.get_shape of <3000x7207 sparse matrix of type '<class 'numpy.int64'>'
	with 93620 stored elements in Compressed Sparse Row format>>

Convert the sparse matrix of Bag of Words model to a numpy array._____no_output_____

review_vector_3k = final_counts.toarray()_____no_output_____

Before applying t-SNE, it's necessary that we standardize our data. _____no_output_____

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(review_vector_3k)/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py:475: DataConversionWarning: Data with input dtype int64 was converted to float64 by StandardScaler.
  warnings.warn(msg, DataConversionWarning)

Let's see the shape of of standardized data of BoW representation. It contains 3000 rows for 3000 reviews and 7207 dimensions._____no_output_____

X_scaled.shape_____no_output_____

Now apply t-SNE_____no_output_____

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, random_state=7, perplexity=40, early_exaggeration = 18, learning_rate = 250, method='exact')

review_vector_3k_tsne = tsne.fit_transform(X_scaled)_____no_output_____

Let's see the shape of the array after reducing the dimensions with t-SNE. It should be 3000 x 2._____no_output_____

print(review_vector_3k_tsne.shape)
print(review_vector_3k_tsne)(3000, 2)
[[  2.217929   -16.504608  ]
 [-39.40018     -1.17793   ]
 [ 10.396667   -11.995523  ]
 ...
 [  2.3932097   -0.36999798]
 [  1.7725201   -1.1499448 ]
 [  3.6150823   -1.3117751 ]]

This model took an hour to run. It's a good idea to store it on disk for later use._____no_output_____

import pickle
pickle_file = open('/gdrive/My Drive/amazon/pickled_3k_reviews_tsne_bow.pkl', 'wb')
pickle.dump(review_vector_3k_tsne, pickle_file)
pickle_file.close()_____no_output_____

Construct a dataframe to help visualize the t-SNE result._____no_output_____

bow_tsne_3k_array = np.vstack((review_vector_3k_tsne.T, df_np['Score'])).T
df_bow_tsne_3k = pd.DataFrame(bow_tsne_3k_array, columns=['First Dimension', 'Second Dimension', 'Label'])_____no_output_____

The following dataframe contains the new dimensions that were created by t-SNE. The original dimensions are lost._____no_output_____

df_bow_tsne_3k.head()_____no_output_____

Display a scatter plot._____no_output_____

sbn.FacetGrid(df_bow_tsne_3k, hue="Label", size=6).map(plt.scatter, 'First Dimension', 'Second Dimension').add_legend()
plt.show()_____no_output_____

The above scatter plot doesn't make any sense. The positive and negative reviews are almost perfectly overlapped.

This suggests that the Bag of Words model isn't really good at distinguishing between the two types of reviews._____no_output_____### BoW Bigrams

This is an extension to the previous model. We consider bi-grams here instead of uni-grams.

First, let's create lists of positive and negative words._____no_output_____

all_positive_words = []

for w in df_p['cleaned_text']:
    all_positive_words.extend(w.split())
#all_positive_words

all_negative_words = []

for w in df_n['cleaned_text']:
    all_negative_words.extend(w.split())
#all_negative_words_____no_output_____

Most used words in positive and negative reviews. _____no_output_____

freq_dist_positive = nltk.FreqDist(all_positive_words)
print('Most common positive words: {}'.format(freq_dist_positive.most_common(10)))

freq_dist_negative = nltk.FreqDist(all_negative_words)
print('Most common negative words: {}'.format(freq_dist_negative.most_common(10)))Most common positive words: [(b'chip', 907), (b'flavor', 728), (b'like', 648), (b'love', 582), (b'great', 567), (b'tast', 552), (b'good', 549), (b'bag', 458), (b'use', 405), (b'one', 404)]
Most common negative words: [(b'tast', 881), (b'like', 847), (b'product', 718), (b'flavor', 551), (b'one', 517), (b'tri', 508), (b'would', 486), (b'use', 417), (b'coffe', 408), (b'good', 389)]
#  Bi-grams
count_vec = CountVectorizer(ngram_range=(1,2))
bigrams = count_vec.fit_transform(df_np['cleaned_text'].values)_____no_output_____

The number of bigrams is far more than the unigrams._____no_output_____

bigrams.shape_____no_output_____bigrams_____no_output_____

Convert the sparse matrix to a dense numpy matrix._____no_output_____

review_vector_3k = bigrams.toarray()_____no_output_____review_vector_3k.shape_____no_output_____

Standardize the data_____no_output_____

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()

X_scaled = scaler.fit_transform(review_vector_3k)/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py:475: DataConversionWarning: Data with input dtype int64 was converted to float64 by StandardScaler.
  warnings.warn(msg, DataConversionWarning)

At this point, memory is about to run out so let's store the variable on Google drive._____no_output_____

# import pickle
# pickle_file = open('/gdrive/My Drive/amazon/X_scaled_standardized_3k_bigram_bow_nparray.pkl', 'wb')
# pickle.dump(X_scaled, pickle_file)
# pickle_file.close()_____no_output_____

Load the pickled X_scaled file._____no_output_____

# import pickle
# file_path = open('/gdrive/My Drive/amazon/X_scaled_standardized_3k_bigram_bow_nparray.pkl', 'rb')
# X_scaled = pickle.load(file_path)_____no_output_____

Create a t-SNE model_____no_output_____

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, random_state=7, perplexity=40, early_exaggeration = 18, learning_rate = 250, method='exact')_____no_output_____review_vector_3k_tsne = tsne.fit_transform(X_scaled)_____no_output_____

Shape of the numpy array after reducing the dimensions to two. _____no_output_____

review_vector_3k_tsne.shape_____no_output_____

Pickle important variables again._____no_output_____

# # Pickle it first!
# import pickle
# pickle_file = open('/gdrive/My Drive/amazon/pickled_3k_reviews_tsne_bigram.pkl', 'wb')
# pickle.dump(review_vector_3k_tsne, pickle_file)
# pickle_file.close()_____no_output_____

Build dataframe_____no_output_____

bigram_tsne_3k_array = np.vstack((review_vector_3k_tsne.T, df_np['Score'])).T
df_bigram_tsne_3k = pd.DataFrame(bigram_tsne_3k_array, columns=['First Dimension', 'Second Dimension', 'Label'])_____no_output_____df_bigram_tsne_3k.head()_____no_output_____

Plot the result._____no_output_____

sbn.FacetGrid(df_bigram_tsne_3k, hue="Label", size=6).map(plt.scatter, 'First Dimension', 'Second Dimension').add_legend()
plt.show()_____no_output_____

Unfortunately, using bigrams also does not yield anything useful. Let's look at TFIDF next._____no_output_____### Tfidf

Term Frequency Inverse Document Frequency.

Let's define the model and fit it to the data._____no_output_____

tfidf_vec = TfidfVectorizer(ngram_range=(1,1))
tfidf = tfidf_vec.fit_transform(df_np['cleaned_text'].values)_____no_output_____tfidf.shape_____no_output_____

There are 3000 rows as expected and 7207 dimensions._____no_output_____

features = tfidf_vec.get_feature_names()
len(features)_____no_output_____

These are some dimensions in the tfidf representation._____no_output_____

features[100:110]_____no_output_____

Convert the tfidf matrix to a dense numpy array._____no_output_____

review_vector_3k_TFIDF=tfidf.toarray()
review_vector_3k_TFIDF.shape_____no_output_____

Standardize the tfidf data._____no_output_____

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()

X_scaled = scaler.fit_transform(review_vector_3k)_____no_output_____X_scaled.shape_____no_output_____

Apply t-SNE_____no_output_____

from sklearn.manifold import TSNE
tsne = TSNE(n_components=2, random_state=7, perplexity=45, early_exaggeration = 17, learning_rate = 300, method='exact')
review_vector_3k_tsne = tsne.fit_transform(X_scaled)_____no_output_____

As expected, the newly create data array has 2 dimensions and 3000 reviews_____no_output_____

review_vector_3k_tsne.shape_____no_output_____

Pickle the tsne object to Google Drive._____no_output_____

import pickle
pickle_file = open('/gdrive/My Drive/amazon/pickled_3k_reviews_tsne_tfidf_1gram.pkl', 'wb')
pickle.dump(review_vector_3k_tsne, pickle_file)
pickle_file.close()_____no_output_____

Construct dataframe_____no_output_____

tfidf_tsne_3k_array = np.vstack((review_vector_3k_tsne.T, df_np['Score'])).T
df_tfidf_tsne_3k = pd.DataFrame(tfidf_tsne_3k_array, columns=['First Dimension', 'Second Dimension', 'Label'])_____no_output_____df_tfidf_tsne_3k.head()_____no_output_____

Plot the result of applying t-SNE on tfidf._____no_output_____

sbn.FacetGrid(df_tfidf_tsne_3k, hue="Label", size=6).map(plt.scatter, 'First Dimension', 'Second Dimension').add_legend()
plt.show()_____no_output_____

The TFIDF model too didn't prove to be of much use here. Let's look at a different technique._____no_output_____### Doc2Vec_____no_output_____

# Install gensim if not installed

!pip install gensim_____no_output_____

Import gensim_____no_output_____

import gensim
from gensim.models import word2vec
from gensim.models import KeyedVectors_____no_output_____

Define a function that reads each review and converts it into TaggedDocument format needed for training a model using Doc2Vec technique._____no_output_____

#train_corpus = gensim.utils.simple_preprocess(df_np['cleaned_text'][1])

def read_corpus(df_np, tokens_only=False):
        for i, review in enumerate(df_np['cleaned_text']):
            if tokens_only:
                yield gensim.utils.simple_preprocess(review)
            else:
                # For training data, add tags
                yield gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(review), [i])

train_corpus = list(read_corpus(df_np))_____no_output_____train_corpus[:3]_____no_output_____

Define D2V model_____no_output_____

model = gensim.models.doc2vec.Doc2Vec(vector_size=50, min_count=2, epochs=45)_____no_output_____

Build vocabulary_____no_output_____

model.build_vocab(train_corpus)_____no_output_____model_____no_output_____

Train D2V model_____no_output_____

%time model.train(train_corpus, total_examples=model.corpus_count, epochs=model.epochs)CPU times: user 23.1 s, sys: 2.82 s, total: 26 s
Wall time: 14.9 s

Create a corpus of vector representation of all reviews under consideration, using D2V technique._____no_output_____

review_vector_3k = np.zeros((1,50))

for review in df_np['cleaned_text']:
    arr = np.reshape(model.infer_vector(str(review).split()), (1,-1))
    review_vector_3k=np.vstack([review_vector_3k, arr])

#model.infer_vector(str(df_np['cleaned_text'][1636]).split())
#df_np['cleaned_text'][1637]
#df_np['cleaned_text']
#np.reshape(model.infer_vector(str(df_np['cleaned_text'][2]).split()), (1,-1)).shape

review_vector_3k = np.delete(review_vector_3k, (0), axis=0)_____no_output_____

A vector is generated that has 3000 rows, each for 1 review and 50 dimensions as defined by the vector_size parameter specified when we defined the model above._____no_output_____

review_vector_3k.shape_____no_output_____

Standardize the data_____no_output_____

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()

review_vector_3k = scaler.fit_transform(review_vector_3k)_____no_output_____

Define t-SNE and transform data_____no_output_____

tsne = TSNE(n_components=2, random_state=7, perplexity=45, early_exaggeration = 13, learning_rate = 300, method='exact')

review_vector_3k_tsne = tsne.fit_transform(review_vector_3k)_____no_output_____

As expected, the resulting matrix has 2 dimensions which are created using t-SNE, from the original vector generated using Doc2Vec._____no_output_____

review_vector_3k_tsne.shape_____no_output_____

Generate a dataframe from it._____no_output_____

d2v_tsne_3k_array = np.vstack((review_vector_3k_tsne.T, df_np['Score'])).T
df_d2v_tsne_3k = pd.DataFrame(d2v_tsne_3k_array, columns=['First Dimension', 'Second Dimension', 'Label'])_____no_output_____df_d2v_tsne_3k.head()_____no_output_____

Plot the result._____no_output_____

sbn.FacetGrid(df_d2v_tsne_3k, hue="Label", size=6).map(plt.scatter, 'First Dimension', 'Second Dimension').add_legend()
plt.show()_____no_output_____

Not much here, either._____no_output_____### Word2Vec_____no_output_____

from gensim.models import Word2Vec
from gensim.models import KeyedVectors
import pickle_____no_output_____

Here, I'm training the model using our own corpus of food reviews. The reason behind not using Google's model trained on google news is - there are several words used in the food reviews that aren't present in the model trained by Google. Creating vectors of reviews using it throws exceptions. _____no_output_____

# Training a Word2Vec model using our own corpus

list_of_reviews = []

for review in df['cleaned_text'].values:
    list_of_reviews.append(review.decode('utf-8').split())_____no_output_____print(df['cleaned_text'][0])
print('------------------')
print(list_of_reviews[0])
b'bought sever vital can dog food product found good qualiti product look like stew process meat smell better labrador finicki appreci product better'
------------------
['bought', 'sever', 'vital', 'can', 'dog', 'food', 'product', 'found', 'good', 'qualiti', 'product', 'look', 'like', 'stew', 'process', 'meat', 'smell', 'better', 'labrador', 'finicki', 'appreci', 'product', 'better']
w2v_model = gensim.models.Word2Vec(list_of_reviews, min_count=4,size=50,workers=4)_____no_output_____words = list(w2v_model.wv.vocab)
print(len(words))24867

The following result is fascinating. It shows the words which are similar to 'smell'. The list of words printed is amazing._____no_output_____

w2v_model.wv.most_similar('smell')/usr/local/lib/python3.6/dist-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.
  if np.issubdtype(vec.dtype, np.int):

Similarly, words similar to 'bad'_____no_output_____

w2v_model.wv.most_similar('bad')/usr/local/lib/python3.6/dist-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.
  if np.issubdtype(vec.dtype, np.int):

Calculating the avg W2V representation for each review._____no_output_____

import sys

corpus_vec = np.zeros(shape=(50))

for review in df_np['cleaned_text'].values:
    review_vector = np.zeros(shape=(50))
    for word in review.decode('utf-8').split():
        try:

            review_vector += w2v_model.wv[word]

        except KeyError:
            continue

    review_vector /= 50
    corpus_vec=np.vstack((corpus_vec,review_vector))
corpus_vec=np.delete(corpus_vec, 0, axis=0)_____no_output_____corpus_vec.shape_____no_output_____

Again, store the files on Google Drive for future use._____no_output_____

import pickle
pickle_file = open('/gdrive/My Drive/amazon/avg_w2v_nparray_3kcorpus_50dim.pkl', 'wb')
pickle.dump(corpus_vec, pickle_file)
pickle_file.close()_____no_output_____corpus_vec.shape_____no_output_____

Appply t-SNE_____no_output_____

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, random_state=7, perplexity=45, early_exaggeration = 13, learning_rate = 250)

review_vector_3k_tsne = tsne.fit_transform(corpus_vec)_____no_output_____review_vector_3k_tsne.shape_____no_output_____

Build a dataframe for visualization_____no_output_____

avg_w2v_tsne_3k_array = np.vstack((review_vector_3k_tsne.T, df_np['Score'])).T
df_avg_w2v_tsne_3k = pd.DataFrame(avg_w2v_tsne_3k_array, columns=['First Dimension', 'Second Dimension', 'Label'])_____no_output_____sbn.FacetGrid(df_avg_w2v_tsne_3k, hue="Label", size=6).map(plt.scatter, 'First Dimension', 'Second Dimension').add_legend()
plt.show()_____no_output_____

The above result is far from ideal, but it's slightly better than the techniques seen so far. The density of positive points is more in the right side of the map as compared to the left side._____no_output_____### Tfidf weighted W2V

In this representation, each review is made up from the tfidf weighted sum of all the words in a review._____no_output_____

corpus_tfidf_weighted_w2v = np.zeros(shape=(50))
review_number = 0

for review in df_np['cleaned_text'].values:
    review_vector_tfidf_weighted = np.zeros(shape=(50))
    tfidf_sum = 0
    for word in review.decode('utf-8').split():
        try:
            tfidf_value = review_vector_3k_TFIDF[review_number, features.index(word)]
            review_vector_tfidf_weighted += w2v_model.wv[word] * tfidf_value
            tfidf_sum += tfidf_value

        except KeyError:
            continue

    review_number += 1

    review_vector_tfidf_weighted /= tfidf_sum
    corpus_tfidf_weighted_w2v=np.vstack((corpus_tfidf_weighted_w2v,review_vector_tfidf_weighted))

corpus_tfidf_weighted_w2v = np.delete(corpus_tfidf_weighted_w2v, 0, axis=0)_____no_output_____

Shape of the data matrix_____no_output_____

corpus_tfidf_weighted_w2v.shape_____no_output_____

Apply t-SNE _____no_output_____

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, random_state=13, perplexity=50, early_exaggeration = 14, learning_rate = 225)

review_vector_3k_tsne = tsne.fit_transform(corpus_tfidf_weighted_w2v)_____no_output_____

The dimensions of the array reduced to 2 as expected._____no_output_____

review_vector_3k_tsne.shape_____no_output_____

Build a dataframe._____no_output_____

avg_tfidf_weighted_w2v_tsne_3k_array = np.vstack((review_vector_3k_tsne.T, df_np['Score'])).T
df_avg_tfidf_weighted_w2v_tsne_3k = pd.DataFrame(avg_tfidf_weighted_w2v_tsne_3k_array, columns=['First Dimension', 'Second Dimension', 'Label'])_____no_output_____sbn.FacetGrid(df_avg_w2v_tsne_3k, hue="Label", size=6).map(plt.scatter, 'First Dimension', 'Second Dimension').add_legend()
plt.show()_____no_output_____

The tfidf weighted W2V plot is not very different from the plot obtained using the avg. W2V representation. _____no_output_____TFIDF values of words in a review_____no_output_____

review = 414

for word in df_np['cleaned_text'][review].decode('utf-8').split():

    print('{} : {}'.format(word, tfidf[review, features.index(word)]))chip : 0.0
right : 0.0
amount : 0.0
crunch : 0.0
enjoy : 0.0
tast : 0.11877744889822464
chip : 0.0
consist : 0.1376015524136066
fairli : 0.0
even : 0.0
distribut : 0.0
season : 0.0
like : 0.11664406503962119
size : 0.0
bag : 0.0
right : 0.0
amount : 0.0
chip : 0.0

## Conclusion

In this exercise, I achieved the following goals:

* Cleaning dirty data.
            The original food reviews dataset contains lots of duplicate entries and HTML tags.
* Vector representation of text using techniques such as
            Bag Of Words (unigram & bigram), Tfidf, Doc2Vec, Avg. W2V, Tfidf weighted W2V

* Applying t-SNE on text data, which are represented in the form of vectors to reduce dimensions from hundreds to just two.

* The cluster seperation may be improved by using larger amount of data. Due to limitations of hardware, I used just 3000 reviews in the dataset which is about 0.57% the size of the original dataset._____no_output_____


# Notebook from topseer/APL_Great
Path: fundamentals_2018.9/inference/overview.ipynb

# Overview

We start with a review of Bayes Rule and then a motivating story for the Bayesian approach to inference.

## Review of Bayes Rule

Let us return to our example application of Bayes Rule from a previous chapter in which we tried to see if it was going to rain on Sam's wedding. For review,

> Sam is getting married tomorrow in an outdoor ceremony in the desert. In recent years, it has only rained 5 days per year. Unfortunately, the meteorologist has predicted rain for tomorrow. Should Sam rent a tent for the ceremony?

> We can solve this problem using Bayes Rule which remember is:

> $$P(A|B) = \frac{P(B|A)P(A)}{P(B)}$$

> But instead what we want is:

> $$P(W|F) = \frac{P(F|W)P(W)}{P(F)}$$

> where $W$ is weather (rain or shine) and $F$ is forecast (rain or shine).

In Bayesian terms, the different values for weather, rain or shine, are *models*. We have one model that says "it rains" and we have another model that says "it shines".

> Remember that $P(W)$ in the numerator is our *prior* probability. What *is* our prior probability? Well, it only rains 5 days a year on average:

>| rain | shine |
|:----:|:-----:|
| 5/365 = 0.0137 | 360/365 = 0.9863 |

> I think this is what Sam had in mind when he planned his wedding.

The prior is how much we believe or how much we find credible each model. We find the "rain" model 1.37% credible and the "shine" model, 98.63% credible.

> But now he needs to take new evidence into account: a forecast of rain. The likelihood $P(F|W)$ is essentially the probability of the meteorologist being correct: given that it rained, what is the probability that it was forecast? Sam looks this up on the Internet.

>|  F  | rain | shine |
|:---:|:----:|:-----:|
|rain |  0.8 | 0.2   |
|shine | 0.2 | 0.8   |

> What does this mean? *Given* that it rained, there is an 80% chance there was a forecast of rain:

> $F(F=rain|W=rain) = 0.8$

This is the consistency of the data with each possible model. In Bayesian statistics, if you don't know something, you condition on it. This has the strange effect of telling you what you want to know. Here we know that there's a forecast for rain but we don't know if it will really rain so we condition on it and ask, what's the probability that we have a forecast for rain tomorrow given that it really rains tomorrow.

It turns out that this is an easier question to answer than the posterior itself.

> Because it *will* be confusing, we do not take shortcuts here. We will use the longhand notation, F=rain and W=rain, to distinguish the two events. Up above, we had Bayes Rule defined over entire random variables.

> What does it look like for the specific outcome we're interested in?

> $$P(W=rain|F=rain) = \frac{P(F=rain|W=rain)P(W=rain)}{P(F=rain)}$$

> We have everything we need except the denominator. We can use total probability for it, though:

> $P(F=rain) = P(F=rain|W=rain)P(W=rain) + P(F=rain|W=shine)P(W=shine)$

> $0.8 \times 0.0137 + 0.2 \times 0.9863 = 0.208$

> and now we have:

> $P(W=rain|F=rain) = \frac{0.8 \times 0.0137}{0.208} = 0.053$

The resulting posterior is the probability of the model (rain) given a forecast for rain tomorrow. Or, the probability of rain, given the data (F=rain) and the prior.

Here we only have two models, though. What if we have more?

Consider the case where the meteorologist doesn't predict rain or shine but says only that there's a 5%, 10%, 15%, etc., to 95% chance of rain tomorrow. The first thing to note is that this means we have 19 models. The other interesting thing is that the meteorologist's forecast is never right or wrong. If there's a 5% chance of rain and it rains...the forecast is right. If there is a 5% chance of rain and it doesn't...the forecast is still right. What we want now is something like the accuracy of the meteorologist. In those cases where the meteorologist forecasts a 5% chance of rain, ideally, we want it to only rain 5% of the time.

Even so, there's nothing that stops us from having prior beliefs about the credibility of P(5%), P(10%), P(15%), etc. And there's nothing that stops us from conditioning our data on those models so that we have P(rain|5%), P(rain|10%), P(rain|15%), etc.

What we end up with is something like P(rain|5%) or the consistency of the forecasts. That's ok, too.

What we really want to drive home is that we can have more than two models. In fact, we can have a *continuous* model of P(1%), P(2%), P(3%), if we want but the math gets very complicated. This actually stymied Bayesian techniques for several centuries...until the invention of the computer.

In order to make clear that Bayesian Inference is just Bayes Rule, we return to the well-trod example of a coin toss and place it in the context of the previous chapters._____no_output_____## Inference, Deduction, and Bayesian Inference

Inference begins to bring the previous chapters together and demonstrate that it is the fundamental problem of science and, hence, data science. In keeping with tradition, we're going to start with the coin example.

1. There is a process that gives rise to an observable event (data) which we call "flipping a coin". We can describe the elements of that process even if we can't create a simulation of the event or measure all the variables involved. One way to handle this is to make a qualitative model (Systems Theory, Causal Loop Diagram).
2. Because all we can observe about the process is the result, we model this process with a parameter $p$, the rate of success, and associate $p$ with the probability of heads. If we know $p$, the probability distribution over the events $\{heads, tails\}$, we can make all kinds of *deductions* about outcomes. One such deduction might be, what is the probability of 10 heads and 12 tails if $p = 0.42$? Note that deductions are always true given their assumptions. (Probability).
3. It turns out that there are many processes like our coin with only two outcomes and that we often ask similar questions of such processes (Mathematical Distributions). These questions might include "what is the probability of 10 successes and 5 failures?" (Bernoulli Distribution), "what is the probability of 10 successes in 15 attempts?" (Binomial Distribution), "how many trials do we need in order to see at least 5 successes?" (Geometric Distribution).
4. Finally, we have observed 6 heads and 4 tails, what is $p$?

This last question is a question of inference. It is intimately connected with the process of science in general as we are moving from data (specifics) to theories and models ($p$). Admittedly, $p$ is not the Theory of Gravity or Evolution, but it is still a model of the real world--our process of interest. And that's what data science is for the most part, building models about the *everyday*. After all, if you build models about biology, you're a biologist.

There are a lot of ways to think about what $p$ "really" is. If you have some familiarity with statistics, you'll have heard $p$ referred to as a *population* parameter or the true value and the question is couched as, how do we discover the population parameter from a *sample*, a finite set of observations that includes 6 heads and 4 tails?

Instead, I would like you to think of $p$ as just our model of the real world process and we just want to know how credible the possible values of $p$ are given the data we've seen. We want good models. So how do we come up with a good model for $p$, based on our data?

So we know that if we have a fair coin ($p=0.5$) we can use *deduction* to answer the question, what is the probability of seeing 6 heads and 4 tails in 10 tosses. This is just a joint probability distribution of 10 independent outcomes. The Binomial distribution will tell us that the probability of this event is 20.5%.

If, on the other hand, we'd like to know what the value of $p$ is when we observe 6 heads and 4 tails in 10 tosses, our task is not as easy. Why? Because many values of $p$ are consistent with 6 heads and 4 tails in 10 tosses. The following table shows the probability of that event for different values of $p$ in 0.1 increments:

| $p$ | probability |
|:--------:|------------:|
| 0.0      | 0.00%       |
| 0.1      | 0.01%       |
| 0.2      | 0.55%       |
| 0.3      | 3.67%       |
| 0.4      | 11.14%      |
| 0.5      | 20.50%       |
| 0.6      | 25.08%      |
| 0.7      | 20.01%      |
| 0.8      | 8.80%       |
| 0.9      | 1.11%       |
| 1.0      | 0.00%       |

so which $p$ is the "real" one? We don't know for certain. Actually, we **cannot** know for certain merely from data.

If we use each possible value of $p$ as a model of the actual process and we assume that all values were initially equally likely to be true (or we had no information that would lead us to favor one of the values--models--over another), then the table above shows us, based on the data, how much we should believe in each model, the one where $p=0.2$, the one where $p=0.3$, the one where $p=0.4$, etc. This probability distribution then becomes a model of our uncertainty over the possible models, $p$. We use probability to model probability.

Suppose now we have a question about the probability of throwing 3 heads in a row. What model do we pick? There are a variety of answers. We could pick the most probable model where $p=0.6$. Or we could use all of the models simultaneously, weighting each prediction by the probability of the $p$ used.

This is the essence of the Bayesian approach to *statistical* inference. It basically says, in the forward direction, we have a model (probability) and we deduce data (events) based on those probabilities. In the reverse or *inverse* direction, we have data and we infer probabilities over the various possible models.

In the forward direction, I have a **model** of $p$ and I deduce events given that model's probability distribution. In the reverse direction, I have events and infer a probability distribution over **models** of $p$.

It's probability all the way down. The main challenge is to see everything we calculate as a model._____no_output_____## Another View

So how do we do this *inverse probability* thing?

We start with Bayes Rule:

$P(\theta|D) = \frac{P(D|\theta)P(\theta)}{P(D)}$

where $\theta$ represents one or more (and sometimes many) "models" of our process and then $P(\theta|D)$ is our *posterior* distribution over $\theta$, our Bayesian model of our models. There's nothing actually mysterious about the approach. Sometimes we use the term *hypothesis* instead of model.

In the Elvis problem, we have a model of twins and two hypotheses: Elvis's twin was a fraternal twin and Elvis's twin was an identical twin. We started with prior probabilities for each of these hypotheses and then using data we adjusted our probabilities according to Bayes Rule.

In the M&M's problem, we had two models: the first bag was from 1994 and the second from 1996 or the first bag was from 1996 and the second was from 1994. We started out with a 50/50 chance for each model and when we were done, our probabilities for each model (or hypothesis) was updated.

And, as we just saw, in the Weather problem, we had two models: rain or shine. We had prior probabilities for these models, and then data (a forecast of rain). We then reasoned out the posterior probabilities of the models based on the data and our priors.

The main difference between what we did then and what we'll do now is that now our models are almost always over continuous numerical values. The coin's "success rate", $p$, can take on infinitely many values. Even if we limit the significance to two decimals, there are 101 possible values. More generally, we will be interested in making inferences not over just discrete, categorical events like which M&M bag is which but over a infinite number of possible means and standard deviations.

We're still going to use Bayes Rule, it's just a bit harder. There are several different ways to do this,

1. Grid Method
2. Exact Method
3. Monte Carlo Method
4. Bootstrap Methods

The Exact Method involves parameterizing mathematical functions that represent the likelihood and prior and multiplying them together exactly as described in Bayes Rule. The math often becomes intractable except for the simplest of problem because with continuous variables especially "counting" means "integral calculus". The integrals involved in $P(D|\theta)$, $P(D)$ and $P(\theta)$ do not often compose well. This was one of the great obstacles to the practical application of Bayesian statistical inference.

The Grid Method involves picking a variety of individual values of $\theta$ and doing the calculations of Bayes Rule "by hand" as it were. This approach can actually be pretty effective but the quality of your results depend on how many values of $\theta$ you're willing to do calculations for...the size of your grid. We will start here because it's easiest to explain and develops the main intuition about the process.

With the advent of the computer, both the Exact and Grid approaches are no longer the only way to perform these calculations. Why? With Markov Chain Monte Carlo methods we can use numerical approaches to estimate the integrals that give the Exact method such problems.

The Monte Carlo Method involves parameterizing mathematical functions as with the Exact method but instead of trying to find an analytical solution for the posterior distribution, we *simulate* events from these distributions and construct a posterior distribution for our parameter which we can analyze.

With a renewed interest in Bayesian estimation, research has deepened our understanding of statistical inference which lead to the development of Bootstrap Methods (both parameteric and non-parameteric) which use simulation and data themselves to estimate the posterior distribution directly. This result follows mostly from the Law of Large Numbers. This is the approach we will focus on in the remainder of the text because its easy. Nevertheless, it almost seems like magic.

We will discuss each of these methods in turn but in this course, we will rely mostly on the Bootstrap in later chapters. You will certainly want to know about the Monte Carlo method if you venture in Bayesian modeling or use actual "Big Data"._____no_output_____


# Notebook from bksahu/statsmodels
Path: examples/notebooks/robust_models_1.ipynb

# M-Estimators for Robust Linear Modeling_____no_output_____

%matplotlib inline

from __future__ import print_function
from statsmodels.compat import lmap
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt

import statsmodels.api as sm_____no_output_____

* An M-estimator minimizes the function

$$Q(e_i, \rho) = \sum_i~\rho \left (\frac{e_i}{s}\right )$$

where $\rho$ is a symmetric function of the residuals

* The effect of $\rho$ is to reduce the influence of outliers
* $s$ is an estimate of scale.
* The robust estimates $\hat{\beta}$ are computed by the iteratively re-weighted least squares algorithm_____no_output_____* We have several choices available for the weighting functions to be used_____no_output_____

norms = sm.robust.norms_____no_output_____def plot_weights(support, weights_func, xlabels, xticks):
    fig = plt.figure(figsize=(12,8))
    ax = fig.add_subplot(111)
    ax.plot(support, weights_func(support))
    ax.set_xticks(xticks)
    ax.set_xticklabels(xlabels, fontsize=16)
    ax.set_ylim(-.1, 1.1)
    return ax_____no_output_____

### Andrew's Wave_____no_output_____

help(norms.AndrewWave.weights)_____no_output_____a = 1.339
support = np.linspace(-np.pi*a, np.pi*a, 100)
andrew = norms.AndrewWave(a=a)
plot_weights(support, andrew.weights, ['$-\pi*a$', '0', '$\pi*a$'], [-np.pi*a, 0, np.pi*a]);_____no_output_____

### Hampel's 17A_____no_output_____

help(norms.Hampel.weights)_____no_output_____c = 8
support = np.linspace(-3*c, 3*c, 1000)
hampel = norms.Hampel(a=2., b=4., c=c)
plot_weights(support, hampel.weights, ['3*c', '0', '3*c'], [-3*c, 0, 3*c]);_____no_output_____

### Huber's t_____no_output_____

help(norms.HuberT.weights)_____no_output_____t = 1.345
support = np.linspace(-3*t, 3*t, 1000)
huber = norms.HuberT(t=t)
plot_weights(support, huber.weights, ['-3*t', '0', '3*t'], [-3*t, 0, 3*t]);_____no_output_____

### Least Squares_____no_output_____

help(norms.LeastSquares.weights)_____no_output_____support = np.linspace(-3, 3, 1000)
lst_sq = norms.LeastSquares()
plot_weights(support, lst_sq.weights, ['-3', '0', '3'], [-3, 0, 3]);_____no_output_____

### Ramsay's Ea_____no_output_____

help(norms.RamsayE.weights)_____no_output_____a = .3
support = np.linspace(-3*a, 3*a, 1000)
ramsay = norms.RamsayE(a=a)
plot_weights(support, ramsay.weights, ['-3*a', '0', '3*a'], [-3*a, 0, 3*a]);_____no_output_____

### Trimmed Mean_____no_output_____

help(norms.TrimmedMean.weights)_____no_output_____c = 2
support = np.linspace(-3*c, 3*c, 1000)
trimmed = norms.TrimmedMean(c=c)
plot_weights(support, trimmed.weights, ['-3*c', '0', '3*c'], [-3*c, 0, 3*c]);_____no_output_____

### Tukey's Biweight_____no_output_____

help(norms.TukeyBiweight.weights)_____no_output_____c = 4.685
support = np.linspace(-3*c, 3*c, 1000)
tukey = norms.TukeyBiweight(c=c)
plot_weights(support, tukey.weights, ['-3*c', '0', '3*c'], [-3*c, 0, 3*c]);_____no_output_____

### Scale Estimators_____no_output_____* Robust estimates of the location_____no_output_____

x = np.array([1, 2, 3, 4, 500])_____no_output_____

* The mean is not a robust estimator of location_____no_output_____

x.mean()_____no_output_____

* The median, on the other hand, is a robust estimator with a breakdown point of 50%_____no_output_____

np.median(x)_____no_output_____

* Analagously for the scale
* The standard deviation is not robust_____no_output_____

x.std()_____no_output_____

Median Absolute Deviation

$$ median_i |X_i - median_j(X_j)|) $$_____no_output_____Standardized Median Absolute Deviation is a consistent estimator for $\hat{\sigma}$

$$\hat{\sigma}=K \cdot MAD$$

where $K$ depends on the distribution. For the normal distribution for example,

$$K = \Phi^{-1}(.75)$$_____no_output_____

stats.norm.ppf(.75)_____no_output_____print(x)_____no_output_____sm.robust.scale.mad(x)_____no_output_____np.array([1,2,3,4,5.]).std()_____no_output_____

* The default for Robust Linear Models is MAD
* another popular choice is Huber's proposal 2_____no_output_____

np.random.seed(12345)
fat_tails = stats.t(6).rvs(40)_____no_output_____kde = sm.nonparametric.KDEUnivariate(fat_tails)
kde.fit()
fig = plt.figure(figsize=(12,8))
ax = fig.add_subplot(111)
ax.plot(kde.support, kde.density);_____no_output_____print(fat_tails.mean(), fat_tails.std())_____no_output_____print(stats.norm.fit(fat_tails))_____no_output_____print(stats.t.fit(fat_tails, f0=6))_____no_output_____huber = sm.robust.scale.Huber()
loc, scale = huber(fat_tails)
print(loc, scale)_____no_output_____sm.robust.mad(fat_tails)_____no_output_____sm.robust.mad(fat_tails, c=stats.t(6).ppf(.75))_____no_output_____sm.robust.scale.mad(fat_tails)_____no_output_____

### Duncan's Occupational Prestige data - M-estimation for outliers_____no_output_____

from statsmodels.graphics.api import abline_plot
from statsmodels.formula.api import ols, rlm_____no_output_____prestige = sm.datasets.get_rdataset("Duncan", "carData", cache=True).data_____no_output_____print(prestige.head(10))_____no_output_____fig = plt.figure(figsize=(12,12))
ax1 = fig.add_subplot(211, xlabel='Income', ylabel='Prestige')
ax1.scatter(prestige.income, prestige.prestige)
xy_outlier = prestige.loc['minister', ['income','prestige']]
ax1.annotate('Minister', xy_outlier, xy_outlier+1, fontsize=16)
ax2 = fig.add_subplot(212, xlabel='Education',
                           ylabel='Prestige')
ax2.scatter(prestige.education, prestige.prestige);_____no_output_____ols_model = ols('prestige ~ income + education', prestige).fit()
print(ols_model.summary())_____no_output_____infl = ols_model.get_influence()
student = infl.summary_frame()['student_resid']
print(student)_____no_output_____print(student.loc[np.abs(student) > 2])_____no_output_____print(infl.summary_frame().loc['minister'])_____no_output_____sidak = ols_model.outlier_test('sidak')
sidak.sort_values('unadj_p', inplace=True)
print(sidak)_____no_output_____fdr = ols_model.outlier_test('fdr_bh')
fdr.sort_values('unadj_p', inplace=True)
print(fdr)_____no_output_____rlm_model = rlm('prestige ~ income + education', prestige).fit()
print(rlm_model.summary())_____no_output_____print(rlm_model.weights)_____no_output_____

### Hertzprung Russell data for Star Cluster CYG 0B1 - Leverage Points_____no_output_____* Data is on the luminosity and temperature of 47 stars in the direction of Cygnus._____no_output_____

dta = sm.datasets.get_rdataset("starsCYG", "robustbase", cache=True).data_____no_output_____from matplotlib.patches import Ellipse
fig = plt.figure(figsize=(12,8))
ax = fig.add_subplot(111, xlabel='log(Temp)', ylabel='log(Light)', title='Hertzsprung-Russell Diagram of Star Cluster CYG OB1')
ax.scatter(*dta.values.T)
# highlight outliers
e = Ellipse((3.5, 6), .2, 1, alpha=.25, color='r')
ax.add_patch(e);
ax.annotate('Red giants', xy=(3.6, 6), xytext=(3.8, 6),
            arrowprops=dict(facecolor='black', shrink=0.05, width=2),
            horizontalalignment='left', verticalalignment='bottom',
            clip_on=True, # clip to the axes bounding box
            fontsize=16,
     )
# annotate these with their index
for i,row in dta.loc[dta['log.Te'] < 3.8].iterrows():
    ax.annotate(i, row, row + .01, fontsize=14)
xlim, ylim = ax.get_xlim(), ax.get_ylim()_____no_output_____from IPython.display import Image
Image(filename='star_diagram.png')_____no_output_____y = dta['log.light']
X = sm.add_constant(dta['log.Te'], prepend=True)
ols_model = sm.OLS(y, X).fit()
abline_plot(model_results=ols_model, ax=ax)_____no_output_____rlm_mod = sm.RLM(y, X, sm.robust.norms.TrimmedMean(.5)).fit()
abline_plot(model_results=rlm_mod, ax=ax, color='red')_____no_output_____

* Why? Because M-estimators are not robust to leverage points._____no_output_____

infl = ols_model.get_influence()_____no_output_____h_bar = 2*(ols_model.df_model + 1 )/ols_model.nobs
hat_diag = infl.summary_frame()['hat_diag']
hat_diag.loc[hat_diag > h_bar]_____no_output_____sidak2 = ols_model.outlier_test('sidak')
sidak2.sort_values('unadj_p', inplace=True)
print(sidak2)_____no_output_____fdr2 = ols_model.outlier_test('fdr_bh')
fdr2.sort_values('unadj_p', inplace=True)
print(fdr2)_____no_output_____

* Let's delete that line_____no_output_____

l =  ax.lines[-1]
l.remove()
del l_____no_output_____weights = np.ones(len(X))
weights[X[X['log.Te'] < 3.8].index.values - 1] = 0
wls_model = sm.WLS(y, X, weights=weights).fit()
abline_plot(model_results=wls_model, ax=ax, color='green')_____no_output_____

* MM estimators are good for this type of problem, unfortunately, we don't yet have these yet.
* It's being worked on, but it gives a good excuse to look at the R cell magics in the notebook._____no_output_____

yy = y.values[:,None]
xx = X['log.Te'].values[:,None]_____no_output_____%load_ext rpy2.ipython_____no_output_____%R library(robustbase)
%Rpush yy xx
%R mod <- lmrob(yy ~ xx);
%R params <- mod$coefficients;
%Rpull params_____no_output_____%R print(mod)_____no_output_____print(params)_____no_output_____abline_plot(intercept=params[0], slope=params[1], ax=ax, color='red')_____no_output_____

### Exercise: Breakdown points of M-estimator_____no_output_____

np.random.seed(12345)
nobs = 200
beta_true = np.array([3, 1, 2.5, 3, -4])
X = np.random.uniform(-20,20, size=(nobs, len(beta_true)-1))
# stack a constant in front
X = sm.add_constant(X, prepend=True) # np.c_[np.ones(nobs), X]
mc_iter = 500
contaminate = .25 # percentage of response variables to contaminate_____no_output_____all_betas = []
for i in range(mc_iter):
    y = np.dot(X, beta_true) + np.random.normal(size=200)
    random_idx = np.random.randint(0, nobs, size=int(contaminate * nobs))
    y[random_idx] = np.random.uniform(-750, 750)
    beta_hat = sm.RLM(y, X).fit().params
    all_betas.append(beta_hat)_____no_output_____all_betas = np.asarray(all_betas)
se_loss = lambda x : np.linalg.norm(x, ord=2)**2
se_beta = lmap(se_loss, all_betas - beta_true)_____no_output_____

#### Squared error loss_____no_output_____

np.array(se_beta).mean()_____no_output_____all_betas.mean(0)_____no_output_____beta_true_____no_output_____se_loss(all_betas.mean(0) - beta_true)_____no_output_____