# Notebook from Arios39/Heart-Disease-Classification-Project
Path: End-to-End-Heart-Disease-Classification.ipynb

# Predicting heart disease using machine learning

This notebook looks into using various Python-based machine learning and data science libraries in an attempt to build a machine learning model capable of predicting whether or not someone has heart disease based on their medical attributes.

We're going to take the following approach:
1. Problem definition
2. Data
3. Evaluation
4. Features
5. Modelling
6. Experimentation

## 1. Problem Definition

In a statement,
> Given clinical parameters about a patient, can we predict whether or not they have heart disease?

## 2. Data

The original data came from the Cleavland data from the UCI Machine Learning Repository. https://archive.ics.uci.edu/ml/datasets/heart+Disease

There is also a version of it available on Kaggle. https://www.kaggle.com/ronitf/heart-disease-uci

## 3. Evaluation

> If we can reach 95% accuracy at predicting whether or not a patient has heart disease during the proof of concept, we'll pursue the project.

## 4. Features

This is where you'll get different information about each of the features in your data. You can do this via doing your own research (such as looking at the links above) or by talking to a subject matter expert (someone who knows about the dataset).
_____no_output_____**Create data dictionary**

1. age - age in years
2. sex - (1 = male; 0 = female)
3. cp - chest pain type
    * 0: Typical angina: chest pain related decrease blood supply to the heart
    * 1: Atypical angina: chest pain not related to heart
    * 2: Non-anginal pain: typically esophageal spasms (non heart related)
    * 3: Asymptomatic: chest pain not showing signs of disease
4. trestbps - resting blood pressure (in mm Hg on admission to the hospital) anything above 130-140 is typically cause for concern
5. chol - serum cholestoral in mg/dl
    * serum = LDL + HDL + .2 * triglycerides
    * above 200 is cause for concern
6. fbs - (fasting blood sugar > 120 mg/dl) (1 = true; 0 = false)
    * '>126' mg/dL signals diabetes
7. restecg - resting electrocardiographic results
    * 0: Nothing to note
    * 1: ST-T Wave abnormality
        * can range from mild symptoms to severe problems
        * signals non-normal heart beat
    * 2: Possible or definite left ventricular hypertrophy
        * Enlarged heart's main pumping chamber
8. thalach - maximum heart rate achieved
9. exang - exercise induced angina (1 = yes; 0 = no)
10. oldpeak - ST depression induced by exercise relative to rest looks at stress of heart during excercise unhealthy heart will stress more
11. slope - the slope of the peak exercise ST segment
    * 0: Upsloping: better heart rate with excercise (uncommon)
    * 1: Flatsloping: minimal change (typical healthy heart)
    * 2: Downslopins: signs of unhealthy heart
12. ca - number of major vessels (0-3) colored by flourosopy
    * colored vessel means the doctor can see the blood passing through
    * the more blood movement the better (no clots)
13. thal - thalium stress result
    * 1,3: normal
    * 6: fixed defect: used to be defect but ok now
    * 7: reversable defect: no proper blood movement when excercising
14. target - have disease or not (1=yes, 0=no) (= the predicted attribute)_____no_output_____## Preparing the tools

We're going to use pandas, Matplotlib and NumPy for data analysis and manipulation._____no_output_____

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns

# we want our plots to appear inside the notebook
%matplotlib inline _____no_output_____# Models from Scikit-Learn
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier

# Model Evaluations
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.model_selection import RandomizedSearchCV, GridSearchCV
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.metrics import precision_score, recall_score, f1_score
from sklearn.metrics import plot_roc_curve_____no_output_____

## Load data_____no_output_____

df = pd.read_csv("heart-disease.csv")
df.shape # (rows, columns)_____no_output_____

## Data Exploration (exploratory data analysis or EDA)

The goal here is to find out more about the data and become a subject matter expert on the dataset you're working with.

1. What question(s) are you trying to solve?
2. What kind of data do we have and how do we treat different types?
3. What's missing from the data and how do you deal with it?
4. Where are the outliers and why should you care about them?
5. How can you add, change or remove features to get more out of your data?_____no_output_____

df.head()_____no_output_____df.tail()_____no_output_____df.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 303 entries, 0 to 302
Data columns (total 14 columns):
 #   Column    Non-Null Count  Dtype
---  ------    --------------  -----
 0   age       303 non-null    int64
 1   sex       303 non-null    int64
 2   cp        303 non-null    int64
 3   trestbps  303 non-null    int64
 4   chol      303 non-null    int64
 5   fbs       303 non-null    int64
 6   restecg   303 non-null    int64
 7   thalach   303 non-null    int64
 8   exang     303 non-null    int64
 9   oldpeak   303 non-null    float64
 10  slope     303 non-null    int64
 11  ca        303 non-null    int64
 12  thal      303 non-null    int64
 13  target    303 non-null    int64
dtypes: float64(1), int64(13)
memory usage: 33.3 KB
df.isna().sum()_____no_output_____#Have disease or not (1=yes, 0=no)
df['target'].value_counts()_____no_output_____df["target"].value_counts().plot(kind="bar", color=["salmon", "lightblue"]);_____no_output_____df.describe()_____no_output_____

### Heart Disease Frequency according to Sex _____no_output_____

df.sex.value_counts()_____no_output_____# Compare target column with sex column
pd.crosstab(df.target, df.sex)_____no_output_____# Create a plot of crosstab
pd.crosstab(df.target, df.sex).plot(kind="bar",
                                    figsize=(10, 6),
                                    color=["salmon", "lightblue"])

plt.title("Heart Disease Frequency for Sex")
plt.xlabel("0 = No Diesease, 1 = Disease")
plt.ylabel("Amount")
plt.legend(["Female", "Male"]);
plt.xticks(rotation=0);_____no_output_____

### Age vs. Max Heart Rate for Heart Disease_____no_output_____

# Create another figure
plt.figure(figsize=(10, 6))

# Scatter with postivie examples
plt.scatter(df.age[df.target==1],
            df.thalach[df.target==1],
            c="salmon")

# Scatter with negative examples
plt.scatter(df.age[df.target==0],
            df.thalach[df.target==0],
            c="lightblue")

# Add some helpful info
plt.title("Heart Disease in function of Age and Max Heart Rate")
plt.xlabel("Age")
plt.ylabel("Max Heart Rate")
plt.legend(["Disease", "No Disease"]);_____no_output_____# Check the distribution of the age column with a histogram
df.age.plot.hist();_____no_output_____

### Heart Disease Frequency per Chest Pain Type

3. cp - chest pain type
    * 0: Typical angina: chest pain related decrease blood supply to the heart
    * 1: Atypical angina: chest pain not related to heart
    * 2: Non-anginal pain: typically esophageal spasms (non heart related)
    * 3: Asymptomatic: chest pain not showing signs of disease_____no_output_____

pd.crosstab(df.cp, df.target)_____no_output_____# Make the crosstab more visual
pd.crosstab(df.cp, df.target).plot(kind="bar",
                                   figsize=(10, 6),
                                   color=["salmon", "lightblue"])

# Add some communication
plt.title("Heart Disease Frequency Per Chest Pain Type")
plt.xlabel("Chest Pain Type")
plt.ylabel("Amount")
plt.legend(["No Disease", "Disease"])
plt.xticks(rotation=0);_____no_output_____df.head()_____no_output_____# Make a correlation matrix
df.corr()_____no_output_____# Let's make our correlation matrix a little prettier
corr_matrix = df.corr()
fig, ax = plt.subplots(figsize=(15, 10))
ax = sns.heatmap(corr_matrix,
                 annot=True,
                 linewidths=0.5,
                 fmt=".2f",
                 cmap="YlGnBu");
bottom, top = ax.get_ylim()
ax.set_ylim(bottom + 0.5, top - 0.5)_____no_output_____

## 5. Modelling _____no_output_____

df.head()_____no_output_____# Split data into X and y
X = df.drop("target", axis=1)

y = df["target"]_____no_output_____X_____no_output_____y_____no_output_____# Split data into train and test sets
np.random.seed(42)

# Split into train & test set
X_train, X_test, y_train, y_test = train_test_split(X,
                                                    y,
                                                    test_size=0.2)        _____no_output_____X_train_____no_output_____y_train, len(y_train)_____no_output_____

Now we've got our data split into training and test sets, it's time to build a machine learning model.

We'll train it (find the patterns) on the training set.

And we'll test it (use the patterns) on the test set.

We're going to try 3 different machine learning models:
1. Logistic Regression
2. K-Nearest Neighbours Classifier
3. Random Forest Classifier_____no_output_____

# Put models in a dictionary
models = {"Logistic Regression": LogisticRegression(),
          "KNN": KNeighborsClassifier(),
          "Random Forest": RandomForestClassifier()}

# Create a function to fit and score models
def fit_and_score(models, X_train, X_test, y_train, y_test):
    """
    Fits and evaluates given machine learning models.
    models : a dict of differetn Scikit-Learn machine learning models
    X_train : training data (no labels)
    X_test : testing data (no labels)
    y_train : training labels
    y_test : test labels
    """
    # Set random seed
    np.random.seed(42)
    # Make a dictionary to keep model scores
    model_scores = {}
    # Loop through models
    for name, model in models.items():
        # Fit the model to the data
        model.fit(X_train, y_train)
        # Evaluate the model and append its score to model_scores
        model_scores[name] = model.score(X_test, y_test)
    return model_scores_____no_output_____model_scores = fit_and_score(models=models,
                             X_train=X_train,
                             X_test=X_test,
                             y_train=y_train,
                             y_test=y_test)

model_scores/opt/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

### Model Comparison_____no_output_____

model_compare = pd.DataFrame(model_scores, index=["accuracy"])
model_compare.T.plot.bar();_____no_output_____

Now we've got a baseline model... and we know a model's first predictions aren't always what we should based our next steps off. What should we do?

Let's look at the following:
* Hypyterparameter tuning
* Feature importance
* Confusion matrix
* Cross-validation
* Precision
* Recall
* F1 score
* Classification report
* ROC curve
* Area under the curve (AUC)

### Hyperparameter tuning (by hand)_____no_output_____

# Let's tune KNN

train_scores = []
test_scores = []

# Create a list of differnt values for n_neighbors
neighbors = range(1, 21)

# Setup KNN instance
knn = KNeighborsClassifier()

# Loop through different n_neighbors
for i in neighbors:
    knn.set_params(n_neighbors=i)

    # Fit the algorithm
    knn.fit(X_train, y_train)

    # Update the training scores list
    train_scores.append(knn.score(X_train, y_train))

    # Update the test scores list
    test_scores.append(knn.score(X_test, y_test))_____no_output_____train_scores_____no_output_____test_scores_____no_output_____plt.plot(neighbors, train_scores, label="Train score")
plt.plot(neighbors, test_scores, label="Test score")
plt.xticks(np.arange(1, 21, 1))
plt.xlabel("Number of neighbors")
plt.ylabel("Model score")
plt.legend()

print(f"Maximum KNN score on the test data: {max(test_scores)*100:.2f}%")Maximum KNN score on the test data: 75.41%

## Hyperparameter tuning with RandomizedSearchCV

We're going to tune:
* LogisticRegression()
* RandomForestClassifier()

... using RandomizedSearchCV_____no_output_____

# Create a hyperparameter grid for LogisticRegression
log_reg_grid = {"C": np.logspace(-4, 4, 20),
                "solver": ["liblinear"]}

# Create a hyperparameter grid for RandomForestClassifier
rf_grid = {"n_estimators": np.arange(10, 1000, 50),
           "max_depth": [None, 3, 5, 10],
           "min_samples_split": np.arange(2, 20, 2),
           "min_samples_leaf": np.arange(1, 20, 2)}_____no_output_____

Now we've got hyperparameter grids setup for each of our models, let's tune them using RandomizedSearchCV..._____no_output_____

# Tune LogisticRegression

np.random.seed(42)

# Setup random hyperparameter search for LogisticRegression
rs_log_reg = RandomizedSearchCV(LogisticRegression(),
                                param_distributions=log_reg_grid,
                                cv=5,
                                n_iter=20,
                                verbose=True)

# Fit random hyperparameter search model for LogisticRegression
rs_log_reg.fit(X_train, y_train)Fitting 5 folds for each of 20 candidates, totalling 100 fits
rs_log_reg.best_params______no_output_____rs_log_reg.score(X_test, y_test)_____no_output_____# Setup random seed
np.random.seed(42)

# Setup random hyperparameter search for RandomForestClassifier
rs_rf = RandomizedSearchCV(RandomForestClassifier(),
                           param_distributions=rf_grid,
                           cv=5,
                           n_iter=20,
                           verbose=True)

# Fit random hyperparameter search model for RandomForestClassifier()
rs_rf.fit(X_train, y_train)Fitting 5 folds for each of 20 candidates, totalling 100 fits
# Find the best hyperparameters
rs_rf.best_params______no_output_____# Evaluate the randomized search RandomForestClassifier model
rs_rf.score(X_test, y_test)_____no_output_____

## Hyperparamter Tuning with GridSearchCV

Since our LogisticRegression model provides the best scores so far, we'll try and improve them again using GridSearchCV..._____no_output_____

# Different hyperparameters for our LogisticRegression model
log_reg_grid = {"C": np.logspace(-4, 4, 30),
                "solver": ["liblinear"]}

# Setup grid hyperparameter search for LogisticRegression
gs_log_reg = GridSearchCV(LogisticRegression(),
                          param_grid=log_reg_grid,
                          cv=5,
                          verbose=True)

# Fit grid hyperparameter search model
gs_log_reg.fit(X_train, y_train);Fitting 5 folds for each of 30 candidates, totalling 150 fits
# Check the best hyperparmaters
gs_log_reg.best_params______no_output_____# Evaluate the grid search LogisticRegression model
gs_log_reg.score(X_test, y_test)_____no_output_____

## Evaluting our tuned machine learning classifier, beyond accuracy

* ROC curve and AUC score
* Confusion matrix
* Classification report
* Precision
* Recall
* F1-score

... and it would be great if cross-validation was used where possible.

To make comparisons and evaluate our trained model, first we need to make predictions._____no_output_____

# Make predictions with tuned model
y_preds = gs_log_reg.predict(X_test)_____no_output_____y_preds_____no_output_____y_test_____no_output_____# Plot ROC curve and calculate and calculate AUC metric
plot_roc_curve(gs_log_reg, X_test, y_test)_____no_output_____# Confusion matrix
print(confusion_matrix(y_test, y_preds))[[25  4]
 [ 3 29]]
sns.set(font_scale=1.5)

def plot_conf_mat(y_test, y_preds):
    """
    Plots a nice looking confusion matrix using Seaborn's heatmap()
    """
    fig, ax = plt.subplots(figsize=(3, 3))
    ax = sns.heatmap(confusion_matrix(y_test, y_preds),
                     annot=True,
                     cbar=False)
    plt.xlabel("True label")
    plt.ylabel("Predicted label")

    bottom, top = ax.get_ylim()
    ax.set_ylim(bottom + 0.5, top - 0.5)

plot_conf_mat(y_test, y_preds)_____no_output_____

Now we've got a ROC curve, an AUC metric and a confusion matrix, let's get a classification report as well as cross-validated precision, recall and f1-score._____no_output_____

print(classification_report(y_test, y_preds))              precision    recall  f1-score   support

           0       0.89      0.86      0.88        29
           1       0.88      0.91      0.89        32

    accuracy                           0.89        61
   macro avg       0.89      0.88      0.88        61
weighted avg       0.89      0.89      0.89        61

### Calculate evaluation metrics using cross-validation

We're going to calculate accuracy, precision, recall and f1-score of our model using cross-validation and to do so we'll be using `cross_val_score()`._____no_output_____

# Check best hyperparameters
gs_log_reg.best_params______no_output_____# Create a new classifier with best parameters
clf = LogisticRegression(C=0.20433597178569418,
                         solver="liblinear")_____no_output_____# Cross-validated accuracy
cv_acc = cross_val_score(clf,
                         X,
                         y,
                         cv=5,
                         scoring="accuracy")
cv_acc_____no_output_____cv_acc = np.mean(cv_acc)
cv_acc_____no_output_____# Cross-validated precision
cv_precision = cross_val_score(clf,
                         X,
                         y,
                         cv=5,
                         scoring="precision")
cv_precision=np.mean(cv_precision)
cv_precision_____no_output_____# Cross-validated recall
cv_recall = cross_val_score(clf,
                         X,
                         y,
                         cv=5,
                         scoring="recall")
cv_recall = np.mean(cv_recall)
cv_recall_____no_output_____# Cross-validated f1-score
cv_f1 = cross_val_score(clf,
                         X,
                         y,
                         cv=5,
                         scoring="f1")
cv_f1 = np.mean(cv_f1)
cv_f1_____no_output_____# Visualize cross-validated metrics
cv_metrics = pd.DataFrame({"Accuracy": cv_acc,
                           "Precision": cv_precision,
                           "Recall": cv_recall,
                           "F1": cv_f1},
                          index=[0])

cv_metrics.T.plot.bar(title="Cross-validated classification metrics",
                      legend=False);_____no_output_____

### Feature Importance

Feature importance is another as asking, "which features contributed most to the outcomes of the model and how did they contribute?"

Finding feature importance is different for each machine learning model. One way to find feature importance is to search for "(MODEL NAME) feature importance".

Let's find the feature importance for our LogisticRegression model..._____no_output_____

# Fit an instance of LogisticRegression
clf = LogisticRegression(C=0.20433597178569418,
                         solver="liblinear")

clf.fit(X_train, y_train);_____no_output_____# Check coef_
clf.coef______no_output_____df.head()_____no_output_____# Match coef's of features to columns
feature_dict = dict(zip(df.columns, list(clf.coef_[0])))
feature_dict_____no_output_____# Visualize feature importance
feature_df = pd.DataFrame(feature_dict, index=[0])
feature_df.T.plot.bar(title="Feature Importance", legend=False);_____no_output_____pd.crosstab(df["sex"], df["target"])_____no_output_____pd.crosstab(df["slope"], df["target"])_____no_output_____

slope - the slope of the peak exercise ST segment
* 0: Upsloping: better heart rate with excercise (uncommon)
* 1: Flatsloping: minimal change (typical healthy heart)
* 2: Downslopins: signs of unhealthy heart_____no_output_____


# Notebook from ashishpatel26/Data_Science_Fundamentals
Path: notebooks/Machine_Learning/Algorithms_From_Scratch/Gradient_Descent.ipynb

## Imports_____no_output_____

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt_____no_output_____

## Style_____no_output_____

plt.style.use('ggplot')_____no_output_____

## Batch Gradient Descent Class_____no_output_____

class BatchGradientDescent:

    def __init__(self, learning_rate=0.001, n_iter=20, seed=None):
        self.learning_rate_ = learning_rate
        self.n_iter_ = n_iter
        self.seed_ = seed

    def fit(self, X, y, fit_intercept=True):
        '''run batch gradient descent on training data'''

        rows = X.shape[0]
        cols = X.shape[1]

        # reproducibility
        if self.seed_:
            np.random.seed(self.seed_)

        # intercept check
        if fit_intercept:
            X = np.c_[np.ones(rows), X]
            cols = X.shape[1]

        # randomly initialize coefficients
        self.coef_ = np.random.normal(loc=0, scale=1.5, size=cols)

        # setup to track cost and coefficients
        self.cost_ = []
        self.coefficient_tracker_ = [self.coef_]

        # main logic
        for i in range(self.n_iter_):
            predictions = self.predict(X)
            residuals = np.subtract(predictions, y)
            update_step = np.multiply(self.learning_rate_, X.T.dot(residuals))
            self.coef_ = np.subtract(self.coef_, update_step)
            cost = (residuals**2).sum()
            self.cost_.append(cost)
            self.coefficient_tracker_.append(self.coef_)

    def predict(self, X):
        '''return predictions for given inputs'''

        # dimensionality check
        if len(X.shape) == 1:
            X = X.reshape(-1,1)

        return np.dot(X, self.coef_) _____no_output_____

## Create Data_____no_output_____

np.random.seed(123)

rows = 100
cols = 2

intercept = 3.2
slope = 1.9

noise = np.random.normal(loc=0, scale=3.5, size=rows)_____no_output_____data = np.linspace(start=0, stop=20, num=rows)_____no_output_____target = np.multiply(slope, data)
target = np.add(intercept, target)
target = np.add(target, noise)_____no_output_____

## Plot Data_____no_output_____

plt.plot(data, target, 'bo', alpha=0.5);_____no_output_____target_no_noise = np.multiply(slope, data)
target_no_noise = np.add(intercept, target_no_noise)
plt.plot(data, target, 'bo', alpha=0.5)
plt.plot(data, target_no_noise, 'k--', label='actual (sans noise)')
plt.legend();_____no_output_____

## Fit w/Sklearn_____no_output_____

from sklearn.linear_model import LinearRegression

lr = LinearRegression()
lr.fit(data.reshape(-1,1), target)_____no_output_____sklearn_intercept = np.round(lr.intercept_, 4)
sklearn_coef = np.round(lr.coef_, 4)[0]
print('intercept: {} | coef: {}'.format(sklearn_intercept, sklearn_coef))intercept: 3.2458 | coef: 1.9049
plt.plot(data, target, 'bo', alpha=0.5)
plt.plot(data, lr.predict(data.reshape(-1,1)), 'g', label='sklearn')
plt.legend();_____no_output_____plt.plot(data, target, 'bo', alpha=0.5)
plt.plot(data, lr.predict(data.reshape(-1,1)), 'g', label='sklearn')
plt.plot(data, target_no_noise, 'k--', label='actual (sans noise)')
plt.legend();_____no_output_____

## Fit w/Gradient Descent_____no_output_____

gd = BatchGradientDescent(learning_rate=0.0001, n_iter=5, seed=2)
gd.fit(data.reshape(-1,1), target, fit_intercept=True)_____no_output_____test_data = np.c_[np.ones(rows), data]
print(test_data[:5])[[ 1.          0.        ]
 [ 1.          0.2020202 ]
 [ 1.          0.4040404 ]
 [ 1.          0.60606061]
 [ 1.          0.80808081]]
def cost_plot(obj):
    '''plot cost as a function of iteration number'''
    x_axis = np.arange(1, len(obj.cost_)+1)
    plt.plot(x_axis, obj.cost_);
    plt.title('Cost vs Iteration')
    plt.xlabel('Iteration')
    plt.xticks(x_axis)
    plt.ylabel('Cost');_____no_output_____cost_plot(gd)_____no_output_____plt.plot(data, target, 'bo', alpha=0.5)
plt.plot(data, gd.predict(test_data), 'r--', label='gd')
plt.legend();_____no_output_____plt.plot(data, target, 'bo', alpha=0.5)
plt.plot(data, gd.predict(test_data), 'r--', label='gd')
plt.plot(data, target_no_noise, 'k--', label='actual (sans noise)')
plt.legend();_____no_output_____

## Evolution of $\theta_0$ & $\theta_1$_____no_output_____

iterations = np.arange(1, len(gd.coefficient_tracker_)+1)

plt.plot(iterations, np.array(gd.coefficient_tracker_)[:,0], 'r--o')
plt.title(r'$\theta_0$')
plt.xlabel('iteration #')

plt.figure()

plt.plot(iterations, np.array(gd.coefficient_tracker_)[:,1], 'b--o')
plt.axhline(y=lr.coef_, color='g')
plt.title(r'$\theta_1$')
plt.xlabel('iteration #');

plt.figure()

plt.plot(np.array(gd.coefficient_tracker_)[:,0], np.array(gd.coefficient_tracker_)[:,1], 'k--o')
plt.title('Coefficient Evolution')
plt.xlabel(r'$\theta_0$')
plt.ylabel(r'$\theta_1$');_____no_output_____

## Evolution of Model by Iteration_____no_output_____

for i, coefficients in enumerate(gd.coefficient_tracker_):
    plt.figure()
    plt.plot(data, target, 'bo', alpha=0.5)
    predictions = np.dot(test_data, coefficients)
    plt.plot(data, predictions, 'r--', label='gd_'+str(i))
    plt.legend();_____no_output_____

## Comparison_____no_output_____

plt.figure(figsize=(9,9))
plt.plot(data, target, 'bo', alpha=0.5)
plt.plot(data, target_no_noise, 'k--', label='actual (sans noise)')
plt.plot(data, lr.predict(data.reshape(-1,1)), 'g', label='sklearn')
plt.plot(data, gd.predict(test_data), 'r--', label='gd')
plt.xlabel('feature 1')
plt.ylabel('target')
plt.title('Comparison: Generative vs Sklearn vs GD')
plt.legend();_____no_output_____from sklearn.metrics import mean_squared_error

sklearnGD = round(mean_squared_error(target, lr.predict(data.reshape(-1,1))),3)
myGD = round(mean_squared_error(target, gd.predict(np.c_[np.ones(rows), data])),3)

print(' '*4 + '[Mean Squared Error]')
print('sklearn: {} | gd: {}'.format(sklearnGD, myGD))    [Mean Squared Error]
sklearn: 15.593 | gd: 19.001


# Notebook from so19/Airbnb-Project
Path: Yelp sentiment analysis.ipynb

# In this project, I found this data on kaggle and decided to conduct a Sentiment Analysis on reviews. In this I have used Tf-Idf and Bag of words techniques to predict number of stars on basis of the review provided. I have also created word clouds of most positive and most negative words used in reviews. Also performed Stemming, Lemmantization on the reviews to get them in proper format for analysis._____no_output_____# Stemming,Lemmantization and Sentiment Analysis of Yelp Reviews(Natural Language Processing)_____no_output_____

# numerical modules
import pandas as pd
import numpy as np
import scipy as sp_____no_output_____# machine learning modules
from sklearn.cross_validation import train_test_split
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn import metricsC:\Users\soham\Anaconda3\lib\site-packages\sklearn\cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
# text processing modules
from textblob import TextBlob, Word
from nltk.stem.porter import PorterStemmer

import heapq
from collections import Counter, OrderedDict_____no_output_____# wordcloud visualization
from wordcloud import WordCloud

# Visualization
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline_____no_output_____%%javascript
IPython.OutputArea.auto_scroll_threshold = 999999;_____no_output_____

# 1) Data Loading_____no_output_____

#Loading the csv file
yelp_df = pd.read_csv('review.csv')_____no_output_____yelp_df.head()_____no_output_____yelp_df.shape_____no_output_____yelp_df['type'].unique()_____no_output_____# business_id, date, user_id, type columns are irrelevant for now
yelp_df.drop(['business_id', 'date', 'review_id', 'type', 'user_id'], axis=1, inplace=True)_____no_output_____yelp_df.head()_____no_output_____

# Now We Will only keep the most positive and most negative reviews _____no_output_____

# Here we used the And function to only select positive and negative reviews i.e. reviews having 1 and 5 stars only
yelp_df = yelp_df[(yelp_df['stars'] == 5) | (yelp_df['stars'] == 1)]_____no_output_____# Here is the text of first 5 positive reviews
yelp_df[yelp_df['stars']==5]['text'].head()_____no_output_____# similarly here is few of negative reviews
yelp_df[yelp_df['stars']==1]['text'].head()_____no_output_____

So, we can see infer from the above texts that some of reviews are bad and some are good_____no_output_____

# Checking how many are positive sentiment and how many are negative
yelp_df['stars'].value_counts()_____no_output_____# This shows graph what we found above graphically
sns.countplot(y='stars', data=yelp_df)
_____no_output_____ratio = float(len(yelp_df[yelp_df['stars'] == 5])) / (len(yelp_df[yelp_df['stars'] == 5]) + len(yelp_df[yelp_df['stars'] == 1]))_____no_output_____print("Null Accuracy: %0.6f" % (ratio))Null Accuracy: 0.816691

# 2. Sentiment Analysis with TextBlob_____no_output_____

# print the first yelp review
print(yelp_df.text[0])My wife took me here on my birthday for breakfast and it was excellent.  The weather was perfect which made sitting outside overlooking their grounds an absolute pleasure.  Our waitress was excellent and our food arrived quickly on the semi-busy Saturday morning.  It looked like the place fills up pretty quickly so the earlier you get here the better.

Do yourself a favor and get their Bloody Mary.  It was phenomenal and simply the best I've ever had.  I'm pretty sure they only use ingredients from their garden and blend them fresh when you order it.  It was amazing.

While EVERYTHING on the menu looks excellent, I had the white truffle scrambled eggs vegetable skillet and it was tasty and delicious.  It came with 2 pieces of their griddled bread with was amazing and it absolutely made the meal complete.  It was the best "toast" I've ever had.

Anyway, I can't wait to go back!
yelp_df.stars[0]_____no_output_____# polarity ranges from -1 (most negative) to 1 (most positive)# polari
review = TextBlob(yelp_review_df.text[0])_____no_output_____review.sentiment.polarity_____no_output_____review.sentiment.subjectivity_____no_output_____# create a new DataFrame column for sentiment# create
yelp_df['texblob_sentiment_score'] = yelp_df.text.apply(lambda x :TextBlob(x).sentiment.polarity)_____no_output_____yelp_df.head()_____no_output_____# let's look at some negative reviews
yelp_df[yelp_df['stars'] == 1].head(10)_____no_output_____# let's check a random negative review, 35 seems to tbe good candidate
print(yelp_review_df.text[35])I've eaten here many times, but none as bad as last night.
Service was excellent, and highly attentive.
Food, absolutely horrible.

My expectation was they would serve a steak on par with their seafood. After all, they were charging 39 bucks for a ribeye.
What I was hoping for was a 1- 1-1/2' thick steak, cooked Pittsburgh style as I had ordered.
What I got a a 3/4 in thick piece of meat that was mostly fat, gristle, and in no way resembled Pittsburgh Style.
Salad, similar to something you could get at Chick Filet
Veggies, blah.
Bread basket, ample, but day old, and if not, it certainly wasn't fresh.

In addition to bad food, we were crammed into a small room where we were nuts to butts with 6 other tables, listening to conversations ranging from someone's recent bout with pinkeye, and another couple who elected to speak entirely in French, until the waiter showed up, then it was like they turned off the French switch and suddenly began speaking English.

I've had it with this place.
If I'm going to pay 150 bucks for dinner, it'll be at Mortons, or Maestro where the steaks are 1-1/2 in thick, cooked to perfection, and half of it doesnt wind up on the plate as fat and gristle
# let's look at some positive reviews
yelp_df[yelp_df['stars'] == 5].head(10)_____no_output_____# let's check a random predicted positive review, 9 seems to tbe good candidate
print(yelp_review_df.text[9])Nobuo shows his unique talents with everything on the menu. Carefully crafted features with much to drink. Start with the pork belly buns and a stout. Then go on until you can no longer.
# reviews with most positive sentiment
yelp_df[yelp_df['texblob_sentiment_score'] == 1]['text'].head()_____no_output_____# reviews with most negative sentiment
yelp_df[yelp_df['texblob_sentiment_score'] == -1]['text'].head()_____no_output_____# what is the minimum sentiment given by textblob for negative reviews?
yelp_df[yelp_df['stars'] == 1]['texblob_sentiment_score'].min()_____no_output_____# what is the maximum sentiment given by textblob for negative reviews?
yelp_df[yelp_df['stars'] == 1]['texblob_sentiment_score'].max()_____no_output_____# positive sentiment in a 1-star review
yelp_df[(yelp_df['stars'] == 1) & (yelp_df['texblob_sentiment_score'] > 0.7)]_____no_output_____print(yelp_review_df.text[8833])The owner has changed hands & this place isn't what it used to be.  If you want up to date paper & quality product...go to Scrap Happy OR Crop Girls!
# what is the minimum sentiment given by textblob for positive reviews?
yelp_df[yelp_df['stars'] == 5]['texblob_sentiment_score'].min()
_____no_output_____# what is the maximum sentiment given by textblob for positive reviews?
yelp_df[yelp_df['stars'] == 5]['texblob_sentiment_score'].max()
_____no_output_____# negative sentiment in a 5-star review# negati
yelp_df[(yelp_df['stars'] == 5) & (yelp_df['texblob_sentiment_score'] < -0.3)]_____no_output_____print(yelp_review_df['text'][9809])I have to tell you....

I had their Jerk Chicken Plate the other night, and it was yuuuuummmmmyyy! You gotta try it!

# Use TextBlob to calculate stars for the reviews_____no_output_____

yelp_df['textblob_predicted_stars'] = yelp_df['texblob_sentiment_score'].apply(lambda x : 5 if x > 0.1 else 1)_____no_output_____yelp_df.head()_____no_output_____print('Textblob Accuracy', metrics.accuracy_score(yelp_df['stars'], yelp_df['textblob_predicted_stars']))Textblob Accuracy 0.8935389133627019

So we where able to predict upto 90% of correct rating stars on basis of the reviews_____no_output_____# 3. Sentiment Classification with Logistic Regression_____no_output_____## 3.1 Tokenization, Stemming & Lemmatization_____no_output_____After tokenizning the string into words, we use nltk PorterStemmer to stem the tokens._____no_output_____

import re, nltk
from nltk.stem.porter import PorterStemmer
from nltk.stem import WordNetLemmatizer
from nltk.corpus import stopwords_____no_output_____stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

english_stops = stopwords.words('english')_____no_output_____print(english_stops)['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]
def simple_tokenizer(text):
    # remove non letters
    text = res.sub("[^a-zA-Z]", " ", text)
    # tokenize
    tokens = nltk.word_tokenize(text)
    tokens = [elem for elem in tokens if (len(elem) > 2 and elem not in english_stops)]

    return tokens_____no_output_____def stem_tokenizer(text):
    # remove non letters
    text = re.sub("[^a-zA-Z]", " ", text)
    # tokenize
    tokens = nltk.word_tokenize(text)
    tokens = [elem for elem in tokens if (len(elem) > 2 and elem not in english_stops)]
    # stem
    stems = []
    for item in tokens:
        stems.append(stemmer.stem(item))

    return stems_____no_output_____def lemma_tokenizer(text):
    # remove non letters
    text = re.sub("[^a-zA-Z]", " ", text)
    # tokenize
    tokens = nltk.word_tokenize(text)
    tokens = [elem for elem in tokens if (len(elem) > 2 and elem not in english_stops)]
    # lemmas
    lemmas = []
    for item in tokens:
        lemmas.append(lemmatizer.lemmatize(item))

    return lemmas_____no_output_____

Bag Of Words Model

Here we will be using here is called a bag-of-words model. In this kind of model we simplify documents to a multi-set of terms frequencies. That means that, for our model, a document sentiment tag will depend on what words appear in that document, discarding any grammar or word order but keeping multiplicity.

_____no_output_____### 3.2.1 Counting Word Frequency_____no_output_____

count_vect = CountVectorizer(analyzer = 'word', tokenizer=simple_tokenizer, lowercase=True)_____no_output_____import nltk
nltk.download('punkt')
[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\soham\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!
review_tf = count_vect.fit_transform(yelp_df['text'])_____no_output_____# Numpy arrays are easy to work with, so convert the result to an array
review_tf_nd = review_tf.toarray()
review_tf_nd.shape_____no_output_____# Take a look at how the Document Term Matrix looks
df = pd.DataFrame(review_tf_nd, columns=count_vect.get_feature_names())
df.head(5)_____no_output_____# let's look at the words in the vocabulary, first 100 and last 100
vocab = count_vect.get_feature_names()
print(vocab[:100]) # First 100 words
print(vocab[-100:]) # Last 100 words
#print(vocab)  # all 18k words['aaa', 'aaaamazing', 'aaammmazzing', 'aaron', 'aback', 'abandoned', 'abandoning', 'abba', 'abbreviate', 'abbreviated', 'abby', 'abc', 'abdomen', 'abhor', 'abiding', 'abilities', 'ability', 'abilty', 'able', 'abodoba', 'abou', 'abound', 'abrasion', 'abroad', 'abrupt', 'absence', 'absent', 'absinthe', 'absoloutely', 'absolut', 'absolute', 'absolutely', 'absolutley', 'absolutly', 'absorb', 'absorption', 'abstain', 'abstained', 'abstract', 'absurd', 'abuelo', 'abundance', 'abundant', 'abuse', 'abused', 'abusive', 'abysmal', 'acacia', 'academy', 'acai', 'acapulco', 'accent', 'accented', 'accents', 'accept', 'acceptable', 'accepted', 'accepting', 'accepts', 'access', 'accessed', 'accessible', 'accessories', 'accessorize', 'accessory', 'accident', 'accidental', 'accidentally', 'accidently', 'acclaimed', 'accolades', 'accommodate', 'accommodated', 'accommodates', 'accommodating', 'accommodation', 'accommodations', 'accomodate', 'accomodated', 'accomodating', 'accompanied', 'accompanies', 'accompaniment', 'accompany', 'accompanying', 'accomplish', 'accomplished', 'accomplishment', 'according', 'accordingly', 'account', 'accountable', 'accounts', 'accoutrement', 'accredited', 'accross', 'accumulated', 'accurate', 'accurately', 'accusation']
['youth', 'youthful', 'youtube', 'yow', 'yowza', 'yqqtg', 'yragui', 'yrs', 'yuck', 'yucky', 'yuk', 'yukgejang', 'yukon', 'yum', 'yuma', 'yumm', 'yummie', 'yummier', 'yumminess', 'yummm', 'yummmayyyy', 'yummmm', 'yummmmm', 'yummmmmm', 'yummmmmmers', 'yummmmmmmm', 'yummmmy', 'yummo', 'yummy', 'yumness', 'yung', 'yup', 'yupha', 'yuppies', 'yur', 'yuri', 'yusefs', 'yuukk', 'yuuuummmmae', 'yuuuuummmmmyyy', 'yuuuuuuum', 'yuyuyummy', 'yuzu', 'yyyeeaahhhh', 'yyyyy', 'zabba', 'zach', 'zam', 'zanella', 'zankou', 'zappos', 'zatsiki', 'zen', 'zero', 'zest', 'zesty', 'zexperience', 'zha', 'zhou', 'zia', 'zichini', 'zihuatenejo', 'zilch', 'zillion', 'zin', 'zinburger', 'zinburgergeist', 'zinc', 'zinfandel', 'zing', 'zip', 'zipcar', 'ziploc', 'zipper', 'zippers', 'zipps', 'ziti', 'zoe', 'zombi', 'zombies', 'zone', 'zoners', 'zones', 'zoning', 'zoo', 'zoom', 'zoyo', 'zse', 'zucca', 'zucchini', 'zuccini', 'zuchinni', 'zumba', 'zupa', 'zupas', 'zuzu', 'zuzus', 'zwftv', 'zwiebel', 'zzed']
# calculate frequency distribution of words in positive reviews
dist = np.sum(review_tf_nd[np.where(yelp_df['stars'] == 5)], axis=0)

counter = Counter()
# For each, print the vocabulary word and the number of times it appears in the data set
for tag, count in zip(vocab, dist):
    counter[tag] = count_____no_output_____# word_count sum sorted by value
counter.most_common(20)
_____no_output_____dicttt={'great': 2011, 'place': 1988, 'food': 1679, 'good': 1663, 'like': 1278, 'one': 1254, 'get': 1108, 'love': 1104, 'time': 1063, 'best': 970, 'service': 901, 'really': 879, 'always': 846, 'also': 841, 'back': 767, 'would': 717, 'well': 689, 'amazing': 618, 'delicious': 579, 'nice': 578}_____no_output_____
wordcloud = WordCloud().fit_words(dicttt)
plt.axis("off")
plt.imshow(wordcloud)_____no_output_____# calculate frequency distribution of words in negative reviews
dist = np.sum(review_tf_nd[np.where(yelp_df['stars'] == 1)], axis=0)

counter = Counter()
# For each, print the vocabulary word and the number of times it appears in the data set
for tag, count in zip(vocab, dist):
    counter[tag] = count_____no_output_____# word_count sum sorted by value
counter.most_common(20)_____no_output_____dictt2={'food': 586,'place':489,'like': 439,'one': 407,'get': 361,'would':331,'service': 325,'back': 320,'good': 301,
 'time':301,
 'even': 278,
 'never': 241,
 'minutes': 218,
 'order': 206,
 'got': 205,
 'ordered': 202,
 'really': 194,
'went': 192,
 'said': 187,
 'restaurant': 185}_____no_output_____wordcloud = WordCloud().fit_words(dictt2)
plt.axis("off")
plt.imshow(wordcloud)_____no_output_____

### 3.2.3 Predict the stars rating - A bag-of-words Linear Classifier_____no_output_____

# split Train and Validation Data Set
X_train, X_test, y_train, y_test  = train_test_split(review_tf_nd, yelp_df['stars'], train_size=0.75)_____no_output_____# Create and train the Logistic Regression Classifier
log_tf_model = LogisticRegression()
log_tf_model = log_tf_model.fit(X=X_train, y=y_train)_____no_output_____# Use the classifier to label our evaluation set. We can use either predict for classes or predict_proba for probabilities.
y_pred = log_tf_model.predict(X=X_test)
y_pred_proba = log_tf_model.predict_proba(X=X_test)[:,1]_____no_output_____#calculates several types of (predictive) scores on a classification model
from sklearn import metrics
print(metrics.classification_report(y_test, y_pred))             precision    recall  f1-score   support

          1       0.91      0.69      0.79       179
          5       0.94      0.99      0.96       843

avg / total       0.93      0.93      0.93      1022

print("accuracy:  %0.6f"% metrics.accuracy_score(y_test, y_pred))accuracy:  0.934442
from pylab import rcParams
labels = ['5', '1']
confmat = metrics.confusion_matrix(y_test, y_pred)
fig, ax = plt.subplots(figsize=(3, 3))
ax.matshow(confmat, cmap=plt.cm.Blues, alpha=0.7)
for i in range(confmat.shape[0]):
    for j in range(confmat.shape[1]):
        ax.text(x=j, y=i, s=confmat[i, j], va='center', ha='center')
tick_marks = np.arange(len(labels))
plt.xticks(tick_marks, labels, rotation=45)
plt.yticks(tick_marks,labels)
plt.xlabel('predicted label')
plt.ylabel('true label')
plt.tight_layout()_____no_output_____fpr_tf, tpr_tf, thresholds = metrics.roc_curve(y_test, y_pred_proba, pos_label=5)
roc_auc_tf = metrics.auc(fpr_tf, tpr_tf)_____no_output_____

# 3.3 TF-IDF Model

## 3.3.1 Calculate Inverse Document Term Frequency_____no_output_____

tfidf_vect = TfidfVectorizer(analyzer = 'word', tokenizer=lemma_tokenizer, lowercase=True)_____no_output_____import nltk
nltk.download('wordnet')[nltk_data] Downloading package wordnet to
[nltk_data]     C:\Users\soham\AppData\Roaming\nltk_data...
[nltk_data]   Unzipping corpora\wordnet.zip.
review_tf = tfidf_vect.fit_transform(yelp_df.text)_____no_output_____# Numpy arrays are easy to work with, so convert the result to an array# Numpy
review_tf_nd = review_tf.toarray()
review_tf_nd.shape_____no_output_____# Take a look at how the Document Term Matrix looks
df = pd.DataFrame(review_tf_nd, columns=tfidf_vect.get_feature_names())
df.head(10)_____no_output_____# let's look at the words in the vocabulary, first 100 and last 100
vocab = tfidf_vect.get_feature_names()
print(vocab[:100]) # First 100 words
print(vocab[-100:]) # Last 100 words
#print(vocab)  # all 18k words['aaa', 'aaaamazing', 'aaammmazzing', 'aaron', 'aback', 'abandoned', 'abandoning', 'abba', 'abbreviate', 'abbreviated', 'abby', 'abc', 'abdomen', 'abhor', 'abiding', 'ability', 'abilty', 'able', 'abodoba', 'abou', 'abound', 'abrasion', 'abroad', 'abrupt', 'absence', 'absent', 'absinthe', 'absoloutely', 'absolut', 'absolute', 'absolutely', 'absolutley', 'absolutly', 'absorb', 'absorption', 'abstain', 'abstained', 'abstract', 'absurd', 'abuelo', 'abundance', 'abundant', 'abuse', 'abused', 'abusive', 'abysmal', 'acacia', 'academy', 'acai', 'acapulco', 'accent', 'accented', 'accept', 'acceptable', 'accepted', 'accepting', 'accepts', 'access', 'accessed', 'accessible', 'accessorize', 'accessory', 'accident', 'accidental', 'accidentally', 'accidently', 'acclaimed', 'accolade', 'accommodate', 'accommodated', 'accommodates', 'accommodating', 'accommodation', 'accomodate', 'accomodated', 'accomodating', 'accompanied', 'accompanies', 'accompaniment', 'accompany', 'accompanying', 'accomplish', 'accomplished', 'accomplishment', 'according', 'accordingly', 'account', 'accountable', 'accoutrement', 'accredited', 'accross', 'accumulated', 'accurate', 'accurately', 'accusation', 'accused', 'accustom', 'accustomed', 'accutemp', 'ace']
['youngtown', 'youre', 'youth', 'youthful', 'youtube', 'yow', 'yowza', 'yqqtg', 'yr', 'yragui', 'yuck', 'yucky', 'yuk', 'yukgejang', 'yukon', 'yum', 'yuma', 'yumm', 'yummie', 'yummier', 'yumminess', 'yummm', 'yummmayyyy', 'yummmm', 'yummmmm', 'yummmmmm', 'yummmmmmers', 'yummmmmmmm', 'yummmmy', 'yummo', 'yummy', 'yumness', 'yung', 'yup', 'yupha', 'yuppie', 'yur', 'yuri', 'yusefs', 'yuukk', 'yuuuummmmae', 'yuuuuummmmmyyy', 'yuuuuuuum', 'yuyuyummy', 'yuzu', 'yyyeeaahhhh', 'yyyyy', 'zabba', 'zach', 'zam', 'zanella', 'zankou', 'zappos', 'zatsiki', 'zen', 'zero', 'zest', 'zesty', 'zexperience', 'zha', 'zhou', 'zia', 'zichini', 'zihuatenejo', 'zilch', 'zillion', 'zin', 'zinburger', 'zinburgergeist', 'zinc', 'zinfandel', 'zing', 'zip', 'zipcar', 'ziploc', 'zipper', 'zipps', 'ziti', 'zoe', 'zombi', 'zombie', 'zone', 'zoners', 'zoning', 'zoo', 'zoom', 'zoyo', 'zse', 'zucca', 'zucchini', 'zuccini', 'zuchinni', 'zumba', 'zupa', 'zupas', 'zuzu', 'zuzus', 'zwftv', 'zwiebel', 'zzed']
# calculate frequency distribution of words in positive reviews
dist = np.sum(review_tf_nd[np.where(yelp_df['stars'] == 5)], axis=0)

counter = Counter()
# For each, print the vocabulary word and the number of times it appears in the data set
for tag, count in zip(vocab, dist):
    counter[tag] = count
    _____no_output_____counter.most_common(20)_____no_output_____d ={'great':131.03597041898036,
 'place': 113.05547307078189,
'food': 102.21339366444383,
 'good': 93.68256495294435,
 'love': 91.23321334839521,
 'best': 74.82220932390386,
 'time': 72.77568672447293,
'service': 67.43993748869981,
 'always': 65.72706284787824,
 'one': 64.33464059443395,
 'get': 62.39477801529317,
 'like': 60.71231744846042,
 'amazing': 51.88201697906908,
  'really': 51.64007315200254,

     'friendly': 50.56508924263671,
 'delicious': 50.047550206690346,
 'also': 48.85090962336725,
 'staff': 47.197221854488994,
 'back': 46.30183075093126,
 'pizza': 45.17805934866678}_____no_output_____wordcloud = WordCloud().fit_words(d)
plt.axis("off")
plt.imshow(wordcloud)_____no_output_____# calculate frequency distribution of words in negative reviews
dist = np.sum(review_tf_nd[np.where(yelp_df['stars'] == 1)], axis=0)

counter = Counter()
# For each, print the vocabulary word and the number of times it appears in the data set
for tag, count in zip(vocab, dist):
    counter[tag] = count
    _____no_output_____counter.most_common(20)_____no_output_____d2 = {'food': 27.413376715920993, 'place': 21.020716023812756, 'service': 19.19022599799789, 'like': 18.700208054365056, 'time': 17.19917295246528, 'one': 16.521516808044964, 'get': 15.69508701284176, 'back': 15.628870302002532, 'would': 15.558851868933054, 'minute': 14.783391134762052, 'bad': 14.585656469358277, 'never': 14.217830698718352, 'even': 13.99430547215625, 'good': 13.370943161933187, 'horrible': 12.458647900787206, 'ordered': 12.313456479282019, 'order': 12.164459042667001, 'told': 11.916779912824232, 'customer': 11.914305719747084, 'table': 11.759847045295585}_____no_output_____wordcloud = WordCloud().fit_words(d2)
plt.axis("off")
plt.imshow(wordcloud)_____no_output_____# split Train and Validation Data Set
X_train, X_test, y_train, y_test  = train_test_split(review_tf_nd, yelp_df['stars'], train_size=0.75)_____no_output_____from sklearn.linear_model import LogisticRegression
# Create and train the Logistic Regression Classifier
log_tfidf_model = LogisticRegression()
log_tfidf_model = log_tfidf_model.fit(X=X_train, y=y_train)_____no_output_____# Use the classifier to label our evaluation set. We can use either predict for classes or predict_proba for probabilities.
y_pred = log_tfidf_model.predict(X=X_test)
y_pred_prob = log_tfidf_model.predict_proba(X=X_test)[:,1]_____no_output_____# compute precision and recall, or their combination (harmonic mean) F1:
from sklearn import metrics
print(metrics.classification_report(y_test, y_pred))             precision    recall  f1-score   support

          1       1.00      0.29      0.45       178
          5       0.87      1.00      0.93       844

avg / total       0.89      0.88      0.85      1022

print("accuracy: %0.6f" % metrics.accuracy_score(y_test, y_pred))accuracy: 0.875734
from pylab import rcParams
labels = ['5', '1']
confmat = metrics.confusion_matrix(y_test, y_pred)
fig, ax = plt.subplots(figsize=(3, 3))
ax.matshow(confmat, cmap=plt.cm.Blues, alpha=0.7)
for i in range(confmat.shape[0]):
    for j in range(confmat.shape[1]):
        ax.text(x=j, y=i, s=confmat[i, j], va='center', ha='center')
tick_marks = np.arange(len(labels))
plt.xticks(tick_marks, labels, rotation=45)
plt.yticks(tick_marks,labels)
plt.xlabel('predicted label')
plt.ylabel('true label')
plt.tight_layout()_____no_output_____

### 3.4 Comparing the the Bag-Of-Words and TF-IDF Models with each other_____no_output_____

fpr_tfidf, tpr_tfidf, thresholds = metrics.roc_curve(y_test, y_pred_prob, pos_label=5)
roc_auc_tfidf = metrics.auc(fpr_tfidf, tpr_tfidf)_____no_output_____# ROC curves of both the Classifiers
sns.set(style='whitegrid', context='notebook')
plt.xlim([-0.05, 1.05])
plt.ylim([-0.05, 1.05])
plt.plot([0, 1], [0, 1], '--', color=(0.6, 0.6, 0.6), label='NULL Accuracy')
plt.plot(fpr_tfidf, tpr_tfidf, lw=1, label='Logistic Regression TF-IDF (AUC = %0.6f)' % roc_auc_tfidf)
plt.plot(fpr_tf, tpr_tf, lw=1, label='Logistic Regression Bag-Of-Words (AUC = %0.6f)' % roc_auc_tf)
plt.title('ROC Sentiment Classifier')
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.legend(loc="lower right")
plt.grid(True)
_____no_output_____


# Notebook from Lucian-N/DataScience
Path: PCA.ipynb

# Principal Component Analysis_____no_output_____PCA is a dimensionality reduction technique; it lets you distill multi-dimensional data down to fewer dimensions, selecting new dimensions that preserve variance in the data as best it can.

We're not talking about Star Trek stuff here; let's make it real - a black & white image for example, contains three dimensions of data: X position, Y position, and brightness at each point. Distilling that down to two dimensions can be useful for things like image compression and facial recognition, because it distills out the information that contributes most to the variance in the data set.

Let's do this with a simpler example: the Iris data set that comes with scikit-learn. It's just a small collection of data that has four dimensions of data for three different kinds of Iris flowers: The length and width of both the petals and sepals of many individual flowers from each species. Let's load it up and have a look:_____no_output_____

from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
import pylab as pl
from itertools import cycle

iris = load_iris()

numSamples, numFeatures = iris.data.shape
print(numSamples)
print(numFeatures)
print(list(iris.target_names))150
4
['setosa', 'versicolor', 'virginica']

So, this tells us our data set has 150 samples (individual flowers) in it. It has 4 dimensions - called features here, and three distinct Iris species that each flower is classified into.

While we can visualize 2 or even 3 dimensions of data pretty easily, visualizing 4D data isn't something our brains can do. So let's distill this down to 2 dimensions, and see how well it works:_____no_output_____

X = iris.data
pca = PCA(n_components=1, whiten=True).fit(X)
X_pca = pca.transform(X)_____no_output_____

What we have done is distill our 4D data set down to 2D, by projecting it down to two orthogonal 4D vectors that make up the basis of our new 2D projection. We can see what those 4D vectors are, although it's not something you can really wrap your head around:_____no_output_____

print(pca.components_)[[ 0.36158968 -0.08226889  0.85657211  0.35884393]]

Let's see how much information we've managed to preserve:_____no_output_____

print(pca.explained_variance_ratio_)
print(sum(pca.explained_variance_ratio_))[ 0.92461621]
0.924616207174

That's pretty cool. Although we have thrown away two of our four dimensions, PCA has chosen the remaining two dimensions well enough that we've captured 92% of the variance in our data in a single dimension alone! The second dimension just gives us an additional 5%; altogether we've only really lost less than 3% of the variance in our data by projecting it down to two dimensions.

As promised, now that we have a 2D representation of our data, we can plot it:_____no_output_____

%matplotlib inline
from pylab import *

colors = cycle('rgb')
target_ids = range(len(iris.target_names))
pl.figure()
for i, c, label in zip(target_ids, colors, iris.target_names):
    pl.scatter(X_pca[iris.target == i, 0], X_pca[iris.target == i, 1],
        c=c, label=label)
pl.legend()
pl.show()
    _____no_output_____

You can see the three different types of Iris are still clustered pretty well. If you think about it, this probably works well because the overall size of an individual flower probably makes both the petal and sepal sizes increase by a similar amount. Although the actual numbers on this graph have no intuitive meaning, what we're probably seeing is measure of the ratio of width to height for petals and sepals - and PCA distilled our data down to that on its own._____no_output_____## Activity_____no_output_____Our results suggest we could actually distill this data down to a single dimension and still preserve most of its variance. Try it! Do a PCA down to one component, and measure the results._____no_output_____


# Notebook from sgspurs/sklearn
Path: notebooks/02.2-Basic-Principles.ipynb

This notebook was put together by [Jake Vanderplas](http://www.vanderplas.com). Source and license info is on [GitHub](https://github.com/jakevdp/sklearn_tutorial/)._____no_output_____# Basic Principles of Machine Learning_____no_output_____Here we'll dive into the basic principles of machine learning, and how to
utilize them via the Scikit-Learn API.

After briefly introducing scikit-learn's *Estimator* object, we'll cover **supervised learning**, including *classification* and *regression* problems, and **unsupervised learning**, including *dimensinoality reduction* and *clustering* problems._____no_output_____

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

# use seaborn for plot defaults
# this can be safely commented out
import seaborn; seaborn.set()_____no_output_____

## The Scikit-learn Estimator Object

Every algorithm is exposed in scikit-learn via an ''Estimator'' object. For instance a linear regression is implemented as so:_____no_output_____

from sklearn.linear_model import LinearRegression_____no_output_____

**Estimator parameters**: All the parameters of an estimator can be set when it is instantiated, and have suitable default values:_____no_output_____

model = LinearRegression(normalize=True)
print(model.normalize)True
print(model)LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=True)

**Estimated Model parameters**: When data is *fit* with an estimator, parameters are estimated from the data at hand. All the estimated parameters are attributes of the estimator object ending by an underscore:_____no_output_____

x = np.arange(10)
y = 2 * x + 1_____no_output_____print(x)
print(y)[0 1 2 3 4 5 6 7 8 9]
[ 1  3  5  7  9 11 13 15 17 19]
plt.plot(x, y, 'o');_____no_output_____# The input data for sklearn is 2D: (samples == 10 x features == 1)
X = x[:, np.newaxis]
print(X)
print(y)[[0]
 [1]
 [2]
 [3]
 [4]
 [5]
 [6]
 [7]
 [8]
 [9]]
[ 1  3  5  7  9 11 13 15 17 19]
# fit the model on our data
model.fit(X, y)_____no_output_____# underscore at the end indicates a fit parameter
print(model.coef_)
print(model.intercept_)[ 2.]
1.0
# residual error around fit
model.residues______no_output_____

The model found a line with a slope 2 and intercept 1, as we'd expect._____no_output_____## Supervised Learning: Classification and Regression

In **Supervised Learning**, we have a dataset consisting of both features and labels.
The task is to construct an estimator which is able to predict the label of an object
given the set of features. A relatively simple example is predicting the species of
iris given a set of measurements of its flower. This is a relatively simple task.
Some more complicated examples are:

- given a multicolor image of an object through a telescope, determine
  whether that object is a star, a quasar, or a galaxy.
- given a photograph of a person, identify the person in the photo.
- given a list of movies a person has watched and their personal rating
  of the movie, recommend a list of movies they would like
  (So-called *recommender systems*: a famous example is the [Netflix Prize](http://en.wikipedia.org/wiki/Netflix_prize)).

What these tasks have in common is that there is one or more unknown
quantities associated with the object which needs to be determined from other
observed quantities.

Supervised learning is further broken down into two categories, **classification** and **regression**.
In classification, the label is discrete, while in regression, the label is continuous. For example,
in astronomy, the task of determining whether an object is a star, a galaxy, or a quasar is a
classification problem: the label is from three distinct categories. On the other hand, we might
wish to estimate the age of an object based on such observations: this would be a regression problem,
because the label (age) is a continuous quantity._____no_output_____### Classification Example
K nearest neighbors (kNN) is one of the simplest learning strategies: given a new, unknown observation, look up in your reference database which ones have the closest features and assign the predominant class.

Let's try it out on our iris classification problem:_____no_output_____

from sklearn import neighbors, datasets

iris = datasets.load_iris()
X, y = iris.data, iris.target

# create the model
knn = neighbors.KNeighborsClassifier(n_neighbors=5)

# fit the model
knn.fit(X, y)

# What kind of iris has 3cm x 5cm sepal and 4cm x 2cm petal?
# call the "predict" method:
result = knn.predict([[3, 5, 4, 2],])

print(iris.target_names[result])['versicolor']

You can also do probabilistic predictions:_____no_output_____

knn.predict_proba([[3, 5, 4, 2],])_____no_output_____from fig_code import plot_iris_knn
plot_iris_knn()/Users/jakevdp/anaconda/envs/python3.4/lib/python3.4/site-packages/matplotlib/collections.py:590: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  if self._edgecolors == str('face'):

---

#### Exercise

Use a different estimator on the same problem: ``sklearn.svm.SVC``.

*Note that you don't have to know what it is do use it. We're simply trying out the interface here*

*If you finish early, try to create a similar plot as above with the SVC estimator.*_____no_output_____

from sklearn.svm import SVC_____no_output_____

---_____no_output_____### Regression Example

One of the simplest regression problems is fitting a line to data, which we saw above.
Scikit-learn also contains more sophisticated regression algorithms_____no_output_____

# Create some simple data
import numpy as np
np.random.seed(0)
X = np.random.random(size=(20, 1))
y = 3 * X.squeeze() + 2 + np.random.randn(20)

plt.plot(X.squeeze(), y, 'o');_____no_output_____

As above, we can plot a line of best fit:_____no_output_____

model = LinearRegression()
model.fit(X, y)

# Plot the data and the model prediction
X_fit = np.linspace(0, 1, 100)[:, np.newaxis]
y_fit = model.predict(X_fit)

plt.plot(X.squeeze(), y, 'o')
plt.plot(X_fit.squeeze(), y_fit);_____no_output_____

Scikit-learn also has some more sophisticated models, which can respond to finer features in the data:_____no_output_____

# Fit a Random Forest
from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor()
model.fit(X, y)

# Plot the data and the model prediction
X_fit = np.linspace(0, 1, 100)[:, np.newaxis]
y_fit = model.predict(X_fit)

plt.plot(X.squeeze(), y, 'o')
plt.plot(X_fit.squeeze(), y_fit);_____no_output_____

Whether either of these is a "good" fit or not depends on a number of things; we'll discuss details of how to choose a model later in the tutorial._____no_output_____---

#### Exercise

Explore the ``RandomForestRegressor`` object using IPython's help features (i.e. put a question mark after the object).
What arguments are available to ``RandomForestRegressor``?
How does the above plot change if you change these arguments?

These class-level arguments are known as *hyperparameters*, and we will discuss later how you to select hyperparameters in the model validation section.

---_____no_output_____## Unsupervised Learning: Dimensionality Reduction and Clustering

**Unsupervised Learning** addresses a different sort of problem. Here the data has no labels,
and we are interested in finding similarities between the objects in question. In a sense,
you can think of unsupervised learning as a means of discovering labels from the data itself.
Unsupervised learning comprises tasks such as *dimensionality reduction*, *clustering*, and
*density estimation*. For example, in the iris data discussed above, we can used unsupervised
methods to determine combinations of the measurements which best display the structure of the
data. As we'll see below, such a projection of the data can be used to visualize the
four-dimensional dataset in two dimensions. Some more involved unsupervised learning problems are:

- given detailed observations of distant galaxies, determine which features or combinations of
  features best summarize the information.
- given a mixture of two sound sources (for example, a person talking over some music),
  separate the two (this is called the [blind source separation](http://en.wikipedia.org/wiki/Blind_signal_separation) problem).
- given a video, isolate a moving object and categorize in relation to other moving objects which have been seen.

Sometimes the two may even be combined: e.g. Unsupervised learning can be used to find useful
features in heterogeneous data, and then these features can be used within a supervised
framework._____no_output_____### Dimensionality Reduction: PCA

Principle Component Analysis (PCA) is a dimension reduction technique that can find the combinations of variables that explain the most variance.

Consider the iris dataset. It cannot be visualized in a single 2D plot, as it has 4 features. We are going to extract 2 combinations of sepal and petal dimensions to visualize it:_____no_output_____

X, y = iris.data, iris.target

from sklearn.decomposition import PCA
pca = PCA(n_components=0.95)
pca.fit(X)
X_reduced = pca.transform(X)
print("Reduced dataset shape:", X_reduced.shape)Reduced dataset shape: (150, 2)
import pylab as plt
plt.scatter(X_reduced[:, 0], X_reduced[:, 1], c=y,
           cmap='RdYlBu')

print("Meaning of the 2 components:")
for component in pca.components_:
    print(" + ".join("%.3f x %s" % (value, name)
                     for value, name in zip(component,
                                            iris.feature_names)))Meaning of the 2 components:
0.362 x sepal length (cm) + -0.082 x sepal width (cm) + 0.857 x petal length (cm) + 0.359 x petal width (cm)
-0.657 x sepal length (cm) + -0.730 x sepal width (cm) + 0.176 x petal length (cm) + 0.075 x petal width (cm)

#### Clustering: K-means

Clustering groups together observations that are homogeneous with respect to a given criterion, finding ''clusters'' in the data.

Note that these clusters will uncover relevent hidden structure of the data only if the criterion used highlights it._____no_output_____

from sklearn.cluster import KMeans
k_means = KMeans(n_clusters=3, random_state=0) # Fixing the RNG in kmeans
k_means.fit(X)
y_pred = k_means.predict(X)

plt.scatter(X_reduced[:, 0], X_reduced[:, 1], c=y_pred,
           cmap='RdYlBu');/Users/jakevdp/anaconda/envs/python3.4/lib/python3.4/site-packages/matplotlib/collections.py:590: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  if self._edgecolors == str('face'):

### Recap: Scikit-learn's estimator interface

Scikit-learn strives to have a uniform interface across all methods,
and we'll see examples of these below. Given a scikit-learn *estimator*
object named `model`, the following methods are available:

- Available in **all Estimators**
  + `model.fit()` : fit training data. For supervised learning applications,
    this accepts two arguments: the data `X` and the labels `y` (e.g. `model.fit(X, y)`).
    For unsupervised learning applications, this accepts only a single argument,
    the data `X` (e.g. `model.fit(X)`).
- Available in **supervised estimators**
  + `model.predict()` : given a trained model, predict the label of a new set of data.
    This method accepts one argument, the new data `X_new` (e.g. `model.predict(X_new)`),
    and returns the learned label for each object in the array.
  + `model.predict_proba()` : For classification problems, some estimators also provide
    this method, which returns the probability that a new observation has each categorical label.
    In this case, the label with the highest probability is returned by `model.predict()`.
  + `model.score()` : for classification or regression problems, most (all?) estimators implement
    a score method.  Scores are between 0 and 1, with a larger score indicating a better fit.
- Available in **unsupervised estimators**
  + `model.predict()` : predict labels in clustering algorithms.
  + `model.transform()` : given an unsupervised model, transform new data into the new basis.
    This also accepts one argument `X_new`, and returns the new representation of the data based
    on the unsupervised model.
  + `model.fit_transform()` : some estimators implement this method,
    which more efficiently performs a fit and a transform on the same input data._____no_output_____## Model Validation

An important piece of machine learning is **model validation**: that is, determining how well your model will generalize from the training data to future unlabeled data. Let's look at an example using the *nearest neighbor classifier*. This is a very simple classifier: it simply stores all training data, and for any unknown quantity, simply returns the label of the closest training point.

With the iris data, it very easily returns the correct prediction for each of the input points:_____no_output_____

from sklearn.neighbors import KNeighborsClassifier
X, y = iris.data, iris.target
clf = KNeighborsClassifier(n_neighbors=1)
clf.fit(X, y)
y_pred = clf.predict(X)
print(np.all(y == y_pred))True

A more useful way to look at the results is to view the **confusion matrix**, or the matrix showing the frequency of inputs and outputs:_____no_output_____

from sklearn.metrics import confusion_matrix
print(confusion_matrix(y, y_pred))[[50  0  0]
 [ 0 50  0]
 [ 0  0 50]]

For each class, all 50 training samples are correctly identified. But this **does not mean that our model is perfect!** In particular, such a model generalizes extremely poorly to new data. We can simulate this by splitting our data into a *training set* and a *testing set*. Scikit-learn contains some convenient routines to do this:_____no_output_____

from sklearn.cross_validation import train_test_split
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y)
clf.fit(Xtrain, ytrain)
ypred = clf.predict(Xtest)
print(confusion_matrix(ytest, ypred))[[12  0  0]
 [ 0  6  0]
 [ 0  2 18]]

This paints a better picture of the true performance of our classifier: apparently there is some confusion between the second and third species, which we might anticipate given what we've seen of the data above.

This is why it's **extremely important** to use a train/test split when evaluating your models.  We'll go into more depth on model evaluation later in this tutorial._____no_output_____## Flow Chart: How to Choose your Estimator

This is a flow chart created by scikit-learn super-contributor [Andreas Mueller](https://github.com/amueller) which gives a nice summary of which algorithms to choose in various situations. Keep it around as a handy reference!_____no_output_____

from IPython.display import Image
Image("http://scikit-learn.org/dev/_static/ml_map.png")_____no_output_____

Original source on the [scikit-learn website](http://scikit-learn.org/stable/tutorial/machine_learning_map/)_____no_output_____## Quick Application: Optical Character Recognition_____no_output_____To demonstrate the above principles on a more interesting problem, let's consider OCR (Optical Character Recognition) – that is, recognizing hand-written digits.
In the wild, this problem involves both locating and identifying characters in an image. Here we'll take a shortcut and use scikit-learn's set of pre-formatted digits, which is built-in to the library._____no_output_____### Loading and visualizing the digits data

We'll use scikit-learn's data access interface and take a look at this data:_____no_output_____

from sklearn import datasets
digits = datasets.load_digits()
digits.images.shape_____no_output_____

Let's plot a few of these:_____no_output_____

fig, axes = plt.subplots(10, 10, figsize=(8, 8))
fig.subplots_adjust(hspace=0.1, wspace=0.1)

for i, ax in enumerate(axes.flat):
    ax.imshow(digits.images[i], cmap='binary', interpolation='nearest')
    ax.text(0.05, 0.05, str(digits.target[i]),
            transform=ax.transAxes, color='green')
    ax.set_xticks([])
    ax.set_yticks([])_____no_output_____

Here the data is simply each pixel value within an 8x8 grid:_____no_output_____

# The images themselves
print(digits.images.shape)
print(digits.images[0])(1797, 8, 8)
[[  0.   0.   5.  13.   9.   1.   0.   0.]
 [  0.   0.  13.  15.  10.  15.   5.   0.]
 [  0.   3.  15.   2.   0.  11.   8.   0.]
 [  0.   4.  12.   0.   0.   8.   8.   0.]
 [  0.   5.   8.   0.   0.   9.   8.   0.]
 [  0.   4.  11.   0.   1.  12.   7.   0.]
 [  0.   2.  14.   5.  10.  12.   0.   0.]
 [  0.   0.   6.  13.  10.   0.   0.   0.]]
# The data for use in our algorithms
print(digits.data.shape)
print(digits.data[0])(1797, 64)
[  0.   0.   5.  13.   9.   1.   0.   0.   0.   0.  13.  15.  10.  15.   5.
   0.   0.   3.  15.   2.   0.  11.   8.   0.   0.   4.  12.   0.   0.   8.
   8.   0.   0.   5.   8.   0.   0.   9.   8.   0.   0.   4.  11.   0.   1.
  12.   7.   0.   0.   2.  14.   5.  10.  12.   0.   0.   0.   0.   6.  13.
  10.   0.   0.   0.]
# The target label
print(digits.target)[0 1 2 ..., 8 9 8]

So our data have 1797 samples in 64 dimensions._____no_output_____### Unsupervised Learning: Dimensionality Reduction

We'd like to visualize our points within the 64-dimensional parameter space, but it's difficult to plot points in 64 dimensions!
Instead we'll reduce the dimensions to 2, using an unsupervised method.
Here, we'll make use of a manifold learning algorithm called *Isomap*, and transform the data to two dimensions._____no_output_____

from sklearn.manifold import Isomap_____no_output_____iso = Isomap(n_components=2)
data_projected = iso.fit_transform(digits.data)_____no_output_____data_projected.shape_____no_output_____plt.scatter(data_projected[:, 0], data_projected[:, 1], c=digits.target,
            edgecolor='none', alpha=0.5, cmap=plt.cm.get_cmap('nipy_spectral', 10));
plt.colorbar(label='digit label', ticks=range(10))
plt.clim(-0.5, 9.5)/Users/jakevdp/anaconda/envs/python3.4/lib/python3.4/site-packages/matplotlib/collections.py:590: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  if self._edgecolors == str('face'):

We see here that the digits are fairly well-separated in the parameter space; this tells us that a supervised classification algorithm should perform fairly well. Let's give it a try._____no_output_____### Classification on Digits

Let's try a classification task on the digits. The first thing we'll want to do is split the digits into a training and testing sample:_____no_output_____

from sklearn.cross_validation import train_test_split
Xtrain, Xtest, ytrain, ytest = train_test_split(digits.data, digits.target,
                                                random_state=2)
print(Xtrain.shape, Xtest.shape)(1347, 64) (450, 64)

Let's use a simple logistic regression which (despite its confusing name) is a classification algorithm:_____no_output_____

from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(penalty='l2')
clf.fit(Xtrain, ytrain)
ypred = clf.predict(Xtest)_____no_output_____

We can check our classification accuracy by comparing the true values of the test set to the predictions:_____no_output_____

from sklearn.metrics import accuracy_score
accuracy_score(ytest, ypred)_____no_output_____

This single number doesn't tell us **where** we've gone wrong: one nice way to do this is to use the *confusion matrix*_____no_output_____

from sklearn.metrics import confusion_matrix
print(confusion_matrix(ytest, ypred))[[42  0  0  0  0  0  0  0  0  0]
 [ 0 45  0  1  0  0  0  0  3  1]
 [ 0  0 47  0  0  0  0  0  0  0]
 [ 0  0  0 42  0  2  0  3  1  0]
 [ 0  2  0  0 36  0  0  0  1  1]
 [ 0  0  0  0  0 52  0  0  0  0]
 [ 0  0  0  0  0  0 42  0  1  0]
 [ 0  0  0  0  0  0  0 48  1  0]
 [ 0  2  0  0  0  0  0  0 38  0]
 [ 0  0  0  1  0  1  0  1  2 34]]
plt.imshow(np.log(confusion_matrix(ytest, ypred)),
           cmap='Blues', interpolation='nearest')
plt.grid(False)
plt.ylabel('true')
plt.xlabel('predicted');_____no_output_____

We might also take a look at some of the outputs along with their predicted labels. We'll make the bad labels red:_____no_output_____

fig, axes = plt.subplots(10, 10, figsize=(8, 8))
fig.subplots_adjust(hspace=0.1, wspace=0.1)

for i, ax in enumerate(axes.flat):
    ax.imshow(Xtest[i].reshape(8, 8), cmap='binary')
    ax.text(0.05, 0.05, str(ypred[i]),
            transform=ax.transAxes,
            color='green' if (ytest[i] == ypred[i]) else 'red')
    ax.set_xticks([])
    ax.set_yticks([])_____no_output_____

The interesting thing is that even with this simple logistic regression algorithm, many of the mislabeled cases are ones that we ourselves might get wrong!

There are many ways to improve this classifier, but we're out of time here. To go further, we could use a more sophisticated model, use cross validation, or apply other techniques.
We'll cover some of these topics later in the tutorial._____no_output_____


# Notebook from glotzerlab/hoomd-examples
Path: 01-Introducing-Molecular-Dynamics/03-Compressing-the-System.ipynb

# Compressing the System

## Overview

### Questions

* How do I compress the system to a target density?

### Objectives

* Describe how **Variants** describe values that vary with the timestep.
* Show how to use **BoxResize** with a **Ramp variant** to slowly compress the system to the target volume.
* Demonstrate the crystallization of Lennard-Jones particles

## Boilerplate code_____no_output_____

import math

import fresnel
import hoomd
import matplotlib

%matplotlib inline
matplotlib.style.use('ggplot')_____no_output_____

The `render` function in the next (hidden) cell will render the system state using **fresnel**.

    This is not intended as a full tutorial on fresnel - see the fresnel user documentation if you would like to learn more.

_____no_output_____

import warnings

import fresnel
import IPython
import packaging.version

device = fresnel.Device()
tracer = fresnel.tracer.Path(device=device, w=300, h=300)

FRESNEL_MIN_VERSION = packaging.version.parse("0.13.0")
FRESNEL_MAX_VERSION = packaging.version.parse("0.14.0")

def render(snapshot):
    if ('version' not in dir(fresnel) or packaging.version.parse(
            fresnel.version.version) < FRESNEL_MIN_VERSION
            or packaging.version.parse(
                fresnel.version.version) >= FRESNEL_MAX_VERSION):
        warnings.warn(
            f"Unsupported fresnel version {fresnel.version.version} - expect errors."
        )
    L = snapshot.configuration.box[0]
    scene = fresnel.Scene(device)
    geometry = fresnel.geometry.Sphere(scene,
                                       N=len(snapshot.particles.position),
                                       radius=0.5)
    geometry.material = fresnel.material.Material(color=fresnel.color.linear(
        [252 / 255, 209 / 255, 1 / 255]),
                                                  roughness=0.5)
    geometry.position[:] = snapshot.particles.position[:]
    geometry.outline_width = 0.04
    box = fresnel.geometry.Box(scene, [L, L, L, 0, 0, 0], box_radius=.02)

    scene.lights = [
        fresnel.light.Light(direction=(0, 0, 1),
                            color=(0.8, 0.8, 0.8),
                            theta=math.pi),
        fresnel.light.Light(direction=(1, 1, 1),
                            color=(1.1, 1.1, 1.1),
                            theta=math.pi / 3)
    ]
    scene.camera = fresnel.camera.Orthographic(position=(L * 2, L, L * 2),
                                               look_at=(0, 0, 0),
                                               up=(0, 1, 0),
                                               height=L * 1.4 + 1)
    scene.background_color = (1, 1, 1)
    return IPython.display.Image(tracer.sample(scene, samples=500)._repr_png_())_____no_output_____

## Procedure

`random.gsd`, generated by the previous section of this tutorial has particles randomly placed in a low density fluid state.
To find the crystalline state, you need to compress the system to a higher density.
If you were to immediately scale the system to the target density, the particle's hard cores would overlap and the integrator would be become unstable.
A more effective strategy gradually resizes the simulation box a little at a time over the course of a simulation, allowing any slight overlaps to relax._____no_output_____## Initialize the simulation

Configure this simulation to run on the CPU._____no_output_____

cpu = hoomd.device.CPU()
sim = hoomd.Simulation(device=cpu)
sim.create_state_from_gsd(filename='random.gsd')_____no_output_____

Set up the molecular dynamics simulation, as discussed in the previous sections of this tutorial:_____no_output_____

integrator = hoomd.md.Integrator(dt=0.005)
cell = hoomd.md.nlist.Cell()
lj = hoomd.md.pair.LJ(nlist=cell)
lj.params[('A', 'A')] = dict(epsilon=1, sigma=1)
lj.r_cut[('A', 'A')] = 2.5
integrator.forces.append(lj)
nvt = hoomd.md.methods.NVT(kT=1.5, filter=hoomd.filter.All(), tau=1.0)
integrator.methods.append(nvt)
sim.operations.integrator = integrator_____no_output_____

## Variants

The **BoxResize** class performs scales the simulation box.
You provide a **Variant** that tells **BoxResize** how to interpolate between two boxes, `box1` and `box2`, over time.
A **Variant** is a scalar-valued function of the simulation timestep.
**BoxResize** sets `box1` when the function is at its minimum, `box2` when the function is at its maximum, and linearly interpolates between the two boxes for intermediate values.

Use the **Ramp variant** to define a function that ramps from one value to another over a given number of steps.
Note that this simulation does not start at timestep 0, `random.gsd` was saved after running a number of steps to randomize the system in the previous section:_____no_output_____

sim.timestep_____no_output_____

Create a **Ramp** that starts at this step and ends 100,000 steps later:_____no_output_____

ramp = hoomd.variant.Ramp(A=0, B=1, t_start=sim.timestep, t_ramp=20000)_____no_output_____

Here is what the ramp looks like as a function of timestep:_____no_output_____

steps = range(0, 40000, 20)
y = [ramp(step) for step in steps]

fig = matplotlib.figure.Figure(figsize=(10, 6.18))
ax = fig.add_subplot()
ax.plot(steps, y)
ax.set_xlabel('timestep')
ax.set_ylabel('ramp')
fig_____no_output_____

## Gradually resize the simulation box

Now, you need to determine the initial simulation box `box1` and final simulation box at the target density `box2`.
This is the initial number density $\rho$ of the current simulation box:_____no_output_____

rho = sim.state.N_particles / sim.state.box.volume
rho_____no_output_____

Create a new **Box** at the target number density:_____no_output_____

initial_box = sim.state.box
final_box = hoomd.Box.from_box(initial_box)  # make a copy of initial_box
final_rho = 1.2
final_box.volume = sim.state.N_particles / final_rho_____no_output_____

To avoid destabilizing the integrator with large forces due to large box size changes, scale the box with a small period._____no_output_____

box_resize_trigger = hoomd.trigger.Periodic(10)_____no_output_____

Construct the **BoxResize updater** and add it to the simulation._____no_output_____

box_resize = hoomd.update.BoxResize(box1=initial_box,
                                    box2=final_box,
                                    variant=ramp,
                                    trigger=box_resize_trigger)
sim.operations.updaters.append(box_resize)_____no_output_____

Run the simulation to compress the box._____no_output_____

sim.run(10001)_____no_output_____

At the half way point of the ramp, the variant is 0.5 and the current box lengths are half way between the initial and final ones:_____no_output_____

ramp(sim.timestep - 1)_____no_output_____current_box = sim.state.box
(current_box.Lx - initial_box.Lx) / (final_box.Lx - initial_box.Lx)_____no_output_____

Finish the rest of the compression:_____no_output_____

sim.run(10000)_____no_output_____

The current box matches the final box:_____no_output_____

sim.state.box == final_box_____no_output_____

The system is at a higher density:_____no_output_____

render(sim.state.get_snapshot())_____no_output_____

Now that the system has reached the target volume, remove the **BoxResize** updater from the simulation:_____no_output_____

sim.operations.updaters.remove(box_resize)_____no_output_____

## Equilibrating the system

Run the simulation for a longer time and see if it self assembles the ordered fcc structure:

This cell may require several minutes to complete.

_____no_output_____

sim.run(5e5)_____no_output_____

Here is what the final state of the system looks like:_____no_output_____

render(sim.state.get_snapshot())_____no_output_____

## Analyzing the simulation results

Is this the fcc structure? Did the system reach equilibrium? These are topics covered in the Introducing HOOMD-blue tutorial. Consider it an exercise to modify this example and implement that analysis here. You will need to save a GSD file with the simulation trajectory, then use freud's **SolidLiquid** order parameter to identify when the ordered structure appears._____no_output_____This is the end of the introducing molecular dynamics tutorial! It described molecular dynamics simulations, demonstrated how to initialize, randomize, compress, and equilibrate a system of Lennard-Jones particles. See the other HOOMD-blue tutorials to learn about other concepts, or browse the reference documentation for more information._____no_output_____[Previous section](02-Initializing-a-Random-System.ipynb)._____no_output_____


# Notebook from rvmuthukumar/Hands-On-Data-Analysis-with-Pandas
Path: solutions/ch_09/exercise_2.ipynb

# Predicting Star Temperature
Using the Open Exoplanet Catalogue database: https://github.com/OpenExoplanetCatalogue/open_exoplanet_catalogue/

## Data License
Copyright (C) 2012 Hanno Rein

Permission is hereby granted, free of charge, to any person obtaining a copy of this database and associated scripts (the "Database"), to deal in the Database without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Database, and to permit persons to whom the Database is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Database. A reference to the Database shall be included in all scientific publications that make use of the Database.

THE DATABASE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE DATABASE OR THE USE OR OTHER DEALINGS IN THE DATABASE._____no_output_____

<code>
%matplotlib inline

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

stars = pd.read_csv('../../ch_09/data/stars.csv')
stars.head()_____no_output_____from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

data = stars[['metallicity', 'temperature', 'magJ', 'radius', 'magB', 'magV', 'magK', 'mass', 'planets']].dropna()
y = data.pop('temperature')
X = data

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0
)

lm = LinearRegression().fit(X_train, y_train)
lm.score(X_test, y_test) # R-squared_____no_output_____from sklearn.metrics import mean_squared_error
mean_squared_error(y_test, lm.predict(X_test))_____no_output_____[(coef, feature) for coef, feature in zip(lm.coef_, X_train.columns)]_____no_output_____lm.intercept______no_output_____from ml_utils.regression import plot_residuals
plot_residuals(y_test, lm.predict(X_test))_____no_output_____
</code>


# Notebook from edeutsch/Community-Notebooks
Path: Notebooks/How_to_convert_10X_bam_to_fastq_scRNA_seq.ipynb

# ISB-CGC Community Notebooks¶
Check out more notebooks at our [Community Notebooks Repository](https://github.com/isb-cgc/Community-Notebooks)!

```
Title:   How to convert 10X bams to fastq files
Author:  David L Gibbs
Created: 2019-08-07
Purpose: Demonstrate how to make fastq files from 10X bams
Notes:
```_____no_output_____# Using 10X bamtofastq to convert a bam file to fastq files._____no_output_____In this example, we'll be using the Google Genomics Pipelines API. The pipelines API makes it easy to run a job without having to  spin up and shut down a VM. It's all done automatically.

The work is uses materials from https://cloud.google.com/genomics/docs/quickstart.

Docs for the genomics pipeline run: https://cloud.google.com/sdk/gcloud/reference/alpha/genomics/pipelines/run

For this to work, we need to make sure that the Google Genomics API is enabled. To do that, from the main menu in the cloud console, select 'APIs & Services'. The API is called: genomics.googleapis.com._____no_output_____
In this case, we're going to be using data and compiled software from 10X,
which is compatible with Ubuntu. Therefore, we'll select an
ubuntu image to run the pipeline.

Software can be found here:
https://support.10xgenomics.com/docs/bamtofastq

Data is found here:
https://support.10xgenomics.com/single-cell-gene-expression/datasets/3.0.0/pbmc_1k_protein_v3
_____no_output_____

# This notebook cell specifies the job parameters
# We are using an ubuntu docker image which doesn't
# contain the 'wget' program, so we install that,
# then use it to get software from 10X, which finally
# is used to convert the bam file to fastqs.
# everything in the outputPath will be copied back to
# cloud storage.

# note on the input files..
# the ${file} contains the full path.
# if there's multiple bam files to process
# we could use
# for file in $(ls /path/*.bam | sed s/^.*\\/\//) # to get the last one
# and we could write those fastq files in /mnt/data/output/{$file}

params='''
name: bamtofastq
description: Run 10X bamtofastq on bams

resources:
  zones:
  - us-west1-b

  disks:
  - name: datadisk
    autoDelete: True

    # Within the Docker container, specify a mount point for the disk.
    mountPoint: /mnt/data

docker:
  imageName: ubuntu:19.10

  # The Pipelines API does not create the output directory.
  cmd: >
    mkdir /mnt/data/output &&
    find /mnt/data/input &&
    for file in $(/bin/ls /mnt/data/input/*.bam); do
        apt-get update;
        apt-get --yes install wget;
        wget http://cf.10xgenomics.com/misc/bamtofastq;
        chmod +x bamtofastq;
        ./bamtofastq ${file} /mnt/data/output/fastq;
        tar -czvf /mnt/data/output/fastq.tar.gz /mnt/data/output/fastq;
    done

inputParameters:
- name: bamFile
  description: the sorted bam file
  localCopy:
    path: input/
    disk: datadisk
- name: bamIndex
  description: bam file index
  localCopy:
    path: input/
    disk: datadisk

outputParameters:
- name: outputPath
  description: Cloud Storage path for where bamtofastq writes
  localCopy:
    path: output/*
    disk: datadisk

'''

fout = open('10X_bamtofastq.yaml','w')
fout.write(params)
fout.close()_____no_output_____

See: https://cloud.google.com/sdk/gcloud/reference/alpha/genomics/pipelines/run  for more parameters.

_____no_output_____

# We're going to save this bucket location to our environment.
# Not necessary, but is useful in some cases.
%set_env TEMP=gs://cgc_scrnaseq_tempenv: TEMP=gs://cgc_scrnaseq_temp

***
The call to the pipelines API takes parameters pointing to the
inputs and output dir, locations to save log files, and can take
parameters that designate the compute environment, such as disk size
and number of processors.
***_____no_output_____

!gcloud alpha genomics pipelines run \
    --pipeline-file 10X_bamtofastq.yaml \
    --inputs bamFile=gs://cgc_bam_bucket_007/pbmc_1k_protein_v3_possorted_genome_bam.bam \
    --inputs bamIndex=gs://cgc_bam_bucket_007/pbmc_1k_protein_v3_possorted_genome_bam.bam.bai \
    --outputs outputPath=gs://cgc_bam_bucket_007/output/ \
    --logging "${TEMP}/10X_bamtofastq.log" \
    --disk-size datadisk:200Running [operations/EIrt7PTGLRj6xq_K45OovFsg0JGr1fcSKg9wcm9kdWN0aW9uUXVldWU].

***
Then you can check the status (describe), or cancel the job using the following command._____no_output_____

!gcloud alpha genomics operations wait EIrt7PTGLRj6xq_K45OovFsg0JGr1fcSKg9wcm9kdWN0aW9uUXVldWUWaiting for [operations/EIrt7PTGLRj6xq_K45OovFsg0JGr1fcSKg9wcm9kdWN0aW9uUXVldWU
]...done.