# Notebook from alexandrebouayad/Cirq
Path: docs/protocols.ipynb

##### Copyright 2020 The Cirq Developers_____no_output_____

#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License._____no_output_____

# Protocols_____no_output_____

    View on QuantumAI

    Run in Google Colab

    View source on GitHub

    Download notebook

_____no_output_____

try:
    import cirq
except ImportError:
    print("installing cirq...")
    !pip install --quiet cirq
    print("installed cirq.")
    import cirq _____no_output_____

# Introduction

Cirq's protocols are very similar concept to Python's built-in protocols that were introduced in [PEP 544](https://www.python.org/dev/peps/pep-0544/).
Python's built-in protocols are extremely convenient. For example, behind all the for loops and list comprehensions you can find the Iterator protocol.
As long as an object has the `__iter__()` magic method that returns an iterator object, it has iterator support.
An iterator object has to define `__iter__()` and `__next__()` magic methods, that defines the iterator protocol.
The `iter(val)` builtin function returns an iterator for `val` if it defines the above methods, otherwise throws a `TypeError`. Cirq protocols work similarly.

A canonical Cirq protocol example is the `unitary` protocol that allows to check the unitary matrix of values that support the protocol by calling `cirq.unitary(val)`._____no_output_____

print(cirq.X)
print("cirq.X unitary:\n", cirq.unitary(cirq.X))

a, b = cirq.LineQubit.range(2)
circuit = cirq.Circuit(cirq.X(a), cirq.Y(b))
print(circuit)
print("circuit unitary:\n", cirq.unitary(circuit))
_____no_output_____

When an object does not support a given protocol, an error is thrown._____no_output_____

try:
    print(cirq.unitary(a)) ## error!
except Exception as e:
    print("As expected, a qubit does not have a unitary. The error: ")
    print(e)
    _____no_output_____

## What is a protocol?

A protocol is a combination of the following two items:
- a `SupportsXYZ` class, which defines and documents all the magic functions that need to be implemented in order to support that given protocol
- the entrypoint function(s), which are exposed to the main cirq namespace as `cirq.xyz()`

Note: While the protocol is technically both of these things, we refer to the public utility functions interchangeably as protocols. See the list of them below.

## Cirq's protocols

For a complete list of Cirq protocols, refer to the `cirq.protocols` package.
Here we provide a list of frequently used protocols for debugging, simulation and testing.

| Protocol | Description |
|----------|-------|
|`cirq.act_on`| Allows an object (operations or gates) to act on a state, particularly within simulators. |
|`cirq.apply_channel`| High performance evolution under a channel evolution. |
|`cirq.apply_mixture`| High performance evolution under a mixture of unitaries evolution. |
|`cirq.apply_unitaries`| Apply a series of unitaries onto a state tensor. |
|`cirq.apply_unitary`| High performance left-multiplication of a unitary effect onto a tensor. |
|`cirq.approx_eq`| Approximately compares two objects. |
|`cirq.circuit_diagram_info`| Retrieves information for drawing operations within circuit diagrams. |
|`cirq.commutes`| Determines whether two values commute. |
|`cirq.control_keys`| Gets the keys that the value is classically controlled by. |
|`cirq.definitely_commutes`| Determines whether two values definitely commute. |
|`cirq.decompose`| Recursively decomposes a value into `cirq.Operation`s meeting a criteria. |
|`cirq.decompose_once`| Decomposes a value into operations, if possible. |
|`cirq.decompose_once_with_qubits`| Decomposes a value into operations on the given qubits. |
|`cirq.equal_up_to_global_phase`| Determine whether two objects are equal up to global phase. |
|`cirq.has_kraus`| Returns whether the value has a Kraus representation. |
|`cirq.has_mixture`| Returns whether the value has a mixture representation. |
|`cirq.has_stabilizer_effect`| Returns whether the input has a stabilizer effect. |
|`cirq.has_unitary`| Determines whether the value has a unitary effect. |
|`cirq.inverse`| Returns the inverse `val**-1` of the given value, if defined. |
|`cirq.is_measurement`| Determines whether or not the given value is a measurement. |
|`cirq.is_parameterized`| Returns whether the object is parameterized with any Symbols. |
|`cirq.kraus`| Returns a Kraus representation of the given channel. |
|`cirq.measurement_key`| Get the single measurement key for the given value. |
|`cirq.measurement_keys`| Gets the measurement keys of measurements within the given value. |
|`cirq.mixture`| Return a sequence of tuples representing a probabilistic unitary. |
|`cirq.num_qubits`| Returns the number of qubits, qudits, or qids `val` operates on. |
|`cirq.parameter_names`| Returns parameter names for this object. |
|`cirq.parameter_symbols`| Returns parameter symbols for this object. |
|`cirq.pauli_expansion`| Returns coefficients of the expansion of val in the Pauli basis. |
|`cirq.phase_by`| Returns a phased version of the effect. |
|`cirq.pow`| Returns `val**factor` of the given value, if defined. |
|`cirq.qasm`| Returns QASM code for the given value, if possible. |
|`cirq.qid_shape`| Returns a tuple describing the number of quantum levels of each |
|`cirq.quil`| Returns the QUIL code for the given value. |
|`cirq.read_json`| Read a JSON file that optionally contains cirq objects. |
|`cirq.resolve_parameters`| Resolves symbol parameters in the effect using the param resolver. |
|`cirq.to_json`| Write a JSON file containing a representation of obj. |
|`cirq.trace_distance_bound`| Returns a maximum on the trace distance between this effect's input |
|`cirq.trace_distance_from_angle_list`| Given a list of arguments of the eigenvalues of a unitary matrix, |
|`cirq.unitary`| Returns a unitary matrix describing the given value. |
|`cirq.validate_mixture`| Validates that the mixture's tuple are valid probabilities. |

### Quantum operator representation protocols

The following family of protocols is an important and frequently used set of features of Cirq and it is worthwhile mentioning them and and how they interact with each other. They are, in the order of increasing generality:

* `*unitary`
* `*kraus`
* `*mixture`

All these protocols make it easier to work with different representations of quantum operators, namely:
- finding that representation (`unitary`, `kraus`, `mixture`),
- determining whether the operator has that representation (`has_*`)
- and applying them (`apply_*`) on a state vector.

#### Unitary

The `*unitary` protocol is the least generic, as only unitary operators should implement it. The `cirq.unitary` function returns the matrix representation of the operator in the computational basis. We saw an example of the unitary protocol above, but let's see the unitary matrix of the Pauli-Y operator as well:
_____no_output_____

print(cirq.unitary(cirq.Y))_____no_output_____

#### Mixture

The `*mixture` protocol should be implemented by operators that are _unitary-mixtures_. These probabilistic operators are represented by a list of tuples ($p_i$, $U_i$), where each unitary effect $U_i$ occurs with a certain probability $p_i$, and $\sum p_i = 1$. Probabilities are a Python float between 0.0 and 1.0, and the unitary matrices are numpy arrays.

Constructing simple probabilistic gates in Cirq is easiest with the `with_probability` method. _____no_output_____

probabilistic_x = cirq.X.with_probability(.3)

for p, op in cirq.mixture(probabilistic_x):
    print(f"probability: {p}")
    print("operator:")
    print(op)_____no_output_____

In case an operator does not implement `SupportsMixture`, but does implement `SupportsUnitary`, `*mixture` functions fall back to the `*unitary` methods. It is easy to see that a unitary operator $U$ is just a "mixture" of a single unitary with probability $p=1$. _____no_output_____

# cirq.Y has a unitary effect but does not implement SupportsMixture
# thus mixture protocols will return ((1, cirq.unitary(Y)))
print(cirq.mixture(cirq.Y))
print(cirq.has_mixture(cirq.Y))_____no_output_____

#### Channel

The `kraus` representation is the operator sum representation of a quantum operator (a channel):

$$
  \rho \rightarrow \sum_{k=0}^{r-1} A_k \rho A_k^\dagger
$$

These matrices are required to satisfy the trace preserving condition

$$
        \sum_{k=0}^{r-1} A_k^\dagger A_k = I
$$

where $I$ is the identity matrix. The matrices $A_k$ are sometimes called Kraus or noise operators.

The `cirq.kraus` returns a tuple of numpy arrays, one for each of the Kraus operators:_____no_output_____

cirq.kraus(cirq.DepolarizingChannel(p=0.3))_____no_output_____

In case the operator does not implement `SupportsKraus`, but it does implement `SupportsMixture`, the `*kraus` protocol will generate the Kraus operators based on the  `*mixture` representation.

$$
((p_0, U_0),(p_1, U_1),\ldots,(p_n, U_n)) \rightarrow (\sqrt{p_0}U_0, \sqrt{p_1}U_1, \ldots, \sqrt{p_n}U_n)
$$

Thus for example `((0.25, X), (0.75, I)) -> (0.5 X, sqrt(0.75) I)`:_____no_output_____

cirq.kraus(cirq.X.with_probability(0.25))_____no_output_____

In the simplest case of a unitary operator, `cirq.kraus` returns a one-element tuple with the same unitary as returned by `cirq.unitary`:_____no_output_____

print(cirq.kraus(cirq.Y))
print(cirq.unitary(cirq.Y))
print(cirq.has_kraus(cirq.Y))_____no_output_____


# Notebook from GavinatorK/pssummitwkshp
Path: byoc/fastAI.ipynb

 ↑   Ensure Kernel is set to  ↑

conda_amazonei_pytorch_latest_p36
_____no_output_____

!pip install -U sagemaker
#restart your kernel_____no_output_____

### Bring your own Container

In this notebook, we will cover how to bring our own container with either a framework or algorithm to train a model on SageMaker.

We will use fastai in this case and build our container with custom training code integrated into the container. The other option is to use script mode which is easily done by changing the entrypoint.

The outline of this notebook is

1. Build docker a image for FastAI and serving and training code (provided).

2. Log into ECR, tag and push docker image to ECR

3. Use the FastAI container image in SageMaker to train our model

4. Deploy model to endpoint using the container image

5. Test inference using an image in couple of possible ways _____no_output_____#### Container Image
Let's start with building a container image locally and then push that to ECR (Elastic Container Registry)_____no_output_____

%cd ~/SageMaker/pssummitwkshp/byoc/docker_____no_output_____!docker build -t fastai ._____no_output_____!docker images_____no_output_____

## Set the ecr details and tags
Lets set a few params here like ecr name space , tag name etc._____no_output_____

from sagemaker import get_execution_role
import boto3
ecr_namespace = "sagemaker-training-containers/"
prefix = "script-mode-container-fastai"

ecr_repository_name = ecr_namespace + prefix
role = get_execution_role()
account_id = role.split(":")[4]
region = boto3.Session().region_name
tag_name=account_id+'.dkr.ecr.'+region+'.amazonaws.com/'+ecr_repository_name+':latest'_____no_output_____tag_name_____no_output_____

Now we tag our image with the tag name we generated above_____no_output_____

!docker tag fastai $tag_name_____no_output_____

### ECR Repository and push steps

All of these can be scripted out but they are laid out this way for transparency and step evolution understanding

First we get a token credential to ECR. This will allow us to perform ECR operations_____no_output_____

!$(aws ecr get-login --no-include-email)_____no_output_____

Here we create an ECR repository_____no_output_____

!aws ecr create-repository --repository-name $ecr_repository_name_____no_output_____

Now that our ECR respoitory has been created, we can now push our docker image to it with the tag name we assigned to it_____no_output_____

!docker push $tag_name_____no_output_____

This is how we get the URI of our uploaded docker image in ECR_____no_output_____

container_image_uri = "{0}.dkr.ecr.{1}.amazonaws.com/{2}:latest".format(
    account_id, region, ecr_repository_name
)
print(container_image_uri)_____no_output_____

#### Call your custom container to train the model

In the cell below, replace **"your-unique-bucket-name"** with the name of bucket you created in the data-prep notebook

**Note:** This cell takes around **20 mins** to run_____no_output_____

%%time
import sagemaker
import json

bucket = "your-unique-bucket-name"

# JSON encode hyperparameters
def json_encode_hyperparameters(hyperparameters):
    return {str(k): json.dumps(v) for (k, v) in hyperparameters.items()}

hyperparameters = json_encode_hyperparameters({"lr":1e-03})

est = sagemaker.estimator.Estimator(
    container_image_uri,
    role,
    instance_count=1,
    #train_instance_type="local",  # we use local mode
    instance_type='ml.m5.12xlarge',
    base_job_name=prefix,
    hyperparameters=hyperparameters,
)

train_config = sagemaker.session.TrainingInput(f's3://{bucket}/train')

est.fit({"train": train_config})_____no_output_____

Finally let us print out the trained FastAI model location. You will need this information for the inference step_____no_output_____

print(f'FastAI Model located at \n{est.output_path}{est._current_job_name}/output/model.tar.gz')_____no_output_____

### Attach to a training job that has been left to run

If your kernel becomes disconnected and your training has already started, you can reattach to the training job.

Simply look up the training job name and replace the **your-training-job-name** and then run the cell below.

Once the training job is finished, you can continue the cells after the training cell_____no_output_____

import sagemaker
import boto3

sess = sagemaker.Session()

training_job_name = 'your-training-job-name'

if training_job_name != 'your-training-job-name':
    est = sagemaker.estimator.Estimator.attach(training_job_name=training_job_name, sagemaker_session=sess)_____no_output_____


# Notebook from MohammedRakib/Heart-Disease-Classification
Path: Heart Disease Classification.ipynb

# Predicting heart disease using machine learning

This notebook looks into using various Python-based machine learning and data science libraries in an attempt to build a machine learning model capable of predicting whether or not someone has heart disease based on their medical attributes.

## Data

The original data came from the Cleavland data from the UCI Machine Learning Repository. https://archive.ics.uci.edu/ml/datasets/heart+Disease

There is also a version of it available on Kaggle. https://www.kaggle.com/ronitf/heart-disease-uci

## Features

1. age - age in years
2. sex - (1 = male; 0 = female)
3. cp - chest pain type
    * 0: Typical angina: chest pain related decrease blood supply to the heart
    * 1: Atypical angina: chest pain not related to heart
    * 2: Non-anginal pain: typically esophageal spasms (non heart related)
    * 3: Asymptomatic: chest pain not showing signs of disease
4. trestbps - resting blood pressure (in mm Hg on admission to the hospital) anything above 130-140 is typically cause for concern
5. chol - serum cholestoral in mg/dl
    * serum = LDL + HDL + .2 * triglycerides
    * above 200 is cause for concern
6. fbs - (fasting blood sugar > 120 mg/dl) (1 = true; 0 = false)
    * '>126' mg/dL signals diabetes
7. restecg - resting electrocardiographic results
    * 0: Nothing to note
    * 1: ST-T Wave abnormality
        * can range from mild symptoms to severe problems
        * signals non-normal heart beat
    * 2: Possible or definite left ventricular hypertrophy
        * Enlarged heart's main pumping chamber
8. thalach - maximum heart rate achieved
9. exang - exercise induced angina (1 = yes; 0 = no)
10. oldpeak - ST depression induced by exercise relative to rest looks at stress of heart during excercise unhealthy heart will stress more
11. slope - the slope of the peak exercise ST segment
    * 0: Upsloping: better heart rate with excercise (uncommon)
    * 1: Flatsloping: minimal change (typical healthy heart)
    * 2: Downslopins: signs of unhealthy heart
12. ca - number of major vessels (0-3) colored by flourosopy
    * colored vessel means the doctor can see the blood passing through
    * the more blood movement the better (no clots)
13. thal - thalium stress result
    * 1,3: normal
    * 6: fixed defect: used to be defect but ok now
    * 7: reversable defect: no proper blood movement when excercising
14. target - have disease or not (1=yes, 0=no) (= the predicted attribute)_____no_output_____# Data Analysis or EDA_____no_output_____

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline

from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier

from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.model_selection import RandomizedSearchCV, GridSearchCV
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.metrics import precision_score, recall_score, f1_score
from sklearn.metrics import plot_roc_curve_____no_output_____df = pd.read_csv("heart-disease.csv")
df.head()_____no_output_____df.shape_____no_output_____df['target'].value_counts()_____no_output_____df['target'].value_counts().plot(kind='bar', color=['salmon', 'lightblue']);_____no_output_____df.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 303 entries, 0 to 302
Data columns (total 14 columns):
 #   Column    Non-Null Count  Dtype
---  ------    --------------  -----
 0   age       303 non-null    int64
 1   sex       303 non-null    int64
 2   cp        303 non-null    int64
 3   trestbps  303 non-null    int64
 4   chol      303 non-null    int64
 5   fbs       303 non-null    int64
 6   restecg   303 non-null    int64
 7   thalach   303 non-null    int64
 8   exang     303 non-null    int64
 9   oldpeak   303 non-null    float64
 10  slope     303 non-null    int64
 11  ca        303 non-null    int64
 12  thal      303 non-null    int64
 13  target    303 non-null    int64
dtypes: float64(1), int64(13)
memory usage: 33.3 KB
df.isna().sum()_____no_output_____df.describe()_____no_output_____df.sex.value_counts()_____no_output_____pd.crosstab(df.target, df.sex)_____no_output_____pd.crosstab(df.target, df.sex).plot(kind='bar', color=['salmon', 'lightblue'], figsize=(10,6));

plt.title("Heart Disease Frequency by Sex")
plt.xlabel("0 = No Disease; 1 = Disease")
plt.ylabel("No.of people")
plt.legend(['Female', 'Male'])
plt.xticks(rotation=0);_____no_output_____plt.figure(figsize=(10,6))

plt.scatter(df.age[df.target==1], df.thalach[df.target==1], c='salmon')
plt.scatter(df.age[df.target==0], df.thalach[df.target==0], c='lightblue')

plt.title("Max Heart Rate vs Age")
plt.xlabel("Age")
plt.ylabel("Max Heart Rate")
plt.legend(["Disease", "No Disease"]);_____no_output_____df.age.plot.hist();_____no_output_____pd.crosstab(df.cp, df.target)_____no_output_____pd.crosstab(df.cp, df.target).plot(kind='bar', color=["salmon", "lightblue"], figsize=(10,6))

plt.title("Count vs Chestpain Type")
plt.xlabel("Chestpain")
plt.ylabel("Count")
plt.legend(['No Disease', 'Disease'])
plt.xticks(rotation=0);_____no_output_____df.head()_____no_output_____df.corr()_____no_output_____corr_matrix = df.corr()
fig, ax = plt.subplots(figsize=(15,10))

ax = sns.heatmap(df.corr(), annot=True, fmt=".2f", linewidths=0.5, cmap="RdYlBu");_____no_output_____

# Data Preprocessing_____no_output_____

X = df.drop('target', axis=1)
y = df['target']_____no_output_____X_____no_output_____y_____no_output_____np.random.seed(42)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)_____no_output_____X_train_____no_output_____y_train_____no_output_____

# Modelling & Comparisons_____no_output_____

models = {"Logistic Regression" : LogisticRegression(),
          "KNN" : KNeighborsClassifier(),
          "Random Forest" : RandomForestClassifier()}

def fit_and_score(models, X_train, X_test, y_train, y_test):

    np.random.seed(42)

    model_scores = {}

    for name, model in models.items():

        model.fit(X_train, y_train)

        model_scores[name] = model.score(X_test, y_test)

    return model_scores_____no_output_____model_scores = fit_and_score(models=models,
                             X_train=X_train,
                             X_test=X_test,
                             y_train=y_train,
                             y_test=y_test)
model_scoresC:\Users\Md Rakib\miniconda3\lib\site-packages\sklearn\linear_model\_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)
model_compare = pd.DataFrame(model_scores, index=['accuracy'])

model_compare.T.plot(kind='bar');_____no_output_____

# Hyperparameter Tuning (Manual)_____no_output_____

train_scores = []

test_scores = []

knn = KNeighborsClassifier()

neighbors = range(1,21)

for i in neighbors:
    knn.set_params(n_neighbors=i)

    knn.fit(X_train, y_train)

    train_scores.append(knn.score(X_train, y_train))

    test_scores.append(knn.score(X_test, y_test))_____no_output_____train_scores_____no_output_____test_scores_____no_output_____plt.plot(neighbors, train_scores, label="Train score")
plt.plot(neighbors, test_scores, label="Test score")
plt.xticks(np.arange(1,21))
plt.xlabel("No. of Neighbors")
plt.ylabel("Accuracy / Model Score")
plt.legend();

print(f"Maximum KNN score on the test data is: {max(test_scores)*100:.2f}%")Maximum KNN score on the test data is: 75.41%

# Hyperparameter Tuning with RandomizedSearchCV_____no_output_____

log_reg_grid = {'C': np.logspace(-4, 4, 20),
                'solver': ["liblinear"]}

rf_grid = {"n_estimators": np.arange(10, 1000, 50),
           "max_depth": [None, 3, 5, 10],
           "min_samples_split": np.arange(2, 20, 2),
           "min_samples_leaf": np.arange(1, 20, 2)}_____no_output_____np.random.seed(42)

rs_log_reg = RandomizedSearchCV(LogisticRegression(),
                                param_distributions=log_reg_grid,
                                cv=5,
                                n_iter=20,
                                verbose=True)

rs_log_reg.fit(X_train, y_train)Fitting 5 folds for each of 20 candidates, totalling 100 fits
rs_log_reg.best_params______no_output_____rs_log_reg.score(X_test, y_test)_____no_output_____np.random.seed(42)

rs_rf = RandomizedSearchCV(RandomForestClassifier(),
                           param_distributions=rf_grid,
                           cv=5,
                           n_iter=20,
                           verbose=True)

rs_rf.fit(X_train, y_train);[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.
rs_rf.best_params______no_output_____rs_rf.score(X_test, y_test)_____no_output_____

# Hyperparameter Tuning with GridSearchCV_____no_output_____

np.random.seed(42)

log_reg_grid = {"C": np.logspace(-4, 4, 20),
                "solver": ["liblinear"]}

gs_log_reg = GridSearchCV(LogisticRegression(),
                          param_grid=log_reg_grid,
                          cv=5,
                          verbose=True)

gs_log_reg.fit(X_train, y_train);Fitting 5 folds for each of 20 candidates, totalling 100 fits
gs_log_reg.best_params______no_output_____gs_log_reg.score(X_test, y_test)_____no_output_____

# Evaluting our tuned machine learning classifier, beyond accuracy

ROC curve and AUC score

* Confusion matrix
* Classification report
* Precision
* Recall
* F1-score_____no_output_____

y_preds = gs_log_reg.predict(X_test)

y_preds_____no_output_____y_test_____no_output_____plot_roc_curve(gs_log_reg, X_test, y_test);_____no_output_____print(confusion_matrix(y_test, y_preds))[[25  4]
 [ 3 29]]
sns.set(font_scale=1.5)

def plot_conf_mat(y_test, y_preds):

    fig, ax = plt.subplots(figsize=(3,3))

    ax =  sns.heatmap(confusion_matrix(y_test, y_preds),
                      annot=True,
                      cbar=False)
    plt.xlabel("True label")
    plt.ylabel("Predicted label")

plot_conf_mat(y_test, y_preds)_____no_output_____print(classification_report(y_test, y_preds))              precision    recall  f1-score   support

           0       0.89      0.86      0.88        29
           1       0.88      0.91      0.89        32

    accuracy                           0.89        61
   macro avg       0.89      0.88      0.88        61
weighted avg       0.89      0.89      0.89        61

gs_log_reg.best_params______no_output_____clf = LogisticRegression(C=0.23357214690901212,
                         solver='liblinear')_____no_output_____cv_acc = cross_val_score(clf,
                         X,
                         y,
                         cv=5,
                         scoring="accuracy")

cv_acc = np.mean(cv_acc)
cv_acc_____no_output_____cv_precision = cross_val_score(clf,
                         X,
                         y,
                         cv=5,
                         scoring="precision")

cv_precision = np.mean(cv_precision)
cv_precision_____no_output_____cv_recall = cross_val_score(clf,
                         X,
                         y,
                         cv=5,
                         scoring="recall")

cv_recall = np.mean(cv_recall)
cv_recall_____no_output_____cv_f1 = cross_val_score(clf,
                         X,
                         y,
                         cv=5,
                         scoring="f1")

cv_f1 = np.mean(cv_f1)
cv_f1_____no_output_____cv_metrics = pd.DataFrame({"Accuracy": cv_acc,
                           "Precision": cv_precision,
                           "Recall": cv_recall,
                           "F1": cv_f1},
                          index=[0])

cv_metrics.T.plot.bar(title="Cross-validated Classification Metrics", legend=False)_____no_output_____

# Feature Importance_____no_output_____

gs_log_reg.best_params______no_output_____clf = LogisticRegression(C=0.23357214690901212,
                         solver="liblinear")

clf.fit(X_train, y_train);_____no_output_____clf.coef______no_output_____feature_dict = dict(zip(df.columns, clf.coef_[0]))

feature_dict_____no_output_____feature_df = pd.DataFrame(feature_dict, index=[0])

feature_df.T.plot(kind='bar', title="Feature Importance", legend=False);_____no_output_____pd.crosstab(df['sex'], df['target'])_____no_output_____pd.crosstab(df['slope'], df['target'])_____no_output_____


# Notebook from josh9191/introtodeeplearning
Path: lab2/Part1_MNIST.ipynb

      Visit MIT Deep Learning

        Run in Google Colab

        View Source on GitHub

# Copyright Information_____no_output_____

# Copyright 2020 MIT 6.S191 Introduction to Deep Learning. All Rights Reserved.
#
# Licensed under the MIT License. You may not use this file except in compliance
# with the License. Use and/or modification of this code outside of 6.S191 must
# reference:
#
# © MIT 6.S191: Introduction to Deep Learning
# http://introtodeeplearning.com
#_____no_output_____

# Laboratory 2: Computer Vision

# Part 1: MNIST Digit Classification

In the first portion of this lab, we will build and train a convolutional neural network (CNN) for classification of handwritten digits from the famous [MNIST](http://yann.lecun.com/exdb/mnist/) dataset. The MNIST dataset consists of 60,000 training images and 10,000 test images. Our classes are the digits 0-9.

First, let's download the course repository, install dependencies, and import the relevant packages we'll need for this lab._____no_output_____

# Import Tensorflow 2.0
# %tensorflow_version 2.x
import tensorflow as tf

# !pip install mitdeeplearning
import mitdeeplearning as mdl

import matplotlib.pyplot as plt
import numpy as np
import random
from tqdm import tqdm

# Check that we are using a GPU, if not switch runtimes
#   using Runtime > Change Runtime Type > GPU
assert len(tf.config.list_physical_devices('GPU')) > 0_____no_output_____

## 1.1 MNIST dataset

Let's download and load the dataset and display a few random samples from it:_____no_output_____

mnist = tf.keras.datasets.mnist
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()
train_images = (np.expand_dims(train_images, axis=-1)/255.).astype(np.float32)
train_labels = (train_labels).astype(np.int64)
test_images = (np.expand_dims(test_images, axis=-1)/255.).astype(np.float32)
test_labels = (test_labels).astype(np.int64)_____no_output_____

Our training set is made up of 28x28 grayscale images of handwritten digits.

Let's visualize what some of these images and their corresponding training labels look like._____no_output_____

plt.figure(figsize=(10,10))
random_inds = np.random.choice(60000,36)
for i in range(36):
    plt.subplot(6,6,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    image_ind = random_inds[i]
    plt.imshow(np.squeeze(train_images[image_ind]), cmap=plt.cm.binary)
    plt.xlabel(train_labels[image_ind])_____no_output_____

## 1.2 Neural Network for Handwritten Digit Classification

We'll first build a simple neural network consisting of two fully connected layers and apply this to the digit classification task. Our network will ultimately output a probability distribution over the 10 digit classes (0-9). This first architecture we will be building is depicted below:

![alt_text](https://raw.githubusercontent.com/aamini/introtodeeplearning/master/lab2/img/mnist_2layers_arch.png "CNN Architecture for MNIST Classification")
_____no_output_____### Fully connected neural network architecture
To define the architecture of this first fully connected neural network, we'll once again use the Keras API and define the model using the [`Sequential`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential) class. Note how we first use a [`Flatten`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Flatten) layer, which flattens the input so that it can be fed into the model.

In this next block, you'll define the fully connected layers of this simple work._____no_output_____

def build_fc_model():
  fc_model = tf.keras.Sequential([
      # First define a Flatten layer
      tf.keras.layers.Flatten(),

      # '''TODO: Define the activation function for the first fully connected (Dense) layer.'''
      tf.keras.layers.Dense(128, activation='relu'),

      # '''TODO: Define the second Dense layer to output the classification probabilities'''
      tf.keras.layers.Dense(10, activation='softmax')

  ])
  return fc_model

model = build_fc_model()_____no_output_____

As we progress through this next portion, you may find that you'll want to make changes to the architecture defined above. **Note that in order to update the model later on, you'll need to re-run the above cell to re-initialize the model. **_____no_output_____Let's take a step back and think about the network we've just created. The first layer in this network, `tf.keras.layers.Flatten`, transforms the format of the images from a 2d-array (28 x 28 pixels), to a 1d-array of 28 * 28 = 784 pixels. You can think of this layer as unstacking rows of pixels in the image and lining them up. There are no learned parameters in this layer; it only reformats the data.

After the pixels are flattened, the network consists of a sequence of two `tf.keras.layers.Dense` layers. These are fully-connected neural layers. The first `Dense` layer has 128 nodes (or neurons). The second (and last) layer (which you've defined!) should return an array of probability scores that sum to 1. Each node contains a score that indicates the probability that the current image belongs to one of the handwritten digit classes.

That defines our fully connected model! _____no_output_____

### Compile the model

Before training the model, we need to define a few more settings. These are added during the model's [`compile`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential#compile) step:

* *Loss function* — This defines how we measure how accurate the model is during training. As was covered in lecture, during training we want to minimize this function, which will "steer" the model in the right direction.
* *Optimizer* — This defines how the model is updated based on the data it sees and its loss function.
* *Metrics* — Here we can define metrics used to monitor the training and testing steps. In this example, we'll look at the *accuracy*, the fraction of the images that are correctly classified.

We'll start out by using a stochastic gradient descent (SGD) optimizer initialized with a learning rate of 0.1. Since we are performing a categorical classification task, we'll want to use the [cross entropy loss](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/sparse_categorical_crossentropy).

You'll want to experiment with both the choice of optimizer and learning rate and evaluate how these affect the accuracy of the trained model. _____no_output_____

'''TODO: Experiment with different optimizers and learning rates. How do these affect
    the accuracy of the trained model? Which optimizers and/or learning rates yield
    the best performance?'''
model.compile(optimizer=tf.keras.optimizers.SGD(learning_rate=1e-1),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])_____no_output_____

### Train the model

We're now ready to train our model, which will involve feeding the training data (`train_images` and `train_labels`) into the model, and then asking it to learn the associations between images and labels. We'll also need to define the batch size and the number of epochs, or iterations over the MNIST dataset, to use during training.

In Lab 1, we saw how we can use `GradientTape` to optimize losses and train models with stochastic gradient descent. After defining the model settings in the `compile` step, we can also accomplish training by calling the [`fit`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential#fit) method on an instance of the `Model` class. We will use this to train our fully connected model
_____no_output_____

# Define the batch size and the number of epochs to use during training
BATCH_SIZE = 64
EPOCHS = 5

model.fit(train_images, train_labels, batch_size=BATCH_SIZE, epochs=EPOCHS)Train on 60000 samples
Epoch 1/5
60000/60000 [==============================] - ETA: 8:12 - loss: 2.4209 - accuracy: 0.03 - ETA: 29s - loss: 1.8236 - accuracy: 0.4748 - ETA: 17s - loss: 1.4829 - accuracy: 0.610 - ETA: 11s - loss: 1.2191 - accuracy: 0.682 - ETA: 8s - loss: 1.0395 - accuracy: 0.729 - ETA: 7s - loss: 0.9206 - accuracy: 0.76 - ETA: 6s - loss: 0.8452 - accuracy: 0.78 - ETA: 5s - loss: 0.7855 - accuracy: 0.79 - ETA: 4s - loss: 0.7295 - accuracy: 0.81 - ETA: 4s - loss: 0.6951 - accuracy: 0.81 - ETA: 3s - loss: 0.6655 - accuracy: 0.82 - ETA: 3s - loss: 0.6377 - accuracy: 0.83 - ETA: 3s - loss: 0.6139 - accuracy: 0.83 - ETA: 3s - loss: 0.5915 - accuracy: 0.84 - ETA: 2s - loss: 0.5727 - accuracy: 0.84 - ETA: 2s - loss: 0.5568 - accuracy: 0.85 - ETA: 2s - loss: 0.5444 - accuracy: 0.85 - ETA: 2s - loss: 0.5328 - accuracy: 0.85 - ETA: 2s - loss: 0.5210 - accuracy: 0.85 - ETA: 2s - loss: 0.5112 - accuracy: 0.86 - ETA: 2s - loss: 0.5024 - accuracy: 0.86 - ETA: 1s - loss: 0.4918 - accuracy: 0.86 - ETA: 1s - loss: 0.4839 - accuracy: 0.86 - ETA: 1s - loss: 0.4741 - accuracy: 0.87 - ETA: 1s - loss: 0.4677 - accuracy: 0.87 - ETA: 1s - loss: 0.4619 - accuracy: 0.87 - ETA: 1s - loss: 0.4550 - accuracy: 0.87 - ETA: 1s - loss: 0.4497 - accuracy: 0.87 - ETA: 1s - loss: 0.4442 - accuracy: 0.87 - ETA: 1s - loss: 0.4382 - accuracy: 0.88 - ETA: 1s - loss: 0.4339 - accuracy: 0.88 - ETA: 1s - loss: 0.4281 - accuracy: 0.88 - ETA: 0s - loss: 0.4226 - accuracy: 0.88 - ETA: 0s - loss: 0.4184 - accuracy: 0.88 - ETA: 0s - loss: 0.4136 - accuracy: 0.88 - ETA: 0s - loss: 0.4103 - accuracy: 0.88 - ETA: 0s - loss: 0.4054 - accuracy: 0.88 - ETA: 0s - loss: 0.4025 - accuracy: 0.88 - ETA: 0s - loss: 0.3994 - accuracy: 0.88 - ETA: 0s - loss: 0.3960 - accuracy: 0.89 - ETA: 0s - loss: 0.3917 - accuracy: 0.89 - ETA: 0s - loss: 0.3881 - accuracy: 0.89 - ETA: 0s - loss: 0.3856 - accuracy: 0.89 - ETA: 0s - loss: 0.3823 - accuracy: 0.89 - ETA: 0s - loss: 0.3794 - accuracy: 0.89 - ETA: 0s - loss: 0.3753 - accuracy: 0.89 - ETA: 0s - loss: 0.3718 - accuracy: 0.89 - 3s 49us/sample - loss: 0.3699 - accuracy: 0.8979
Epoch 2/5
60000/60000 [==============================] - ETA: 2s - loss: 0.2789 - accuracy: 0.90 - ETA: 2s - loss: 0.2202 - accuracy: 0.93 - ETA: 2s - loss: 0.2127 - accuracy: 0.93 - ETA: 2s - loss: 0.2177 - accuracy: 0.93 - ETA: 2s - loss: 0.2238 - accuracy: 0.93 - ETA: 2s - loss: 0.2175 - accuracy: 0.93 - ETA: 2s - loss: 0.2131 - accuracy: 0.93 - ETA: 2s - loss: 0.2109 - accuracy: 0.93 - ETA: 1s - loss: 0.2144 - accuracy: 0.93 - ETA: 1s - loss: 0.2154 - accuracy: 0.93 - ETA: 1s - loss: 0.2143 - accuracy: 0.93 - ETA: 1s - loss: 0.2122 - accuracy: 0.93 - ETA: 1s - loss: 0.2093 - accuracy: 0.94 - ETA: 1s - loss: 0.2122 - accuracy: 0.93 - ETA: 1s - loss: 0.2110 - accuracy: 0.94 - ETA: 1s - loss: 0.2124 - accuracy: 0.93 - ETA: 1s - loss: 0.2121 - accuracy: 0.93 - ETA: 1s - loss: 0.2111 - accuracy: 0.93 - ETA: 1s - loss: 0.2088 - accuracy: 0.94 - ETA: 1s - loss: 0.2088 - accuracy: 0.94 - ETA: 1s - loss: 0.2074 - accuracy: 0.94 - ETA: 1s - loss: 0.2067 - accuracy: 0.94 - ETA: 1s - loss: 0.2066 - accuracy: 0.94 - ETA: 1s - loss: 0.2060 - accuracy: 0.94 - ETA: 1s - loss: 0.2064 - accuracy: 0.94 - ETA: 1s - loss: 0.2068 - accuracy: 0.94 - ETA: 1s - loss: 0.2063 - accuracy: 0.94 - ETA: 1s - loss: 0.2057 - accuracy: 0.94 - ETA: 0s - loss: 0.2047 - accuracy: 0.94 - ETA: 0s - loss: 0.2052 - accuracy: 0.94 - ETA: 0s - loss: 0.2040 - accuracy: 0.94 - ETA: 0s - loss: 0.2025 - accuracy: 0.94 - ETA: 0s - loss: 0.2023 - accuracy: 0.94 - ETA: 0s - loss: 0.2028 - accuracy: 0.94 - ETA: 0s - loss: 0.2024 - accuracy: 0.94 - ETA: 0s - loss: 0.2024 - accuracy: 0.94 - ETA: 0s - loss: 0.2019 - accuracy: 0.94 - ETA: 0s - loss: 0.2013 - accuracy: 0.94 - ETA: 0s - loss: 0.2010 - accuracy: 0.94 - ETA: 0s - loss: 0.2005 - accuracy: 0.94 - ETA: 0s - loss: 0.1997 - accuracy: 0.94 - ETA: 0s - loss: 0.1994 - accuracy: 0.94 - ETA: 0s - loss: 0.1992 - accuracy: 0.94 - ETA: 0s - loss: 0.1983 - accuracy: 0.94 - ETA: 0s - loss: 0.1973 - accuracy: 0.94 - ETA: 0s - loss: 0.1965 - accuracy: 0.94 - ETA: 0s - loss: 0.1959 - accuracy: 0.94 - 2s 40us/sample - loss: 0.1958 - accuracy: 0.9436
Epoch 3/5
60000/60000 [==============================] - ETA: 2s - loss: 0.2652 - accuracy: 0.90 - ETA: 2s - loss: 0.1619 - accuracy: 0.95 - ETA: 2s - loss: 0.1486 - accuracy: 0.96 - ETA: 2s - loss: 0.1544 - accuracy: 0.95 - ETA: 2s - loss: 0.1584 - accuracy: 0.95 - ETA: 2s - loss: 0.1618 - accuracy: 0.95 - ETA: 2s - loss: 0.1625 - accuracy: 0.95 - ETA: 2s - loss: 0.1610 - accuracy: 0.95 - ETA: 1s - loss: 0.1566 - accuracy: 0.95 - ETA: 1s - loss: 0.1541 - accuracy: 0.95 - ETA: 1s - loss: 0.1561 - accuracy: 0.95 - ETA: 1s - loss: 0.1544 - accuracy: 0.95 - ETA: 1s - loss: 0.1547 - accuracy: 0.95 - ETA: 1s - loss: 0.1525 - accuracy: 0.95 - ETA: 1s - loss: 0.1513 - accuracy: 0.95 - ETA: 1s - loss: 0.1525 - accuracy: 0.95 - ETA: 1s - loss: 0.1509 - accuracy: 0.95 - ETA: 1s - loss: 0.1498 - accuracy: 0.95 - ETA: 1s - loss: 0.1516 - accuracy: 0.95 - ETA: 1s - loss: 0.1512 - accuracy: 0.95 - ETA: 1s - loss: 0.1511 - accuracy: 0.95 - ETA: 1s - loss: 0.1507 - accuracy: 0.95 - ETA: 1s - loss: 0.1501 - accuracy: 0.95 - ETA: 1s - loss: 0.1501 - accuracy: 0.95 - ETA: 1s - loss: 0.1496 - accuracy: 0.95 - ETA: 1s - loss: 0.1502 - accuracy: 0.95 - ETA: 1s - loss: 0.1500 - accuracy: 0.95 - ETA: 1s - loss: 0.1492 - accuracy: 0.95 - ETA: 0s - loss: 0.1493 - accuracy: 0.95 - ETA: 0s - loss: 0.1500 - accuracy: 0.95 - ETA: 0s - loss: 0.1499 - accuracy: 0.95 - ETA: 0s - loss: 0.1497 - accuracy: 0.95 - ETA: 0s - loss: 0.1492 - accuracy: 0.95 - ETA: 0s - loss: 0.1482 - accuracy: 0.95 - ETA: 0s - loss: 0.1479 - accuracy: 0.95 - ETA: 0s - loss: 0.1477 - accuracy: 0.95 - ETA: 0s - loss: 0.1469 - accuracy: 0.95 - ETA: 0s - loss: 0.1457 - accuracy: 0.95 - ETA: 0s - loss: 0.1459 - accuracy: 0.95 - ETA: 0s - loss: 0.1457 - accuracy: 0.95 - ETA: 0s - loss: 0.1468 - accuracy: 0.95 - ETA: 0s - loss: 0.1467 - accuracy: 0.95 - ETA: 0s - loss: 0.1467 - accuracy: 0.95 - ETA: 0s - loss: 0.1464 - accuracy: 0.95 - ETA: 0s - loss: 0.1461 - accuracy: 0.95 - ETA: 0s - loss: 0.1458 - accuracy: 0.95 - ETA: 0s - loss: 0.1454 - accuracy: 0.95 - 2s 40us/sample - loss: 0.1451 - accuracy: 0.9590
Epoch 4/5

As the model trains, the loss and accuracy metrics are displayed. With five epochs and a learning rate of 0.01, this fully connected model should achieve an accuracy of approximatley 0.97 (or 97%) on the training data._____no_output_____### Evaluate accuracy on the test dataset

Now that we've trained the model, we can ask it to make predictions about a test set that it hasn't seen before. In this example, the `test_images` array comprises our test dataset. To evaluate accuracy, we can check to see if the model's predictions match the labels from the `test_labels` array.

Use the [`evaluate`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential#evaluate) method to evaluate the model on the test dataset!_____no_output_____

test_loss, test_acc = model.evaluate(x=test_images, y=test_labels, verbose=2)
print('Test accuracy:', test_acc)10000/10000 - 1s - loss: 0.1047 - accuracy: 0.9698
Test accuracy: 0.9698

You may observe that the accuracy on the test dataset is a little lower than the accuracy on the training dataset. This gap between training accuracy and test accuracy is an example of *overfitting*, when a machine learning model performs worse on new data than on its training data.

What is the highest accuracy you can achieve with this first fully connected model? Since the handwritten digit classification task is pretty straightforward, you may be wondering how we can do better...

![Deeper...](https://i.kym-cdn.com/photos/images/newsfeed/000/534/153/f87.jpg)_____no_output_____## 1.3 Convolutional Neural Network (CNN) for handwritten digit classification_____no_output_____As we saw in lecture, convolutional neural networks (CNNs) are particularly well-suited for a variety of tasks in computer vision, and have achieved near-perfect accuracies on the MNIST dataset. We will now build a CNN composed of two convolutional layers and pooling layers, followed by two fully connected layers, and ultimately output a probability distribution over the 10 digit classes (0-9). The CNN we will be building is depicted below:

![alt_text](https://raw.githubusercontent.com/aamini/introtodeeplearning/master/lab2/img/convnet_fig.png "CNN Architecture for MNIST Classification")_____no_output_____### Define the CNN model

We'll use the same training and test datasets as before, and proceed similarly as our fully connected network to define and train our new CNN model. To do this we will explore two layers we have not encountered before: you can use  [`keras.layers.Conv2D` ](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D) to define convolutional layers and [`keras.layers.MaxPool2D`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/MaxPool2D) to define the pooling layers. Use the parameters shown in the network architecture above to define these layers and build the CNN model._____no_output_____

def build_cnn_model():
    cnn_model = tf.keras.Sequential([

        # TODO: Define the first convolutional layer
        tf.keras.layers.Conv2D(filters=24, kernel_size=(3, 3), strides=(1,1 )),

        # TODO: Define the first max pooling layer
        tf.keras.layers.MaxPool2D(pool_size=(2, 2)),

        # TODO: Define the second convolutional layer
        tf.keras.layers.Conv2D(filters=36, kernel_size=(3, 3), strides=(1, 1)),

        # TODO: Define the second max pooling layer
        tf.keras.layers.MaxPool2D(pool_size=(2, 2)),

        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(128, activation=tf.nn.relu),

        # TODO: Define the last Dense layer to output the classification
        # probabilities. Pay attention to the activation needed a probability
        # output
        tf.keras.layers.Dense(10, activation='softmax')
    ])

    return cnn_model

cnn_model = build_cnn_model()
# Initialize the model by passing some data through
cnn_model.predict(train_images[[0]])
# Print the summary of the layers in the model.
print(cnn_model.summary())Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_2 (Conv2D)            multiple                  240
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 multiple                  0
_________________________________________________________________
conv2d_3 (Conv2D)            multiple                  7812
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 multiple                  0
_________________________________________________________________
flatten_2 (Flatten)          multiple                  0
_________________________________________________________________
dense_3 (Dense)              multiple                  115328
_________________________________________________________________
dense_4 (Dense)              multiple                  1290
=================================================================
Total params: 124,670
Trainable params: 124,670
Non-trainable params: 0
_________________________________________________________________
None

### Train and test the CNN model

Now, as before, we can define the loss function, optimizer, and metrics through the `compile` method. Compile the CNN model with an optimizer and learning rate of choice:_____no_output_____

'''TODO: Define the compile operation with your optimizer and learning rate of choice'''
opt = tf.keras.optimizers.Adam(learning_rate=0.005)
cnn_model.compile(optimizer=opt,
                                  loss='sparse_categorical_crossentropy',
                                  metrics=['accuracy']) # TODO_____no_output_____

As was the case with the fully connected model, we can train our CNN using the `fit` method via the Keras API._____no_output_____

'''TODO: Use model.fit to train the CNN model, with the same batch_size and number of epochs previously used.'''
BATCH_SIZE = 64
EPOCHS = 5

cnn_model.fit(train_images, train_labels, batch_size=BATCH_SIZE, epochs=EPOCHS)Train on 60000 samples
Epoch 1/5
60000/60000 [==============================] - ETA: 12s - loss: 0.0323 - accuracy: 0.984 - ETA: 5s - loss: 0.1254 - accuracy: 0.985 - ETA: 4s - loss: 0.0938 - accuracy: 0.98 - ETA: 4s - loss: 0.0820 - accuracy: 0.98 - ETA: 4s - loss: 0.1142 - accuracy: 0.98 - ETA: 3s - loss: 0.1085 - accuracy: 0.98 - ETA: 3s - loss: 0.0975 - accuracy: 0.98 - ETA: 3s - loss: 0.0901 - accuracy: 0.98 - ETA: 3s - loss: 0.0882 - accuracy: 0.98 - ETA: 3s - loss: 0.0806 - accuracy: 0.98 - ETA: 3s - loss: 0.0742 - accuracy: 0.98 - ETA: 3s - loss: 0.0703 - accuracy: 0.99 - ETA: 3s - loss: 0.0692 - accuracy: 0.99 - ETA: 3s - loss: 0.0679 - accuracy: 0.99 - ETA: 3s - loss: 0.0633 - accuracy: 0.99 - ETA: 3s - loss: 0.0608 - accuracy: 0.99 - ETA: 3s - loss: 0.0639 - accuracy: 0.99 - ETA: 3s - loss: 0.0662 - accuracy: 0.99 - ETA: 3s - loss: 0.0635 - accuracy: 0.99 - ETA: 3s - loss: 0.0639 - accuracy: 0.99 - ETA: 2s - loss: 0.0621 - accuracy: 0.99 - ETA: 2s - loss: 0.0607 - accuracy: 0.99 - ETA: 2s - loss: 0.0594 - accuracy: 0.99 - ETA: 2s - loss: 0.0585 - accuracy: 0.99 - ETA: 2s - loss: 0.0570 - accuracy: 0.99 - ETA: 2s - loss: 0.0549 - accuracy: 0.99 - ETA: 2s - loss: 0.0541 - accuracy: 0.99 - ETA: 2s - loss: 0.0524 - accuracy: 0.99 - ETA: 2s - loss: 0.0521 - accuracy: 0.99 - ETA: 2s - loss: 0.0530 - accuracy: 0.99 - ETA: 2s - loss: 0.0527 - accuracy: 0.99 - ETA: 2s - loss: 0.0548 - accuracy: 0.99 - ETA: 2s - loss: 0.0547 - accuracy: 0.99 - ETA: 2s - loss: 0.0550 - accuracy: 0.99 - ETA: 2s - loss: 0.0559 - accuracy: 0.99 - ETA: 2s - loss: 0.0563 - accuracy: 0.99 - ETA: 2s - loss: 0.0549 - accuracy: 0.99 - ETA: 2s - loss: 0.0561 - accuracy: 0.99 - ETA: 2s - loss: 0.0572 - accuracy: 0.99 - ETA: 1s - loss: 0.0567 - accuracy: 0.99 - ETA: 1s - loss: 0.0556 - accuracy: 0.99 - ETA: 1s - loss: 0.0548 - accuracy: 0.99 - ETA: 1s - loss: 0.0540 - accuracy: 0.99 - ETA: 1s - loss: 0.0533 - accuracy: 0.99 - ETA: 1s - loss: 0.0526 - accuracy: 0.99 - ETA: 1s - loss: 0.0519 - accuracy: 0.99 - ETA: 1s - loss: 0.0515 - accuracy: 0.99 - ETA: 1s - loss: 0.0508 - accuracy: 0.99 - ETA: 1s - loss: 0.0507 - accuracy: 0.99 - ETA: 1s - loss: 0.0522 - accuracy: 0.99 - ETA: 1s - loss: 0.0518 - accuracy: 0.99 - ETA: 1s - loss: 0.0510 - accuracy: 0.99 - ETA: 1s - loss: 0.0501 - accuracy: 0.99 - ETA: 1s - loss: 0.0500 - accuracy: 0.99 - ETA: 1s - loss: 0.0507 - accuracy: 0.99 - ETA: 1s - loss: 0.0502 - accuracy: 0.99 - ETA: 1s - loss: 0.0504 - accuracy: 0.99 - ETA: 1s - loss: 0.0503 - accuracy: 0.99 - ETA: 1s - loss: 0.0503 - accuracy: 0.99 - ETA: 0s - loss: 0.0499 - accuracy: 0.99 - ETA: 0s - loss: 0.0500 - accuracy: 0.99 - ETA: 0s - loss: 0.0500 - accuracy: 0.99 - ETA: 0s - loss: 0.0496 - accuracy: 0.99 - ETA: 0s - loss: 0.0498 - accuracy: 0.99 - ETA: 0s - loss: 0.0496 - accuracy: 0.99 - ETA: 0s - loss: 0.0495 - accuracy: 0.99 - ETA: 0s - loss: 0.0496 - accuracy: 0.99 - ETA: 0s - loss: 0.0490 - accuracy: 0.99 - ETA: 0s - loss: 0.0500 - accuracy: 0.99 - ETA: 0s - loss: 0.0502 - accuracy: 0.99 - ETA: 0s - loss: 0.0499 - accuracy: 0.99 - ETA: 0s - loss: 0.0507 - accuracy: 0.99 - ETA: 0s - loss: 0.0509 - accuracy: 0.99 - ETA: 0s - loss: 0.0507 - accuracy: 0.99 - ETA: 0s - loss: 0.0510 - accuracy: 0.99 - ETA: 0s - loss: 0.0509 - accuracy: 0.99 - ETA: 0s - loss: 0.0502 - accuracy: 0.99 - ETA: 0s - loss: 0.0504 - accuracy: 0.99 - 4s 68us/sample - loss: 0.0507 - accuracy: 0.9924
Epoch 2/5
60000/60000 [==============================] - ETA: 4s - loss: 4.5461e-06 - accuracy: 1.00 - ETA: 4s - loss: 0.0267 - accuracy: 0.9952   - ETA: 4s - loss: 0.0598 - accuracy: 0.99 - ETA: 4s - loss: 0.0653 - accuracy: 0.99 - ETA: 4s - loss: 0.0721 - accuracy: 0.99 - ETA: 3s - loss: 0.0615 - accuracy: 0.99 - ETA: 3s - loss: 0.0657 - accuracy: 0.99 - ETA: 3s - loss: 0.0610 - accuracy: 0.99 - ETA: 3s - loss: 0.0571 - accuracy: 0.99 - ETA: 3s - loss: 0.0597 - accuracy: 0.99 - ETA: 3s - loss: 0.0574 - accuracy: 0.99 - ETA: 3s - loss: 0.0620 - accuracy: 0.99 - ETA: 3s - loss: 0.0609 - accuracy: 0.99 - ETA: 3s - loss: 0.0622 - accuracy: 0.99 - ETA: 3s - loss: 0.0638 - accuracy: 0.99 - ETA: 3s - loss: 0.0667 - accuracy: 0.99 - ETA: 3s - loss: 0.0651 - accuracy: 0.99 - ETA: 3s - loss: 0.0651 - accuracy: 0.99 - ETA: 3s - loss: 0.0630 - accuracy: 0.99 - ETA: 3s - loss: 0.0620 - accuracy: 0.99 - ETA: 3s - loss: 0.0607 - accuracy: 0.99 - ETA: 2s - loss: 0.0613 - accuracy: 0.99 - ETA: 2s - loss: 0.0619 - accuracy: 0.99 - ETA: 2s - loss: 0.0608 - accuracy: 0.99 - ETA: 2s - loss: 0.0587 - accuracy: 0.99 - ETA: 2s - loss: 0.0569 - accuracy: 0.99 - ETA: 2s - loss: 0.0554 - accuracy: 0.99 - ETA: 2s - loss: 0.0564 - accuracy: 0.99 - ETA: 2s - loss: 0.0575 - accuracy: 0.99 - ETA: 2s - loss: 0.0568 - accuracy: 0.99 - ETA: 2s - loss: 0.0560 - accuracy: 0.99 - ETA: 2s - loss: 0.0563 - accuracy: 0.99 - ETA: 2s - loss: 0.0559 - accuracy: 0.99 - ETA: 2s - loss: 0.0544 - accuracy: 0.99 - ETA: 2s - loss: 0.0541 - accuracy: 0.99 - ETA: 2s - loss: 0.0532 - accuracy: 0.99 - ETA: 2s - loss: 0.0533 - accuracy: 0.99 - ETA: 2s - loss: 0.0524 - accuracy: 0.99 - ETA: 1s - loss: 0.0519 - accuracy: 0.99 - ETA: 1s - loss: 0.0524 - accuracy: 0.99 - ETA: 1s - loss: 0.0522 - accuracy: 0.99 - ETA: 1s - loss: 0.0516 - accuracy: 0.99 - ETA: 1s - loss: 0.0523 - accuracy: 0.99 - ETA: 1s - loss: 0.0534 - accuracy: 0.99 - ETA: 1s - loss: 0.0531 - accuracy: 0.99 - ETA: 1s - loss: 0.0523 - accuracy: 0.99 - ETA: 1s - loss: 0.0543 - accuracy: 0.99 - ETA: 1s - loss: 0.0542 - accuracy: 0.99 - ETA: 1s - loss: 0.0571 - accuracy: 0.99 - ETA: 1s - loss: 0.0575 - accuracy: 0.99 - ETA: 1s - loss: 0.0609 - accuracy: 0.99 - ETA: 1s - loss: 0.0642 - accuracy: 0.99 - ETA: 1s - loss: 0.0651 - accuracy: 0.99 - ETA: 1s - loss: 0.0676 - accuracy: 0.99 - ETA: 1s - loss: 0.0696 - accuracy: 0.99 - ETA: 1s - loss: 0.0715 - accuracy: 0.99 - ETA: 0s - loss: 0.0728 - accuracy: 0.99 - ETA: 0s - loss: 0.0745 - accuracy: 0.99 - ETA: 0s - loss: 0.0742 - accuracy: 0.99 - ETA: 0s - loss: 0.0748 - accuracy: 0.99 - ETA: 0s - loss: 0.0762 - accuracy: 0.99 - ETA: 0s - loss: 0.0796 - accuracy: 0.99 - ETA: 0s - loss: 0.0828 - accuracy: 0.99 - ETA: 0s - loss: 0.0840 - accuracy: 0.99 - ETA: 0s - loss: 0.0845 - accuracy: 0.99 - ETA: 0s - loss: 0.0863 - accuracy: 0.99 - ETA: 0s - loss: 0.0858 - accuracy: 0.99 - ETA: 0s - loss: 0.0860 - accuracy: 0.99 - ETA: 0s - loss: 0.0865 - accuracy: 0.99 - ETA: 0s - loss: 0.0865 - accuracy: 0.99 - ETA: 0s - loss: 0.0863 - accuracy: 0.99 - ETA: 0s - loss: 0.0871 - accuracy: 0.99 - ETA: 0s - loss: 0.0879 - accuracy: 0.99 - ETA: 0s - loss: 0.0869 - accuracy: 0.99 - ETA: 0s - loss: 0.0876 - accuracy: 0.99 - 4s 65us/sample - loss: 0.0871 - accuracy: 0.9909
Epoch 3/5

Great! Now that we've trained the model, let's evaluate it on the test dataset using the [`evaluate`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential#evaluate) method:_____no_output_____

'''TODO: Use the evaluate method to test the model!'''
test_loss, test_acc = cnn_model.evaluate(x=test_images, y=test_labels, verbose=2)

print('Test accuracy:', test_acc)10000/10000 - 1s - loss: 0.3903 - accuracy: 0.9825
Test accuracy: 0.9825

What is the highest accuracy you're able to achieve using the CNN model, and how does the accuracy of the CNN model compare to the accuracy of the simple fully connected network? What optimizers and learning rates seem to be optimal for training the CNN model? _____no_output_____### Make predictions with the CNN model

With the model trained, we can use it to make predictions about some images. The [`predict`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential#predict) function call generates the output predictions given a set of input samples.
_____no_output_____

predictions = cnn_model.predict(test_images)_____no_output_____

With this function call, the model has predicted the label for each image in the testing set. Let's take a look at the prediction for the first image in the test dataset:_____no_output_____

predictions[0]_____no_output_____

As you can see, a prediction is an array of 10 numbers. Recall that the output of our model is a probability distribution over the 10 digit classes. Thus, these numbers describe the model's "confidence" that the image corresponds to each of the 10 different digits.

Let's look at the digit that has the highest confidence for the first image in the test dataset:_____no_output_____

'''TODO: identify the digit with the highest confidence prediction for the first
    image in the test dataset. '''
prediction = np.argmax(predictions, axis=1)
print(prediction)[7 2 1 ... 4 5 6]

So, the model is most confident that this image is a "???". We can check the test label (remember, this is the true identity of the digit) to see if this prediction is correct:_____no_output_____

print("Label of this digit is:", test_labels[0])
plt.imshow(test_images[0,:,:,0], cmap=plt.cm.binary)Label of this digit is: 7

It is! Let's visualize the classification results on the MNIST dataset. We will plot images from the test dataset along with their predicted label, as well as a histogram that provides the prediction probabilities for each of the digits:_____no_output_____

#@title Change the slider to look at the model's predictions! { run: "auto" }

image_index = 79 #@param {type:"slider", min:0, max:100, step:1}
plt.subplot(1,2,1)
mdl.lab2.plot_image_prediction(image_index, predictions, test_labels, test_images)
plt.subplot(1,2,2)
mdl.lab2.plot_value_prediction(image_index, predictions,  test_labels)_____no_output_____

We can also plot several images along with their predictions, where correct prediction labels are blue and incorrect prediction labels are red. The number gives the percent confidence (out of 100) for the predicted label. Note the model can be very confident in an incorrect prediction!_____no_output_____

# Plots the first X test images, their predicted label, and the true label
# Color correct predictions in blue, incorrect predictions in red
num_rows = 5
num_cols = 4
num_images = num_rows*num_cols
plt.figure(figsize=(2*2*num_cols, 2*num_rows))
for i in range(num_images):
  plt.subplot(num_rows, 2*num_cols, 2*i+1)
  mdl.lab2.plot_image_prediction(i, predictions, test_labels, test_images)
  plt.subplot(num_rows, 2*num_cols, 2*i+2)
  mdl.lab2.plot_value_prediction(i, predictions, test_labels)
_____no_output_____

## 1.4 Training the model 2.0

Earlier in the lab, we used the [`fit`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential#fit) function call to train the model. This function is quite high-level and intuitive, which is really useful for simpler models. As you may be able to tell, this function abstracts away many details in the training call, and we have less control over training model, which could be useful in other contexts.

As an alternative to this, we can use the [`tf.GradientTape`](https://www.tensorflow.org/api_docs/python/tf/GradientTape) class to record differentiation operations during training, and then call the [`tf.GradientTape.gradient`](https://www.tensorflow.org/api_docs/python/tf/GradientTape#gradient) function to actually compute the gradients. You may recall seeing this in Lab 1 Part 1, but let's take another look at this here.

We'll use this framework to train our `cnn_model` using stochastic gradient descent._____no_output_____

# Rebuild the CNN model
cnn_model = build_cnn_model()

batch_size = 12
loss_history = mdl.util.LossHistory(smoothing_factor=0.95) # to record the evolution of the loss
plotter = mdl.util.PeriodicPlotter(sec=2, xlabel='Iterations', ylabel='Loss', scale='semilogy')
optimizer = tf.keras.optimizers.SGD(learning_rate=1e-2) # define our optimizer

if hasattr(tqdm, '_instances'): tqdm._instances.clear() # clear if it exists
for idx in tqdm(range(0, train_images.shape[0], batch_size)):
  # First grab a batch of training data and convert the input images to tensors
  (images, labels) = (train_images[idx:idx+batch_size], train_labels[idx:idx+batch_size])
  images = tf.convert_to_tensor(images, dtype=tf.float32)

  # GradientTape to record differentiation operations
  with tf.GradientTape() as tape:
    #'''TODO: feed the images into the model and obtain the predictions'''
    logits = cnn_model(images)

    #'''TODO: compute the categorical cross entropy loss
    loss_value = tf.keras.backend.sparse_categorical_crossentropy(labels, logits) # TODO

  loss_history.append(loss_value.numpy().mean()) # append the loss to the loss_history record
  plotter.plot(loss_history.get())

  # Backpropagation
  '''TODO: Use the tape to compute the gradient against all parameters in the CNN model.
      Use cnn_model.trainable_variables to access these parameters.'''
  grads = tape.gradient(loss_value, cnn_model.trainable_variables)
  optimizer.apply_gradients(zip(grads, cnn_model.trainable_variables))
_____no_output_____

## 1.5 Conclusion
In this part of the lab, you had the chance to play with different MNIST classifiers with different architectures (fully-connected layers only, CNN), and experiment with how different hyperparameters affect accuracy (learning rate, etc.). The next part of the lab explores another application of CNNs, facial detection, and some drawbacks of AI systems in real world applications, like issues of bias. _____no_output_____


# Notebook from mariolpantunes/ml101
Path: projects/00 refresher/00_linear_regression_00.ipynb

# Linear Regression

The idea of this project is to develop three different versions of the Linear Regression model:
1. Using the non-differential objective function (Differential Evolution)
2. Using the gradient descent (manual derivation)
3. Using the gradient descent (with autograd)

The linear regression models is defined by:
$$
h(x,m,b) = m\times x + b
$$

The tipical cost function to compute fit the linear regression is the following:
$$
e = \frac{\sum_{i=0}^{m}(y_i-h(x_i, m, b))}{2m}
$$_____no_output_____

%matplotlib inline
import matplotlib.pyplot as plt
import tqdm
import pandas as pd
import numpy as np
import jax.numpy as jnp
from jax import grad, jit
import optimization.de as de
plt.rcParams['figure.figsize'] = [10, 5]_____no_output_____

## Load the dataset_____no_output_____

data = pd.read_csv('https://media.githubusercontent.com/media/mariolpantunes/ml101/main/datasets/heart.csv')
data = data[['age', 'trestbps']]
data = data.rename(columns={"age":"age", "trestbps":"bps"})
data.head()_____no_output_____# divide the dataset
x = data[['age']]
y = data[['bps']]

# convert to numpy arrays
x = x.to_numpy()[:,0]
y = y.to_numpy()[:,0]

# plot the data
plt.plot(x, y, 'o')
plt.show()_____no_output_____def rmse(a, b):
    return np.sqrt(np.mean(np.power(a-b,2)))_____no_output_____

## Differential Evolution_____no_output_____

class LRDE:
    def fit(self, x, y):
        pass

    def predict(self, x):
        return x*self.w[0]+self.w[1]

    def params(self):
        return self.w_____no_output_____lr = LRDE()
lr.fit(x,y)_____no_output_____y_hat = f(lr.params())
# plot the data
plt.plot(x, y, 'o')
plt.plot(x, y_hat, 'ro')
plt.show()
_____no_output_____cost = rmse(y,y_hat)
print(f'RMSE = {cost}')_____no_output_____

## Gradient Descent (Manual)_____no_output_____

class LR:
    def fit(self, X, Y, L=0.0001):
        self.m = 0.0
        self.c = 0.0
        for _ in tqdm.tqdm(range(1000)):
            pass

    def predict(self, x):
        return x*self.m+self.c

    def params(self):
        return (self.m, self.c)_____no_output_____lr = LR()
lr.fit(x,y)
print(f'LR {lr.params()}')_____no_output_____y_hat = f(lr.params())
# plot the data
plt.plot(x, y, 'o')
plt.plot(x, y_hat, 'ro')
plt.show()_____no_output_____cost = rmse(y,y_hat)
print(f'RMSE = {cost}')_____no_output_____

### Gradient Descent (Autograd)_____no_output_____

class LRAG:
    def fit(self, X, Y, L=0.0001):
        self.m = 0.0
        self.b = 0.0
        for _ in tqdm.tqdm(range(1000)):
            pass

    def predict(self, x):
        return x*self.m+self.b

    def params(self):
        return (self.m, self.b)_____no_output_____lr = LRAG()
lr.fit(x,y)
print(f'LR {lr.params()}')_____no_output_____y_hat = f(lr.params())
# plot the data
plt.plot(x, y, 'o')
plt.plot(x, y_hat, 'ro')
plt.show()_____no_output_____cost = rmse(y,y_hat)
print(f'RMSE = {cost}')_____no_output_____


# Notebook from Merck/BioPhi-2021-publication
Path: notebooks/reports/02_oas_germline_prevalence.ipynb

<code>
import pandas as pd
import numpy as np
import os
import seaborn as sns
from matplotlib import pyplot as plt
from abnumber import Chain
from bin.evaluation import plot_oasis_curves

%config InlineBackend.figure_format = 'retina'
%matplotlib inline_____no_output_____oasis_curves = pd.read_excel('../../data/germlines/imgt_human_v_genes_raw_oasis.xlsx', sheet_name='OASis Curves', index_col=0)
oasis_curves.insert(0, 'Family', oasis_curves.index.map(lambda i: i.split('-')[0].split('/')[0]))
oasis_curves.insert(1, 'Chain', oasis_curves.index.map(lambda i: i[2]))
oasis_curves.head()_____no_output_____COLORS = [plt.get_cmap('tab20')(i) for i in range(20)]
COLORS_____no_output_____COLORS = [
     (0.12156862745098039, 0.4666666666666667, 0.7058823529411765, 1.0),
     (0.6823529411764706, 0.7803921568627451, 0.9098039215686274, 1.0),
     (1.0, 0.4980392156862745, 0.054901960784313725, 1.0),
     (1.0, 0.7333333333333333, 0.47058823529411764, 1.0),
     (0.17254901960784313, 0.6274509803921569, 0.17254901960784313, 1.0),
     (0.596078431372549, 0.8745098039215686, 0.5411764705882353, 1.0),
     (0.8392156862745098, 0.15294117647058825, 0.1568627450980392, 1.0),
     (1.0, 0.596078431372549, 0.5882352941176471, 1.0),
     (0.5803921568627451, 0.403921568627451, 0.7411764705882353, 1.0),
     (0.7725490196078432, 0.6901960784313725, 0.8352941176470589, 1.0),
     (0.5490196078431373, 0.33725490196078434, 0.29411764705882354, 1.0),
     (0.7686274509803922, 0.611764705882353, 0.5803921568627451, 1.0),
     (0.8901960784313725, 0.4666666666666667, 0.7607843137254902, 1.0),
     (0.9686274509803922, 0.7137254901960784, 0.8235294117647058, 1.0),
     (0.4980392156862745, 0.4980392156862745, 0.4980392156862745, 1.0),
     (0.7803921568627451, 0.7803921568627451, 0.7803921568627451, 1.0),
     (0.7372549019607844, 0.7411764705882353, 0.13333333333333333, 1.0),
     (0.8588235294117647, 0.8588235294117647, 0.5529411764705883, 1.0),
     (0.09019607843137255, 0.7450980392156863, 0.8117647058823529, 1.0),
     (0.6196078431372549, 0.8549019607843137, 0.8980392156862745, 1.0),
]_____no_output_____fig, axes = plt.subplots(1, 3, figsize=(15, 5))
for chain, ax in zip('HKL', axes):
    mean = oasis_curves[oasis_curves['Chain'] == chain].groupby('Family').mean()
    mean = mean.loc[sorted(mean.index, key=lambda s: int(''.join(a for a in s if a.isnumeric())))]
    colors = dict(zip(mean.index, COLORS))
    plot_oasis_curves(mean, errorbars=False, ax=ax, lw=2, colors=colors, markers='osx+vD<>^1234')
    ax.set_ylim(0, 1.02)
    ax.legend(loc='lower center', bbox_to_anchor=(0.5, 1), frameon=False, handletextpad=1, ncol=3)
fig.tight_layout();_____no_output_____fig, axes = plt.subplots(1, 3, figsize=(15, 5))
for chain, ax in zip('HKL', axes):
    mean = oasis_curves[oasis_curves['Chain'] == chain].groupby('Family').mean()
    mean = mean.loc[sorted(mean.index, key=lambda s: int(''.join(a for a in s if a.isnumeric())))]
    colors = dict(zip(mean.index, COLORS))
    plot_oasis_curves(oasis_curves, groupby='Family', ax=ax, lw=2, colors=colors, fill_alpha=0.05, markers='osx+vD<>^1234')
    ax.set_ylim(0, 1.02)
    ax.legend(loc='lower center', bbox_to_anchor=(0.5, 1), frameon=False, handletextpad=1, ncol=3)
fig.tight_layout();/Users/prihodad/Documents/bioinformatics/projects/github/BioPhi-2021-publication/bin/evaluation.py:230: FutureWarning: Dropping invalid columns in DataFrameGroupBy.quantile is deprecated. In a future version, a TypeError will be raised. Before calling .quantile, select only columns which should be valid for the function.
  curves_low = grouped.quantile(0.25)[columns]
/Users/prihodad/Documents/bioinformatics/projects/github/BioPhi-2021-publication/bin/evaluation.py:231: FutureWarning: Dropping invalid columns in DataFrameGroupBy.quantile is deprecated. In a future version, a TypeError will be raised. Before calling .quantile, select only columns which should be valid for the function.
  curves_high = grouped.quantile(0.75)[columns]
/Users/prihodad/Documents/bioinformatics/projects/github/BioPhi-2021-publication/bin/evaluation.py:230: FutureWarning: Dropping invalid columns in DataFrameGroupBy.quantile is deprecated. In a future version, a TypeError will be raised. Before calling .quantile, select only columns which should be valid for the function.
  curves_low = grouped.quantile(0.25)[columns]
/Users/prihodad/Documents/bioinformatics/projects/github/BioPhi-2021-publication/bin/evaluation.py:231: FutureWarning: Dropping invalid columns in DataFrameGroupBy.quantile is deprecated. In a future version, a TypeError will be raised. Before calling .quantile, select only columns which should be valid for the function.
  curves_high = grouped.quantile(0.75)[columns]
/Users/prihodad/Documents/bioinformatics/projects/github/BioPhi-2021-publication/bin/evaluation.py:230: FutureWarning: Dropping invalid columns in DataFrameGroupBy.quantile is deprecated. In a future version, a TypeError will be raised. Before calling .quantile, select only columns which should be valid for the function.
  curves_low = grouped.quantile(0.25)[columns]
/Users/prihodad/Documents/bioinformatics/projects/github/BioPhi-2021-publication/bin/evaluation.py:231: FutureWarning: Dropping invalid columns in DataFrameGroupBy.quantile is deprecated. In a future version, a TypeError will be raised. Before calling .quantile, select only columns which should be valid for the function.
  curves_high = grouped.quantile(0.75)[columns]
# in how many subjects do we find all the peptides?
prevalence_subjects = (oasis_curves[[c for c in oasis_curves if '%' in c]] >= 0.9999).apply(lambda row: row[row].index[-1], axis=1)
prevalence_subjects = prevalence_subjects.str.replace('%','').astype(float)
prevalence_subjects.sort_values()_____no_output_____sns.kdeplot(
    x=prevalence_subjects,
    hue=prevalence_subjects.index.map(lambda c: c[2]),
    bw_adjust=0.2,
)_____no_output_____prevalence_best_by_family = prevalence_subjects.groupby([oasis_curves['Family'], oasis_curves['Chain']])
prevalence_best_by_family = prevalence_best_by_family.apply(lambda rows: rows.sort_values(ascending=False)[:3])
prevalence_best_by_family = prevalence_best_by_family.rename('Subjects').reset_index()
prevalence_best_by_family.head()_____no_output_____with sns.axes_style('whitegrid'):
    fig, axes = plt.subplots(1, 3, figsize=(10, 6))
    for i, (chain_type, prevalence) in enumerate(prevalence_best_by_family.groupby('Chain')):
        families = prevalence['Family'].unique().tolist()
        ax = sns.barplot(
            data=prevalence,
            y='Antibody',
            x='Subjects',
            palette=[COLORS[families.index(f)] for f in prevalence['Family']],
            ax=axes[i]
        )
        ax.set_xticks([0, 25, 50, 75, 100])
        ax.set_xlabel('Subjects (%)')
        ax.set_ylabel('Germline')
    fig.tight_layout();_____no_output_____#germline_peptides = pd.read_excel('../../data/germlines/imgt_human_v_genes_raw_oasis.xlsx', sheet_name=None)_____no_output_____
</code>


# Notebook from CompBiochBiophLab/IntroBioinfo
Path: WorkingWithGenomes-FASTQ.ipynb

Adapted from [https://github.com/PacktPublishing/Bioinformatics-with-Python-Cookbook-Second-Edition](https://github.com/PacktPublishing/Bioinformatics-with-Python-Cookbook-Second-Edition), Chapter 2.

First, we will download the genome for the Yoruba poulation. You can check reference genomes for different populations in the [1000 genomes database](https://www.internationalgenome.org/data-portal/sample)_____no_output_____

!rm -f data/SRR003265.filt.fastq.gz 2>/dev/null
!wget -P data/ -nd ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/NA18489/sequence_read/SRR003265.filt.fastq.gz --2020-04-23 13:52:31--  ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/NA18489/sequence_read/SRR003265.filt.fastq.gz
           => «data/SRR003265.filt.fastq.gz»
S'està resolent ftp.1000genomes.ebi.ac.uk (ftp.1000genomes.ebi.ac.uk)… 193.62.197.77
S'està connectant a ftp.1000genomes.ebi.ac.uk (ftp.1000genomes.ebi.ac.uk)|193.62.197.77|:21… connectat.
S'està entrant com a «anonymous» … S'ha entrat amb èxit!
==> SYST ... fet.     ==> PWD ... fet.
==> TYPE I ... fet.   ==> CWD (1) /vol1/ftp/phase3/data/NA18489/sequence_read ... fet.
==> SIZE SRR003265.filt.fastq.gz ... 28919712
==> PASV ... fet.     ==> RETR SRR003265.filt.fastq.gz ... fet.
Mida: 28919712 (28M) (no autoritatiu)

SRR003265.filt.fast 100%[===================>]  27,58M   478KB/s    in 88s

2020-04-23 13:54:01 (323 KB/s) - s'ha desat «data/SRR003265.filt.fastq.gz» [28919712]

First let us open the file. It is gzipped, so we need to take this into account
The [FASTQ](https://drive5.com/usearch/manual/fastq_files.html) file contains info with quality scores. For example:
```
@SRR003265.818 3042NAAXX:3:1:1926:1464 length=51
GAAAAAAATCCGTGTATAGATGGACCTGCACAGTTTAAACCTGTGTTGTTC
+
IIIIIIIIIIIIIIIIIII:IIIIIIIIIDIIIIIICI-7BII?IAIIII8
```

where the symbols in the last row contain the Phred quality scores. From lower to higher quality:
```
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
```
_____no_output_____

import gzip

%matplotlib inline
import seaborn as sns
import matplotlib.pyplot as plt

from Bio import SeqIO

recs = SeqIO.parse(gzip.open('data/SRR003265.filt.fastq.gz', 'rt', encoding='utf-8'), 'fastq')
rec = next(recs)
print(rec.id, rec.description, rec.seq)
print(rec.letter_annotations)SRR003265.31 SRR003265.31 3042NAAXX:3:1:1252:1819 length=51 GGGAAAAGAAAAACAAACAAACAAAAACAAAACACAGAAACAAAAAAACCA
{'phred_quality': [40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 30, 23, 40, 32, 35, 29, 40, 16, 40, 40, 32, 35, 31, 40, 40, 39, 22, 40, 24, 20, 28, 31, 12, 31, 10, 22, 28, 13, 26, 20, 23, 23]}

let us have a look at the distribution of the nucleotide reads_____no_output_____

from collections import defaultdict
recs = SeqIO.parse(gzip.open('data/SRR003265.filt.fastq.gz', 'rt', encoding='utf-8'), 'fastq')

# let us define a default dictionary of integers.
# The defauldict is a useful way to simplfy the creation of dictionaries
cnt = defaultdict(int)
for rec in recs:
    for letter in rec.seq:
        cnt[letter] += 1
tot = sum(cnt.values())
for letter, cnt in cnt.items():
    print('%s: %.2f %d' % (letter, 100. * cnt / tot, cnt))G: 20.68 5359334
A: 28.60 7411965
C: 21.00 5444053
T: 29.58 7666885
N: 0.14 37289

The amount of Ns will be much higher in an unfiltered FASTQ file that directly comes from the
sequencer. Let us plot the distribution of Ns according to its read position:_____no_output_____

recs = SeqIO.parse(gzip.open('data/SRR003265.filt.fastq.gz', 'rt', encoding='UTF-8'), 'fastq')
n_cnt = defaultdict(int)
for rec in recs:
    for i, letter in enumerate(rec.seq):
        pos = i + 1
        if letter == 'N':
            n_cnt[pos] += 1
seq_len = max(n_cnt.keys())
positions = range(1, seq_len + 1)
fig, ax = plt.subplots(figsize=(16,9))
ax.plot(positions, [n_cnt[x] for x in positions])
ax.set_xlim(1, seq_len)_____no_output_____

Let us evaluate the quality_____no_output_____

recs = SeqIO.parse(gzip.open('data/SRR003265.filt.fastq.gz', 'rt', encoding='utf-8'), 'fastq')
cnt_qual = defaultdict(int)
for rec in recs:
    for i, qual in enumerate(rec.letter_annotations['phred_quality']):
        if i < 25:
            continue
        cnt_qual[qual] += 1
tot = sum(cnt_qual.values())
for qual, cnt in cnt_qual.items():
    print('%d: %.2f %d' % (qual, 100. * cnt / tot, cnt))40: 22.76 3007221
16: 2.05 270431
32: 1.03 136763
35: 0.91 120704
31: 1.08 142464
39: 0.78 102568
22: 1.59 209905
24: 1.46 193259
20: 1.73 228923
28: 1.23 162226
12: 2.35 309918
10: 4.95 654154
13: 2.28 301033
26: 1.33 176263
23: 1.52 201164
37: 0.84 111179
34: 0.95 125624
21: 1.66 219602
30: 1.13 149449
33: 0.99 131291
25: 1.40 184846
17: 1.97 259779
38: 0.80 106290
4: 4.77 630923
36: 0.88 115701
18: 1.88 248982
11: 2.41 318303
19: 1.81 239621
5: 4.88 645266
9: 2.51 332259
3: 4.04 533458
27: 1.28 168902
8: 2.53 334410
29: 1.17 154892
15: 2.12 280719
7: 2.51 331743
2: 3.77 498679
14: 2.20 291341
6: 2.50 330834
1: 1.52 200558
0: 0.40 52229
recs = SeqIO.parse(gzip.open('data/SRR003265.filt.fastq.gz', 'rt', encoding='utf-8'), 'fastq')
qual_pos = defaultdict(list)
for rec in recs:
    for i, qual in enumerate(rec.letter_annotations['phred_quality']):
        if i < 25 or qual == 40:
            continue
        pos = i + 1
        qual_pos[pos].append(qual)
vps = []
poses = list(qual_pos.keys())
poses.sort()
for pos in poses:
    vps.append(qual_pos[pos])
fig, ax = plt.subplots(figsize=(16,9))
sns.boxplot(data=vps, ax=ax)
ax.set_xticklabels([str(x) for x in range(26, max(qual_pos.keys()) + 1)])
pass_____no_output_____


# Notebook from g2nb/workshop-notebooks
Path: 2021_12_15_ccmi_single_cell_analysis_workshop/Reusable Notebook Cell Library/Reusable Notebook Cell Library.ipynb

# Requires GenePattern Notebook: pip install genepattern-notebook
import gp
import genepattern

# Username and password removed for security reasons.
genepattern.display(genepattern.session.register("https://cloud.genepattern.org/gp", "", ""))_____no_output_____

# Reusable Notebook Cell Library

Below are a collection of self-contained cells that are useful for reuse in other notebooks.  Each is self-contained (i.e. you can cut and paste it as is within the GenePattern Notebook Environment with no additional dependencies).

When pasted into a new notebook you will need a GenePattern login cell to be present, and then you will need to 'Run' the cell to cause the UI to be displayed._____no_output_____## Uploading Data to GenePattern_____no_output_____

@genepattern.build_ui(name="Upload from Notebook server to GenePattern",collapse=False,
                      description="Upload a file from the Notebook to the GenePattern server. Displays a URL on the GenePattern server as output.",
                      parameters={  "output_var":{"hide": True,},})
# upload the local file onto the GenePattern server so we can use it in the module
def uploadLocalNotebookFile(path_to_file_on_GPNB):
    filename = filenameAndPath.split("/")[1]
    uploaded = genepattern.get_session(0).upload_file(filename, path_to_file_on_GPNB)
    uploaded_url = uploaded.get_url()
    display(genepattern.GPUIOutput(files=[uploaded_url]))_____no_output_____import genepattern
import urllib
import os
import re
import shutil
from IPython.core.display import HTML

def isUrl(anUrl):
    regex = re.compile(
        r'^(?:http|ftp)s?://' # http:// or https://
        r'(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+(?:[A-Z]{2,6}\.?|[A-Z0-9-]{2,}\.?)|' #domain...
        r'localhost|' #localhost...
        r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})' # ...or ip
        r'(?::\d+)?' # optional port
        r'(?:/?|[/?]\S+)$', re.IGNORECASE)

    return (re.match(regex, anUrl) is not None )

def downloadLinkToGPNB(anURL, FileName):
    print('a')
    urllib.request.urlretrieve(anURL, FileName)
    return './'+FileName  # the link path

def downloadLinkToGenePatternServer(anURL, filename):

    downloadurl_task = gp.GPTask(genepattern.get_session(0), 'urn:lsid:broad.mit.edu:cancer.software.genepattern.module.analysis:00034')
    downloadurl_task.param_load()
    downloadurl_job_spec = downloadurl_task.make_job_spec()
    downloadurl_job_spec.set_parameter("URL", anURL)
    downloadurl_job_spec.set_parameter("output.file", filename)
    job = genepattern.get_session(0).run_job(downloadurl_job_spec)
    output_list = job.get_output_files()

    for file in output_list:     # Loop through each output file
        return file.get_url()

def uploadFileToGenePatternServer(File_or_URL, FileName):
    uploaded = genepattern.get_session(0).upload_file(FileName, File_or_URL)
    my_local_url = uploaded.get_url()
    return my_local_url

#@genepattern.build_ui( name="Upload file",
#                      description="Upload a file or URL to the GenePattern server or the GPNB server",
#                parameters={
#               "File_or_URL": {
#                    "type": "file"
#                },
#                "FileName": {
#                    "type": "String"
#                },
#                "Where_To": {
#                    "type": "choice",
#                    "choices" : {"GPNB":"GPNB", "GPServer":"GP"}
#                },
#                "output_var": {
#                    "hide": True
#                }
#})
def _uploadFileAndGetLocalUrl(File_or_URL, FileName, Where_To):
    link = None

    if isUrl(File_or_URL):
        if (Where_To == "GPNB"):
            print("Save URL to GPNB " )
            link = downloadLinkToGPNB(File_or_URL, FileName)

        else:
            print("Save URL to GenePattern ")
            link = downloadLinkToGenePatternServer(File_or_URL, FileName)

    else:
        # its a file, ui builder should already have uploaded it locally
        if (Where_To == "GPNB"):
            print("Save file: all done its here in the current dir " )
            link = './'+FileName
        else:
            print("Save file to GenePattern ")
            link = uploadFileToGenePatternServer(File_or_URL, FileName)

    # now save it back as a new file local to the Notebook server
    # with open(output_gct_filename, 'w') as f:
    #    f.writelines('#1.2\n{}\t{}\n'.format(df2.shape[0], df2.shape[1] - 2))
    #    df2.to_csv(f, sep='\t', index= False)

    # upload the local file onto the GenePattern server so we can use it in the next module
    #
    # return "my_local_url"
    #return HTML(''+FileName+'')
    #return HTML('
'+FileName+'

'+FileName+'Download FileOpen in New TabSend to Code
Send to Existing GenePattern Cell<select class="form-control gp-widget-job-existing-task" style="margin-left: 0px;"><option>----</option></select>

Send to New GenePattern Cell<select class="form-control gp-widget-job-new-task" style="margin-left: 0px;"><option>----</option></select>

')
    return genepattern.GPUIOutput(files=[link])

def printAndFlush(str):
    print(str)
    sys.stdout.flush()

def DownloadJobResultFile(Job_File_Url, File_Name):
    if os.path.isfile(File_Name):
        print("file already exists!")
        return

    # extract job number and file name from url
    job_num = Job_File_Url.split("/")[-2]
    remote_file_name = Job_File_Url.split("/")[-1]

    # get the job based on the job number passed as the second argument
    job = gp.GPJob(genepattern.get_session(0), job_num)

    # fetch a specific file from the job
    remote_file = job.get_file(remote_file_name)

    printAndFlush("downloading file...")
    response = remote_file.open()
    CHUNK = 16 * 1024
    with open(File_Name, 'wb') as f:
        while True:
            chunk = response.read(CHUNK)
            if not chunk:
                break
            f.write(chunk)
    printAndFlush("done!")

uploadFileUI = genepattern.GPUIBuilder(_uploadFileAndGetLocalUrl, name="Upload file",
                      description="Upload a file or URL to the GenePattern server or the GPNB server",
                parameters={
                "File_or_URL": {
                    "type": "file"
                },
                "FileName": {
                    "type": "String"
                },
                "Where_To": {
                    "type": "choice",
                    "choices" : {"GPNB":"GPNB", "GPServer":"GP"}
                },
                "output_var": {
                    "hide": True
                }
})

uploadFileUI_____no_output_____

## Displaying Data_____no_output_____

import os
import requests
import genepattern
from IPython.display import IFrame
@genepattern.build_ui(name="Display PDF", parameters={
    "image": {
        "name": "PDF to display:",
        "description": "PDF file (typically named Rplots.pdf) from the Seurat.QC module",
        "type": "file",
        "kinds": ["pdf"]
    },
    "height":{"default":850, "hide":True},
    "width":{"default":850, "hide":True},
    "output_var": {
        "default":"output_var",
        "hide": True
    }
})
def displayPdf(image, height, width):
    job_widget = nbtools.UIOutput(status="Getting file from the GenePattern server...")
    display(job_widget)
    f = gp.GPFile(genepattern.session.get(0), image)
    basename=os.path.basename(image)
    resp = requests.get(image, headers={
        'Authorization': f.server_data.authorization_header(),
        'User-Agent': 'GenePatternRest'})
    job_widget.status = 'Writing pdf file to your workspace. This may take a minute.'
    with open(basename, "wb") as f:
        f.write(resp.content)

    job_widget.status = basename+' successfully written to the same folder as this notebook!'
    display(IFrame(basename,width, height))
    return_____no_output_____from IPython.core.display import display, HTML, display_html
from IPython.display import Image

%matplotlib inline

displayImageDescription = "Display an image within a cell of the notebook."
displayImageParamDef = {"image":{  "name":"Image file URL", "type":"file", "kinds":["png", "jpg","gif"]},
                        "image_1":{"name":"First image file URL", "type":"file", "kinds":["png", "jpg","gif"]},
                        "image_2":{"name":"Second image file URL", "type":"file", "kinds":["png", "jpg","gif"]},
                        "height": {"type": "number", "optional": True},
                        "width": {"type":"number", "optional": True},
                        "output_var":{"hide":True }}

def displayImage(image, height, width):
    return Image(url= image, width=width, height=height)

genepattern.GPUIBuilder(displayImage, collapse=False,
                        name="Display Image",
                        description=displayImageDescription,
                        parameters=displayImageParamDef)_____no_output_____from IPython.core.display import display, HTML, display_html
from IPython.display import Image

%matplotlib inline

displayImagePairDescription = "Display a pair of images side-by-side within a cell of the notebook."
displayImagePairParamDef = {"image":{  "name":"Image file URL", "type":"file", "kinds":["png", "jpg","gif"]},
                        "image_1":{"name":"First image file URL", "type":"file", "kinds":["png", "jpg","gif"]},
                        "image_2":{"name":"Second image file URL", "type":"file", "kinds":["png", "jpg","gif"]},
                        "height": {"type": "number", "optional": True},
                        "width": {"type":"number", "optional": True},
                        "output_var":{"hide":True }}

def displayImagePair(image_1, image_2, height, width):

    # display(Image(url= image_1, width=width, height=height), Image(url= image_2, width=width, height=height))
    # display(Image(url= image_2, width=width, height=height))
    tdstyle=style='vertical-align: bottom;background-color:white'
    html_str = "

"
    display_html(html_str, raw=True)
    return None

genepattern.GPUIBuilder(displayImagePair, collapse=False,
                        name="Display Image Pair",
                        description=displayImagePairDescription,
                        parameters=displayImagePairParamDef)_____no_output_____@genepattern.build_ui(name="List Column Names", collapse=False, description="Display the column names from a tsv file.",
                      parameters={  "output_var":{"hide": True,},})
def listColumnNames(path_to_file_on_GPNB):
    df = pd.read_csv( path_to_file_on_GPNB, sep='\t', header=0, index_col=False)
    display(list(df.columns.values))_____no_output_____

## Utility functions_____no_output_____

import pandas as pd
import gp.data as gpdata

@genepattern.build_ui(name="Convert TSV File into GCT", collapse=False, description="Extracts data from TSV file into GCT format and upload gct to the server ",
                      parameters={  "output_var":{"hide": True,},})
# upload the local file onto the GenePattern server so we can use it in the module
def convertTSVDataIntoGCT(input_filename, output_filename, index_column, description_column, values_column_suffix):
    # read TSV into a data frame
    df = pd.read_csv(input_filename, sep='\t', header=0, index_col=False)
    # drop the default numeric row index
    df.reset_index(drop=True, inplace=True)
    # rename the first 2 columns to match norman GenePattern conventions of Name and Description
    df.rename(columns={index_column: "Name", description_column: "Description"}, inplace=True)
    # Set the Name and description as indexes which a GCT file expects in a dataframe
    df.set_index(["Name", "Description"], inplace=True);

    df2 = df.loc[:, df.columns.str.endswith(values_column_suffix)]
    # Show the top of the file
    display(df2.head())
    # Now write the dataframe out as a gct formatted file
    gpdata.write_gct(df2, output_filename)
    print("Wrote "+ output_filename)
    _____no_output_____import gp, gp.data

@genepattern.build_ui(name="Read GCT file into DataFrame",
                      description="Reads a gct file on the GPNB server into a Pandas Dataframe in the current notebook.",
                      collapse=False, parameters={
    "output_var": {
        "default": "aDataFrame",
        "hide": False,
    }
})
def readGctIntoDataFrame(filename):
    gpfile = gp.GPFile(None, filename)
    df = gp.data.GCT(gpfile.uri)
    display(df.head())
    return df_____no_output_____import urllib

@genepattern.build_ui(collapse=False, name="Read GCT URL into DataFrame",
    description="Reads a GCT file from a remote URL into a Pandas Dataframe in the current notebook.", parameters={

    "output_var": {
        "default": "aDataFrameFromURL",
        "hide": False,
    }
})
def readGct_FromURL(link):
    df = gp.data.GCT(link)
    display(df.head())
    print("Loaded")
    return df_____no_output_____@genepattern.build_ui(description="Strip version numbers from ensemble ids and uploads to the GenePattern server.",
                      parameters={  "output_var":{"hide": True,},})
# strip the version # from the ensemble gene id and upload
# the local file onto the GenePattern server so we can use it in the module
def stripEnsembleVersionIds(filename, output_filename):
    df = gpdata.GCT(filename)
    df.reset_index(drop=False, inplace=True)
    df['Name'] = df['Name'].apply(lambda x: x.split(".")[0])
    df.set_index(["Name", "Description"], inplace=True);
    display(df.head())
    gpdata.write_gct(df, output_filename)

    # and finally upload it to the GenePattern server from the Notebook server and display the resulting link
    uploaded = genepattern.get_session(0).upload_file(output_filename, output_filename)
    uploaded_url = uploaded.get_url()
    display(genepattern.GPUIOutput(files=[uploaded_url]))_____no_output_____import gp, gp.data

@genepattern.build_ui(collapse=False, parameters={
    "output_var": {
        "hide": True,
    }
})
def readFile(filename, number_of_lines_to_print):
    count = 0;
    for line in open(filename):
        count = count + 1
        if (count > number_of_lines_to_print):
            break
        print(line)

           _____no_output_____