# Notebook from binblee/Data-Analysis-and-Machine-Learning-Projects
Path: Example Machine Learning Notebook.ipynb

#An example machine learning notebook

###Notebook by [Randal S. Olson](http://www.randalolson.com/)
####Supported by [Jason H. Moore](http://www.epistasis.org/)
####[University of Pennsylvania Institute for Bioinformatics](http://upibi.org/)_____no_output_____**It is recommended to [view this notebook in nbviewer](http://nbviewer.ipython.org/github/rhiever/Data-Analysis-and-Machine-Learning-Projects/blob/master/example-data-science-notebook/Example%20Machine%20Learning%20Notebook.ipynb) for the best viewing experience.**_____no_output_____##Table of contents

1. [Introduction](#Introduction)

2. [License](#License)

3. [Required libraries](#Required-libraries)

4. [The problem domain](#The-problem-domain)

5. [Step 1: Answering the question](#Step-1:-Answering-the-question)

6. [Step 2: Checking the data](#Step-2:-Checking-the-data)

7. [Step 3: Tidying the data](#Step-3:-Tidying-the-data)

    - [Bonus: Testing our data](#Bonus:-Testing-our-data)

8. [Step 4: Exploratory analysis](#Step-4:-Exploratory-analysis)

9. [Step 5: Classification](#Step-5:-Classification)

    - [Cross-validation](#Cross-validation)

    - [Parameter tuning](#Parameter-tuning)

10. [Step 6: Reproducibility](#Step-6:-Reproducibility)

11. [Conclusions](#Conclusions)

12. [Further reading](#Further-reading)

13. [Acknowledgements](#Acknowledgements)_____no_output_____##Introduction

[[ go back to the top ]](#Table-of-contents)

In the time it took you to read this sentence, terabytes of data have been collectively generated across the world — more data than any of us could ever hope to process, much less make sense of, on the machines we're using to read this notebook.

In response to this massive influx of data, the field of Data Science has come to the forefront in the past decade. Cobbled together by people from a diverse array of fields — statistics, physics, computer science, design, and many more — the field of Data Science represents our collective desire to understand and harness the abundance of data around us to build a better world.

In this notebook, I'm going to go over a basic Python data analysis pipeline from start to finish to show you what a typical data science workflow looks like.

In addition to providing code examples, I also hope to imbue in you a sense of good practices so you can be a more effective — and more collaborative — data scientist.

I will be following along with the data analysis checklist from [The Elements of Data Analytic Style](https://leanpub.com/datastyle), which I strongly recommend reading as a free and quick guidebook to performing outstanding data analysis.

**This notebook is intended to be a public resource. As such, if you see any glaring inaccuracies or if a critical topic is missing, please feel free to point it out or (preferably) submit a pull request to improve the notebook.**_____no_output_____##License

[[ go back to the top ]](#Table-of-contents)

Please see the [repository README file](https://github.com/rhiever/Data-Analysis-and-Machine-Learning-Projects#license) for the licenses and usage terms for the instructional material and code in this notebook. In general, I have licensed this material so that it is as widely usable and shareable as possible._____no_output_____##Required libraries

[[ go back to the top ]](#Table-of-contents)

If you don't have Python on your computer, you can use the [Anaconda Python distribution](http://continuum.io/downloads) to install most of the Python packages you need. Anaconda provides a simple double-click installer for your convenience.

This notebook uses several Python packages that come standard with the Anaconda Python distribution. The primary libraries that we'll be using are:

* **NumPy**: Provides a fast numerical array structure and helper functions.
* **pandas**: Provides a DataFrame structure to store data in memory and work with it easily and efficiently.
* **scikit-learn**: The essential Machine Learning package in Python.
* **matplotlib**: Basic plotting library in Python; most other Python plotting libraries are built on top of it.
* **Seaborn**: Advanced statistical plotting library.

To make sure you have all of the packages you need, install them with `conda`:

    conda install numpy pandas scikit-learn matplotlib seaborn

`conda` may ask you to update some of them if you don't have the most recent version. Allow it to do so.

**Note:** I will not be providing support for people trying to run this notebook outside of the Anaconda Python distribution._____no_output_____##The problem domain

[[ go back to the top ]](#Table-of-contents)

For the purposes of this exercise, let's pretend we're working for a startup that just got funded to create a smartphone app that automatically identifies species of flowers from pictures taken on the smartphone. We're working with a moderately-sized team of data scientists and will be building part of the data analysis pipeline for this app.

We've been tasked by our company's Head of Data Science to create a demo machine learning model that takes four measurements from the flowers (sepal length, sepal width, petal length, and petal width) and identifies the species based on those measurements alone.

We've been given a [data set](https://github.com/rhiever/Data-Analysis-and-Machine-Learning-Projects/raw/master/example-data-science-notebook/iris-data.csv) from our field researchers to develop the demo, which only includes measurements for three types of *Iris* flowers:

###*Iris setosa*

###*Iris versicolor*

###*Iris virginica*

The four measurements we're using currently come from hand-measurements by the field researchers, but they will be automatically measured by an image processing model in the future.

**Note:** The data set we're working with is the famous [*Iris* data set](https://archive.ics.uci.edu/ml/datasets/Iris) — included with this notebook — which I have modified slightly for demonstration purposes._____no_output_____##Step 1: Answering the question

[[ go back to the top ]](#Table-of-contents)

The first step to any data analysis project is to define the question or problem we're looking to solve, and to define a measure (or set of measures) for our success at solving that task. The data analysis checklist has us answer a handful of questions to accomplish that, so let's work through those questions.

>Did you specify the type of data analytic question (e.g. exploration, association causality) before touching the data?

We're trying to classify the species (i.e., class) of the flower based on four measurements that we're provided: sepal length, sepal width, petal length, and petal width.

>Did you define the metric for success before beginning?

Let's do that now. Since we're performing classification, we can use [accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precision) — the fraction of correctly classified flowers — to quantify how well our model is performing. Our company's Head of Data has told us that we should achieve at least 90% accuracy.

>Did you understand the context for the question and the scientific or business application?

We're building part of a data analysis pipeline for a smartphone app that will be able to classify the species of flowers from pictures taken on the smartphone. In the future, this pipeline will be connected to another pipeline that automatically measures from pictures the traits we're using to perform this classification.

>Did you record the experimental design?

Our company's Head of Data has told us that the field researchers are hand-measuring 50 randomly-sampled flowers of each species using a standardized methodology. The field researchers take pictures of each flower they sample from pre-defined angles so the measurements and species can be confirmed by the other field researchers at a later point. At the end of each day, the data is compiled and stored on a private company GitHub repository.

>Did you consider whether the question could be answered with the available data?

The data set we currently have is only for three types of *Iris* flowers. The model built off of this data set will only work for those *Iris* flowers, so we will need more data to create a general flower classifier.

Notice that we've spent a fair amount of time working on the problem without writing a line of code or even looking at the data.

**Thinking about and documenting the problem we're working on is an important step to performing effective data analysis that often goes overlooked.** Don't skip it._____no_output_____##Step 2: Checking the data

[[ go back to the top ]](#Table-of-contents)

The next step is to look at the data we're working with. Even curated data sets from the government can have errors in them, and it's vital that we spot these errors before investing too much time in our analysis.

Generally, we're looking to answer the following questions:

* Is there anything wrong with the data?
* Are there any quirks with the data?
* Do I need to fix or remove any of the data?

Let's start by reading the data into a pandas DataFrame._____no_output_____

import pandas as pd

iris_data = pd.read_csv('iris-data.csv')
iris_data.head()_____no_output_____

We're in luck! The data seems to be in a usable format.

The first row in the data file defines the column headers, and the headers are descriptive enough for us to understand what each column represents. The headers even give us the units that the measurements were recorded in, just in case we needed to know at a later point in the project.

Each row following the first row represents an entry for a flower: four measurements and one class, which tells us the species of the flower.

**One of the first things we should look for is missing data.** Thankfully, the field researchers already told us that they put a 'NA' into the spreadsheet when they were missing a measurement.

We can tell pandas to automatically identify missing values if it knows our missing value marker._____no_output_____

iris_data = pd.read_csv('iris-data.csv', na_values=['NA'])_____no_output_____

Voilà! Now pandas knows to treat rows with 'NA' as missing values._____no_output_____Next, it's always a good idea to look at the distribution of our data — especially the outliers.

Let's start by printing out some summary statistics about the data set._____no_output_____

iris_data.describe()_____no_output_____

We can see several useful values from this table. For example, we see that five `petal_width_cm` entries are missing.

If you ask me, though, tables like this are rarely useful unless we know that our data should fall in a particular range. It's usually better to visualize the data in some way. Visualization makes outliers and errors immediately stand out, whereas they might go unnoticed in a large table of numbers.

Since we know we're going to be plotting in this section, let's set up the notebook so we can plot inside of it._____no_output_____

# This line tells the notebook to show plots inside of the notebook
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sb_____no_output_____

Next, let's create a **scatterplot matrix**. Scatterplot matrices plot the distribution of each column along the diagonal, and then plot a scatterplot matrix for the combination of each variable. They make for an efficient tool to look for errors in our data.

We can even have the plotting package color each entry by its class to look for trends within the classes._____no_output_____

# We have to temporarily drop the rows with 'NA' values
# because the Seaborn plotting function does not know
# what to do with them
sb.pairplot(iris_data.dropna(), hue='class')_____no_output_____

From the scatterplot matrix, we can already see some issues with the data set:

1. There are five classes when there should only be three, meaning there were some coding errors.

2. There are some clear outliers in the measurements that may be erroneous: one `sepal_width_cm` entry for `Iris-setosa` falls well outside its normal range, and several `sepal_length_cm` entries for `Iris-versicolor` are near-zero for some reason.

3. We had to drop those rows with missing values.

In all of these cases, we need to figure out what to do with the erroneous data. Which takes us to the next step..._____no_output_____##Step 3: Tidying the data

[[ go back to the top ]](#Table-of-contents)

Now that we've identified several errors in the data set, we need to fix them before we proceed with the analysis.

Let's walk through the issues one-by-one.

>There are five classes when there should only be three, meaning there were some coding errors.

After talking with the field researchers, it sounds like one of them forgot to add `Iris-` before their `Iris-versicolor` entries. The other extraneous class, `Iris-setossa`, was simply a typo that they forgot to fix.

Let's use the DataFrame to fix these errors._____no_output_____

iris_data.loc[iris_data['class'] == 'versicolor', 'class'] = 'Iris-versicolor'
iris_data.loc[iris_data['class'] == 'Iris-setossa', 'class'] = 'Iris-setosa'

iris_data['class'].unique()_____no_output_____

Much better! Now we only have three class types. Imagine how embarrassing it would've been to create a model that used the wrong classes.

>There are some clear outliers in the measurements that may be erroneous: one `sepal_width_cm` entry for `Iris-setosa` falls well outside its normal range, and several `sepal_length_cm` entries for `Iris-versicolor` are near-zero for some reason.

Fixing outliers can be tricky business. It's rarely clear whether the outlier was caused by measurement error, recording the data in improper units, or if the outlier is a real anomaly. For that reason, we should be judicious when working with outliers: if we decide to exclude any data, we need to make sure to document what data we excluded and provide solid reasoning for excluding that data. (i.e., "This data didn't fit my hypothesis" will not stand peer review.)

In the case of the one anomalous entry for `Iris-setosa`, let's say our field researchers know that it's impossible for `Iris-setosa` to have a sepal width below 2.5 cm. Clearly this entry was made in error, and we're better off just scrapping the entry than spending hours finding out what happened._____no_output_____

# This line drops any 'Iris-setosa' rows with a separal width less than 2.5 cm
iris_data = iris_data.loc[(iris_data['class'] != 'Iris-setosa') | (iris_data['sepal_width_cm'] >= 2.5)]
iris_data.loc[iris_data['class'] == 'Iris-setosa', 'sepal_width_cm'].hist()_____no_output_____

Excellent! Now all of our `Iris-setosa` rows have a sepal width greater than 2.5.

The next data issue to address is the several near-zero sepal lengths for the `Iris-versicolor` rows. Let's take a look at those rows._____no_output_____

iris_data.loc[(iris_data['class'] == 'Iris-versicolor') &
              (iris_data['sepal_length_cm'] < 1.0)]_____no_output_____

How about that? All of these near-zero `sepal_length_cm` entries seem to be off by two orders of magnitude, as if they had been recorded in meters instead of centimeters.

After some brief correspondence with the field researchers, we find that one of them forgot to convert those measurements to centimeters. Let's do that for them._____no_output_____

iris_data.loc[(iris_data['class'] == 'Iris-versicolor') &
              (iris_data['sepal_length_cm'] < 1.0),
              'sepal_length_cm'] *= 100.0

iris_data.loc[iris_data['class'] == 'Iris-versicolor', 'sepal_length_cm'].hist()_____no_output_____

Phew! Good thing we fixed those outliers. They could've really thrown our analysis off.

>We had to drop those rows with missing values.

Let's take a look at the rows with missing values:_____no_output_____

iris_data.loc[(iris_data['sepal_length_cm'].isnull()) |
              (iris_data['sepal_width_cm'].isnull()) |
              (iris_data['petal_length_cm'].isnull()) |
              (iris_data['petal_width_cm'].isnull())]_____no_output_____

It's not ideal that we had to drop those rows, especially considering they're all `Iris-setosa` entries. Since it seems like the missing data is systematic — all of the missing values are in the same column for the same *Iris* type — this error could potentially bias our analysis.

One way to deal with missing data is **mean imputation**: If we know that the values for a measurement fall in a certain range, we can fill in empty values with the average of that measurement.

Let's see if we can do that here._____no_output_____

iris_data.loc[iris_data['class'] == 'Iris-setosa', 'petal_width_cm'].hist()_____no_output_____

Most of the petal widths for `Iris-setosa` fall within the 0.2-0.3 range, so let's fill in these entries with the average measured petal width._____no_output_____

average_petal_width = iris_data.loc[iris_data['class'] == 'Iris-setosa', 'petal_width_cm'].mean()

iris_data.loc[(iris_data['class'] == 'Iris-setosa') &
              (iris_data['petal_width_cm'].isnull()),
              'petal_width_cm'] = average_petal_width

iris_data.loc[(iris_data['class'] == 'Iris-setosa') &
              (iris_data['petal_width_cm'] == average_petal_width)]_____no_output_____iris_data.loc[(iris_data['sepal_length_cm'].isnull()) |
              (iris_data['sepal_width_cm'].isnull()) |
              (iris_data['petal_length_cm'].isnull()) |
              (iris_data['petal_width_cm'].isnull())]_____no_output_____

Great! Now we've recovered those rows and no longer have missing data in our data set.

**Note:** If you don't feel comfortable imputing your data, you can drop all rows with missing data with the `dropna()` call:

    iris_data.dropna(inplace=True)

After all this hard work, we don't want to repeat this process every time we work with the data set. Let's save the tidied data file *as a separate file* and work directly with that data file from now on._____no_output_____

iris_data.to_csv('iris-data-clean.csv', index=False)

iris_data_clean = pd.read_csv('iris-data-clean.csv')_____no_output_____

Now, let's take a look at the scatterplot matrix now that we've tidied the data._____no_output_____

sb.pairplot(iris_data_clean, hue='class')_____no_output_____

Of course, I purposely inserted numerous errors into this data set to demonstrate some of the many possible scenarios you may face while tidying your data.

The general takeaways here should be:

* Make sure your data is encoded properly

* Make sure your data falls within the expected range, and use domain knowledge whenever possible to define that expected range

* Deal with missing data in one way or another: replace it if you can or drop it

* Never tidy your data manually because that is not easily reproducible

* Use code as a record of how you tidied your data

* Plot everything you can about the data at this stage of the analysis so you can *visually* confirm everything looks correct_____no_output_____##Bonus: Testing our data

[[ go back to the top ]](#Table-of-contents)

At SciPy 2015, I was exposed to a great idea: We should test our data. Just how we use unit tests to verify our expectations from code, we can similarly set up unit tests to verify our expectations about a data set.

We can quickly test our data using `assert` statements: We assert that something must be true, and if it is, then nothing happens and the notebook continues running. However, if our assertion is wrong, then the notebook stops running and brings it to our attention. For example:_____no_output_____

assert 1 == 2_____no_output_____

Let's test a few things that we know about our data set now._____no_output_____

# We know that we should only have three classes
assert len(iris_data_clean['class'].unique()) == 3_____no_output_____# We know that sepal lengths for 'Iris-versicolor' should never be below 2.5 cm
assert iris_data_clean.loc[iris_data_clean['class'] == 'Iris-versicolor', 'sepal_length_cm'].min() >= 2.5_____no_output_____# We know that our data set should have no missing measurements
assert len(iris_data_clean.loc[(iris_data_clean['sepal_length_cm'].isnull()) |
                               (iris_data_clean['sepal_width_cm'].isnull()) |
                               (iris_data_clean['petal_length_cm'].isnull()) |
                               (iris_data_clean['petal_width_cm'].isnull())]) == 0_____no_output_____

And so on. If any of these expectations are violated, then our analysis immediately stops and we have to return to the tidying stage._____no_output_____##Step 4: Exploratory analysis

[[ go back to the top ]](#Table-of-contents)

Now after spending entirely too much time tidying our data, we can start analyzing it!

Exploratory analysis is the step where we start delving deeper into the data set beyond the outliers and errors. We'll be looking to answer questions such as:

* How is my data distributed?

* Are there any correlations in my data?

* Are there any confounding factors that explain these correlations?

This is the stage where we plot all the data in as many ways as possible. Create many charts, but don't bother making them pretty — these charts are for internal use.

Let's return to that scatterplot matrix that we used earlier._____no_output_____

sb.pairplot(iris_data_clean)_____no_output_____

Our data is normally distributed for the most part, which is great news if we plan on using any modeling methods that assume the data is normally distributed.

There's something strange going on with the petal measurements. Maybe it's something to do with the different `Iris` types. Let's color code the data by the class again to see if that clears things up._____no_output_____

sb.pairplot(iris_data_clean, hue='class')_____no_output_____

Sure enough, the strange distribution of the petal measurements exist because of the different species. This is actually great news for our classification task since it means that the petal measurements will make it easy to distinguish between `Iris-setosa` and the other `Iris` types.

Distinguishing `Iris-versicolor` and `Iris-virginica` will prove more difficult given how much their measurements overlap.

There are also correlations between petal length and petal width, as well as sepal length and sepal width. The field biologists assure us that this is to be expected: Longer flower petals also tend to be wider, and the same applies for sepals.

We can also make **violin plots** of the data to compare the measurement distributions of the classes. Violin plots contain the same information as [box plots](https://en.wikipedia.org/wiki/Box_plot), but also scales the box according to the density of the data._____no_output_____

plt.figure(figsize=(10, 10))

for column_index, column in enumerate(iris_data_clean.columns):
    if column == 'class':
        continue
    plt.subplot(2, 2, column_index + 1)
    sb.violinplot(x='class', y=column, data=iris_data_clean)_____no_output_____

Enough flirting with the data. Let's get to modeling._____no_output_____##Step 5: Classification

[[ go back to the top ]](#Table-of-contents)

Wow, all this work and we *still* haven't modeled the data!

As tiresome as it can be, tidying and exploring our data is a vital component to any data analysis. If we had jumped straight to the modeling step, we would have created a faulty classification model.

Remember: **Bad data leads to bad models.** Always check your data first.

Assured that our data is now as clean as we can make it — and armed with some cursory knowledge of the distributions and relationships in our data set — it's time to make the next big step in our analysis: Splitting the data into training and testing sets.

A **training set** is a random subset of the data that we use to train our models.

A **testing set** is a random subset of the data (mutually exclusive from the training set) that we use to validate our models on unforseen data.

Especially in sparse data sets like ours, it's easy for models to **overfit** the data: The model will learn the training set so well that it won't be able to handle most of the cases it's never seen before. This is why it's important for us to build the model with the training set, but score it with the testing set.

Note that once we split the data into a training and testing set, we should treat the testing set like it no longer exists: We cannot use any information from the testing set to build our model or else we're cheating.

Let's set up our data first._____no_output_____

iris_data_clean = pd.read_csv('iris-data-clean.csv')

# We're using all four measurements as inputs
# Note that scikit-learn expects each entry to be a list of values, e.g.,
# [ [val1, val2, val3],
#   [val1, val2, val3],
#   ... ]
# such that our input data set is represented as a list of lists

# We can extract the data in this format from pandas like this:
all_inputs = iris_data_clean[['sepal_length_cm', 'sepal_width_cm',
                             'petal_length_cm', 'petal_width_cm']].values

# Similarly, we can extract the classes
all_classes = iris_data_clean['class'].values

# Make sure that you don't mix up the order of the entries
# all_inputs[5] inputs should correspond to the class in all_classes[5]

# Here's what a subset of our inputs looks like:
all_inputs[:5]_____no_output_____

Now our data is ready to be split._____no_output_____

from sklearn.cross_validation import train_test_split

(training_inputs,
 testing_inputs,
 training_classes,
 testing_classes) = train_test_split(all_inputs, all_classes, train_size=0.75, random_state=1)_____no_output_____

With our data split, we can start fitting models to our data. Our company's Head of Data is all about decision tree classifiers, so let's start with one of those.

Decision tree classifiers are incredibly simple in theory. In their simplest form, decision tree classifiers ask a series of Yes/No questions about the data — each time getting closer to finding out the class of each entry — until they either classify the data set perfectly or simply can't differentiate a set of entries. Think of it like a game of [Twenty Questions](https://en.wikipedia.org/wiki/Twenty_Questions), except the computer is *much*, *much* better at it.

Here's an example decision tree classifier:

Notice how the classifier asks Yes/No questions about the data — whether a certain feature is <= 1.75, for example — so it can differentiate the records. This is the essence of every decision tree.

The nice part about decision tree classifiers is that they are **scale-invariant**, i.e., the scale of the features does not affect their performance, unlike many Machine Learning models. In other words, it doesn't matter if our features range from 0 to 1 or 0 to 1,000; decision tree classifiers will work with them just the same.

There are several [parameters](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) that we can tune for decision tree classifiers, but for now let's use a basic decision tree classifier._____no_output_____

from sklearn.tree import DecisionTreeClassifier

# Create the classifier
decision_tree_classifier = DecisionTreeClassifier()

# Train the classifier on the training set
decision_tree_classifier.fit(training_inputs, training_classes)

# Validate the classifier on the testing set using classification accuracy
decision_tree_classifier.score(testing_inputs, testing_classes)_____no_output_____

Heck yeah! Our model achieves 97% classification accuracy without much effort.

However, there's a catch: Depending on how our training and testing set was sampled, our model can achieve anywhere from 80% to 100% accuracy:_____no_output_____

model_accuracies = []

for repetition in range(1000):
    (training_inputs,
     testing_inputs,
     training_classes,
     testing_classes) = train_test_split(all_inputs, all_classes, train_size=0.75)

    decision_tree_classifier = DecisionTreeClassifier()
    decision_tree_classifier.fit(training_inputs, training_classes)
    classifier_accuracy = decision_tree_classifier.score(testing_inputs, testing_classes)
    model_accuracies.append(classifier_accuracy)

sb.distplot(model_accuracies)_____no_output_____

It's obviously a problem that our model performs quite differently depending on the subset of the data it's trained on. This phenomenon is known as **overfitting**: The model is learning to classify the training set so well that it doesn't generalize and perform well on data it hasn't seen before.

###Cross-validation

[[ go back to the top ]](#Table-of-contents)

This problem is the main reason that most data scientists perform ***k*-fold cross-validation** on their models: Split the original data set into *k* subsets, use one of the subsets as the testing set, and the rest of the subsets are used as the training set. This process is then repeated *k* times such that each subset is used as the testing set exactly once.

10-fold cross-validation is the most common choice, so let's use that here. Performing 10-fold cross-validation on our data set looks something like this:

(each square is an entry in our data set)_____no_output_____

import numpy as np
from sklearn.cross_validation import StratifiedKFold

def plot_cv(cv, n_samples):
    masks = []
    for train, test in cv:
        mask = np.zeros(n_samples, dtype=bool)
        mask[test] = 1
        masks.append(mask)

    plt.figure(figsize=(15, 15))
    plt.imshow(masks, interpolation='none')
    plt.ylabel('Fold')
    plt.xlabel('Row #')

plot_cv(StratifiedKFold(all_classes, n_folds=10), len(all_classes))_____no_output_____

You'll notice that we used **Stratified *k*-fold cross-validation** in the code above. Stratified *k*-fold keeps the class proportions the same across all of the folds, which is vital for maintaining a representative subset of our data set. (e.g., so we don't have 100% `Iris setosa` entries in one of the folds.)

We can perform 10-fold cross-validation on our model with the following code:_____no_output_____

from sklearn.cross_validation import cross_val_score

decision_tree_classifier = DecisionTreeClassifier()

# cross_val_score returns a list of the scores, which we can visualize
# to get a reasonable estimate of our classifier's performance
cv_scores = cross_val_score(decision_tree_classifier, all_inputs, all_classes, cv=10)
sb.distplot(cv_scores)
plt.title('Average score: {}'.format(np.mean(cv_scores)))_____no_output_____

Now we have a much more consistent rating of our classifier's general classification accuracy.

###Parameter tuning

[[ go back to the top ]](#Table-of-contents)

Every Machine Learning model comes with a variety of parameters to tune, and these parameters can be vitally important to the performance of our classifier. For example, if we severely limit the depth of our decision tree classifier:_____no_output_____

decision_tree_classifier = DecisionTreeClassifier(max_depth=1)

cv_scores = cross_val_score(decision_tree_classifier, all_inputs, all_classes, cv=10)
sb.distplot(cv_scores, kde=False)
plt.title('Average score: {}'.format(np.mean(cv_scores)))_____no_output_____

the classification accuracy falls tremendously.

Therefore, we need to find a systematic method to discover the best parameters for our model and data set.

The most common method for model parameter tuning is **Grid Search**. The idea behind Grid Search is simple: explore a range of parameters and find the best-performing parameter combination. Focus your search on the best range of parameters, then repeat this process several times until the best parameters are discovered.

Let's tune our decision tree classifier. We'll stick to only two parameters for now, but it's possible to simultaneously explore dozens of parameters if we want._____no_output_____

from sklearn.grid_search import GridSearchCV

decision_tree_classifier = DecisionTreeClassifier()

parameter_grid = {'max_depth': [1, 2, 3, 4, 5],
                  'max_features': [1, 2, 3, 4]}

cross_validation = StratifiedKFold(all_classes, n_folds=10)

grid_search = GridSearchCV(decision_tree_classifier,
                           param_grid=parameter_grid,
                           cv=cross_validation)

grid_search.fit(all_inputs, all_classes)
print('Best score: {}'.format(grid_search.best_score_))
print('Best parameters: {}'.format(grid_search.best_params_))Best score: 0.959731543624161
Best parameters: {'max_features': 4, 'max_depth': 3}

Now let's visualize the grid search to see how the parameters interact._____no_output_____

grid_visualization = []

for grid_pair in grid_search.grid_scores_:
    grid_visualization.append(grid_pair.mean_validation_score)

grid_visualization = np.array(grid_visualization)
grid_visualization.shape = (5, 4)
sb.heatmap(grid_visualization, cmap='Blues')
plt.xticks(np.arange(4) + 0.5, grid_search.param_grid['max_features'])
plt.yticks(np.arange(5) + 0.5, grid_search.param_grid['max_depth'][::-1])
plt.xlabel('max_features')
plt.ylabel('max_depth')_____no_output_____

Now we have a better sense of the parameter space: We know that we need a `max_depth` of at least 2 to allow the decision tree to make more than a one-off decision.

`max_features` doesn't really seem to make a big difference here as long as we have 2 of them, which makes sense since our data set has only 4 features and is relatively easy to classify. (Remember, one of our data set's classes was easily separable from the rest based on a single feature.)

Let's go ahead and use a broad grid search to find the best settings for a handful of parameters._____no_output_____

decision_tree_classifier = DecisionTreeClassifier()

parameter_grid = {'criterion': ['gini', 'entropy'],
                  'splitter': ['best', 'random'],
                  'max_depth': [1, 2, 3, 4, 5],
                  'max_features': [1, 2, 3, 4]}

cross_validation = StratifiedKFold(all_classes, n_folds=10)

grid_search = GridSearchCV(decision_tree_classifier,
                           param_grid=parameter_grid,
                           cv=cross_validation)

grid_search.fit(all_inputs, all_classes)
print('Best score: {}'.format(grid_search.best_score_))
print('Best parameters: {}'.format(grid_search.best_params_))Best score: 0.959731543624161
Best parameters: {'max_features': 4, 'max_depth': 3, 'splitter': 'best', 'criterion': 'gini'}

Now we can take the best classifier from the Grid Search and use that:_____no_output_____

decision_tree_classifier = grid_search.best_estimator_
decision_tree_classifier_____no_output_____

We can even visualize the decision tree with [GraphViz](http://www.graphviz.org/) to see how it's making the classifications:_____no_output_____

import sklearn.tree as tree
from sklearn.externals.six import StringIO

with open('iris_dtc.dot', 'w') as out_file:
    out_file = tree.export_graphviz(decision_tree_classifier, out_file=out_file)_____no_output_____

_____no_output_____(This classifier may look familiar from earlier in the notebook.)

Alright! We finally have our demo classifier. Let's create some visuals of its performance so we have something to show our company's Head of Data._____no_output_____

dt_scores = cross_val_score(decision_tree_classifier, all_inputs, all_classes, cv=10)

sb.boxplot(dt_scores)
sb.stripplot(dt_scores, jitter=True, color='white')_____no_output_____

Hmmm... that's a little boring by itself though. How about we compare another classifier to see how they perform?

We already know from previous projects that Random Forest classifiers usually work better than individual decision trees. A common problem that decision trees face is that they're prone to overfitting: They complexify to the point that they classify the training set near-perfectly, but fail to generalize to data they have not seen before.

**Random Forest classifiers** work around that limitation by creating a whole bunch of decision trees (hence "forest") — each trained on random subsets of training samples (drawn with replacement) and features (drawn without replacement) — and have the decision trees work together to make a more accurate classification.

Let that be a lesson for us: **Even in Machine Learning, we get better results when we work together!**

Let's see if a Random Forest classifier works better here.

The great part about scikit-learn is that the training, testing, parameter tuning, etc. process is the same for all models, so we only need to plug in the new classifier._____no_output_____

from sklearn.ensemble import RandomForestClassifier

random_forest_classifier = RandomForestClassifier()

parameter_grid = {'n_estimators': [5, 10, 25, 50],
                  'criterion': ['gini', 'entropy'],
                  'max_features': [1, 2, 3, 4],
                  'warm_start': [True, False]}

cross_validation = StratifiedKFold(all_classes, n_folds=10)

grid_search = GridSearchCV(random_forest_classifier,
                           param_grid=parameter_grid,
                           cv=cross_validation)

grid_search.fit(all_inputs, all_classes)
print('Best score: {}'.format(grid_search.best_score_))
print('Best parameters: {}'.format(grid_search.best_params_))

grid_search.best_estimator_Best score: 0.9731543624161074
Best parameters: {'n_estimators': 5, 'max_features': 3, 'warm_start': True, 'criterion': 'gini'}

Now we can compare their performance:_____no_output_____

random_forest_classifier = grid_search.best_estimator_

rf_df = pd.DataFrame({'accuracy': cross_val_score(random_forest_classifier, all_inputs, all_classes, cv=10),
                       'classifier': ['Random Forest'] * 10})
dt_df = pd.DataFrame({'accuracy': cross_val_score(decision_tree_classifier, all_inputs, all_classes, cv=10),
                      'classifier': ['Decision Tree'] * 10})
both_df = rf_df.append(dt_df)

sb.boxplot(x='classifier', y='accuracy', data=both_df)
sb.stripplot(x='classifier', y='accuracy', data=both_df, jitter=True, color='white')_____no_output_____

How about that? They both seem to perform about the same on this data set. This is probably because of the limitations of our data set: We have only 4 features to make the classification, and Random Forest classifiers excel when there's hundreds of possible features to look at. In other words, there wasn't much room for improvement with this data set._____no_output_____##Step 6: Reproducibility

[[ go back to the top ]](#Table-of-contents)

Ensuring that our work is reproducible is the last and — arguably — most important step in any analysis. **As a rule, we shouldn't place much weight on a discovery that can't be reproduced**. As such, if our analysis isn't reproducible, we might as well not have done it.

Notebooks like this one go a long way toward making our work reproducible. Since we documented every step as we moved along, we have a written record of what we did and why we did it — both in text and code.

Beyond recording what we did, we should also document what software and hardware we used to perform our analysis. This typically goes at the top of our notebooks so our readers know what tools to use.

[Sebastian Raschka](http://sebastianraschka.com/) created a handy [notebook tool](https://github.com/rasbt/watermark) for this:_____no_output_____

%install_ext https://raw.githubusercontent.com/rasbt/watermark/master/watermark.pyInstalled watermark.py. To use it, type:
  %load_ext watermark
%load_ext watermark_____no_output_____%watermark -a 'Randal S. Olson' -nmv --packages numpy,pandas,scikit-learn,matplotlib,SeabornRandal S. Olson Fri Aug 21 2015

CPython 3.4.3
IPython 3.2.1

numpy 1.9.2
pandas 0.16.2
scikit-learn 0.16.1
matplotlib 1.4.3
Seaborn 0.6.0

compiler   : GCC 4.2.1 (Apple Inc. build 5577)
system     : Darwin
release    : 14.5.0
machine    : x86_64
processor  : i386
CPU cores  : 8
interpreter: 64bit

Finally, let's extract the core of our work from Steps 1-5 and turn it into a single pipeline._____no_output_____

%matplotlib inline
import pandas as pd
import seaborn as sb
from sklearn.ensemble import RandomForestClassifier
from sklearn.cross_validation import train_test_split
from sklearn.cross_validation import cross_val_score

# We can jump directly to working with the clean data because we saved our cleaned data set
iris_data_clean = pd.read_csv('iris-data-clean.csv')

# Testing our data: Our analysis will stop here if any of these assertions are wrong

# We know that we should only have three classes
assert len(iris_data_clean['class'].unique()) == 3

# We know that sepal lengths for 'Iris-versicolor' should never be below 2.5 cm
assert iris_data_clean.loc[iris_data_clean['class'] == 'Iris-versicolor', 'sepal_length_cm'].min() >= 2.5

# We know that our data set should have no missing measurements
assert len(iris_data_clean.loc[(iris_data_clean['sepal_length_cm'].isnull()) |
                               (iris_data_clean['sepal_width_cm'].isnull()) |
                               (iris_data_clean['petal_length_cm'].isnull()) |
                               (iris_data_clean['petal_width_cm'].isnull())]) == 0

all_inputs = iris_data_clean[['sepal_length_cm', 'sepal_width_cm',
                             'petal_length_cm', 'petal_width_cm']].values

all_classes = iris_data_clean['class'].values

# This is the classifier that came out of Grid Search
random_forest_classifier = RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
                                max_depth=None, max_features=3, max_leaf_nodes=None,
                                min_samples_leaf=1, min_samples_split=2,
                                min_weight_fraction_leaf=0.0, n_estimators=5, n_jobs=1,
                                oob_score=False, random_state=None, verbose=0, warm_start=True)

# All that's left to do now is plot the cross-validation scores
rf_classifier_scores = cross_val_score(random_forest_classifier, all_inputs, all_classes, cv=10)
sb.boxplot(rf_classifier_scores)
sb.stripplot(rf_classifier_scores, jitter=True, color='white')

# ...and show some of the predictions from the classifier
(training_inputs,
 testing_inputs,
 training_classes,
 testing_classes) = train_test_split(all_inputs, all_classes, train_size=0.75)

random_forest_classifier.fit(training_inputs, training_classes)

for input_features, prediction, actual in zip(testing_inputs[:10],
                                              random_forest_classifier.predict(testing_inputs[:10]),
                                              testing_classes[:10]):
    print('{}\t-->\t{}\t(Actual: {})'.format(input_features, prediction, actual))/Users/libin/.virtualenvs/jupyter/lib/python3.6/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)

There we have it: We have a complete and reproducible Machine Learning pipeline to demo to our company's Head of Data. We've met the success criteria that we set from the beginning (>90% accuracy), and our pipeline is flexible enough to handle new inputs or flowers when that data set is ready. Not bad for our first week on the job!_____no_output_____##Conclusions

[[ go back to the top ]](#Table-of-contents)

I hope you found this example notebook useful for your own work and learned at least one new trick by reading through it.

If you've spotted any errors or would like to contribute to this notebook, please don't hestitate to get in touch. I can be reached in the following ways:

* [Email](http://www.randalolson.com/contact/) me

* [Tweet](https://twitter.com/randal_olson/) at me

* [Submit an issue](https://github.com/rhiever/Data-Analysis-and-Machine-Learning-Projects/issues) on GitHub

* Fork the [notebook repository](https://github.com/rhiever/Data-Analysis-and-Machine-Learning-Projects/), make the fix/addition yourself, then send over a pull request_____no_output_____##Further reading

[[ go back to the top ]](#Table-of-contents)

This notebook covers a broad variety of topics but skips over many of the specifics. If you're looking to dive deeper into a particular topic, here's some recommended reading.

**Data Science**: William Chen compiled a [list of free books](http://www.wzchen.com/data-science-books/) for newcomers to Data Science, ranging from the basics of R & Python to Machine Learning to interviews and advice from prominent data scientists.

**Machine Learning**: /r/MachineLearning has a useful [Wiki page](https://www.reddit.com/r/MachineLearning/wiki/index) containing links to online courses, books, data sets, etc. for Machine Learning. There's also a [curated list](https://github.com/josephmisiti/awesome-machine-learning) of Machine Learning frameworks, libraries, and software sorted by language.

**Unit testing**: Dive Into Python 3 has a [great walkthrough](http://www.diveintopython3.net/unit-testing.html) of unit testing in Python, how it works, and how it should be used

**pandas** has [several tutorials](http://pandas.pydata.org/pandas-docs/stable/tutorials.html) covering its myriad features.

**scikit-learn** has a [bunch of tutorials](http://scikit-learn.org/stable/tutorial/index.html) for those looking to learn Machine Learning in Python. Andreas Mueller's [scikit-learn workshop materials](https://github.com/amueller/scipy_2015_sklearn_tutorial) are top-notch and freely available.

**matplotlib** has many [books, videos, and tutorials](http://matplotlib.org/resources/index.html) to teach plotting in Python.

**Seaborn** has a [basic tutorial](http://stanford.edu/~mwaskom/software/seaborn/tutorial.html) covering most of the statistical plotting features._____no_output_____##Acknowledgements

[[ go back to the top ]](#Table-of-contents)

Many thanks to [Andreas Mueller](http://amueller.github.io/) for some of his [examples](https://github.com/amueller/scipy_2015_sklearn_tutorial) in the Machine Learning section. I drew inspiration from several of his excellent examples.

The photo of a flower with annotations of the petal and sepal was taken by [Eric Guinther](https://commons.wikimedia.org/wiki/File:Petal-sepal.jpg).

The photos of the various *Iris* flower types were taken by [Ken Walker](http://www.signa.org/index.pl?Display+Iris-setosa+2) and [Barry Glick](http://www.signa.org/index.pl?Display+Iris-virginica+3)._____no_output_____


# Notebook from caramirezs/TeachOpenCADD
Path: talktorials/10_binding_site_similarity/T10_off-targets.ipynb

# Talktorial 10

# Binding site similarity and off-target prediction

#### Developed in the CADD seminars 2017 and 2018, AG Volkamer, Charité/FU Berlin

Angelika Szengel, Marvis Sydow and Dominique Sydow_____no_output_____**Note**: Please run this notebook cell by cell. Running all cells in one is possible also, however, a few PyMol images might not turn out as intended._____no_output_____## Aim of this talktorial

In this talktorial, we use the structural similarity of whole proteins and binding sites to predict off-targets, i.e. proteins that are not intended targets of a drug, which may lead to unwanted side effects or enable desired alternate applications of a drug (drug repositioning).
We discuss the main steps for binding site comparison and implement a basic method, i.e. the geometrical variation between structures (the root mean square deviation of two structures).

## Learning goals

### Theory

* Off-target proteins
* Computational off-target prediction: binding site comparison
* Pairwise RMSD as simple measure for similarity
* Imatinib, a tyrosine kinase inhibitor

### Practical

* Load and visualize the ligand of interest (Imatinib/STI)
* Get all protein-STI complexes from the PDB
  * Query the PDB
  * Filter the PDB data set
  * Save the filtered PDB IDs
* Visualize the PDB structures
* Align the PDB structures (whole protein)
* Get pairwise RMSD (whole protein)
* Align the PDB structures (binding site)
* Get pairwise RMSD (binding site)

## References

Binding site comparison:

* Binding site comparison reviews:
([Curr. Comput. Aided Drug Des.  (2008), 4, 209-20](https://www.eurekaselect.com/67606/article/how-measure-similarity-between-protein-ligand-binding-sites))
and
([J. Med. Chem.  (2016), 9, 4121-51](https://pubs.acs.org/doi/10.1021/acs.jmedchem.6b00078))
* Documentation on PyMol `align` command
([PyMolWiki: `align`](https://pymolwiki.org/index.php/Align))
* Wikipedia article on root mean square deviation (RMSD)
([Wikipedia: RMSD](https://en.wikipedia.org/wiki/Root-mean-square_deviation_of_atomic_positions)) and structural superposition ([Wikipedia: structural superposition](https://en.wikipedia.org/wiki/Structural_alignment))
* Structural superposition ([Book chapter: Algorithms, Applications, and Challenges of Protein Structure Alignment in *Advances in Protein Chemistry and Structural Biology* (2014), **94**, 121-75](https://www.sciencedirect.com/science/article/pii/B9780128001684000056?via%3Dihub))

Imatinib:

* Review on Imatinib
([Nat. Rev. Clin. Oncol. (2016), 13, 431-46](https://www.nature.com/articles/nrclinonc.2016.41))
* Promiscuity of imatinib
([J. Biol. (2009), 8, 10.1186/jbiol134](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2689438/))
* ChEMBL information on Imatinib
([ChEMBL: Imatinib](https://www.ebi.ac.uk/chembl/compound/inspect/CHEMBL941))
* PDB information on Imatinib
([PDB: STI](https://www3.rcsb.org/ligand/STI))
* Side effects of Imatinib
([Drugs.com: Imatinib](https://www.drugs.com/cdi/imatinib.html))
* Side effects of Imatinib
 ([BMC Struct. Biol. (2009), 9, 10.1186/1472-6807-9-7](https://bmcstructbiol.biomedcentral.com/articles/10.1186/1472-6807-9-7))_____no_output_____## Theory

### Off-target proteins

An off-target can be any protein which interacts with a drug or (one of) its metabolite(s) without being the designated target protein.
The molecular reaction caused by the off-target can lead to unwanted side effects, ranging from a rather harmless to extremely harmful impact.
Off-targets mainly occur because on- and off-targets share similar structural motifs with each other in their binding site and therefore can bind similar ligands.

### Computational off-target prediction: binding site comparison

Computation-aided prediction of potential off-targets is aimed at minimizing the risk of developing potentially dangerous substances for medical treatment.
There are several algorithmic approaches to assess binding site similarity but they always consist of three main steps:

1. **Binding site encoding**: binding sites are encoded using different descriptor techniques and stored in a target database.
2. **Binding site comparison**: a query binding site is compared with the target database, using different similarity measures.
3. **Target ranking**: targets are ranked based on a suitable scoring approach.

For detailed information on different similarity measures and existing tools, we refer to two excellent reviews on binding site comparison ([Curr. Comput. Aided Drug Des.  (2008), 4, 209-20](https://www.eurekaselect.com/67606/article/how-measure-similarity-between-protein-ligand-binding-sites) and [J. Med. Chem.  (2016), 9, 4121-51](https://pubs.acs.org/doi/10.1021/acs.jmedchem.6b00078)).

 Figure 1: Main steps of binding site comparison methods (figure by Dominique Sydow).

### Pairwise RMSD as simple measure for similarity

A simple and straightforward method for scoring the similarity is to use the calculated root mean square deviation (RMSD), which is the square root of the mean of the square of the distances between the atoms of two aligned structures ([Wikipedia: RMSD](https://en.wikipedia.org/wiki/Root-mean-square_deviation_of_atomic_positions)).

In order to find the respective atoms that are compared between two structures, they need to be aligned first based on sequence-based or sequence-independent alignment algorithms ([Book chapter: Algorithms, Applications, and Challenges of Protein Structure Alignment in *Advances in Protein Chemistry and Structural Biology* (2014), **94**, 121-75](https://www.sciencedirect.com/science/article/pii/B9780128001684000056?via%3Dihub)).

### Imatinib, a tyrosine kinase inhibitor

Kinases transfer a phosphate group from ATP to proteins, and thereby regulate various cellular processes such as signal transduction, metabolism, and protein regulation.
If these kinases are constitutively active (due to genomic mutations), they can distort regulation processes and cause cancer.
An example for cancer treatment is Imatinib ([Nat. Rev. Clin. Oncol. (2016), 13, 431-46](https://www.nature.com/articles/nrclinonc.2016.41)), a small molecule tyrosine kinase inhibitor used to treat cancer, more specifically chronic myeloid leukaemia (CML) and gastrointestinal stromal tumour (GIST).

Imatinib was shown to be not entirely specific and to target tyrosine kinases other than its main target. This was used for drug repositioning, i.e. Imatinib was approved for alternate cancer types, ([J. Biol. (2009), 8, 10.1186/jbiol134](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2689438/)), however can also show unwanted side effects such as signs of an allergic reaction, infection, bleeding, or headache ([Drugs.com: Imatinib](https://www.drugs.com/cdi/imatinib.html))._____no_output_____## Practical

In the following, we will fetch and filter PDB structures that bind Imatinib. We will investigate the structure similarity of Imatinib-binding proteins (those with a solved protein structure).
The similarity measure used is a pairwise RMSD calculation (as a simple similarity measure), in order to show that this simple method can be used as an initial test for potential off-targets._____no_output_____

import os
import pprint
import pickle
import glob
import time

from rdkit import Chem
from rdkit.Chem.Draw import IPythonConsole
from rdkit.Chem import Draw, AllChem
from rdkit.Chem import PyMol
IPythonConsole.ipython_useSVG=True

import pandas as pd
import numpy as np
import random
import matplotlib.pyplot as plt
import seaborn as sns

from pypdb import *
from biopandas.pdb import PandasPdb_____no_output_____

### Load and visualize the ligand of interest (Imatinib/STI)

The SMILES format for Imatinib (common abbreviation: STI) can be retrieved from e.g. the ChEMBL database
([ChEMBL: Imatinib](https://www.ebi.ac.uk/chembl/compound/inspect/CHEMBL941))
or the PDB database by its common abbreviation STI
([PDB: STI](https://www3.rcsb.org/ligand/STI)).
We simply copy the string from the "Isomeric SMILES" entry of the Chemical Component Summary table, and load the ligand here by hand._____no_output_____

sti = Chem.MolFromSmiles('CN1CCN(Cc2ccc(cc2)C(=O)Nc2ccc(C)c(Nc3nccc(n3)-c3cccnc3)c2)CC1')
Draw.MolToImage(sti)_____no_output_____

In order to inspect the 3D structure of STI, we use the open source tool PyMol (see introduction in talktorial T8).
Before we can view STI in PyMol, we need to compute its 3D coordinates.

First, we add hydrogen atoms to the molecule, which are not always explicitly denoted in the SMILES format.
Second, we use the distance geometry to obtain initial coordinates for the molecule and optimize the structure of the molecule using the force field UFF (Universal Force Field)._____no_output_____

sti_mol = Chem.AddHs(sti)
sti_mol_____no_output_____AllChem.EmbedMolecule(sti_mol)
AllChem.UFFOptimizeMolecule(sti_mol)
sti_mol_____no_output_____

Now, we are ready to roll in PyMol!

We open PyMol from the terminal (here controlled from the Jupyter notebook)._____no_output_____

os.popen('pymol -R')_____no_output_____

We need to wait until PyMol is launched completely to link PyMol with the Jupyter notebook via `objPMV = PyMol.MolViewer()`._____no_output_____

# Error handling: wait until PyMol is loaded

nrTry = 0  # Number of current attempt
ttw = 10  # Time to wait in seconds

while nrTry < ttw:  # Try until PyMol is loaded and the object can be saved
    nrTry += 1
    try:
        objPMV = PyMol.MolViewer()  # Save the PyMol object
        break  # Stop the loop when PyMol is loaded
    except ConnectionRefusedError:  # Exception handling if PyMol is not loaded yet
        time.sleep(1)  # Wait...

if nrTry == ttw:  # After ttw trys: print error message
    print("Error: PyMol did not start correctly.\n" +
          "Try again and/or check if PyMol is installed completely.")_____no_output_____

We load STI to PyMol and display the PyMol frame here in the Jupyter notebook._____no_output_____

# Initialize PyMol (in case we already have objects loaded in PyMol)
objPMV.server.do("reinitialize")

# Load and show the molecule in PyMol
objPMV.ShowMol(sti_mol)
# Show as sticks
objPMV.server.do('cmd.show("sticks","molecule")')

# Set PyMol background to white
objPMV.server.do("bg_color white")
# Creates a ray-traced image of the current PyMol frame
objPMV.server.do("ray 900, 900")

objPMV.GetPNG(h=300)_____no_output_____

### Get all protein-STI complexes from the PDB

We can look up Imatinib/STI in open databases like the PDB and search for proteins which are reported targets. In the PDB, Imatinib is usually abbreviated by STI. We will search for both terms and merge the results in the following.

#### Query the PDB

First, we retrieve all proteins from the PDB that bind the ligand of interest (STI)._____no_output_____

search_dict = make_query('STI')  # Query PDB for proteins bound to the ligand STI
found_pbd_ids = do_search(search_dict)

print(found_pbd_ids)
print("\nNumber of structures connected with STI in the PDB: " + str(len(found_pbd_ids)))['1AVU', '1AVW', '1AVX', '1BA7', '1FPU', '1IEP', '1M52', '1OPJ', '1R8N', '1R8O', '1T45', '1T46', '1XBB', '2BEA', '2BEB', '2DRE', '2ESU', '2ET2', '2HYY', '2OIQ', '2PL0', '3EA7', '3EA8', '3FW1', '3GVU', '3HEC', '3I2A', '3I2X', '3K5V', '3M3V', '3MS9', '3MSS', '3OEZ', '3PYY', '3S8J', '3S8K', '4BKJ', '4CSV', '4GCN', '4GCO', '4H9W', '4HA2', '4R7I', '4TLP', '5FNW', '5FNX', '5FZU', '5FZY', '5FZZ', '5G00', '5MQT', '6HD4', '6HD6']

Number of structures connected with STI in the PDB: 53

Note that the query results can differ depending on the term used for the query ligand (here: Imatinib). _____no_output_____

search_dict2 = make_query('Imatinib')  # Query PDB for proteins bound to the ligand Imatinib
found_pbd_ids2 = do_search(search_dict2)

print(found_pbd_ids2)
print("\nNumber of structures connected with Imatinib in the PDB: " + str(len(found_pbd_ids2)))['1IEP', '1M52', '1OPJ', '1T46', '1XBB', '2F4J', '2GQG', '2HYY', '2OIQ', '2PL0', '2XYN', '3EL7', '3EL8', '3FW1', '3G0E', '3G0F', '3G6G', '3G6H', '3GVU', '3HEC', '3HEG', '3K5V', '3MS9', '3MSS', '3OEZ', '3OF0', '3PYY', '3QLF', '3QLG', '4BKJ', '4CSV', '4R7I', '5MQT', '6HD4', '6HD6']

Number of structures connected with Imatinib in the PDB: 35

We merge both query results and keep only unique entries._____no_output_____

pdb_ids = list(set(found_pbd_ids + found_pbd_ids))
print("Number of structures connected with STI/Imatinib in the PDB: " + str(len(pdb_ids)))Number of structures connected with STI/Imatinib in the PDB: 53

#### Filter the PDB data set

We retrieve meta information on the PDB structures using the `pypdb` function `get_entity_info`, in order to filter the data set based on the following criteria:

1. Filter by experimental method (`xray`).
2. Filter by resolution (equal or lower than 3 Å).
3. Retain only PDB structures with a single chain (for simplicity).
4. Retain only Imatinib-bound structures (e.g. some PDB structures are returned that are associated with Imatinib but not bound to it).
5. Retain only PDB IDs deposited before 2019 (data set resource at the time of the talktorial publication).

For more info on how to query the PDB see **talktorial 8**._____no_output_____

# Get meta information from the PDB
entity_info = []
for i in pdb_ids:
    entity_info.append(get_entity_info(i))
entity_info[0]_____no_output_____# Transform list to DataFrame
entity_info_pd = pd.DataFrame(entity_info)
a = [int(i.split(" ")[5]) for i in entity_info_pd["release_date"].tolist()]
entity_info_pd.head()_____no_output_____# 1. Filter by experimental method
entity_info_pd = entity_info_pd[entity_info_pd["Method"] == {'@name': 'xray'}]

# 2. Filter by resolution
entity_info_pd = entity_info_pd[entity_info_pd["resolution"].astype(float) <= 3.0]

# 3. Retain only structures with a single chain (for simplicity)
entity_info_pd = entity_info_pd[[type(i) == dict for i in entity_info_pd["Entity"]]]
entity_info_pd = entity_info_pd[[type(i["Chain"]) == dict for i in entity_info_pd["Entity"]]]

pdb_ids = entity_info_pd["structureId"].tolist()

print("Number of structures after filtering: " + str(len(pdb_ids)))Number of structures after filtering: 22

In the following, we will use a package called `BioPandas`, which provides useful functions to load molecular structures of biological macromolecules (from PDB and MOL2 files) in pandas DataFrames. We will use the `PandasPdb` object to facilitate our work with PDB files._____no_output_____

# 4. Retain only Imatinib-bound structures

def check_if_ligand_present(pdb_id, ligand_name):
    ppdb = PandasPdb().fetch_pdb(pdb_id)  # Fetch PDB (atom info, coordinates)
    return sum(ppdb.df["HETATM"]["residue_name"] == ligand_name) > 0  # Check for existence of STI entries

entity_info_pd = entity_info_pd[[check_if_ligand_present(pdb_id, "STI") for pdb_id in pdb_ids]]  # Apply function

pdb_ids = entity_info_pd["structureId"].tolist()

print("Number of structures after filtering: " + str(len(pdb_ids)))Number of structures after filtering: 8
# 5. Retain only PDB IDs deposited before 2019

entity_info_pd = entity_info_pd[[int(i.split()[5]) < 2019 for i in entity_info_pd["release_date"].tolist()]]

pdb_ids = entity_info_pd["structureId"].tolist()

print("Number of structures after filtering: " + str(len(pdb_ids)))Number of structures after filtering: 8
# 6. After manual visual inspection remove 3GVU (contains 2x STI: not feasable for automatic workflow below)
pdb_ids.remove("3GVU")_____no_output_____pdb_ids
#random.shuffle(pdb_ids)  # In case you would like to change the order of IDs_____no_output_____

#### Save the filtered PDB IDs

We save the PDB IDs of the filtered data set for further analysis (we will use PyMol python scripts later on that will process PDB IDs according to this file)._____no_output_____

pickle.dump(pdb_ids, open("../data/T10/pdb_ids.p", "wb"))_____no_output_____

### Visualize the PDB structures

First, we load all structures to PyMol for visual inspection of the 3D structure of the protein data set.

Besides the visualization here in this Jupyter notebook in form of fixed images of a PyMol frame, it is advised to also view and interact with the structures in 3D directly within the PyMol application, which should be opened and manipulated in parallel to this talktorial. _____no_output_____

# Reinitialize PyMol
objPMV.server.do("reinitialize")

# Load proteins files
for pdb_id in pdb_ids:
    cmd = "fetch " + pdb_id
    objPMV.server.do(cmd)

# Show proteins as cartoon (may be necessary depending on your pymol version)
objPMV.server.do('cmd.show("cartoon","all")')

# Hide objects
objPMV.server.do("hide polar_contacts")

# Set background to white
objPMV.server.do("bg_color white")
# Remove water and ions
objPMV.server.do("remove solvent")
# Center and zoom
objPMV.server.do("center all")
objPMV.server.do("zoom all")
objPMV.server.do("ray 400,400")

# Display PyMol frame in Jupyter notebook
objPMV.GetPNG(h=500)_____no_output_____

Though this image is beautifully colorful and curly, it is not informative yet. We align the structures to each other in the next step._____no_output_____### Align the PDB structures (whole protein)

PyMol offers different alignment methods suitable for different levels of sequence and structural similarity:

* The [`super`](https://pymolwiki.org/index.php/Super) command is preferred for proteins with *decent structural similarity* (sequence-independent).
* The [`align`](https://pymolwiki.org/index.php/Align) command is preferred for proteins with *decent sequence similarity* (sequence-dependent).
* The [`cealign`](https://pymolwiki.org/index.php/Cealign) command is very robust for proteins with *little to no sequence similarity* (twilight zone).

In this talktorial, we choose the `align` command to superimpose the structures (based on sequence) by minimizing their RMSD.

Note: This approach biases the analysis towards structures with similar sequences (the `align` command perform better for protein pairs with decent sequence similarity). For some comparisons with lower sequence similarity the `super` or `cealign` command could be a better choice. For an automated workflow (where we do not know the sequence or structural similarity of protein pairs) a solution could be to calculate the RMSD based on all three measures and retain the best for further analysis.

First, we show the alignment of all structures to the first structure in the list `pdb_ids`._____no_output_____

immobile_pdb_id = pdb_ids[0]
for mobile_pdb_id in pdb_ids[1:]:
    objPMV.server.do("align " + mobile_pdb_id + ", " + immobile_pdb_id)

objPMV.server.do("center all")
objPMV.server.do("zoom all")
objPMV.server.do("ray 400,400")

objPMV.GetPNG(h=500)_____no_output_____

One of the proteins, i.e. 3FW1, is poorly aligned in comparison to the other proteins. We hide this protein, in order to visually show the well aligned proteins._____no_output_____

objPMV.HideObject("3FW1")

objPMV.server.do("center all")
objPMV.server.do("zoom all")
objPMV.server.do("ray 400,400")

objPMV.GetPNG(h=500)_____no_output_____

The structural alignment for many e.g. helices is high, whereas lower or poor for others._____no_output_____### Get pairwise RMSD (whole protein)

Since we could not find a function to return the RMSD values from PyMol in this Jupyter notebook (which *is* possible within PyMol and within a PyMol python script), we call here a PyMol python script: the superposition and final RMSD refinement is performed for all protein pairs. The resulting RMSD values are saved for further analysis.

First, we take a look at the python script for PyMol, which we will use for this task._____no_output_____

f = open("./pymol_scripts/pymol_align_proteins.py")
file_content = f.readlines()
for i in file_content:
    print(i, end="")'''
This script aligns all pairs of input proteins.
'''

import pymol
import pickle
import pandas as pd
import sys
import numpy as np

# Launch PyMol
pymol.finish_launching()

# Load PDB IDs
pdb_ids = pickle.load(open("../data/T10/pdb_ids.p", "rb"))

# Create DataFrame to store align results
align_df = pd.DataFrame(index=pdb_ids, columns=pdb_ids)

def align_structures(m1, m2, align_df):
	# Fetch PDB structures
	pymol.cmd.fetch(m1, "i")
	pymol.cmd.fetch(m2, "m")

	# Align proteins
	aln = pymol.cmd.align(mobile="m", target="i", quiet=1, object="{}_{}".format(m1, m2))

	# Save align results to DataFrame
	align_df.loc[m1, m2] = aln

	# Save sequence alignment to file
	pymol.cmd.save(filename="../data/T10/alignment/alignment_proteins_" + "{}_{}".format(m1, m2) + ".aln", selection="{}_{}".format(m1, m2))

	# Reinitialize PyMol for next structure pair to be loaded
	pymol.cmd.reinitialize()

	return align_df

# Iterate over all structure pairs
for n, immobile in enumerate(pdb_ids):
	for mobile in pdb_ids[n:]:
		align_df = align_structures(mobile, immobile, align_df)
		align_df = align_structures(immobile, mobile, align_df)

# Quit PyMol
pymol.cmd.quit()

# Save results to file
pickle.dump(align_df, open("../data/T10/align_df_proteins.p", "wb"))
os.popen("python ./pymol_scripts/pymol_align_proteins.py")_____no_output_____

This PyMol python script saves all pairwise RMSD results to a file, which we load now to the Jupyter notebook._____no_output_____

align_df_proteins = pickle.load(open("../data/T10/align_df_proteins.p", "rb"))_____no_output_____

`align_df_proteins` is a DataFrame and contains for each pairwise comparison the return values of the `align` command in PyMol, i.e. a tuple with 7 items:

1. RMSD after refinement
2. Number of aligned atoms after refinement
3. Number of refinement cycles
4. RMSD before refinement
5. Number of aligned atoms before refinement
6. Raw alignment score
7. Number of residues aligned

We familiarize ourselves with this data structure for an example protein pair:_____no_output_____

print("Structures in the data set:")
cols = align_df_proteins.columns
rows = align_df_proteins.index
print(cols)
print(rows)

example_col = cols[0]
example_row = rows[2]
print("\nExample align return values for the {}-{} pair: ".format(example_col, example_row))
print(align_df_proteins.loc[example_col, example_row])

print("\nRMSD value (in Angström) after refinement for the {}-{} pair: ".format(example_col, example_row))
print(align_df_proteins.loc[example_col, example_row][0])Structures in the data set:
Index(['4R7I', '3HEC', '2PL0', '4CSV', '1T46', '3FW1', '1XBB'], dtype='object')
Index(['4R7I', '3HEC', '2PL0', '4CSV', '1T46', '3FW1', '1XBB'], dtype='object')

Example align return values for the 4R7I-2PL0 pair:
(1.1611641645431519, 1283, 5, 4.181900501251221, 1734, 514.0, 267)

RMSD value (in Angström) after refinement for the 4R7I-2PL0 pair:
1.1611641645431519

Now, we extract the RMSD after refinement (or any other position of the `align` return value) for all pairs in form of a pandas DataFrame for further analysis.

Therefore, we define a function that takes as input the `align` function return values for all pairs `pymol_align_results` and the position of the return value of interest `position`._____no_output_____

def extract_align_info(align_df, position):
    if position in [0, 3, 5]:
        return align_df.applymap(lambda x: np.float(x[position]))
    if position in [1, 2, 4, 6]:
        return align_df.applymap(lambda x: np.int(x[position]))
    else:
        print("Position not available.")_____no_output_____

For example, we can check the number of aligned residues per protein pair (last position of the `align` return value)._____no_output_____

extract_align_info(align_df_proteins, 6)_____no_output_____

Here, we can see that for most protein pairs a large proportion of their residues could be aligned (EGFR sequence lengths range here from about 270 to 350). However, 3FW1 shows only low sequence alignment._____no_output_____FYI: You can check the sequence alignment of all pairs here: `../data/T10/alignment/`. An example is shown below:_____no_output_____

example_alignment_path = glob.glob("../data/T10/alignment/*")[3]
f = open(example_alignment_path)
file_content = f.readlines()
for i in file_content:
    print(i, end="")CLUSTAL

i            MKKGHHHHHHGQKPKYQVRWKIIESYEGNSYTFIDPTQLPYNEKWEFPRNNLQFGKTLGAGAF
m            MKKGHHHHHHGQKPKYQVRWKIIESYEGNSYTFIDPTQLPYNEKWEFPRNNLQFGKTLGAGAF
                                                 ***************************

i            GKVVEATAFGLGKEDAVLKVAVKMLKSTAHADEKEALMSELKIMSHLGQHENIVNLLGACTHG
m            GKVVEATAFGLGKEDAVLKVAVKMLKSTAHADEKEALMSELKIMSHLGQHENIVNLLGACTHG
             ***************************************************************

i            GPVLVITEYCTYGDLLNFLRRKAEAMLGPSLSPGQDPEGLDKEDGRPLELRDLLHFSSQVAQG
m            GPVLVITEYCTYGDLLNFLRRKAEAMLGPSLSPGQDPEGLDKEDGRPLELRDLLHFSSQVAQG
             ********************************            *******************

i            MAFLASKNCIHRDVAARNVLLTNGHVAKIGDFGLARDIMNDSNYIVKGNARLPVKWMAPESIF
m            MAFLASKNCIHRDVAARNVLLTNGHVAKIGDFGLARDIMNDSNYIVKGNARLPVKWMAPESIF
             ***************************************************************

i            DSVYTVQSDVWSYGILLWEIFSLGLNPYPGILVNSKFYKLVKDGYQMAQPAFAPKNIYSIMQA
m            DSVYTVQSDVWSYGILLWEIFSLGLNPYPGILVNSKFYKLVKDGYQMAQPAFAPKNIYSIMQA
             ***************************************************************

i            CWALEPTHRPTFQQITSFLQEQAQEDRR
m            CWALEPTHRPTFQQITSFLQEQAQEDRR
             ***********************

In the next step, we generate a DataFrame for the RMSD values after refinement (first position of the `align` return value)._____no_output_____

rmsd = extract_align_info(align_df_proteins, 0)
rmsd_____no_output_____

We visualize the results of this RMSD refinement as heatmap._____no_output_____

sns.heatmap(rmsd, linewidths=1, annot=True, cbar_kws={"label": "RMSD ($\AA$)"}, cmap="Blues")
plt.show()_____no_output_____

We cluster the heatmap in order to see protein similarity based on the RMSD refinement._____no_output_____

def plot_clustermap(rmsd, title):
    g = sns.clustermap(rmsd, linewidths=1, annot=True, cbar_kws={"label": "RMSD ($\AA$)"}, cmap="Blues")
    plt.setp(g.ax_heatmap.get_xticklabels(), rotation=0)
    plt.setp(g.ax_heatmap.get_yticklabels(), rotation=0)
    sns.set(font_scale=1.5)

    # Save plot - use bbox_inches to include text boxes:
    # https://stackoverflow.com/questions/44642082/text-or-legend-cut-from-matplotlib-figure-on-savefig?rq=1
    plt.savefig("../data/T10/bsc_{}.png".format(title), dpi=300, bbox_inches="tight", transparent=True)

    plt.show()_____no_output_____plot_clustermap(rmsd, "protein")/home/dominique/anaconda2/envs/py36/lib/python3.6/site-packages/seaborn/matrix.py:603: ClusterWarning: scipy.cluster: The symmetric non-negative hollow observation matrix looks suspiciously like an uncondensed distance matrix
  metric=self.metric)

The RMSD comparison shows that one protein differs from the other proteins, i.e. 3FW1 (as already discussed based on the visual 3D inspection of the alignment and on the number of aligned residues).

Proteins are classified by the chemical reactions they catalyze with so called EC (Enzyme Commission) numbers, which we will use here to check the enzymatic groups the proteins belong to._____no_output_____

# Get EC numbers for PDB IDs from PDB
pdb_all_info = [get_all_info(pdb_id) for pdb_id in pdb_ids]
ec_numbers = [i["polymer"]["enzClass"]["@ec"] for i in pdb_all_info]
target_set = {"pdb_id": pdb_ids,
              "ec_number": ec_numbers}
target_set = pd.DataFrame(target_set)
target_set_____no_output_____

We can see that 3FW1, the human quinone reductase 2 (NQO2), belongs to EC class 1, i.e. oxidoreductases, whereas the other proteins belong to class 2.7, i.e. phosphorus transferases, which contain the tyrosine kinases (EC 2.7.10.2), the designated targets for Imatinib. 3FW1 is a reported off-target "with potential implications for drug design and treatment of chronic myelogenous leukemia in patients" ([BMC Struct. Biol. (2009), 9, 10.1186/1472-6807-9-7](https://bmcstructbiol.biomedcentral.com/articles/10.1186/1472-6807-9-7)).

_____no_output_____### Align PDB structures (binding sites)_____no_output_____So far we have used the whole protein structure for the alignment and RMSD refinement. However, the ligand binds only at the protein binding site and therefore the similarity of binding sites rather than of whole protein structures is a more putative basis for off-target prediction.

We define a binding site of a protein by selecting all residues that are within 10 Å of any ligand atom. These binding site residues are used for alignment and only their Cɑ atoms (protein backbone) are used for the RMSD refinement. Here, we show the alignment of all structures to the first structure in the list `pdb_ids`._____no_output_____

# Reinitialize PyMol
objPMV.server.do("reinitialize")

# Load proteins files
for pdb_id in pdb_ids:
    cmd = "fetch " + pdb_id
    objPMV.server.do(cmd)

# Show proteins as cartoon (may be necessary depending on your pymol version)
#objPMV.server.do('cmd.show("cartoon","all")')

# Hide objects
objPMV.server.do("hide polar_contacts")

# Set background to white
objPMV.server.do("bg_color white")
# Remove water and ions
objPMV.server.do("remove solvent")

# Align binding sites
immobile_pdb_id = pdb_ids[0]
for mobile_pdb_id in pdb_ids[1:]:
    # Select atoms within a certain radius of any atom of STI and extend selection to full residues
    objPMV.server.do("select mobile_bs, byres " + mobile_pdb_id + " within 10 of (" + mobile_pdb_id + " and resn STI)")
    objPMV.server.do("select immobile_bs, byres " + immobile_pdb_id + " within 10 of (" + immobile_pdb_id + " and resn STI)")
    # Perform alignment
    objPMV.server.do("align mobile_bs, immobile_bs")

# Center and zoom
objPMV.server.do("center all")
objPMV.server.do("zoom all")
objPMV.server.do("ray 400,400")

# Display PyMol frame in Jupyter notebook
objPMV.GetPNG(h=500)_____no_output_____

### Get pairwise RMDS (binding sites)_____no_output_____As shown before for the alignment and RMSD refinement of the whole protein structure, we here run a PyMol python script that calculates the alignment and RMSD for all protein binding site pairs as describe above.

The PyMol commands used are explained within the PyMol python script._____no_output_____

f = open("./pymol_scripts/pymol_align_bindingsites.py")
file_content = f.readlines()
for i in file_content:
    print(i, end="")'''
This script aligns all pairs of input proteins (binding sites).
'''

import pymol
import pickle
import pandas as pd
import sys
import numpy as np

# Launch PyMol
pymol.finish_launching()

# Get the parameter (radius from ligand)
radius = sys.argv[1]

# Load PDB IDs
pdb_ids = pickle.load(open("../data/T10/pdb_ids.p", "rb"))

# Create DataFrame to store align results
align_df = pd.DataFrame(index=pdb_ids, columns=pdb_ids)

def align_structures(m1, m2, align_df):
	# Fetch PDB structures
	pymol.cmd.fetch(m1, "i")
	pymol.cmd.fetch(m2, "m")

	# Select binding site
	pymol.cmd.select("i_bs", "byres i within " + radius + " of (i and resn STI)")
	pymol.cmd.select("m_bs", "byres m within " + radius + " of (m and resn STI)")

	# Align binding sites
	aln = pymol.cmd.align(mobile="m_bs", target="i_bs", quiet=1)

	# Save align results to DataFrame
	align_df.loc[m1, m2] = aln

	# Reinitialize PyMol for next structure pair to be loaded
	pymol.cmd.reinitialize()

	return align_df

# Iterate over all structure pairs
for n, immobile in enumerate(pdb_ids):
	for mobile in pdb_ids[n:]:
		align_df = align_structures(mobile, immobile, align_df)
		align_df = align_structures(immobile, mobile, align_df)

# Quit PyMol
pymol.cmd.quit()

# Save results to file
pickle.dump(align_df, open("../data/T10/align_df_bindingsites.p", "wb"))

We run the PyMol python script via the terminal (ligand-protein radius as input for binding site atom definition)._____no_output_____

# Perform binding site comparison using the align function in PyMol
os.popen("python ./pymol_scripts/pymol_align_bindingsites.py 10")_____no_output_____

We load the `align` DataFrame for binding site comparisons._____no_output_____

align_df_bindingsites = pickle.load(open("../data/T10/align_df_bindingsites.p", "rb"))_____no_output_____

We extract the RMSD values from that DataFrame._____no_output_____

rmsd_bs = extract_align_info(align_df_bindingsites, 0)_____no_output_____extract_align_info(align_df_bindingsites, 6)_____no_output_____

We show the clustered heatmap for the RMSD results._____no_output_____

# Show the pairwise RMSD values as clustered heatmap
plot_clustermap(rmsd_bs, "bs")/home/dominique/anaconda2/envs/py36/lib/python3.6/site-packages/seaborn/matrix.py:603: ClusterWarning: scipy.cluster: The symmetric non-negative hollow observation matrix looks suspiciously like an uncondensed distance matrix
  metric=self.metric)

RMSD values of aligned binding sites shows a dissimilarity of 3FW1 (EC number 1.10.5.1) within the dataset (EC number 2.7) - visual inspection in PyMol shows that STI is binding to the surface of the protein. The pairs 1XBB-4CVS and 1XBB-3HEC also show dissimilarities, whereas the rest of the dataset shows low RMSD values.

RMSD values as calculated here are dependend on the residue selection (binding site definition) and the quality of the a priori sequence alignment._____no_output_____

# Clean up directory (remove PDB downloads and PyMol alignment files)
os.popen("rm ./*.cif")
os.popen("rm ../data/T10/alignment/*.aln")_____no_output_____

## Discussion

In this talktorial, we have used sequence alignment and subsequent RMSD refinement of whole protein structures and binding sites to assess the similarity and dissimilarity of a set of Imatinib-binding proteins.
However, off-target prediction for Imatinib requires to compare the binding site of an intended target of Imatinib (a tyrosine kinase) with a large database of resolved structures (PDB).
Since this results in the comparison of sequences also with low similarity, more sophisticated methods should be invoked
that use a sequence-independent alignment algorithm and that include the physico-chemical properties of the binding site in order to enable a more sophisticated search.

## Quiz

1. Explain the terms on- and off-targets of a drug.
2. Explain why binding site similarity can be used to find off-targets to a query target.
3. Discuss how useful the RMSD value of (i) whole proteins and (ii) protein binding sites is for off-target prediction.
4. Think of alternate approaches for binding site information (how to encode a binding site for binding site comparison?)._____no_output_____


# Notebook from erdc/pynirom
Path: examples/PODNODE_burgers2d.ipynb

### Notebook for constructiong POD-NODE NIROM approximation for a 2D Burgers' example

A collection of high-fidelity snapshots are generated that sufficiently capture the time-dynamics of the simulation. POD is adopted to define a reduced basis space for the high-fidelity snaphosts. The evolution of the time dynamics in the POD-latent space is modeled using Neural ODEs (NODE).

Firedrake is used as the high-fidelity model for solving the 2D viscous Burgers equation to generate snapshots
_____no_output_____

<code>
### Loading modules
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import time
from sklearn.preprocessing import MinMaxScaler, StandardScaler
import scipy
import os
import gc
import argparse
import platform
print("Python "+str(platform.python_version()))
import importlib
from importlib import reload as reload

import tensorflow as tf
print("Tensorflow "+ str(tf.__version__))
if tf.__version__ == '1.15.0':
    tf.compat.v1.enable_eager_execution()
elif tf.__version__.split('.')[0] == 2:
    print("Setting Keras backend datatype")
    tf.keras.backend.set_floatx('float64')

from tfdiffeq import odeint,odeint_adjoint
from tfdiffeq.adjoint import odeint as adjoint_odeint
tf.keras.backend.set_floatx('float64')
print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
np.random.seed(0)

basedir   = os.getcwd()
srcdir    = os.path.join(basedir,'../pynirom/')
workdir   = os.path.join(basedir,'../examples/')
datadir   = os.path.join(basedir,'../data/')
figdir    = os.path.join(basedir,'../figures/podnode')
nodedir   = os.path.join(basedir,'../data/')
savedir = nodedir+'burgers2d/current'

if not os.path.exists(savedir):
    os.makedirs(savedir)
if not os.path.exists(figdir):
    os.makedirs(figdir)

import pynirom
from pynirom.pod import pod_utils as pod
from pynirom.rbf import plotting as rpu
from pynirom.utils import data_utils as du
from pynirom.node import main as nd
from pynirom.node import plotting as pu
from pynirom.node import node as node

_____no_output_____device = 'cpu:0'           # select gpu:# or cpu:#
purpose= 'train'           #'train' to train a new model,
                           # 'retrain' to start training from an existing model, and
                           # 'eval' to load a pre-trained model for evaluation
pre_trained_dir = savedir+'/model_weights/' #If 'eval' specify path for pretrained model
stacking = True            #If True, Specify new stacking order of latent space vector
stack_order = 'v_x,v_y'    #If stacking = True, specify the stacking order of the latent space vector
scale_time = True          #Scale time or not (Normalize)
scale_states = False       #Scale states or not
scaling_method = 'centered' #Scaling method: 'centered', 'minmax' or 'maxabs'
augmented,aug_dims = (False,5) #Augmented or not and size of augmentation
N_layers = int(1)          #Only four layers supported as of now.
N_neurons = int(40)        #Number of neurons per layer
act_f = 'tanh'             #Activation Function ('linear', 'tanh', 'sigmoid',...), default='tanh'
learning_rate_decay = True #Use decaying learning rate or not
initial_learning_rate = float(0.01) #If 'learning_rate_decay = False' then this is the fixed learning rate
decay_steps = int(401)     #Number of steps for learning rate decay
decay_rate = float(0.9)    #Rate of learning rate decay
staircase_opt = True       #True for staircase decay and False for exponential
optimizer = 'RMSprop'      #See pynirom.node.node.set_optimizer() for options
use_adjoint = False        #Use adjoint method or not
solver = 'euler'           #Specify ODE solver. See tfdiffeq README for available options
use_minibatch, batch_size = (False,256) #Use minibatch or not and batch size
epochs = int(2000)         #Number of epochs of training

print("\n***** Runtime parameters: ******\n")
print(f'Mode = {purpose}, Scaling = {scale_states}, Augmenting = {augmented}, Adjoint = {use_adjoint}')
print(f'Solver = {solver}, Optimizer = {optimizer}, Stacking order = {stack_order}, Epochs = {epochs}')
print(f'# Layers = {N_layers}, # Neurons per layer = {N_neurons}, Activation fn = {act_f}')
if use_minibatch:
    print(f'Use minibatch = {use_minibatch}, Batch size = {batch_size}')
if learning_rate_decay:
    print(f'Init LR = {initial_learning_rate}, # LR decay steps = {decay_steps}, LR decay rate = {decay_rate}')
else:
    print(f'Fixed LR = {initial_learning_rate}')
print('**********************************\n')

_____no_output_____### ------ Import Snapshot data -------------------
data = np.load(datadir + 'Burgers2D_Re1000_Nn3721_Nt1500.npz')

print('HFM data has {0} snapshots of dimension {1} for p,u and v, spanning times [{2}, {3}]'.format(
                    data['time'].shape[0],data['v_x'].shape[0],
                    data['time'][0], data['time'][-1]))

## ------- Prepare training snapshots ----------------
print('\n-------Prepare training and testing data---------')
soln_names = ['v_x', 'v_y']
nodes = data['nodes'];
triangles = data['elems'];
snap_start = 10
T_end = 1500.0        ### 5 seconds
snap_incr = 4

snap_train, times_train = du.prepare_data(data, soln_names, start_skip=snap_start, T_end=T_end, incr=snap_incr)
print('Using {0} training snapshots for time interval [{1},{2}] seconds'.format(times_train.shape[0],
                                        times_train[0], times_train[-1]))

## ------- Prepare testing snapshots ----------------
pred_incr = snap_incr - 3
snap_pred_true, times_predict = du.prepare_data(data, soln_names, start_skip=snap_start, incr=pred_incr)
print('Using {0} testing snapshots for time interval [{1},{2}] seconds'.format(times_predict.shape[0],
                                        times_predict[0], times_predict[-1]))

del data
gc.collect()_____no_output_____### ------ Compute the POD basis using the training snapshots------------------
trunc_lvl = 0.99
snap_norm, snap_mean, U, D, W = pod.compute_pod_multicomponent(snap_train)
nw, U_r = pod.compute_trunc_basis(D, U, eng_cap = trunc_lvl)

### ------ Compute the POD coefficients for training snapshots------------------
Z_train = pod.project_onto_basis(snap_train, U_r, snap_mean)

### ------ Compute the POD coefficients for the truth snapshots on the prediction interval------------------
Z_pred_true = pod.project_onto_basis(snap_pred_true, U_r, snap_mean)

npod_total = 0
for key in soln_names:
    npod_total+=nw[key]

    _____no_output_____rpu.plot_sing_val(D)
_____no_output_____### ---- Setup NODE input data

NODE = nd.NODEBase(device=device)
true_state_array, true_pred_state_array, init_state, state_len, dt_train, dt_predict = \
        NODE.prepare_input_data(Z_train, nw, times_train, stack_order, times_predict, Z_pred_true)

print("Training NODE using %d modes for %d time steps with %.3f <= t <= %.3f and dt = %.3f"%(state_len,
                             true_state_array.shape[0], times_train[0],
                             times_train[-1], dt_train))
print("Predicting NODE solutions using %d modes for %d time steps with %.3f <= t <= %.3f and dt = %.3f"%(
                            state_len, true_pred_state_array.shape[0], times_predict[0],
                            times_predict[-1], dt_predict))
_____no_output_____### Preprocess training data (scale time and/or states, augment states if using ANODE)
### Set up learning rate scheduler and optimizer for training of the NODE model

true_state_tensor, times_tensor, init_tensor, learn_rate, optim = \
                NODE.preprocess_data(scale_states=scale_states, scale_time=scale_time, augmented=augmented,
                        lr_decay=learning_rate_decay, init_lr=initial_learning_rate, opt=optimizer,
                        scaling_method=scaling_method, aug_dim=aug_dims,
                        decay_steps=decay_steps, decay_rate=decay_rate, staircase=staircase_opt, )

_____no_output_____%%time

### ---- Model Training ------
train_loss_results, train_lr, saved_ep = \
NODE.train_model(true_state_tensor, times_tensor, init_tensor, epochs, savedir, \
                 solver=solver, purpose=purpose, adjoint=use_adjoint, \
                 minibatch=use_minibatch, pre_trained_dir = pre_trained_dir)
_____no_output_____## --- Generate NODE predictions ---

predicted_states, times_predict = NODE.predict_time(times_predict, init_tensor, pre_trained_dir,)

## ---- Compute Mean Square Error of predictions
Z_pred = {}
ctr= 0
for key in stack_order.split(','):
    Z_pred[key] = np.array(predicted_states)[:,ctr:ctr+nw[key]].T
    ctr += nw[key]
snap_pred = pod.reconstruct_from_rom(Z_pred, U_r, snap_mean, nw)

error_vx = np.mean(np.square(snap_pred['v_x']-snap_pred_true['v_x']))
error_vy = np.mean(np.square(snap_pred['v_y']-snap_pred_true['v_y']))

print("\n---- Mean Square Error of NODE predictions ----\n")
print('Vx MSE: ' + str(error_vx))
print('Vy MSE: ' + str(error_vy))_____no_output_____def set_label(key):
    if key == 'v_x':
        return 'u'
    elif key == 'v_y':
        return 'v'
_____no_output_____## ----- Visualize true and predicted POD coefficients -------

comp = 0
Nc = len(soln_names)
# Visualization fluff here
fig, ax = plt.subplots(nrows=Nc,ncols=3,figsize=(15,10))
mnum = comp
for iy, key in enumerate(soln_names):
    for ix in range(3):
        tt = ax[iy,ix].plot(times_predict[:],true_pred_state_array[:,mnum+ix],label='True',marker='o',
                        markevery=100, markersize=8)
        # Visualization of modal evolution using NODE
        ln, = ax[iy,ix].plot(times_predict[:],predicted_states[:,mnum+ix],label='PODNODE',color='orange',marker='D',
                        markevery=150, markersize=8)

        ax[iy,ix].set_xlabel('Time (seconds)', fontsize=18)
        sv = set_label(key)+', mode '+str(comp+ix)
        ax[iy,ix].set_ylabel(sv,fontsize=18)
        ax[iy,ix].legend(fontsize=14)
    mnum = mnum + nw[key]
fig.suptitle("POD coefficients of the HFM and PODNODE solutions for 2D Burgers' equations", fontsize=20)
fig.tight_layout(rect=[0, 0.03, 1, 0.99])

# os.chdir(figdir)
# plt.savefig('burgers_podnode_coeff_tskip%d_oskip%d_epochs%d'%(snap_incr,pred_incr,epochs),bbox_inches='tight')_____no_output_____fig = plt.figure(figsize=(6,5))
plt.semilogy(range(epochs), train_loss_results, label='Training')
if use_minibatch:
    plt.semilogy(range(epochs), val_loss_results,label='Validation')
plt.legend(fontsize=16); plt.xticks(fontsize=14); plt.yticks(fontsize=14)
plt.ylabel('$\log$ | MSE Loss |',fontsize=16); plt.xlabel('epochs',fontsize=16)

# os.chdir(figdir)
# plt.savefig('burgers_podnode_train_loss_epochs%d.pdf'%(epochs),bbox_inches='tight')_____no_output_____## Visualize PODNODE and HFM solutions

ky = soln_names[0]
itime=767
vmin = np.minimum(snap_pred[ky][:,itime].min(),snap_pred_true[ky][:,itime].min())
vmax = np.maximum(snap_pred[ky][:,itime].max(),snap_pred_true[ky][:,itime].max())

fig,axs = plt.subplots(1, 2, figsize=(10,6), constrained_layout=True)
cf = axs[0].tripcolor(nodes[:, 0], nodes[:, 1],triangles, snap_pred[ky][:,itime],
                   cmap=plt.cm.jet, shading='gouraud', vmin=vmin, vmax=vmax)
axs[0].axis('equal'); axs[0].axis('off')
axs[0].set_title('PODNODE $\mathbf{%s}$ solution\n at t=%.2f seconds'%(set_label(ky),times_predict[itime]))

cf = axs[1].tripcolor(nodes[:, 0], nodes[:, 1],triangles, snap_pred_true[ky][:,itime],
                   cmap=plt.cm.jet, shading='gouraud', vmin=vmin, vmax=vmax)
axs[1].axis('equal'); axs[1].axis('off')
axs[1].set_title('HFM $\mathbf{%s}$ solution\n at t=%.2f seconds'%(set_label(ky),times_predict[itime]))
fig.colorbar(cf, ax=axs[:], shrink=0.7, location='bottom', pad=0.04, aspect=30)

# os.chdir(figdir)
# plt.savefig('burgers_podnode_%s_t%.2f_tskip%d_oskip%d.pdf'%(ky,times_predict[itime],snap_incr,pred_incr),bbox_inches='tight')
_____no_output_____### Visualize solution error
fig = plt.figure(figsize=(8,6))
plt.tripcolor(nodes[:, 0], nodes[:, 1],triangles,
                    snap_pred[ky][:,itime]-snap_pred_true[ky][:,itime],
                    cmap=plt.cm.jet, shading='flat')
plt.axis('equal'); plt.axis('off')
# rpu.viz_err(snap_pred[ky][:,itime],snap_pred_true[ky][:,itime],nodes,triangles)

plt.title('PODNODE $\mathbf{%s}$ error\n at t=%.2f'%(ky,times_predict[itime]))
plt.colorbar(shrink=0.9, aspect=20, pad=0.05)

# os.chdir(figdir)
# plt.savefig('burgers_podnode_%s_err_t%.2f_tskip%d_oskip%d.pdf'%(ky,times_predict[itime],snap_incr,pred_incr),bbox_inches='tight')_____no_output_____## ---- Compute spatial RMS/Relative error
reload(nd)
reload(pu)
metric = 'rel'
err = NODE.compute_error(snap_pred_true, snap_pred, soln_names, metric=metric)

vstring = {}
for key in soln_names:
    vstring[key] = set_label(key)_____no_output_____ky1 = soln_names[0]; ky2 = soln_names[1];
t_unit = 'seconds'
tseries = times_predict
freq = tseries.size//20

fig = plt.figure(figsize=(16,4))
ax1 = fig.add_subplot(1, 2, 1)
ax1.plot(tseries[:], err[ky1][:], 'r-s', markersize=8,
                label='$\mathbf{%s}$'%(vstring[ky1]),lw=2, markevery=freq)
ymax_ax1 = err[ky1][:].max()
ax1.set_xlabel('Time (%s)'%t_unit);lg=plt.legend(ncol=2, fancybox=True,)

ax2 = fig.add_subplot(1, 2, 2)
ax2.plot(tseries[:], err[ky2][:], 'b-o', markersize=8,
                label='$\mathbf{%s}$'%(vstring[ky2]), lw=2, markevery=freq)
ymax_ax2 = err[ky2][:].max()
ax2.set_xlabel('Time (%s)'%t_unit);lg=plt.legend(ncol=2, fancybox=True,)
if metric == 'rms':
    fig.suptitle('Spatial RMS errors of PODNODE NIROM solutions for 2D Burgers example',fontsize=18)
elif metric == 'rel':
    fig.suptitle('Relative errors of PODNODE NIROM solutions for 2D Burgers example',fontsize=18)_____no_output_____#### ----- Save predicted solutions -------
save_nirom_solutions = False

if save_nirom_solutions:
    os.chdir(nodedir)
    print("Saving results in %s"%(os.getcwd()))

    np.savez_compressed('burgers2d_online_node',
                        v_x=snap_pred['v_x'], v_y=snap_pred['v_y'],
                        time=times_predict, loss=train_loss_results)
_____no_output_____
</code>