# Notebook from test-user-18/deepchem
Path: examples/notebooks/Exploring_Quantum_Chemistry_with_GDB1k.ipynb

# Exploring Quantum Chemistry with GDB1k_____no_output_____Most of the tutorials we've walked you through so far have focused on applications to the drug discovery realm, but DeepChem's tool suite works for molecular design problems generally. In this tutorial, we're going to walk through an example of how to train a simple molecular machine learning for the task of predicting the atomization energy of a molecule. (Remember that the atomization energy is the energy required to form 1 mol of gaseous atoms from 1 mol of the molecule in its standard state under standard conditions).

To get started, we'll do a few basic imports._____no_output_____

import os
import unittest
import numpy as np
import deepchem as dc
import numpy.random
from deepchem.utils.evaluate import Evaluator
from sklearn.ensemble import RandomForestRegressor
from sklearn.kernel_ridge import KernelRidge_____no_output_____

The ntext step we want to do is load our dataset. We're using a small dataset we've prepared that's pulled out of the larger GDB benchmarks. The dataset contains the atomization energies for 1K small molecules._____no_output_____

tasks = ["atomization_energy"]
dataset_file = "../../datasets/gdb1k.sdf"
smiles_field = "smiles"
mol_field = "mol"_____no_output_____

We now need a way to transform molecules that is useful for prediction of atomization energy. This representation draws on foundational work [1] that represents a molecule's 3D electrostatic structure as a 2D matrix $C$ of distances scaled by charges, where the $ij$-th element is represented by the following charge structure.

$C_{ij} = \frac{q_i q_j}{r_{ij}^2}$

If you're observing carefully, you might ask, wait doesn't this mean that molecules with different numbers of atoms generate matrices of different sizes? In practice the trick to get around this is that the matrices are "zero-padded." That is, if you're making coulomb matrices for a set of molecules, you pick a maximum number of atoms $N$, make the matrices $N\times N$ and set to zero all the extra entries for this molecule. (There's a couple extra tricks that are done under the hood beyond this. Check out reference [1] or read the source code in DeepChem!)

DeepChem has a built in featurization class `dc.feat.CoulombMatrixEig` that can generate these featurizations for you._____no_output_____

featurizer = dc.feat.CoulombMatrixEig(23, remove_hydrogens=False)_____no_output_____

Note that in this case, we set the maximum number of atoms to $N = 23$. Let's now load our dataset file into DeepChem. As in the previous tutorials, we use a `Loader` class, in particular `dc.data.SDFLoader` to load our `.sdf` file into DeepChem. The following snippet shows how we do this:_____no_output_____

loader = dc.data.SDFLoader(
      tasks=["atomization_energy"], smiles_field="smiles",
      featurizer=featurizer,
      mol_field="mol")
dataset = loader.featurize(dataset_file)_____no_output_____

For the purposes of this tutorial, we're going to do a random split of the dataset into training, validation, and test. In general, this split is weak and will considerably overestimate the accuracy of our models, but for now in this simple tutorial isn't a bad place to get started._____no_output_____

random_splitter = dc.splits.RandomSplitter()
train_dataset, valid_dataset, test_dataset = random_splitter.train_valid_test_split(dataset)_____no_output_____

One issue that Coulomb matrix featurizations have is that the range of entries in the matrix $C$ can be large. The charge $q_1q_2/r^2$ term can range very widely. In general, a wide range of values for inputs can throw off learning for the neural network. For this, a common fix is to normalize the input values so that they fall into a more standard range. Recall that the normalization transform applies to each feature $X_i$ of datapoint $X$

$\hat{X_i} = \frac{X_i - \mu_i}{\sigma_i}$

where $\mu_i$ and $\sigma_i$ are the mean and standard deviation of the $i$-th feature. This transformation enables the learning to proceed smoothly. A second point is that the atomization energies also fall across a wide range. So we apply an analogous transformation normalization transformation to the output to scale the energies better. We use DeepChem's transformation API to make this happen:_____no_output_____

transformers = [
    dc.trans.NormalizationTransformer(transform_X=True, dataset=train_dataset),
    dc.trans.NormalizationTransformer(transform_y=True, dataset=train_dataset)]

for dataset in [train_dataset, valid_dataset, test_dataset]:
  for transformer in transformers:
      dataset = transformer.transform(dataset)_____no_output_____

Now that we have the data cleanly transformed, let's do some simple machine learning. We'll start by constructing a random forest on top of the data. We'll use DeepChem's hyperparameter tuning module to do this._____no_output_____

def rf_model_builder(model_params, model_dir):
  sklearn_model = RandomForestRegressor(**model_params)
  return dc.models.SklearnModel(sklearn_model, model_dir)
params_dict = {
    "n_estimators": [10, 100],
    "max_features": ["auto", "sqrt", "log2", None],
}

metric = dc.metrics.Metric(dc.metrics.mean_absolute_error)
optimizer = dc.hyper.HyperparamOpt(rf_model_builder)
best_rf, best_rf_hyperparams, all_rf_results = optimizer.hyperparam_search(
    params_dict, train_dataset, valid_dataset, transformers,
    metric=metric)_____no_output_____

Let's build one more model, a kernel ridge regression, on top of this raw data._____no_output_____

def krr_model_builder(model_params, model_dir):
  sklearn_model = KernelRidge(**model_params)
  return dc.models.SklearnModel(sklearn_model, model_dir)

params_dict = {
    "kernel": ["laplacian"],
    "alpha": [0.0001],
    "gamma": [0.0001]
}

metric = dc.metrics.Metric(dc.metrics.mean_absolute_error)
optimizer = dc.hyper.HyperparamOpt(krr_model_builder)
best_krr, best_krr_hyperparams, all_krr_results = optimizer.hyperparam_search(
    params_dict, train_dataset, valid_dataset, transformers,
    metric=metric)_____no_output_____

**Bibliography:**

[1] https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.98.146401_____no_output_____


# Notebook from deartanapol10/ddear10-portfolio
Path: NLP Project yelp.ipynb

# Natural Language Processing Project

Welcome to the NLP Project for this section of the course. In this NLP project you will be attempting to classify Yelp Reviews into 1 star or 5 star categories based off the text content in the reviews. This will be a simpler procedure than the lecture, since we will utilize the pipeline methods for more complex tasks.

We will use the [Yelp Review Data Set from Kaggle](https://www.kaggle.com/c/yelp-recsys-2013).

Each observation in this dataset is a review of a particular business by a particular user.

The "stars" column is the number of stars (1 through 5) assigned by the reviewer to the business. (Higher stars is better.) In other words, it is the rating of the business by the person who wrote the review.

The "cool" column is the number of "cool" votes this review received from other Yelp users.

All reviews start with 0 "cool" votes, and there is no limit to how many "cool" votes a review can receive. In other words, it is a rating of the review itself, not a rating of the business.

The "useful" and "funny" columns are similar to the "cool" column.

Let's get started! Just follow the directions below!_____no_output_____## Imports
 **Import the usual suspects. :) **_____no_output_____

import numpy as np
import pandas as pd_____no_output_____

## The Data

**Read the yelp.csv file and set it as a dataframe called yelp.**_____no_output_____

yelp = pd.read_csv('yelp.csv')_____no_output_____

** Check the head, info , and describe methods on yelp.**_____no_output_____

yelp.head()_____no_output_____yelp.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 10 columns):
business_id    10000 non-null object
date           10000 non-null object
review_id      10000 non-null object
stars          10000 non-null int64
text           10000 non-null object
type           10000 non-null object
user_id        10000 non-null object
cool           10000 non-null int64
useful         10000 non-null int64
funny          10000 non-null int64
dtypes: int64(4), object(6)
memory usage: 781.3+ KB
yelp.describe()_____no_output_____

**Create a new column called "text length" which is the number of words in the text column.**_____no_output_____

yelp['text length'] = yelp['text'].apply(len)_____no_output_____

# EDA

Let's explore the data

## Imports

**Import the data visualization libraries if you haven't done so already.**_____no_output_____

import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('white')
%matplotlib inline_____no_output_____

**Use FacetGrid from the seaborn library to create a grid of 5 histograms of text length based off of the star ratings. Reference the seaborn documentation for hints on this**_____no_output_____

g = sns.FacetGrid(yelp,col='stars')
g.map(plt.hist,'text length')_____no_output_____

**Create a boxplot of text length for each star category.**_____no_output_____

sns.boxplot(x='stars',y='text length',data=yelp,palette='rainbow')_____no_output_____

**Create a countplot of the number of occurrences for each type of star rating.**_____no_output_____

sns.countplot(x='stars',data=yelp,palette='rainbow')_____no_output_____

** Use groupby to get the mean values of the numerical columns, you should be able to create this dataframe with the operation:**_____no_output_____

stars = yelp.groupby('stars').mean()
stars_____no_output_____

**Use the corr() method on that groupby dataframe to produce this dataframe:**_____no_output_____

stars.corr()_____no_output_____

**Then use seaborn to create a heatmap based off that .corr() dataframe:**_____no_output_____

sns.heatmap(stars.corr(),cmap='coolwarm',annot=True)_____no_output_____

## NLP Classification Task

Let's move on to the actual task. To make things a little easier, go ahead and only grab reviews that were either 1 star or 5 stars.

**Create a dataframe called yelp_class that contains the columns of yelp dataframe but for only the 1 or 5 star reviews.**_____no_output_____

yelp_class = yelp[(yelp.stars==1) | (yelp.stars==5)]_____no_output_____

** Create two objects X and y. X will be the 'text' column of yelp_class and y will be the 'stars' column of yelp_class. (Your features and target/labels)**_____no_output_____

X = yelp_class['text']
y = yelp_class['stars']_____no_output_____

**Import CountVectorizer and create a CountVectorizer object.**_____no_output_____

from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer()_____no_output_____

** Use the fit_transform method on the CountVectorizer object and pass in X (the 'text' column). Save this result by overwriting X.**_____no_output_____

X = cv.fit_transform(X)_____no_output_____

## Train Test Split

Let's split our data into training and testing data.

** Use train_test_split to split up the data into X_train, X_test, y_train, y_test. Use test_size=0.3 and random_state=101 **_____no_output_____

from sklearn.model_selection import train_test_split_____no_output_____X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.3,random_state=101)_____no_output_____

## Training a Model

Time to train a model!

** Import MultinomialNB and create an instance of the estimator and call is nb **_____no_output_____

from sklearn.naive_bayes import MultinomialNB
nb = MultinomialNB()_____no_output_____

**Now fit nb using the training data.**_____no_output_____

nb.fit(X_train,y_train)_____no_output_____

## Predictions and Evaluations

Time to see how our model did!

**Use the predict method off of nb to predict labels from X_test.**_____no_output_____

predictions = nb.predict(X_test)_____no_output_____

** Create a confusion matrix and classification report using these predictions and y_test **_____no_output_____

from sklearn.metrics import confusion_matrix,classification_report_____no_output_____print(confusion_matrix(y_test,predictions))
print('\n')
print(classification_report(y_test,predictions))[[159  69]
 [ 22 976]]

             precision    recall  f1-score   support

          1       0.88      0.70      0.78       228
          5       0.93      0.98      0.96       998

avg / total       0.92      0.93      0.92      1226

**Great! Let's see what happens if we try to include TF-IDF to this process using a pipeline.**_____no_output_____# Using Text Processing

** Import TfidfTransformer from sklearn. **_____no_output_____

from sklearn.feature_extraction.text import  TfidfTransformer_____no_output_____

** Import Pipeline from sklearn. **_____no_output_____

from sklearn.pipeline import Pipeline_____no_output_____

** Now create a pipeline with the following steps:CountVectorizer(), TfidfTransformer(),MultinomialNB()**_____no_output_____

pipeline = Pipeline([
    ('bow', CountVectorizer()),  # strings to token integer counts
    ('tfidf', TfidfTransformer()),  # integer counts to weighted TF-IDF scores
    ('classifier', MultinomialNB()),  # train on TF-IDF vectors w/ Naive Bayes classifier
])_____no_output_____

## Using the Pipeline

**Time to use the pipeline! Remember this pipeline has all your pre-process steps in it already, meaning we'll need to re-split the original data (Remember that we overwrote X as the CountVectorized version. What we need is just the text**_____no_output_____### Train Test Split

**Redo the train test split on the yelp_class object.**_____no_output_____

X = yelp_class['text']
y = yelp_class['stars']
X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.3,random_state=101)_____no_output_____

**Now fit the pipeline to the training data. Remember you can't use the same training data as last time because that data has already been vectorized. We need to pass in just the text and labels**_____no_output_____

# May take some time
pipeline.fit(X_train,y_train)_____no_output_____

### Predictions and Evaluation

** Now use the pipeline to predict from the X_test and create a classification report and confusion matrix. You should notice strange results.**_____no_output_____

predictions = pipeline.predict(X_test)_____no_output_____print(confusion_matrix(y_test,predictions))
print(classification_report(y_test,predictions))[[  0 228]
 [  0 998]]
             precision    recall  f1-score   support

          1       0.00      0.00      0.00       228
          5       0.81      1.00      0.90       998

avg / total       0.66      0.81      0.73      1226

Looks like Tf-Idf actually made things worse! That is it for this project. But there is still a lot more you can play with:

**Some other things to try....**
Try going back and playing around with the pipeline steps and seeing if creating a custom analyzer like we did in the lecture helps (note: it probably won't). Or recreate the pipeline with just the CountVectorizer() and NaiveBayes. Does changing the ML model at the end to another classifier help at all?_____no_output_____# Reference
_________
http://www.pieriandata.com_____no_output_____


# Notebook from lgalbany/IAA_School2019
Path: lectures/Day3-ZeljkoIvezic/notebooks/dimensionality_reduction.ipynb

[The 1st IAA-CSIC Severo Ochoa School on Statistics, Data Mining and Machine Learning](https://www.granadacongresos.com/sostat)

[Zeljko Ivezic, University of Washington](http://faculty.washington.edu/ivezic/)

[This notebook](https://github.com/carmensg/IAA_School2019/tree/master/lectures/Day3-ZeljkoIvezic/notebooks/dimensionality_reduction.ipynb)
### November 6, 2019

# Lecture 4: Dimensionality Reduction

### o Introduction to multi-dimensional distributions
### o Principal Component Analysis (PCA)
### o Independent Component Analysis (ICA)
### o Non-negative matrix factorization
### o Manifold Learning (example: Local Linear Embedding)
_____no_output_____

# first things first...
import numpy as np
from matplotlib import pyplot as plt
from scipy import stats
from scipy import integrate
from scipy.stats import norm
from scipy.stats import cauchy
# astroML tools
from astroML.plotting import hist
from astroML.plotting.mcmc import convert_to_stdev
from astroML.plotting import setup_text_plots
from astroML.plotting.mcmc import convert_to_stdev
setup_text_plots(fontsize=8, usetex=True)
import warnings; warnings.simplefilter('ignore')_____no_output_____

## Dimensionality Reduction##

Fitting and overfitting get worse with ''curse of dimensionality'' Bellman 1961

Think about a hypersphere. Its volume is  given by

\begin{equation}
  V_D(r) = \frac{2r^D\pi^{D/2}}{D\  \Gamma(D/2)},
\end{equation}

where $\Gamma(z)$ is the complete gamma function, $D$ is the dimension, and $r$ the radius of the sphere.

If you populated a hypercube of size $2r$ how much data would be enclosed by the hypersphere
- as $D$ increases the fractional volume enclosed by the hypersphere goes to 0!

For example: the SDSS comprises a sample of 357 million sources.
- each source has 448 measured attributes
- selecting just 30 (e.g., magnitude, size..) and normalizing the data range $-1$ to $1$

probability of having one of the 357 million sources reside within a unit hypersphere 1 in 1.4$\times 10^5$._____no_output_____

%matplotlib inline
import numpy as np
import scipy.special as sp
from matplotlib import pyplot as plt

def unitVolume(dimension, radius=1.):
    return 2*(radius**dimension *np.pi**(dimension/2.))/(dimension*sp.gamma(dimension/2.))

dim = np.linspace(1,100)

#------------------------------------------------------------
# Plot the results
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(dim,unitVolume(dim)/2.**dim)

ax.set_yscale('log')
ax.set_xlabel('$Dimension$')
ax.set_ylabel('$Volume$')

plt.show()_____no_output_____D=30;unitVolume(D)/(2.**D)_____no_output_____

## What can we do to reduce the dimensions?

### Principal component analysis or the Karhunen-Loeve transform

![slide](figures/pcascatter.png) _____no_output_____Points are  correlated along a particular direction which doesn't align with the initial choice of axes.
- we should rotate our axes to align with this correlation.
- rotation preserves the relative ordering of data

Choose  rotation to maximize the ability to discriminate between the data points
*   first axis, or principal component, is direction of maximal variance
*   second principal component is orthogonal to the first component and maximizes the residual variance
*   ..._____no_output_____
## Derivation of principal component analyses##

Set of data $X$: $N$ observations by $K$ measurements

Center data by subtracting the mean **Why?**

The covariance is

>$
C_X=\frac{1}{N-1}X^TX,
$

$N-1$ as the sample covariance matrix

We want a projection, $R$,  aligned with the directions of maximal variance ($Y= X R$) with covariance

>$
C_{Y} = R^T X^T X R = R^T C_X R
$

Derive  principal component by maximizing its variance (using Lagrange multipliers and constraint)

> $
\phi(r_1,\lambda_1) = r_1^TC_X r_1 - \lambda_1(r_1^Tr_1-1).
$

derivative of $\phi(r_1,\lambda)$ with respect to $r_1$ set to 0

> $
C_Xr_1 - \lambda_1 r_1 = 0.
$

$\lambda_1$ is the root of the equation $\det(C_X -
\lambda_1 {\bf I})=0$ and the largest eigenvalue

>$
\lambda_1 =  r_1^T C_X r_1
$

Other  principal components  derived by
applying additional constraint that components are uncorrelated (e.g., $r^T_2 C_X r_1 = 0$)._____no_output_____## Lagrangian mulitpliers
![slide](figures/1000px-LagrangeMultipliers2D.png)_____no_output_____## Computation of principal components##

Common approach is eigenvalue decomposition of the covariance or correlation matrix,
or singular value decomposition (SVD) of the data matrix

** SVD given by**

>$
U \Sigma V^T = \frac{1}{\sqrt{N - 1}} X,
$

columns of $U$ are  _left-singular vectors_

columns of $V$ are the _right-singular vectors_

The columns of $U$ and $V$ form orthonormal bases ($U^TU = V^TV = I$)

Covariance matrix is

> $
\begin{eqnarray}
  C_X &=& \left[\frac{1}{\sqrt{N - 1}}X\right]^T \left[\frac{1}{\sqrt{N - 1}}X\right]\nonumber\\
      &=& V \Sigma U^T U \Sigma V^T\nonumber\\
      &=& V \Sigma^2 V^T.
\end{eqnarray}
$

right singular vectors $V$ are the principal components. We can calculate principal components from the SVD of $X$ - we dont need $C_X$._____no_output_____![slide](figures/fig_svd_visual_1.png)

Singular value decomposition (SVD) can factorize an N x K matrix into $U \Sigma V^T$. There are different conventions for computing the SVD in the literature, and this figure illustrates the convention used in this text. The matrix of singular values $\Sigma$ is always a square matrix of size [R x R] where R = min(N, K). The shape of the resulting U and V matrices depends on whether N or K is larger. The columns of the matrix U are called the left-singular vectors, and the columns of the matrix V are called the right-singular vectors. The columns are orthonormal bases, and satisfy $U^T U = V^T V = I$.

## Preparing data for PCA##

- Center data by subtracting the mean of each dimension
- For heterogeneous data (e.g., galaxy shape and flux) divide by  variance (whitening). **why?**
- For spectra or images normalize each row so integrated flux of each object is one.
_____no_output_____

from sklearn.decomposition import PCA
from sklearn.decomposition import RandomizedPCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#----------------------------------------------------------------------
# Compute PCA
def compute_PCA(n_components=5):
#    np.random.seed(500)
    nrows = 500
    ind = np.random.randint(spectra.shape[0], size=nrows)

    spec_mean = spectra[ind].mean(0)
#    spec_mean = spectra[:50].mean(0)

    # PCA: use randomized PCA for speed
    pca = PCA(n_components - 1)
    pca.fit(spectra[ind])
    pca_comp = np.vstack([spec_mean,
                          pca.components_])
    evals = pca.explained_variance_ratio_

    return pca_comp, evals

n_components = 5
decompositions, evals = compute_PCA(n_components)

#----------------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 8))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = 'PCA components'

for j in range(n_components):
    ax = fig.add_subplot(n_components, 2, 2*j+2)

    ax.yaxis.set_major_formatter(plt.NullFormatter())
    ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax.set_xlabel(r'wavelength ${\rm (\AA)}$')
    ax.plot(wavelengths, decompositions[j], '-k', lw=1)

    # plot zero line
    xlim = [3000, 7999]
    ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax.set_xlim(xlim)

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    ax2 = fig.add_subplot(n_components, 2, 2*j+1)
    ax2.yaxis.set_major_formatter(plt.NullFormatter())
    ax2.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax2.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax2.set_xlabel(r'wavelength ${\rm (\AA)}$')
    ax2.plot(wavelengths, spectra[j], '-k', lw=1)

    # plot zero line
    ax2.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax2.set_xlim(xlim)

    if j == 0:
        ax.set_title(titles, fontsize='medium')

    if j == 0:
        label = 'mean'
    else:
        label = 'component %i' % j

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax2.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    ax.text(0.02, 0.95, label, transform=ax.transAxes,
            ha='left', va='top', bbox=dict(ec='w', fc='w'),
            fontsize='small')

plt.show()_____no_output_____

## Interpreting the PCA##

Reconstruction of spectrum, ${x}(k)$, from the
eigenvectors, ${e}_i(k)$

>$ \begin{equation}
  {x}_i(k) = {\mu}(k) + \sum_j^R \theta_{ij} {e}_j(k),
\end{equation}
$

Truncating this expansion (i.e., $r<R$)

>$\begin{equation}
{x}_i(k) = {\mu}(k) + \sum_i^{r<R} \theta_i {e}_i(k),
\end{equation}
$

- eigenvectors ordered by their associated eigenvalues
- eigenvalues reflect variance  within each eigenvector (sum of the eigenvalues is total variance of the system).
- project a each spectrum onto these first few eigenspectra is a compression of the data

This is the sense in which PCA gives for dimensionality reduction._____no_output_____

from sklearn.decomposition import PCA, KernelPCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#------------------------------------------------------------
# Compute PCA components

# Eigenvalues can be computed using PCA as in the commented code below:

#from sklearn.decomposition import PCA
#pca = PCA()
#pca.fit(spectra)
#evals = pca.explained_variance_ratio_
#evals_cs = evals.cumsum()

#  because the spectra have been reconstructed from masked values, this
#  is not exactly correct in this case: we'll use the values computed
#  in the file compute_sdss_pca.py
evals = data['evals'] ** 2
evals_cs = evals.cumsum()
evals_cs /= evals_cs[-1]
evecs = data['evecs']
spec_mean = spectra.mean(0)

#------------------------------------------------------------
# Find the coefficients of a particular spectrum
spec = spectra[1]
coeff = np.dot(evecs, spec - spec_mean)

#------------------------------------------------------------
# Plot the sequence of reconstructions
fig = plt.figure(figsize=(8, 8))
fig.subplots_adjust(hspace=0)

for i, n in enumerate([0, 4, 8, 20]):
    ax = fig.add_subplot(411 + i)
    ax.plot(wavelengths, spec, '-', c='gray')
    ax.plot(wavelengths, spec_mean + np.dot(coeff[:n], evecs[:n]), '-k')

    if i < 3:
        ax.xaxis.set_major_formatter(plt.NullFormatter())

    ax.set_ylim(-2, 21)
    ax.set_ylabel('flux')

    if n == 0:
        text = "mean"
    elif n == 1:
        text = "mean + 1 component\n"
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]
    else:
        text = "mean + %i components\n" % n
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]

    ax.text(0.01, 0.95, text, ha='left', va='top', transform=ax.transAxes)

fig.axes[-1].set_xlabel(r'${\rm wavelength\ (\AA)}$')
plt.show()_____no_output_____

## Comparing dimensionality reduction (PCA vs LLE)
![slide](figures/fig_PCA_LLE_1.png)

![slide](figures/fig_PCA_LLE_2.png) _____no_output_____*  Comparisons between the eigenvectors derived from PCA and known spectral types of galaxies
have shown that these statistically orthogonal components correlate
strongly with specific physical properties
*  given the cumulative nature of the sum of
variances used in PCA, astrophysically interesting components within
the spectra (e.g., sharp spectral lines or transient features for
certain galaxy populations) may not be reflected in the largest PCA
components.
- sum of
linear components can efficiently reconstruct the features within the
data does not always hold.  An example of this is the variation in
broad emission lines (such as those from quasars). The
variation in line width is an inherently nonlinear process and can
require a large number of components to fully characterize:
for broad line quasars over 30
components are required to reproduce the underlying spectra compared
to the 10 required for quiescent and star-forming galaxies.  _____no_output_____## PCA with missing data##

Observed spectrum, $x^o$ is the true spectrum, ${x}$ plus a wavelength-dependent weight, ${w}$.

Weight is zero where data are missing and $1/{\sigma}^2$ for rest

Minimizing the quadratic deviation between  ${x}^o$ truncated reconstruction, $\sum_i \theta_i {e}_i$
and solving for $\theta_i$ gives

>$\begin{equation}
	\sum_k \theta_i {w}(k) {e}_i(k) {e}_j(k) =
	\sum_k {w}(k) {x}^o(k) {e}_j(k),
\end{equation}
$

If $M_{ij} = \sum_k {w}(k) {e}_i(k) {e}_j(k)$ and $F_i = \sum_k {w}(k) {x}^o(k) {e}_i(k)$ then

>$\begin{equation}
	\theta_i = \sum_j M_{ij}^{-1} F_{j},
\end{equation}
$

- $F_j$ are  coefficients derived from  gappy data
- $M_{ij}^{-1}$ shows how correlated  eigenvectors are over the missing regions.

An estimate of the uncertainty on the
reconstruction coefficients is given by

>$\begin{equation}
%Cov(\theta_i,\theta_j) = \frac{1}{N}M_{ij}^{-1}
{\rm Cov}(\theta_i,\theta_j) = M_{ij}^{-1}.
\end{equation}
$

Accuracy of this reconstruction will depend on the distribution of
the gaps within the data vector._____no_output_____

from astroML.datasets import fetch_sdss_corrected_spectra
from astroML.datasets import sdss_corrected_spectra
from matplotlib import ticker

#------------------------------------------------------------
# Get spectra and eigenvectors used to reconstruct them
data = fetch_sdss_corrected_spectra()
spec = sdss_corrected_spectra.reconstruct_spectra(data)
lam = sdss_corrected_spectra.compute_wavelengths(data)
evecs = data['evecs']
mu = data['mu']
norms = data['norms']
mask = data['mask']

#------------------------------------------------------------
# plot the results
i_plot = ((lam > 5750) & (lam < 6350))
lam = lam[i_plot]

specnums = [20, 8, 9]
subplots = [311, 312, 313]

fig = plt.figure(figsize=(12, 15))
fig.subplots_adjust(hspace=0)

for subplot, i in zip(subplots, specnums):
    ax = fig.add_subplot(subplot)

    # compute eigen-coefficients
    spec_i_centered = spec[i] / norms[i] - mu
    coeffs = np.dot(spec_i_centered, evecs.T)

    # blank out masked regions
    spec_i = spec[i]
    mask_i = mask[i]
    spec_i[mask_i] = np.nan

    # plot the raw masked spectrum
    ax.plot(lam, spec_i[i_plot], '-', color='k', lw=2,
            label='True spectrum')

    # plot two levels of reconstruction
    for nev in [10]:
        if nev == 0:
            label = 'mean'
        else:
            label = 'nev=%i' % nev
        spec_i_recons = norms[i] * (mu + np.dot(coeffs[:nev], evecs[:nev]))
        ax.plot(lam, spec_i_recons[i_plot], label=label)

    # plot shaded background in masked region
    ylim = ax.get_ylim()
    mask_shade = ylim[0] + mask[i][i_plot].astype(float) * ylim[1]
    plt.fill(np.concatenate([lam[:1], lam, lam[-1:]]),
             np.concatenate([[ylim[0]], mask_shade, [ylim[0]]]),
             lw=0, fc='k', alpha=0.2)

    ax.set_xlim(lam[0], lam[-1])
    ax.set_ylim(ylim)
    ax.yaxis.set_major_formatter(ticker.NullFormatter())

    if subplot == 311:
        ax.legend(loc=1, prop=dict(size=14))

    ax.set_xlabel('$\lambda\ (\AA)$')
    ax.set_ylabel('normalized flux')

plt.show()_____no_output_____

 ## Independent Component Analysis (ICA)

Independent component analysis (ICA) solves the ``cocktail party problem'', where there are multiple microphones situated through out a room containing
$N$ people. Each microphone picks up a linear combination of the $N$
voices. The goal of ICA is to use the concept of statistical
independence to isolate (or unmix) the individual signals.

Each spectrum, $x_i(k)$, can be described by

>$x_1(k) = a_{11}s_1(k) + a_{12}s_2(k) + a_{13}s_3(k) + \cdots\\
x_2(k) = a_{21}s_1(k) + a_{22}s_2(k) + a_{23}s_3(k) + \cdots \\
x_3(k) = a_{31}s_1(k) + a_{32}s_2(k) + a_{33}s_3(k) + \cdots \\
$

where $s_i(k)$ are the individual  spectra and $a_{ij}$
the appropriate mixing amplitudes.

In matrix form,
>$ X = A S$,

 Extracting these signal spectra is equivalent to estimating the appropriate weight matrix,
$W$, such that
>$ S = W X$.

Underlying ICA is the observation that the
input signals, $s_i(k)$, should be **statistically
independent** - the joint probability distribution, $f(x,y)$, can be
fully described by  $f(x^p,y^q)=f(x^p)f(y^q)$

 The rationale for ICA is that the
sum of any two independent random variables will always be more
Gaussian than either of the individual random variables (i.e., from the
central limit theorem). If we identify an unmixing matrix, $W$, that
maximizes the non-Gaussianity of the distributions, then we would be
identifying the input signals. _____no_output_____

# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import FastICA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#----------------------------------------------------------------------
# Compute ICA components

def compute_ICA(n_components=5):
    spec_mean = spectra.mean(0)

    # ICA treats sequential observations as related.  Because of this, we need
    # to fit with the transpose of the spectra
    ica = FastICA(n_components - 1, algorithm='deflation')
    ica.fit(spectra.T)
    ica_comp = np.vstack([spec_mean,
                          ica.transform(spectra.T).T])

    return ica_comp

n_components = 5
decompositions = compute_ICA(n_components)

#----------------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 8))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = ['ICA components',]

for j in range(n_components):
    ax = fig.add_subplot(n_components, 1, j + 1)

    ax.yaxis.set_major_formatter(plt.NullFormatter())
    ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax.set_xlabel(r'wavelength ${\rm (\AA)}$')

    ax.plot(wavelengths, comp[j], '-k', lw=1)

    # plot zero line
    xlim = [3000, 7999]
    ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax.set_xlim(xlim)

    if j == 0:
        ax.set_title(titles[0])
        label = 'mean'
    else:
        label = 'component %i' % j

    ax.text(0.03, 0.94, label, transform=ax.transAxes, ha='left', va='top')

    for l in ax.get_xticklines() + ax.get_yticklines():
        l.set_markersize(2)

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

plt.show()_____no_output_____

 ## Non-negative matrix factorization

Eigenvectors are defined relative to  mean data vector. Principal components that can be positive or negative but for many physical systems we know data are a linear sum of positive components (e.g. galaxy spectrum is a linear sum of stellar components

Nonnegative matrix factorization (NMF) applies positivity constraint

Assumes  any data matrix can be factored into two matrices, $W$ and $Y$, with
>$\begin{equation}
X=W Y,
\end{equation}
$

and both $W$ and $Y$ are nonnegative

$WY$ is an approximation of $X$. Minimizing the reconstruction error $|| (X - W Y)^2 ||$
nonnegative bases can be derived using a simple update rule (Lee and Seung),
>$
\begin{eqnarray}
W_{ki} &=& W_{ki}\frac{[XY^T]_{ki}}{[WYY^T]_{ki}},\\
Y_{in} &=& Y_{in}\frac{[W^TX]_{in}}{[W^TWY]_{in}},
\end{eqnarray}
$

Iterative process does not guarantee nonlocal minima (like $K$-means and EM).

Use random initialization and cross-validation _____no_output_____

# Author: Jake VanderPlas
# License: BSD
%matplotlib inline

import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import NMF
from sklearn.decomposition import RandomizedPCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#----------------------------------------------------------------------
# Compute PCA, and NMF components
#  we'll save the results so that they can be re-used
@pickle_results('spec_decompositions.pkl')
def compute_PCA_NMF(n_components=5):
    spec_mean = spectra.mean(0)

    # PCA: use randomized PCA for speed
    pca = RandomizedPCA(n_components - 1)
    pca.fit(spectra)
    pca_comp = np.vstack([spec_mean,
                          pca.components_])

    # NMF requires all elements of the input to be greater than zero
    spectra[spectra < 0] = 0
    nmf = NMF(n_components)
    nmf.fit(spectra)
    nmf_comp = nmf.components_

    return pca_comp, nmf_comp

n_components = 5
decompositions = compute_PCA_NMF(n_components)

#----------------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 8))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = ['PCA components', 'NMF components']

for i, comp in enumerate(decompositions):
    for j in range(n_components):
        ax = fig.add_subplot(n_components, 3, 3 * j + 1 + i)

        ax.yaxis.set_major_formatter(plt.NullFormatter())
        ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
        if j < n_components - 1:
            ax.xaxis.set_major_formatter(plt.NullFormatter())
        else:
            ax.set_xlabel(r'wavelength ${\rm (\AA)}$')

        ax.plot(wavelengths, comp[j], '-k', lw=1)

        # plot zero line
        xlim = [3000, 7999]
        ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
        ax.set_xlim(xlim)

        if j == 0:
            ax.set_title(titles[i])

        if titles[i].startswith('PCA') or titles[i].startswith('ICA'):
            if j == 0:
                label = 'mean'
            else:
                label = 'component %i' % j
        else:
            label = 'component %i' % (j + 1)

        ax.text(0.03, 0.94, label, transform=ax.transAxes,
                ha='left', va='top')

        for l in ax.get_xticklines() + ax.get_yticklines():
            l.set_markersize(2)

        # adjust y limits
        ylim = plt.ylim()
        dy = 0.05 * (ylim[1] - ylim[0])
        ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

plt.show()
 @pickle_results: computing results and saving to 'spec_decompositions.pkl'

## Manifold Learning

Real-world data sets often have very nonlinear features (e.g. QSOs with broad lines) which are hard to describe compactly using linear eigenvectors. Manifold learning techniques search for a representation of these data within a lower dimensional space

_____no_output_____ **Local Linear Embedding:**

Step 1: define local geometry
- local neighborhoods determined from $k$ nearest neighbors.
- for each point calculate weights that reconstruct a point from its $k$ nearest
neighbors
>$
\begin{equation}
  \mathcal{E}_1(W) = \left|X - WX\right|^2
\end{equation}
$
- essentially this is finding the hyperplane that describes the local surface at each point within the data set.

Step 2: embed within a lower dimensional space
- set all $W_{ij}=0$ except when point $j$ is one of the $k$ nearest neighbors of point $i$.
- $W$ becomes very sparse for $k \ll N$.
- minimize
>$\begin{equation}
  \mathcal{E}_2(Y) = \left|Y - W Y\right|^2,
\end{equation}
$

with $W$ fixed to find an $N$ by $d$ matrix ($d$ is the new dimensionality)

Step 1 requires a nearest-neighbor search.

Step 2 requires an
eigenvalue decomposition of the matrix $C_W \equiv (I-W)^T(I-W)$,

LLE has been applied to data as diverse as galaxy spectra, stellar spectra, and photometric light curves. _____no_output_____ _____no_output_____

# Author: Jake VanderPlas
# License: BSD
import numpy as np
from matplotlib import pyplot as plt
from sklearn import manifold, neighbors

from astroML.datasets import sdss_corrected_spectra
from astroML.datasets import fetch_sdss_corrected_spectra
from astroML.plotting.tools import discretize_cmap
from astroML.decorators import pickle_results

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

#------------------------------------------------------------
# Set up color-map properties
clim = (1.5, 6.5)
cmap = discretize_cmap(plt.cm.jet, 5)
cdict = ['unknown', 'star', 'absorption galaxy',
         'galaxy', 'emission galaxy',
         'narrow-line QSO', 'broad-line QSO']
cticks = [2, 3, 4, 5, 6]
formatter = plt.FuncFormatter(lambda t, *args: cdict[int(np.round(t))])

#------------------------------------------------------------
# Fetch the data; PCA coefficients have been pre-computed
data = fetch_sdss_corrected_spectra()
coeffs_PCA = data['coeffs']
c_PCA = data['lineindex_cln']
spec = sdss_corrected_spectra.reconstruct_spectra(data)
color = data['lineindex_cln']

#------------------------------------------------------------
# Compute the LLE projection; save the results
@pickle_results("spec_LLE.pkl")
def compute_spec_LLE(n_neighbors=10, out_dim=3):
    # Compute the LLE projection
    LLE = manifold.LocallyLinearEmbedding(n_neighbors, out_dim,
                                          method='modified',
                                          eigen_solver='dense')
    Y_LLE = LLE.fit_transform(spec)
    print (" - finished LLE projection")

    # remove outliers for the plot
    BT = neighbors.BallTree(Y_LLE)
    dist, ind = BT.query(Y_LLE, n_neighbors)
    dist_to_n = dist[:, -1]
    dist_to_n -= dist_to_n.mean()
    std = np.std(dist_to_n)
    flag = (dist_to_n > 0.25 * std)
    print (" - removing %i outliers for plot" % flag.sum())

    return Y_LLE[~flag], color[~flag]

coeffs_LLE, c_LLE = compute_spec_LLE(10, 3)

#----------------------------------------------------------------------
# Plot the results:
for (c, coeffs, xlim) in zip([c_PCA, c_LLE],
                             [coeffs_PCA, coeffs_LLE],
                             [(-1.2, 1.0), (-0.01, 0.014)]):
    fig = plt.figure(figsize=(5, 3.75))
    fig.subplots_adjust(hspace=0.05, wspace=0.05)

    # axes for colorbar
    cax = plt.axes([0.525, 0.525, 0.02, 0.35])

    # Create scatter-plots
    scatter_kwargs = dict(s=4, lw=0, edgecolors='none', c=c, cmap=cmap)

    ax1 = plt.subplot(221)
    im1 = ax1.scatter(coeffs[:, 0], coeffs[:, 1], **scatter_kwargs)
    im1.set_clim(clim)
    ax1.set_ylabel('$c_2$')

    ax2 = plt.subplot(223)
    im2 = ax2.scatter(coeffs[:, 0], coeffs[:, 2], **scatter_kwargs)
    im2.set_clim(clim)
    ax2.set_xlabel('$c_1$')
    ax2.set_ylabel('$c_3$')

    ax3 = plt.subplot(224)
    im3 = ax3.scatter(coeffs[:, 1], coeffs[:, 2], **scatter_kwargs)
    im3.set_clim(clim)
    ax3.set_xlabel('$c_2$')

    fig.colorbar(im3, ax=ax3, cax=cax,
                 ticks=cticks,
                 format=formatter)

    ax1.xaxis.set_major_formatter(plt.NullFormatter())
    ax3.yaxis.set_major_formatter(plt.NullFormatter())

    ax1.set_xlim(xlim)
    ax2.set_xlim(xlim)

    for ax in (ax1, ax2, ax3):
        ax.xaxis.set_major_locator(plt.MaxNLocator(5))
        ax.yaxis.set_major_locator(plt.MaxNLocator(5))

plt.show()
 @pickle_results: computing results and saving to 'spec_LLE.pkl'
 - finished LLE projection
 - removing 33 outliers for plot


# Notebook from rhallez/ColabFold
Path: verbose/alphafold_noTemplates_noMD.ipynb

_____no_output_____#AlphaFold_____no_output_____

#################
# WARNING
#################
# - This notebook is intended as a "quick" demo, it disables many aspects of the full alphafold2 pipeline
#   (input MSA/templates, amber MD refinement, and number of models). For best results, we recommend using the
#   full pipeline: https://github.com/deepmind/alphafold
# - That being said, it was found that input templates and amber-relax did not help much.
#   The key input features are the MSA (Multiple Sequence Alignment) of related proteins. Where you see a
#   significant drop in predicted accuracy when MSA < 30, but only minor improvements > 100.
# - This notebook does NOT include the alphafold2 MSA generation pipeline, and is designed to work with a
#   single sequence, custom MSA input (that you can upload) or MMseqs2 webserver.
# - Single sequence mode is particularly useful for denovo designed proteins (where there are no sequence
#   homologs by definition). For natural proteins, an MSA input will make a huge difference.

#################
# EXTRA
#################
# For other related notebooks see: https://github.com/sokrypton/ColabFold_____no_output_____%%bash
# download model
git clone https://github.com/deepmind/alphafold.git
mv alphafold alphafold_
mv alphafold_/alphafold .Cloning into 'alphafold'...
%%bash
# download model params (~1 min)
wget -qnc https://storage.googleapis.com/alphafold/alphafold_params_2021-07-14.tar
mkdir params
tar -xf alphafold_params_2021-07-14.tar -C params/
rm alphafold_params_2021-07-14.tar_____no_output_____%%bash
pip -q install biopython
pip -q install dm-haiku
pip -q install ml-collections
pip -q install py3Dmol_____no_output_____import os
import numpy as np
import pickle
import py3Dmol
import matplotlib.pyplot as plt_____no_output_____from alphafold.common import protein
from alphafold.data import pipeline
from alphafold.data import templates
from alphafold.model import data
from alphafold.model import config
from alphafold.model import model_____no_output_____# setup which model params to use
# note: for this demo, we only use model 1, for all five models uncomments the others!
model_runners = {}
models = ["model_1"] #,"model_2","model_3","model_4","model_5"]
for model_name in models:
  model_config = config.model_config(model_name)
  model_config.data.eval.num_ensemble = 1
  model_params = data.get_model_haiku_params(model_name=model_name, data_dir=".")
  model_runner = model.RunModel(model_config, model_params)
  model_runners[model_name] = model_runner_____no_output_____def mk_mock_template(query_sequence):
  # since alphafold's model requires a template input
  # we create a blank example w/ zero input, confidence -1
  ln = len(query_sequence)
  output_templates_sequence = "-"*ln
  output_confidence_scores = np.full(ln,-1)
  templates_all_atom_positions = np.zeros((ln, templates.residue_constants.atom_type_num, 3))
  templates_all_atom_masks = np.zeros((ln, templates.residue_constants.atom_type_num))
  templates_aatype = templates.residue_constants.sequence_to_onehot(output_templates_sequence,
                                                                    templates.residue_constants.HHBLITS_AA_TO_ID)
  template_features = {'template_all_atom_positions': templates_all_atom_positions[None],
                       'template_all_atom_masks': templates_all_atom_masks[None],
                       'template_sequence': [f'none'.encode()],
                       'template_aatype': np.array(templates_aatype)[None],
                       'template_confidence_scores': output_confidence_scores[None],
                       'template_domain_names': [f'none'.encode()],
                       'template_release_date': [f'none'.encode()]}
  return template_features_____no_output_____def predict_structure(prefix, feature_dict, model_runners, random_seed=0):
  """Predicts structure using AlphaFold for the given sequence."""

  # Run the models.
  plddts = {}
  for model_name, model_runner in model_runners.items():
    processed_feature_dict = model_runner.process_features(feature_dict, random_seed=random_seed)
    prediction_result = model_runner.predict(processed_feature_dict)
    unrelaxed_protein = protein.from_prediction(processed_feature_dict,prediction_result)
    unrelaxed_pdb_path = f'{prefix}_unrelaxed_{model_name}.pdb'
    plddts[model_name] = prediction_result['plddt']

    print(f"{model_name} {plddts[model_name].mean()}")

    with open(unrelaxed_pdb_path, 'w') as f:
      f.write(protein.to_pdb(unrelaxed_protein))
  return plddts_____no_output_____

# Single sequence input (no MSA)_____no_output_____

# Change this line to sequence you want to predict
query_sequence = "GWSTELEKHREELKEFLKKEGITNVEIRIDNGRLEVRVEGGTERLKRFLEELRQKLEKKGYTVDIKIE"_____no_output_____%%time
feature_dict = {
    **pipeline.make_sequence_features(sequence=query_sequence,
                                      description="none",
                                      num_res=len(query_sequence)),
    **pipeline.make_msa_features(msas=[[query_sequence]],
                                 deletion_matrices=[[[0]*len(query_sequence)]]),
    **mk_mock_template(query_sequence)
}
plddts = predict_structure("test",feature_dict,model_runners)model_1 83.72061523404395
CPU times: user 2min 16s, sys: 2.83 s, total: 2min 18s
Wall time: 1min 53s
# confidence per position
plt.figure(dpi=100)
for model,value in plddts.items():
  plt.plot(value,label=model)
plt.legend()
plt.ylim(0,100)
plt.ylabel("predicted LDDT")
plt.xlabel("positions")
plt.show()_____no_output_____  p = py3Dmol.view(js='https://3dmol.org/build/3Dmol.js')
  p.addModel(open("test_unrelaxed_model_1.pdb",'r').read(),'pdb')
  p.setStyle({'cartoon': {'color':'spectrum'}})
  p.zoomTo()
  p.show()_____no_output_____  p = py3Dmol.view(js='https://3dmol.org/build/3Dmol.js')
  p.addModel(open("test_unrelaxed_model_1.pdb",'r').read(),'pdb')
  p.setStyle({'cartoon': {'color':'spectrum'},'stick':{}})
  p.zoomTo()
  p.show()_____no_output_____

# Custom MSA input_____no_output_____

%%bash
# for this demo we will download a premade MSA input
wget -qnc https://gremlin2.bakerlab.org/db/ECOLI/fasta/P0A8I3.fas_____no_output_____msa, deletion_matrix = pipeline.parsers.parse_a3m("".join(open("P0A8I3.fas","r").readlines()))
query_sequence = msa[0]_____no_output_____%%time
feature_dict = {
    **pipeline.make_sequence_features(sequence=query_sequence,
                                      description="none",
                                      num_res=len(query_sequence)),
    **pipeline.make_msa_features(msas=[msa],deletion_matrices=[deletion_matrix]),
    **mk_mock_template(query_sequence)
}
plddts = predict_structure("yaaa",feature_dict,model_runners)model_1 97.32114827157228
CPU times: user 2min 48s, sys: 16.4 s, total: 3min 4s
Wall time: 2min 43s
# confidence per position
plt.figure(dpi=100)
for model,value in plddts.items():
  plt.plot(value,label=model)
plt.legend()
plt.ylim(0,100)
plt.ylabel("predicted LDDT")
plt.xlabel("positions")
plt.show()_____no_output_____  p = py3Dmol.view(js='https://3dmol.org/build/3Dmol.js')
  p.addModel(open("yaaa_unrelaxed_model_1.pdb",'r').read(),'pdb')
  p.setStyle({'cartoon': {'color':'spectrum'}})
  p.zoomTo()
  p.show()_____no_output_____

#MSA from MMseqs2_____no_output_____

##############################
# Where do I get an MSA?
##############################
# For any "serious" use, I would recommend using the alphafold2 pipeline to make the MSAs,
# since this is what it was trained on.

# That being said, part of the MSA generation pipeline (specifically searching against uniprot database using hhblits)
# can be done here: https://toolkit.tuebingen.mpg.de/tools/hhblits
# Alternatively, using the SUPER FAST MMseqs2 pipeline below

# for a HUMAN FRIENDLY version see:
# https://colab.research.google.com/drive/1LVPSOf4L502F21RWBmYJJYYLDlOU2NTL_____no_output_____%%bash
apt-get -qq -y update 2>&1 1>/dev/null
apt-get -qq -y install jq curl zlib1g gawk 2>&1 1>/dev/null_____no_output_____# save query sequence to file
name = "YAII"
query_sequence = "MTIWVDADACPNVIKEILYRAAERMQMPLVLVANQSLRVPPSRFIRTLRVAAGFDVADNEIVRQCEAGDLVITADIPLAAEAIEKGAAALNPRGERYTPATIRERLTMRDFMDTLRASGIQTGGPDSLSQRDRQAFAAELEKWWLEVQRSRG"
with open(f"{name}.fasta","w") as out: out.write(f">{name}\n{query_sequence}\n")_____no_output_____%%bash -s "$name"
# build msa using the MMseqs2 search server
ID=$(curl -s -F q=@$1.fasta -F mode=all https://a3m.mmseqs.com/ticket/msa | jq -r '.id')
STATUS=$(curl -s https://a3m.mmseqs.com/ticket/${ID} | jq -r '.status')
while [ "${STATUS}" == "RUNNING" ]; do
    STATUS=$(curl -s https://a3m.mmseqs.com/ticket/${ID} | jq -r '.status')
    sleep 1
done
if [ "${STATUS}" == "COMPLETE" ]; then
    curl -s https://a3m.mmseqs.com/result/download/${ID}  > $1.tar.gz
    tar xzf $1.tar.gz
    tr -d '\000' < uniref.a3m > $1.a3m
else
    echo "MMseqs2 server did not return a valid result."
    exit 1
fi
echo "Found $(grep -c ">" $1.a3m) sequences (after redundacy filtering)"Found 3090 sequences (after redundacy filtering)
msa, deletion_matrix = pipeline.parsers.parse_a3m("".join(open(f"{name}.a3m","r").readlines()))
query_sequence = msa[0]_____no_output_____%%time
feature_dict = {
    **pipeline.make_sequence_features(sequence=query_sequence,
                                      description="none",
                                      num_res=len(query_sequence)),
    **pipeline.make_msa_features(msas=[msa],deletion_matrices=[deletion_matrix]),
    **mk_mock_template(query_sequence)
}
plddts = predict_structure(name,feature_dict,model_runners)model_1 91.07940053970414
CPU times: user 2min 24s, sys: 6.47 s, total: 2min 31s
Wall time: 2min 5s
plt.figure(dpi=100)
plt.plot((feature_dict["msa"] != 21).sum(0))
plt.plot([0,len(query_sequence)],[30,30])
plt.xlabel("positions")
plt.ylabel("number of sequences")
plt.show()_____no_output_____# confidence per position
plt.figure(dpi=100)
for model,value in plddts.items():
  plt.plot(value,label=model)
plt.legend()
plt.ylim(0,100)
plt.ylabel("predicted LDDT")
plt.xlabel("positions")
plt.show()_____no_output_____  p = py3Dmol.view(js='https://3dmol.org/build/3Dmol.js')
  p.addModel(open(f"{name}_unrelaxed_model_1.pdb",'r').read(),'pdb')
  p.setStyle({'cartoon': {'color':'spectrum'}})
  p.zoomTo()
  p.show()_____no_output__________no_output_____


# Notebook from kant/GlennOPT
Path: test/Rosenbrock/RosenbrockExample.ipynb

#Single Objective Rosenbrock Example
In this example you will learn how to set up a single objective differential evolution solver to solving for the minimum of the rosenbrock function.

> Note: When working with an optimizer or with any code. It is important to know what folder you are launching the optimization on. You need to know where your working folder is. In Linux terminal or powershell you can type `pwd` to figure this out.

> The paths that I use in the examples are all relative paths. This is so that the code is easily shared with others.

> Using absolute paths should only be reserved for programs that are sensitive and are not meant to be shared.

## About Rosenbrock
Rosenbrock is an interesting function with steep gradients near the bounds but as you get close to the minimum (blue zone) the gradients become less steep and close to 0. This makes it a hard function to find the minimum of.

The equation for Rosenbrock

$f(x,y) = (a-x)^2 + b(y-x^2)^2$

Multi-variable version is

$f(x) = f(x_1,x_2,x_3,...,x_n) = \sum_{i=1}^{N/2} [100(x_{2i-1}^2 - x_{2i})^2 + (x_{2i-1}-1)^2]$

Link to wikipedia article https://en.wikipedia.org/wiki/Rosenbrock_function

[Example from Wikpedia](https://en.wikipedia.org/wiki/Rosenbrock_function#/media/File:Rosenbrock_function.svg)

## Methodology
1. This tutorial will cover the following steps cloning of the project. This downloads the code to evaluate the rosenbrock function.
2. Is the setting up of glennopt to copy the evaluation function and execute a design of experiments(DOE). This evaluates the design space and lets the optimizer know where the minimum might be and what parameters.
3. Optimization from the DOE. From the randomized design of experiments or non random if you use fullfactorial. The optimizer will proceed with taking the best design and perform mutation and crossover with other designs to generate a new population of potentially better designs. The loop will continue indefinitely; however it may reach something that is close to a minimum by the 50th population.
_____no_output_____###Step 1: Cloning the Project and Rosenbrock codes
Lets clone the test project in Glennopt. We will need the test folder located in GlennOPT/test/Rosenbrock/serial/Evaluation/._____no_output_____

# Clone the source code for GlennOPT
!git clone https://github.com/nasa/GlennOPT.git
# Little Housekeeping
!cp -r GlennOPT/test/Rosenbrock/serial/Evaluation/ .    # Copy the folder we need
!rm GlennOPT/ -r        # Deletes GlennOPT source code. We don't need this anymore
!rm sample_data -rCloning into 'GlennOPT'...
remote: Enumerating objects: 1391, done.
remote: Counting objects: 100% (722/722), done.
remote: Compressing objects: 100% (443/443), done.
remote: Total 1391 (delta 412), reused 521 (delta 246), pack-reused 669
Receiving objects: 100% (1391/1391), 2.75 MiB | 11.53 MiB/s, done.
Resolving deltas: 100% (791/791), done.
# Install GlennOPT
!python --version
!pip install --upgrade glennopt pillow matplotlib numpyPython 3.7.10
Collecting glennopt
  Downloading https://files.pythonhosted.org/packages/a1/c3/ed861e2fa828a606d83a812aab69eeb4e7750c1895eb8eba63f5f37ba9f2/GlennOPT-1.2.1-py3-none-any.whl
Collecting pillow
  Downloading https://files.pythonhosted.org/packages/8e/7a/b047f6f80fdb02c0cca1d3761d71e9800bcf6d4874b71c9e6548ec59e156/Pillow-8.3.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (3.0MB)
     |████████████████████████████████| 3.0MB 7.4MB/s
Collecting matplotlib
  Downloading https://files.pythonhosted.org/packages/24/33/5568d443ba438d95d4db635dd69958056f087e57e1026bee56f959d53f9d/matplotlib-3.4.2-cp37-cp37m-manylinux1_x86_64.whl (10.3MB)
     |████████████████████████████████| 10.3MB 44.2MB/s
Collecting numpy
  Downloading https://files.pythonhosted.org/packages/3f/03/c3526fb4e79a793498829ca570f2f868204ad9a8040afcd72d82a8f121db/numpy-1.21.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7MB)
     |████████████████████████████████| 15.7MB 190kB/s
Requirement already satisfied, skipping upgrade: psutil in /usr/local/lib/python3.7/dist-packages (from glennopt) (5.4.8)
Collecting dataclasses_json
  Downloading https://files.pythonhosted.org/packages/c3/89/390710a3ad24833f2ad11295caddd37f1bc4e60a278de1c9059727a92482/dataclasses_json-0.5.4-py3-none-any.whl
Requirement already satisfied, skipping upgrade: pandas in /usr/local/lib/python3.7/dist-packages (from glennopt) (1.1.5)
Collecting doepy
  Downloading https://files.pythonhosted.org/packages/a3/a7/7c2abcafe7e3142e053a9ebbff1ca9cf78b7f7f39895137f6f421491b335/doepy-0.0.1-py3-none-any.whl
Collecting diversipy
  Downloading https://files.pythonhosted.org/packages/56/9d/b147d06b469051bd90528468c57c7044bc51927a2ee010a8ce35f5df2604/diversipy-0.8.tar.gz
Collecting pydoe
  Downloading https://files.pythonhosted.org/packages/bc/ac/91fe4c039e2744466621343d3b8af4a485193ed0aab53af5b1db03be0989/pyDOE-0.3.8.zip
Requirement already satisfied, skipping upgrade: tqdm in /usr/local/lib/python3.7/dist-packages (from glennopt) (4.41.1)
Requirement already satisfied, skipping upgrade: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (0.10.0)
Requirement already satisfied, skipping upgrade: pyparsing>=2.2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (2.4.7)
Requirement already satisfied, skipping upgrade: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (1.3.1)
Requirement already satisfied, skipping upgrade: python-dateutil>=2.7 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (2.8.1)
Collecting typing-inspect>=0.4.0
  Downloading https://files.pythonhosted.org/packages/ad/fa/77565f652ce57ed56d3a63d537c885a18e4451cf2d56d944991aeb3c82bd/typing_inspect-0.7.1-py3-none-any.whl
Collecting marshmallow-enum<2.0.0,>=1.5.1
  Downloading https://files.pythonhosted.org/packages/c6/59/ef3a3dc499be447098d4a89399beb869f813fee1b5a57d5d79dee2c1bf51/marshmallow_enum-1.5.1-py2.py3-none-any.whl
Collecting marshmallow<4.0.0,>=3.3.0
  Downloading https://files.pythonhosted.org/packages/c2/65/7cd7a1da0f52de85d3ad5b440bcc33b052fb53876b20cce8cad629a58bc6/marshmallow-3.12.2-py2.py3-none-any.whl (47kB)
     |████████████████████████████████| 51kB 6.2MB/s
Collecting stringcase<2.0.0,==1.2.0
  Downloading https://files.pythonhosted.org/packages/f3/1f/1241aa3d66e8dc1612427b17885f5fcd9c9ee3079fc0d28e9a3aeeb36fa3/stringcase-1.2.0.tar.gz
Requirement already satisfied, skipping upgrade: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->glennopt) (2018.9)
Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.7/dist-packages (from diversipy->glennopt) (1.4.1)
Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.7/dist-packages (from cycler>=0.10->matplotlib) (1.15.0)
Collecting mypy-extensions>=0.3.0
  Downloading https://files.pythonhosted.org/packages/5c/eb/975c7c080f3223a5cdaff09612f3a5221e4ba534f7039db34c35d95fa6a5/mypy_extensions-0.4.3-py2.py3-none-any.whl
Requirement already satisfied, skipping upgrade: typing-extensions>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from typing-inspect>=0.4.0->dataclasses_json->glennopt) (3.7.4.3)
Building wheels for collected packages: diversipy, pydoe, stringcase
  Building wheel for diversipy (setup.py) ... done
  Created wheel for diversipy: filename=diversipy-0.8-cp37-none-any.whl size=27531 sha256=88815252f698d124e9d0ffaddd6c9eb843610535d65f77863b3b75e79c2923bc
  Stored in directory: /root/.cache/pip/wheels/49/89/f4/1c64e470a0af08f4aefb99efec615e5da7907435b975ab64a6
  Building wheel for pydoe (setup.py) ... done
  Created wheel for pydoe: filename=pyDOE-0.3.8-cp37-none-any.whl size=18184 sha256=96871d706961ae7ccb1e175632c6f6a33377f5bdb5fe9d4e322baa961d7352d0
  Stored in directory: /root/.cache/pip/wheels/7c/c8/58/a6493bd415e8ba5735082b5e0c096d7c1f2933077a8ce34544
  Building wheel for stringcase (setup.py) ... done
  Created wheel for stringcase: filename=stringcase-1.2.0-cp37-none-any.whl size=3588 sha256=d8f914069ded44e731b08d318ac7e414afee68d330fa35242978c11e7e70733f
  Stored in directory: /root/.cache/pip/wheels/a0/16/a0/16e2c81dbd47503b5a35583dfabde5a93b4cf98dbf0033dad5
Successfully built diversipy pydoe stringcase
ERROR: tensorflow 2.5.0 has requirement numpy~=1.19.2, but you'll have numpy 1.21.0 which is incompatible.
ERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.
ERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.
Installing collected packages: mypy-extensions, typing-inspect, marshmallow, marshmallow-enum, stringcase, dataclasses-json, numpy, diversipy, pydoe, doepy, pillow, matplotlib, glennopt
  Found existing installation: numpy 1.19.5
    Uninstalling numpy-1.19.5:
      Successfully uninstalled numpy-1.19.5
  Found existing installation: Pillow 7.1.2
    Uninstalling Pillow-7.1.2:
      Successfully uninstalled Pillow-7.1.2
  Found existing installation: matplotlib 3.2.2
    Uninstalling matplotlib-3.2.2:
      Successfully uninstalled matplotlib-3.2.2
Successfully installed dataclasses-json-0.5.4 diversipy-0.8 doepy-0.0.1 glennopt-1.2.1 marshmallow-3.12.2 marshmallow-enum-1.5.1 matplotlib-3.4.2 mypy-extensions-0.4.3 numpy-1.21.0 pillow-8.3.1 pydoe-0.3.8 stringcase-1.2.0 typing-inspect-0.7.1

Import relevant libraries_____no_output_____

from glennopt.base import Parameter
from glennopt.optimizers import SODE
from glennopt.helpers import de_mutation_type, mutation_parameters, parallel_settings
from glennopt.DOE import Default,CCD,FullFactorial,LatinHyperCube
import os_____no_output_____# Setting up the optimizer
current_dir = os.getcwd()
pop_size = 20
sode = SODE(eval_command = "python evaluation.py", eval_folder="Evaluation",pop_size=pop_size,optimization_folder=current_dir)

# Selecting the DOE type
# doe = Default(15) # Default
# doe = CCD()
# doe = FullFactorial(levels=8)
doe = LatinHyperCube(128)

doe.add_parameter(name="x1",min_value=-3,max_value=3)
doe.add_parameter(name="x2",min_value=-3,max_value=3)
sode.add_eval_parameters(eval_params=doe.eval_parameters) # add in the parameters to sode

doe.add_objectives(name='objective1')
sode.add_objectives(objectives=doe.objectives)  # add in the objectives into sode

# No performance Parameters
doe.add_perf_parameter(name='p1')
doe.add_perf_parameter(name='p2')
sode.add_performance_parameters(performance_params=doe.perf_parameters) # add performance parameters to sode

# params = mutation_parameters
sode.mutation_params.mutation_type = de_mutation_type.de_rand_1_bin
sode.mutation_params.F = 0.8
sode.mutation_params.C = 0.7_____no_output_____

Enable Parallel Execution (OPTIONAL) _____no_output_____

# Parallel Settings (You don't need to run this block if you only want serial execution)
parallel_settings(concurrent_executions = 8,cores_per_execution = 1, execution_timeout=30) # 8 simulations at the same time, 1 core per simulation, 30 second timeout
sode.parallel_settings = parallel_settings # pass it into the SODE (Single Objective Differential Evolution)_____no_output_____

##Run the Design of Experiments
Design of experiments is used to sample the evaluation space. Say you have 5 variables and f(x[1-5]) = y[1,2] and each x1 through x5 have min and max bounds. The design of experiments is used to evaluate different combinations of x1 to x5 which are used as the starting population (pop_start=-1) _____no_output_____

# Run the Design of Experiments
sode.start_doe(doe.generate_doe())  # This generates individuals that have evaluation parameters randomly distributed between the bounds 100%|██████████| 128/128 [00:00<00:00, 13084.52it/s]

### Viewing the DOE Results in a Plot _____no_output_____To view the results, we plot the rosenbrock function (Optional) and then the scattered points from the DOE evaluated. This shows where the optimizer is sampling the design space_____no_output_____

from Evaluation.rosenbrock import rosenbrock
import numpy as np

# Generate the Rosenbrock
xmin = -3; xmax = 3
ymin = -3; ymax = 3
x = np.linspace(xmin,xmax,50)
y = np.linspace(ymin,ymax,50)
X,Y = np.meshgrid(x,y)
Z = X*0
for i in range(50):
  for j in range(50):
    Z[i,j] = rosenbrock([X[i,j],Y[i,j]] )

# Read in the doe results
individuals = sode.read_calculation_folder()
eval_x = list(); eval_y = list(); eval_z = list()
for ind in individuals[0]:          # Read the DOE individuals
  eval_params = ind.eval_parameters
  eval_x.append(eval_params[0]); eval_y.append(eval_params[1])
  eval_z.append(ind.objectives[0])

# Plotting the function
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter

fig = plt.figure()
ax = fig.gca(projection='3d')
# Rosenbrock function
surf = ax.plot_surface(X, Y, Z, cmap=cm.rainbow,
                       linewidth=0, alpha=0.6, antialiased=True)
# Plot the DOE's evaluation parameters (x,y) and objective value (z)
ax.scatter(eval_x,eval_y,eval_z,color="r",s=50)

fig.colorbar(surf, shrink=0.5, aspect=5)
# rotate the axes and update
ax.view_init(40, 100)/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:32: MatplotlibDeprecationWarning: Calling gca() with keyword arguments was deprecated in Matplotlib 3.4. Starting two minor releases later, gca() will take no keyword arguments. The gca() function should only be used to get the current axes, or if no axes exist, create new axes with default keyword arguments. To create a new axes with non-default arguments, use plt.axes() or plt.subplot().

##Run the Optimization_____no_output_____The code below executes the optimization. `-1` means we are starting from the DOE. If you wanted to start from a different population say population 5 just change `-1` to `5. The execution should take a few minutes._____no_output_____

# Execute the Optimization
sode.optimize_from_population(pop_start=-1,n_generations=50) # -1 means read the doe in and run the simulation100%|██████████| 49/49 [01:52<00:00,  2.30s/it]

#Analyzing and Visualizing the Results_____no_output_____Plotting the best individuals and tracking that vs population. The code below shows how you can use get_best and get_pop_best to find the best individual vs population.

**get_best** shows how the best performing individual changes as the population advances.

**get_pop_best** looks at the individuals inside each population folder and finds the best one.

Below is an example of get_best:_____no_output_____

import matplotlib.pyplot as plt
import matplotlib.cm as cm
import numpy as np
from glennopt.helpers import get_best,get_pop_best

# Plotting get best
individuals = sode.read_calculation_folder()
objectives, pop, _ = get_best(individuals,pop_size=20)
objective_index = 0 # Single objective use 0, if you wanted the second or 3rd then just change to 1 and 2
_, ax = plt.subplots()
ax.scatter(pop, objectives[:,objective_index],color='blue',s=10)
ax.set_xlabel('Population')
ax.set_ylabel('Objective {0} Value'.format(objective_index))
ax.set_yscale('log',base=10) _____no_output_____# Plot Best individual at each population
best_individuals, best_fronts = get_pop_best(individuals)

nobjectives = len(best_individuals[0][0].objectives)
objective_data = list()
for pop,best_individual in best_individuals.items():
    objective_data.append(best_individual[objective_index].objectives[objective_index])

_,ax = plt.subplots()
colors = cm.rainbow(np.linspace(0, 1, len(best_individuals.keys())))
ax.scatter(list(best_individuals.keys()), objective_data, color='blue',s=10)
ax.set_xlabel('Population')
ax.set_ylabel('Objective {0} Value'.format(objective_index))
ax.set_title('Best individual at each population')
ax.set_yscale('log',base=10)
_____no_output_____

## Pareto Front_____no_output_____Because this is a single objective optimization. There really isn't a pareto front. _____no_output_____