# Notebook from hnoamany/course-content
Path: tutorials/W1D5_DimensionalityReduction/student/W1D5_Tutorial1.ipynb

_____no_output_____

# Mount Google Drive
from google.colab import drive # import drive from google colab

ROOT = "/content/drive"     # default location for the drive
print(ROOT)                 # print content of ROOT (Optional)

drive.mount(ROOT,force_remount=True)      /content/drive
Mounted at /content/drive

# Neuromatch Academy: Week 1, Day 5, Tutorial 1
# Dimensionality Reduction: Geometric view of data

__Content creators:__ Alex Cayco Gajic, John Murray

__Content reviewers:__ Roozbeh Farhoudi, Matt Krause, Spiros Chavlis, Richard Gao, Michael Waskom
_____no_output_____---
# Tutorial Objectives

In this notebook we'll explore how multivariate data can be represented in different orthonormal bases. This will help us build intuition that will be helpful in understanding PCA in the following tutorial.

Overview:
 - Generate correlated multivariate data.
 - Define an arbitrary orthonormal basis.
 - Project the data onto the new basis._____no_output_____

# @title Video 1: Geometric view of data
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="THu9yHnpq9I", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
videoVideo available at https://youtube.com/watch?v=THu9yHnpq9I

---
# Setup_____no_output_____

# Import
import numpy as np
import matplotlib.pyplot as plt_____no_output_____# @title Figure Settings
import ipywidgets as widgets  # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")_____no_output_____# @title Helper functions

def get_data(cov_matrix):
  """
  Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian.

  Note that samples are sorted in ascending order for the first random variable

  Args:
    cov_matrix (numpy array of floats): desired covariance matrix

  Returns:
    (numpy array of floats) : samples from the bivariate Gaussian, with each
                              column corresponding to a different random
                              variable
  """

  mean = np.array([0, 0])
  X = np.random.multivariate_normal(mean, cov_matrix, size=1000)
  indices_for_sorting = np.argsort(X[:, 0])
  X = X[indices_for_sorting, :]

  return X

def plot_data(X):
  """
  Plots bivariate data. Includes a plot of each random variable, and a scatter
  plot of their joint activity. The title indicates the sample correlation
  calculated from the data.

  Args:
    X (numpy array of floats) :   Data matrix each column corresponds to a
                                  different random variable

  Returns:
    Nothing.
  """

  fig = plt.figure(figsize=[8, 4])
  gs = fig.add_gridspec(2, 2)
  ax1 = fig.add_subplot(gs[0, 0])
  ax1.plot(X[:, 0], color='k')
  plt.ylabel('Neuron 1')
  plt.title('Sample var 1: {:.1f}'.format(np.var(X[:, 0])))
  ax1.set_xticklabels([])
  ax2 = fig.add_subplot(gs[1, 0])
  ax2.plot(X[:, 1], color='k')
  plt.xlabel('Sample Number')
  plt.ylabel('Neuron 2')
  plt.title('Sample var 2: {:.1f}'.format(np.var(X[:, 1])))
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(X[:, 0], X[:, 1], '.', markerfacecolor=[.5, .5, .5],
           markeredgewidth=0)
  ax3.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(X[:, 0], X[:, 1])[0, 1]))
  plt.show()

def plot_basis_vectors(X, W):
  """
  Plots bivariate data as well as new basis vectors.

  Args:
    X (numpy array of floats) :   Data matrix each column corresponds to a
                                  different random variable
    W (numpy array of floats) :   Square matrix representing new orthonormal
                                  basis each column represents a basis vector

  Returns:
    Nothing.
  """

  plt.figure(figsize=[4, 4])
  plt.plot(X[:, 0], X[:, 1], '.', color=[.5, .5, .5], label='Data')
  plt.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.plot([0, W[0, 0]], [0, W[1, 0]], color='r', linewidth=3,
           label='Basis vector 1')
  plt.plot([0, W[0, 1]], [0, W[1, 1]], color='b', linewidth=3,
           label='Basis vector 2')
  plt.legend()
  plt.show()

def plot_data_new_basis(Y):
  """
  Plots bivariate data after transformation to new bases.
  Similar to plot_data but with colors corresponding to projections onto
  basis 1 (red) and basis 2 (blue). The title indicates the sample correlation
  calculated from the data.

  Note that samples are re-sorted in ascending order for the first
  random variable.

  Args:
    Y (numpy array of floats):   Data matrix in new basis each column
                                 corresponds to a different random variable

  Returns:
    Nothing.
  """
  fig = plt.figure(figsize=[8, 4])
  gs = fig.add_gridspec(2, 2)
  ax1 = fig.add_subplot(gs[0, 0])
  ax1.plot(Y[:, 0], 'r')
  plt.xlabel
  plt.ylabel('Projection \n basis vector 1')
  plt.title('Sample var 1: {:.1f}'.format(np.var(Y[:, 0])))
  ax1.set_xticklabels([])
  ax2 = fig.add_subplot(gs[1, 0])
  ax2.plot(Y[:, 1], 'b')
  plt.xlabel('Sample number')
  plt.ylabel('Projection \n basis vector 2')
  plt.title('Sample var 2: {:.1f}'.format(np.var(Y[:, 1])))
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(Y[:, 0], Y[:, 1], '.', color=[.5, .5, .5])
  ax3.axis('equal')
  plt.xlabel('Projection basis vector 1')
  plt.ylabel('Projection basis vector 2')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(Y[:, 0], Y[:, 1])[0, 1]))
  plt.show()_____no_output_____

---
# Section 1: Generate correlated multivariate data_____no_output_____

# @title Video 2: Multivariate data
video = YouTubeVideo(id="jcTq2PgU5Vw", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
videoVideo available at https://youtube.com/watch?v=jcTq2PgU5Vw

To gain intuition, we will first use a simple model to generate multivariate data. Specifically, we will draw random samples from a *bivariate normal distribution*. This is an extension of the one-dimensional normal distribution to two dimensions, in which each $x_i$ is marginally normal with mean $\mu_i$ and variance $\sigma_i^2$:

\begin{align}
x_i \sim \mathcal{N}(\mu_i,\sigma_i^2).
\end{align}

Additionally, the joint distribution for $x_1$ and $x_2$ has a specified correlation coefficient $\rho$. Recall that the correlation coefficient is a normalized version of the covariance, and ranges between -1 and +1:

\begin{align}
\rho = \frac{\text{cov}(x_1,x_2)}{\sqrt{\sigma_1^2 \sigma_2^2}}.
\end{align}

For simplicity, we will assume that the mean of each variable has already been subtracted, so that $\mu_i=0$. The remaining parameters can be summarized in the covariance matrix, which for two dimensions has the following form:

\begin{equation*}
{\bf \Sigma} =
\begin{pmatrix}
 \text{var}(x_1) & \text{cov}(x_1,x_2) \\
 \text{cov}(x_1,x_2) &\text{var}(x_2)
\end{pmatrix}.
\end{equation*}

In general, $\bf \Sigma$ is a symmetric matrix with the variances $\text{var}(x_i) = \sigma_i^2$ on the diagonal, and the covariances on the off-diagonal. Later, we will see that the covariance matrix plays a key role in PCA.

_____no_output_____
## Exercise 1: Draw samples from a distribution

We have provided code to draw random samples from a zero-mean bivariate normal distribution. Throughout this tutorial, we'll imagine these samples represent the activity (firing rates) of two recorded neurons on different trials. Fill in the function below to calculate the covariance matrix given the desired variances and correlation coefficient. The covariance can be found by rearranging the equation above:

\begin{align}
\text{cov}(x_1,x_2) = \rho \sqrt{\sigma_1^2 \sigma_2^2}.
\end{align}

Use these functions to generate and plot data while varying the parameters. You should get a feel for how changing the correlation coefficient affects the geometry of the simulated data.

**Steps**
* Fill in the function `calculate_cov_matrix` to calculate the desired covariance.
* Generate and plot the data for $\sigma_1^2 =1$, $\sigma_1^2 =1$, and $\rho = .8$. Try plotting the data for different values of the correlation coefficent: $\rho = -1, -.5, 0, .5, 1$._____no_output_____

help(plot_data)
help(get_data)Help on function plot_data in module __main__:

plot_data(X)
    Plots bivariate data. Includes a plot of each random variable, and a scatter
    plot of their joint activity. The title indicates the sample correlation
    calculated from the data.

    Args:
      X (numpy array of floats) :   Data matrix each column corresponds to a
                                    different random variable

    Returns:
      Nothing.

Help on function get_data in module __main__:

get_data(cov_matrix)
    Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian.

    Note that samples are sorted in ascending order for the first random variable

    Args:
      cov_matrix (numpy array of floats): desired covariance matrix

    Returns:
      (numpy array of floats) : samples from the bivariate Gaussian, with each
                                column corresponding to a different random
                                variable

def calculate_cov_matrix(var_1, var_2, corr_coef):
  """
  Calculates the covariance matrix based on the variances and correlation
  coefficient.

  Args:
    var_1 (scalar)          : variance of the first random variable
    var_2 (scalar)          : variance of the second random variable
    corr_coef (scalar)      : correlation coefficient

  Returns:
    (numpy array of floats) : covariance matrix
  """

  #################################################
  ## TODO for students: calculate the covariance matrix
  # Fill out function and remove
  #raise NotImplementedError("Student excercise: calculate the covariance matrix!")
  #################################################

  # Calculate the covariance from the variances and correlation
  cov = corr_coef*np.sqrt([var_1*var_2])

  cov_matrix = np.array([[var_1, cov], [cov, var_2]])

  return cov_matrix

###################################################################
## TO DO for students: generate and plot bivariate Gaussian data with variances of 1
## and a correlation coefficients of: 0.8
## repeat while varying the correlation coefficient from -1 to 1
###################################################################
np.random.seed(2020)  # set random seed
variance_1 = 1
variance_2 = 1
corr_coef = 0.8

# Uncomment to test your code and plot
cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)
X = get_data(cov_matrix)
plot_data(X)_____no_output_____

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial1_Solution_57497711.py)

*Example output:*

_____no_output_____---
# Section 2: Define a new orthonormal basis
_____no_output_____

# @title Video 3: Orthonormal bases
video = YouTubeVideo(id="PC1RZELnrIg", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
videoVideo available at https://youtube.com/watch?v=PC1RZELnrIg

Next, we will define a new orthonormal basis of vectors ${\bf u} = [u_1,u_2]$ and ${\bf w} = [w_1,w_2]$. As we learned in the video, two vectors are orthonormal if:

1.   They are orthogonal (i.e., their dot product is zero):
\begin{equation}
{\bf u\cdot w} = u_1 w_1 + u_2 w_2 = 0
\end{equation}
2.   They have unit length:
\begin{equation}
||{\bf u} || = ||{\bf w} || = 1
\end{equation}

In two dimensions, it is easy to make an arbitrary orthonormal basis. All we need is a random vector ${\bf u}$, which we have normalized. If we now define the second basis vector to be ${\bf w} = [-u_2,u_1]$, we can check that both conditions are satisfied:
\begin{equation}
{\bf u\cdot w} = - u_1 u_2 + u_2 u_1 = 0
\end{equation}
and
\begin{equation}
{|| {\bf w} ||} = \sqrt{(-u_2)^2 + u_1^2} = \sqrt{u_1^2 + u_2^2} = 1,
\end{equation}
where we used the fact that ${\bf u}$ is normalized. So, with an arbitrary input vector, we can define an orthonormal basis, which we will write in matrix by stacking the basis vectors horizontally:

\begin{equation}
{{\bf W} } =
\begin{pmatrix}
 u_1 & w_1 \\
 u_2 & w_2
\end{pmatrix}.
\end{equation}

 _____no_output_____## Exercise 2: Find an orthonormal basis

In this exercise you will fill in the function below to define an orthonormal basis, given a single arbitrary 2-dimensional vector as an input.

**Steps**
* Modify the function `define_orthonormal_basis` to first normalize the first basis vector $\bf u$.
* Then complete the function by finding a basis vector $\bf w$ that is orthogonal to $\bf u$.
* Test the function using initial basis vector ${\bf u} = [3,1]$. Plot the resulting basis vectors on top of the data scatter plot using the function `plot_basis_vectors`. (For the data, use  $\sigma_1^2 =1$, $\sigma_2^2 =1$, and $\rho = .8$)._____no_output_____

help(plot_basis_vectors)Help on function plot_basis_vectors in module __main__:

plot_basis_vectors(X, W)
    Plots bivariate data as well as new basis vectors.

    Args:
      X (numpy array of floats) :   Data matrix each column corresponds to a
                                    different random variable
      W (numpy array of floats) :   Square matrix representing new orthonormal
                                    basis each column represents a basis vector

    Returns:
      Nothing.

a=[1,2]
b = np.flip(a)
b[0] = -b[0]
print(b)[-2  1]
def define_orthonormal_basis(u):
  """
  Calculates an orthonormal basis given an arbitrary vector u.

  Args:
    u (numpy array of floats) : arbitrary 2-dimensional vector used for new
                                basis

  Returns:
    (numpy array of floats)   : new orthonormal basis
                                columns correspond to basis vectors
  """

  #################################################
  ## TODO for students: calculate the orthonormal basis
  # Fill out function and remove
  #raise NotImplementedError("Student excercise: implement the orthonormal basis function")
  #################################################

  # normalize vector u
  u = u/np.sqrt(np.transpose(u)@u)
  # calculate vector w that is orthogonal to w
  w = [-u[1],u[0]]

  W = np.column_stack([u, w])
  print(W)

  return W

np.random.seed(2020)  # set random seed
variance_1 = 1
variance_2 = 1
corr_coef = 0.8

cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)
X = get_data(cov_matrix)
u = np.array([3, 1])

# Uncomment and run below to plot the basis vectors
W = define_orthonormal_basis(u)
plot_basis_vectors(X, W)[[ 0.9486833  -0.31622777]
 [ 0.31622777  0.9486833 ]]

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial1_Solution_7a9640ef.py)

*Example output:*

_____no_output_____---
# Section 3: Project data onto new basis_____no_output_____

# @title Video 4: Change of basis
video = YouTubeVideo(id="Mj6BRQPKKUc", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
videoVideo available at https://youtube.com/watch?v=Mj6BRQPKKUc

Finally, we will express our data in the new basis that we have just found. Since $\bf W$ is orthonormal, we can project the data into our new basis using simple matrix multiplication :

\begin{equation}
{\bf Y = X W}.
\end{equation}

We will explore the geometry of the transformed data $\bf Y$ as we vary the choice of basis._____no_output_____## Exercise 3: Define an orthonormal basis
In this exercise you will fill in the function below to define an orthonormal basis, given a single arbitrary vector as an input.

**Steps**
* Complete the function `change_of_basis` to project the data onto the new basis.
* Plot the projected data using the function `plot_data_new_basis`.
* What happens to the correlation coefficient in the new basis? Does it increase or decrease?
* What happens to variance?

_____no_output_____

def change_of_basis(X, W):
  """
  Projects data onto new basis W.

  Args:
    X (numpy array of floats) : Data matrix each column corresponding to a
                                different random variable
    W (numpy array of floats) : new orthonormal basis columns correspond to
                                basis vectors

  Returns:
    (numpy array of floats)    : Data matrix expressed in new basis
  """

  #################################################
  ## TODO for students: project the data onto o new basis W
  # Fill out function and remove
  #raise NotImplementedError("Student excercise: implement change of basis")
  #################################################

  # project data onto new basis described by W
  Y = X@W

  return Y

# Unomment below to transform the data by projecting it into the new basis
Y = change_of_basis(X, W)
plot_data_new_basis(Y)_____no_output_____

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial1_Solution_a1124bbc.py)

*Example output:*

_____no_output_____## Interactive Demo: Play with the basis vectors
To see what happens to the correlation as we change the basis vectors, run the cell below. The parameter $\theta$ controls the angle of $\bf u$ in degrees. Use the slider to rotate the basis vectors. _____no_output_____

# @title

# @markdown Make sure you execute this cell to enable the widget!

def refresh(theta=0):
  u = [1, np.tan(theta * np.pi / 180)]
  W = define_orthonormal_basis(u)
  Y = change_of_basis(X, W)
  plot_basis_vectors(X, W)
  plot_data_new_basis(Y)

_ = widgets.interact(refresh, theta=(0, 90, 5))_____no_output_____

## Questions

* What happens to the projected data as you rotate the basis?
* How does the correlation coefficient change? How does the variance of the projection onto each basis vector change?
* Are you able to find a basis in which the projected data is **uncorrelated**?_____no_output_____---
# Summary

- In this tutorial, we learned that multivariate data can be visualized as a cloud of points in a high-dimensional vector space. The geometry of this cloud is shaped by the covariance matrix.

- Multivariate data can be represented in a new orthonormal basis using the dot product. These new basis vectors correspond to specific mixtures of the original variables - for example, in neuroscience, they could represent different ratios of activation  across a population of neurons.

- The projected data (after transforming into the new basis) will generally have a different geometry from the original data. In particular, taking basis vectors that are aligned with the spread of cloud of points decorrelates the data.

* These concepts - covariance, projections, and orthonormal bases - are key for understanding PCA, which we be our focus in the next tutorial._____no_output_____


# Notebook from AdeboyeML/lincs-profiling-comparison
Path: 1.Data-exploration/Profiles_level4/L1000/L1000_calculate_cpd_median_score.ipynb

#### Level-4 data of L1000 gene-expression profiling is calculated as:

Level 4 - Differential Expression (ZSPC):

To obtain a measure of relative gene expression, we use a robust z-scoring procedure to generate differential expression values from normalized profiles (Level-3). We compute the **differential expression of gene x in the ith sample(xi) on the plate (zi)** as

                              zi = xi - median(X) / 1.4826 * MAD(X)

***where X is the vector of normalized gene expression of gene x across all samples on the plate, MAD is the median absolute deviation of X, and the factor of 1.4826 makes the denominator a consistent estimator of scale for normally distributed data.***

[Subramanian et al 2017](https://www.cell.com/action/showPdf?pii=S0092-8674%2817%2931309-0)

- Level-4 data - are replicate level data i.e. where you have multiple profiles been perturbed by the same compound (pertubagen)

#### The goal here:

-- is to determine the median score of each compound per dose based on taking the median of the correlation values between replicates of the same compound._____no_output_____

import os
import requests
import pickle
import argparse
import pandas as pd
import numpy as np
import re
from os import walk
from collections import Counter
import random
import shutil
from statistics import median
import cmapPy.pandasGEXpress.parse_gct as pg
from cmapPy.pandasGEXpress.parse import parse
from io import BytesIO
from urllib.request import urlopen
from zipfile import ZipFile

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
np.warnings.filterwarnings('ignore', category=np.VisibleDeprecationWarning)_____no_output_____data_dir = 'D:\Documents\L1000' #os.getcwd() ##current dir
zipurl = "https://ndownloader.figshare.com/articles/13181966/versions/1"
def download_L1000_data(data_dir, zipurl):
    """
    Download L1000 data from figshare and extract
    the zipped files into a directory
    """
    if not os.path.exists(data_dir):
        os.mkdir(data_dir)

    with urlopen(zipurl) as zipresp:
        with ZipFile(BytesIO(zipresp.read())) as zfile:
            zfile.extractall(data_dir)_____no_output_____#download_L1000_data(data_dir, zipurl)_____no_output_____os.listdir(data_dir) ##files in L1000 downloaded dataset_____no_output_____pertinfo_file = '../aligned_moa_CP_L1000.csv'_____no_output_____

Two level-4 L1000 Data:

- 'level_4W_zspc_n27837x978.gctx'
- 'level_4_zspc_n27837x978.gctx',_____no_output_____

def construct_lvl4_data(data_dir, level4_dir, pertinfo_file):
    """
    This function returns L1000 Level-4 Data that is aligned with
    the important metadata information to compute compound's median scores
    """

    lvl4_data = parse(os.path.join(data_dir, level4_dir))
    lvl4_data = lvl4_data.data_df.rename_axis(None).T
    lvl4_data = lvl4_data.rename_axis(None).reset_index()
    lvl4_data.rename(columns={"index": "replicate_id"}, inplace = True)
    df_metalvl_5 = pd.read_csv(os.path.join(data_dir, 'col_meta_level_5_REP.A_A549_only_n9482.txt'), delimiter = "\t")
    lvl4_data['sig_id'] = lvl4_data['replicate_id'].apply(lambda x: ''.join(re.sub('(24H.+(\_|\.)[A-Z0-9]+.)\:', '24H:', x)))
    df_meta_features = df_metalvl_5[['sig_id', 'pert_id', 'pert_idose', 'det_plate', 'det_well']].copy()
    df_meta_features['dose'] = df_meta_features['pert_idose'].map({'-666' : 0, '0.04 uM' : 1, '0.12 uM' : 2, '0.37 uM' : 3,
                                                                   '1.11 uM' : 4, '3.33 uM' : 5, '10 uM' : 6, '20 uM' : 7})
    df_pertinfo = pd.read_csv(pertinfo_file)
    df_pertinfo.rename(columns={"broad_id": "pert_id"}, inplace = True)
    df_meta_features = pd.merge(df_meta_features, df_pertinfo, on=['pert_id'], how = 'left')
    lvl4_data = pd.merge(lvl4_data, df_meta_features, on='sig_id')

    return lvl4_data_____no_output_____df_level4 = construct_lvl4_data(data_dir, 'level_4_zspc_n27837x978.gctx', pertinfo_file)_____no_output_____df_level4.head()_____no_output_____df_level4.shape_____no_output_____len(df_level4['det_plate'].unique())_____no_output_____len(df_level4['det_well'].unique())_____no_output_____

### - Remove highly correlated landmark genes and samples with Null compound values_____no_output_____

def feature_selection(df_data):

    """
    Perform feature selection by dropping columns with null compounds values,
    and highly correlated landmark genes from the data.
    """

    df_data_genes = df_data.drop(['replicate_id', 'Metadata_broad_sample', 'pert_id', 'dose', 'pert_idose',
                                  'pert_iname', 'moa', 'sig_id', 'det_plate', 'det_well'], axis = 1).copy()
    df_data_corr = df_data_genes.corr(method = 'spearman')
    drop_cols = []
    n_cols = len(df_data_corr.columns)
    for i in range(n_cols):
        for k in range(i+1, n_cols):
            val = df_data_corr.iloc[k, i]
            col = df_data_corr.columns[i]
            if abs(val) >= 0.9:
                drop_cols.append(col)
    df_data.drop(set(drop_cols), axis = 1, inplace = True)
    df_data.drop(df_data[df_data['pert_iname'].isnull()].index).reset_index(drop = True, inplace = True)

    return df_data_____no_output_____df_level4 = feature_selection(df_level4)_____no_output_____df_level4.shape_____no_output_____def get_median_score(cpds_list, df):
    """
    This function calculates the median score for each compound based on its replicates
    """

    cpds_median_score = {}
    for cpd in cpds_list:
        cpd_replicates = df[df['pert_iname'] == cpd].copy()
        cpd_replicates.drop(['replicate_id', 'Metadata_broad_sample', 'pert_id', 'dose', 'pert_idose',
                             'pert_iname', 'moa', 'sig_id', 'det_plate', 'det_well'], axis = 1, inplace = True)
        cpd_replicates_corr = cpd_replicates.astype('float64').T.corr(method = 'spearman').values
        if len(cpd_replicates_corr) == 1:
            median_val = 1
        else:
            median_val = median(list(cpd_replicates_corr[np.triu_indices(len(cpd_replicates_corr), k = 1)]))

        cpds_median_score[cpd] = median_val

    return cpds_median_score_____no_output_____def check_compounds(cpd_med_score, df):
    """
    Check if all distinct compounds in the Level-4 dataframe are present
    in the cpd_med_score dictionary, if not add the compounds as keys to the dictionary
    and give them a null value.
    """
    cpd_list = df['pert_iname'].unique().tolist()
    cpd_keys = cpd_med_score.keys()
    for cpd in cpd_list:
        if cpd not in cpd_keys:
            cpd_med_score[cpd] = np.nan

    return cpd_med_score_____no_output_____def get_cpd_medianscores(df):

    """This function computes median scores for all compounds found in the Level-4 dataframe PER DOSE (1-6)"""

    dose_list = list(set(df['dose'].unique().tolist()))[1:7]

    for dose in dose_list:
        df_dose = df[df['dose'] == dose].copy()
        cpds_list = df_dose['pert_iname'].unique().tolist()
        cpds_median_score = get_median_score(cpds_list, df_dose)
        cpds_median_score = check_compounds(cpds_median_score, df)
        sorted_med_score = {key:value for key, value in sorted(cpds_median_score.items(), key=lambda item: item[0])}
        if dose == 1:
            df_cpd_med_score = pd.DataFrame.from_dict(sorted_med_score, orient='index', columns = ['dose_1'])
        else:
            df_cpd_med_score['dose_' + str(dose)] = sorted_med_score.values()

    return df_cpd_med_score_____no_output_____df_cpd_med_score = get_cpd_medianscores(df_level4)_____no_output_____df_cpd_med_score.head(10)_____no_output_____def drop_cpds_with_null(df):
    """
    This function drop compounds with median scores of 1
    or null values in any of the dose points (1-6)
    """
    cpds_with_null = []
    for cpd in df.index:
        if any(df.loc[cpd] == 1) | any(df.loc[cpd].isnull()):
            cpds_with_null.append(cpd)
    df.drop(cpds_with_null, axis = 0, inplace = True)

    return df_____no_output_____df_cpd_med_score = drop_cpds_with_null(df_cpd_med_score)_____no_output_____df_cpd_med_score.shape_____no_output_____df_cpd_med_score.head(10)_____no_output_____def no_of_replicates_per_cpd(df, df_lvl4):
    """This function computes the numbers of replicates for each compound (cpd_size)"""

    dose_list = list(set(df_lvl4['dose'].unique().tolist()))[1:7]
    cpds_no_of_reps = {}
    for cpd in df.index:
        num_of_reps = 0
        df_cpd = df_lvl4[df_lvl4['pert_iname'] == cpd].copy()
        for dose in dose_list:
            df_dose = df_cpd[df_cpd['dose'] == dose].copy()
            num_of_reps += df_dose.shape[0]
        cpds_no_of_reps[cpd] = num_of_reps // len(dose_list)
    df['no_of_replicates'] = cpds_no_of_reps.values()
    return df_____no_output_____df_cpd_med_score = no_of_replicates_per_cpd(df_cpd_med_score, df_level4)_____no_output_____df_cpd_med_score.head(10)_____no_output_____def save_to_csv(df, path, file_name, compress=None):
    """saves dataframes to csv"""

    if not os.path.exists(path):
        os.mkdir(path)

    df.to_csv(os.path.join(path, file_name), index=False, compression=compress)_____no_output_____save_to_csv(df_cpd_med_score.reset_index().rename({'index':'cpd'}, axis = 1),
            'L1000_lvl4_cpd_replicate_datasets', 'cpd_replicate_median_scores.csv')_____no_output_____save_to_csv(df_level4, 'L1000_lvl4_cpd_replicate_datasets',
            'L1000_level4_cpd_replicates.csv.gz', compress="gzip")_____no_output_____


# Notebook from CLima86/Physics_5300_CDL
Path: 2020_week_3/Logistic_map_explorations.ipynb

# Logistic map explorations (Taylor and beyond)

Adapted by Dick Furnstahl from the Ipython Cookbook by Cyrille Rossant.
Lyapunov plot modified on 29-Jan-2019 after discussion with Michael Heinz.

Here we consider the *logistic map*, which illustrates how chaos can arise from a simple nonlinear equation.  The logistic map models the evolution of a population including reproduction and mortality (see Taylor section 12.9).

The defining function for the logistic map is

$ f_r(x) = r x (1-x) \;, $

where $r$ is the control parameter analogous to $\gamma$ for the damped, driven pendulum.  To study the pendulum, we looked at $\phi(t)$ at $t_0$, $t_0 + \tau$, $t_0 + 2\tau$, and so on, where $\tau$ is the driving frequency and we found $\phi(t_0 + \tau)$ from $\phi(t_0)$ by solving the differential equation for the pendulum.  After transients had died off, we characterized the value of $\gamma$ being considered by the periodicity of the $\phi(t_n)$ values: period one (all the same), period two (alternating), chaotic (no repeats) and so on.

Here instead of a differential equation telling us how to generate a trajectory, we use $f_r(x)$:

$\begin{align}
  x_1 = f_r(x_0) \ \rightarrow\ x_2 = f_r(x_1) \ \rightarrow\  x_3 = f_r(x_2) \ \rightarrow\  \ldots
\end{align}$

There will be transients at the beginning, but this sequence of $x_i$ values may reach a point $x = x^*$ such that $f(x^*) = x^*$. We call this a *fixed point*.  If instead it ends up bouncing between two values of $x$, it is period two and we call it a two-cycle.  We can have a cascade of period doubling, as found for the damped, driven pendulum, leading to chaos, which is characterized by the mapping never repeating.  We can make a corresponding bifurcation diagram and identify Lyapunov exponents for each initial condition and $r$._____no_output_____To adapt this notebook to a different map (such as the sine map), you will need to:
* Change the map function and its derivative.
* Change the range of the $r$ parameter to an appropriate range for the new map.
* Possibly change the initial $x$ value ($x_0$).
* Modify the limits of the plots appropriately._____no_output_____

%matplotlib inline

# standard numpy and matplotlib imports
import numpy as np
import matplotlib.pyplot as plt
_____no_output_____def logistic(r, x):
    """Logistic map function: f(x) = r x(1-x)
    """
    return r * x * (1.-x)

def logistic_deriv(r, x):
    """Logistic map derivative: f'(x) = r(1-2x)
    """
    return r * (1. - 2.*x)_____no_output_____

## Explore the logistic map and its fixed points

Start with a simple plot of the logistic function.  **What does changing $r$ do?**_____no_output_____

x, step = np.linspace(0,1, num=101, endpoint=True, retstep=True)
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

r_param = 2
ax.plot(x, logistic(r_param, x), 'k-')
ax.set_xlabel('x')
ax.set_ylabel('f(x)')

fig.tight_layout()_____no_output_____

Make plots showing the approach to fixed points.

**Play with the number of steps taken and the values of $r$.**
1. **Increase the number of steps in the plots to more cleanly identify the long-time trend for that value of $r$.**
2. **Try smaller values of $r$.  Are there always non-zero fixed points for small $r$?**
_____no_output_____

def plot_system(r, x0, n, ax=None):
    """Plot the function and the y=x diagonal line."""
    t = np.linspace(0,1, num=101)
    ax.plot(t, logistic(r,t), 'k', lw=2)  # black, linewidth 2
    ax.plot([0,1], [0,1], 'k', lw=2)  # x is an array of 0 and 1,
                                      # y is the same array, so this plots
                                      #  a straight line from (0,0) to (1,1).

    # Recursively apply y=f(x) and plot two additional straight lines:
    #   line from (x, x) to (x, y)
    #   line from (x, y) to (y, y)
    x = x0
    for i in range(n):   # do n iterations, i = 0, 1, ..., n-1
        y = logistic(r, x)
        # Plot the two lines
        ax.plot([x,x], [x,y], color='blue', lw=1)
        ax.plot([x,y], [y,y], color='blue', lw=1)
        # Plot the positions with increasing opacity of the circles
        ax.plot([x], [y], 'or', ms=10, alpha=(i+1)/n)
        x = y  # recursive: reset x to y for the next iteration
    ax.set_xlim(0,1)
    ax.set_ylim(0,1)
    ax.set_title(f'$r={r:.1f}$, $x_0={x0:.1f}$')

fig = plt.figure(figsize=(12,12))

ax1 = fig.add_subplot(2,2,1)
# start at 0.1, r parameter is 2.5, take n steps
plot_system(r=0.7, x0=0.9, n=100, ax=ax1)

ax2 = fig.add_subplot(2,2,2)
# start at 0.1, r parameter is 3.2, take n steps
plot_system(r=3.2, x0=0.9, n=100, ax=ax2)

ax3 = fig.add_subplot(2,2,3)
# start at 0.1, r parameter is 2.6, take n steps
plot_system(r=3.5, x0=0.1, n=10, ax=ax3)

ax4 = fig.add_subplot(2,2,4)
# start at 0.1, r parameter is 3.7, take n steps
plot_system(r=3.7, x0=0.1, n=10, ax=ax4)_____no_output_____

**What periods do these exhibit?  Are any chaotic?** _____no_output_____## Find the unstable point numerically_____no_output_____To find the value of $r$ at which the second fixed point becomes unstable, we must solve simultaneously the equations (see Taylor 12.9 in the "A Test for Stability" subsection):

$\begin{align}
  &f_r(x^*) = x^* \\
  &f_r'(x^*) = -1
\end{align}$

where $f'$ means the derivative with respect to $x$, or

$\begin{align}
  &f_r(x^*) - x^* = 0 \\
  &f_r'(x^*) + 1 = 0
\end{align}$

It is the latter form of the equations that we will pass to the
scipy function `fsolve` to find the values of $r$ and $x*$._____no_output_____

from scipy.optimize import fsolve  # Google this to learn more!

def equations(passed_args):
    """Return the two expressions that must equal to zero."""
    r, x = passed_args
    return (logistic(r, x) - x, logistic_deriv(r, x) + 1)

# Call fsolve with initial values for r and x
r, x = fsolve(equations, (0.5, 0.5))

print_string = f'x* = {x:.10f},  r = {r:.10f}'
print(print_string)_____no_output_____

**Verify analytically that these are the correct values for the logistic map.**  We use this same procedure for other maps where only numerical solutions are known. _____no_output_____## Now make the bifurcation diagram and calculate Lyapunov exponents_____no_output_____To make a bifurcation diagram, we:
* define an array of closely spaced $r$ values;
* for each $r$, iterate the map $x_{i+1} = f_r(x_i)$ a large number of times (e.g., 1000) to ensure convergence;
* plot the last 100 or so iterations as points at each $r$._____no_output_____To think about how to calculate a Lyapunov exponent, consider two iterations of the map starting from two values of $x$ that are close together.  Call these initial values $x_0$ and $x_0 + \delta x_0$.  These are mapped by $f_r$ to $x_1$ and  $x_1 + \delta x_1$, then $x_2$ and $x_2 + \delta x_2$, up to $x_n + \delta x_n$.

How are the $\delta x_i$ related?  Each $\delta x_i$ is small, so use a Taylor expansion!
Expanding $f(x)$ about $x_n$:

$\begin{align}
  f(x_i &+ \delta x_i) = &f(x_i) + \delta x_i f'(x_i) + \frac{1}{2}(\delta x_i)^2 f''(x_i) + \cdots \\
\end{align}$

so, neglecting terms of order $(\delta x_i)^2$ and higher,

$\begin{align}
    x_{i+1} + \delta x_{i+1} \approx x_{i+1} + \delta x_i f'(x_i)
    \quad\mbox{or}\quad  \delta x_{i+1} \approx f'(x_i)\, \delta x_i \;.
\end{align}$

Iterating this result we get

$\begin{align}
  \delta x_1 &= f'(x_0)\, \delta x_0  \\
  \delta x_2 &= f'(x_1)\, \delta x_1 = f'(x_1)\times f'(x_0)\, \delta x_0  \\
  & \qquad\vdots  \\
  |\delta x_n| &= \left(\prod_{i=0}^{n-1} |f'(x_i)|\right) |\delta x_0| \;,
\end{align}$

where the last equation gives us the separation of two trajectories after $n$ steps $\delta x_n$ in terms of their
initial separation, $\delta x_0$.  (Why can we just take the absolute value of all the terms?)

We expect that this will vary exponentially at large $n$ like

$\begin{align}
  \left| \frac{\delta x_n}{\delta x_0} \right| = e^{\lambda n}
\end{align}$

and so we define the Lypunov exponent $\lambda$ by

$\begin{align}
  \lambda = \lim_{n\rightarrow\infty} \frac{1}{n} \sum_{i=1}^{n} \ln |f'(x_i)| \;,
\end{align}$

which in practice is well approximated by the sum for large $n$.

If $\lambda > 0$, then nearby trajectories diverge from each other exponentially at large $n$, which corresponds to chaos. However, if the
trajectories converge to a fixed point or a limit cycle, they will get closer together with increasing $n$, which corresponds to $\lambda<0$._____no_output_____Ok, let's do it!_____no_output_____

n = 10000
# Here we'll use n values of r linearly spaced between 2.8 and 4.0
###  You may want to change the range of r for other maps
r = np.linspace(2.8, 4.0, n)

iterations = 1000   # iterations of logistic map; keep last iterations
last = 100          #  where results should have stabilized

x = 0.1 * np.ones(n)  # x_0 initial condition
                      ### (you may want to change for other maps)

lyapunov = np.zeros(n) # initialize vector used for the Lyapunov sums_____no_output_____fig = plt.figure(figsize=(8,9))
ax1 = fig.add_subplot(2,1,1)   # bifurcation diagram
ax2 = fig.add_subplot(2,1,2)   # Lyapunov exponent

# Display the bifurcation diagram with one pixel per point x_n^(r) for last iterations
for i in range(iterations):
    x = logistic(r,x)  # just iterate: x_{i+1} = f_r(x_i)

    # Compute the partial sum of the Lyapunov exponent, which is the sum
    #  of derivatives of the logistic function (absolute value)
    lyapunov += np.log(abs(logistic_deriv(r,x)))
    # Display the bifurcation diagram.
    if i >= (iterations-last):  # only plot the last iterations
        ax1.plot(r, x, ',k', alpha=0.25)
ax1.set_xlim(2.8, 4)
ax1.set_xlabel(r'$r$')
ax1.set_title("Bifurcation")

# Display the Lyapunov exponent

# Negative Lyapunov exponent
ax2.plot(r[lyapunov < 0],
         lyapunov[lyapunov < 0] / iterations, 'o',
         color='black', alpha=0.5, ms=0.5)
# Positive Lyapunov exponent
ax2.plot(r[lyapunov >= 0],
         lyapunov[lyapunov >= 0] / iterations, 'o',
         color='red', alpha=0.5, ms=0.5)

# Add a zero line (lightened with alpha=0.5)
ax2.axhline(0, color='k', lw=0.5, alpha=0.5)

ax2.set_xlim(2.8, 4)
ax2.set_ylim(-2, 1)
ax2.set_xlabel(r'$r$')
ax2.set_title("Lyapunov exponent")

plt.tight_layout()_____no_output_____

We see there is a fixed point for $r < 3$, then two and four periods and a chaotic behavior when $r$ belongs to certain areas of the parameter space.  **Do the values of $r$ where these behaviors occur agree with what you found in the plots with particular $r$ values?**

As for the pendulum, the Lyapunov exponent is positive when the system is chaotic (in red).  **Do these regions look consistent with the characteristics of chaos?**_____no_output_____


# Notebook from ucsd-ccbb/mali-dual-crispr-pipeline
Path: dual_crispr/distributed_files/notebooks/Dual CRISPR 4-Count Combination.ipynb

# Dual CRISPR Screen Analysis
# Step 4: Count Combination
Amanda Birmingham, CCBB, UCSD (abirmingham@ucsd.edu)

## Instructions

To run this notebook reproducibly, follow these steps:
1. Click **Kernel** > **Restart & Clear Output**
2. When prompted, click the red **Restart & clear all outputs** button
3. Fill in the values for your analysis for each of the variables in the [Input Parameters](#Input-Parameters) section
4. Click **Cell** > **Run All**

## Input Parameters_____no_output_____

g_dataset_name = "Notebook4Test"
g_fastq_counts_run_prefix = "TestSet4"
g_fastq_counts_dir = '~/dual_crispr/test_data/test_set_4'
g_collapsed_counts_run_prefix = ""
g_collapsed_counts_dir = '~/dual_crispr/test_outputs/test_set_4'
g_combined_counts_dir = ""
g_combined_counts_run_prefix = ""_____no_output_____

## Automated Set-Up_____no_output_____

import inspect

import ccbb_pyutils.analysis_run_prefixes as ns_runs
import ccbb_pyutils.files_and_paths as ns_files
import ccbb_pyutils.notebook_logging as ns_logs

def describe_var_list(input_var_name_list):
    description_list =  ["{0}: {1}\n".format(name, eval(name)) for name in input_var_name_list]
    return "".join(description_list)

ns_logs.set_stdout_info_logger()_____no_output_____g_fastq_counts_dir = ns_files.expand_path(g_fastq_counts_dir)
g_collapsed_counts_run_prefix = ns_runs.check_or_set(g_collapsed_counts_run_prefix,
                                                     ns_runs.generate_run_prefix(g_dataset_name))
g_collapsed_counts_dir = ns_files.expand_path(ns_runs.check_or_set(g_collapsed_counts_dir, g_fastq_counts_dir))
g_combined_counts_run_prefix = ns_runs.check_or_set(g_combined_counts_run_prefix, g_collapsed_counts_run_prefix)
g_combined_counts_dir = ns_files.expand_path(ns_runs.check_or_set(g_combined_counts_dir, g_collapsed_counts_dir))

print(describe_var_list(['g_fastq_counts_dir','g_collapsed_counts_run_prefix','g_collapsed_counts_dir',
                         'g_combined_counts_run_prefix','g_combined_counts_dir']))

ns_files.verify_or_make_dir(g_collapsed_counts_dir)
ns_files.verify_or_make_dir(g_combined_counts_dir)_____no_output_____

## Count Combination Functions_____no_output_____

import dual_crispr.construct_counter as ns_counter
print(inspect.getsource(ns_counter.get_counts_file_suffix))_____no_output_____import dual_crispr.count_combination as ns_combine
print(inspect.getsource(ns_combine))_____no_output_____

## Input Count Filenames_____no_output_____

print(ns_files.summarize_filenames_for_prefix_and_suffix(g_fastq_counts_dir, g_fastq_counts_run_prefix,
                                                         ns_counter.get_counts_file_suffix()))_____no_output_____

## Count Combination Execution_____no_output_____

ns_combine.write_collapsed_count_files(g_fastq_counts_dir, g_collapsed_counts_dir, g_collapsed_counts_run_prefix,
                            g_fastq_counts_run_prefix, ns_counter.get_counts_file_suffix(),
                            ns_combine.get_collapsed_counts_file_suffix())_____no_output_____print(ns_files.check_file_presence(g_collapsed_counts_dir, g_collapsed_counts_run_prefix,
                                   ns_combine.get_collapsed_counts_file_suffix(),
                                   check_failure_msg="Count combination failed to produce collapsed count file(s)."))_____no_output_____ns_combine.write_combined_count_file(g_collapsed_counts_dir, g_combined_counts_dir, g_collapsed_counts_run_prefix,
                          g_combined_counts_run_prefix, ns_combine.get_collapsed_counts_file_suffix(),
                          ns_combine.get_combined_counts_file_suffix())_____no_output_____print(ns_files.check_file_presence(g_combined_counts_dir, g_collapsed_counts_run_prefix,
                                   ns_combine.get_collapsed_counts_file_suffix(),
                                   check_failure_msg="Count combination failed to produce combined count file."))_____no_output_____


# Notebook from alexander-camuto/SpectralBias
Path: ICML_Robustness.ipynb

# Robustness to Parameter Perturbations_____no_output_____In this notebook, we see how the lower-frequencies are less vulnerable to random perturbatations in the parameters. This means that _high-frequency parameters_ could be hard to find. _____no_output_____

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import numpy as np

from argparse import Namespace
from functools import reduce
from copy import deepcopy

import matplotlib
from matplotlib import pyplot as plt
import seaborn as sns
sns.set()
%matplotlib inline_____no_output_____

## Data Generation

The code for generating the dataset and evaluating FFT. _____no_output_____

def make_phased_waves(opt):
    t = np.arange(0, 1, 1./opt.N)
    if opt.A is None:
        yt = reduce(lambda a, b: a + b,
                    [np.sin(2 * np.pi * ki * t + 2 * np.pi * phi) for ki, phi in zip(opt.K, opt.PHI)])
    else:
        yt = reduce(lambda a, b: a + b,
                    [Ai * np.sin(2 * np.pi * ki * t + 2 * np.pi * phi) for ki, Ai, phi in zip(opt.K, opt.A, opt.PHI)])
    return t, yt

def fft(opt, yt):
    n = len(yt) # length of the signal
    k = np.arange(n)
    T = n/opt.N
    frq = k/T # two sides frequency range
    frq = frq[range(n//2)] # one side frequency range
    # -------------
    FFTYT = np.fft.fft(yt)/n # fft computing and normalization
    FFTYT = FFTYT[range(n//2)]
    fftyt = abs(FFTYT)
    return frq, fftyt

def to_torch_dataset_1d(opt, t, yt):
    t = torch.from_numpy(t).view(-1, 1).float()
    yt = torch.from_numpy(yt).view(-1, 1).float()
    if opt.CUDA:
        t = t.cuda()
        yt = yt.cuda()
    return t, yt_____no_output_____

## Model Training_____no_output_____

def make_model(opt):
    layers = []
    layers.append(nn.Linear(opt.INP_DIM, opt.WIDTH))
    layers.append(nn.ReLU())
    for _ in range(opt.DEPTH - 2):
        layers.append(nn.Linear(opt.WIDTH, opt.WIDTH))
        layers.append(nn.ReLU())
    layers.extend([nn.Linear(opt.WIDTH, opt.OUT_DIM)])
    model = nn.Sequential(*layers)
    if opt.CUDA:
        model = model.cuda()
    return model_____no_output_____def power_iteration(A, num_simulations=10):
    # Ideally choose a random vector
    # To decrease the chance that our vector
    # Is orthogonal to the eigenvector
    A = A.data
    b_k = A.new(A.shape[1], 1).normal_()
    for _ in range(num_simulations):
        # calculate the matrix-by-vector product Ab
        b_k1 = A @ b_k
        # calculate the norm
        b_k1_norm = torch.norm(b_k1)
        # re normalize the vector
        b_k = b_k1 / b_k1_norm
    return ((b_k.t() @ A @ b_k) / b_k.t() @ b_k).squeeze().abs()
    # return torch.dot(torch.dot(b_k.t(), A), b_k) / torch.dot(b_k.t(), b_k)

def spectral_norm(model):
    norms = []
    for layer in model:
        if isinstance(layer, nn.Linear):
            if layer.in_features == layer.out_features:
                norms.append(power_iteration(layer.weight).cpu().numpy())
            elif layer.in_features == 1 or layer.out_features == 1:
                norms.append(torch.norm(layer.weight.data))
    return norms_____no_output_____def train_model(opt, model, input_, target):
    # Build loss
    loss_fn = nn.MSELoss()
    # Build optim
    optim = torch.optim.Adam(model.parameters(), lr=opt.LR)
    # Rec
    frames = []
    model.train()
    # To cuda
    if opt.CUDA:
        input_ = input_.cuda()
        target = target.cuda()
    # Loop!
    for iter_num in range(opt.NUM_ITER):
        if iter_num % (opt.NUM_ITER // 100) == 0:
            print(">", end='')
        x = input_
        yt = target.view(-1, opt.OUT_DIM)
        optim.zero_grad()
        y = model(x)
        loss = loss_fn(y, yt)
        loss.backward()
        optim.step()
        if iter_num % opt.REC_FRQ == 0:
            # Measure spectral norm
            frames.append(Namespace(iter_num=iter_num,
                                    prediction=y.data.cpu().numpy(),
                                    loss=loss.item(),
                                    spectral_norms=spectral_norm(model)))
    # Done
    model.eval()
    return frames_____no_output_____

## Parameter Perturbations_____no_output_____

def param_norm(params):
    return np.sqrt(sum([p.data.pow(2).sum().item() for p in params]))

def unit_params(like):
    new_params = [Variable(p.data.new(*p.size()).normal_(), requires_grad=True) for p in like]
    norm = param_norm(new_params)
    for p in new_params:
        p.data.div_(norm)
    return new_params

def axis_aligned_unit_params(like):
    new_params = [Variable(p.data.new(*p.size()).zero_(), requires_grad=True) for p in like]
    # Choose a random weight matrix
    rand_ps = [p for p in new_params if p.dim() == 2]
    rand_p = rand_ps[np.random.randint(len(rand_ps))]
    # Light one up at a random position
    y_idx = np.random.randint(rand_p.size(0))
    x_idx = np.random.randint(rand_p.size(1))
    rand_p.data[y_idx, x_idx] = 1
    return new_params

def set_params(model, params, inplace=False):
    if inplace:
        model_cp = model
    else:
        model_cp = deepcopy(model)
    for p, new_p in zip(model_cp.parameters(), params):
        p.data.copy_(new_p.data)
    return model_cp

def scale_params(params, scale, inplace=False):
    if not inplace:
        params = deepcopy(params)
    for p in params:
        p.data.mul_(scale)
    return params

def sum_params(param1, param2, inplace=False):
    if not inplace:
        params = deepcopy(param1)
    else:
        params = param1
    for p, p2 in zip(params, param2):
        p.data.add_(p2.data)
    return params

def scale_params_(params, scale):
    return scale_params(params, scale, True)_____no_output_____def measure_model(opt, frames, model):
    # Sample random parameter
    x, _ = to_torch_dataset_1d(opt, *make_phased_waves(opt))
    if opt.CUDA:
        model = model.cuda()
    specs_at_norm = []
    for perturb_norm in opt.PERTURB_NORMS:
        samples = []
        for samp_num in range(opt.NUM_SAMPLES):
            rand_unit_params = unit_params(model.parameters())
            scaled_params = scale_params_(rand_unit_params, perturb_norm)
            perturbed_params = sum_params(list(model.parameters()), scaled_params, inplace=False)
            # Set params
            new_model = set_params(model, perturbed_params, inplace=False)
            # Predict at x
            preds = new_model(Variable(x, requires_grad=False)).data.cpu().numpy().squeeze()
            frq, spectrum = fft(opt, preds.squeeze())
            samples.append(spectrum)
        mean_spec = np.array(samples).mean(0)
        specs_at_norm.append(mean_spec)
    specs_at_norm = np.array(specs_at_norm)
    # Normalize
    preds = frames[-1].prediction
    frq, model_spectrum = fft(opt, preds.squeeze())
    K_arr = np.array(opt.K).reshape(1, -1)
    A_arr = model_spectrum[np.sum(frq.reshape(-1, 1) == K_arr, axis=-1, dtype='bool')]
    normalized_specs_at_norm = specs_at_norm[:, np.sum(frq.reshape(-1, 1) == K_arr, axis=-1, dtype='bool')]/A_arr
    return normalized_specs_at_norm

def measure_all_models(opt, all_frames, models):
    all_specs_at_norm = []
    for frames, model in zip(all_frames, models):
        print(">", end='')
        specs_at_norm = measure_model(opt, frames, model)
        all_specs_at_norm.append(specs_at_norm)
    all_specs_at_norm = np.array(all_specs_at_norm).mean(0)
    return all_specs_at_norm_____no_output_____

## Visualization_____no_output_____

def plot_wave_and_spectrum(opt, x, yox):
    # Btw, "yox" --> "y of x"
    # Compute fft
    k, yok = fft(opt, yox)
    # Plot
    fig, (ax0, ax1) = plt.subplots(1, 2, figsize=(9, 4))
    ax0.set_title("Function")
    ax0.plot(x, yox)
    ax0.set_xlabel("x")
    ax0.set_ylabel("f(x)")
    ax1.set_title("FT of Function")
    ax1.plot(k, yok)
    ax1.set_xlabel("k")
    ax1.set_ylabel("f(k)")
    plt.show()_____no_output_____def compute_spectra(opt, frames):
    # Make array for heatmap
    dynamics = []
    xticks = []
    for iframe, frame in enumerate(frames):
        # Compute fft of prediction
        frq, yfft = fft(opt, frame.prediction.squeeze())
        dynamics.append(yfft)
        xticks.append(frame.iter_num)
    return np.array(frq), np.array(dynamics), np.array(xticks)

def plot_spectral_dynamics(opt, all_frames):
    all_dynamics = []
    # Compute spectra for all frames
    for frames in all_frames:
        frq, dynamics, xticks = compute_spectra(opt, frames)
        all_dynamics.append(dynamics)
    # Average dynamics over multiple frames
    # mean_dynamics.shape = (num_iterations, num_frequencies)
    mean_dynamics = np.array(all_dynamics).mean(0)
    # Select the frequencies which are present in the target spectrum
    freq_selected = mean_dynamics[:, np.sum(frq.reshape(-1, 1) == np.array(opt.K).reshape(1, -1),
                                            axis=-1, dtype='bool')]
    # Normalize by the amplitude. Remember to account for the fact that the measured spectra
    # are single-sided (positive freqs), so multiply by 2 accordingly
    norm_dynamics = 2 * freq_selected / np.array(opt.A).reshape(1, -1)
    # Plot heatmap
    plt.figure(figsize=(7, 6))
    # plt.title("Evolution of Frequency Spectrum (Increasing Amplitudes)")
    sns.heatmap(norm_dynamics[::-1],
                xticklabels=opt.K,
                yticklabels=[(frame.iter_num if frame.iter_num % 10000 == 0 else '')
                             for _, frame in zip(range(norm_dynamics.shape[0]), frames)][::-1],
                vmin=0., vmax=1.,
                cmap=sns.cubehelix_palette(8, start=.5, rot=-.75, reverse=True, as_cmap=True))
    plt.xlabel("Frequency [Hz]")
    plt.ylabel("Training Iteration")
    plt.show()

def plot_multiple_spectral_norms(all_frames):
    iter_nums = np.array([frame.iter_num for frame in all_frames[0]])
    norms = np.array([np.array(list(zip(*[frame.spectral_norms for frame in frames]))).squeeze() for frames in all_frames])
    means = norms.mean(0)
    stds = norms.std(0)
    plt.xlabel("Training Iteration")
    plt.ylabel("Spectral Norm of Layer Weights")
    for layer_num, (mean_curve, std_curve) in enumerate(zip(means, stds)):
        p = plt.plot(iter_nums, mean_curve, label=f'Layer {layer_num + 1}')
        plt.fill_between(iter_nums, mean_curve + std_curve, mean_curve - std_curve, color=p[0].get_color(), alpha=0.15)
    plt.legend()
    plt.show()

def plot_robustness(opt, mean_spectra):
    # Plot heatmap
    plt.figure(figsize=(7, 6))
    sns.heatmap(mean_spectra[::-1],
                xticklabels=opt.K,
                yticklabels=[(np.round(norm) if i % 20 == 0 else '') for i, norm in enumerate(opt.PERTURB_NORMS)][::-1],
                vmin=0., vmax=1.,
               )
    plt.xlabel("Frequency [Hz]")
    plt.ylabel("Perturbation Norm")
    plt.show()_____no_output_____

## Play_____no_output_____

opt = Namespace()_____no_output_____# Data Generation
opt.N = 200
opt.K = [5, 10, 15, 20, 25, 30, 35, 40, 45, 50]
opt.A = [1 for _ in opt.K]
opt.PHI = [np.random.rand() for _ in opt.K]
# Model parameters
opt.INP_DIM = 1
opt.OUT_DIM = 1
opt.WIDTH = 256
opt.DEPTH = 6
# Training
opt.CUDA = True
opt.NUM_ITER = 60000
opt.REC_FRQ = 100
opt.LR = 0.0003_____no_output_____

### Plot the Functions

... as a sanity check. _____no_output_____

x, y = make_phased_waves(opt)
plot_wave_and_spectrum(opt, x, y)_____no_output_____

Note that we only plot the positive frequencies, which is why the peaks in the spectrum are at $0.5$ (half the power is in the negative frequencies). _____no_output_____### Train the Model_____no_output_____

def go(opt, repeats=10):
    all_frames = []
    models = []
    for _ in range(repeats):
        # Sample random phase
        opt.PHI = [np.random.rand() for _ in opt.K]
        # Generate data
        x, y = to_torch_dataset_1d(opt, *make_phased_waves(opt))
        # Make model
        model = make_model(opt)
        # Train
        frames = train_model(opt, model, x, y)
        all_frames.append(frames)
        models.append(model.cpu())
        print('', end='\n')
    return all_frames, models_____no_output_____

Train 5 models, as usual. _____no_output_____

opt.K = [5, 10, 15, 20, 25, 30, 35, 40, 45, 50]
opt.A = [1 for _ in opt.K]_____no_output_____frames, models = go(opt, 5)>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Draw 100 random unit vectors in the parameter space, and measure the frequency spectrum of the network as the parameters move along these unit vectors. Finally, average the results over the 100 unit vectors and the 5 models. _____no_output_____

opt.PERTURB_NORMS = np.linspace(0, 5.001, 100)
opt.NUM_SAMPLES = 100_____no_output_____mean_spectra = measure_all_models(opt, frames, models)>>>>>SIZE = 15
matplotlib.rc('xtick', labelsize=SIZE)
matplotlib.rc('ytick', labelsize=SIZE)
matplotlib.rc('axes', labelsize=SIZE, titlesize=SIZE)
matplotlib.rc('legend', fontsize=SIZE)

plot_robustness(opt, mean_spectra)_____no_output_____


# Notebook from ChanLIM/ColBERT
Path: ColBERT_official_new_api.ipynb

# ColBERT: Indexing & Search Notebook v0.1_____no_output_____We start by importing the relevant classes. As we'll see below, `Indexer` and `Searcher` are the key actors here. _____no_output_____

!git clone -b new_api https://github.com/stanford-futuredata/ColBERT.gitCloning into 'ColBERT'...
remote: Enumerating objects: 516, done.
remote: Counting objects: 100% (356/356), done.
remote: Compressing objects: 100% (264/264), done.
remote: Total 516 (delta 169), reused 220 (delta 87), pack-reused 160
Receiving objects: 100% (516/516), 255.34 KiB | 3.99 MiB/s, done.
Resolving deltas: 100% (224/224), done.
%cd ColBERT/content/ColBERT
!apt-get update
!apt-get install git_____no_output_____!pip install ujson
!pip install GitPython
!pip install transformers==3.0.2
!pip install faiss-gpu==1.6.3Requirement already satisfied: ujson in /usr/local/lib/python3.7/dist-packages (5.1.0)
Requirement already satisfied: GitPython in /usr/local/lib/python3.7/dist-packages (3.1.24)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from GitPython) (3.10.0.2)
Requirement already satisfied: gitdb<5,>=4.0.1 in /usr/local/lib/python3.7/dist-packages (from GitPython) (4.0.9)
Requirement already satisfied: smmap<6,>=3.0.1 in /usr/local/lib/python3.7/dist-packages (from gitdb<5,>=4.0.1->GitPython) (5.0.0)
Requirement already satisfied: transformers==3.0.2 in /usr/local/lib/python3.7/dist-packages (3.0.2)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (2019.12.20)
Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (3.4.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (21.3)
Requirement already satisfied: tokenizers==0.8.1.rc1 in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (0.8.1rc1)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (2.23.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (1.19.5)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (4.62.3)
Requirement already satisfied: sacremoses in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (0.0.46)
Requirement already satisfied: sentencepiece!=0.1.92 in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (0.1.96)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers==3.0.2) (3.0.6)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.0.2) (2021.10.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.0.2) (1.24.3)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.0.2) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.0.2) (2.10)
Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==3.0.2) (7.1.2)
Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==3.0.2) (1.1.0)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==3.0.2) (1.15.0)
Collecting faiss-gpu==1.6.3
  Downloading faiss_gpu-1.6.3-cp37-cp37m-manylinux2010_x86_64.whl (35.5 MB)
     |████████████████████████████████| 35.5 MB 265 kB/s
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from faiss-gpu==1.6.3) (1.19.5)
Installing collected packages: faiss-gpu
Successfully installed faiss-gpu-1.6.3
import os
import sys
sys.path.insert(0, '../')

from colbert.infra import Run, RunConfig
from colbert.data import Queries, Collection
from colbert import Indexer, Searcher_____no_output_____

The workflow here assumes an IR dataset: a set of queries and a corresponding collection of passages.

The classes `Queries` and `Collection` provide a convenient interface for working with such datasets.

We'll load the answer posts of the English Language Learners (ELL) StackExchange community as our collection, and use relevant GooAQ questions as our queries._____no_output_____

dataroot = '/future/u/okhattab/data/tmp/stackexchange/'
dataset = 'ell'

queries = os.path.join(dataroot, dataset, 'questions.tsv')
collection = os.path.join(dataroot, dataset, 'collection.answeronly.tsv')

queries = Queries(path=queries)
collection = Collection(path=collection)

f'Loaded {len(queries)} queries and {len(collection):,} passages'[Dec 23, 02:52:41] #> Loading the queries from /future/u/okhattab/data/tmp/stackexchange/ell/questions.tsv ...

This loaded 441 queries and 214,300 passages, numbered (in this dataset) from 0 onwathrough 440. Let's inspect one query and one passage._____no_output_____

print(queries[23])
print()
print(collection[79852])
print()are disease names proper nouns?

No, “chance” and “get a chance” do not mean the same thing. “Get a chance” means “have an opportunity”, but the verb “chance” alone doesn't. The verb “chance” has several meanings, but none of them work here. In the sense of something happening by luck, it's normally used in a past tense (“I never chanced to meet him” = “I was not lucky enough to meet him”) or hypothetically (“If you chance to meet him, say hello from me” = “If you luck into meeting him, say hello from me”). It doesn't make sense here since by definition chance implies that

## Indexing

For efficient search, we can pre-compute the ColBERT representation of each passage and index them.

Below, the `Indexer` take a model checkpoint and writes a (compressed) index to disk. We then prepare a `Searcher` for retrieval from this index.

(On future machines with four Titan V GPUs, indexing should take 6--7 minutes. The output is fairly ugly at the moment!)_____no_output_____

with Run().context(RunConfig(nranks=4, experiment='notebook')):  # nranks specifies the number of GPUs to use.
    index_name = f'{dataset}.index'

    indexer = Indexer(checkpoint='/dfs/scratch0/okhattab/OpenQA/colbert-400000.dnn')  # MS MARCO ColBERT checkpoint
    indexer.index(name=index_name, collection=collection, overwrite=True)

    searcher = Searcher(index=index_name){
    "DocSettings": {
        "dim": 128,
        "doc_maxlen": 220,
        "mask_punctuation": true
    },
    "IndexingSettings": {
        "centroid_fraction_of_sample": 0.03,
        "chunksize": 2.0,
        "compression_level": 1,
        "compression_thresholds": "\/future\/u\/keshav2\/compression_thresholds.csv",
        "index_root": null,
        "kmeans_niters": 20,
        "kmeans_spherical": true,
        "partitions": null,
        "sample": 0.05
    },
    "QuerySettings": {
        "query_maxlen": 32
    },
    "ResourceSettings": {
        "checkpoint": "\/dfs\/scratch0\/okhattab\/OpenQA\/colbert-400000.dnn",
        "collection": "\/future\/u\/okhattab\/data\/tmp\/stackexchange\/ell\/collection.answeronly.tsv",
        "index_name": "ell.index",
        "queries": null,
        "triples": null
    },
    "RunSettings": {
        "amp": true,
        "experiment": "notebook",
        "gpus": [
            0,
            1,
            2,
            3
        ],
        "name": "2021-09-21\/19.49.08",
        "nranks": 4,
        "overwrite": false,
        "rank": -1,
        "root": "\/future\/u\/okhattab\/repos\/ColBERT-private-releases\/experiments"
    },
    "SearchSettings": {
        "faiss_depth": 16384,
        "nprobe": 2
    },
    "TrainingSettings": {
        "accumsteps": 1,
        "bsize": 64,
        "lr": 3e-6,
        "maxsteps": 500000,
        "resume": false,
        "save_every": null,
        "similarity": "cosine"
    }
}

[Sep 21, 19:49:08] #> Note: Output directory /future/u/okhattab/repos/ColBERT-private-releases/experiments/notebook/indexes/ell.index already exists

#> Preparing...
#> Preparing...
#> Preparing...
#> Preparing...
[Sep 21, 19:49:09] [Pre-Emptying] GPU memory check: r=0, a=0, f=0
[Sep 21, 19:49:09] [Post-Emptying] GPU memory check: r=0, a=0, f=0
#> Starting...
#> Starting...
#> Starting...
#> Starting...
nranks = 4 	 num_gpus = 4 	 device=1
[Sep 21, 19:50:07] [1] 		 #> Encoding 18994 passages..
nranks = 4 	 num_gpus = 4 	 device=3
[Sep 21, 19:50:07] [3] 		 #> Encoding 18983 passages..
nranks = 4 	 num_gpus = 4 	 device=0
[Sep 21, 19:50:07] [0] 		 # of sampled PIDs = 81138 	 sampled_pids[:3] = [109214, 192069, 2665]
nranks = 4 	 num_gpus = 4 	 device=2
[Sep 21, 19:50:07] [2] 		 #> Encoding 18845 passages..
[Sep 21, 19:50:07] [0] 		 #> Encoding 24316 passages..
[Sep 21, 19:50:57] [0] 		 avg_doclen_est = 78.97882843017578 	 len(local_sample) = 24,316
[Sep 21, 19:50:57] [2] 		 avg_doclen_est = 78.97882843017578 	 len(local_sample) = 18,845
[Sep 21, 19:50:57] [3] 		 avg_doclen_est = 78.97882843017578 	 len(local_sample) = 18,983
[Sep 21, 19:50:57] [1] 		 avg_doclen_est = 78.97882843017578 	 len(local_sample) = 18,994
[Sep 21, 19:51:01] [0] 		 Creaing 65,536 partitions. *Estimated* 16,925,162.93258667 embeddings.
[Sep 21, 19:51:01] [0] 		 #> Saving the indexing plan to /future/u/okhattab/repos/ColBERT-private-releases/experiments/notebook/indexes/ell.index/plan.json ..
Clustering 6168239 points in 128D to 65536 clusters, redo 1 times, 20 iterations
  Preprocessing in 0.65 s
  Iteration 19 (138.63 s, search 133.16 s): objective=5.0599e+06 imbalance=1.333 nsplit=0
[0.041, 0.044, 0.043, 0.044, 0.046, 0.046, 0.044, 0.043, 0.043, 0.045, 0.045, 0.043, 0.041, 0.045, 0.043, 0.041, 0.044, 0.044, 0.044, 0.046, 0.043, 0.041, 0.047, 0.041, 0.046, 0.049, 0.044, 0.043, 0.043, 0.042, 0.043, 0.044, 0.045, 0.044, 0.044, 0.043, 0.039, 0.049, 0.042, 0.041, 0.043, 0.039, 0.043, 0.046, 0.043, 0.045, 0.045, 0.042, 0.043, 0.041, 0.044, 0.042, 0.042, 0.043, 0.042, 0.043, 0.04, 0.04, 0.041, 0.043, 0.042, 0.047, 0.044, 0.043, 0.05, 0.05, 0.046, 0.044, 0.043, 0.046, 0.04, 0.041, 0.043, 0.045, 0.045, 0.045, 0.044, 0.04, 0.044, 0.043, 0.044, 0.043, 0.041, 0.042, 0.043, 0.04, 0.041, 0.042, 0.044, 0.046, 0.041, 0.039, 0.046, 0.043, 0.043, 0.039, 0.041, 0.048, 0.042, 0.048, 0.042, 0.047, 0.044, 0.044, 0.042, 0.048, 0.042, 0.039, 0.04, 0.042, 0.044, 0.043, 0.043, 0.046, 0.044, 0.042, 0.041, 0.045, 0.042, 0.039, 0.043, 0.041, 0.039, 0.041, 0.045, 0.04, 0.042, 0.048]
[Sep 21, 19:53:35] avg_residual = 0.043212890625

## Search

Having built the index and prepared our `searcher`, we can search for individual query strings.

We can use the `queries` set we loaded earlier — or you can supply your own questions, say, "how large is the vocabulary of the average person?"

Feel free to get creative! But keep in mind this set of ~200k ELL passages can only answer a small, focused set of questions on English Language Learning._____no_output_____

query = queries[30]   # or supply your own query

print(f"#> {query}")

# Find the top-3 passages for this query
results = searcher.search(query, k=3)

# Print out the top-k retrieved passages
for passage_id, passage_rank, passage_score in zip(*results):
    print(f"\t [{passage_rank}] \t\t {passage_score:.1f} \t\t {searcher.collection[passage_id]}")#> are morphology and structure the same?
	 [1] 		 21.3 		 This is tricky one, I'll try and explain my perspective as a native British English speaker. I have a biology background as opposed to physics, so that might come into play as well here. Some defintions: Structure: Arrangement of and relations between the parts or elements of something complex or a piece of construction. Whereas morphology: Morphology: A particular form, shape, or structure or the study of something's form of shape. As you can see, they both have one usage where they are near enough synonymous, and another where they are not. In most situations they can be
	 [2] 		 19.6 		 Since each program has one structure, the correct sentence is Program A and program B have the same structure: they both have a sequential structure. There is a single structure, which is shared by the two programs. The “same structure” in the first sentence is the sequential structure, there is only one. Other ways to formulate this idea include The structure of program A is the same as the structure of program B. Both program A and program B have a sequential structure. The structures of program A and program B are the same. In this
	 [3] 		 19.4 		 The structure is of the form of X. The structure is in the form of X. What is the difference in meaning and which one is grammatically correct? I think 1 means the structure is united with the form of X and 2 means the structure is shaped in the form of X So, as for 1, I don't think the entire shape of the structure must look in the form of X (maybe ,part of it is united with the form of X?), but as for 2, I do think the entire shape of the

## Batch Search

In many applications, you have a large batch of queries and you need to maximize the overall throughput. For that, you can use the `searcher.search_all(queries, k)` method, which returns a `Ranking` object that organizes the results across all queries.

(Batching provides many opportunities for higher-throughput search, though we have not implemented most of those optimizations for compressed indexes yet.)_____no_output_____

rankings = searcher.search_all(queries, k=5).todict()_____no_output_____rankings[30]  # For query 30, a list of (passage_id, rank, score) for the top-k passages_____no_output__________no_output__________no_output_____


# Notebook from shashankdholakia/starry
Path: notebooks/EclipsingBinary_PyMC3.ipynb

# Eclipsing binary: `pymc3` solution for the maps_____no_output_____In this notebook, we're going to do MCMC to infer the surface maps of two stars in an eclipsing binary given the light curve of the system. We generated the data in [this notebook](EclipsingBinary_Generate.ipynb). Note that here we assume we know everything else about the system (the orbital parameters, the limb darkening coefficients, etc.), so the only unknown parameters are the maps of the two stars, which are expressed in `starry` as vectors of spherical harmonic coefficients. In a future tutorial we'll explore a more complex inference problem where we have uncertainties on all the parameters.

Let's begin with some imports. **Note that in order to do inference with pymc3, we need to enable lazy evaluation.** That's because `pymc3` requires derivatives of the likelihood function, so we need to use the fancy `theano` computational graph to perform backpropagation on the `starry` model. All this means in practice is that we'll have to call `.eval()` in some places to get numerical values out of the parameters._____no_output_____

%matplotlib inline_____no_output_____%run notebook_setup.py_____no_output_____import matplotlib.pyplot as plt
import numpy as np
import pymc3 as pm
import pymc3_ext as pmx
import exoplanet as xo
import os
import starry
from corner import corner

np.random.seed(12)
starry.config.lazy = True
starry.config.quiet = True_____no_output_____

## Load the data

Let's load the EB dataset:_____no_output_____

# Run the Generate notebook if needed
if not os.path.exists("eb.npz"):
    import nbformat
    from nbconvert.preprocessors import ExecutePreprocessor

    with open("EclipsingBinary_Generate.ipynb") as f:
        nb = nbformat.read(f, as_version=4)
    ep = ExecutePreprocessor(timeout=600, kernel_name="python3")
    ep.preprocess(nb);_____no_output_____data = np.load("eb.npz", allow_pickle=True)
A = data["A"].item()
B = data["B"].item()
t = data["t"]
flux = data["flux"]
sigma = data["sigma"]_____no_output_____

Next, we instantiate the primary, secondary, and system objects. Recall that we assume we know the true values of all the orbital parameters and star properties, *except* for the two surface maps. Note that we are instantiating the `starry` objects within a `pm.Model()` context so that `pymc3` can keep track of all the variables._____no_output_____

with pm.Model() as model:

    # Primary
    pri = starry.Primary(
        starry.Map(ydeg=A["ydeg"], udeg=A["udeg"], inc=A["inc"]),
        r=A["r"],
        m=A["m"],
        prot=A["prot"],
    )
    pri.map[1:] = A["u"]

    # Secondary
    sec = starry.Secondary(
        starry.Map(ydeg=B["ydeg"], udeg=B["udeg"], inc=B["inc"]),
        r=B["r"],
        m=B["m"],
        porb=B["porb"],
        prot=B["prot"],
        t0=B["t0"],
        inc=B["inc"],
    )
    sec.map[1:] = B["u"]

    # System
    sys = starry.System(pri, sec)_____no_output_____

Here's the light curve we're going to do inference on:_____no_output_____

fig, ax = plt.subplots(1, figsize=(12, 5))
ax.plot(t, flux, "k.", alpha=0.5, ms=4)
ax.set_xlabel("time [days]", fontsize=24)
ax.set_ylabel("normalized flux", fontsize=24);_____no_output_____

## Define the `pymc3` model

Now we define the full `pymc3` model. If you've never used `pymc3` before, Dan Foreman-Mackey's [exoplanet package documentation](https://exoplanet.dfm.io/en/stable/) has lots of nice tutorials on how to use `pymc3` to do inference. The basic idea here is we define our variables by assigning priors to them; we use a `pm.MvNormal` for both the primary and secondary maps. This is a multi-variate normal (Gaussian) distribution, which happens to be a convenient prior to place on spherical harmonic coefficients because of its close relationship to the *power spectrum* of the map. In particular, if the Gaussian prior is zero-mean and its covariance is diagonal with constant entries for each degree $l$ (as we assume below), this is equivalent to an isotropic prior whose power spectrum is given by those entries on the diagonal. Note that for simplicity we are assuming a *flat* power spectrum, meaning we place the same prior weight on all spatial scales. So the covariance of our Gaussian is as simple as it can be: it's just $\lambda I$, where $\lambda = 10^{-2}$ is the prior variance of the spherical harmonic coefficients and $I$ is the identity matrix. The scalar $\lambda$ is essentially a regularization parameter: by making it small, we ensure that the spherical harmonic coefficients stay close to zero, which is usually what we want for physical maps.

You'll note there's also a call to `pm.Deterministic`, which just keeps track of variables for later (in this case, we'll have access to the value of `flux_model` for every iteration of the chain once we're done; this is useful for plotting). And finally, there's a call to `pm.Normal` in which we specify our `observed` values, their standard deviation `sd`, and the mean vector `mu`, which is our `starry` flux model. This normal distribution is our chi-squared term: we're telling `pymc3` that our data is normally distributed about our model with some (observational) uncertainty._____no_output_____

with pm.Model() as model:

    # The amplitude of the primary
    pri.map.amp = pm.Normal("pri_amp", mu=1.0, sd=0.1)

    # The Ylm coefficients of the primary
    # with a zero-mean isotropic Gaussian prior
    ncoeff = pri.map.Ny - 1
    pri_mu = np.zeros(ncoeff)
    pri_cov = 1e-2 * np.eye(ncoeff)
    pri.map[1:, :] = pm.MvNormal("pri_y", pri_mu, pri_cov, shape=(ncoeff,))

    # The amplitude of the secondary
    sec.map.amp = pm.Normal("sec_amp", mu=0.1, sd=0.01)

    # The Ylm coefficients of the secondary
    # with a zero-mean isotropic Gaussian prior
    ncoeff = sec.map.Ny - 1
    sec_mu = np.zeros(ncoeff)
    sec_cov = 1e-2 * np.eye(ncoeff)
    sec.map[1:, :] = pm.MvNormal("sec_y", sec_mu, sec_cov, shape=(ncoeff,))

    # Compute the flux
    flux_model = sys.flux(t=t)

    # Track some values for plotting later
    pm.Deterministic("flux_model", flux_model)

    # Save our initial guess
    flux_model_guess = pmx.eval_in_model(flux_model)

    # The likelihood function assuming known Gaussian uncertainty
    pm.Normal("obs", mu=flux_model, sd=sigma, observed=flux)_____no_output_____

Now that we've specified the model, it's a good idea to run a quick gradient descent to find the MAP (maximum a posteriori) solution. This will give us a decent starting point for the inference problem._____no_output_____

with model:
    map_soln = pmx.optimize()_____no_output_____

Note the dramatic increase in the value of the log posterior!
Let's plot the MAP model alongside the data and the initial guess (note that we're doing quite well)._____no_output_____

plt.figure(figsize=(12, 5))
plt.plot(t, flux, "k.", alpha=0.3, ms=2, label="data")
plt.plot(t, flux_model_guess, "C1--", lw=1, alpha=0.5, label="Initial")
plt.plot(
    t, pmx.eval_in_model(flux_model, map_soln, model=model), "C1-", label="MAP", lw=1
)
plt.legend(fontsize=10, numpoints=5)
plt.xlabel("time [days]", fontsize=24)
plt.ylabel("relative flux", fontsize=24);_____no_output_____

We can also plot the corresponding maps: note that we recover the spots *really well*!_____no_output_____

map = starry.Map(ydeg=A["ydeg"])
map.inc = A["inc"]
map.amp = map_soln["pri_amp"]
map[1:, :] = map_soln["pri_y"]
map.show(theta=np.linspace(0, 360, 50))_____no_output_____map = starry.Map(ydeg=B["ydeg"])
map.inc = B["inc"]
map.amp = map_soln["sec_amp"]
map[1:, :] = map_soln["sec_y"]
map.show(theta=np.linspace(0, 360, 50))_____no_output_____

## MCMC sampling

We have an optimum solution, but we're really interested in the *posterior* over surface maps (i.e., an understanding of the uncertainty of our solution). We're therefore going to do MCMC sampling with `pymc3`. This is easy: within the `model` context, we just call `pmx.sample`. The number of tuning and draw steps below are quite small since I wanted this notebook to run quickly; try increasing them by a factor of a few to get more faithful posteriors._____no_output_____

with model:
    trace = pmx.sample(tune=500, draws=500, start=map_soln, chains=4, target_accept=0.9)_____no_output_____

We can look at `pm.summary` to check if things converged. In particular, we're looking for a large number of effective samples `ess` for all parameters and a value of `r_hat` that is very close to one._____no_output_____

varnames = ["pri_amp", "pri_y", "sec_amp", "sec_y"]
display(pm.summary(trace, var_names=varnames).head())
display(pm.summary(trace, var_names=varnames).tail())_____no_output_____

The number of effective samples for some of the parameters is quite small, so in practice we should run this chain for longer (an exercise for the reader!) But let's carry on for now, keeping in mind that our posteriors will be quite noisy._____no_output_____Let's plot the model for 24 random samples from the chain. Note that the lines are so close together that they're indistinguishable!_____no_output_____

plt.figure(figsize=(12, 5))
plt.plot(t, flux, "k.", alpha=0.3, ms=2, label="data")
label = "samples"
for i in np.random.choice(range(len(trace["flux_model"])), 24):
    plt.plot(t, trace["flux_model"][i], "C0-", alpha=0.3, label=label)
    label = None
plt.legend(fontsize=10, numpoints=5)
plt.xlabel("time [days]", fontsize=24)
plt.ylabel("relative flux", fontsize=24);_____no_output_____

Let's compare the *mean* map and a *random* sample to the true map for each star:_____no_output_____

# Random sample
np.random.seed(0)
i = np.random.randint(len(trace["pri_y"]))

map = starry.Map(ydeg=A["ydeg"])
map[1:, :] = np.mean(trace["pri_y"], axis=0)
map.amp = np.mean(trace["pri_amp"])
pri_mu = map.render(projection="rect").eval()
map[1:, :] = trace["pri_y"][i]
map.amp = trace["pri_amp"][i]
pri_draw = map.render(projection="rect").eval()
map[1:, :] = A["y"]
map.amp = A["amp"]
pri_true = map.render(projection="rect").eval()

map = starry.Map(ydeg=B["ydeg"])
map[1:, :] = np.mean(trace["sec_y"], axis=0)
map.amp = np.mean(trace["sec_amp"])
sec_mu = map.render(projection="rect").eval()
map[1:, :] = trace["sec_y"][i]
map.amp = trace["sec_amp"][i]
sec_draw = map.render(projection="rect").eval()
map[1:, :] = B["y"]
map.amp = B["amp"]
sec_true = map.render(projection="rect").eval()

fig, ax = plt.subplots(3, 2, figsize=(8, 7))
ax[0, 0].imshow(
    pri_true,
    origin="lower",
    extent=(-180, 180, -90, 90),
    cmap="plasma",
    vmin=0,
    vmax=0.4,
)
ax[1, 0].imshow(
    pri_mu,
    origin="lower",
    extent=(-180, 180, -90, 90),
    cmap="plasma",
    vmin=0,
    vmax=0.4,
)
ax[2, 0].imshow(
    pri_draw,
    origin="lower",
    extent=(-180, 180, -90, 90),
    cmap="plasma",
    vmin=0,
    vmax=0.4,
)
ax[0, 1].imshow(
    sec_true,
    origin="lower",
    extent=(-180, 180, -90, 90),
    cmap="plasma",
    vmin=0,
    vmax=0.04,
)
ax[1, 1].imshow(
    sec_mu,
    origin="lower",
    extent=(-180, 180, -90, 90),
    cmap="plasma",
    vmin=0,
    vmax=0.04,
)
ax[2, 1].imshow(
    sec_draw,
    origin="lower",
    extent=(-180, 180, -90, 90),
    cmap="plasma",
    vmin=0,
    vmax=0.04,
)
ax[0, 0].set_title("primary")
ax[0, 1].set_title("secondary")
ax[0, 0].set_ylabel("true", rotation=0, labelpad=20)
ax[1, 0].set_ylabel("mean", rotation=0, labelpad=20)
ax[2, 0].set_ylabel("draw", rotation=0, labelpad=20);_____no_output_____

Looks pretty good! There are obvious artifacts (there are tons of degeneracies in this problem), but we've definitely recovered the spots, with some uncertainty. Recall that our chains weren't well converged! Run this notebook for longer to get more faithful posteriors.

Finally, here's a corner plot for the first several coefficients of the primary map. You can see that all the posteriors are nice and Gaussian, with some fairly strong correlations (the degeneracies I mentioned above):_____no_output_____

fig, ax = plt.subplots(9, 9, figsize=(7, 7))
labels = [r"$\alpha$"] + [
    r"$Y_{%d,%d}$" % (l, m)
    for l in range(1, pri.map.ydeg + 1)
    for m in range(-l, l + 1)
]
samps = np.hstack((trace["pri_amp"].reshape(-1, 1), trace["pri_y"][:, :8]))
corner(samps, fig=fig, labels=labels)
for axis in ax.flatten():
    axis.xaxis.set_tick_params(labelsize=6)
    axis.yaxis.set_tick_params(labelsize=6)
    axis.xaxis.label.set_size(12)
    axis.yaxis.label.set_size(12)
    axis.xaxis.set_label_coords(0.5, -0.6)
    axis.yaxis.set_label_coords(-0.6, 0.5)_____no_output_____

That's it! While sampling with `pymc3` is fairly fast, the problem of inferring a surface map when all other parameters are known is a **linear problem**, which means it actually has an *analytic* solution! In the following [notebook](EclipsingBinary_Linear.ipynb), we show how to take advantage of this within `starry` to do extremely fast inference._____no_output_____


# Notebook from MLVPRASAD/Na-ve-Bees-Predict-Species-from-Images
Path: notebook.ipynb

## 1. Import Python libraries

A honey bee (Apis).

Can a machine identify a bee as a honey bee or a bumble bee? These bees have different behaviors and appearances, but given the variety of backgrounds, positions, and image resolutions, it can be a challenge for machines to tell them apart.

Being able to identify bee species from images is a task that ultimately would allow researchers to more quickly and effectively collect field data. Pollinating bees have critical roles in both ecology and agriculture, and diseases like colony collapse disorder threaten these species. Identifying different species of bees in the wild means that we can better understand the prevalence and growth of these important insects.

A bumble bee (Bombus).

After loading and pre-processing images, this notebook walks through building a model that can automatically detect honey bees and bumble bees.
_____no_output_____

# used to change filepaths
import os

import matplotlib as mpl
import matplotlib.pyplot as plt
from IPython.display import display
%matplotlib inline

import pandas as pd
import numpy as np

# import Image from PIL
# ... YOUR CODE FOR TASK 1 ...

from PIL import Image
from skimage.feature import hog
from skimage.color import rgb2grey

from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
# import train_test_split from sklearn's model selection module
# ... YOUR CODE FOR TASK 1 ...
from sklearn.svm import SVC
# import SVC from sklearn's svm module
# ... YOUR CODE FOR TASK 1 ...

# import accuracy_score from sklearn's metrics module
from sklearn.metrics import roc_curve, auc,accuracy_score_____no_output_____

## 2. Display image of each bee type

Now that we have all of our imports ready, it is time to look at some images. We will load our labels.csv file into a dataframe called labels, where the index is the image name (e.g. an index of 1036 refers to an image named 1036.jpg) and the genus column tells us the bee type. genus takes the value of either 0.0 (Apis or honey bee) or 1.0 (Bombus or bumble bee).

The function get_image converts an index value from the dataframe into a file path where the image is located, opens the image using the Image object in Pillow, and then returns the image as a numpy array.

We'll use this function to load the sixth Apis image and then the sixth Bombus image in the dataframe.
_____no_output_____

# load the labels using pandas
labels = pd.read_csv("datasets/labels.csv", index_col=0)

# show the first five rows of the dataframe using head
display(labels.head())

def get_image(row_id, root="datasets/"):
    """
    Converts an image number into the file path where the image is located,
    opens the image, and returns the image as a numpy array.
    """
    filename = "{}.jpg".format(row_id)
    file_path = os.path.join(root, filename)
    img = Image.open(file_path)
    return np.array(img)

# subset the dataframe to just Apis (genus is 0.0) get the value of the sixth item in the index
apis_row = labels[labels.genus == 0.0].index[5]

# show the corresponding image of an Apis
plt.imshow(get_image(apis_row))
plt.show()

# subset the dataframe to just Bombus (genus is 1.0) get the value of the sixth item in the index
bombus_row = labels[labels.genus == 1.0].index[5]

# show the corresponding image of a Bombus
plt.imshow(get_image(bombus_row))
plt.show()_____no_output_____

## 3. Image manipulation with rgb2grey

scikit-image has a number of image processing functions built into the library, for example, converting an image to greyscale. The rgb2grey function computes the luminance of an RGB image using the following formula Y = 0.2125 R + 0.7154 G + 0.0721 B.

Image data is represented as a matrix, where the depth is the number of channels. An RGB image has three channels (red, green, and blue) whereas the returned greyscale image has only one channel. Accordingly, the original color image has the dimensions 100x100x3 but after calling rgb2grey, the resulting greyscale image has only one channel, making the dimensions 100x100x1.
_____no_output_____

# load a bombus image using our get_image function and bombus_row from the previous cell
bombus = get_image(bombus_row)

# print the shape of the bombus image
print('Color bombus image has shape: ', bombus.shape)

# convert the bombus image to greyscale
grey_bombus = rgb2grey(bombus)

# show the greyscale image
plt.imshow(grey_bombus, cmap=mpl.cm.gray)

# greyscale bombus image only has one channel
print('Greyscale bombus image has shape: ', grey_bombus.shape)_____no_output_____

## 4. Histogram of oriented gradients

Now we need to turn these images into something that a machine learning algorithm can understand. Traditional computer vision techniques have relied on mathematical transforms to turn images into useful features. For example, you may want to detect edges of objects in an image, increase the contrast, or filter out particular colors.

We've got a matrix of pixel values, but those don't contain enough interesting information on their own for most algorithms. We need to help the algorithms along by picking out some of the salient features for them using the histogram of oriented gradients (HOG) descriptor. The idea behind HOG is that an object's shape within an image can be inferred by its edges, and a way to identify edges is by looking at the direction of intensity gradients (i.e. changes in luminescence).

An image is divided in a grid fashion into cells, and for the pixels within each cell, a histogram of gradient directions is compiled. To improve invariance to highlights and shadows in an image, cells are block normalized, meaning an intensity value is calculated for a larger region of an image called a block and used to contrast normalize all cell-level histograms within each block. The HOG feature vector for the image is the concatenation of these cell-level histograms.
_____no_output_____

# run HOG using our greyscale bombus image
hog_features, hog_image = hog(grey_bombus,
                              visualize=True,
                              block_norm='L2-Hys',
                              pixels_per_cell=(16, 16))

# show our hog_image with a grey colormap
plt.imshow(hog_image, cmap=mpl.cm.gray)_____no_output_____

## 5. Create image features and flatten into a single row

Algorithms require data to be in a format where rows correspond to images and columns correspond to features. This means that all the information for a given image needs to be contained in a single row.

We want to provide our model with the raw pixel values from our images as well as the HOG features we just calculated. To do this, we will write a function called create_features that combines these two sets of features by flattening the three-dimensional array into a one-dimensional (flat) array.
_____no_output_____

def create_features(img):
    # flatten three channel color image
    color_features = img.flatten()
    # convert image to greyscale
    grey_image = rgb2grey(img)
    # get HOG features from greyscale image
    hog_features = hog(grey_image, block_norm='L2-Hys', pixels_per_cell=(16, 16))
    # combine color and hog features into a single array
    flat_features = np.hstack((color_features, hog_features))
    return flat_features

bombus_features = create_features(bombus)
bombus_features.shape
# print shape of bombus_features
# ... YOUR CODE FOR TASK 5 ..._____no_output_____

## 6. Loop over images to preprocess

Above we generated a flattened features array for the bombus image. Now it's time to loop over all of our images. We will create features for each image and then stack the flattened features arrays into a big matrix we can pass into our model.

In the create_feature_matrix function, we'll do the following:

Load an image

Generate a row of features using the create_features function above

Stack the rows into a features matrix

In the resulting features matrix, rows correspond to images and columns to features.
_____no_output_____

def create_feature_matrix(label_dataframe):
    features_list = []

    for img_id in label_dataframe.index:
        # load image
        img = get_image(img_id)
        # get features for image
        image_features = create_features(img)
        features_list.append(image_features)

    # convert list of arrays into a matrix
    feature_matrix = np.array(features_list)
    return feature_matrix

# run create_feature_matrix on our dataframe of images
feature_matrix = create_feature_matrix(labels)_____no_output_____

## 7. Scale feature matrix + PCA

Our features aren't quite done yet. Many machine learning methods are built to work best with data that has a mean of 0 and unit variance. Luckily, scikit-learn provides a simple way to rescale your data to work well using StandardScaler. They've got a more thorough explanation of why that is in the linked docs.

Remember also that we have over 31,000 features for each image and only 500 images total. To use an SVM, our model of choice, we also need to reduce the number of features we have using principal component analysis (PCA).

PCA is a way of linearly transforming the data such that most of the information in the data is contained within a smaller number of features called components. Below is a visual example from an image dataset containing handwritten numbers. The image on the left is the original image with 784 components. We can see that the image on the right (post PCA) captures the shape of the number quite effectively even with only 59 components.

In our case, we will keep 500 components. This means our feature matrix will only have 500 columns rather than the original 31,296.
_____no_output_____

# get shape of feature matrix
print('Feature matrix shape is: ', feature_matrix.shape)

# define standard scaler
ss = StandardScaler()
# run this on our feature matrix
bees_stand = ss.fit_transform(feature_matrix)

pca = PCA(n_components=500)
# use fit_transform to run PCA on our standardized matrix
bees_pca = pca.fit_transform(bees_stand)
# look at new shape
print('PCA matrix shape is: ', bees_pca.shape)_____no_output_____

## 8. Split into train and test sets

Now we need to convert our data into train and test sets. We'll use 70% of images as our training data and test our model on the remaining 30%. Scikit-learn's train_test_split function makes this easy.
_____no_output_____

X_train, X_test, y_train, y_test = train_test_split(bees_pca,
                                                    labels.genus.values,
                                                    test_size=.3,
                                                    random_state=1234123)

# look at the distrubution of labels in the train set
pd.Series(y_train).value_counts()
# ... YOUR CODE FOR TASK 8_____no_output_____

## 9. Train model

It's finally time to build our model! We'll use a support vector machine (SVM), a type of supervised machine learning model used for regression, classification, and outlier detection." An SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall."

Here's a visualization of the maximum margin separating two classes using an SVM classifier with a linear kernel.

Since we have a classification task -- honey or bumble bee -- we will use the support vector classifier (SVC), a type of SVM. We imported this class at the top of the notebook.
_____no_output_____

svm = SVC(kernel='linear', probability=True, random_state=42)

# fit model
svm.fit(X_train, y_train)_____no_output_____

## 10. Score model

Now we'll use our trained model to generate predictions for our test data. To see how well our model did, we'll calculate the accuracy by comparing our predicted labels for the test set with the true labels in the test set. Accuracy is the number of correct predictions divided by the total number of predictions. Scikit-learn's accuracy_score function will do math for us. Sometimes accuracy can be misleading, but since we have an equal number of honey and bumble bees, it is a useful metric for this problem.
_____no_output_____

y_pred = svm.predict(X_test)

# calculate accuracy
accuracy = accuracy_score(y_pred, y_test)
print('Model accuracy is: ', accuracy)_____no_output_____

## 11. ROC curve + AUC

Above, we used svm.predict to predict either 0.0 or 1.0 for each image in X_test. Now, we'll use svm.predict_proba to get the probability that each class is the true label. For example, predict_proba returns [0.46195176, 0.53804824] for the first image, meaning there is a 46% chance the bee in the image is an Apis (0.0) and a 53% chance the bee in the image is a Bombus (1.0). Note that the two probabilities for each image always sum to 1.

Using the default settings, probabilities of 0.5 or above are assigned a class label of 1.0 and those below are assigned a 0.0. However, this threshold can be adjusted. The receiver operating characteristic curve (ROC curve) plots the false positive rate and true positive rate at different thresholds. ROC curves are judged visually by how close they are to the upper lefthand corner.

The area under the curve (AUC) is also calculated, where 1 means every predicted label was correct. Generally, the worst score for AUC is 0.5, which is the performance of a model that randomly guesses. See the scikit-learn documentation for more resources and examples on ROC curves and AUC.
_____no_output_____

# predict probabilities for X_test using predict_proba
probabilities = ...

# select the probabilities for label 1.0
y_proba = ...

# calculate false positive rate and true positive rate at different thresholds
false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_proba, pos_label=1)

# calculate AUC
roc_auc = auc(...)

plt.title('Receiver Operating Characteristic')
# plot the false positive rate on the x axis and the true positive rate on the y axis
roc_plot = plt.plot(...,
                    ...,
                    label='AUC = {:0.2f}'.format(roc_auc))

plt.legend(loc=0)
plt.plot([0,1], [0,1], ls='--')
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate');_____no_output_____