# Notebook from Michael-Monaldi/LectureNotes
Path: Lecture07_KNN/Lecture 07 Non-parametric K-Nearest Neighbors Algorithm.ipynb

# K-Nearest Neighbors Algorithm

* Last class, we introduced the probabilistic generative classifier.
* As discussed, the probabilistic generative classifier requires us to assume a parametric form for each class (e.g., each class is represented by a multi-variate Gaussian distribution, etc..).  Because of this, the probabilistic generative classifier is a *parametric* approach
* Parametric approaches have the drawback that the functional parametric form needs to be decided/assumed in advance and, if chosen poorly, might be a poor model of the distribution that generates the data resulting in poor performance.
* Non-parametric methods are those that do not assume a particular generating distribution for the data.  The $K$-nearest neighbors algorithm is one example of a non-parametric classifier.

_____no_output_____* Nearest neighbor methods compare test point to the $k$ nearest training data points and then estimate an output value based on the desired/true output values of the $k$ nearest training points
* Essentially, there is no ``training'' other than storing the training data points and their desired outputs
* In test, you need to: (1) determine which $k$ training data points are closest to the test point; and (2) determine the output value for the test point
* In order to find the $k$ nearest neighbors in the training data, you need to define a *similarity measure* or a *dissimilarity measure*.  The most common dissimilarity measure is Euclidean distance.
    * Euclidean distance: $d_E = \sqrt{\left(\mathbf{x}_1-\mathbf{x}_2\right)^T\left(\mathbf{x}_1-\mathbf{x}_2\right)}$
    * City block distance: $d_C = \sum_{i=1}^d \left| x_{1i} - x_{2i} \right|$
    * Mahalanobis distance: $\left(\mathbf{x}_1-\mathbf{x}_2\right)^T\Sigma^{-1}\left(\mathbf{x}_1-\mathbf{x}_2\right)$
    * Geodesic distance
    * Cosine angle similarity: $\cos \theta = \frac{\mathbf{x}_1^T\mathbf{x}_2}{\left\|\mathbf{x}_1\right\|_2\left\|\mathbf{x}_2\right\|_2}$
    * and many more...
_____no_output_____* If you are doing classification, once you find the $k$ nearest neighbors to your test point in the training data, then you can determine the class label of your test point using (most commonly) *majority vote*
* If there are ties, they can be broken randomly or using schemes like applying the label of the closest data point in the neighborhood
*  Of course, there are MANY modifications to you can make to this.  A common one is to weight the votes of each of the nearest neighbors by their distance/similarity measure value.  If they are closer, they get more weight.
_____no_output_____

<code>
import numpy as np
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_moons, make_circles, make_classification
from sklearn import neighbors
%matplotlib inline

#figure params
h = .02  # step size in the mesh
figure = plt.figure(figsize=(17, 9))

#set up classifiers
n_neighbors = 3
classifiers = []
classifiers.append(neighbors.KNeighborsClassifier(n_neighbors, weights='uniform'))
classifiers.append(neighbors.KNeighborsClassifier(n_neighbors, weights='distance'))
names = ['K-NN_Uniform', 'K-NN_Weighted']

#Put together datasets
n_samples = 300
X, y = make_classification(n_samples, n_features=2, n_redundant=0, n_informative=2,
                           random_state=0, n_clusters_per_class=1)
rng = np.random.RandomState(2)
X += 2 * rng.uniform(size=X.shape)
linearly_separable = (X, y)

datasets = [make_moons(n_samples, noise=0.3, random_state=0),
            make_circles(n_samples,  noise=0.2, factor=0.5, random_state=1),
            linearly_separable]

i = 1
# iterate over datasets
for X, y in datasets:
    # preprocess dataset, split into training and test part
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.8)  #split into train/test folds

    #set up meshgrid for figure
    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

    # just plot the dataset first
    cm = plt.cm.RdBu
    cm_bright = ListedColormap(['#FF0000', '#0000FF'])
    ax = plt.subplot(len(datasets), len(classifiers) + 1, i)
    # Plot the training points
    ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright)
    # and testing points
    ax.scatter(X_test[:, 0], X_test[:, 1], marker='+', c=y_test, cmap=cm_bright, alpha=0.6)
    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    ax.set_xticks(())
    ax.set_yticks(())
    i += 1

    # iterate over classifiers
    for name, clf in zip(names, classifiers):
        ax = plt.subplot(len(datasets), len(classifiers) + 1, i)
        clf.fit(X_train, y_train)
        score = clf.score(X_test, y_test)

        # Plot the decision boundary. For that, we will assign a color to each
        # point in the mesh [x_min, x_max]x[y_min, y_max].
        Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])

        # Put the result into a color plot
        Z = Z.reshape(xx.shape)
        ax.contourf(xx, yy, Z, cmap=cm, alpha=.8)

        # Plot also the training points
        ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright)
        # and testing points
        ax.scatter(X_test[:, 0], X_test[:, 1], marker='+', c=y_test, cmap=cm_bright,
                   alpha=0.4)

        ax.set_xlim(xx.min(), xx.max())
        ax.set_ylim(yy.min(), yy.max())
        ax.set_xticks(())
        ax.set_yticks(())
        ax.set_title(name)
        ax.text(xx.max() - .3, yy.min() + .3, ('%.2f' % score).lstrip('0'),
                size=15, horizontalalignment='right')
        i += 1

figure.subplots_adjust(left=.02, right=.98)
plt.show()_____no_output_____
</code>

# Error and Evaluation Metrics

* A key step in machine learning algorithm development and testing is determining a good error and evaluation metric.

* Evaluation metrics help us to estimate how well our model is trained and it is important to pick a metric that matches our overall goal for the system.

* Some common evaluation metrics include precision, recall, receiver operating curves, and confusion matrices._____no_output_____### Classification Accuracy and Error

* Classification accuracy is defined as the number of correctly classified samples divided by all samples:

\begin{equation}
\text{accuracy} = \frac{N_{cor}}{N}
\end{equation}
where $N_{cor}$ is the number of correct classified samples and $N$ is the total number of samples.

* Classification error is defined as the number of incorrectly classified samples divided by all samples:

\begin{equation}
\text{error} = \frac{N_{mis}}{N}
\end{equation}
where $N_{mis}$ is the number of misclassified samples and $N$ is the total number of samples.

* Suppose there is a 3-class classification problem, in which we would like to classify each training sample (a fish) to one of the three classes (A = salmon or B = sea bass or C = cod).

* Let's assume there are 150 samples, including 50 salmon, 50 sea bass and 50 cod.  Suppose our model misclassifies 3 salmon, 2 sea bass and 4 cod.

* Prediction accuracy of our binary classification model is calculated as:

\begin{equation}
\text{accuracy} = \frac{47+48+46}{50+50+50} = \frac{47}{50}
\end{equation}

* Prediction error is calculated as:

\begin{equation}
\text{error} = \frac{N_{mis}}{N} = \frac{3+2+4}{50+50+50} = \frac{3}{50}
\end{equation}

_____no_output_____### Confusion Matrices

* A confusion matrix summarizes the classification accuracy across several classes. It shows the ways in which our classification model is confused when it makes predictions, allowing visualization of the performance of our algorithm. Generally, each row represents the instances of a actual class while each column represents the instances in an predicted class.

* If our classifier is trained to distinguish between salmon, sea bass and cod. We can summarize the prediction result in the confusion matrix as follows:

| Actual/Predicted | Salmon | Sea bass | Cod  |
| --- | --- | --- | --- |
| Salmon | 47 | 2 | 1 |
| Sea Bass | 2 | 48 | 0 |
| Cod | 0 | 0 | 50 |

* In this confusion matrix, of the 50 actual salmon, the classifier predicted that 2 are sea bass, 1 is cod incorrectly and 47 are labeled salmon correctly. All correct predictions are located in the diagonal of the table. So it is easy to visually inspect the table for prediction errors, as they will be represented by values outside the diagonal.

### TP, FP, TN, and FN

* True positive (TP): correctly predicting event values
* False positive (FP): incorrectly calling non-events as an event
* True negative (TN): correctly predicting non-event values
* False negative (FN): incorrectly labeling events as non-event

* Precision is also called positive predictive value.

\begin{equation}
\text{Precision} = \frac{\text{TP}}{\text{TP}+\text{FP}}
\end{equation}

* Recall is also called true positive rate, probability of detection

\begin{equation}
\text{Recall} = \frac{\text{TP}}{\text{TP}+\text{FN}}
\end{equation}

* Fall-out is also called false positive rate, probability of false alarm.

\begin{equation}
\text{Fall-out} = \frac{\text{FP}}{\text{N}}= \frac{\text{FP}}{\text{FP}+\text{TN}}
\end{equation}

* *Consider the salmon/non-salmon classification problem, what are the TP, FP, TN, FN values?*

| Actual/Predicted | Salmon | Non-Salmon  |
| --- | --- | --- |
| Salmon | 47 | 3 |
| Non-Salmon | 2 | 98 |
_____no_output_____### ROC curves

* The Receiver Operating Characteristic (ROC) curve is a plot between the true positive rate (TPR) and the false positive rate (FPR), where the TPR is defined on the $x$-axis and FPR is defined on the $y$-axis.

* $TPR = TP/(TP+FN)$ is defined as ratio between true positive prediction and all real positive samples. The definition used for $FPR$ in a ROC curve is often problem dependent.  For example, for detection of targets in an area, FPR may be defined as the ratio between the number of false alarms per unit area ($FA/m^2$).  In another example, if you have a set number of images and you are looking for targets in these collection of images, FPR may be defined as the number of false alarms per image.  In some cases, it may make the most sense to simply use the Fall-out or false positive rate.

* Given a binary classifier and its threshold, the (x,y) coordinates of ROC space can be calculated from all the prediction result.  You trace out a ROC curve by varying the threshold to get all of the points on the ROC.

* The diagonal between (0,0) and (1,1) separates the ROC space into two areas, which are left up area and right bottom area. The points above the diagonal represent good classification (better than random guess) which below the diagonal represent bad classification (worse than random guess).

* *What is the perfect prediction point in a ROC curve?*

_____no_output_____### MSE and MAE

* *Mean Square Error* (MSE) is the average of the squared error between prediction and actual observation.

* For each sample $\mathbf{x}_i$, the prediction value is $y_i$ and the actual output is $d_i$. The MSE is

\begin{equation}
MSE = \sum_{i=1}^n \frac{(d_i - y_i)^2}{n}
\end{equation}

* *Root Mean Square Error* (RMSE) is simply the square root the MSE.

\begin{equation}
RMSE = \sqrt{MSE}
\end{equation}

* *Mean Absolute Error* (MAE) is the average of the absolute error.
\begin{equation}
MAE = \frac{1}{n} \sum_{i=1}^n \lvert d_i - y_i \rvert
\end{equation}
_____no_output_____


# Notebook from MichaelHopwood/HopML
Path: HopML/Bootstrapping.ipynb

# Bootstrapping

**Can we estimate the mean of a distribution without using a parametric model?**

Key idea is to first estimate the distribution non-parametrically, and then get an estimate of the mean.

**What about the standard error of that estimator? How to get a C.I.**

Bootstrapping!

## Bootstrap principle

## Empirical distribution

Let $x_1, ..., x_n$ be an independent real-valued random variable with distribution $P$.

Define a pdf $\hat{P}$ by

$$\hat{P}(A) = \frac{1}{n} \sum_{i=1}^n I_A(x_i)$$

where $I_A (x_i) = \begin{cases}
1 & if x_i \in A\\
0 & o.w.\\
\end{cases}$

$\hat{P}$ is called the empirical distribution of the sample $X$.  $\hat{P}$ can be thought as the distribution which puts mass $\frac{1}{n}$ on each observation $X_i$. It can be shown that $\hat{P}$ is a nonparametric likelihood estimator of $P$. This justifies estimating $P$ by $\hat{P}$ if no other information about $P$ is available.

**Theorem:** Let $A \leq \mathbf{R}$ (s.t. $P(A)$ is defined, i.e. $A$ belongs to the Borel $\sigma$-algebra), then $\hat{P}(A) \xrightarrow{} P(A)$ as $n \xrightarrow{} \infty$. This result can proved using the Law of Large Numbers.

**Proof:**

$$n \hat{P}(A) = \sum_{i=1}^n I_A (x_i) \sim Bin(n, P(A))$$

This is a binomial because of our indicator function above.  We then use the law of large numbers to say $\hat{P}(A) \xrightarrow{} P(A)$, its expectation, as $n\xrightarrow{} \infty$. In other words, the distribution $P(A)$ can be approximated by $\hat{P}(A)$ equally well. for all $A \in I$ where $I$ is the set of all intervals of $\mathbf{R}$. This is the empirical distribution.

## Empirical distribution function

Move in the direction of a CDF. Let $x_1, ..., x_n \sim F$ where $F(x) = P(X \leq x)$. Now because we cna relate the $F$ to the $P$, which was defined above, we can relate to the content above.

We can estimate $F$ with the empirical distribution function $\hat{F}_n$, the CDF that puts mass $\frac{1}{n}$ at each data point $x_i$.

$$\hat{F}_n (x) = \frac{1}{n} \sum_{i=1}^n I(X_i \leq x)$$

where $I(x_i \leq x) = \begin{cases}
1 & if X_i \leq x\\
0 & o.w.\\
\end{cases}$

According to **Glivenko-Cautelli theorem**

$$sup_x |\hat{F}_n(x) - F(x)| \xrightarrow[a.s.]{} 0$$

Here, $\hat{F}_n(x) \xrightarrow[a.s.]{} F(x)$ as $n \xrightarrow{} \infty$. This is called a **consistent** estimator of $F$. In fact, the convergence is fast.

## Sampling from the empirical distribution $\hat{F}_n$

To recap, we have a distribution $F$ which we do not know. So, we can use the estimator $\hat{F}_n$, and we know that this estimator is a consistent estimator of $F$. So, how do we get a sample from $\hat{F}_n$?

Suppose we want to draw an iid sample $\vec{X}^\star = (X_1^\star, ..., X_n^\star)^T$ from $\hat{F}_n$. When sampling from $\hat{F}_n$, the $i$th observation $X_i$ in the original sample is selected with probability $\frac{1}{n}$. If we use this idea, we can define a two-step procedure to do our sampling.

Step 1: Draw $i_1, i_2, ..., i_n$ independently from the uniform distribution. $\{ 1, 2, 3, ..., n \}$.

Step 2: Set $X_j^\star = X_{ij}$ and $\vec{X}^\star = (X_{i_1}^\star, ..., X_{i_n}^\star)^T$

For example, let's say $\vec{X} = (3,2,7,8,10,25)^T$ is a sample coming from a distribution that we do not know. Now, we want a sample from the empirical distribution $\hat{F}_n$ of $X$.

Step 1: Draw $i_1, i_2, i_3, i_4, i_5, i_6$ from the uniform $\{ 1,2,3,4,5,6 \}$. Assume that $i_1 = 6,  i_2=3, i_3 = 1, i_4 = 2, i_5=1, i_6 = 3$ comes from the first drawing of the uniform, for instance.

Step 2: We assign $X_j^\star = X_{ij}$. So, in our first sample $\vec{X}_1^\star = (x_{i_1}, x_{i_2}, x_{i_3}, x_{i_4}, x_{i_5}, x_{i_6}) = (X_6, X_3, X_1, X_2, X_1, X_3) = (25, 7, 3, 2, 3, 7)$

Each sample must have $n$ samples! We sample with replacement from the original sample $X_1, X_2, ..., X_n$.

So, sampling from $\hat{F}_n$ is the same as saying sampling with replacement from the original sample.

_____no_output_____## Bootstrap principle

Let $\vec{X} = (X_1, ..., X_n)^T$ be a random sample from $F$. Let $\theta = t(F)$ be some parameter of the distribution $F$.

$\hat{\theta} = s(\vec{X})$ is an estimate of $\theta$.

Example: $\vec{X} = (X_1, ..., X_n) \sim F$,

$\theta = \mu ( = t(F))$, which is a parameter

$\hat{\theta} = \bar{X} (= s(\vec{X}))$, which is a function of the sample, and is an estimator of $\theta$

To evaluate statistical properties (bias or standard error) of $\hat{\theta}$, we need to estimate the sampling distribution of $\hat{\theta}$. The bootstrap method mimics the data generating process by sampling an estimate $\hat{F}_n$ of $F$. The role of the above real quantities is taken by their analogous quantities in the "bootstrap world".

$\vec{X}^\star = (X_1^\star, ..., X_n^\star)^T$ is a bootstrap sample from $\hat{F}_n$.

$\theta^\star = t(\hat{F}_n)$ is the parameter in the bootstrap world.

$\hat{\theta}^\star = s(\vec{X}^\star)$ is the bootstrap replication for $\theta$

So, we have our bootstrap sample data $\vec{X}^\star$ is found using sample with replacement. Then, we have $\theta$ which we are trying to understand. We have $\theta^\star$ which is the parameter in the bootstrap world. $\hat{\theta}^\star$ is the estimated $\theta$.

So, what is the sampling distribution of $\hat{\theta}$?  This is estimated by its bootstrap equivalent $\hat{\theta}^\star$.
_____no_output_____

import numpy as np
n = 1000
sets = 100
i = np.random.uniform(0,1,(sets, n))_____no_output_____

The bootstrap principal can be summarized as follows:

**In the real world**, we see some unknown probabldy distribution and we have a sample, with the goal of getting a statistic from the underlying distribution.

We have an unknown probability distribution (Usually denoted as $P$ or $F$) and an observed random sample $\vec{x}$.

$$P, F \xrightarrow{} \vec{x} = (x_1, ..., x_u)$$

producing $\hat{\theta} = s(\vec{x})$, which is the **statistic of interest**

**In the bootstrap world**, we have an empirical distribution (usually denoted as $\hat{P}$ or $\hat{F}_n$) and a bootstrap sample $\vec{x}^\star$. The bootstrap sample is derived from the observed random sample $\vec{x}$. In the bootstrap world, the observed random sample $\vec{x}$ acts as the "population". Some people call the $\vec{x}^\star$ a "sample of a sample".

$$\hat{P}, \hat{F}_n \xrightarrow{} \vec{x}^\star = (x_1^\star, ..., x_n^\star)$$

producing $\hat{\theta}^\star = s(\vec{x}^\star)$, which is the **bootstrap replication**

Even though the distribution of the boostrap sample $\vec{x}^\star$ is known, the evaluation of the exact bootstrap sampling distribution of $\hat{\theta}^\star$ can still be intractable (aka, a tough problem).

### Example: Bootstrap median

Let's say our stat of interest is the median. Our bootstrap sample can still be generated and we can get the bootstrap median. But, what is the sampling distribution of the sample median?

In general, the bootstrap estimate of the sampling distribution of $\hat{\theta}^\star$ is computed using the _monte carlo method_.

**Algorithm**

Step 1: Draw $B$ independent bootstrap samples $\vec{X}^{(\star) (1)}, ..., \vec{X}^{(\star) (B)}$ from $\hat{F}_n$ (i.e. samples of size $n$ (with replacement) repeated $B$ times)

This will generate a lot of $\hat{\theta}^\star$ and we can visualize it in a distribution.

Step 2: Evaluate the bootstrap replications $\hat{\theta}^\star = s(\vec{X}^{\star (b)})$, $b = 1, ..., B$

Step 3: Estimate the sampling distribution of $\hat{\theta}$ by the empirical distribution of the bootstrap replications $\hat{\theta}^{\star (1)}, ..., \hat{\theta}^{\star (B)}$

**Question: Why should bootstrap work?**

Glivenko-Cantelli Theorem says that $\hat{F}_n \xrightarrow[a.s.]{} F$ as $n \xrightarrow{} \infty$. So, iid sampling from $\hat{F}_n$ should be approximately the same as iid sampling from $F$ when $n$ is large.
_____no_output_____## Applications

### Bootstrap for standard error (s.e.)

Let $\theta = s(\vec{x})$ be an estimator for $\theta$ and suppose we want to know the s.e. of $\hat{\theta}$.

**Algorithm:**

Step 1: Draw $B$ independent samples $\vec{X}^{\star (1)}, ..., \vec{X}^{\star (B)}$ from $\hat{F}_n$.

Step 2: Evaluate the bootstrap replications $\hat{\theta}^\star = s(\vec{X}^{\star (b)})$, $b = 1, ..., B$

Step 3: Estimate the s.e., $se(\hat{\theta})$, by the standard deviation of the $B$ replications.

$$se_{boot}(\hat{\theta}) = [\frac{1}{B-1} \sum_{b=1}^B (\hat{\theta}^{\star (b)} - \hat{\theta}^{\star (o)})^2]^{1/2}$$

where $\hat{\theta}^{\star o} = \frac{1}{B} \sum_{b=1}^B \hat{\theta}^{\star (b)}$

Remark: $\bar{\hat{\theta}}^\star$ or $\hat{\theta}^{\star (o)}$: mean of the bootstrap replications_____no_output_____

import numpy as np
from numpy.random import choice
import matplotlib.pyplot as plt

n=1000
rv = np.random.randint(0,10,n)

# TODO: FINISH
def bootstrap(samples, statistic_func, nboot=1000):
    """Conduct bootstrap statistic estimation, including the estimate,
    standard error, and confidence interval

    Parameters
    ----------
    samples : array (n,)
        1D array of samples
    statistic_func : function
        Function which summarizes an array
        theta = f(samples)
        Should have parameter `axis` where a specification of 0 allows
        row-wise operations.
    nboot : int
        Number of bootstrap samples
    """
    # Draw nboot independent samples from the available samples
    samples_star = choice(samples, size=(len(samples),nboot))
    # Evaluate the bootstrap replications
    theta_hat = statistic_func(samples_star, axis=0)
    # Analyze!
    # 1. Evaluate standard error, se(theta-hat)
    theta_hat_star_dot = (1 / nboot) * np.sum(theta_hat)
    se_boot = ((1 / (nboot - 1)) * np.sum(np.square(theta_hat - theta_hat_star_dot)) )**(1/2)
    # 2. Visualize sampling distribution
    plt.hist(samples_star)
    plt.show()

    return theta_hat#, se_boot

bootstrap(rv, np.std, nboot=100), np.std(rv)_____no_output_____

### Bootstrap estimate of bias

Suppose we estimate the parameter $\theta = t(F)$ by the statistic $\hat{\theta} = s(\vec{x})$

The bias of $\hat{\theta}$ is defined as

$$bias(\hat{\theta}) = E(\hat{\theta}) - \theta$$

Recall: if $bias(\hat{\theta}) = 0$, then $E(\hat{\theta}) = \theta$, and $\hat{\theta}$ is an unbiased estimator.

Example: Let $\theta = \mu$, and $\hat{\theta} = \bar{x}$.

$E[\bar{x}] = \mu$, $\bar{x}$ is an unbiased estimator for $\mu$.

Substituting the empirical distribution $\hat{F}_n$ for $F$, the bootstrap estimate of the bias is
$$\hat{bias (\hat{\theta})} = bias^\star (\hat{\theta}^\star) = E^\star [\hat{\theta}^\star] - \hat{\theta} = \hat{\theta}^{\star (o)} - \hat{\theta}$$_____no_output_____### Confidence Intervals

From the sampling distribution of $\hat{\theta}$, we can construct CIs for $\theta$

**Standard CI**

Suppose that $\hat{\theta}$ is approximately normally distributed with mean $\theta$ and variance $se(\hat{\theta})^2$.

An approximate $(1 - \alpha) 100 \%$ CI for $\theta$ is $\hat{\theta}_L = \hat{\theta} - Z_{\alpha/2} \hat{se}_{boot}(\hat{\theta})$ and $\hat{\theta}_U = \hat{\theta} + Z_{\alpha/2} \hat{se}_{boot}(\hat{\theta})$, where $Z_\alpha$ is the $\alpha$ critical value of the standard normal.

**Bootstrap t-interval**

See function in Rizzo textbook - TODO: replicate it in python.

Suppose that $\hat{\theta}$ is approximately normal distribution. Also, suppose that $\hat{\sigma}_n$ is an estimate of the standard deviation of $\hat{\theta}$. Therefore, $\hat{\sigma}_n = \frac{s}{\sqrt{n}}$ in the normal mean problem. We can also use the **delta theorem approximation** for $\hat{\sigma}_n$.

Let $T = \frac{\hat{\theta} - \theta}{\hat{\sigma}_n}$ be the "t - statistic".

We can also use the bootstrap appromimation of $\hat{\sigma}_n$!

From the bootstrap samples $X^{\star (b)}$, we calculate $T^{\star (b)} = \frac{\hat{\theta^{\star (b)}} - \theta}{\hat{\sigma}_n^\star}$ as the $T$ counterpart.

Then the bootstrap + CI for $\theta$ is given by $[\hat{\theta} + C^\star_{\alpha /2} \hat{\sigma}_n, \hat{\theta} + C^\star_{1 - \alpha /2} \hat{\sigma}_n]$ where $C^\star_p$ is the 100 $p$th percentile of $T^{\star (b)}$

**Bootstrap percentile interval**

Let $\hat{\theta}^{\star (1)}, \hat{\theta}^{\star (B)}$ be a bootstrap sample. **Order** the bootstrap replicates $\hat{\theta}^\star_{(1)}, \hat{\theta}^\star_{(2)}, ..., \hat{\theta}^\star_{(B)}$. Let $m = [\frac{\alpha}{2} \times B]$, where $[u]$ is the largest integer less than or equal to $u$.

An approximation $(1 - \alpha) 100 \%$ CI for $\theta$ is $(\hat{\theta}_{(m)}^\star, \hat{\theta}_{(B-m)}^\star)$.

Generally, we want $B\geq 1000$

This construction is simple and intuitive, but there are "better" methods

Takeaway: While the standard CI and bootstrap t-interval need normal distribution specifications, the bootstrap percentile interval does not.

## Bias-corrected percentile interval (BCPI)

Correct the percentile interval by using the bias!

The CI should have equal probability to both sides of $\hat{\theta}$, that is $P(\hat{\theta} < \theta < \hat{\theta}_U) = P(\hat{\theta}_L < \theta < \hat{\theta})$   (Condition *).

**Example:** For $\theta = \mu$, our $\bar{x}$ is in the middle of the interval.

If $\hat{\theta}$ is not the median of the bootstrap distribution, Condition * is not fulfilled.

The bias-corrected (BC) percentile interval is one approach to fix this issue.

The 2-sided $100(1-\alpha)\%$ BCPI boils down to picking different quantiles of the bootstrap distribution of $\hat{\theta}$.

Instead of using $[\zeta^\star_{\alpha/2}, \zeta^\star_{1 - \alpha/2}]$ where $\zeta^\star_{p}$ is the 100 $p$th percentile in the bootstrap sample, use $[\zeta_{\beta_1}^\star, \zeta_{\beta_2}^\star]$ where $\beta_1$ and $\beta_2$ are quantities that depend on user-specified constants $(a,b)$. (Formulas for $\beta_1$ and $\beta_2$ are in Rizzo textbook along with discussion on choices)

Idea: does the above principle also apply to the $C^\star_{\alpha/2}$ from Bootstrap t-interval?

## Parametric bootstrap

$\lambda$ : parameter

$\theta$: statistic

Parametric bootstrap is a variation of the standard (non-parametric) bootstrap discussed previously. Suppose that we knwo that the distribution $F$ belongs to a parametric family of distribution $F_\lambda$ with densities $p(x|\lambda)$. If $\hat{\lambda}$ is an estimate of the true parameter $\lambda$, an obvious estimate of $F$ is the distribution $F = F_{\hat{\lambda}}$ with density $p(x|\hat{\lambda})$. In this case, we can still use the bootstrapping method to obtain an estimate of the sampling distribution of $\hat{\theta}$ (or any function of $\hat{\theta}$: $p(\hat{\theta})$). The parametric bootstrap replaces sampling iid from $\hat{F}_n$ (empirical distribution) with sampling iid from $F_{\hat{\theta}}$.

Algorithm:

Step 1: Draw $B$ independent bootstrap samples $\vec{X}^{\star (1)}, ..., \vec{X}^{\star (B)}$ from $F_{\hat{\lambda}}$

Step 2: Evaluate bootstrap replications $\hat{\theta}^{\star (b)} = s(\vec{X}^{\star (b)}), b = 1, ..., B$

Step 3: Estimate the sampling distribution of $\hat{\theta}$ by the empirical distribution of the bootstrap replications $\hat{\theta}^{\star (1)}, ..., \hat{\theta}^{\star (B)}$

This is potentially more complicated than nonparametric because sampling from $F_{\hat{\lambda}}$ might be more difficult than sampling from $\hat{F}_n$

Question: Is $\hat{F}_n$ or $F_{\hat{\lambda}}$ better? Why?

**Example:** Assume a normal population $N(\mu, \sigma^2)$ and $H_0: \mu = 2$ vs. $H_1: \mu \neq 2$.

If $\sigma^2$ is unknown, we can use a $t$-statistic (but here, we try something else).

We can also use the likelihood ratio statistic $\Lambda$. Get the sampling distribution of $\Lambda$ when $H_0$ is true. If $H_0$ is true, $\mu=2$ and we have now $N(2, \sigma^2)$

As a note, $\lambda = \mu$ and $\hat{\theta} = \Lambda$, according to the algorithm presented above

Want to use Monte Carlo simulation to get the sampling distribution of $\Lambda$ when $\mu = 2$. However, monte carlo hypothesis testing will not work here since we do not know $\sigma^2$. So, an alternative method is to use **parameteric bootstrap**. Use $s^2$ from the initial sample as an estimate of $\sigma^2$. Sample from $N(2, s^2)$, then compute $\lambda$ and sampling distribution $\hat{\Lambda}^{\star (1)}, ..., \hat{\Lambda}^{\star (B)}$.
_____no_output_____## Bootstrap Failures

**Example:** Let iid $X_1, ..., X_n \sim Unif(0,\theta)$

Consider statistic $T_n = n(\theta - \hat{\theta}_n)$, where $\hat{\theta} = X_{(n)} = \max X_i$.

Let the bootstrap stat $T^\star_n = n(\hat{\theta}_n - \hat{\theta}^\star_n)$, where $\hat{\theta}^\star_n = X^\star_{(n)}$

**Claim:** The distribution of $T^\star_n$ and $T_n$ are not close when $n \xrightarrow{} \infty$. Therefore, we do not have a consistent estimator.

**Remedy:** Surprisingly, one way to fix bootstrap failure is to take bootstrap samples of size $m = o(n)$ (little o, not big-O) instead of size $n$.

This is called "m-out-of-n bootstrap", and is consistent in cases such as the $unif(0, \theta)$ model. We use bootstrap samples of size $m$ instead of $n$. It is using sampling with replacement.

Intuition: https://stats.stackexchange.com/questions/476018/intuition-behind-m-out-of-n-bootstrap_____no_output_____## Remarks

1. The bootstrap is nonparametric but it does require some assumptions. You can't always assume it is always valid.

2. The bootstrap is an asymptotic method. It is very accurate when $n \xrightarrow{} \infty$ (because each bootstrap samples use all data)

3. There is related method called "Jacknife". However, the bootstrap is valid under weaker conditions.

4. There are many cases where the bootstrap is not formally justified. This is especially true with discrete structures like trees and graphs. It is an informal way to get some intuition on the variability of the procedure. But, keep in mind that the formal guarantees may not apply in these cases.

5. There is a method related to the bootstrap called subsampling. In this case, we draw samples of size $m < n$ without replacement (because without replacement, it is not the same as bootstrap). Subsampling produces valid CI under weaker conditions than the bootstrap. _____no_output_____


# Notebook from ML4DS/ML4all
Path: C3.Classification_LogReg/RegresionLogistica_student.ipynb

# Logistic Regression_____no_output_____    Notebook version: 2.0 (Nov 21, 2017)
                      2.1 (Oct 19, 2018)
                      2.2 (Oct 09, 2019)
                      2.3 (Oct 27, 2020)

    Author: Jesús Cid Sueiro (jcid@tsc.uc3m.es)
            Jerónimo Arenas García (jarenas@tsc.uc3m.es)

    Changes: v.1.0 - First version
             v.1.1 - Typo correction. Prepared for slide presentation
             v.2.0 - Prepared for Python 3.0 (backcompmatible with 2.7)
                     Assumptions for regression model modified
             v.2.1 - Minor changes regarding notation and assumptions
             v.2.2 - Updated notation
             v.2.3 - Improved slides format. Backward compatibility removed_____no_output_____

# To visualize plots in the notebook
%matplotlib inline

# Imported libraries
import csv
import random
import matplotlib
import matplotlib.pyplot as plt
import pylab

import numpy as np
from mpl_toolkits.mplot3d import Axes3D
from sklearn.preprocessing import PolynomialFeatures
from sklearn import linear_model
_____no_output_____

## 1. Introduction

### 1.1. Binary classification

The **goal** of a classification problem is to assign a *class* or *category* to every *instance* or *observation* of a data collection. _____no_output_____Here, we will assume that

   * every instance ${\bf x}$ is an $N$-dimensional vector in $\mathbb{R}^N$, and
   * the class $y$ of sample ${\bf x}$ is an element of a binary set ${\mathcal Y} = \{0, 1\}$.

The goal of a classifier is to predict the true value of $y$ after observing ${\bf x}$._____no_output_____We will denote as $\hat{y}$ the classifier output or **decision**. If $y=\hat{y}$, the decision is a **hit**, otherwise $y\neq \hat{y}$ and the decision is an **error**._____no_output_____### 1.2. Decision theory: the MAP criterion

**Decision theory** provides a solution to the classification problem in situations where the relation between instance ${\bf x}$ and its class $y$ is given by a known probabilistic model._____no_output_____Assume that every tuple $({\bf x}, y)$ is an outcome of a random vector $({\bf X}, Y)$ with joint distribution $p_{{\bf X},Y}({\bf x}, y)$. A natural criteria for classification is to select predictor $\hat{Y}=f({\bf x})$ in such a way that the **probability or error**, $P\{\hat{Y} \neq Y\}$ is minimum. _____no_output_____Noting that

$$
P\{\hat{Y} \neq Y\} = \int P\{\hat{Y} \neq Y | {\bf x}\} p_{\bf X}({\bf x}) d{\bf x}
$$

the optimal decision maker should take, for every sample ${\bf x}$, the decision minimizing the **conditional error probability**:

\begin{align}
\hat{y}^* &= \arg\min_{\hat{y}} P\{Y \neq \hat{y} |{\bf x}\} \\
          &= \arg\max_{\hat{y}} P\{Y = \hat{y} |{\bf x}\} \\
\end{align}_____no_output_____
Thus, the **optimal decision rule** can be expressed as

$$
P_{Y|{\bf X}}(1|{\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad  P_{Y|{\bf X}}(0|{\bf x})
$$_____no_output_____or, equivalently

$$
P_{Y|{\bf X}}(1|{\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad  \frac{1}{2}
$$

The classifier implementing this decision rule is usually referred to as the MAP (*Maximum A Posteriori*) classifier. As we have seen, the MAP classifier minimizes the error probability for binary classification, but the result can also be generalized to multiclass classification problems._____no_output_____### 1.3. Learning

**Classical decision theory** is grounded on the assumption that the probabilistic model relating the observed sample ${\bf X}$ and the true hypothesis $Y$ is known.

Unfortunately, this is unrealistic in many applications, where the only available information to construct the classifier is a **dataset** $\mathcal D = \{{\bf x}_k, y_k\}_{k=0}^{K-1}$ of instances and their respective class labels.

A more realistic formulation of the classification problem is the following: given a dataset $\mathcal D = \{({\bf x}_k, y_k) \in {\mathbb{R}}^N \times {\mathcal Y}, \, k=0,\ldots,{K-1}\}$ of independent and identically distributed (i.i.d.) samples from an ***unknown*** distribution $p_{{\bf X},Y}({\bf x}, y)$, predict the class $y$ of a new sample ${\bf x}$ with the minimum probability of error.
_____no_output_____### 1.4. Parametric classifiers

Since the probabilistic model generating the data is unknown, the MAP decision rule cannot be applied. However, we can use the dataset to **estimate the a posterior class probability model**, and apply it to approximate the MAP decision maker.

**Parametric classifiers** based on this idea assume, additionally, that the posterior class probabilty satisfies some parametric formula:

$$
P_{Y|X}(1|{\bf x},{\bf w}) = f_{\bf w}({\bf x})
$$

where ${\bf w}$ is a vector of parameters. Given the expression of the MAP decision maker, classification consists in comparing the value of $f_{\bf w}({\bf x})$ with the threshold $\frac{1}{2}$, and each parameter vector would be associated to a different decision maker.
_____no_output_____

In practice, the dataset ${\mathcal D}$ is used to select a particular parameter vector $\hat{\bf w}$ according to certain criterion. Accordingly, the decision rule becomes

$$
f_{\hat{\bf w}}({\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad  \frac{1}{2}
$$

In this notebook, we explore one of the most popular model-based parametric classification methods: **logistic regression**.

_____no_output_____## 2. Logistic regression.

### 2.1. The logistic function

The **logistic regression model** assumes that the binary class label $Y \in \{0,1\}$ of observation $X\in \mathbb{R}^N$ satisfies the expression.

$$P_{Y|{\bf X}}(1|{\bf x}, {\bf w}) = g({\bf w}^\intercal{\bf x})$$
$$P_{Y|{\bf,X}}(0|{\bf x}, {\bf w}) = 1-g({\bf w}^\intercal{\bf x})$$

where ${\bf w}$ is a parameter vector and $g(·)$ is the **logistic** function, which is defined by

$$g(t) = \frac{1}{1+\exp(-t)}$$
_____no_output_____The code below defines and plots the logistic function:_____no_output_____

# Define the logistic function
def logistic(t):
    #<SOL>
    #</SOL>

# Plot the logistic function
t = np.arange(-6, 6, 0.1)
z = logistic(t)
plt.plot(t, z)
plt.xlabel('$t$', fontsize=14)
plt.ylabel('$g(t)$', fontsize=14)
plt.title('The logistic function')
plt.grid()_____no_output_____

It is straightforward to see that the logistic function has the following properties:_____no_output_____- **P1**: Probabilistic output: $\quad 0 \le g(t) \le 1$
- **P2**: Symmetry: $\quad g(-t) = 1-g(t)$
- **P3**: Monotonicity: $\quad g'(t) = g(t)\cdot [1-g(t)] \ge 0$
_____no_output_____**Exercise 1**: Verify properties P2 and P3.

**Exercise 2**: Implement a function to compute the logistic function, and use it to plot such function in the inverval $[-6,6]$._____no_output_____### 2.2. Classifiers based on the logistic model.

The MAP classifier under a logistic model will have the form

$$P_{Y|{\bf X}}(1|{\bf x}, {\bf w}) = g({\bf w}^\intercal{\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0} \quad \frac{1}{2} $$_____no_output_____Therefore
$$
2 \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0} \quad
1 + \exp(-{\bf w}^\intercal{\bf x}) $$_____no_output_____which is equivalent to
$${\bf w}^\intercal{\bf x}
\quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad
0 $$

Thus, the classifiers based on the logistic model are given by **linear decision boundaries** passing through the origin, ${\bf x} = {\bf 0}$. _____no_output_____

# Weight vector:
w = [4, 8]   # Try different weights

# Create a rectangular grid.
x_min = -1
x_max = 1
h = (x_max - x_min) / 200
xgrid = np.arange(x_min, x_max, h)
xx0, xx1 = np.meshgrid(xgrid, xgrid)

# Compute the logistic map for the given weights, and plot
Z = logistic(w[0]*xx0 + w[1]*xx1)

fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot_surface(xx0, xx1, Z, cmap=plt.cm.copper)
ax.contour(xx0, xx1, Z, levels=[0.5], colors='b', linewidths=(3,))
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
ax.set_zlabel('P(1|x,w)')
plt.show()_____no_output_____

The next code fragment represents the output of the same classifier, representing the output of the logistic function in the $x_0$-$x_1$ plane, encoding the value of the logistic function in the color map._____no_output_____

CS = plt.contourf(xx0, xx1, Z)
CS2 = plt.contour(CS, levels=[0.5], colors='m', linewidths=(3,))
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
plt.colorbar(CS, ticks=[0, 0.5, 1])
plt.show()_____no_output_____

### 3.3. Nonlinear classifiers.

The logistic model can be extended to construct non-linear classifiers by using **non-linear data transformations**. A general form for a nonlinear logistic regression model is

$$P_{Y|{\bf X}}(1|{\bf x}, {\bf w}) = g[{\bf w}^\intercal{\bf z}({\bf x})] $$

where ${\bf z}({\bf x})$ is an arbitrary nonlinear transformation of the original variables. The boundary decision in that case is given by equation
$$
{\bf w}^\intercal{\bf z} = 0
$$_____no_output_____**Exercise 3**: Modify the code above to generate a 3D surface plot of the polynomial logistic regression model given by

$$
P_{Y|{\bf X}}(1|{\bf x}, {\bf w}) = g(1 + 10 x_0 + 10 x_1 - 20 x_0^2 + 5 x_0 x_1 + x_1^2)
$$_____no_output_____

# Weight vector:
w = [1, 10, 10, -20, 5, 1]   # Try different weights

# Create a regtangular grid.
x_min = -1
x_max = 1
h = (x_max - x_min) / 200
xgrid = np.arange(x_min, x_max, h)
xx0, xx1 = np.meshgrid(xgrid, xgrid)

# Compute the logistic map for the given weights
# Z = <FILL IN>
# Plot the logistic map
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot_surface(xx0, xx1, Z, cmap=plt.cm.copper)
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
ax.set_zlabel('P(1|x,w)')
plt.show()_____no_output_____CS = plt.contourf(xx0, xx1, Z)
CS2 = plt.contour(CS, levels=[0.5], colors='m', linewidths=(3,))
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
plt.colorbar(CS, ticks=[0, 0.5, 1])
plt.show()_____no_output_____

## 3. Inference

Remember that the idea of parametric classification is to use the training data set $\mathcal D = \{({\bf x}_k, y_k) \in {\mathbb{R}}^N \times \{0,1\}, k=0,\ldots,{K-1}\}$ to estimate ${\bf w}$. The estimate, $\hat{\bf w}$, can be used to compute the label prediction for any new observation as

$$\hat{y} = \arg\max_y P_{Y|{\bf X}}(y|{\bf x},\hat{\bf w}).$$

_____no_output_____In this notebook, we will discuss two different approaches to the estimation of ${\bf w}$:

   * **Maximum Likelihood** (ML): $\hat{\bf w}_{\text{ML}} = \arg\max_{\bf w} P_{{\mathcal D}|{\bf W}}({\mathcal D}|{\bf w})$
   * **Maximum *A Posteriori** (MAP): $\hat{\bf w}_{\text{MAP}} = \arg\max_{\bf w} p_{{\bf W}|{\mathcal D}}({\bf w}|{\mathcal D})$
_____no_output_____
For the mathematical derivation of the logistic regression algorithm, the following representation of the logistic model will be useful: using the **symmetry** property of the logistic function, we can write

$$P_{Y|{\bf X}}(0|{\bf x}, {\bf w}) = 1-g\left({\bf w}^\intercal{\bf z}({\bf x})\right)
= g\left(-{\bf w}^\intercal{\bf z}({\bf x})\right)$$

thus

$$P_{Y|{\bf X}}(y|{\bf x}, {\bf w}) = g\left(\overline{y}{\bf w}^\intercal{\bf z}({\bf x})\right)$$

where $\overline{y} = 2y-1$ is a **symmetrized label** ($\overline{y}\in\{-1, 1\}$). _____no_output_____### 3.1. Model assumptions

In the following, we will make the following assumptions:

- **A1**. (Logistic Regression): We assume a logistic model for the *a posteriori* probability of ${Y}$ given ${\bf X}$, i.e.,

$$P_{Y|{\bf X}}(y|{\bf x}, {\bf w}) = g\left({\bar y}\cdot {\bf w}^\intercal{\bf z}({\bf x})\right).$$_____no_output_____- **A2**. All samples in ${\mathcal D}$ have been generated from the same distribution, $p_{{\bf X}, Y| {\bf W}}({\bf x}, y| {\bf w})$._____no_output_____- **A3**. Input variables $\bf x$ do not depend on $\bf w$. This implies that  $p({\bf x}|{\bf w}) = p({\bf x})$_____no_output_____- **A4**. Targets $y_0, \cdots, y_{K-1}$ are statistically independent given $\bf w$ and the inputs ${\bf x}_0, \cdots, {\bf x}_{K-1}$, that is:

$$P(y_0, \cdots, y_{K-1} | {\bf x}_0, \cdots, {\bf x}_{K-1}, {\bf w}) = \prod_{k=0}^{K-1} P(y_k | {\bf x}_k, {\bf w})$$_____no_output_____### 3.2. ML estimation.

The ML estimate is defined as

$$\hat{\bf w}_{\text{ML}} = \arg\max_{\bf w} P_{{\mathcal D}|{\bf W}}({\mathcal D}|{\bf w})$$_____no_output_____Ussing assumptions A2 and A3 above, we have that

\begin{align}
P_{{\mathcal D}|{\bf W}}({\mathcal D}|{\bf w}) & =  p(y_0, \cdots, y_{K-1},{\bf x}_0, \cdots, {\bf x}_{K-1}| {\bf w}) \\
& =  P(y_0, \cdots, y_{K-1}|{\bf x}_0, \cdots, {\bf x}_{K-1}, {\bf w}) \; p({\bf x}_0, \cdots, {\bf x}_{K-1}| {\bf w}) \\
& = P(y_0, \cdots, y_{K-1}|{\bf x}_0, \cdots, {\bf x}_{K-1}, {\bf w}) \; p({\bf x}_0, \cdots, {\bf x}_{K-1})\end{align}_____no_output_____Finally, using assumption A4, we can formulate the ML estimation of $\bf w$ as the resolution of the following **optimization problem**

\begin{align}
\hat {\bf w}_\text{ML} & = \arg \max_{\bf w} P(y_0, \cdots, y_{K-1}|{\bf x}_0, \cdots, {\bf x}_{K-1}, {\bf w}) \\
& = \arg \max_{\bf w} \prod_{k=0}^{K-1} P(y_k|{\bf x}_k, {\bf w}) \\
& = \arg \max_{\bf w} \sum_{k=0}^{K-1} \log P(y_k|{\bf x}_k, {\bf w}) \\
& = \arg \min_{\bf w} \sum_{k=0}^{K-1} - \log P(y_k|{\bf x}_k, {\bf w})
\end{align}

where the arguments of the maximization or minimization problems of the last three lines are usually referred to as the **likelihood**, **log-likelihood** $\left[L(\bf w)\right]$, and **negative log-likelihood** $\left[\text{NLL}(\bf w)\right]$, respectively._____no_output_____
Now, using A1 (the logistic model)

\begin{align}
\text{NLL}({\bf w})
    &= - \sum_{k=0}^{K-1}\log\left[g\left(\overline{y}_k{\bf w}^\intercal {\bf z}_k\right)\right]   \\
    &= \sum_{k=0}^{K-1}\log\left[1+\exp\left(-\overline{y}_k{\bf w}^\intercal {\bf z}_k\right)\right]
\end{align}

where ${\bf z}_k={\bf z}({\bf x}_k)$.
_____no_output_____
It can be shown that $\text{NLL}({\bf w})$ is a **convex** and **differentiable** function of ${\bf w}$. Therefore, its minimum is a point with zero gradient.

\begin{align}
\nabla_{\bf w} \text{NLL}(\hat{\bf w}_{\text{ML}})
    &= - \sum_{k=0}^{K-1}
       \frac{\exp\left(-\overline{y}_k\hat{\bf w}_{\text{ML}}^\intercal {\bf z}_k\right) \overline{y}_k {\bf z}_k}
       {1+\exp\left(-\overline{y}_k\hat{\bf w}_{\text{ML}}^\intercal {\bf z}_k
       \right)} = \\
    &= - \sum_{k=0}^{K-1} \left[y_k-g(\hat{\bf w}_{\text{ML}}^T {\bf z}_k)\right] {\bf z}_k = 0
\end{align}

Unfortunately, $\hat{\bf w}_{\text{ML}}$ cannot be taken out from the above equation, and some iterative optimization algorithm must be used to search for the minimum._____no_output_____### 3.3. Gradient descent.

A simple iterative optimization algorithm is  gradient descent.

\begin{align}
{\bf w}_{n+1} = {\bf w}_n - \rho_n \nabla_{\bf w} \text{NLL}({\bf w}_n)
\end{align}

where $\rho_n >0$ is the *learning step*.

Applying the gradient descent rule to logistic regression, we get the following algorithm:

\begin{align}
{\bf w}_{n+1} &= {\bf w}_n
    + \rho_n \sum_{k=0}^{K-1} \left[y_k-g({\bf w}_n^\intercal {\bf z}_k)\right] {\bf z}_k
\end{align}
_____no_output_____#### Gradient descent in matrix form

Defining vectors

\begin{align}
{\bf y} &= [y_0,\ldots,y_{K-1}]^\top \\
\hat{\bf p}_n &= [g({\bf w}_n^\top {\bf z}_0), \ldots, g({\bf w}_n^\top {\bf z}_{K-1})]^\top
\end{align}
and matrix
\begin{align}
{\bf Z} = \left[{\bf z}_0,\ldots,{\bf z}_{K-1}\right]^\top
\end{align}

we can write

\begin{align}
{\bf w}_{n+1} &= {\bf w}_n
    + \rho_n {\bf Z}^\top \left({\bf y}-\hat{\bf p}_n\right)
\end{align}

In the following, we will explore the behavior of the gradient descend method using the Iris Dataset._____no_output_____

# Adapted from a notebook by Jason Brownlee
def loadDataset(filename, split):
    xTrain, cTrain, xTest, cTest = [], [], [], []

    with open(filename, 'r') as csvfile:
        lines = csv.reader(csvfile)
        dataset = list(lines)
    for i in range(len(dataset)-1):
        for y in range(4):
            dataset[i][y] = float(dataset[i][y])
        item = dataset[i]
        if random.random() < split:
            xTrain.append(item[0:4])
            cTrain.append(item[4])
        else:
            xTest.append(item[0:4])
            cTest.append(item[4])
    return xTrain, cTrain, xTest, cTest

xTrain_all, cTrain_all, xTest_all, cTest_all = loadDataset('iris.data', 0.66)
nTrain_all = len(xTrain_all)
nTest_all = len(xTest_all)
print('Train:', nTrain_all)
print('Test:', nTest_all)_____no_output_____

Now, we select two classes and two attributes._____no_output_____

# Select attributes
i = 0 # Try 0,1,2,3
j = 1 # Try 0,1,2,3 with j!=i

# Select two classes
c0 = 'Iris-versicolor'
c1 = 'Iris-virginica'
# Select two coordinates
ind = [i, j]

# Take training test
X_tr = np.array([[xTrain_all[n][i] for i in ind] for n in range(nTrain_all)
                  if cTrain_all[n]==c0 or cTrain_all[n]==c1])
C_tr = [cTrain_all[n] for n in range(nTrain_all)
          if cTrain_all[n]==c0 or cTrain_all[n]==c1]
Y_tr = np.array([int(c==c1) for c in C_tr])
n_tr = len(X_tr)

# Take test set
X_tst = np.array([[xTest_all[n][i] for i in ind] for n in range(nTest_all)
                 if cTest_all[n]==c0 or cTest_all[n]==c1])
C_tst = [cTest_all[n] for n in range(nTest_all)
         if cTest_all[n]==c0 or cTest_all[n]==c1]
Y_tst = np.array([int(c==c1) for c in C_tst])
n_tst = len(X_tst)_____no_output_____

#### 3.2.2. Data normalization

Normalization of data is a common pre-processing step in many machine learning algorithms. Its goal is to get a dataset where all input coordinates have a similar scale. Learning algorithms usually show less instabilities and convergence problems when data are normalized.

We will define a normalization function that returns a training data matrix with zero sample mean and unit sample variance._____no_output_____

def normalize(X, mx=None, sx=None):
    # Compute means and standard deviations
    if mx is None:
        mx = np.mean(X, axis=0)
    if sx is None:
        sx = np.std(X, axis=0)

    # Normalize
    X0 = (X-mx)/sx

    return X0, mx, sx_____no_output_____

Now, we can normalize training and test data. Observe in the code that **the same transformation should be applied to training and test data**. This is the reason why normalization with the test data is done using the means and the variances computed with the training set._____no_output_____

# Normalize data
Xn_tr, mx, sx = normalize(X_tr)
Xn_tst, mx, sx = normalize(X_tst, mx, sx)_____no_output_____

The following figure generates a plot of the normalized training data._____no_output_____

# Separate components of x into different arrays (just for the plots)
x0c0 = [Xn_tr[n][0] for n in range(n_tr) if Y_tr[n]==0]
x1c0 = [Xn_tr[n][1] for n in range(n_tr) if Y_tr[n]==0]
x0c1 = [Xn_tr[n][0] for n in range(n_tr) if Y_tr[n]==1]
x1c1 = [Xn_tr[n][1] for n in range(n_tr) if Y_tr[n]==1]

# Scatterplot.
labels = {'Iris-setosa': 'Setosa', 'Iris-versicolor': 'Versicolor',
          'Iris-virginica': 'Virginica'}
plt.plot(x0c0, x1c0,'r.', label=labels[c0])
plt.plot(x0c1, x1c1,'g+', label=labels[c1])
plt.xlabel('$x_' + str(ind[0]) + '$')
plt.ylabel('$x_' + str(ind[1]) + '$')
plt.legend(loc='best')
plt.axis('equal')
plt.show()_____no_output_____

In order to apply the gradient descent rule, we need to define two methods:
 - A `fit` method, that receives the training data and returns the model weights and the value of the negative log-likelihood during all iterations.
 - A `predict` method, that receives the model weight and a set of inputs, and returns the posterior class probabilities for that input, as well as their corresponding class predictions._____no_output_____

def logregFit(Z_tr, Y_tr, rho, n_it):

    # Data dimension
    n_dim = Z_tr.shape[1]
    # Initialize variables
    nll_tr = np.zeros(n_it)
    pe_tr = np.zeros(n_it)
    Y_tr2 = 2*Y_tr - 1     # Transform labels into binary symmetric.
    w = np.random.randn(n_dim,1)

    # Running the gradient descent algorithm
    for n in range(n_it):
        # Compute posterior probabilities for weight w
        p1_tr = logistic(np.dot(Z_tr, w))
        # Compute negative log-likelihood
        # (note that this is not required for the weight update, only for nll tracking)
        nll_tr[n] = np.sum(np.log(1 + np.exp(-np.dot(Y_tr2*Z_tr, w))))
        # Update weights
        w += rho*np.dot(Z_tr.T, Y_tr - p1_tr)

    return w, nll_tr

def logregPredict(Z, w):
    # Compute posterior probability of class 1 for weights w.
    p = logistic(np.dot(Z, w)).flatten()
    # Class
    D = [int(round(pn)) for pn in p]

    return p, D_____no_output_____

We can test the behavior of the gradient descent method by fitting a logistic regression model with ${\bf z}({\bf x}) = (1, {\bf x}^\top)^\top$._____no_output_____

# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 200   # Number of iterations

# Compute Z's
Z_tr = np.c_[np.ones(n_tr), Xn_tr]
Z_tst = np.c_[np.ones(n_tst), Xn_tst]
n_dim = Z_tr.shape[1]

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit(Z_tr, Y_tr2, rho, n_it)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst_____no_output_____# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()

print(f'The optimal weights are: {w}')
print('The final error rates are:')
print(f'- Training: {pe_tr}')
print(f'- Test: {pe_tst}')
print(f'The NLL after training is {nll_tr[len(nll_tr)-1]}')_____no_output_____

#### 3.2.3. Free parameters

Under certain conditions, the gradient descent method can be shown to converge asymptotically (i.e. as the number of iterations goes to infinity) to the ML estimate of the logistic model. However, in practice, the final estimate of the weights ${\bf w}$ depend on several factors:

- Number of iterations
- Initialization
- Learning step_____no_output_____**Exercise 4**: Visualize the variability of gradient descent caused by initializations. To do so, fix the number of iterations to 200 and the learning step, and execute the gradient descent 100 times, storing the training error rate of each execution. Plot the histogram of the error rate values.

Note that you can do this exercise with a loop over the 100 executions, including the code in the previous code slide inside the loop, with some proper modifications. To plot a histogram of the values in array `p` with `n`bins, you can use `plt.hist(p, n)`_____no_output_____##### 3.2.3.1. Learning step

The learning step, $\rho$, is a free parameter of the algorithm. Its choice is critical for the convergence of the algorithm. Too large values of $\rho$ make the algorithm diverge. For too small values, the convergence gets very slow and more iterations are required for a good convergence.
_____no_output_____**Exercise 5**: Observe the evolution of the negative log-likelihood with the number of iterations for different values of $\rho$. It is easy to check that, for large enough $\rho$, the gradient descent method does not converge. Can you estimate (through manual observation) an approximate value of $\rho$ stating a boundary between convergence and divergence?_____no_output_____**Exercise 6**: In this exercise we explore the influence of the learning step more sistematically. Use the code in the previouse exercises to compute, for every value of $\rho$, the average error rate over 100 executions. Plot the average error rate vs. $\rho$.

Note that you should explore the values of $\rho$ in a logarithmic scale. For instance, you can take $\rho = 1, \frac{1}{10}, \frac{1}{100}, \frac{1}{1000}, \ldots$_____no_output_____In practice, the selection of $\rho$ may be a matter of trial an error. Also there is some theoretical evidence that the learning step should decrease along time up to cero, and the sequence $\rho_n$ should satisfy two conditions:
- C1: $\sum_{n=0}^{\infty} \rho_n^2 < \infty$ (decrease slowly)
- C2: $\sum_{n=0}^{\infty} \rho_n = \infty$ (but not too slowly)

For instance, we can take $\rho_n= \frac{1}{n}$. Another common choice is $\rho_n = \frac{\alpha}{1+\beta n}$ where $\alpha$ and $\beta$ are also free parameters that can be selected by trial and error with some heuristic method._____no_output_____#### 3.2.4. Visualizing the posterior map.

We can also visualize the posterior probability map estimated by the logistic regression model for the estimated weights._____no_output_____

# Create a regtangular grid.
x_min, x_max = Xn_tr[:, 0].min(), Xn_tr[:, 0].max()
y_min, y_max = Xn_tr[:, 1].min(), Xn_tr[:, 1].max()
dx = x_max - x_min
dy = y_max - y_min
h = dy /400
xx, yy = np.meshgrid(np.arange(x_min - 0.1 * dx, x_max + 0.1 * dx, h),
                     np.arange(y_min - 0.1 * dx, y_max + 0.1 * dy, h))
X_grid = np.array([xx.ravel(), yy.ravel()]).T

# Compute Z's
Z_grid = np.c_[np.ones(X_grid.shape[0]), X_grid]

# Compute the classifier output for all samples in the grid.
pp, dd = logregPredict(Z_grid, w)_____no_output_____# Paint output maps
pylab.rcParams['figure.figsize'] = 6, 6  # Set figure size
# Color plot
plt.plot(x0c0, x1c0,'r.', label=labels[c0])
plt.plot(x0c1, x1c1,'g+', label=labels[c1])
plt.xlabel('$x_' + str(ind[0]) + '$')
plt.ylabel('$x_' + str(ind[1]) + '$')
plt.legend(loc='best')
plt.axis('equal')
pp = pp.reshape(xx.shape)
CS = plt.contourf(xx, yy, pp, cmap=plt.cm.copper)
plt.contour(xx, yy, pp, levels=[0.5], colors='b', linewidths=(3,))
plt.colorbar(CS, ticks=[0, 0.5, 1])
plt.show()_____no_output_____

#### 3.2.5. Polynomial Logistic Regression

The error rates of the logistic regression model can be potentially reduced by using polynomial transformations.

To compute the polynomial transformation up to a given degree, we can use the `PolynomialFeatures` method in `sklearn.preprocessing`._____no_output_____

# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 500   # Number of iterations
g = 5 # Degree of polynomial

# Compute Z_tr
poly = PolynomialFeatures(degree=g)
Z_tr = poly.fit_transform(Xn_tr)
# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr,1)), Zn), axis=1)

# Compute Z_tst
Z_tst = poly.fit_transform(Xn_tst)
Zn, mz, sz = normalize(Z_tst[:,1:], mz, sz)
Z_tst = np.concatenate((np.ones((n_tst,1)), Zn), axis=1)

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit(Z_tr, Y_tr2, rho, n_it)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst_____no_output_____# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()
print(f'The optimal weights are: {w.T}')
print('The final error rates are:')
print(f'- Training: {pe_tr} \n- Test: {pe_tst}')
print('The NLL after training is', nll_tr[len(nll_tr)-1])_____no_output_____

Visualizing the posterior map we can se that the polynomial transformation produces nonlinear decision boundaries._____no_output_____

# Compute Z_grid
Z_grid = poly.fit_transform(X_grid)
Zn, mz, sz = normalize(Z_grid[:,1:], mz, sz)
Z_grid = np.concatenate((np.ones((Z_grid.shape[0],1)), Zn), axis=1)

# Compute the classifier output for all samples in the grid.
pp, dd = logregPredict(Z_grid, w)
pp = pp.reshape(xx.shape)_____no_output_____# Paint output maps
pylab.rcParams['figure.figsize'] = 6, 6  # Set figure size
plt.plot(x0c0, x1c0,'r.', label=labels[c0])
plt.plot(x0c1, x1c1,'g+', label=labels[c1])
plt.xlabel('$x_' + str(ind[0]) + '$')
plt.ylabel('$x_' + str(ind[1]) + '$')
plt.axis('equal')
plt.legend(loc='best')
CS = plt.contourf(xx, yy, pp, cmap=plt.cm.copper)
plt.contour(xx, yy, pp, levels=[0.5], colors='b', linewidths=(3,))
plt.colorbar(CS, ticks=[0, 0.5, 1])
plt.show()_____no_output_____

## 4. Regularization and MAP estimation.

### 4.1 MAP estimation

An alternative to the ML estimation of the weights in logistic regression is Maximum A Posteriori estimation. Modelling the logistic regression weights as a random variable with prior distribution $p_{\bf W}({\bf w})$, the **MAP estimate** is defined as

$$
\hat{\bf w}_{\text{MAP}} = \arg\max_{\bf w} p({\bf w}|{\mathcal D})
$$

The posterior density $p({\bf w}|{\mathcal D})$ is related to the likelihood function and the prior density of the weights, $p_{\bf W}({\bf w})$ through the **Bayes rule**

$$
p({\bf w}|{\mathcal D}) =
    \frac{P\left({\mathcal D}|{\bf w}\right) \; p_{\bf W}({\bf w})}
         {p\left({\mathcal D}\right)}
$$_____no_output_____In general, the denominator in this expression cannot be computed analytically. However, it is not required for MAP estimation because it does not depend on ${\bf w}$. Therefore, the MAP solution is given by

\begin{align}
\hat{\bf w}_{\text{MAP}} & = \arg\max_{\bf w} \left\{ P\left({\mathcal D}|{\bf w}\right) \; p_{\bf W}({\bf w}) \right\}\\
& = \arg\max_{\bf w} \left\{ L({\mathbf w}) + \log p_{\bf W}({\bf w})\right\} \\
& = \arg\min_{\bf w} \left\{ \text{NLL}({\mathbf w}) - \log p_{\bf W}({\bf w})\right\}
\end{align}

In the light of this expression, we can conclude that the MAP solution is affected by two terms:
   - The likelihood, which takes large values for parameter vectors $\bf w$ that fit well the training data (smaller $\text{NLL}$ values)
   - The prior distribution of weights $p_{\bf W}({\bf w})$, which expresses our *a priori* preference for some solutions.
   _____no_output_____### 4.2. Regularization

Even though the prior distribution has a natural interpretation as a model of our knowledge about $p({\bf w})$ before observing the data, its choice is frequenty motivated by the need to avoid data **overfitting**.

**Data overfitting** is a frequent problem in ML estimation when the dimension of ${\bf w}$ is much higher that the dimension of the input ${\bf x}$: the ML solution can be too adjusted to the training data, while the test error rate is large.

In practice **we recur to prior distributions that take large values when $\|{\bf w}\|$ is small** (associated to smooth classification borders). This helps to improve **generalization**.

In this way, the MAP criterion adds a **penalty term** to the ML objective, that penalizes parameter vectors for which the prior distribution of weights takes small values.

In machine learning, the process of introducing penalty terms to avoid overfitting is usually named **regularization**. _____no_output_____### 4.3 MAP estimation with Gaussian prior

If we assume that ${\bf W}$ follows a **zero-mean Gaussian** random variable with variance matrix $v{\bf I}$,

$$
p_{\bf W}({\bf w}) = \frac{1}{(2\pi v)^{N/2}} \exp\left(-\frac{1}{2v}\|{\bf w}\|^2\right)
$$

the **MAP estimate** becomes

\begin{align}
\hat{\bf w}_{\text{MAP}}
  &= \arg\min_{\bf w} \left\{\text{NLL}({\bf w}) + \frac{1}{C}\|{\bf w}\|^2
         \right\}
\end{align}

where $C = 2v$. Note that the **regularization term** associated to the prior penalizes parameter vectors with large components. Parameter $C$ controls the regularizatin, and it is named the **inverse regularization strength**._____no_output_____Noting that

$$\nabla_{\bf w}\left\{\text{NLL}({\bf w}) + \frac{1}{C}\|{\bf w}\|^2\right\}
= - {\bf Z} \left({\bf y}-\hat{\bf p}_n\right) + \frac{2}{C}{\bf w},
$$

we obtain the following **gradient descent rule** for MAP estimation

\begin{align}
{\bf w}_{n+1} &= \left(1-\frac{2\rho_n}{C}\right){\bf w}_n
    + \rho_n {\bf Z} \left({\bf y}-\hat{\bf p}_n\right)
\end{align}

Note that the regularization term "pushes" the weights towards zero.
_____no_output_____### 4.4 MAP estimation with Laplacian prior

If we assume that ${\bf W}$ follows a multivariate zero-mean Laplacian distribution given by

$$
p_{\bf W}({\bf w}) = \frac{1}{(2 C)^{N}} \exp\left(-\frac{1}{C}\|{\bf w}\|_1\right)
$$

(where $\|{\bf w}\|=|w_1|+\ldots+|w_N|$ is the $L_1$ norm of ${\bf w}$), the MAP estimate becomes

\begin{align}
\hat{\bf w}_{\text{MAP}}
  &= \arg\min_{\bf w} \left\{\text{NLL}({\bf w}) + \frac{1}{C}\|{\bf w}\|_1
         \right\}
\end{align}

Parameter $C$ is named the *inverse regularization strength*._____no_output_____**Exercise 7**: Derive the gradient descent rules for MAP estimation of the logistic regression weights with Laplacian prior._____no_output_____## 5. Other optimization algorithms

### 5.1. Stochastic Gradient descent.

Stochastic gradient descent (SGD) is based on the idea of using a single sample at each iteration of the learning algorithm. The SGD rule for ML logistic regression is

\begin{align}
{\bf w}_{n+1} &= {\bf w}_n
    + \rho_n {\bf z}_n \left(y_n-\hat{p}_n\right)
\end{align}

Once all samples in the training set have been applied, the algorith can continue by applying the training set several times.

The computational cost of each iteration of SGD is much smaller than that of gradient descent, though it usually needs many more iterations to converge._____no_output_____**Exercise 8**: Modify logregFit to implement an algorithm that applies the SGD rule._____no_output_____### 5.2. Newton's method

Assume that the function to be minimized, $C({\bf w})$, can be approximated by its **second order Taylor series expansion** around ${\bf w}_0$

$$
C({\bf w}) \approx C({\bf w}_0)
+ \nabla_{\bf w}^\top C({\bf w}_0)({\bf w}-{\bf w}_0)
+ \frac{1}{2}({\bf w}-{\bf w}_0)^\top{\bf H}({\bf w}_0)({\bf w}-{\bf w}_0)
$$

where ${\bf H}({\bf w})$ is the  **Hessian matrix** of $C$ at ${\bf w}$. Taking the gradient of $C({\bf w})$, and setting the result to ${\bf 0}$, the minimum of C around ${\bf w}_0$ can be approximated as

$$
{\bf w}^* = {\bf w}_0 - {\bf H}({\bf w}_0)^{-1} \nabla_{\bf w}^\top C({\bf w}_0)
$$

Since the second order polynomial is only an approximation to $C$, ${\bf w}^*$ is only an approximation to the optimal weight vector, but we can expect ${\bf w}^*$ to be closer to the minimizer of $C$ than ${\bf w}_0$. Thus, we can repeat the process, computing a second order approximation around ${\bf w}^*$ and a new approximation to the minimizer._____no_output_____
 **Newton's method** is based on this idea. At each optimization step, the function to be minimized is approximated by a second order approximation using a Taylor series expansion around the current estimate. As a result, the learning rule becomes

$$\hat{\bf w}_{n+1} = \hat{\bf w}_{n} - \rho_n {\bf H}({\bf w}_n)^{-1} \nabla_{{\bf w}}C({\bf w}_n)
$$_____no_output_____#### 5.2.1. Example: MAP estimation with Gaussian prior.

For instance, for the MAP estimate with Gaussian prior, the *Hessian* matrix becomes

$$
{\bf H}({\bf w})
  = \frac{2}{C}{\bf I}
  + \sum_{k=0}^{K-1} g({\bf w}^\top {\bf z}_k)
                     \left[1-g({\bf w}^\top {\bf z}_k)\right]{\bf z}_k {\bf z}_k^\top
$$

Defining diagonal matrix

$$
{\mathbf S}({\bf w}) = \text{diag}\left[g({\bf w}^\top {\bf z}_k) \left(1-g({\bf w}^\top {\bf z}_k)\right)\right]
$$

the Hessian matrix can be written in more compact form as

$$
{\bf H}({\bf w})
  = \frac{2}{C}{\bf I} + {\bf Z}^\top {\bf S}({\bf w}) {\bf Z}
$$

Therefore, the Newton's algorithm for logistic regression becomes

\begin{align}
{\bf w}_{n+1} = {\bf w}_{n} +
\rho_n
\left(\frac{2}{C}{\bf I} + {\bf Z}^\top {\bf S}({\bf w}_{n})
{\bf Z}
\right)^{-1}
{\bf Z}^\top \left({\bf y}-\hat{\bf p}_n\right)
\end{align}

Some variants of the Newton method are implemented in the  Scikit-learn  package.

_____no_output_____

def logregFit2(Z_tr, Y_tr, rho, n_it, C=1e4):

    # Compute Z's
    r = 2.0/C
    n_dim = Z_tr.shape[1]

    # Initialize variables
    nll_tr = np.zeros(n_it)
    pe_tr = np.zeros(n_it)
    w = np.random.randn(n_dim,1)

    # Running the gradient descent algorithm
    for n in range(n_it):
        p_tr = logistic(np.dot(Z_tr, w))

        sk = np.multiply(p_tr, 1-p_tr)
        S = np.diag(np.ravel(sk.T))

        # Compute negative log-likelihood
        nll_tr[n] = - np.dot(Y_tr.T, np.log(p_tr)) - np.dot((1-Y_tr).T, np.log(1-p_tr))

        # Update weights
        invH = np.linalg.inv(r*np.identity(n_dim) + np.dot(Z_tr.T, np.dot(S, Z_tr)))

        w += rho*np.dot(invH, np.dot(Z_tr.T, Y_tr - p_tr))

    return w, nll_tr_____no_output_____# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 500   # Number of iterations
C = 1000
g = 4

# Compute Z_tr
poly = PolynomialFeatures(degree=g)
Z_tr = poly.fit_transform(X_tr)
# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr,1)), Zn), axis=1)

# Compute Z_tst
Z_tst = poly.fit_transform(X_tst)
Zn, mz, sz = normalize(Z_tst[:,1:], mz, sz)
Z_tst = np.concatenate((np.ones((n_tst,1)), Zn), axis=1)

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit2(Z_tr, Y_tr2, rho, n_it, C)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst_____no_output_____# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()

print('The final error rates are:')
print('- Training:', str(pe_tr))
print('- Test:', str(pe_tst))
print('The NLL after training is:', str(nll_tr[len(nll_tr)-1]))_____no_output_____

## 6. Logistic regression in Scikit Learn.

The  scikit-learn  package includes an efficient implementation of  logistic regression. To use it, we must first create a classifier object, specifying the parameters of the logistic regression algorithm._____no_output_____

# Create a logistic regression object.
LogReg = linear_model.LogisticRegression(C=1.0)

# Compute Z_tr
poly = PolynomialFeatures(degree=g)
Z_tr = poly.fit_transform(Xn_tr)
# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr,1)), Zn), axis=1)

# Compute Z_tst
Z_tst = poly.fit_transform(Xn_tst)
Zn, mz, sz = normalize(Z_tst[:,1:], mz, sz)
Z_tst = np.concatenate((np.ones((n_tst,1)), Zn), axis=1)

# Fit model to data.
LogReg.fit(Z_tr, Y_tr)

# Classify training and test data
D_tr = LogReg.predict(Z_tr)
D_tst = LogReg.predict(Z_tst)_____no_output_____# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

print('The final error rates are:')
print('- Training:', str(pe_tr))
print('- Test:', str(pe_tst))

# Compute Z_grid
Z_grid = poly.fit_transform(X_grid)
n_grid = Z_grid.shape[0]
Zn, mz, sz = normalize(Z_grid[:,1:], mz, sz)
Z_grid = np.concatenate((np.ones((n_grid,1)), Zn), axis=1)_____no_output_____# Compute the classifier output for all samples in the grid.
dd = LogReg.predict(Z_grid)
pp = LogReg.predict_proba(Z_grid)[:,1]
pp = pp.reshape(xx.shape)

# Paint output maps
pylab.rcParams['figure.figsize'] = 6, 6  # Set figure size
plt.plot(x0c0, x1c0,'r.', label=labels[c0])
plt.plot(x0c1, x1c1,'g+', label=labels[c1])
plt.xlabel('$x_' + str(ind[0]) + '$')
plt.ylabel('$x_' + str(ind[1]) + '$')
plt.axis('equal')
plt.contourf(xx, yy, pp, cmap=plt.cm.copper)
plt.legend(loc='best')
plt.contour(xx, yy, pp, levels=[0.5], colors='b', linewidths=(3,))
plt.colorbar(CS, ticks=[0, 0.5, 1])
plt.show()_____no_output_____


# Notebook from ingolia/mcb280-2021
Path: 02_statistics/02_b_allele from class.ipynb

### Allele-specific expression

RNA sequencing can distinguish mRNAs expressed from from different copies of genes on homologous chromosomes when single-nucleotide polymorphisms (perhaps silent) distinguish the two alleles.

Linkage between these distinctive SNPs and _cis_-regulatory sequences can provide information on regulatory variation within a shared cellular context.

In the example below, the maternally derived copy of the gene has a `TGC` serine codon at the 2nd position and a functioning enhancer. The paternally derived allele has a `TGT` serine codon and a mutated enhancer.

```
Maternal: ----enhancer----ATGTGCACT...

Paternal: ----exhxnxer----ATGTGTACT...
```

Because the paternal enhancer is mutated, we would expect most mRNA to derive from the maternal allele and thus most RNA-Seq reads should have a `C` rather than a `T`._____no_output_____### Null hypothesis testing

The _null hypothesis_ in allele-specific expression analysis is that the alleles are expressed equally, and so each read is equally likely to be derived from each allele.

Our actual data comes from _counting_ how many RNA-seq reads have the maternally-derived `C` variant and how many have the paternally-derived `T` variant.

We want to build a complex probabilistic model of these read counts from a simpler model:
* each read is equally likely to be a `C` or a `T`
* each read is independent

We will use this to ask — if we see an imbalance between maternally- and paternally-derived alleles — whether this imbalance is very unlikely to occur in our null model that reads are equally likely to come from each allele.

We will do this in two ways: first we will actually perform random sampling, and second we will use a mathematical formula that fits this kind of model._____no_output_____#### Random sampling

We begin by importing pandas_____no_output_____

import pandas as pd_____no_output_____

Now we create a pandas `Series` object containing the two possible alleles.

The _values_ will be the two possible genotypes and the _indexes_ will help us remember that one variant is maternally-derived and the other paternally-derived._____no_output_____

alleles = pd.Series(['C', 'T'], index=['mat', 'pat'])
alleles_____no_output_____

We can use the `sample` method to take one (by default) or more (using the `n=` parameter) random allele choices.

When we're taking more than one random sample, we also need to use the `replace=True` parameter because the default is sampling without replacement, but sampling with replacement — that is, being able to randomly choose a given allele more than once — better models gene expression.

We can then use the `value_counts` method to count up the number of maternal and paternal alleles seen across the randomly sampled reads._____no_output_____

alleles.sample(n=8, replace=True).value_counts()['T']_____no_output_____

Next we want to run this sampling procedure 10,000 times and count up how often we see zero `T` alleles out of eight reads, how often we see one `T` allele, and so forth.

To do this, we will:
* write a `for` loop to run 10,000 times
* sample alleles from 8 reads
* count `T` alleles in our sample (use the `get` method with default value 0)
* add the count of `T` alleles to a list of sample counts
* after generating all of our sample counts, convert them to a `Series` and find the value counts._____no_output_____

sample_paternal_counts = []
for _ in range(0,10000):
    paternal_count = alleles.sample(n=8, replace=True).value_counts().get('T', 0)
    sample_paternal_counts.append(paternal_count)
paternal_freqs = pd.Series(sample_paternal_counts).value_counts()
paternal_freqs_____no_output_____

We can divide these by the total number of random samples we generated in order to get a normalized probability.

We can also sort them for easier reading using `sort_index`_____no_output_____

paternal_probs = paternal_freqs / 10000
paternal_probs = paternal_probs.sort_index()
paternal_probs_____no_output_____

We can compute the probability of seeing _exactly_ 2 paternally-derived alleles_____no_output_____

paternal_probs[2]_____no_output_____

We can also compute the probability of seeing _no more than_ 2 paternally-derived alelles_____no_output_____

paternal_probs[0:3].sum()_____no_output_____

Lastly, we'll use matplotlib to plot this distribution._____no_output_____

import matplotlib.pyplot as plt
plt.plot(paternal_probs)_____no_output_____

Later, we'll want to look at random samples taken with different numbers of reads. We'll also want to control the number of samples we take. To simplify this, we'll package up all of our random sampling code into a function that has parameters to control the number of reads in each sample and the total number of samples._____no_output_____

# We didn't get to this in class

def sample_paternal_probs(nread=8, nsample=10000):
    alleles = pd.Series(['C', 'T'])
    sample_counts = []
    for _ in range(0, nsample):
        maternal_count = alleles.sample(n = nread, replace=True).value_counts().get('T', 0)
        sample_counts.append(maternal_count)
    sample_freqs = pd.Series(sample_counts).value_counts().sort_index()
    return(sample_freqs / nsample)

_____no_output_____

To test our function, we will sample probabilities for 16 reads._____no_output_____

sample_paternal_counts = []
for _ in range(0,10000):
    paternal_count = alleles.sample(n=16, replace=True).value_counts().get('T', 0)
    sample_paternal_counts.append(paternal_count)
paternal_freqs = pd.Series(sample_paternal_counts).value_counts()
paternal_probs_16 = paternal_freqs / 10000
paternal_probs_16 = paternal_probs_16.sort_index()
paternal_probs_16_____no_output_____

#### The binomial distribution

Our probabilistic model of counting up the results of several independent trials is very common in probability. From a mathematical perspective, it's the same as counting the number of heads that come up out of _N_ total coin tosses.

This number follows a _binomial distribution_. We can compute
$$P(\;k\;\text{heads out of}\;N\;\text{coin tosses})$$
using
```
binom.pmf(k, N, 0.5)
```
The `0.5` parameter here is the probability of a head for each coin toss — the `binom.pmf` function can compute probabilities for a _biased_ coin where the chances of a head are not exactly $\frac{1}{2}$.

The `binom.pmf` function comes from the `binom` module in `scipy.stats`. We'll import this and then compute a probability under the binomial model for exactly 2 paternally-derived reads out of 8 coin tosses._____no_output_____

from scipy.stats import binom
binom.pmf(2, 8, 0.5)_____no_output_____

This binomial probability is very close to the value we got by random sampling!

We can also construct the whole table of binomial probabilities by using a `range()` for the $k$ parameter._____no_output_____

binom_probs_8 = pd.Series(binom.pmf(range(0, 9), 8, 0.5), index=range(0,9))
binom_probs_8_____no_output_____

We can plot the binomial probabilities and the random sampling probabilities on the same graph and compare them.

We can also control the plot styles: I'll use the `'b-'` parameter for blue lines and the `'ro'` parameter for red dots. _____no_output_____

plt.plot(binom_probs_8, 'b-')
plt.plot(paternal_probs, 'ro')_____no_output_____

#### More reads give more certainty

In this binomial model — and in other more realistic models of deep sequencing data — greater sequencing depth gives more statistical power to reject the null hypothesis.

To see this effect in action for the binomial model of allele-specific expression, consider two possibilities:
* No more than one quarter of 8 reads are paternally-derived
* No more than one quarter of 16 reads are paternally-derived

In both cases, the _magnitude_ of the paternal vs maternal skew is 1:3._____no_output_____

# We didn't get to this in class
print('8 reads: ' + str(binom.pmf(range(0,3), 8, 0.5).sum()))
print('16 reads: ' + str(binom.pmf(range(0, 5), 16, 0.5).sum()))
8 reads: 0.14453125000000003
16 reads: 0.0384063720703125


# Notebook from callysto/curriculum-notebooks
Path: Science/BioinformaticsofGMOs/GMOs-in-space.ipynb

![Callysto.ca Banner](https://github.com/callysto/curriculum-notebooks/blob/master/callysto-notebook-banner-top.jpg?raw=true)

_____no_output_____# 🚀 Spacer's Challenge : Using Data and Biology to Decide What We Should Grow On Mars

## 📌 Mission Statement

You are a top scientist sent from Earth tasked with the most important mission of all: ensuring our survival on other planets. Before sending you off to space, your team gave you a variety of crop seeds, however there's a problem! The bag containing the crop seeds split open and spilled everywhere during your space travel! You picked up the ones you could find, but you now have no idea what it grows or what is special about them.

Now on Mars, it is your job to [decide which crops to grow](https://www.sciencealert.com/growing-potatoes-on-mars-might-actually-work-hints-a-new-experiment). Consider crops that are tolerant to extreme conditions like those they would experience on Mars. We will be using some basic bioinformatics concepts to try and find this out. To help you with this, you will work with a special robot-helper who will help write code for you, named ***Crop-3PO***.
![](http://biowyse.eu/wp-content/uploads/2016/05/agrospace_01.jpg)_____no_output_____
**🤖 Crop-3PO:** Welcome! The people of Mars are counting on you! In order to use the special command, we need to import a special library by typing `import pandas`. We'll also need a few other libraries.

**🤖 Crop-3PO:** I wrote it down in the code cell below, however because I'm just a robot I can't execute the code! I'll need your help to do that. The easiest way to do this is by clicking `cell -> Run All`. If you don't see anything or think something's broken, you can click `Run`in the toolbar above while you have the code cell selected.
_____no_output_____

# ❗ Run this cell!
# import libraries
import pandas
import plotly.express as px
import plotly.graph_objects as go
from Bio import SeqIO_____no_output_____

### 📌 Data Sources

Data in this learning module has been obtained from the following sources:
1. Approval data [research article](https://doi.org/10.1080/21645698.2015.1056972).
2. Crop data [research article](https://doi.org/10.1038/s41598-019-51668-x).
3. Trait data from the International Service for the Acquisition of [Agri-Biotech Applications Genetically Modified Approval Database](https://www.isaaa.org/gmapprovaldatabase/geneslist/default.asp)._____no_output_____

# ❗ Run this cell!
# import data
approvaldata = pandas.read_csv("./data/foodapprovals.csv")
cropdata = pandas.read_csv("./data/cropdatafromearth.csv")
traitdata = pandas.read_csv("./data/isaaa_croptraits.csv")_____no_output_____

## 🌽 Before we get started, what even is a GMO?

*According to [gmotesting.com](http://www.gmotesting.com/GMOs/What-is-a-GMO):*

> **GMO** is an acronym for **g**enetically **m**odified **o**rganism and is commonly used to refer to genetically altered crops that are grown in many areas around the globe.  Genetic engineering provides the ability to confer desired traits on plants such as herbicide tolerance and/or virus or insect resistance.

> The process of genetic modification involves splicing, or cutting, genes from one organism, such as a bacterium, virus, or animal, and inserting them into a recipient organism, such as a plant, so that the recipient is now able to express new traits provided by the donor genes.  The genetic material (commonly called a transgene) is inserted into the nucleus of a plant cell where it integrates into the plant DNA.  If integration of the DNA is successful, the plant cell, now described as a transgenic cell, divides and grows into a genetically modified (or transgenic) plant.  The genetic modification is permanent and will be passed on to the seeds of the transgenic (GMO) plant._____no_output_____### **🤖 Crop-3PO:** GMOs have grown in popularity internationally over the last 30 years as our scientific methods become more reliable. Take a look at this chart that shows which countries have approved GMO crops for cultivation. _____no_output_____

# ❗ Run this cell!
# Display a histogram of the number of approved GMOs by country
fig = px.bar(approvaldata, x="Country", y = 'Total', title="Number of Approved GMO Crops for Cultivation between 1992-2014")
fig.update_yaxes(range=[0, 400])
fig.show()_____no_output_____

### 📚 Look at the trends of approved GMOs by countries over the last three decades. What differences do you see? Hover over the graph to see more details.

Fill in the boxes marked with ✏ by double clicking on it and filling in your answer. Remember to save your answers by clicking `File > Save and Checkpoint`!_____no_output_____✏

✏

✏_____no_output_____### 📚 List some reasons why there might be differences in approved GMOs by countries.

Fill in the boxes marked with ✏ by double clicking on it and filling in your answer. Remember to save your answers by clicking `File > Save and Checkpoint`!_____no_output_____✏

✏

✏_____no_output_____## 🌽 How are GMOs made?

![](https://i1.wp.com/sitn.hms.harvard.edu/wp-content/uploads/2015/08/Untitled2.png?w=900)
Image from: http://sitn.hms.harvard.edu/flash/2015/how-to-make-a-gmo/_____no_output_____## 🌽  What are plant traits? How are traits selected for?

Plant traits are desirable **phenotypes** (observable expression of a gene, it's functional use), and can be almost anything as long as they are useful. Century after century, farmers and botanists have selected plants that have desirable traits (sweet taste, size, etc) and cross bred them over time to produce more of that crop. With the use of modern technology however, we have successfully determined various genes that result in different plant traits. Some traits are not just the addition or enhancement of something, but can also be a suppression (for instead, delayed ripening). Instead of breeding over many generations, we can directly insert the gene to create a genetically engineered (GE) or modified plant, and have more direct and reliable results.

To learn more about the science of how this is actually done, read here: https://www.nationalgeographic.com/environment/future-of-food/food-technology-gene-editing/_____no_output_____

# ❗ Run this cell!
# Display a histogram breaking down the GMO varieties in most popular crops
fig = px.histogram(cropdata, x="Crop", color="Variety", title = "Popular Plant Traits in Common GMO Crops",
                   labels={"Variety":"Plant Traits","Crop":"GMO Crop Types"})
fig.show()_____no_output_____

### 📚 Look at the number (count) of popular plant traits in common GMO crops. What differences in plant traits between GMO crops do you see? Hover over the graph to see more details.

Note that GE is an acronym for genetically engineered.

Fill in the boxes marked with ✏ by double clicking on it and filling in your answer. Remember to save your answers by clicking `File > Save and Checkpoint`!_____no_output_____✏

✏

✏_____no_output_____### 📚 List some reasons why there might be differences in popular plant traits between GMO crop types.

Fill in the boxes marked with ✏ by double clicking on it and filling in your answer. Remember to save your answers by clicking `File > Save and Checkpoint`!_____no_output_____✏

✏

✏_____no_output_____## 🌽 How are plant genes and proteins related to traits?

Genes are actually only one part of the story. The way a gene becomes useful is by being translated into a protein in the cell. This protein then goes on to have different functions based on it's physical properties.

*From [The University of Kentucky's College of Agriculture:](https://entomology.ca.uky.edu/ef130)*
> To transform a plant into a GMO plant, the gene that produces a genetic trait of interest is identified and separated from the rest of the genetic material from a donor organism. Most organisms have thousands of genes, a single gene represents only a tiny fraction of the total genetic makeup of an organism.

>A donor organism may be a bacterium, fungus or even another plant. In the case of Bt corn, the donor organism is a naturally occurring soil bacterium, Bacillus thuringiensis, and the gene of interest produces a protein that kills Lepidoptera larvae, in particular, European corn borer. This protein is called the Bt delta endotoxin. Growers use Bt corn as an alternative to spraying insecticides for control of European and southwestern corn borer. _____no_output_____

# ❗ Run this cell!
# histogram of different genes associated with traits
fig = px.histogram(traitdata, x="Trait", title="Number of genes associated with traits",
                   labels={'Trait':'Phenotypic Trait'}).update_xaxes(categoryorder="total descending")
fig.show()_____no_output_____

### 📚 Look at the distribution of traits in the above graph. Which 3 traits are the most popular and why do you think that is?

Fill in the boxes marked with ✏ by double clicking on it and filling in your answer. Remember to save your answers by clicking `File > Save and Checkpoint`!_____no_output_____✏

✏

✏_____no_output_____### **🤖 Crop-3PO:** As you can see below, there are MANY genes associated with traits!

In this table, the columns are:
- "Trait" - refers to the phenotypic trait that we are interested in.
- "Gene" - refers to the name of the gene that codes for the phenotypic trait.
- "Source" - refers to original source organism where this "Gene" comes from.
- "Product" - refers to the protein that is translated from the "Gene".
- "Function" - refers to what the "Product" or protein of interest actually does when expressed. This ability is then conferred to the genetically modified or engineered plant that expresses it.

The table is sorted alphabetically (with numbers first) by the "Trait" column._____no_output_____

# ❗ Run this cell!
# View the ENTIRE data frame
#pandas.set_option('display.max_rows', None)
traitdata_____no_output_____

### 📚 Look at the "Trait" column in the table above. Look for the row(s) with the "Lepidopteran insect resistance" entry AND the "cry1Ab" gene. What is the "Function" (last column) associated with this trait?

Fill in the boxes marked with ✏ by double clicking on it and filling in your answer. Remember to save your answers by clicking `File > Save and Checkpoint`!_____no_output_____✏_____no_output_____### 📚 Locate another "Trait" that caught your attention. Write down what the "Trait" is and why you think it might be useful for plants to have on Mars.

Fill in the boxes marked with ✏ by double clicking on it and filling in your answer. Remember to save your answers by clicking `File > Save and Checkpoint`!_____no_output_____✏_____no_output_____# 🧬 💻  Bioinformatics Hands-on portion_____no_output_____Now that you understand a little more of what GMO's are, it's time to try and find out what our seeds are. Luckily, you have a sequencing machine that can tell you the DNA sequence of a seed. Crop-3PO also has a list of genes, but only knows a few of their DNA sequences. (Note that these sequences are made up, and that actual DNA sequences are much longer!)

You sort your seeds out into 10 different groups based on phenotypic traits (e.g. colour, size, and hardness). Since you only a few seeds in each group you don't want to waste them, so you tell Crop-3PO to sequence only one from each group. You call these your samples._____no_output_____### **🤖 Crop-3PO:** I know lots of genes and sequences! Here is an example of what the `cry1Ab` gene DNA sequence looks like:_____no_output_____

# ❗ Run this cell!
# Here, we can see a gene sequence associated with cry1AB_1
genes_dict = SeqIO.to_dict(SeqIO.parse("./data/genes.fa", "fasta"))
print(genes_dict["cry1Ab_1"].format("fasta"))_____no_output_____

### **🤖 Crop-3PO:** Here are all the genes I know, some I only know part of the sequence for. If it looks similar, you can act like they are the same as the trait. For instance `tNOS` can be considered `nos` for our purposes._____no_output_____

# ❗ Run this cell!
#Here is a list of all the genes we are working with
print("Here is a list of genes Crop-3PO knows:")
for key in genes_dict.keys():
    print(key) _____no_output_____

### **🤖 Crop-3PO:** Here is a list of all the samples I successfully sequenced!_____no_output_____

# ❗ Run this cell!
# list of our total samples
samples_dict = SeqIO.to_dict(SeqIO.parse("./data/samples.fa", "fasta"))
print("Here is a list of samples Crop-3PO sequenced:")
for key in samples_dict.keys():
    print(key)_____no_output_____

### **🤖 Crop-3PO:** Here's an example of what the sequence for `sample1` looks like._____no_output_____

# ❗ Run this cell!
# let's look at some of the information in our first sample
print(samples_dict["sample1"].format("fasta"))_____no_output_____

### **🤖 Crop-3PO:** That's probably pretty difficult for a human to read. Luckily, as a machine I can very quickly match the sequences of our genes and tell you whether a sample is likely to have the gene or not. Let me try doing this, and you can see my output down below._____no_output_____

# ❗ Run this cell!
# now we're going to match the genes to each sample
x=[]
y=[]
for sample in samples_dict:
    x.append(samples_dict[sample].name)
    y.append(samples_dict[sample].seq)
    temp2 = []
    for gene in genes_dict:
        temp1 = []
        if genes_dict[gene].seq in samples_dict[sample].seq:
            temp1.append(genes_dict[gene].name)
            temp2.append(temp1)
    print("🔬\033[1;36m", samples_dict[sample].name, "\033[1;0m contains matches for the genes \033[1;32m", temp2, "\033[1;0m")_____no_output_____

### **🤖 Crop-3PO:** Pretty cool trick huh? Now all that's left to do is try to figure out which ones are useful for us. You'll probably want to look at the genes in each sample and think about which traits they're associated with, and whether or not that is useful for us to have here on Mars. I've listed them down below._____no_output_____

# ❗ Run this cell!
# trait data narrowed down to our genes
traitdata.loc[traitdata['Gene'].isin(['cry1Ab','nos','cry1Ac','cry2Ab2','epsps (Ag)','prsv_cp','cel1','bar','pat'])]_____no_output_____

### 📚 Based on the information available to us, what traits does each sample have?

Fill in the boxes marked with ✏ by double clicking on it and filling in your answer. Remember to save your answers by clicking `File > Save and Checkpoint`!_____no_output_____✏ sample1:

✏ sample2:

✏ sample3:

✏ sample4:

✏ sample5:

✏ sample6:

✏ sample7:

✏ sample8:

✏ sample9:

✏ sample10:_____no_output_____### 📚 Out of all of our `samples`, which ones do you think would be most suitable to grow on Mars? Explain your reasoning. Keep in mind, what might be beneficial on Earth might not make much of a difference on Mars!

Fill in the boxes marked with ✏ by double clicking on it and filling in your answer. Remember to save your answers by clicking `File > Save and Checkpoint`!_____no_output_____✏

✏

✏_____no_output_____### 📚 Assuming the `samples` are of the same plant species (e.g. Corn), explain which two `sample` plants would you try to cross first to achieve a "stacked" trait plant that can thrive on Mars?

### [Gene stacking](https://www.isaaa.org/resources/publications/pocketk/42/#:~:text=Gene%20stacking%20refers%20to%20the,process%20are%20called%20stacked%20traits.) refers to combining multiple genes in a single GMO plant to confer multiple traits.

Fill in the boxes marked with ✏ by double clicking on it and filling in your answer. Remember to save your answers by clicking `File > Save and Checkpoint`!_____no_output_____✏
_____no_output_____[![Callysto.ca License](https://github.com/callysto/curriculum-notebooks/blob/master/callysto-notebook-banner-bottom.jpg?raw=true)](https://github.com/callysto/curriculum-notebooks/blob/master/LICENSE.md)_____no_output_____