# Notebook from kryvokhyzha/examples-and-courses
Path: some-ml-examples/pattern_classification-master/dimensionality_reduction/projection/principal_component_analysis.ipynb

%load_ext watermark_____no_output_____%watermark -v -d -a 'Sebastian Raschka' -p scikit-learn,matplotlib,numpy,pandas/Users/sebastian/miniconda3/lib/python3.6/site-packages/watermark/watermark.py:155: DeprecationWarning: Importing scikit-learn as `scikit-learn` has been depracated and will not be supported anymore in v1.7.0. Please use the package name `sklearn` instead.
  DeprecationWarning)
%matplotlib inline_____no_output_____

_____no_output_____# Principal Component Analysis in 3 Simple Steps_____no_output_____Principal Component Analysis (PCA) is a simple yet popular and useful linear transformation technique that is used in numerous applications, such as stock market predictions, the  analysis of gene expression data, and many more. In this tutorial, we will see that PCA is not just a "black box", and we are going to unravel its internals in 3 basic steps._____no_output_____This article just got a complete overhaul, the original version is still available at [principal_component_analysis_old.ipynb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/dimensionality_reduction/projection/principal_component_analysis.ipynb)._____no_output_____

_____no_output_____
_____no_output_____## Sections_____no_output_____- [Introduction](#Introduction)
    - [PCA Vs. LDA](#PCA-Vs.-LDA)
    - [PCA and Dimensionality Reduction](#PCA-and-Dimensionality-Reduction)
    - [A Summary of the PCA Approach](#A-Summary-of-the-PCA-Approach)
- [Preparing the Iris Dataset](#Preparing-the-Iris-Dataset)
    - [About Iris](#About-Iris)
    - [Loading the Dataset](#Loading-the-Dataset)
    - [Exploratory Visualization](#Exploratory-Visualization)
    - [Standardizing](#Standardizing)
- [1 - Eigendecomposition - Computing Eigenvectors and Eigenvalues](#1---Eigendecomposition---Computing-Eigenvectors-and-Eigenvalues)
    - [Covariance Matrix](#Covariance-Matrix)
    - [Correlation Matrix](#Correlation-Matrix)
    - [Singular Vector Decomposition](#Singular-Vector-Decomposition)
- [2 - Selecting Principal Components](#2---Selecting-Principal-Components)
    - [Sorting Eigenpairs](#Sorting-Eigenpairs)
    - [Explained Variance](#Explained-Variance)
    - [Projection Matrix](#Projection-Matrix)
- [3 - Projection Onto the New Feature Space](#3---Selecting-Principal-Components)
- [Shortcut - PCA in scikit-learn](#Shortcut---PCA-in-scikit-learn)_____no_output_____

_____no_output_____
_____no_output_____## Introduction_____no_output_____[[back to top](#Sections)]_____no_output_____The sheer size of data in the modern age is not only a challenge for computer hardware but also a main bottleneck for the performance of many machine learning algorithms. The main goal of a PCA analysis is to identify patterns in data; PCA aims to detect the correlation between variables. If a strong correlation between variables exists, the attempt to reduce the dimensionality only makes sense. In a nutshell, this is what PCA is all about: Finding the directions of maximum variance in high-dimensional data and project it onto a smaller dimensional subspace while retaining most of the information._____no_output_____

_____no_output_____### PCA Vs. LDA_____no_output_____[[back to top](#Sections)]_____no_output_____Both Linear Discriminant Analysis (LDA) and PCA are linear transformation methods. PCA yields the directions (principal components) that maximize the variance of the data, whereas LDA also aims to find the directions that maximize the separation (or discrimination) between different classes, which can be useful in pattern classification problem (PCA "ignores" class labels).
***In other words, PCA projects the entire dataset onto a different feature (sub)space, and LDA tries to determine a suitable feature (sub)space in order to distinguish between patterns that belong to different classes.***  _____no_output_____

_____no_output_____### PCA and Dimensionality Reduction_____no_output_____[[back to top](#Sections)]_____no_output_____Often, the desired goal is to reduce the dimensions of a $d$-dimensional dataset by projecting it onto a $(k)$-dimensional subspace (where $k\;<\;d$) in order to increase the computational efficiency while retaining most of the information. An important question is "what is the size of $k$ that represents the data 'well'?"

Later, we will compute eigenvectors (the principal components) of a dataset and collect them in a projection matrix. Each of those eigenvectors is associated with an eigenvalue which can be interpreted as the "length" or "magnitude" of the corresponding eigenvector. If some eigenvalues have a significantly larger magnitude than others that the reduction of the dataset via PCA onto a smaller dimensional subspace by dropping the "less informative" eigenpairs is reasonable.
_____no_output_____

_____no_output_____### A Summary of the PCA Approach_____no_output_____[[back to top](#Sections)]_____no_output_____-  Standardize the data.
-  Obtain the Eigenvectors and Eigenvalues from the covariance matrix or correlation matrix, or perform Singular Vector Decomposition.
-  Sort eigenvalues in descending order and choose the $k$ eigenvectors that correspond to the $k$ largest eigenvalues where $k$ is the number of dimensions of the new feature subspace ($k \le d$)/.
-  Construct the projection matrix $\mathbf{W}$ from the selected $k$ eigenvectors.
-  Transform the original dataset $\mathbf{X}$ via $\mathbf{W}$ to obtain a $k$-dimensional feature subspace $\mathbf{Y}$._____no_output_____

_____no_output_____## Preparing the Iris Dataset_____no_output_____[[back to top](#Sections)]_____no_output_____

_____no_output_____### About Iris_____no_output_____[[back to top](#Sections)]_____no_output_____For the following tutorial, we will be working with the famous "Iris" dataset that has been deposited on the UCI machine learning repository
([https://archive.ics.uci.edu/ml/datasets/Iris](https://archive.ics.uci.edu/ml/datasets/Iris)).

The iris dataset contains measurements for 150 iris flowers from three different species.

The three classes in the Iris dataset are:

1. Iris-setosa (n=50)
2. Iris-versicolor (n=50)
3. Iris-virginica (n=50)

And the four features of in Iris dataset are:

1. sepal length in cm
2. sepal width in cm
3. petal length in cm
4. petal width in cm

_____no_output_____

_____no_output_____### Loading the Dataset_____no_output_____[[back to top](#Sections)]_____no_output_____In order to load the Iris data directly from the UCI repository, we are going to use the superb [pandas](http://pandas.pydata.org) library. If you haven't used pandas yet, I want encourage you to check out the [pandas tutorials](http://pandas.pydata.org/pandas-docs/stable/tutorials.html). If I had to name one Python library that makes working with data a wonderfully simple task, this would definitely be pandas!_____no_output_____

import pandas as pd

df = pd.read_csv(
    filepath_or_buffer='https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data',
    header=None,
    sep=',')

df.columns=['sepal_len', 'sepal_wid', 'petal_len', 'petal_wid', 'class']
df.dropna(how="all", inplace=True) # drops the empty line at file-end

df.tail()_____no_output_____# split data table into data X and class labels y

X = df.ix[:,0:4].values
y = df.ix[:,4].values/Users/sebastian/miniconda3/lib/python3.6/site-packages/ipykernel_launcher.py:3: DeprecationWarning:
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing

See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#ix-indexer-is-deprecated
  This is separate from the ipykernel package so we can avoid doing imports until
/Users/sebastian/miniconda3/lib/python3.6/site-packages/pandas/core/indexing.py:890: DeprecationWarning:
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing

See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#ix-indexer-is-deprecated
  retval = getattr(retval, self.name)._getitem_axis(key, axis=i)
/Users/sebastian/miniconda3/lib/python3.6/site-packages/ipykernel_launcher.py:4: DeprecationWarning:
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing

See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#ix-indexer-is-deprecated
  after removing the cwd from sys.path.

Our iris dataset is now stored in form of a  $150 \times 4$ matrix where the columns are the different features, and every row represents a separate flower sample.
Each sample row $\mathbf{x}$ can be pictured as a 4-dimensional vector

$\mathbf{x^T} = \begin{pmatrix} x_1 \\ x_2 \\ x_3 \\ x_4 \end{pmatrix}
= \begin{pmatrix} \text{sepal length} \\ \text{sepal width} \\\text{petal length} \\ \text{petal width} \end{pmatrix}$_____no_output_____

_____no_output_____### Exploratory Visualization_____no_output_____[[back to top](#Sections)]_____no_output_____To get a feeling for how the 3 different flower classes are distributes along the 4 different features, let us visualize them via histograms._____no_output_____

from matplotlib import pyplot as plt
import numpy as np
import math

label_dict = {1: 'Iris-Setosa',
              2: 'Iris-Versicolor',
              3: 'Iris-Virgnica'}

feature_dict = {0: 'sepal length [cm]',
                1: 'sepal width [cm]',
                2: 'petal length [cm]',
                3: 'petal width [cm]'}

with plt.style.context('seaborn-whitegrid'):
    plt.figure(figsize=(8, 6))
    for cnt in range(4):
        plt.subplot(2, 2, cnt+1)
        for lab in ('Iris-setosa', 'Iris-versicolor', 'Iris-virginica'):
            plt.hist(X[y==lab, cnt],
                     label=lab,
                     bins=10,
                     alpha=0.3,)
        plt.xlabel(feature_dict[cnt])
    plt.legend(loc='upper right', fancybox=True, fontsize=8)

    plt.tight_layout()
    plt.show()_____no_output_____

_____no_output_____### Standardizing_____no_output_____[[back to top](#Sections)]_____no_output_____Whether to standardize the data prior to a PCA on the covariance matrix depends on the measurement scales of the original features. Since PCA yields a feature subspace that maximizes the variance along the axes, it makes sense to standardize the data, especially, if it was measured on different scales. Although, all features in the Iris dataset were measured in centimeters, let us continue with the transformation of the data onto unit scale (mean=0 and variance=1), which is a requirement for the optimal performance of many machine learning algorithms._____no_output_____

from sklearn.preprocessing import StandardScaler
X_std = StandardScaler().fit_transform(X)_____no_output_____

_____no_output_____## 1 - Eigendecomposition - Computing Eigenvectors and Eigenvalues_____no_output_____[[back to top](#Sections)]_____no_output_____The eigenvectors and eigenvalues of a covariance (or correlation) matrix represent the "core" of a PCA: The eigenvectors (principal components) determine the directions of the new feature space, and the eigenvalues determine their magnitude. In other words, the eigenvalues explain the variance of the data along the new feature axes._____no_output_____

_____no_output_____### Covariance Matrix_____no_output_____[[back to top](#Sections)]_____no_output_____The classic approach to PCA is to perform the eigendecomposition on the covariance matrix $\Sigma$, which is a $d \times d$ matrix where each element represents the covariance between two features. The covariance between two features is calculated as follows:

$\sigma_{jk} = \frac{1}{n-1}\sum_{i=1}^{N}\left(  x_{ij}-\bar{x}_j \right)  \left( x_{ik}-\bar{x}_k \right).$

We can summarize the calculation of the covariance matrix via the following matrix equation:
$\Sigma = \frac{1}{n-1} \left( (\mathbf{X} - \mathbf{\bar{x}})^T\;(\mathbf{X} - \mathbf{\bar{x}}) \right)$
where $\mathbf{\bar{x}}$ is the mean vector
$\mathbf{\bar{x}} = \sum\limits_{i=1}^n x_{i}.$
The mean vector is a $d$-dimensional vector where each value in this vector represents the sample mean of a feature column in the dataset._____no_output_____

import numpy as np
mean_vec = np.mean(X_std, axis=0)
cov_mat = (X_std - mean_vec).T.dot((X_std - mean_vec)) / (X_std.shape[0]-1)
print('Covariance matrix \n%s' %cov_mat)Covariance matrix
[[ 1.00671141 -0.11010327  0.87760486  0.82344326]
 [-0.11010327  1.00671141 -0.42333835 -0.358937  ]
 [ 0.87760486 -0.42333835  1.00671141  0.96921855]
 [ 0.82344326 -0.358937    0.96921855  1.00671141]]
import numpy as np
mean_vec = np.mean(X_std, axis=0)
cov_mat = (X_std - mean_vec).T.dot((X_std - mean_vec)) / (X_std.shape[0]-1)
print('Covariance matrix \n%s' %cov_mat)Covariance matrix
[[ 1.00671141 -0.11010327  0.87760486  0.82344326]
 [-0.11010327  1.00671141 -0.42333835 -0.358937  ]
 [ 0.87760486 -0.42333835  1.00671141  0.96921855]
 [ 0.82344326 -0.358937    0.96921855  1.00671141]]

The more verbose way above was simply used for demonstration purposes, equivalently, we could have used the numpy `cov` function:_____no_output_____

print('NumPy covariance matrix: \n%s' %np.cov(X_std.T))NumPy covariance matrix:
[[ 1.00671141 -0.11010327  0.87760486  0.82344326]
 [-0.11010327  1.00671141 -0.42333835 -0.358937  ]
 [ 0.87760486 -0.42333835  1.00671141  0.96921855]
 [ 0.82344326 -0.358937    0.96921855  1.00671141]]

_____no_output_____Next, we perform an eigendecomposition on the covariance matrix:_____no_output_____

cov_mat = np.cov(X_std.T)

eig_vals, eig_vecs = np.linalg.eig(cov_mat)

print('Eigenvectors \n%s' %eig_vecs)
print('\nEigenvalues \n%s' %eig_vals)Eigenvectors
[[ 0.52237162 -0.37231836 -0.72101681  0.26199559]
 [-0.26335492 -0.92555649  0.24203288 -0.12413481]
 [ 0.58125401 -0.02109478  0.14089226 -0.80115427]
 [ 0.56561105 -0.06541577  0.6338014   0.52354627]]

Eigenvalues
[2.93035378 0.92740362 0.14834223 0.02074601]

_____no_output_____### Correlation Matrix_____no_output_____[[back to top](#Sections)]_____no_output_____Especially, in the field of "Finance," the correlation matrix typically used instead of the covariance matrix. However, the eigendecomposition of the covariance matrix (if the input data was standardized) yields the same results as a eigendecomposition on the correlation matrix, since the correlation matrix can be understood as the normalized covariance matrix._____no_output_____

_____no_output_____Eigendecomposition of the standardized data based on the correlation matrix:_____no_output_____

cor_mat1 = np.corrcoef(X_std.T)

eig_vals, eig_vecs = np.linalg.eig(cor_mat1)

print('Eigenvectors \n%s' %eig_vecs)
print('\nEigenvalues \n%s' %eig_vals)Eigenvectors
[[ 0.52237162 -0.37231836 -0.72101681  0.26199559]
 [-0.26335492 -0.92555649  0.24203288 -0.12413481]
 [ 0.58125401 -0.02109478  0.14089226 -0.80115427]
 [ 0.56561105 -0.06541577  0.6338014   0.52354627]]

Eigenvalues
[2.91081808 0.92122093 0.14735328 0.02060771]

_____no_output_____Eigendecomposition of the raw data based on the correlation matrix:_____no_output_____

cor_mat2 = np.corrcoef(X.T)

eig_vals, eig_vecs = np.linalg.eig(cor_mat2)

print('Eigenvectors \n%s' %eig_vecs)
print('\nEigenvalues \n%s' %eig_vals)Eigenvectors
[[ 0.52237162 -0.37231836 -0.72101681  0.26199559]
 [-0.26335492 -0.92555649  0.24203288 -0.12413481]
 [ 0.58125401 -0.02109478  0.14089226 -0.80115427]
 [ 0.56561105 -0.06541577  0.6338014   0.52354627]]

Eigenvalues
[2.91081808 0.92122093 0.14735328 0.02060771]

_____no_output_____We can clearly see that all three approaches yield the same eigenvectors and eigenvalue pairs:

- Eigendecomposition of the covariance matrix after standardizing the data.
- Eigendecomposition of the correlation matrix.
- Eigendecomposition of the correlation matrix after standardizing the data._____no_output_____

_____no_output_____### Singular Vector Decomposition_____no_output_____[[back to top](#Sections)]_____no_output_____While the eigendecomposition of the covariance or correlation matrix may be more intuitiuve, most PCA implementations perform a Singular Vector Decomposition (SVD) to improve the computational efficiency. So, let us perform an SVD to  confirm that the result are indeed the same:_____no_output_____

u,s,v = np.linalg.svd(X_std.T)
print('Vectors U:\n', u)Vectors U:
 [[-0.52237162 -0.37231836  0.72101681  0.26199559]
 [ 0.26335492 -0.92555649 -0.24203288 -0.12413481]
 [-0.58125401 -0.02109478 -0.14089226 -0.80115427]
 [-0.56561105 -0.06541577 -0.6338014   0.52354627]]

_____no_output_____## 2 - Selecting Principal Components_____no_output_____[[back to top](#Sections)]_____no_output_____

_____no_output_____### Sorting Eigenpairs_____no_output_____[[back to top](#Sections)]_____no_output_____The typical goal of a PCA is to reduce the dimensionality of the original feature space by projecting it onto a smaller subspace, where the eigenvectors will form the axes. However, the eigenvectors only define the directions of the new axis, since they have all the same unit length 1, which can confirmed by the following two lines of code:_____no_output_____

for ev in eig_vecs.T:
    np.testing.assert_array_almost_equal(1.0, np.linalg.norm(ev))
print('Everything ok!')Everything ok!

_____no_output_____In order to decide which eigenvector(s) can dropped without losing too much information
for the construction of lower-dimensional subspace, we need to inspect the corresponding eigenvalues: The eigenvectors with the lowest eigenvalues bear the least information about the distribution of the data; those are the ones can be dropped.
In order to do so, the common approach is to rank the eigenvalues from highest to lowest in order choose the top $k$ eigenvectors._____no_output_____

# Make a list of (eigenvalue, eigenvector) tuples
eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:,i]) for i in range(len(eig_vals))]

# Sort the (eigenvalue, eigenvector) tuples from high to low
eig_pairs.sort(key=lambda x: x[0], reverse=True)

# Visually confirm that the list is correctly sorted by decreasing eigenvalues
print('Eigenvalues in descending order:')
for i in eig_pairs:
    print(i[0])Eigenvalues in descending order:
2.91081808375
0.921220930707
0.147353278305
0.0206077072356

_____no_output_____### Explained Variance_____no_output_____[[back to top](#Sections)]_____no_output_____After sorting the eigenpairs, the next question is "how many principal components are we going to choose for our new feature subspace?" A useful measure is the so-called "explained variance," which can be calculated from the eigenvalues. The explained variance tells us how much information (variance) can be attributed to each of the principal components._____no_output_____

tot = sum(eig_vals)
var_exp = [(i / tot)*100 for i in sorted(eig_vals, reverse=True)]
cum_var_exp = np.cumsum(var_exp)_____no_output_____with plt.style.context('seaborn-whitegrid'):
    plt.figure(figsize=(6, 4))

    plt.bar(range(4), var_exp, alpha=0.5, align='center',
            label='individual explained variance')
    plt.step(range(4), cum_var_exp, where='mid',
             label='cumulative explained variance')
    plt.ylabel('Explained variance ratio')
    plt.xlabel('Principal components')
    plt.legend(loc='best')
    plt.tight_layout()
    plt.savefig('/Users/Sebastian/Desktop/pca2.pdf')_____no_output_____

The plot above clearly shows that most of the variance (72.77% of the variance to be precise) can be explained by the first principal component alone. The second principal component still bears some information (23.03%) while the third and fourth principal components can safely be dropped without losing to much information. Together, the first two principal components contain 95.8% of the information._____no_output_____

_____no_output_____### Projection Matrix_____no_output_____[[back to top](#Sections)]_____no_output_____It's about time to get to the really interesting part: The construction of the projection matrix that will be used to transform the Iris data onto the new feature subspace. Although, the name "projection matrix" has a nice ring to it, it is basically just a matrix of our concatenated top *k* eigenvectors.

Here, we are reducing the 4-dimensional feature space to a 2-dimensional feature subspace, by choosing the "top 2" eigenvectors with the highest eigenvalues to construct our $d \times k$-dimensional eigenvector matrix $\mathbf{W}$._____no_output_____

matrix_w = np.hstack((eig_pairs[0][1].reshape(4,1),
                      eig_pairs[1][1].reshape(4,1)))

print('Matrix W:\n', matrix_w)Matrix W:
 [[ 0.52237162 -0.37231836]
 [-0.26335492 -0.92555649]
 [ 0.58125401 -0.02109478]
 [ 0.56561105 -0.06541577]]

_____no_output_____## 3 - Projection Onto the New Feature Space_____no_output_____[[back to top](#Sections)]_____no_output_____In this last step we will use the $4 \times 2$-dimensional projection matrix $\mathbf{W}$ to transform our samples onto the new subspace via the equation
$\mathbf{Y} = \mathbf{X} \times  \mathbf{W}$, where $\mathbf{Y}$ is a $150\times 2$ matrix of our transformed samples._____no_output_____

Y = X_std.dot(matrix_w)_____no_output_____with plt.style.context('seaborn-whitegrid'):
    plt.figure(figsize=(6, 4))
    for lab, col in zip(('Iris-setosa', 'Iris-versicolor', 'Iris-virginica'),
                        ('blue', 'red', 'green')):
        plt.scatter(Y[y==lab, 0],
                    Y[y==lab, 1],
                    label=lab,
                    c=col)
    plt.xlabel('Principal Component 1')
    plt.ylabel('Principal Component 2')
    plt.legend(loc='lower center')
    plt.tight_layout()

    plt.savefig('/Users/Sebastian/Desktop/pca1.pdf')_____no_output_____

_____no_output_____

 _____no_output_____## Shortcut - PCA in scikit-learn_____no_output_____[[back to top](#Sections)]_____no_output_____For educational purposes, we went a long way to apply the PCA to the Iris dataset. But luckily, there is already implementation in scikit-learn. _____no_output_____

from sklearn.decomposition import PCA as sklearnPCA
sklearn_pca = sklearnPCA(n_components=2)
Y_sklearn = sklearn_pca.fit_transform(X_std)_____no_output_____with plt.style.context('seaborn-whitegrid'):
    plt.figure(figsize=(6, 4))
    for lab, col in zip(('Iris-setosa', 'Iris-versicolor', 'Iris-virginica'),
                        ('blue', 'red', 'green')):
        plt.scatter(Y_sklearn[y==lab, 0],
                    Y_sklearn[y==lab, 1],
                    label=lab,
                    c=col)
    plt.xlabel('Principal Component 1')
    plt.ylabel('Principal Component 2')
    plt.legend(loc='lower center')
    plt.tight_layout()
    plt.show()_____no_output_____


# Notebook from Bluefog-Lib/EastCoastTutorial2021
Path: tutorial/Parallel-and-Distributed-Optimization.ipynb

## 1. Parallel and Distributed Optimization

This notebook provides concise BlueFog demos to concepts or algorithms introduced in the tutorial.

### 1.1 Finite-sum optimizaiton example: distributed least square

Suppose $m$ computing nodes collaborate to solve the following problem:

\begin{align*}
\min_{x\in \mathbb{R}^d}\ \sum_{i=1}^n h_i(x) \quad \mbox{where} \quad h_i(x) = \frac{1}{2}\|A_i x - b_i\|^2 \hspace{1cm} \mbox{(Opt-Problem)}
\end{align*}

where $h_i(x): \mathbb{R}^d \to \mathbb{R}$ is a local cost function held by node $i$ and $\{A_i, b_i\}$ are local data. Each node $i$ can evaluate its own data and gradient, but it has to communicate to achieve information from the other node. We let $x^\star$ denote the global solution to the above problem

### 1.2 All-reduce

BlueFog supports the Ring-Allreduce operation as follows:
```
x_bar = bf.allreduce(x_local)
```

In the following code, we activate 8 computing nodes (CPUs) and label them as $0,1,\cdots,7$. We target to let all nodes collaborate to compute the global average of their labels $(\sum_{i=0}^7 i)/8 = 3.5$._____no_output_____

%%writefile Allreduce.py

import bluefog.torch as bf
import torch

bf.init()
x_local = torch.ones(1) * bf.rank()
x_bar = bf.allreduce(x_local)
print('Node {} achieved the global average {}'.format(bf.rank(), x_bar[0]))Overwriting Allreduce.py
# You can change NUM_PROC into any value smaller than the number of CPUs you have.
import os
NUM_PROC = 2 if os.getenv("TEST_ENV") else 8
NUM_PROC_____no_output_____! bfrun -np $NUM_PROC python Allreduce.pyNode 0 achieved the global average 3.5
Node 1 achieved the global average 3.5
Node 2 achieved the global average 3.5
Node 4 achieved the global average 3.5
Node 5 achieved the global average 3.5
Node 6 achieved the global average 3.5
Node 7 achieved the global average 3.5
Node 3 achieved the global average 3.5

### 1.3 Distributed Gradient Descent_____no_output_____The distributed gradient descent is $x^{k+1} = x^k - \frac{\alpha}{n}\sum_{i=1}^n \nabla h_i(x^k)$. This process can be implemented as

```python
# core distributed gradient descent snippet
grad = A.T.mm(A.mm(x) - b)
next_x = x - alpha * bf.allreduce(grad)
```_____no_output_____

%%writefile DistributedGD.py

import bluefog.torch as bf
import torch

bf.init()
# Make sure different agent has different random seed.
torch.manual_seed(12345 * bf.rank())

def generate_data(m, d):
    A = torch.randn(m, d).to(torch.double)
    ns = 0.1*torch.randn(m, 1).to(torch.double)
    x_o = torch.rand(d,1).to(torch.double)
    b = A.mm(x_o) + ns

    return A, b

def check_opt_cond(x, A, b):

    grad_local = A.t().mm(A.mm(x) - b)
    grad = bf.allreduce(grad_local, name='gradient')  # global gradient

    # the norm of global gradient is expected to be 0 (optimality condition)
    global_grad_norm = torch.norm(grad, p=2)
    print("[Distributed Grad Descent] Rank {}: global gradient norm: {}".format(bf.rank(), global_grad_norm))

    return

def distributed_grad_descent(A, b, maxite=5000, alpha=1e-1):

    m, d = A.shape

    x_opt = torch.zeros(d, 1, dtype=torch.double)

    for _ in range(maxite):
        # calculate local gradient
        grad_local = A.t().mm(A.mm(x_opt) - b)

        # global gradient
        grad = bf.allreduce(grad_local, name='gradient')

        # distributed gradient descent
        x_opt = x_opt - alpha*grad

    check_opt_cond(x_opt, A, b)

    return x_opt

if __name__ == "__main__":
    m, d = 20, 5 # dimension of A
    A, b = generate_data(m, d)
    x_opt = distributed_grad_descent(A, b, maxite=200, alpha=1e-2)Overwriting DistributedGD.py

If $x^k\to x^\star$, it holds that $\sum_{i=1}^m \nabla h_i(x^k) \to 0$. We can use $\|\sum_{i=1}^m \nabla h_i(x)\|$ as a metric to gauge whether distributed gradient descent converge or not._____no_output_____

! bfrun -np $NUM_PROC python DistributedGD.py[Distributed Grad Descent] Rank 2: global gradient norm: 7.786811775375865e-15
[Distributed Grad Descent] Rank 3: global gradient norm: 7.786811775375865e-15
[Distributed Grad Descent] Rank 4: global gradient norm: 7.786811775375865e-15
[Distributed Grad Descent] Rank 5: global gradient norm: 7.786811775375865e-15
[Distributed Grad Descent] Rank 6: global gradient norm: 7.786811775375865e-15
[Distributed Grad Descent] Rank 7: global gradient norm: 7.786811775375865e-15
[Distributed Grad Descent] Rank 0: global gradient norm: 7.786811775375865e-15
[Distributed Grad Descent] Rank 1: global gradient norm: 7.786811775375865e-15

### 1.4 Primal Decomposition

We consider the following linearly-constrained resource sharing problem:

\begin{align*}
\min_{\{x_i\}, y}&\quad \|y\|_1 + \frac{1}{2}\sum_{i=1}^n \|A_i x_i - b_i\|^2 \\
\mathrm{s.t.} &\quad\sum_{i=1}^n B_i x_i = y
\end{align*}

where $x_i \in \mathbb{R}^d$, $A_i \in \mathbb{R}^{m\times d}$, and $B_i \in \mathbb{R}^{d\times d}$. For simplicity, we assume each $B_i$ is a invertible matrix. Each node $i$ has local data $\{A_i, B_i, b_i\}$. To solve the above problem in a distributed manner, we introduce $y_i = B_i x_i$ so that $y=\sum_{i=1}^n y_i$. Since $B_i$ is invertible, we have $x_i = B_i^{-1} y_i$. Substituting these facts into the above problem, we achieve

\begin{align*}
\min_{\{y_i\}}&\quad \|y_1 + \cdots + y_n\|_1 + \frac{1}{2}\sum_{i=1}^n \|A_i B_i^{-1} y_i - b_i\|^2
\end{align*}

which can be solved in the following distributed manner. For notation simplicity, we let $C_i = A_i B_i^{-1}$ and $g(y_1,\cdots, y_n) = \|y_1 + \cdots + y_n\|_1$. Following proximal gradient descent, we have

\begin{align*}
z_i^{k+1} &= y_i^k - \alpha C_i^T(C_i y_i^k - b_i), \quad \forall\ i=1,\cdots, n\\
y_i^{k+1} &= [\mathrm{Prox}_{\alpha g}(z_1^{k+1},\cdots, z_n^{k+1})]_{i} =z_i^{k+1} - \alpha v^{k+1}, \quad \forall\ i=1,\cdots, n \\
x_i^{k+1} &= B_i^{-1} y_i^{k+1}, \quad \forall\ i=1,\cdots, n
\end{align*}

where
$$v^{k+1} = \frac{1}{n\alpha^2}\big(\alpha(z^{k+1}_1 + \cdots + z^{k+1}_n) - \mathrm{Prox}_{n \alpha^2 \|\cdot\|_1}(\alpha z^{k+1}_1 + \cdots + \alpha z^{k+1}_n)\big) \quad \mbox{(Need Allreduce Communication)}$$_____no_output_____

%%writefile PrimalDecompose.py

import bluefog.torch as bf
import torch

bf.init()
# Make sure different agent has different random seed.
torch.manual_seed(12345 * bf.rank())

def generate_data(m, d):
    A = torch.randn(m, d).to(torch.double)
    B_inv = torch.randn(d, d).to(torch.double)
    ns = 0.1*torch.randn(m, 1).to(torch.double)
    x_o = torch.rand(d,1).to(torch.double)
    b = A.mm(x_o) + ns

    return A, B_inv, b

def soft_threshold(x, kappa):

    zeros = torch.zeros(d,1).to(torch.double)
    x = torch.max(x - kappa, zeros)
    x = torch.min(x + kappa, zeros)

    return x

def check_opt_cond(y, A, B_inv, b, alpha):

    m, d = A.shape
    n = bf.size()
    C = A.mm(B_inv)

    # update z
    grad_local = C.t().mm(C.mm(y) - b)
    z = y - alpha*grad_local
    z_bar = bf.allreduce(z)
    z_sum = z_bar * n

    # update v
    v = (1/(n*alpha*alpha)) * (alpha*z_sum - soft_threshold(alpha*z_sum, n*alpha*alpha))

    # update y
    y_next = z - alpha*v

    # the norm of global gradient is expected to be 0 (optimality condition)
    global_grad_norm = torch.norm((y - y_next), p=2)
    print("[Primal Decomposition] Rank {}: optimality metric norm: {}".format(bf.rank(), global_grad_norm))

    return

def primal_decomposition(A, B_inv, b, maxite=5000, alpha=1e-1):

    m, d = A.shape
    n = bf.size()
    C = A.mm(B_inv)

    y = torch.zeros(d, 1, dtype=torch.double)

    for _ in range(maxite):
        # update z
        grad_local = C.t().mm(C.mm(y) - b)
        z = y - alpha*grad_local
        z_bar = bf.allreduce(z)
        z_sum = z_bar * n

        # update v
        v = (1/(n*alpha*alpha)) * (alpha*z_sum - soft_threshold(alpha*z_sum, n*alpha*alpha))

        # update y
        y = z - alpha*v

        # update x
        x = B_inv.mm(y)

    check_opt_cond(y, A, B_inv, b, alpha)

    return y

if __name__ == "__main__":
    m, d = 20, 5 # dimension of A
    maxite = 5000 if os.getenv("TEST_ENV") else 100
    A, B_inv, b = generate_data(m, d)
    y = primal_decomposition(A, B_inv, b, maxite=maxite, alpha=3e-3)Overwriting PrimalDecompose.py

We define $G^k_i = \frac{1}{\alpha}\big(y_i^k - [\mathrm{Prox}_{\alpha g}(z_1^{k+1},\cdots, z_n^{k+1})]_i\big)$ and use $\|G^k_i\|$ as the metric to evaluate the optimality of $y_i^k$._____no_output_____

! bfrun -np $NUM_PROC python PrimalDecompose.py[Primal Decomposition] Rank 2: optimality metric norm: 1.526888906516414e-05
[Primal Decomposition] Rank 3: optimality metric norm: 0.00010111695583238024
[Primal Decomposition] Rank 5: optimality metric norm: 2.66226578246676e-06
[Primal Decomposition] Rank 6: optimality metric norm: 3.1143558672548934e-05
[Primal Decomposition] Rank 4: optimality metric norm: 1.2605012098981551e-05
[Primal Decomposition] Rank 1: optimality metric norm: 0.00013532193550425833
[Primal Decomposition] Rank 0: optimality metric norm: 6.13283876701975e-05
[Primal Decomposition] Rank 7: optimality metric norm: 4.335475665174773e-06

### 1.5 Distributed ADMM

Suppose we want to optimize the following consensus problem:
$$
    \min_{x} \sum_{i=1}^n h_i(x)
$$
Each machine $i$ can access the function $h_i$ only. Note the problem has a shared $x$ cross all machines. To relax this condition, we can reformat the problem into the following equivalent form:
\begin{align}
    \min_{\{x_i\}, y} & \;\;\; \sum_{i=1}^n h_i(x_i),\\
    {\rm subject\ to} & \;\;\;  x_i = y,\;\; \forall i.
\end{align}

Applying ADMM, we obtain:

\begin{align}
    x^{k+1}_i =&\;\; {\rm argmin}_{x_{i}} \left\{ h_i(x_i) + \langle u_i^k, x_i - y^k\rangle + \frac{\alpha}{2}  \left\|x_i - y^k \right\|^2\right\} \\
    y^{k+1} =&\;\; \frac{1}{n} \left(x_i^{k+1} + \frac{1}{\alpha} u^k_i\right)\\
    u^{k+1}_i = &\;\; u^k_i + \alpha\left(x_i^{k+1} - y^{k+1} \right)
\end{align}
_____no_output_____We can further simplify it by noticing that $u^k_1,\ldots,u^k_n$ has mean 0. Here we provide the code snippet of a simple quadratic cost function again.

``` python
# For each agent, it owns data A and b differently.
def CentralizedADMMStepL2(A, b, x, y, u, alpha):
    next_x = ProximalStepL2(A, b, x, y, u, alpha)
    # We use allreduce to mimic the centralized behavior
    # It should be based on PS architecture and using gather and broadcast.
    next_y = bf.allreduce(next_x)   # Without u is okay since allreudce(u) == 0
    next_u = u + alpha * (next_x - next_y)
    return next_x, next_y, next_u
```_____no_output_____

! bfrun -np $NUM_PROC python CentralizedADMM.py[Centralized ADMM] Rank 5: ADMM residue gradient norm: 5.6263947243994325e-15
[Centralized ADMM] Rank 0: ADMM residue gradient norm: 5.6263947243994325e-15
[Centralized ADMM] Rank 1: ADMM residue gradient norm: 5.6263947243994325e-15
[Centralized ADMM] Rank 2: ADMM residue gradient norm: 5.6263947243994325e-15
[Centralized ADMM] Rank 4: ADMM residue gradient norm: 5.6263947243994325e-15
[Centralized ADMM] Rank 6: ADMM residue gradient norm: 5.6263947243994325e-15
[Centralized ADMM] Rank 3: ADMM residue gradient norm: 5.6263947243994325e-15
[Centralized ADMM] Rank 7: ADMM residue gradient norm: 5.6263947243994325e-15
Last three entries of x_ar:
 tensor([[0.4883],
        [0.6776],
        [0.6447]], dtype=torch.float64)
Last three entries of x_admm:
 tensor([[0.4883],
        [0.6776],
        [0.6447]], dtype=torch.float64)
import matplotlib.pyplot as plt
import scipy.io as sio
import os
%matplotlib inline
mse_records_dict = sio.loadmat('results/CentralizedADMM.mat')

plt.semilogy(mse_records_dict["mse"][0], label="Centralized ADMM")
plt.title("Centralized ADMM")
plt.xlabel("iteration")
plt.ylabel("Mean-Square-Error")
plt.grid("on")
plt.legend()
dirname = 'images'
if not os.path.exists(dirname):
    os.makedirs(dirname)
plt.savefig(os.path.join(dirname, 'centralized_admm.png'))_____no_output_____

You can find the full code in the `CentralizedADMM.py`

**Exercise**: Modify the loss function, (sub-)gradient, and proximal step for $\ell_1 + \ell_2$ cases._____no_output_____


# Notebook from mganahl/Cirq
Path: docs/protocols.ipynb

##### Copyright 2020 The Cirq Developers_____no_output_____

#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License._____no_output_____

# Protocols_____no_output_____

    View on QuantumAI

    Run in Google Colab

    View source on GitHub

    Download notebook

_____no_output_____

try:
    import cirq
except ImportError:
    print("installing cirq...")
    !pip install --quiet cirq
    print("installed cirq.")_____no_output_____

# Introduction

Cirq's protocols are very similar concept to Python's built-in protocols that were introduced in [PEP 544](https://www.python.org/dev/peps/pep-0544/).
Python's built-in protocols are extremely convenient, for example behind all the for loops and list comprehensions you can find the Iterator protocol.
As long as an object has the `__iter__()` magic method that returns an iterator object, it has iterator support.
An iterator object has to define `__iter__()` and `__next__()` magic methods, that defines the iterator protocol.
The `iter(val)` builtin function returns an iterator for `val` if it defines the above methods, otherwise throws a `TypeError`. Cirq protocols work similarly.

A canonical Cirq protocol example is the `unitary` protocol that allows to check the unitary matrix of values that support the protocol by calling `cirq.unitary(val)`._____no_output_____

import cirq

print(cirq.X)
print("cirq.X unitary:\n", cirq.unitary(cirq.X))

a, b = cirq.LineQubit.range(2)
circuit = cirq.Circuit(cirq.X(a), cirq.Y(b))
print(circuit)
print("circuit unitary:\n", cirq.unitary(circuit))
_____no_output_____

When an object does not support a given protocol, an error is thrown._____no_output_____

try:
    print(cirq.unitary(a)) ## error!
except Exception as e:
    print("As expected, a qubit does not have a unitary. The error: ")
    print(e)
    _____no_output_____

## What is a protocol?

A protocol is a combination of the following two items:
- a `SupportsXYZ` class, which defines and documents all the magic functions that need to be implemented in order to support that given protocol
- the entrypoint function(s), which are exposed to the main cirq namespace as `cirq.xyz()`

Note: While the protocol is technically both of these things, we refer to the public utility functions interchangeably as protocols. See the list of them below.

## Cirq's protocols

For a complete list of Cirq protocols, refer to the `cirq.protocols` package.
Here we provide a list of frequently used protocols for debugging, simulation and testing.

| Protocol | Description |
|----------|-------|
|`cirq.apply_channel`| High performance evolution under a channel evolution. |
|`cirq.apply_mixture`| High performance evolution under a mixture of unitaries evolution. |
|`cirq.apply_unitaries`| Apply a series of unitaries onto a state tensor. |
|`cirq.apply_unitary`| High performance left-multiplication of a unitary effect onto a tensor. |
|`cirq.approx_eq`| Approximately compares two objects. |
|`cirq.channel`| Returns a list of matrices describing the channel for the given value. |
|`cirq.commutes`| Determines whether two values commute. |
|`cirq.definitely_commutes`| Determines whether two values definitely commute. |
|`cirq.decompose`| Recursively decomposes a value into `cirq.Operation`s meeting a criteria. |
|`cirq.decompose_once`| Decomposes a value into operations, if possible. |
|`cirq.decompose_once_with_qubits`| Decomposes a value into operations on the given qubits. |
|`cirq.equal_up_to_global_phase`| Determine whether two objects are equal up to global phase. |
|`cirq.has_channel`| Returns whether the value has a channel representation. |
|`cirq.has_mixture`| Returns whether the value has a mixture representation. |
|`cirq.has_unitary`| Determines whether the value has a unitary effect. |
|`cirq.inverse`| Returns the inverse `val**-1` of the given value, if defined. |
|`cirq.is_measurement`| Determines whether or not the given value is a measurement. |
|`cirq.is_parameterized`| Returns whether the object is parameterized with any Symbols. |
|`cirq.measurement_key`| Get the single measurement key for the given value. |
|`cirq.measurement_keys`| Gets the measurement keys of measurements within the given value. |
|`cirq.mixture`| Return a sequence of tuples representing a probabilistic unitary. |
|`cirq.num_qubits`| Returns the number of qubits, qudits, or qids `val` operates on. |
|`cirq.parameter_names`| Returns parameter names for this object. |
|`cirq.parameter_symbols`| Returns parameter symbols for this object. |
|`cirq.pauli_expansion`| Returns coefficients of the expansion of val in the Pauli basis. |
|`cirq.phase_by`| Returns a phased version of the effect. |
|`cirq.pow`| Returns `val**factor` of the given value, if defined. |
|`cirq.qasm`| Returns QASM code for the given value, if possible. |
|`cirq.qid_shape`| Returns a tuple describing the number of quantum levels of each |
|`cirq.quil`| Returns the QUIL code for the given value. |
|`cirq.read_json`| Read a JSON file that optionally contains cirq objects. |
|`cirq.resolve_parameters`| Resolves symbol parameters in the effect using the param resolver. |
|`cirq.to_json`| Write a JSON file containing a representation of obj. |
|`cirq.trace_distance_bound`| Returns a maximum on the trace distance between this effect's input |
|`cirq.trace_distance_from_angle_list`| Given a list of arguments of the eigenvalues of a unitary matrix, |
|`cirq.unitary`| Returns a unitary matrix describing the given value. |
|`cirq.validate_mixture`| Validates that the mixture's tuple are valid probabilities. |

### Quantum operator representation protocols

The following family of protocols is an important and frequently used set of features of Cirq and it is worthwhile mentioning them and and how they interact with each other. They are, in the order of increasing generality:

* `*unitary`
* `*mixture`
* `*channel`

All these protocols make it easier to work with different representations of quantum operators, namely:
- finding that representation (`unitary`, `mixture`, `channel`),
- determining whether the operator has that representation (`has_*`)
- and applying them (`apply_*`) on a state vector.

#### Unitary

The `*unitary` protocol is the least generic, as only unitary operators should implement it. The `cirq.unitary` function returns the matrix representation of the operator in the computational basis. We saw an example of the unitary protocol above, but let's see the unitary matrix of the Pauli-Y operator as well:
_____no_output_____

print(cirq.unitary(cirq.Y))[[0.+0.j 0.-1.j]
 [0.+1.j 0.+0.j]]

#### Mixture

The `*mixture` protocol should be implemented by operators that are _unitary-mixtures_. These probabilistic operators are represented by a list of tuples ($p_i$, $U_i$), where each unitary effect $U_i$ occurs with a certain probability $p_i$, and $\sum p_i = 1$. Probabilities are a Python float between 0.0 and 1.0, and the unitary matrices are numpy arrays.

Constructing simple probabilistic gates in Cirq is easiest with the `with_probability` method. _____no_output_____

probabilistic_x = cirq.X.with_probability(.3)

for p, op in cirq.mixture(probabilistic_x):
    print(f"probability: {p}")
    print("operator:")
    print(op)probability: 0.3
operator:
[[0.+0.j 1.+0.j]
 [1.+0.j 0.+0.j]]
probability: 0.7
operator:
[[1. 0.]
 [0. 1.]]

In case an operator does not implement `SupportsMixture`, but does implement `SupportsUnitary`, `*mixture` functions fall back to the `*unitary` methods. It is easy to see that a unitary operator $U$ is just a "mixture" of a single unitary with probability $p=1$. _____no_output_____

# cirq.Y has a unitary effect but does not implement SupportsMixture
# thus mixture protocols will return ((1, cirq.unitary(Y)))
print(cirq.mixture(cirq.Y))
print(cirq.has_mixture(cirq.Y))((1.0, array([[0.+0.j, 0.-1.j],
       [0.+1.j, 0.+0.j]])),)
True

#### Channel

The `channel` representation is the operator sum representation of a quantum operator (a channel):

        $$
        \rho \rightarrow \sum_{k=0}^{r-1} A_k \rho A_k^\dagger
        $$

These matrices are required to satisfy the trace preserving condition

        $$
        \sum_{k=0}^{r-1} A_k^\dagger A_k = I
        $$

where $I$ is the identity matrix. The matrices $A_k$ are sometimes called Kraus or noise operators.

The `cirq.channel` method returns a tuple of numpy arrays, one for each of the Kraus operators:_____no_output_____

cirq.channel(cirq.DepolarizingChannel(p=0.3))_____no_output_____

In case the operator does not implement `SupportsChannel`, but it does implement `SupportsMixture`, the `*channel` protocol will generate the Kraus operators based on the  `*mixture` representation.

$$
((p_0, U_0),(p_1, U_1),\ldots,(p_n, U_n)) \rightarrow (\sqrt{p_0}U_0, \sqrt{p_1}U_1, \ldots, \sqrt{p_n}U_n)
$$

Thus for example `((0.25, X), (0.75, I)) -> (0.5 X, sqrt(0.75) I)`:_____no_output_____

cirq.channel(cirq.X.with_probability(0.25))_____no_output_____

In the simplest case of a simple unitary operator, `cirq.channel` returns the exact unitary matrix as `cirq.unitary`:_____no_output_____

print(cirq.channel(cirq.Y))
print(cirq.unitary(cirq.Y))
print(cirq.has_channel(cirq.Y))(array([[0.+0.j, 0.-1.j],
       [0.+1.j, 0.+0.j]]),)
[[0.+0.j 0.-1.j]
 [0.+1.j 0.+0.j]]
True


# Notebook from anderson-github-classroom/csc-448-project
Path: students/zwinger/Project3.ipynb

# Week 3
For this week's analysis I decided to move my focus to trying to figure out what country the SARS-Cov-2 may have traveled to the USA from. Using my work from last week, I decided to continue using the data provided by https://covid19.galaxyproject.org/genomics/4-Variation/current_complete_ncov_genomes.fasta. I also am using the same multiple sequence alignment and position table as I did for my analysis last week.

However, in order to perform my analysis this week, I needed a litle more information for my data set, such as the country of origin and date of collection for each sequence in the data set. This data was provided by https://www.ncbi.nlm.nih.gov/core/assets/genbank/files/ncov-sequences.yaml. In addition to the information I was looking for, the file also contains the accesion of the sequence, accesion listing, and the state it was collected in, if that information is available. _____no_output_____

import wget

url = 'https://www.ncbi.nlm.nih.gov/core/assets/genbank/files/ncov-sequences.yaml'
filename = 'ncov-sequences.yaml'
wget.download(url, filename)_____no_output_____import yaml

#Read in the information for the sequences
file = open(filename)
info = yaml.load(file, Loader=yaml.FullLoader)
file.close()_____no_output_____

## Pulling the data out of the yaml file
Once I had read in the sequence information, I needed to next pull out what I needed and store it in a dicitonary. Using the accession of sequence as the key, I stored the country of origin and collection date of the sequence together for use later._____no_output_____

#Pull out the country and date information for each sequence
countries_and_dates = {}
for data in info['genbank-sequences']:
    country = data['country']
    #We want just the country, not the state
    if country is not None:
        country = country.split(':')[0]
    date = data['collection_date']
    countries_and_dates[data['accession']] = (country, date)_____no_output_____

## Modifying the position table
Since I was using the same position table I used for my previous analysis, I needed to read in the position table I had saved._____no_output_____

import pandas as pd
import numpy as np

#read in the position table
position_table = pd.read_csv('../../data/position_table.csv')_____no_output_____

Once I had read in the position table, I needed to add in the information I had extracted from the yaml file for each of the sequences. I also decided to add an additional name column that matched the accession for each sequence so I would not constanly neeed to remove the '.1' from the end of the seqid of the sequence in order to reference the information in the future._____no_output_____

#Add the country and date information to the position table for each sequence
for i in range(len(position_table)):
    #Add the name of the sequence without the '.1' ending
    name = position_table.loc[i, 'seqid'][:-2]
    position_table.loc[i, 'name'] = name
    if name in countries_and_dates:
        country, date = countries_and_dates[position_table.loc[i, 'name']]
        position_table.loc[i, 'country'] = country
        position_table.loc[i, 'date'] = date
    else:
        position_table.loc[i, 'country'] = None
        position_table.loc[i, 'date'] = None_____no_output_____position_table_____no_output_____

In order to clean up any missing data, I removed any entry from the position table that did not have a date and/or a country._____no_output_____

#Remove any entry without a date or country
position_table = position_table[~position_table['date'].isna()]
position_table = position_table[~position_table['country'].isna()]
position_table['date'] = pd.to_datetime(position_table['date'])_____no_output_____

In order begin finding the data to use in my analysis from the partition table, I selected the sequence from the position table with the earliest collection date in the USA as a starting point. With this sequence selected, I then wanted to find all other seuquences that had a collection date of before or on the collection date of the selected USA sequence._____no_output_____

#Find the eqrliest USA genome sequence
us_genome = sorted(position_table[position_table['country'] == 'USA']['date'])[0]
us_genome_____no_output_____genome_table = position_table.set_index("seqid")
subset_seqs = genome_table[genome_table["date"] <= us_genome].index_____no_output_____

In order to use these sequences to create a tree, I decided to create a distance matrix in order to construct a distance based phylogenetic tree._____no_output_____

#Construct the distance matrix
distances = {}
for i,seqid1 in enumerate(subset_seqs):
    distances[seqid1,seqid1]=0
    for j in range(i+1,len(subset_seqs)):
        seqid2 = subset_seqs[j]
        distances[seqid1,seqid2] = sum(genome_table.loc[seqid1] != genome_table.loc[seqid2])
        distances[seqid2,seqid1] = distances[seqid1,seqid2]
distances = pd.Series(distances).unstack()

distances_____no_output_____

## Tree Construction
Same as how I created my trees in week 2, I again utilized biopython for constructing my phylogenetic trees._____no_output_____

from Bio.Phylo.TreeConstruction import DistanceMatrix
from Bio.Phylo.TreeConstruction import DistanceTreeConstructor
from Bio import Phylo

#Convert to biopython distance matrix
matrix = np.tril(distances.values).tolist()
for i in range(len(matrix)):
    matrix[i] = matrix[i][:i+1]
dm = DistanceMatrix(list(distances.index), matrix)_____no_output_____

### Neighbor Joining Tree
The first tree I wanted to make was a distacne tree using the Neighbor Joining algorithm. Using biopython's DistanceTreeConstructor class, I was able to do so_____no_output_____

import matplotlib.pylab as plt

#Construct the NJ tree
constructor = DistanceTreeConstructor()
tree = constructor.nj(dm)

%matplotlib inline

tree.ladderize()   # Flip branches so deeper clades are displayed at top
fig = plt.figure(figsize=(10, 10))
axes = fig.add_subplot(1, 1, 1)
Phylo.draw(tree, axes=axes)_____no_output_____

### UPGA Tree
I next also wanted to compare the results of using the same data set to construct a UPGMA tree, again using biopython_____no_output_____

#Construct the UPGMA tree
constructor = DistanceTreeConstructor()
tree = constructor.upgma(dm)

%matplotlib inline

tree.ladderize()   # Flip branches so deeper clades are displayed at top
fig = plt.figure(figsize=(10, 10))
axes = fig.add_subplot(1, 1, 1)
Phylo.draw(tree, axes=axes)_____no_output_____

Both trees are quite similar in their result. As I had expected, it appears that China, specifically Wuhan, is the source of SARS-Cov-2. It also seems to appear that Wuhan is also the source of the transmission of Sars-Cov-2 to the USA as well._____no_output_____


# Notebook from BillmanH/exoplanets
Path: notebooks/Build Home System.ipynb

# Creating The base system and planets_____no_output_____

import numpy as np
import pandas as pd
from numpy import random as r

import pickle

%load_ext lab_black_____no_output_____

importing the universe module from the actual game. _____no_output_____

import sys

sys.path.insert(0, "../web/app/creators")

from universe import *_____no_output_____

Data from the userform when creating a new game:_____no_output_____

data = {
    "planet_name": "Earth",
    "num_planets": "6",
    "num_moons": "24",
    "home_has_moons": "on",
    "starting_pop": "7",
    "population_conformity": "0.3",
    "population_literacy": "0.7",
    "population_aggression": "0.5",
    "population_constitution": "0.5",
}_____no_output_____nodes, edges = build_homeSystem(data, "billmanh")_____no_output_____# nodes_____no_output_____# edges_____no_output_____

data from my stats about planet types:_____no_output_____

pdata_____no_output_____

data about moon types:_____no_output_____

mdata_____no_output_____

## The star system_____no_output_____

system = {
    "name": make_word(rnd(2, 1)),
    "label": "system",
    "objid": uuid(n=13),
}
system_____no_output_____

## the sun
just using a binary star system at this time_____no_output_____

star = {"name": make_word(rnd(2, 1)), "label": "star", "objid": uuid(n=13)}
star_____no_output_____

now a list of planets
_____no_output_____

pd.DataFrame([make_planet(p, star) for p in pdata.keys()])_____no_output_____planets = [
    make_planet(
        r.choice(list(pdata.keys()), p=[pdata[t]["prob"] for t in pdata.keys()]), star
    )
    for p in range(int(data["num_planets"]))
]
pd.DataFrame(planets).sort_values("order")_____no_output_____moons = [
    make_moon(
        r.choice(list(mdata.keys()), p=[mdata[t]["prob"] for t in mdata.keys()]),
        planets,
    )
    for p in range(int(data["num_moons"]))
]
pd.DataFrame(moons).sort_values("orbitsId")_____no_output_____

## Putting all of these objects together_____no_output_____

pd.concat(
    [
        pd.DataFrame([system]),
        pd.DataFrame([star]),
        pd.DataFrame(moons),
        pd.DataFrame(planets),
    ]
).reset_index(drop=True)_____no_output_____nodes = [system] + [star] + moons + planets_____no_output_____

## Connecting the nodes_____no_output_____all objects are connected to the system. _____no_output_____

system_____no_output_____edges = [
    {"node1": p["objid"], "node2": system["objid"], "label": "isInSystem"}
    for p in nodes
    if p["label"] != "system"
]
edges_____no_output_____orbits = [
    {"node1": p["objid"], "node2": p["orbitsId"], "label": "orbits"}
    for p in nodes
    if p.get("orbitsId")
]
orbits_____no_output_____


# Notebook from oliverslott97/lectures-2022
Path: web/04/Random_numbers_and_simulation.ipynb

# Lecture 04: Random numbers and simulation_____no_output_____[Download on GitHub](https://github.com/NumEconCopenhagen/lectures-2022)

[](https://mybinder.org/v2/gh/NumEconCopenhagen/lectures-2022/master?urlpath=lab/tree/04/Random_numbers_and_simulation.ipynb)_____no_output_____1. [Exchange economy with many consumers](#Exchange-economy-with-many-consumers)
2. [Random numbers](#Random-numbers)
3. [Demand](#Demand)
4. [Interactive figures](#Interactive-figures)
5. [Equilibrium](#Equilibrium)
6. [Numerical integration by Monte Carlo](#Numerical-integration-by-Monte-Carlo)
7. [Load and save](#Load-and-save)
8. [Summary](#Summary)
_____no_output_____You will learn how to use a random number generator with a seed and produce simulation results (**numpy.random**, **scipy.stats**), and calcuate the expected value of a random variable through Monte Carlo integration. You will learn how to save your results for later use (**pickle**). Finally, you will learn how to make your figures interactive (**ipywidgets**)._____no_output_____**Links:**

* [numpy.random](https://docs.scipy.org/doc/numpy-1.13.0/reference/routines.random.html)
* [scipy.stats](https://docs.scipy.org/doc/scipy/reference/stats.html)
* [ipywidgets](https://ipywidgets.readthedocs.io/en/stable/examples/Widget%20List.html)
* datacamp on [pickle](https://www.datacamp.com/community/tutorials/pickle-python-tutorial)_____no_output_____**Imports:** We now import all the modules, we need for this notebook. Importing everything in the beginning makes it more clear what modules the notebook relies on._____no_output_____

import math
import pickle
import numpy as np

from scipy.stats import norm # normal distribution

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
import ipywidgets as widgets_____no_output_____

# 1. Exchange economy with many consumers_____no_output_____Consider an **exchange economy** with_____no_output_____1. 2 goods, $(x_1,x_2)$
2. $N$ consumers indexed by $j \in \{1,2,\dots,N\}$
3. Preferences are Cobb-Douglas with uniformly *heterogenous* coefficients

$$
\begin{aligned}
u^{j}(x_{1},x_{2}) & = x_{1}^{\alpha_{j}}x_{2}^{1-\alpha_{j}}\\
 & \,\,\,\alpha_{j}\sim\mathcal{U}(\underline{\mu},\overline{\mu})\\
 & \,\,\,0<\underline{\mu}<\overline{\mu}<1
\end{aligned}
$$

4. Endowments are *homogenous* and given by

$$
\boldsymbol{e}^{j}=(e_{1}^{j},e_{2}^{j})=(k,1),\,k>0
$$_____no_output_____The implied **demand functions** are:

$$
\begin{aligned}
x_{1}^{\star j}(p_{1},p_{2},e^{j})&=&\alpha_{j}\frac{I}{p_{1}}=\alpha_{j}\frac{kp_{1}+p_{2}}{p_{1}} \\
x_{2}^{\star j}(p_{1},p_{2},e^{j})&=&(1-\alpha_{j})\frac{I}{p_{2}}=(1-\alpha_{j})\frac{kp_{1}+p_{2}}{p_{2}}
\end{aligned}
$$_____no_output_____The **equilibrium** for a random draw of $\alpha = \{\alpha_1,\alpha_2,\dots,\alpha_N\}$ is a set of **prices** $p_1$ and $p_2$ satifying:

$$
\begin{aligned}
x_1(p_1,p_2) = \sum_{j=1}^N x_{1}^{\star j}(p_{1},p_{2},e^{j}) &= \sum_{j=1}^N e_1^j = Nk \\
x_2(p_1,p_2) = \sum_{j=1}^N x_{2}^{\star j}(p_{1},p_{2},e^{j}) &= \sum_{j=1}^N e_2^j = N
\end{aligned}
$$_____no_output_____**Problem:** Solve for this equilibrium. But how do we handle the randomness? We need a random number generator (RNG)._____no_output_____**Warm-up**: Choose parameters and define demand functions._____no_output_____

# a. parameters
N = 1000
k = 2 # endowment
mu_low = 0.1 # lower bound on alpha
mu_high = 0.9 # upper bound on alpha

# b. demand functions
def demand_good_1_func(alpha,p1,p2,k):
    I = k*p1+p2
    return alpha*I/p1

def demand_good_2_func(alpha,p1,p2,k):
    I = k*p1+p2
    return (1-alpha)*I/p2_____no_output_____

**Quizz:** take a quick [quizz](https://forms.office.com/Pages/ResponsePage.aspx?id=kX-So6HNlkaviYyfHO_6kckJrnVYqJlJgGf8Jm3FvY9UMFpSRTIzUlJKMkdFQlpIN1VZUE9EVTBaMSQlQCN0PWcu) regarding the demand functions._____no_output_____

# 2. Random numbers_____no_output_____The two main approaches to generating random numbers are:

1. **Physical observations** of random processes (radioactive decay,  atmospheric noise, roulette wheels, etc.)
2. **Algorithms** creating pseudo-random numbers_____no_output_____**Pseudo-random numbers** satisfy propoerties such that they are as good as random. It should be impossible (for all practical purposes) to calculate, or otherwise guess, from any given subsequence, any previous or future values in the sequence._____no_output_____**More information:** See this [video](https://www.youtube.com/watch?v=C82JyCmtKWg&app=desktop#fauxfullscreen) by Infinite Series._____no_output_____## 2.1 Simple example: Middle-square method_____no_output_____Proposed by **John von Neumann**:_____no_output_____1. Start with a $N$ digit number
2. Square the number
3. Pad the number with leading zeros making it a $2N$ digit number
4. Extract the middle $N$ digits (*your random number*)
5. Return to step 1 to generate one more_____no_output_____> **Pro:** Simple and easy to implement. Conceptually somewhat similar to more advanced methods (e.g. *Mersenne-Twister* used by *numpy*).
>
> **Con:** Cycles can be no longer than $8^N$ periods. Many repeating cycles are very short. Internal state is directly observable.
>
> **Conclusion:** Can not be used in practice._____no_output_____**Code:** An implementation in Python for $N = 4$ digit random integers:_____no_output_____

def rng(number,max_iter=100):

    already_seen = [] # list of seen numbers

    i = 0
    while number not in already_seen and i < max_iter:

        already_seen.append(number)
        squared = number**2
        padded = str(squared).zfill(8) # add leading zeros
        number = int(padded[2:6]) # extract middle 4 numbers

        print(f"square = {squared:8d}, padded = {padded} -> {number:4d}")
        i += 1_____no_output_____

A reasonable cycle:_____no_output_____

rng(4653)_____no_output_____

A short cycle:_____no_output_____

rng(540)_____no_output_____

No cycle at all:_____no_output_____

rng(3792)_____no_output_____

## 2.2 Numpy_____no_output_____Numpy provides various functions for drawing random numbers. We can, for example, draw random integers between 0 and 10000:_____no_output_____

X = np.random.randint(0,10000,size=5)
print(X)_____no_output_____

**Problem:** How can we reproduce our results the next time we open Python?

**Solution:** Use a seed! Choose the seed, and reset the random number generator:_____no_output_____

print('set seed to 2000 and create numbers:')
np.random.seed(2000)
print(np.random.uniform(size=5))
print(np.random.uniform(size=5))

print('\nreset algorithm by stating the same seed again:')
np.random.seed(2000)
print(np.random.uniform(size=5))_____no_output_____

> **Note:** The first and third draws above are exactly the same._____no_output_____We can also **save and load the state** of the random number generator._____no_output_____

# a. save state
state = np.random.get_state()

# b. draw some random number
print('generate numbers from current state:')
print(np.random.uniform(size=5))
print(np.random.uniform(size=5))

# c. reset state
np.random.set_state(state)

# d. draw the same random numbers again
print('\ngenerate numbers from past state by reloading it:')
print(np.random.uniform(size=5))
print(np.random.uniform(size=5))_____no_output_____

> **Note**: You should *only set the seed once* per program. Changing seed might brake randomness._____no_output_____## 2.3 Different distributions_____no_output_____Draw random numbers from various distributions:_____no_output_____

X = np.random.normal(loc=0,scale=1,size=10**6)
Y = np.random.beta(a=5,b=2,size=10**6)
Z = np.random.uniform(low=-2,high=2,size=10**6)

vec = np.array([-2.5,-2.0,-1.5,-1.0,-0.5,0,0.5,1.0,1.5,2,2.5])
prob = (np.linspace(-1,1,vec.size)+0.1)**2 # all positive numbers
prob /= np.sum(prob) # make them sum to one
K = np.random.choice(vec,size=10**6,p=prob)_____no_output_____

Plot the various distributions:_____no_output_____

fig = plt.figure(dpi=100)
ax = fig.add_subplot(1,1,1)
ax.hist(X,bins=100,density=True,alpha=0.5,label='normal') # alpha < 1 = transparent
ax.hist(Y,bins=100,density=True,alpha=0.5,label='beta')
ax.hist(Z,bins=100,density=True,alpha=0.5,label='uniform')
ax.hist(K,bins=100,density=True,alpha=0.5,label='choice')
ax.set_xlim([-3,3])
ax.legend(loc='upper left'); # note: the ; stops output from being printed_____no_output_____

**Task:** Follow this [link](https://docs.scipy.org/doc/numpy-1.13.0/reference/routines.random.html). Choose a distribution and add it to the figure above._____no_output_____## 2.4 Analytical results_____no_output_____How close are our draws to a normal distribution?_____no_output_____

from scipy.stats import norm

# a. create analytical distribution
loc_guess = 0.25
scale_guess = 0.75
# loc_guess, scale_guess = norm.fit(X)
F = norm(loc=loc_guess,scale=scale_guess)
rnd = F.rvs(5) # example: create 5 random draws from the distribution F
print(f'F pdf at 0.0: {F.pdf(0.0): 1.3f} \nF cdf at 0.0: {F.cdf(0.0): 1.3f}') # the object F has several useful functions available

# b. vector of x values
x_low = F.ppf(0.001) # x value where cdf is 0.001
x_high = F.ppf(0.999) # x value where cdf is 0.999
x = np.linspace(x_low,x_high,100)

# c. compare
fig = plt.figure(dpi=100)
ax = fig.add_subplot(1,1,1)
ax.plot(x,F.pdf(x),lw=2,label='estimated')
ax.hist(X,bins=100,density=True,histtype='stepfilled');_____no_output_____

**Task:** Make the pdf fit the historgram._____no_output_____## 2.5 Permutations_____no_output_____

class dice_cup:

    def __init__(self,ndice):
        self.ndice = ndice

    def roll(self):
        self.dice = np.random.randint(1,7,size=self.ndice)
        print(self.dice)

    def shuffle(self):
        np.random.shuffle(self.dice)
        print(self.dice)

    def roll_and_sum(self):
        self.roll()
        print(self.dice.sum())

my_dice_cup = dice_cup(4)
my_dice_cup.roll()
my_dice_cup.shuffle()
my_dice_cup.roll_and_sum()_____no_output_____

**Task:** Add a method ``roll_and_sum()`` to the class above, which rolls and print the sum of the dice. Compare the value of your roll to your neighbor.
*(You can delete the pass statement when starting to code. It's there to inform Python that roll_and_sum() is well defined as Python cannot handle a totally codeless function)*_____no_output_____

# 3. Demand_____no_output_____$$
x_1(p_1,p_2) = \sum_{j=1}^N x_{1}^{\star j}(p_{1},p_{2},e^{j}) = \alpha_{j}\frac{kp_{1}+p_{2}}{p_{1}}
$$_____no_output_____Find demand distribution and total demand:_____no_output_____

def find_demand_good_1(alphas,p1,p2,k):
    distr = demand_good_1_func(alphas,p1,p2,k) # Notice we are passing in arrays of alphas together with scalars! It works because of numpy broadcasting.
    total = distr.sum()
    return distr,total_____no_output_____

Calculate for various prices:_____no_output_____

# a. draw alphas
alphas = np.random.uniform(low=mu_low,high=mu_high,size=N)

# b. prices
p1_vec = [0.5,1,2,5]
p2 = 1

# c. demand
dists = np.empty((len(p1_vec),N))
totals = np.empty(len(p1_vec))
for i,p1 in enumerate(p1_vec):
    dist,total = find_demand_good_1(alphas,p1,p2,k)
    dists[i,:] = dist
    totals[i] = total_____no_output_____

Plot the results:_____no_output_____

fig = plt.figure(figsize=(10,4))

ax_left = fig.add_subplot(1,2,1)
ax_left.set_title('Distributions of demand')
for i,p1 in enumerate(p1_vec):
    ax_left.hist(dists[i],density=True,alpha=0.5,label=f'$p_1 = {p1}$')
ax_left.legend(loc='upper right')

ax_right = fig.add_subplot(1,2,2)
ax_right.set_title('Level of demand')
ax_right.grid(True)
ax_right.plot(p1_vec,totals)_____no_output_____

# 4. Interactive figures_____no_output_____Create a function constructing a figure:_____no_output_____

def interactive_figure(alphas,p1,p2,k):

    # a. calculations
    dist,_total = find_demand_good_1(alphas,p1,p2,k)

    # b. figure
    fig = plt.figure(dpi=100)
    ax = fig.add_subplot(1,1,1)
    ax.hist(dist,density=True)
    ax.set_xlim([0,4]) # fixed x range
    ax.set_ylim([0,0.8]) # fixed y range_____no_output_____

**Case 1:** Make it interactive with a **slider**_____no_output_____

widgets.interact(interactive_figure,
    alphas=widgets.fixed(alphas),
    p1=widgets.FloatSlider(description="$p_1$", min=0.1, max=5, step=0.05, value=2),
    p2=widgets.fixed(p2),
    k=widgets.fixed(k)
);_____no_output_____

**Case 2:** Make it interactive with a **textbox**:_____no_output_____

widgets.interact(interactive_figure,
    alphas=widgets.fixed(alphas),
    p1=widgets.FloatText(description="$p_1$", value=2),
    p2=widgets.fixed(p2),
    k=widgets.fixed(k)
);_____no_output_____

**Case 3:** Make it interactive with a **dropdown menu**_____no_output_____

widgets.interact(interactive_figure,
    alphas=widgets.fixed(alphas),
    p1=widgets.Dropdown(description="$p_1$", options=[0.5,1,1.5,2.0,2.5,3], value=2),
    p2=widgets.fixed(p2),
    k=widgets.fixed(k)
);_____no_output_____

**Task:** Add a slider for \\(k\\) to the interactive figure below._____no_output_____

# change this code
widgets.interact(interactive_figure,
    alphas=widgets.fixed(alphas),
    p1=widgets.FloatSlider(description="$p_1$", min=0.1, max=5, step=0.05, value=2),
    p2=widgets.fixed(p2),
    k=widgets.fixed(k)
);_____no_output_____

# 5. Equilibrium_____no_output_____The equilibrium conditions (demand = supply) were:

$$
\begin{aligned}
\sum_{j=1}^N x_{1}^{\star j}(p_{1},p_{2},e^{j}) &= Nk \Leftrightarrow Z_1 \equiv \sum_{j=1}^N x_{1}^{\star j}(p_{1},p_{2},e^{j}) - Nk = 0 \\
\sum_{j=1}^N x_{2}^{\star j}(p_{1},p_{2},e^{j}) &= N \Leftrightarrow Z_2 \equiv \sum_{j=1}^N x_{2}^{\star j}(p_{1},p_{2},e^{j}) - N = 0
\end{aligned}
$$_____no_output_____**Idea:** Solve the first equation. The second is then satisfied due to Walras's law._____no_output_____**Excess demand functions:**_____no_output_____

def excess_demand_good_1_func(alphas,p1,p2,k):

    # a. demand
    demand = np.sum(demand_good_1_func(alphas,p1,p2,k))

    # b. supply
    supply = k*alphas.size

    # c. excess demand
    excess_demand = demand-supply

    return excess_demand

def excess_demand_good_2_func(alphas,p1,p2,k):

    # a. demand
    demand = np.sum(demand_good_2_func(alphas,p1,p2,k))

    # b. supply
    supply = alphas.size

    # c. excess demand
    excess_demand = demand-supply

    return excess_demand_____no_output_____

**Algorithm:**
First choose a tolerance $\epsilon > 0$ and an adjustment factor $\kappa$, and a guess on $p_1 > 0$.

Then find the equilibrium price by:

1. Calculate excess demand $Z_1 = \sum_{j=1}^N x_{1}^{\star j}(p_{1},p_{2},e^{j}) - Nk$
2. If $|Z_1| < \epsilon $ stop
3. If $|Z_1| \geq \epsilon $ set $p_1 = p_1 + \kappa \cdot \frac{Z_1}{N}$
4. Return to step 1

That is, if if excess demand is positive and far from 0, then increase the price. If excess demand is negative and far from 0, decrease the price.  _____no_output_____

def find_equilibrium(alphas,p1,p2,k,kappa=0.5,eps=1e-8,maxiter=500):

    t = 0
    while True:

        # a. step 1: excess demand
        Z1 = excess_demand_good_1_func(alphas,p1,p2,k)

        # b: step 2: stop?
        if  np.abs(Z1) < eps or t >= maxiter:
            print(f'{t:3d}: p1 = {p1:12.8f} -> excess demand -> {Z1:14.8f}')
            break

        # c. step 3: update p1
        p1 = p1 + kappa*Z1/alphas.size

        # d. step 4: print only every 25th iteration using the modulus operator
        if t < 5 or t%25 == 0:
            print(f'{t:3d}: p1 = {p1:12.8f} -> excess demand -> {Z1:14.8f}')
        elif t == 5:
            print('   ...')

        t += 1

    return p1_____no_output_____

Find the equilibrium price:_____no_output_____

p1 = 1.4
p2 = 1
kappa = 0.1
eps = 1e-8
p1 = find_equilibrium(alphas,p1,p2,k,kappa=kappa,eps=eps)_____no_output_____

**Check:** Ensure that excess demand of both goods are (almost) zero._____no_output_____

Z1 = excess_demand_good_1_func(alphas,p1,p2,k)
Z2 = excess_demand_good_2_func(alphas,p1,p2,k)
print(Z1,Z2)
assert np.abs(Z1) < eps
assert np.abs(Z2) < eps_____no_output_____

**Quizz:** take a quick quizz on the algorithm [here](https://forms.office.com/Pages/ResponsePage.aspx?id=kX-So6HNlkaviYyfHO_6kckJrnVYqJlJgGf8Jm3FvY9UMjRVRkEwQTRGVVJPVzRDS0dIV1VJWjhJVyQlQCN0PWcu)_____no_output_____

# 6. Numerical integration by Monte Carlo_____no_output_____Numerical integration is the task of computing

$$
\mathbb{E}[g(x)]  \text{ where } x \sim F
$$

and $F$ is a known probability distribution and $g$ is a function.

Relying on the law of large numbers we approximate this integral with

$$
\mathbb{E}[g(x)] \approx \frac{1}{N}\sum_{i=1}^{N} g(x_i)
$$

where $x_i$ is drawn from $F$ using a random number generator. This is also called **numerical integration by Monte Carlo**._____no_output_____**Monte Carlo function:**_____no_output_____

def g(x):
    return (x-1)**2

def MC(N,g,F):
    X = F.rvs(size=N) # rvs = draw N random values from F
    return np.mean(g(X))_____no_output_____

**Example** with a normal distribution:_____no_output_____

N = 1000
mu = 0.1
sigma = 0.5
F = norm(loc=mu,scale=sigma)
print(MC(N,g,F))_____no_output_____

Function for drawning \\( K \\) Monte Carlo samples:_____no_output_____

def MC_sample(N,g,F,K):

    results = np.empty(K)
    for i in range(K):
        results[i] = MC(N,g,F)

    return results_____no_output_____

The variance across Monte Carlo samples falls with larger $N$:_____no_output_____

K = 1000
for N in [10**2,10**3,10**4,10**5]:
    results = MC_sample(N,g,F,K)
    print(f'N = {N:8d}: {results.mean():.6f} (std: {results.std():.4f})')_____no_output_____

## 6.1 Advanced: Gauss-Hermite quadrature_____no_output_____**Problem:** Numerical integration by Monte Carlo is **slow**.

**Solution:** Use smarter integration formulas on the form

$$
\mathbb{E}[g(x)] \approx \sum_{i=1}^{n} w_ig(x_i)
$$

where $(x_i,w_i), \forall n \in \{1,2,\dots,N\}$, are called **quadrature nodes and weights** and are provided by some theoretical formula depending on the distribution of $x$.

**Example I, Normal:** If $x \sim \mathcal{N}(\mu,\sigma)$ then we can use [Gauss-Hermite quadrature](https://en.wikipedia.org/wiki/Gauss%E2%80%93Hermite_quadrature) as implemented below._____no_output_____

def gauss_hermite(n):
    """ gauss-hermite nodes

    Args:

        n (int): number of points

    Returns:

        x (numpy.ndarray): nodes of length n
        w (numpy.ndarray): weights of length n

    """

    # a. calculations
    i = np.arange(1,n)
    a = np.sqrt(i/2)
    CM = np.diag(a,1) + np.diag(a,-1)
    L,V = np.linalg.eig(CM)
    I = L.argsort()
    V = V[:,I].T

    # b. nodes and weights
    x = L[I]
    w = np.sqrt(math.pi)*V[:,0]**2

    return x,w_____no_output_____def normal_gauss_hermite(sigma, n=7, mu=None, exp=False):
    """ normal gauss-hermite nodes

    Args:

        sigma (double): standard deviation
        n (int): number of points
        mu (double,optinal): mean
        exp (bool,optinal): take exp and correct mean (if not specified)

    Returns:

        x (numpy.ndarray): nodes of length n
        w (numpy.ndarray): weights of length n

    """

    if sigma == 0.0 or n == 1:
        x = np.ones(n)
        if mu is not None:
            x += mu
        w = np.ones(n)
        return x,w

    # a. GaussHermite
    x,w = gauss_hermite(n)
    x *= np.sqrt(2)*sigma

    # b. log-normality
    if exp:
        if mu is None:
            x = np.exp(x - 0.5*sigma**2)
        else:
            x = np.exp(x + mu)
    else:
        if mu is None:
            x = x
        else:
            x = x + mu

    w /= np.sqrt(math.pi)

    return x,w_____no_output_____

**Results:** Becuase the function is "nice", very few quadrature points are actually needed (*not generally true*)._____no_output_____

for n in [1,2,3,5,7,9,11]:
    x,w = normal_gauss_hermite(mu=mu,sigma=sigma,n=n)
    result = np.sum(w*g(x))
    print(f'n = {n:3d}: {result:.10f}')_____no_output_____

**Example II, log-normal ([more info](https://en.wikipedia.org/wiki/Log-normal_distribution)):**

1. Let $\log x \sim \mathcal{N}(\mu,\sigma)$.
2. Gauss-Hermite quadrature nodes and weights can be used with the option `exp=True`.
3. To ensure $\mathbb{E}[x] = 1$ then $\mu = -0.5\sigma^2$._____no_output_____

z = np.random.normal(size=1_000_000,scale=sigma)

print('mean(x) when mu = 0')
x,w = normal_gauss_hermite(mu=0,sigma=sigma,n=7,exp=True)
print(f'MC: {np.mean(np.exp(z)):.4f}')
print(f'Gauss-Hermite: {np.sum(x*w):.4f}')
print('')

print('mean(x), mu = -0.5*sigma^2')
x,w = normal_gauss_hermite(sigma=sigma,n=7,exp=True)
print(f'MC: {np.mean(np.exp(z)-0.5*sigma**2):.4f}')
print(f'Gauss-Hermite: {np.sum(x*w):.4f}')_____no_output_____

# 7. Load and save_____no_output_____## 7.1 Pickle_____no_output_____A good allround method for loading and saving is to use **pickle**. Here is how to save:_____no_output_____

# a. variables
my_dict = {'a':1,'b':2}
my_vec = np.array([1,2,3])
my_tupple = (1,4,2)

# b. put them in a dictionary
my_data = {}
my_data['my_dict'] = my_dict
my_data['my_vec'] = my_vec
my_data['my_tupple'] = my_tupple

# c. save the dictionary in a file
with open(f'data.p', 'wb') as f: # wb = write binary
    pickle.dump(my_data, f)_____no_output_____

Delete the variables:_____no_output_____

del my_dict
del my_vec
del my_tupple_____no_output_____

Load the data again:_____no_output_____

# a. try
try:
    print(my_tupple)
except:
    print('my_vec does not exist')

# b. load
with open(f'data.p', 'rb') as f: # rb = read binary
    data = pickle.load(f)
    my_dict = data['my_dict']
    my_vec = data['my_vec']
    my_tupple = data['my_tupple']

# c. try again
print(my_vec)
print(my_tupple)_____no_output_____

## 7.2 Saving with numpy_____no_output_____When only saving/loading **numpy arrays**, an alternative is to use ``np.savez`` (or ``np.savez_compressed``). This is typically faster than pickle._____no_output_____Here is how to save some data:_____no_output_____

my_data = {}
my_data['A'] = np.array([1,2,3])
my_data['B'] = np.zeros((5,8))
my_data['C'] = np.ones((7,3,8))
np.savez(f'data.npz', **my_data)
# '**' unpacks the dictionary_____no_output_____

Here is how to load the data again:_____no_output_____

# a. delete
del my_data

# a. load all
my_data = {}
with np.load(f'data.npz') as data_obj:
    for key in data_obj.files:
        my_data[key] = data_obj[key]
print(my_data['A'])

# b. load single array
X = np.load(f'data.npz')['A']
print(X)_____no_output_____

# 8. Summary_____no_output_____**This lecture:** We have talked about:

1. numpy.random: Drawing (pseudo-)random numbers (seed, state, distributions)
2. scipy.stats: Using analytical random distributions (ppf, pdf, cdf, rvs)
3. ipywidgets: Making interactive figures
4. pickle and np.savez: Saving and loading data_____no_output_____The method you learned for finding the equilibrium can be used in a lot of models. For example, a simple method can be applied with multiple goods._____no_output_____**Your work:** Before solving Problem Set 2 read through this notebook and play around with the code._____no_output_____**Next lecture:** Workflow and debugging. Go through these guides beforehand:

1. [Installing Python and VSCode](https://numeconcopenhagen.netlify.com//guides/python-setup)
2. [Running Python in JupyterLab](https://numeconcopenhagen.netlify.com//guides/jupyterlab)
3. [Running Python in VSCode](https://numeconcopenhagen.netlify.com//guides/vscode-basics)_____no_output_____You must have installed **git** and have a **GitHub account!** (step 2 in [Installing Python and VSCode](https://numeconcopenhagen.netlify.com//guides/python-setup))._____no_output_____**Finally:** You can begin to think about who you want to work together with for the group assignments. We will talk more about inaugural project next-time._____no_output_____


# Notebook from NCBI-Hackathons/robustSingleCell
Path: analysis/PhenoGraph_clustering/03_sensitive_DE.ipynb

# Sensitive DE_____no_output_____

suppressMessages({
    library(tidyverse)
    library(limma)
    library(hashmap)
    library(magrittr)
    source('../../R/gene_signature.R')
})_____no_output_____t3k.path = '../../data/assafNormedData/T3k.RData'
load(t3k.path)
t3.tree = tree
t3.normalized = normalized _____no_output_____t4k.path = '../../data/assafNormedData/T4k.RData'
load(t4k.path)
t4.tree = tree
t4.normalized = normalized _____no_output_____concatLabels = function(unique_labels) { # unique labels (n x 2)
    label.df = unique_labels %>%
      group_by(id) %>%
      mutate(n = seq(n())) %>%
      spread(n, desc) %>%
      unite(desc, colnames(.)[-1]) %>%
      mutate(desc = sapply(desc, function(x) gsub('_NA', '', x)))
      name.map = hashmap(unique_labels$id, unique_labels$desc)
    return(label.df)
}_____no_output_____

## Load dataset_____no_output_____

t3.desc = t3.tree[, ncol(t3.tree)]
t3.id = t3.tree[,-ncol(t3.tree)]_____no_output_____head(t3.id)_____no_output_____t4.desc = t4.tree[, ncol(t4.tree)]
t4.id = t4.tree[,-ncol(t4.tree)]_____no_output_____head(t3.id)_____no_output_____

## create word version of the cluster ancestor matrix_____no_output_____

t3.desc.df = word_ancestor_matrix(t3.id, t3.desc)_____no_output_____head(t3.desc.df)_____no_output_____t4.desc.df = word_ancestor_matrix(t4.id, t4.desc)_____no_output_____head(t4.desc.df)_____no_output_____

## DE at every level_____no_output_____

sensDE_limma = function(expression, unique_cluster_membership, q_thresh = 0.1, FC_thresh = 1.2) {
    design = model.matrix(~unique_cluster_membership)
    my.lm = limma::lmFit(expression, design=design)
    my.lm = eBayes(my.lm)

    FC_genes = topTable(my.lm, number=Inf, coef = seq(2,ncol(design)))
    FC_genes = FC_genes[FC_genes$adj.P.Val < q_thresh & FC_genes$logFC > log(FC_thresh, base=2), c('logFC', 'adj.P.Val')]
    FC_genes = FC_genes[order(FC_genes$logFC, decreasing =T),]
    return(FC_genes)
}_____no_output_____DE_level_k = function(cluster_name, expression, q_thresh, FC_thresh) {
    unique_clusters = unique(cluster_name)
    for (unique_cluster in unique_clusters) {
        unique_cluster_membership = (cluster_name == unique_cluster)
        FC_genes = sensDE_limma(expression, unique_cluster_membership, q_thresh, FC_thresh)
        FC_genes$node = unique_cluster
        cluster_DE = rbind(cluster_DE, FC_genes)
    }
    return(cluster_DE)
}_____no_output_____sens_DE = function(expression.all, cell_ancestor_matrix, q_thresh, FC_thresh) {
    cluster_DE_all = c()
    for (k in K) {
    if (k == 1) {
        expression = expression.all
        cluster_name = cell_ancestor_matrix[,k]
    } else {
        summary_matrix = table(cell_ancestor_matrix[,k-1], cell_ancestor_matrix[,k])
        parent_split_idx = apply(summary_matrix, 1, function(x) sum(x > 0) == 2)
        parent_split = names(parent_split_idx)[parent_split_idx]
        subset_idx = cell_ancestor_matrix[, k-1] == parent_split
        expression = expression.all[, subset_idx]
        cluster_name = cell_ancestor_matrix[subset_idx, k]
    }
    cluster_DE_k = DE_level_k(cluster_name, expression, q_thresh, FC_thresh)
    cluster_DE_all = rbind(cluster_DE_all, cluster_DE_k)
    }
    return(cluster_DE_all)
}_____no_output_____get_jaccard<-function(inputA, inputB){
 splitA<-split(inputA,inputA$node)
 splitB<-split(inputB,inputB$node)
 num<-as.numeric(length(splitA)*length(splitB))
 jc<-data.frame(matrix(ncol=3,nrow=num))
 k=0
 for (i in seq(length(splitA))){
   v1<-splitA[[i]]$gene
   for (j in seq(length(splitB))){
     k=k+1
     print
     v2<-splitB[[j]]$gene
     jaccard_calc<-length(intersect(v1,v2))/length(union(v1,v2))
     jc[k,]<-c(names(splitA)[i],names(splitB)[j],jaccard_calc)
   }
 }
 jc$X3<-as.numeric(jc$X3)
 jc2<-matrix(jc$X3,nrow=length(unique(jc$X1)),ncol=length(unique(jc$X2)), byrow = TRUE)
 colnames(jc)<-c("node_exp1","node_exp2","j_indx")
 rownames(jc2)<-names(splitA)
 colnames(jc2)<-names(splitB)
 return(list(jc=jc,jc2=jc2))
}_____no_output_____q_thresh = 0.1
FC_thresh = 1.2
expression = t3.normalized
cell_ancestor_matrix = t3.desc.df
t3_DE = sens_DE(expression, cell_ancestor_matrix, q_thresh, FC_thresh)Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”dim(t3_DE)_____no_output_____table(t3_DE$node)_____no_output_____expression = t4.normalized
cell_ancestor_matrix = t4.desc.df
t4_DE = sens_DE(expression, cell_ancestor_matrix, q_thresh, FC_thresh)Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”Warning message:
“Zero sample variances detected, have been offset away from zero”table(t4_DE$node)_____no_output_____t3_DE$gene = rownames(t3_DE)
t4_DE$gene = rownames(t4_DE)_____no_output_____

## Create the Jaccard coefficients map between t3 and t4 clusters_____no_output_____

j <- get_jaccard(t3_DE, t4_DE)
p.jaccard = j$jc %>%
  ggplot(data=., aes(node_exp1,node_exp2)) +
  geom_tile(aes(fill=j_indx), colour="white") +
  scale_fill_gradient(low="white", high="steelblue", name='Jaccard') + xlab('3K T cell 10X dataset') + ylab('4K T cell 10X dataset') +
  theme(axis.text.x = element_text(angle=90, hjust = 1))_____no_output_____options(repr.plot.width=14, repr.plot.height=10)
p.jaccard_____no_output_____ggsave(p.jaccard, file='../../figures/jaccard_complicated_DE.pdf', width=14, height=10)_____no_output_____