# Notebook from Salad-bowl-of-knowledge/hp
Path: _notebooks/2020-05-01-sparse-coding.ipynb

# Sparse coding (Olshausen & Field, 1996) モデルの実装
>

- toc: true
- badges: true
- comments: true
- author: 山拓
- categories: [neuroscience]_____no_output_____このノートブックでは視覚におけるSparse codingのモデル ([Olshausen & Field, *Nature*. 1996](https://www.nature.com/articles/381607a0))を解説し、Pythonによる実装を行います。コードを実行したい場合は<https://github.com/takyamamoto/SparseCoding-OlshausenField-Model>をCloneして`sparse-coding.ipynb`か、`train.py`を実行してください。_____no_output_____## 画像のsparse coding

画像 $\boldsymbol{I} (\boldsymbol{x})$が基底関数(basis function) $\Phi = [\phi_i(\boldsymbol{x})]$の線形和 (係数は$\boldsymbol{r}=[r_i] $)で表されるとします。

$$
\boldsymbol{I}(\boldsymbol{x}) = \sum_i r_i \phi_i (\boldsymbol{x}) + \epsilon(\boldsymbol{x})= \Phi \boldsymbol{r} + \epsilon(\boldsymbol{x})
\tag{1}
$$

ただし、$\boldsymbol{x}$は画像上の座標, $\epsilon$は平均0のGaussianノイズを表します。また、$\boldsymbol{r}$は入力よりも高次の神経細胞の活動、$\Phi$は重み行列とします。

Sparse codingは、少数の基底で画像 (や目的変数)を表すことを目的とします。要は(1)式において、ほとんどが0で、一部だけ0以外の値を取るという疎 (=sparse, スカスカ)な係数$\boldsymbol{r}$を求めたい、ということです。なお、sparse codingは統計学や機械学習において一般に使用される語であることを念のために言っておきます。_____no_output_____## 目的関数の設定
Sparse codingのための目的関数(cost function) $E$は(2)式のようになります。

$$
E = \underbrace{\left\|\boldsymbol{I}-\Phi \boldsymbol{r}\right\|^2}_{\text{preserve information}} + \lambda \underbrace{\sum_i S\left(\frac{r_i}{\sigma}\right)}_{\text{sparseness of}\ r_i} \tag{2}
$$

ただし、$\lambda$は正則化係数、$\sigma$は定数(scaling constant)です。ここで、第一項が復元損失、第二項が罰則項 (係数が大きな値とならないようにする項)となっています。

$S(x)$としては $-\exp(-x^2), \ln(1+x^2), |x|$ などの関数が用いられます。これらの関数は原点において尖った形状をしており、解が0になりやすくなっています。描画すると以下のようになります。ただし、スケールを合わせるために $-\exp(-x^2)$には$1$を加算しました。_____no_output_____

import numpy as np
import matplotlib.pyplot as plt

x = np.arange(-5, 5, 0.1)

plt.figure(figsize=(5,4))
plt.plot(x, np.log(1+x**2), label=r"$\ln(1+x^2)$")
plt.plot(x, np.abs(x), label=r"$|x|$")
plt.plot(x, 1-np.exp(-x**2), label=r"$1-\exp(-x^2)$")
plt.xlabel("x")
plt.ylabel(r"$S(x)$")
plt.legend()
plt.tight_layout()
plt.show()_____no_output_____

次節からはSparse codingに基づいて最適な$\boldsymbol{r}$と$\Phi$を得る方法について説明していきます。_____no_output_____##  Locally Competitive Algorithm (LCA)
$\boldsymbol{r}$の勾配法による更新則は、目的関数 $E$の微分により次のように得られます。

$$
\begin{aligned}
\frac{d \boldsymbol{r}}{dt} &= -\frac{\eta_\boldsymbol{r}}{2}\frac{\partial E}{\partial \boldsymbol{r}}\\
&=\eta_\boldsymbol{r} \cdot\left[\Phi^T (\boldsymbol{I}-\Phi\boldsymbol{r})- \frac{\lambda}{2\sigma}S'\left(\frac{r_i}{\sigma}\right)\right]
\end{aligned}\tag{3}
$$

ただし、$\eta_{\boldsymbol{r}}$は学習率です。この式により$\boldsymbol{r}$が収束するまで最適化しますが、単なる勾配法ではなく、(Olshausen & Field, 1996)では**共役勾配法** (conjugate gradient method)を用いています。しかし、共役勾配法は実装が煩雑で非効率であるため、より効率的かつ生理学的な妥当性の高い学習法として、**LCA**  (locally competitive algorithm)が提案されています ([Rozell et al., *Neural Comput*. 2008](https://www.ece.rice.edu/~eld1/papers/Rozell08.pdf))。LCAは**側抑制** (local competition, lateral inhibition)と**閾値関数** (thresholding function)を用いる更新則です。LCAによる更新を行うRNNは通常のRNNとは異なり、コスト関数(またはエネルギー関数)を最小化する動的システムです。このような機構はHopfield networkで用いられているために、Olshausenは"the Hopfield trick"と呼んでいます。_____no_output_____### 軟判定閾値関数を用いる場合 (ISTA)
$S(x)=|x|$とした場合の閾値関数を用いる手法として**ISTA**(Iterative Shrinkage Thresholding Algorithm)があります。ISTAはL1-norm正則化項に対する近接勾配法で、要はLasso回帰に用いる勾配法です。

解くべき問題は(4)式で表されます。

$$
\boldsymbol{r} = \mathop{\rm arg~min}\limits_{\boldsymbol{r}}\left\{\|\boldsymbol{I}-\Phi\boldsymbol{r}\|^2_2+\lambda\|\boldsymbol{r}\|_1\right\}\tag{4}
$$

詳細は後述しますが、次のように更新することで解が得られます。

1. $\boldsymbol{r}(0)$を要素が全て0のベクトルで初期化
2. $\boldsymbol{r}_*(t+1)=\boldsymbol{r}(t)+\eta_\boldsymbol{r}\cdot \Phi^T(\boldsymbol{I}-\Phi\boldsymbol{r}(t))$
3. $\boldsymbol{r}(t+1) = S_\lambda(\boldsymbol{r}_*(t+1))$
4. $\boldsymbol{r}$が収束するまで2と3を繰り返す

ここで$S_\lambda(\cdot)$は**軟判定閾値関数** (Soft thresholding function)と呼ばれ、(5)式で表されます。

$$
S_\lambda(y)=
\begin{cases}
y-\lambda & (y>\lambda)\\
0 & (-\lambda\leq y\leq\lambda)\\
 y+\lambda & (y<-\lambda)
\end{cases}\tag{5}
$$

$S_\lambda(\cdot)$を描画すると次のようになります。_____no_output_____

import numpy as np
import matplotlib.pyplot as plt

# thresholding function of S(x)=|x|
def soft_thresholding_func(x, lmda):
    return np.maximum(x - lmda, 0) - np.maximum(-x - lmda, 0)

plt.figure(figsize=(5,4))
x = np.arange(-5, 5, 0.1)
y = soft_thresholding_func(x, 1)
plt.plot(x, x, "k--", label=r"$y=x$")
plt.plot(x, y, label=r"Soft thresholding ($\lambda=1$)")
plt.xlabel("x")
plt.ylabel(r"$S_\lambda (x)$")
plt.legend()
plt.tight_layout()
plt.show()_____no_output_____

なお、Soft thresholding関数は次の目的関数$C$を最小化する$x$を求めることで導出できます。

$$
C=\frac{1}{2}(y-x)^2+\lambda |x| \tag{6}
$$

ただし、$x, y, \lambda$はスカラー値であるとします。$|x|$が微分できませんが、これは場合分けを考えることで解決します。$x\geq 0$を考えると、(6)式は

$$
C=\frac{1}{2}(y-x)^2+\lambda x = \{x-(y-\lambda)\}^2+\lambda(y-\lambda) \tag{7}
$$

となります。ここで、(7)式の最小値を与える$x$は場合分けをして考えると、$y-\lambda\geq0$のとき二次関数の頂点を考えて$x=y-\lambda$となります。 一方で$y-\lambda<0$のときは$x\geq0$において単調増加な関数となるので、最小となるのは$x=0$のときです。同様の議論を$x\leq0$に対しても行うことで (5)式が得られます。_____no_output_____### Cauchy分布の閾値関数を用いる場合
次に、$S(x)=\ln(1+x^2)$を用いた場合を考えます。これはCauchy分布を$\boldsymbol{r}$の事前分布 $p(\boldsymbol{r})=\prod_i p(r_i)=\prod_i \exp\left[-\alpha \ln(1+r_i^2)\right]$としたときの負の対数事前分布 $g(\boldsymbol{r}):=-\ln p(\boldsymbol{r})=\alpha \sum_i \ln(1+r_i^2)$に対応しています。

前節と同様に次式で表される、1次元の目的関数$C$を最小にする$x$を求めます。

$$
C=(y-x)^2+\lambda \ln(1+x^2) \tag{8}
$$

今度は微分可能なので(8)式の勾配が0となるような$x$を求めます。

$$
\frac{\partial C}{\partial x}=-2(y-x)+\frac{2\lambda x}{1+x^2} \tag{9}
$$

(9)式 $=0$とおいて$x$について整理したいところですが、まずは$y$について解いてみると、

$$
y=x+\frac{\lambda x}{1+x^2} \tag{10}
$$

となります。後は(10)式の逆関数を求めればよいのですが、どう考えても複雑なのでWolfram alphaを使いましょう ([結果](https://ja.wolframalpha.com/input/?i=y%3Dax%2F%281%2Bx%5E2%29%2Bx%E3%81%AE%E9%80%86%E9%96%A2%E6%95%B0))。解を整理すると(11)式のように表されます。

$$
\begin{aligned}
S_\lambda(y)&=\frac{1}{3}\left[y-\frac{h_\lambda(y)}{\sqrt[3]{2}}+\frac{\sqrt[3]{2} \cdot g_\lambda(y)}{h_\lambda(y)}\right]\\
f_\lambda(y) &:= 9\lambda y - 2y^3-18y\\
g_\lambda(y) &:= 3\lambda - y^2+3\\
h_\lambda(y) &:= \left[\sqrt{f_\lambda(y)^2+4g_\lambda(y)^3}+f_\lambda(y)\right]^\frac{1}{3}
\end{aligned}\tag{11}
$$

(11)式を実装してみると次のようになります。_____no_output_____

# thresholding function of S(x)=ln(1+x^2)
def ln_thresholding_func(x, lmda):
    f = 9*lmda*x - 2*np.power(x, 3) - 18*x
    g = 3*lmda - np.square(x) + 3
    h = np.cbrt(np.sqrt(np.square(f) + 4*np.power(g, 3)) + f)
    two_croot = np.cbrt(2) # cubic root of two
    return (1/3)*(x - h / two_croot + two_croot*g / (1e-8+h))

plt.figure(figsize=(5,4))
x = np.arange(-5, 5, 0.1)
y = ln_thresholding_func(x, 1)
plt.plot(x, x, "k--", label=r"$y=x$")
plt.plot(x, y, label=r"$\ln(1+x^2)$ thresholding ($\lambda=1$)")
plt.xlabel("x")
plt.ylabel(r"$S_\lambda (x)$")
plt.legend()
plt.tight_layout()
plt.show()_____no_output_____

Soft thresholding 関数を緩やかにしたような形状であることが分かります。しかし、これをそのまま用いてもスパースな表現は得られませんでした (ハイパーパラメータの調整不足かもしれませんが)。そこで、([Mayo et al., arXiv. 2020](https://arxiv.org/abs/2003.12507))で提案されている**Cauchy thresholding**関数を用いてみましょう。

Cauchy thresholding関数は次式で表されます。

$$
S_\lambda(y)=
\begin{cases}
\dfrac{1}{2}\left(y+\sqrt{y^2-4\lambda}\right) & (y>2\lambda)\\
0 & (-2\lambda\leq y\leq 2\lambda)\\
\dfrac{1}{2}\left(y-\sqrt{y^2-4\lambda}\right) & (y<-2\lambda)
\end{cases}\tag{12}
$$

実装は次のようになります。_____no_output_____

def cauchy_thresholding_func(x, lmda):
    f = 0.5*(x + np.sqrt(np.maximum(x**2 - lmda,0)))
    g = 0.5*(x - np.sqrt(np.maximum(x**2 - lmda,0)))
    return f*(x>=lmda) + g*(x<=-lmda)

plt.figure(figsize=(5,4))
x = np.arange(-5, 5, 0.1)
ln_thresholding_func
y = cauchy_thresholding_func(x, 1)
plt.plot(x, x, "k--", label=r"$y=x$")
plt.plot(x, y, label=r"Cauchy thresholding ($\lambda=1$)")
plt.xlabel("x")
plt.ylabel(r"$S_\lambda (x)$")
plt.legend()
plt.tight_layout()
plt.show()_____no_output_____

硬判定閾値関数 (hard thresholding function)に似た形状をしていることが分かります。_____no_output_____## 重み行列の更新則
$\boldsymbol{r}$が収束したら勾配法により$\Phi$を更新します。

$$
\begin{aligned}
\Delta \phi_i(\boldsymbol{x}) &= -\eta \frac{\partial E}{\partial \Phi}\\
&=\eta\cdot\left[\left([\boldsymbol{I}-\Phi\boldsymbol{r}\right)\boldsymbol{r}^T\right]
\end{aligned}\tag{13}
$$_____no_output_____## Sparse coding networkの実装
### ネットワークの構造
入力層を含め2層の単純な構造です。今回は、入力はランダムに切り出した16×16 (＝256)の画像パッチとし、これを入力層の256個のニューロンが受け取るとします。入力層のニューロンは次層の100個のニューロンに投射するとします。100個のニューロンが入力をSparseに符号化するようにその活動および重み行列を最適化します。下図はネットワークの構造図です。ただし、$a_i$は本記事における$r_i$に対応します。_____no_output_____![fig2a](images/sparse-coding_figs/fig2a.JPG)

 (Olshausen, 2002; Fig. 2a)
_____no_output_____### ライブラリのimport_____no_output_____

import numpy as np
import matplotlib.pyplot as plt
import network
from tqdm.notebook import tqdm
import scipy.io as sio

np.random.seed(0)_____no_output_____

### 画像データの読み込み
データは<http://www.rctn.org/bruno/sparsenet/>からダウンロードできます。MATLABコードもダウンロードできますが、手元では動かすことができませんでした。`IMAGES.mat`には白色化された10枚の自然画像が記録されています。画像の描画などは『[Predictive coding (Rao & Ballard, 1999) モデルの実装](https://salad-bowl-of-knowledge.github.io/hp/neuroscience/2020/04/25/predictive-coding.html)』を参照してください。_____no_output_____

# datasets from http://www.rctn.org/bruno/sparsenet/
mat_images = sio.loadmat('datasets/IMAGES.mat')
imgs = mat_images['IMAGES']_____no_output_____

### ネットワークの実装
LCAに基づき、ネットワークを実装します。基本的には`__call__`関数を実行して係数`r`と重み行列`Phi`を更新します。外部から呼び出し、`r`を収束するまで更新し、収束したときに`training`を`True`にして`Phi`を更新します。また、閾値関数としては`soft_thresholding_func`, `ln_thresholding_func`, `cauchy_thresholding_func`の3つを使用できるようにしています。どれを用いるかはコメントアウトを外すなどして変更してください。_____no_output_____

class OlshausenField1996Model:
    def __init__(self, num_inputs, num_units, batch_size,
                 lr_r=1e-2, lr_Phi=1e-2, lmda=5e-3):
        self.lr_r = lr_r # learning rate of r
        self.lr_Phi = lr_Phi # learning rate of Phi
        self.lmda = lmda # regularization parameter

        self.num_inputs = num_inputs
        self.num_units = num_units
        self.batch_size = batch_size

        # Weights
        Phi = np.random.randn(self.num_inputs, self.num_units).astype(np.float32)
        self.Phi = Phi * np.sqrt(1/self.num_units)

        # activity of neurons
        self.r = np.zeros((self.batch_size, self.num_units))

    def initialize_states(self):
        self.r = np.zeros((self.batch_size, self.num_units))

    def normalize_rows(self):
        self.Phi = self.Phi / np.maximum(np.linalg.norm(self.Phi, ord=2, axis=0, keepdims=True), 1e-8)

    # thresholding function of S(x)=|x|
    def soft_thresholding_func(self, x, lmda):
        return np.maximum(x - lmda, 0) - np.maximum(-x - lmda, 0)

    # thresholding function of S(x)=ln(1+x^2)
    def ln_thresholding_func(self, x, lmda):
        f = 9*lmda*x - 2*np.power(x, 3) - 18*x
        g = 3*lmda - np.square(x) + 3
        h = np.cbrt(np.sqrt(np.square(f) + 4*np.power(g, 3)) + f)
        two_croot = np.cbrt(2) # cubic root of two
        return (1/3)*(x - h / two_croot + two_croot*g / (1e-8+h))

    # thresholding function https://arxiv.org/abs/2003.12507
    def cauchy_thresholding_func(self, x, lmda):
        f = 0.5*(x + np.sqrt(np.maximum(x**2 - lmda,0)))
        g = 0.5*(x - np.sqrt(np.maximum(x**2 - lmda,0)))
        return f*(x>=lmda) + g*(x<=-lmda)

    def calculate_total_error(self, error):
        recon_error = np.mean(error**2)
        sparsity_r = self.lmda*np.mean(np.abs(self.r))
        return recon_error + sparsity_r

    def __call__(self, inputs, training=True):
        # Updates
        error = inputs - self.r @ self.Phi.T

        r = self.r + self.lr_r * error @ self.Phi
        self.r = self.soft_thresholding_func(r, self.lmda)
        #self.r = self.cauchy_thresholding_func(r, self.lmda)

        if training:
            error = inputs - self.r @ self.Phi.T
            dPhi = error.T @ self.r
            self.Phi += self.lr_Phi * dPhi

        return error, self.r_____no_output_____

### モデルの定義
上で実装したネットワークを`model`として定義します。また、シミュレーションで用いる定数などを初期化しておきます。_____no_output_____

# Simulation constants
H, W, num_images = imgs.shape
num_iter = 500 # number of iterations
nt_max = 1000 # Maximum number of simulation time
batch_size = 250 # Batch size

sz = 16 # image patch size
num_units = 100 # number of neurons (units)

eps = 1e-2 # small value which determines convergence
error_list = [] # List to save errors

# Define model
model = OlshausenField1996Model(num_inputs=sz**2, num_units=num_units,
                                batch_size=batch_size)_____no_output_____

### シミュレーションの実行
シミュレーションを実行します。外側の`for loop`では画像パッチの作成と`r`の初期化を行います。内側の`for loop`では`r`が収束するまで更新を行い、収束したときに重み行列`Phi`を更新します。_____no_output_____

# Run simulation
for iter_ in tqdm(range(num_iter)):
    # Get the coordinates of the upper left corner of clopping image randomly.
    beginx = np.random.randint(0, W-sz, batch_size)
    beginy = np.random.randint(0, H-sz, batch_size)

    inputs_list = []

    # Get images randomly
    for i in range(batch_size):
        idx = np.random.randint(0, num_images)
        img = imgs[:, :, idx]
        clop = img[beginy[i]:beginy[i]+sz, beginx[i]:beginx[i]+sz].flatten()
        inputs_list.append(clop - np.mean(clop))

    inputs = np.array(inputs_list) # Input image patches

    model.initialize_states() # Reset states
    model.normalize_rows() # Normalize weights

    # Input an image patch until latent variables are converged
    r_tm1 = model.r # set previous r (t minus 1)

    for t in range(nt_max):
        # Update r without update weights
        error, r = model(inputs, training=False)
        dr = r - r_tm1

        # Compute norm of r
        dr_norm = np.linalg.norm(dr, ord=2) / (eps + np.linalg.norm(r_tm1, ord=2))
        r_tm1 = r # update r_tm1

        # Check convergence of r, then update weights
        if dr_norm < eps:
            error, r = model(inputs, training=True)
            break

        # If failure to convergence, break and print error
        if t >= nt_max-2:
            print("Error at patch:", iter_)
            print(dr_norm)
            break

    error_list.append(model.calculate_total_error(error)) # Append errors

    # Print moving average error
    if iter_ % 100 == 99:
        print("iter: "+str(iter_+1)+"/"+str(num_iter)+", Moving error:",
              np.mean(error_list[iter_-99:iter_]))_____no_output_____

### 訓練中の損失の描画_____no_output_____訓練中の損失の変化を描画してみましょう。損失が低下し、学習が進行したことが分かります。_____no_output_____

# Plot error
plt.figure(figsize=(5, 3))
plt.ylabel("Error")
plt.xlabel("Iterations")
plt.plot(np.arange(len(error_list)), np.array(error_list))
plt.tight_layout()
plt.show()_____no_output_____

### 重み行列 (受容野)の描画
学習後の重み行列 `Phi` ($\Phi$)を可視化してみましょう。_____no_output_____

# Plot Receptive fields
fig = plt.figure(figsize=(6, 6))
plt.subplots_adjust(hspace=0.1, wspace=0.1)
for i in tqdm(range(num_units)):
    plt.subplot(10, 10, i+1)
    plt.imshow(np.reshape(model.Phi[:, i], (sz, sz)), cmap="gray")
    plt.axis("off")

fig.suptitle("Receptive fields", fontsize=20)
plt.subplots_adjust(top=0.9)
plt.show()_____no_output_____

白色が**ON領域**(興奮)、黒色が**OFF領域**(抑制)を表します。Gaborフィルタ様の局所受容野が得られており、これは一次視覚野(V1)における単純型細胞(simple cells)の受容野に類似しています。

なお、`cauchy_thresholding_func`を用い、`lmda=1e-3`とした結果は次図のようになりました。_____no_output_____![RF_cauchy_thresholding](images/sparse-coding_figs/RF_cauchy_thresholding.png)_____no_output_____## ICAとPCAによる自然画像の分析
参考までに自然画像に対してICAとPCAを実行してみます。ICAを実行することはSparse codingを行うことと同じであり、結果としてGaborフィルタ様の局所受容野が得られます。一方で、PCAを実行しても広域的な受容野しか得られません ([Sanger TD, 1989](https://papers.nips.cc/paper/139-an-optimality-principle-for-unsupervised-learning))。

### ライブラリのimport
`sklearn.decomposition`を持ちます。_____no_output_____

from sklearn.decomposition import FastICA, PCA_____no_output_____

### 画像パッチの作成
白色化前の画像を用いた方がPCAの結果が綺麗です。_____no_output_____

mat_images_raw = sio.loadmat('datasets/IMAGES_RAW.mat')
imgs_raw = mat_images_raw['IMAGESr']

# Simulation constants
H, W, num_images = imgs_raw.shape

num_patches = 15000
patchs_list = []
w, h = 16, 16 # patch size

# generate patches
for i in tqdm(range(num_patches)):
    i = np.random.randint(0, num_images)
    # Get the coordinates of the upper left corner of clopping image randomly.
    beginx = np.random.randint(0, W-w-1)
    beginy = np.random.randint(0, H-h-1)
    img_clopped = imgs_raw[beginy:beginy+h, beginx:beginx+w, i]
    patchs_list.append(img_clopped.flatten())

patches = np.array(patchs_list)_____no_output_____

### ICAの実行_____no_output_____

# perform ICA
n_comp = 100
ica = FastICA(n_components=n_comp)
ica.fit(patches)
ica_filters = ica.components_

# plot filters
plt.figure(figsize=(6,6))
plt.subplots_adjust(hspace=0.1, wspace=0.1)
for i in tqdm(range(n_comp)):
    plt.subplot(10, 10, i+1)
    plt.imshow(np.reshape(ica_filters[i], (w, h)), cmap="gray")
    plt.axis("off")
plt.suptitle("ICA", fontsize=20)
plt.subplots_adjust(top=0.9)
plt.show()_____no_output_____

他にはK-means法を用いることでもGaborフィルタを得ることはできます。
- [Gabors / Primary Visual Cortex “Simple Cells” from an Image — skimage v0.13.1 docs](https://scikit-image.org/docs/0.13.x/auto_examples/features_detection/plot_gabors_from_astronaut.html#sphx-glr-auto-examples-features-detection-plot-gabors-from-astronaut-py)_____no_output_____### PCAの実行_____no_output_____

# perform PCA
pca = PCA(n_components=n_comp)
pca.fit(patches)
pca_filters = pca.components_

# plot filters
plt.figure(figsize=(6,6))
plt.subplots_adjust(hspace=0.1, wspace=0.1)
for i in tqdm(range(n_comp)):
    plt.subplot(10, 10, i+1)
    plt.imshow(np.reshape(pca_filters[i], (w, h)), cmap="gray")
    plt.axis("off")
plt.suptitle("PCA", fontsize=20)
plt.subplots_adjust(top=0.9)
plt.show()_____no_output_____

## まとめ
- Sparse codingのモデルを実装した。
- LCA (locally competitive algorithm)は効率良くsparce codingをneural networkで実現するための手法。
- Sparce codingによりV1の単純型細胞と同様のGaborフィルタが得られる。
- 脳においても発火の抑制などによる正則化によりsparse codingが実現されていると考えられる。_____no_output_____## 参考文献
### 論文
- Olshausen BA, Field DJ. [Emergence of simple-cell receptive field properties by learning a sparse code for natural images](https://www.nature.com/articles/381607a0). *Nature*. 1996;381(6583):607–609. [Data and Code](http://www.rctn.org/bruno/sparsenet/), [pdf](https://courses.cs.washington.edu/courses/cse528/11sp/Olshausen-nature-paper.pdf)
- Olshausen BA. [Principles of Image Representation in Visual Cortex](https://groups.oist.jp/sites/default/files/img/ocnc/2004/Olshausen.pdf). *The Visual Neurosciences*. 2002.
- Rozell CJ, Johnson DH, Baraniuk RG, Olshausen BA. [Sparse coding via thresholding and local competition in neural circuits](http://www.mit.edu/~9.54/fall14/Classes/class07/Palm.pdf). *Neural Comput*. 2008;20(10):2526‐2563.
- Mayo P, Holmes R, Achim A. [Iterative Cauchy Thresholding: Regularisation with a heavy-tailed prior](https://arxiv.org/abs/2003.12507). arXiv. 2020.
- Paiton DM. [Analysis and applications of the Locally Competitive Algorithm](https://escholarship.org/uc/item/1wz289gt). UC Berkeley. 2019.

### GitHubで確認した他の実装例
- https://github.com/lpjiang97/sparse-coding

### Olshausen先生の講義
- [Bruno Olshausen: “Sparse coding in brains and machines”(Stanford talks)](https://talks.stanford.edu/bruno-olshausen-sparse-coding-in-brains-and-machines/), [Slide](http://www.rctn.org/bruno/public/Simons-sparse-coding.pdf)
- https://redwood.berkeley.edu/wp-content/uploads/2018/08/sparse-coding-LCA.pdf

### スパースモデリング
- [今日からできるスパースモデリング](http://www-adsys.sys.i.kyoto-u.ac.jp/mohzeki/Presentation/lecturenote20150909.pdf)
- [スパースモデリングを超初歩から始めてみた。(ISTAを実装した) - MAEA2’s diary](http://maea2.hatenablog.com/entry/2017/03/06/235519)_____no_output_____


# Notebook from skoltech-nlp/coqas
Path: generation/Student/TextRank.ipynb

_____no_output_____# Load Data_____no_output_____We have set of 538 samples generated by CAM. Each sample contains numerical score, objects and supportive sentences retrived by elastic search engine. For more detailed information please visit: link _____no_output_____

import json_____no_output_____data = []
for line in open('./mined_bow_str.json', 'r'):
    data.append(json.loads(line))_____no_output_____# usefull function that extracts all supporting sentences from the CAM output
def write_sentences(sample, sentences = None):
    if sentences is None:
        sentences = []
    for s in sample['object1']['sentences']:
        sentences.append(s['text'] + '\n')
    for s in sample['object2']['sentences']:
        sentences.append(s['text'] + '\n')
    return sentences_____no_output_____e =data[6]['object1']['sentences']_____no_output_____

Let's choose some samples to compare model's performance_____no_output_____

samples = [data[4], data[155], data[228]]

for s in samples:
  print(s['object1']['name'] + ' vs ' + s['object2']['name'])python vs java
toyota vs nissan
tea vs juice
for s in samples:
    write_sentences(s)_____no_output_____

# Graph Routine

TextRank is a graph method used for summarisation. It extracts vetices of the graph that are most important in terms of similarity function to all other verticies. We use networkx library implementation of PageRank algorithm, then we sort vertices and get top k sentences with the highest PageRank score.

In Graph-based Text Summarization Using Modified TextRank, C. Mallick et al., 2018, https://www.researchgate.net/publication/327136473_Graph-based_Text_Summarization_Using_Modified_TextRank there are two modifications of TextRank algorithm suggested for summarization task:
- prune graph so that only egdes with weights higher than average are left
- initialize TextRank with node weigt proportional to the importance of the edges incident to it_____no_output_____

import networkx as nx_____no_output_____def prune_graph(scores):
    mean = np.array([s[2] for s in scores]).mean()
    pruned = []
    for s in scores:
        if s[2] > mean:
            pruned.append(s)
    return pruned_____no_output_____def init_pagerank(scores):
    weights = {}
    n_edges = {}
    for i, j, w in scores:
        prev_score, prev_n = weights.get(i, 0.0), n_edges.get(i, 0)
        weights[i] = w + prev_score
        n_edges[i] = 1 + prev_n
        prev_score, prev_n = weights.get(j, 0.0), n_edges.get(j, 0)
        weights[j] = w + prev_score
        n_edges[j] = 1 + prev_n
    for k in weights:
        v = weights[k]
        weights[k] = v/n_edges[k]
    return weights_____no_output_____import networkx as nx

def init_pagerank(scores):
    weights = {}
    n_edges = {}
    for i, j, w in scores:
        prev_score, prev_n = weights.get(i, 0.0), n_edges.get(i, 0)
        weights[i] = w + prev_score
        n_edges[i] = 1 + prev_n
        prev_score, prev_n = weights.get(j, 0.0), n_edges.get(j, 0)
        weights[j] = w + prev_score
        n_edges[j] = 1 + prev_n
    for k in weights:
        v = weights[k]
        weights[k] = v/n_edges[k]
    return weights

def extract_top(scores, k = 10, weighted_init = False):
    g = nx.Graph()
    g.add_weighted_edges_from(scores)
    if weighted_init:
        pr = nx.pagerank(g, nstart = init_pagerank(scores))
    else:
        pr = nx.pagerank(g)
    idxes = sorted(pr, key = lambda x: -pr[x])[:k]
    return idxes_____no_output_____

# Chinese Whispers
(Chris Biemann, 2006, https://www.aclweb.org/anthology/W06-3812.pdf)

Is an efficient graph clustering algorithm that on each iteration sets class of one of the vertices equal to the highest ranked class among the vertices neighbours. It is shown efficient for NLP tasks and does not requires pre-defined number of clusters.

It is suggested to cluster graph and build separate TextRank models for each cluster. We assume that each clusster will correspond to one important agrument  or topic. As a result, we may extract only the most high-ranked sentence from each cluster and use them as a complete summary of CAM output.

Here we use original implementation of Chinese Whispers availiable in PyPi. As edges weights we may use any similarity function presented below._____no_output_____

!pip install chinese_whispersRequirement already satisfied: chinese_whispers in /home/user/.local/lib/python3.6/site-packages (0.6.1)
Requirement already satisfied: networkx<3.0,>=2.1 in /home/user/.local/lib/python3.6/site-packages (from chinese_whispers) (2.4)
Requirement already satisfied: decorator>=4.3.0 in /opt/conda/lib/python3.6/site-packages (from networkx<3.0,>=2.1->chinese_whispers) (4.3.0)
from chinese_whispers import chinese_whispers, aggregate_clusters_____no_output_____def cw_summarize(scores, k = None, weighted_init = False):
    g = nx.Graph()
    g.add_weighted_edges_from(scores)
    pos = nx.spring_layout(g)
    edges,weights = zip(*nx.get_edge_attributes(g,'weight').items())
    nx.draw(g, pos, node_color='b', edgelist=edges, edge_color=weights)
    plt.show()
    chinese_whispers(g, weighting='log', iterations = 1)

    scores = {(s[0], s[1]): s[2] for s in scores}

    clusters = []
    for label, cluster in sorted(aggregate_clusters(g).items(), key=lambda e: len(e[1]), reverse=True):
        clusters.append(cluster)

    print(len(clusters))
    tops = []
    for c in clusters:
        if len(c) == 1:
            tops.append(list(c)[0])
        else:
            scores_c = []
            for i in c:
                for j in c:
                    if (i, j) in scores:
                        scores_c.append((i, j, scores[(i, j)]))
            top_k = extract_top(scores_c, k = 1, weighted_init=weighted_init)
            tops.append(top_k[0])
    return tops_____no_output_____

# Similarity Functions:

We use an original version of PageRank algorithm, we also compare it with suggested heuristic based on Chinese Whispers Algorithm.

Similarity between sentences can be defined in different ways, here we apply some of them:_____no_output_____### Words Co-occurence

In this approach we use number of words in common normalized by the cumulative length of both sentences. Here we use stemmer so that different forms of the same word are counted as the same token. We also do not take into consideration number of word occurence in sentences.

Let's define sentence representation
$$s = \{w: w = stemmer(\tilde{w}), \tilde{w} \in sentence \}$$
where $s$ is a set without repetition, then similarity function is
$$f(s_1, s_2) = \frac {\sum_{w \in s_1} \mathbb{1}(w \in s_2)}{|s_1| + |s_2|}$$
$$\mathbb{1}(w \in s_2) = 1 \ \text{if} \ w \in s_2, \ \text{else} \ 0$$
_____no_output_____

def similarity1(s1, s2):
    if len(s1) == 0 and len(s2) == 0:
        return 0.0

    inter = 0.0
    for w in s1:
        inter += 1
    return inter/((len(s1) + len(s2)))_____no_output_____

It is also possible to exclude common words from the similarity function calculation. It can be usefull because only meaningfull words are used and common phrases do not affect the similarity. We use nltk stopwords here._____no_output_____

#make sure that stopwords are downloaded
import nltk
nltk.download('stopwords')[nltk_data] Error loading stopwords: <urlopen error [Errno 110]
[nltk_data]     Connection timed out>
from nltk.corpus import stopwords
from nltk.tokenize import TweetTokenizer, RegexpTokenizer
from nltk.stem.snowball import SnowballStemmer
from collections import Counter
import numpy as np
import matplotlib.pyplot as plt_____no_output_____#use for Chinese Whispers:
#summarize = cw_summarize

#use for TextRank:
summarize = extract_top

exclude_common = True
prune = True
weighted_init = True

stemmer = SnowballStemmer(language='english')
tokenizer = RegexpTokenizer(r'\w+')
summaries = []
k = 10

for s in samples:
    raw_sentences = write_sentences(s)
    sentences = []
    for sentence in raw_sentences:
        words = tokenizer.tokenize(sentence)
        sentences.append(' '.join(words))
    sentences = list(set(sentences))
    word_sets = []
    for sentence in sentences:
        words = sentence.split()
        common = stopwords.words('english') + [s['object1']['name'], s['object2']['name']]
        if exclude_common:
            words = [w.lower() for w in words if w.lower() not in common]

        words = set([stemmer.stem(word) for word in words])
        word_sets.append(words)

    scores = []
    for i in range(len(sentences)):
        for j in range(len(sentences)):

            scores.append((i, j, similarity1(word_sets[i], word_sets[j])))

    if prune:
        scores = prune_graph(scores)

    top_k = summarize(scores, k = k, weighted_init = weighted_init)

    summaries.append([sentences[i] for i in top_k])_____no_output_____for i, s in enumerate(samples):
    print(s['object1']['name'] + ' vs ' + s['object2']['name'] + ':')
    for sent in summaries[i]:
        print(sent + '.')
    print()python vs java:
python than for java.
Aside from the perils of late binding the other major concern I have about Python for mission critical apps is that the Python designers didn t include a posse of security paranoids the way Java and JEE did so exploits are potentially more possible.
In practice I find that my Python programs perform somewhat better than Java probably because Python has many libraries implemented in C whereas Java has many pure Java libraries.
Using the decimal library in python with 90 significant digits is massive overkill but I plan to use those results as a baseline for comparison with both the java app and the python app.
most humorously the major reason python is touted over java is the reason you give for java over c python is even more high level than java meaning higher productivity.
BTW AFAIK the python language evolution is managed much in the same way that Java JSR for Java vs PEPs for Python JCP for Java vs PSF for Python.
Python is slower than Java.
Python is better than JAVA.
Python is no more interpreted than Java.
Java is faster while python is.

toyota vs nissan:
I chose the Toyota over the Nissan.
More than Toyota more than Nissan or any of the brands.
i will not purchase another toyota product because of the way s e toyota dealt with us over the alignment issue but s e toyota gets 500 from the buyer for each car nissan gm ford chrysler etc do not have such a fee.
And Toyota was smaller than Nissan and Honda.
Nissan is a good machine but not as good as a Toyota.
The context of your message would indicate that the engine would be more in sync in the TCH than the NAH e g Toyota with Toyota rather than Nissan with Toyota.
Nissan Altima Nissan sold an astonishing 41 050 Altimas during March just a handful less than the segment powerhouse Toyota Camry.
To explain why I liken Giant to Nissan rather than Toyota is because Toyota is revered in my opinion too much for quality while Nissan has a more benign reputation.
However since the Nissan CC appears to be considerably less expensive than the Toyota DC my vote goes to Nissan.
My limited experience tells me that overall Toyota is better than Nissan especially in terms of attention in details quietness.

tea vs juice:
We don t drink juice but we do drink tea or water.
POM Wonderful Facts POM Wonderful 100 Pomegranate Juice is the super juice that contains more antioxidants than red wine grape juice acai juice and even green tea.
What is so wonderful about this Pomegranate juice is its antioxidant potency where better than green tea cranberry juice red wine concord grape juice and blueberry juice.
According to POM their juice has more antioxidant power than other drinks including red wine blueberry juice green tea and cranberry juice.
In the summer its juice is offered instead of tea.
This tea is nothing more than hot leaf juice.
Pomegranate juice has more antioxidants than green tea.
Iroh This tea is nothing more than hot leaf juice.
Only problem is that this juice isn t as good as the green tea cartos.
Juice Frozen orange juice is cheaper but usually I just stick to tea and buy the GOOD orange juice for special occasions.

### TF-IDF

Term frequency – inverse document frequency is a common statistic used in number of natural language processing models. As a similarity function here we use cosine distance between tf-idf representations of the sentences:

$$f(s_1, s_2) = \frac {\left<s_1, s_2\right>}{\|s_1\|\|s_2\|}$$

where tf-idf of a sentence $s$ is a product of tf representation of a sentence and idf statistic of a set of documents retrieved by CAM for the given query:
$$s = tf \times idf$$
$$tf(t, s) = \frac{n_t}{|s|}$$,
where $t$ is a term, s is a sentence and n_t is a number of occurence of term t in sentence s. Here we also use stemmer.

$$idf(t, S) = log \frac{|S|}{|\{s: t \in s \ \& \ s \in S\}|}$$,
where S is a set of documents, t is a term and s is a sentence from S._____no_output_____

from nltk.tokenize import TweetTokenizer, RegexpTokenizer
from nltk.stem.snowball import SnowballStemmer
from collections import Counter
import numpy as np_____no_output_____def make_vocab(sentences):
    counter = Counter()
    for s in sentences:
        words = s.split(' ')
        words = [stemmer.stem(word) for word in words]
        counter.update(words)

    vocab = {}
    i = 0
    for key in counter:
        vocab[key] = i
        i += 1
    return vocab

def compute_idf(sentences, vocab):
    d = np.zeros(len(vocab))
    word_sets = []
    for s in sentences:
        words = s.split()
        word_sets.append(set([stemmer.stem(word) for word in words]))

    for s in word_sets:
        for w in s:
            d[vocab[w]] += 1

    idf = np.log(len(sentences)/d)
    return idf

def tf_idf(sentence, vocab, idf):
    res = np.zeros(len(vocab))
    for w in sentence.split():
        res[vocab[stemmer.stem(w)]] += 1
    res /= res.sum()
    res *= idf
    return res_____no_output_____def similarity(s1, s2):
    return s1.dot(s2)/np.linalg.norm(s1)/np.linalg.norm(s2) _____no_output_____len(samples)_____no_output_____#use for Chinese Whispers:
summarize = cw_summarize

#use for TextRank:
#summarize = extract_top

prune = True
weighted_init = True

stemmer = SnowballStemmer(language='english')
tokenizer = RegexpTokenizer(r'\w+')

summaries = []
k = 10

for s in samples:
    raw_sentences = write_sentences(s)
    sentences = []
    counter = Counter()

    for s in raw_sentences:
        words = tokenizer.tokenize(s)
        sentences.append(' '.join(words))
    sentences = list(set(sentences))

    vocab = make_vocab(sentences)
    idf = compute_idf(sentences, vocab)
    vectors = [tf_idf(s, vocab, idf) for s in sentences]

    scores = []

    for i in range(len(sentences)):
        for j in range(len(sentences)):
            if i != j:
                scores.append((i, j, similarity(vectors[i], vectors[j])))

    if prune:
        scores = prune_graph(scores)

    # tops of clusters obtained by PageRank
    top_k = summarize(scores, k = k, weighted_init = weighted_init)

    summaries.append([sentences[i] for i in top_k])_____no_output_____for i, s in enumerate(samples):
    print(s['object1']['name'] + ' vs ' + s['object2']['name'] + ':')
    for sent in summaries[i]:
        print(sent + '.')
    print()python vs java:
python than for java.
Aside from the perils of late binding the other major concern I have about Python for mission critical apps is that the Python designers didn t include a posse of security paranoids the way Java and JEE did so exploits are potentially more possible.
In practice I find that my Python programs perform somewhat better than Java probably because Python has many libraries implemented in C whereas Java has many pure Java libraries.
Using the decimal library in python with 90 significant digits is massive overkill but I plan to use those results as a baseline for comparison with both the java app and the python app.
most humorously the major reason python is touted over java is the reason you give for java over c python is even more high level than java meaning higher productivity.
BTW AFAIK the python language evolution is managed much in the same way that Java JSR for Java vs PEPs for Python JCP for Java vs PSF for Python.
Python is slower than Java.
Python is better than JAVA.
Python is no more interpreted than Java.
Java is faster while python is.

toyota vs nissan:
I chose the Toyota over the Nissan.
More than Toyota more than Nissan or any of the brands.
i will not purchase another toyota product because of the way s e toyota dealt with us over the alignment issue but s e toyota gets 500 from the buyer for each car nissan gm ford chrysler etc do not have such a fee.
And Toyota was smaller than Nissan and Honda.
Nissan is a good machine but not as good as a Toyota.
The context of your message would indicate that the engine would be more in sync in the TCH than the NAH e g Toyota with Toyota rather than Nissan with Toyota.
Nissan Altima Nissan sold an astonishing 41 050 Altimas during March just a handful less than the segment powerhouse Toyota Camry.
To explain why I liken Giant to Nissan rather than Toyota is because Toyota is revered in my opinion too much for quality while Nissan has a more benign reputation.
However since the Nissan CC appears to be considerably less expensive than the Toyota DC my vote goes to Nissan.
My limited experience tells me that overall Toyota is better than Nissan especially in terms of attention in details quietness.

tea vs juice:
We don t drink juice but we do drink tea or water.
POM Wonderful Facts POM Wonderful 100 Pomegranate Juice is the super juice that contains more antioxidants than red wine grape juice acai juice and even green tea.
What is so wonderful about this Pomegranate juice is its antioxidant potency where better than green tea cranberry juice red wine concord grape juice and blueberry juice.
According to POM their juice has more antioxidant power than other drinks including red wine blueberry juice green tea and cranberry juice.
In the summer its juice is offered instead of tea.
This tea is nothing more than hot leaf juice.
Pomegranate juice has more antioxidants than green tea.
Iroh This tea is nothing more than hot leaf juice.
Only problem is that this juice isn t as good as the green tea cartos.
Juice Frozen orange juice is cheaper but usually I just stick to tea and buy the GOOD orange juice for special occasions.

### Pretrained Language Model

Language Model inn general is a function that is fitted to maximize probabilty of the tokens sequences from some domain. In this approach we use a reccurent neural model that also learns hidden representation for every token in a given sequence that is aimed to store information about all previous words in the text. Then the next token probability is a function of a privious token representation and  a hidden state:
$$p(x_{t+1}) = f(h_t, x_t)$$

In this experiment we use LSTM model pretrained on the set retrieved by CAM for all 512 queries. As input representations we use pretrained embeddings from BERT model with original BERT tokenizer. As our domain is pretty wide, it is efficient to use BPE tokens instead of building word-level vocabulary.

We initialize model in two different ways: randomly and by the language model pretrained on WikiText103 dataset. WikiText103 is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia.

As a similarity function here we use cosine distance between hidden representations of the last tokens of sentences:

$$f(h^1_{t_1}, h^2_{t_2}) = \frac {\left<h^1_{t_1}, h^2_{t_2}\right>}{\|h^1_{t_1}\|\|h^2_{t_2}\|}$$,
where $h^i$ is a sequence of hidden states of sentence $i$ and $t_i$ is number of tokens in sentence $i$._____no_output_____

# we will need BERT Tokenizer
!pip install pytorch-pretrained-bertRequirement already satisfied: pytorch-pretrained-bert in /home/user/.local/lib/python3.6/site-packages (0.6.2)
Requirement already satisfied: regex in /opt/conda/lib/python3.6/site-packages (from pytorch-pretrained-bert) (2018.1.10)
Requirement already satisfied: boto3 in /opt/conda/lib/python3.6/site-packages (from pytorch-pretrained-bert) (1.9.83)
Requirement already satisfied: tqdm in /opt/conda/lib/python3.6/site-packages (from pytorch-pretrained-bert) (4.28.1)
Requirement already satisfied: torch>=0.4.1 in /home/user/.local/lib/python3.6/site-packages (from pytorch-pretrained-bert) (1.3.1)
Requirement already satisfied: requests in /opt/conda/lib/python3.6/site-packages (from pytorch-pretrained-bert) (2.19.1)
Requirement already satisfied: numpy in /home/user/.local/lib/python3.6/site-packages (from pytorch-pretrained-bert) (1.18.1)
Requirement already satisfied: botocore<1.13.0,>=1.12.83 in /opt/conda/lib/python3.6/site-packages (from boto3->pytorch-pretrained-bert) (1.12.83)
Requirement already satisfied: s3transfer<0.2.0,>=0.1.10 in /opt/conda/lib/python3.6/site-packages (from boto3->pytorch-pretrained-bert) (0.1.13)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.6/site-packages (from boto3->pytorch-pretrained-bert) (0.9.3)
Requirement already satisfied: idna<2.8,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests->pytorch-pretrained-bert) (2.7)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests->pytorch-pretrained-bert) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests->pytorch-pretrained-bert) (2018.11.29)
Requirement already satisfied: urllib3<1.24,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests->pytorch-pretrained-bert) (1.22)
Requirement already satisfied: docutils>=0.10 in /opt/conda/lib/python3.6/site-packages (from botocore<1.13.0,>=1.12.83->boto3->pytorch-pretrained-bert) (0.14)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in /opt/conda/lib/python3.6/site-packages (from botocore<1.13.0,>=1.12.83->boto3->pytorch-pretrained-bert) (2.7.5)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.6/site-packages (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore<1.13.0,>=1.12.83->boto3->pytorch-pretrained-bert) (1.11.0)
import numpy as np
import torch
import torch.nn.functional as F
import torch.nn as nn
from pytorch_pretrained_bert import *
import matplotlib.pyplot as plt
from IPython.display import clear_output_____no_output_____print (6)6
from tokenizer_custom_bert import BertTokenizer_____no_output_____tokenizer = BertTokenizer(vocab_file = 'vocab.txt')_____no_output_____

Here we just load pretrained model that was initially used for Encoder-Decoder architecture. Training code can be found in MeanSum part of the project._____no_output_____

data_____no_output_____data_____no_output_____class RNNModel(nn.Module):
    """Container module with an encoder, a recurrent module, and a decoder."""

    def __init__(self, ntoken, ninp = 768, nhid = 512, nlayers=1, dropout=0.5, tie_weights=False):
        super(RNNModel, self).__init__()
        self.drop = nn.Dropout(dropout)
        self.encoder = nn.Embedding(ntoken, ninp)
        self.rnn = nn.LSTM(ninp, nhid, nlayers, dropout=dropout)
        self.decoder = nn.Linear( nhid, ntoken)

        self.init_weights()

        self.nhid = nhid
        self.nlayers = nlayers

    def init_weights(self):
        initrange = 0.1
        self.decoder.bias.data.zero_()
        self.decoder.weight.data.uniform_(-initrange, initrange)

    def forward(self, input, hidden, input_lengths = None):
        emb = self.drop(self.encoder(input))
        if input_lengths is not None:
            emb = torch.nn.utils.rnn.pack_padded_sequence(emb, input_lengths, enforce_sorted = True)
            outputs, hidden = self.rnn(emb, hidden)
            output, _ = torch.nn.utils.rnn.pad_packed_sequence(outputs)
        else:
            output, hidden = self.rnn(emb, hidden)
        output = self.drop(output)
        decoded = self.decoder(output)
        return decoded, hidden

    def init_hidden(self, bsz):
        weight = next(self.parameters())
        return (weight.new_zeros(self.nlayers, bsz, self.nhid),
                weight.new_zeros(self.nlayers, bsz, self.nhid))

def detach_hidden(hidden):
    return (hidden[0].detach(),
            hidden[1].detach())_____no_output_____!ls -lhtotal 187M
-rw-r--r-- 1 user user1 380K Jan 29 20:01 TextRank.ipynb
drwxr-xr-x 2 user user1 4.0K Jan 29 19:59 __pycache__
-rw-r--r-- 1 user user1 7.3K Jan 29 19:59 file_utils_custom.py
-rw-r--r-- 1 user user1  20M Jan 29 18:21 mined_bow_str.json
-rw-r--r-- 1 user user1  16K Jan 29 19:59 tokenizer_custom_bert.py
-rw-r--r-- 1 user user1 227K Jan 29 20:04 vocab.txt
-rw-r--r-- 1 user user1 168M Jan 29 19:27 wikitext_lm_finetuned
device = torch.device("cuda")

# one-layered LSTM model pretraied on CAM outputs

#LM = RNNModel(30522, nlayers=1, dropout = 0.0)
#LM.load_state_dict(torch.load('drive/My Drive/summarization/simple_lm'))

# two-layered LSTM pretrained on WikiText103 and finetuned on CAM outputs

LM = RNNModel(30522, nlayers=2, dropout = 0.0)
LM.load_state_dict(torch.load('./wikitext_lm_finetuned'))

LM = LM.to(device)_____no_output_____def similarity(s1, s2):
    return s1.dot(s2)/np.linalg.norm(s1,2)/np.linalg.norm(s2, 2)_____no_output_____import requests

URL = 'http://ltdemos.informatik.uni-hamburg.de/cam-api'
proxies = {"http": "http://185.46.212.97:10015/","https": "https://185.46.212.98:10015/",}
params = {
            'objectA': 'Moscow',
            'objectB': 'London',
            'fs': str(True).lower()}
response = requests.get(url=URL, params=params, proxies=proxies)
s = response.json()_____no_output_____response_json = s
comparing_pair = {}

if (response_json['object1']['name'] == response_json['winner']):
    comparing_pair['winner_aspects'] = response_json['extractedAspectsObject1'][:4]
    comparing_pair['loser_aspects'] = response_json['extractedAspectsObject2'][:4]
    comparing_pair['winner'] = response_json['object1']['name']
    comparing_pair['loser'] = response_json['object2']['name']
else:
    comparing_pair['winner_aspects'] = response_json['extractedAspectsObject2'][:4]
    comparing_pair['loser_aspects'] = response_json['extractedAspectsObject1'][:4]
    comparing_pair['winner'] = response_json['object2']['name']
    comparing_pair['loser'] = response_json['object1']['name']_____no_output_____s = response.json()
#raw_sentences = write_sentences(s)_____no_output_____from template_generation import generate_template
generate_template(comparing_pair, mode="extended")_____no_output_____from cam_summarize import cam_summarizeWarning:  apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
Warning:  apex was installed without --cuda_ext. Fused syncbn kernels will be unavailable.  Python fallbacks will be used instead.
Warning:  apex was installed without --cuda_ext.  FusedAdam will be unavailable.
Warning:  apex was installed without --cuda_ext.  FusedLayerNorm will be unavailable.
cam_summarize(s)[0]_____no_output_____summaries = []
sentences = []
for s in raw_sentences:
    s = ["[CLS]"] + tokenizer.tokenize(s) + ["[SEP]"]
    sentences.append(tokenizer.convert_tokens_to_ids(s))

hiddens = []
for s in sentences:
    hidden = LM.init_hidden(1)
    batch = torch.LongTensor([s]).transpose(0, 1).to(device)
    preds, h = LM(batch, hidden, input_lengths = None)
    hiddens.append(h[1].view(-1).cpu().detach().numpy())

scores = []

for i in range(len(sentences)):
    for j in range(len(sentences)):
        scores.append((i, j, similarity(hiddens[i], hiddens[j])))

if prune:
    scores = prune_graph(scores)

top_k = summarize(scores, k = k, weighted_init = weighted_init)

summaries.append([raw_sentences[i] for i in top_k])_____no_output_____summaries_____no_output_____for sent in summaries:
    print(sent[:-1])
print()_____no_output_____#use for Chinese Whispers:
summarize = cw_summarize

#use for TextRank:
#summarize = extract_top

prune = True
weighted_init = True

summaries = []
k = 10

for s in samples:
    raw_sentences = write_sentences(s)
    sentences = []
    for s in raw_sentences:
        s = ["[CLS]"] + tokenizer.tokenize(s) + ["[SEP]"]
        sentences.append(tokenizer.convert_tokens_to_ids(s))

    hiddens = []
    for s in sentences:
        hidden = LM.init_hidden(1)
        batch = torch.LongTensor([s]).transpose(0, 1).to(device)
        preds, h = LM(batch, hidden, input_lengths = None)
        hiddens.append(h[1].view(-1).cpu().detach().numpy())

    scores = []

    for i in range(len(sentences)):
        for j in range(len(sentences)):
            scores.append((i, j, similarity(hiddens[i], hiddens[j])))

    if prune:
        scores = prune_graph(scores)

    top_k = summarize(scores, k = k, weighted_init = weighted_init)

    summaries.append([raw_sentences[i] for i in top_k])_____no_output_____for i, s in enumerate(samples):
    print(s['object1']['name'] + ' vs ' + s['object2']['name'] + ':')
    for sent in summaries[i]:
        print(sent[:-1])
    print()python vs java:
Python is easier to program in than Java.
Writing it in python is probably faster than java.
Learn Python instead of Java as your first language .
compliant with Java than Python 2.5.)
I'm writing this project in Java instead of Python.
Hell, Python runtime is much smaller than Java or .
I'm using the Python version of the API, so a Python solution would be preferred, but I know Java well enough that I could translate a Java solution into Python.

toyota vs nissan:
The Nissan provided better ride and handling than the Toyota.
I would pefer the toyota over the nissan anyday!
I know Toyota owners are smarter than Nissan owners.)
Needless to say Nissan do not even compare to the appointed Toyota dealership.

tea vs juice:
Pomegranate juice has more polyphenol antioxidants than red wine, green tea, cranberry juice and orange juice.
But I was in hurry to taste her love juice first rather than tea or lichi juice etc.
Variations: Top with cranberry juice instead of cola to make a Long Beach Iced Tea, orange juice instead of cola for a California Iced Tea, or grapefruit juice instead of cola for a Florida Iced Tea.
Afternoon Snack: The Juice That's Healthier Than Wine, Green Tea .


# Notebook from joaop9425/DataScience
Path: mentoria/Aula+Exercicio_01_Py/me-linguagem-python-exercicios-01.ipynb

# Mentoria Evolution - Exercícios Python_____no_output_____https://minerandodados.com.br_____no_output_____* Para executar uma célula digite **Control + enter** ou clique em **Run**.
* As celulas para rodar script Python devem ser do tipo code.
* Crie células abaixo das celulas que foram escrito o enunciado das questões com as respostas._____no_output_____**Obs**: Caso de dúvidas, volte na aula anterior de Python. Não desista :)_____no_output_____## Exercícios de Fixação_____no_output_____1) Imprima a frase **"Eu sou Minerador de Dados!"**_____no_output_____

print ("Eu sou Minerador de Dados")Eu sou Minerador de Dados

2) Faça operações matemáticas de soma, subtração, multiplicação e divisão._____no_output_____

soma = 3 + 2
subt = 3 - 2
mult = 3 * 2
divi = 3 / 2
print(soma, subt, mult, divi)5 1 6 1.5

3) Imprima o tipo de um determinado do valor **10**._____no_output_____

dez = 10
type(dez)_____no_output_____

4) Converta o valor passado anteriormente para o tipo **float**_____no_output_____

float(dez)_____no_output_____

5) Faça a conversão do valor float recebido anteriormente para o tipo **string**_____no_output_____

str(dez)_____no_output_____

6) Utilizando variáveis faça:
- a) Defina uma variável com o **seu nome** e imprima o seu valor.
- b) Imprima o **tipo** dessa variável.
- c) Substitua o valor da sua variável nome com a concatenação do seu valor e o seu sobrenome, depois, imprima o novo valor.
    - obs: O resultado dessa variável será Nome Sobrenome, exemplo, **Felipe Santana**.
- d) Defina duas variaveis com os valores do seu email e idade com apenas uma linha de comando, depois, imprima seus valores._____no_output_____

nome = "João Paulo"
print(nome)
type(nome)

nome += " Neves"
print(nome)
email, idade = "joao.p9425@gmail.com", 23
print(email, idade)João Paulo
João Paulo Neves
joao.p9425@gmail.com 23

7) Utilizando funções de arredonamento faça:
- a) Arredonde o valor 2.898
- b) Arredonde o valor 2.4545455 e exiba apenas 4 cadas decimais_____no_output_____

round(2.898)_____no_output_____round(2.4545455, 4)_____no_output_____

9) Sobre os objetos Strings, faça:
- a) Defina uma variavel com a string: "Python é uma maldição!" e imprima
- b) Imprima o primeiro caracter da variavel.
- c) Imprima os valores do primeiro caracter até o 6.
- d) Faça a contagem de todos os caracteres utilizando uma única linha de comando.
- e) Faça uma contagem de quantas vezes a palavra dados aparece na seguinte frase:
    - "dados é o novo petróleo, em Deus eu confio, para o resto me traga dados.."
    - Obs: Crie uma variável chamada "frase" com a frase acima.
- f) Substitua o primeiro caracter da frase acima para maiúsculo
- g) Quebre as palavras da frase separado por virgula, criando uma lista._____no_output_____

var = "Python é uma maldição!"
print(var)Python é uma maldição!
print(var[0])P
print(var[0:6])Python
len(var)_____no_output_____frase = "dados é o novo petróleo, em Deus eu confio, para o resto me traga dados.."
frase.count("dados")_____no_output_____frase.capitalize()_____no_output_____frase.split(',')_____no_output_____

- Ao concluir, salve seu notebook e envie suas respostas para **contato@minerandodados.com.br**_____no_output_____


# Notebook from zoeydy/CMEESamraat
Path: content/notebooks/Appendix-Maths.ipynb

# Mathematical models in Jupyter_____no_output_____## Introduction

Jupyter is a fully-functional alternative to Mathematica or Maple notebooks for developing and analyzing mathematical models in biology (or any other discipine, for that matter). For this, you will need to use a [Computer Algebra System](https://en.wikipedia.org/wiki/Computer_algebra_system) (CAS). A CAS is software that emulates manual (pen-and-paper) manipulations of mathematical expressions. Yes, it can be done, and very effectively, for a vast array of mathematical problems! A CAS combined with a graphing/plotting package like matplotlib gives you a powerful tool for mathematical modelling using a Jupyter notebook.

We will use Python's [SymPy](http://sympy.org/en/index.html) package, which provides powerful CAS features for most common mathematical modelling problems.

There is also [Sage](http://www.sagemath.org/), a more capable CAS. We will not use it here because is not a regular Python package, but rather, uses Python as its programming language. So unlike SymPy it cannot just be loaded in a Jupyter nb with a Python kernel. Instead, you will need to [install its Jupyter kernel](http://doc.sagemath.org/html/en/reference/repl/sage/repl/ipython_kernel/install.html). You can [install and try sage](http://www.sagemath.org/download.html) outside of Jupyter if you want.

So let's use SymPy in Jupyter._____no_output_____If you used [Anaconda](http://continuum.io/downloads) to install Jupyter, it should already include SymPy, Matplotlib, IPython, NumPy, and other useful packages for scientific computing. If you don't have SymPy for some other reason, install it in Linux/mac using:
```bash
$ sudo apt install python3-sympy
```
Otherwise, follow the instructions [here](http://docs.sympy.org/latest/install.html).

We also need to rum some commands so that the plots appear correctly:_____no_output_____

%matplotlib inline
import matplotlib.pyplot as p_____no_output_____

And import scipy and sympy:_____no_output_____

from sympy import *
import scipy as sc
init_printing() # for pretty-printing equations etc_____no_output_____

## Some preliminaries

Before we get started with our mathematical modelling session in Jupyter, some SymPy preliminaries.

### Symbolic variables_____no_output_____In CAS' like SymPy, we need to create symbolic variables for the mathematical variables we want to work with. A new symbolic variable can be created using `var`. Try this:_____no_output_____

x = var('x')
type(x) # check it's class_____no_output_____

You can also define multiple symbolic variables at one go:_____no_output_____

a, b, c = var("a, b, c")_____no_output_____

For more info on symbolic variables, [have a look at this](http://docs.sympy.org/latest/gotchas.html#variables)._____no_output_____It is often important to add assumptions (constraints) to our symbolic vars:_____no_output_____

x = var('x', real=True)_____no_output_____

Now check:_____no_output_____

x.is_imaginary_____no_output_____x = Symbol('x', positive=True)_____no_output_____

Again, check:_____no_output_____

x > 0_____no_output_____x < 0_____no_output_____

### Symbolic equations

We can define the mathematical equations (functions) that we will be using/manipulating as follows:  _____no_output_____

MyFun = (pi + x)**2; MyFun_____no_output_____MyFun = N_0 + (N_max - N_0) * exp(-exp(r_max * exp(1) * (t_lag - t)/((N_max - N_0) * log(10)) + 1))_____no_output_____

See the nice $\LaTeX$ - formatted output: this is where `init_printing()` comes handy.

SymPy has predefined expressions for a number of mathematical constants, such as: `pi` ($\pi$), `e` (exponential), `oo` (infinity).

You can also get your equation in latex syntax! Try:_____no_output_____

latex(MyFun)_____no_output_____

That has extra escape slashes for Python to be able to parse it correctly. To display it in its actual form (that you can directly use in a $\LaTeX$ document), `print` it: _____no_output_____

print(latex(MyFun))\left(x + \pi\right)^{2}

### Numerical evaluation_____no_output_____To evaluate an expression numerically we can use the `evalf` function (or `N`). It takes an argument `n` which specifies the number of significant digits._____no_output_____

pi.evalf(n=100) # pi to a 100 places after decimal!_____no_output_____

`N()` is shorthand alias for `evalf()`:_____no_output_____

N(pi, 50)_____no_output_____

So let's try evaluating our function:_____no_output_____

N(MyFun, 5)_____no_output_____

When we numerically evaluate algebraic expressions we often want to substitute a symbol with a numerical value. In SymPy we do that using the `subs` function:_____no_output_____

MyFun.subs(x, 1.5)_____no_output_____

Now let's evaluate it:_____no_output_____

MyFun.subs(x, 1.5)_____no_output_____

The `subs` function can also be used to substitute mathematical variables or expressions. Let's substitute $x$ with $a+\pi$:_____no_output_____

MyFun.subs(x, a+pi)_____no_output_____

And assign it as a new symbolic equation for using later:_____no_output_____

MyFun_new = MyFun.subs(x, a+pi); MyFun_new_____no_output_____

We can also numerically evaluate the function over a range of values using NumPy arrays:_____no_output_____

x_vec = sc.arange(0, 10, 0.1)_____no_output_____MyFun_vec = sc.array([N(MyFun.subs(x, xx)) for xx in x_vec]) #Note: using a list comprehension!_____no_output_____

We can also evaluate the new function `MyFun_new` we created by substitution above:_____no_output_____

MyFun_new_vec = sc.array([N((MyFun_new).subs(a, xx)) for xx in x_vec])_____no_output_____

Now plot the two functions that you evaluated (try adding axes and a legend to these basic plots). _____no_output_____

fig, ax = p.subplots()
ax.plot(x_vec, MyFun_vec)
ax.plot(x_vec, MyFun_new_vec)_____no_output_____

However, numerical evaluation using `evalf()` can be very slow. There is a much more efficient way to do it by using [`lambdify()`](http://docs.sympy.org/latest/modules/utilities/lambdify.html) to "compile" a Sympy expression into a function that is much more efficient to evaluate numerically:_____no_output_____

MyFun_lamb = lambdify([x], MyFun, 'numpy')_____no_output_____

The first argument is a (python) list of variables that `MyFun_lamb` will be a function of. In this case its only $x$. Now we can directly pass a numpy array and MyFun is evaluated more efficiently:_____no_output_____

MyFun_vec = MyFun_lamb(x_vec)_____no_output_____

The speedup when using "lambdified" functions instead of direct numerical evaluation can be significant, often several orders of magnitude. Even in this simple example we get a significant speed up:_____no_output_____

%%timeit #remember this?

MyFun_vec = sc.array([N(((x + pi)**2).subs(x, xx)) for xx in x_vec])27.7 ms ± 7.48 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
%%timeit

MyFun_vec = MyFun_lamb(x_vec)2.64 µs ± 684 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

## Fundamental mathematical operations _____no_output_____Let's look at some fundamental methematical operations in SymPy that you will almost certainly use at some point in biological models. You can find a full list and addtional tutorials and examples at [SymPy documentation](http://docs.sympy.org/latest/index.html).  _____no_output_____### Expand and factor_____no_output_____

expand(MyFun)_____no_output_____

The `expand` function takes a number of keywords arguments which we can tell the functions what kind of expansions to want to perform; use `help(expand)` (or the [SymPy documentation](http://docs.sympy.org/latest/index.html)) for more info._____no_output_____You can also print the result of an manipulation in its raw python form:  _____no_output_____

print(expand(MyFun))x**2 + 2*pi*x + pi**2

You can also factor using, well, `factor()`:_____no_output_____

factor(x**2 + 2*pi*x + pi**2)_____no_output_____

### Apart and together_____no_output_____To manipulate symbolic expressions of fractions, you can use the `apart` and `together` functions:_____no_output_____

f1 = 1/((a+1)*(a+2)); f1_____no_output_____apart(f1)_____no_output_____f2 = 1/(a+2) + 1/(a+3); f2_____no_output_____together(f2)_____no_output_____

### Simplification

The `simplify` tries to simplify an expression into a nice looking expression, using various techniques. More specific alternatives to the `simplify` functions also exist: `trigsimp`, `powsimp`, `logcombine`, etc. Applying `simplify` to the above example will give the same result as `together`: _____no_output_____

simplify(f2)_____no_output_____

Note that simplify usually combines fractions but does not factor. _____no_output_____In addition to algebraic manipulations, the other main use of CAS is to do calculus, like derivatives and integrals of algebraic expressions._____no_output_____### Differentiation_____no_output_____Differentiation is usually simple. Use the `diff` function. The first argument is the expression to take the derivative of, and the second argument is the symbol by which to take the derivative:_____no_output_____

diff(MyFun_new, a)_____no_output_____

For higher order derivatives we can do:_____no_output_____

diff(MyFun_new, a, a)_____no_output_____diff(MyFun_new**2, a, 2) # same as above_____no_output_____

You can directly apply another manipulation to the result of a previous operation:_____no_output_____

expand(diff(MyFun_new**2, a, 2))_____no_output_____

Calculate the derivative of a multivariate expression:_____no_output_____

x, y, z = var("x,y,z")_____no_output_____f = sin(x*y) + cos(y*z)_____no_output_____

$\frac{d^3f}{dxdy^2}$_____no_output_____

diff(f, x, 1, y, 2)_____no_output_____

### Integration_____no_output_____Integration is done in a similar fashion:_____no_output_____

MyFun_____no_output_____integrate(MyFun, x)_____no_output_____

By providing limits for the integration variable we can evaluate definite integrals:_____no_output_____

integrate(MyFun, (x, -1, 1))_____no_output_____

and also improper integrals_____no_output_____

integrate(exp(-x**2), (x, -oo, oo))_____no_output_____

Remember, `oo` is the SymPy notation for inifinity._____no_output_____### Sums and products_____no_output_____You can evaluate sums and products using `Sum`. Note that this function is named `Sum` and not `sum` to avoid namespace conflict._____no_output_____

n = var("n")_____no_output_____Sum(1/n**2, (n, 1, 10))_____no_output_____Sum(1/n**2, (n,1, 10)).evalf()_____no_output_____Sum(1/n**2, (n, 1, oo)).evalf()_____no_output_____

Products work much the same way:_____no_output_____

Product(n, (n, 1, 10)) # 10!_____no_output_____

### Limits_____no_output_____Limits can be evaluated using the `limit` function. For example, _____no_output_____

limit(sin(x)/x, x, 0)_____no_output_____

We can use 'limit' to check the result of derivation using the `diff` function:_____no_output_____

f_____no_output_____diff(f, x)_____no_output_____

$\displaystyle \frac{\mathrm{d}f(x,y)}{\mathrm{d}x} = \frac{f(x+h,y)-f(x,y)}{h}$_____no_output_____

h = var("h")_____no_output_____limit((f.subs(x, x+h) - f)/h, h, 0)_____no_output_____

OK!_____no_output_____We can change the direction from which we approach the limiting point using the `dir` keywork argument:_____no_output_____

limit(1/x, x, 0, dir="+")_____no_output_____limit(1/x, x, 0, dir="-")_____no_output_____

### Series_____no_output_____Series expansion is also one of the most useful features of a CAS. In SymPy we can perform a series expansion of an expression using the `series` function:_____no_output_____

series(exp(x), x) # this is a classic!_____no_output_____

By default it expands the expression around $x=0$, but we can expand around any value of $x$ by explicitly include a value in the function call:_____no_output_____

series(exp(x), x, 1)_____no_output_____

Or try:_____no_output_____

series(log(x), x, 0) # will not work why?_____no_output_____series(log(x), x,1) # this will work, however _____no_output_____

And we can explicitly define to which order the series expansion should be carried out:_____no_output_____

series(exp(x), x, 0, 3)_____no_output_____

Another way to do the same:_____no_output_____

exp(x).series(x,0,3)_____no_output_____

The series expansion includes the order of the approximation, which is very useful for keeping track of the order of validity when we do calculations with series expansions of different orders:_____no_output_____

s1 = cos(x).series(x, 0, 5); s1_____no_output_____s2 = sin(x).series(x, 0, 2); s2_____no_output_____expand(s1 * s2)_____no_output_____

If we want to get rid of the order information we can use the `removeO` method:_____no_output_____

expand(s1.removeO() * s2.removeO())_____no_output_____

### Matrix algebra_____no_output_____Matrices are defined using the `Matrix` class:_____no_output_____

m11, m12, m21, m22 = var("m11, m12, m21, m22")
b1, b2 = var("b1, b2")_____no_output_____A = Matrix([[m11, m12],[m21, m22]]) # Again, note: capital M for to avoid namespace conflict
A_____no_output_____b = Matrix([[b1], [b2]]); b_____no_output_____

With `Matrix` class instances we can do the usual matrix algebra operations:_____no_output_____

A**2_____no_output_____A * b_____no_output_____

And calculate determinants and inverses, and the like:_____no_output_____

A.det()_____no_output_____A.inv()_____no_output_____

### Solving equations_____no_output_____For solving equations and systems of equations we can use the `solve` function:_____no_output_____

solve(x**2 - 1, x)_____no_output_____solve(x**4 - x**2 - 1, x)_____no_output_____

System of equations:_____no_output_____

solve([x + y - 1, x - y - 1], [x,y])_____no_output_____

In terms of other symbolic expressions:_____no_output_____

solve([x + y - a, x - y - c], [x,y])_____no_output_____

You can also solve a single, or a system of ordinary differential equations (ODEs) using
[dsolve](http://docs.sympy.org/latest/modules/solvers/ode.html). WE will use this in a couple of the biological examples below.  _____no_output_____## Some biological examples_____no_output_____Here are some examples of development and analysis of some fundamental mathematical models in biology._____no_output_____### One population: Exponential growth

Let's look at how populations grow when there are no environmental constraints. The differential equation model is:

\begin{equation}\label{eq:exp_growth}
	\frac{\text{d}N}{\text{d}t} = r_m N
\end{equation}

where $r_m$ is the intrinsic, constant rate of population gowth (units of 1/time), and $N$ is population size (or biomass abundance). I use the subscript $m$ in $r_m$ to denote both [Malthusian](https://en.wikipedia.org/wiki/Malthusianism) and maximal population growth rate because, in theory, without any constraints, this growth rate is expected to at its theoretical/biological maximum.

Let's solve equation \ref{eq:exp_growth} so that we can calculate population size $N_t$ at any given time $t$ given a starting population size  $N_0$ at time 0.

First assign the symbolic variables:_____no_output_____

r_max, N_0, K, t_lag, t  = var("r_max N_0 K t_lag t",real = True) # the real bit is not really necessary here_____no_output_____N_0 + (K - N_0) * exp(-exp(r_max * exp(1) * (t_lag - t)/((K - N_0) * log(10)) + 1))_____no_output_____r_m, N, t = var("r_m N t",real = True) # the real bit is not really necessary here_____no_output_____

Now tell SymPy that $N$ is a function:_____no_output_____

N = Function('N')_____no_output_____

Define $N$ is a derivative of $t$_____no_output_____

dN_dt = Derivative(N(t), t) - r_m*N(t); dN_dt_____no_output_____

Note that we have simply re-written the condition that LHS = RHS in eqn \ref{eq:exp_growth}. Now that we have the differential equation set up, we can solve it using `dsolve`. Since this is a simple ODE, Sympy can do it on its own with no hints or guesses (unlike more complex ODEs; [see the documentation](http://docs.sympy.org/latest/modules/solvers/ode.html)):_____no_output_____

MyEq_sol = dsolve(dN_dt); MyEq_sol_____no_output_____

If you remember your high-school calculus, you might recall that $C_1$ here is an arbitrary constant. We now need to re-express it in terms of the initial conditions. We can do so by setting $t = 0$, and then setting that equal to $N(0)$ at time 0:

$$C_1 = N_0$$

That is,

\begin{equation}\label{eq:exp_growth_sol}
N{\left (t \right )} = N_0 e^{r_m t}
\end{equation}

We could use Sympy for this last step as well (using `subs()` like you learned above), but that would be just plain silly -- like using a sledge-hammer to drive in a nail!

We can now have a go at plotting the model (eqn. \ref{eq:exp_growth}), and also the solution eqn. \ref{eq:exp_growth_sol}. First, let's get an approximate solution by using numerical integration (which you learnt in the Advanced Python chapter):_____no_output_____

from scipy  import integrate

# parameters
r_m = 1.

# initial conditions
N_0 = 0.1

# The time vector
t_vec = sc.arange(0, 10., 0.01)

def exp_pop(N, t, r_m):
    """The right-hand side of the exponential growth ODE"""
    return r_m*N

N_vec = integrate.odeint(exp_pop, N_0, t_vec, args=(r_m,)) # the comma is needed!

# plot the numerical solution
p.plot(t_vec, N_vec)
p.xlabel('Time') ; p.ylabel('$N$')

# plot analytical solution
p.plot(t_vec, N_0 * sc.exp(r_m * t_vec),'k--')
p.legend(['numerical approximation', 'analytical solution'], loc='best') # draw legend_____no_output_____

These look practically identical. But they are not:_____no_output_____

N_vec - N_0 * sc.exp(r_m * t_vec)_____no_output_____

(Logistic-Population-Growth)=
### One population: Logistic Population growth

Populations eventually run into contraints, even if they can grow exponentially at the start (eqn \ref{eq:exp_growth}). The classical model for logistic growth in population density ($N$) captures this dynamic:

$$
	\frac{\text{d}N}{\text{d}t} = r_m N \left(1-\frac{N}{K}\right)
$$(eq:logist_growth)

where $K$ is the carrying capacity of the environment, while $r_m$ is the same parameter same as above. Let's solve this one as well.

As in the case of the exponential growth above, let's find the solution to this equation for any arbitrary time point $t$.

Again, first we define the vars and the function:_____no_output_____

r_m, K, N, t = var("r_m K N t",real = True) # the real bit is not really necessary here

N = Function('N')_____no_output_____dN_dt = Derivative(N(t), t) - r_m * N(t) * (1 - N(t) / K); dN_dt_____no_output_____

Again, as in the exponential growth example, we have simply re-written the condition that LHS = RHS in eqn \ref{eq:logist_growth}. Now we can solve the ODE:_____no_output_____

MyEq_sol = dsolve(dN_dt); MyEq_sol_____no_output_____

This is a bit more complicated than the solution for exponential growth above. But we can solve it the same way. First substitute $t = 0$, and then solve the resulting equation for the initital condition of $N_0$, which then gives This the time-dependent solution:
\begin{equation}
    N_t = \frac{N_0 K\mathrm{e}^{r_m t}}{K + N_0(\mathrm{e}^{r_m t}-1)}
\end{equation}

You can do the last steps to obtain this solution using Sympy as well (I leave it to you to try it)._____no_output_____No let's again compare the analytical solution against the numerical one:_____no_output_____

from scipy  import integrate

# parameters
r_m = 1.
K = 10.
# initial condition
N_0 = 0.1

#The time vector
t_vec = sc.arange(0, 10., 0.01)

def log_pop(N, t, r_m, K):
    """The right-hand side of the logistic ODE"""
    return r_m*N*(1-N/K)

N_vec = integrate.odeint(log_pop, N_0, t_vec, args=(r_m, K));

p.plot(t_vec, N_vec) # plot the solution
p.xlabel('Time') ; p.ylabel('$N$')

# plot analytical solution
p.plot(t_vec, K * N_0 * sc.exp(r_m * t_vec)/(K + N_0 * (sc.exp(r_m * t_vec) - 1.)),'k--')
p.legend(['numerical approximation', 'analytical solution'], loc='best') # draw legend_____no_output_____

### Two interacting populations: The Lotka-Volterra predator-prey model

Now for the classical Lotka-Volterra model that you encountered in the advanced Python week (without logistic growth for the consumer, $C$).

\begin{align}
\frac{dN}{dt} &= r_m N \left(1-\frac{N}{K}\right) - a N C\\
\frac{dC}{dt} &= e a N C - z C
\end{align}

here $r_m$ and $K$ is the Resource's growth rate and carrying capacity respectively as in the Logistic equation, $a$ is the consumer's search rate for the resource, $e$ is consumer's biomass conversion efficiency, and $z$ is it's mortality rate.

To solve this system of ODEs, we will take a different approach -- We will solve for the equilibrum (steady state for the two species' populations). Again, we start by define the vars:_____no_output_____

r_m, a, e, z, K, N, C, t = var("r_m, a, e, z, K, N, C, t",real = True)_____no_output_____

Now define the sysyem of ODEs for Sympy:_____no_output_____

dN_dt = r_m * N *(1-N/K) - a * N * C
dC_dt = e * a * N * C - z * C

dC_dt, dN_dt_____no_output_____

Now define the equilibrium state:_____no_output_____

N_eqlb = Eq(dN_dt, 0)
C_eqlb = Eq(dC_dt, 0)
N_eqlb, C_eqlb_____no_output_____

Solve it:_____no_output_____

N_eqlb_sol = solve(N_eqlb, C)
C_eqlb_sol = solve(C_eqlb, N)

N_eqlb_sol, C_eqlb_sol_____no_output_____

So there is one equilibrium  solution where both species maintain their non-zero populations. That was easy! Now you don't need to guess what parameter values will give you coexistence (both populations remain > 0). Just substitute parameter combinations that satisfy the conditions that

$$ \frac{r_{m} \left(K - N\right)}{K a} > 0, \textrm{ and } \frac{z}{a e} > 0$$

As an exercise try plotting this exact solution for the steady state along with the one you would obtain (asymptotically) using numerical integration of the system of ODEs. Below is code that does the numerical integration (essentially, same as the `LV.py` script).  _____no_output_____

from scipy import integrate

t_vec = sc.arange(0, 100., 0.01)

# parameters
r_m = 1.
a = 1
e = 0.5
z = .5
K =10

# initial condition: this is an array now!
N0C0 = sc.array([1., 1.])

# the function still receives only `x`, but it will be an array, not a number
def LV(NC, t, r_m, K, a, e, z):
    # Unlike the esponental and logistic growth model, we now need to convert
    # the output to a numpy array as it has two populations.
    return sc.array([ r_m * NC[0]*(1-NC[0]/K) - a * NC[0] * NC[1],
                   e * a * NC[0] * NC[1] - z * NC[1] ])

NC_vec = integrate.odeint(LV, N0C0, t_vec, (r_m, K, a, e, z))_____no_output_____

Check NC_vec's dimensions:_____no_output_____

print(NC_vec.shape)(10000, 2)

Now let's plot the solution. But first, just for fun, let's change the plot style:_____no_output_____

print(p.style.available)['bmh', 'seaborn-darkgrid', 'fast', 'seaborn-dark', 'seaborn-notebook', 'classic', 'seaborn', 'ggplot', 'seaborn-white', 'seaborn-pastel', 'seaborn-deep', 'seaborn-dark-palette', 'dark_background', 'seaborn-paper', 'grayscale', 'seaborn-ticks', '_classic_test', 'seaborn-whitegrid', 'Solarize_Light2', 'seaborn-talk', 'fivethirtyeight', 'seaborn-poster', 'seaborn-muted', 'seaborn-colorblind', 'seaborn-bright']
p.style.use('seaborn-darkgrid')_____no_output_____p.plot(t_vec, NC_vec)
p.xlabel('Time'); p.ylabel('Population size') # and of y-axis
p.legend(['Resource ($N$)', 'Consumer ($C$)'], loc='best')_____no_output_____

An useful thing to do here is take a look at the *phase space*, that is, plot only the dependent variables, without respect to time:_____no_output_____

p.plot(NC_vec[0,0], NC_vec[0,1], 'o')
print('Initial condition:', NC_vec[0])

p.plot(NC_vec[:,0], NC_vec[:,1])

#Another solution with a different initial condition:
#NC_vec2 = odeint(LV, [2., 4.], t_vec, (r_m, K, a, e, z))
#p.plot(NC_vec2[:,0], NC_vec2[:,1])
#p.plot(NC_vec2[0,0], NC_vec2[0,1], 'o')
#p.xlabel('Resource Population size'); p.ylabel('Consumer Population size') # and of y-axisInitial condition: [ 1.  1.]

## Readings and Resources_____no_output_____* [The SymPy documentation](http://sympy.org/en/index.html)
* [SymPy online](http://live.sympy.org)_____no_output_____