# Notebook from cemozen/pattern_formation_in_reaction-diffusion_systems
Path: .ipynb_checkpoints/brusselator_hopf-checkpoint.ipynb

# Hopf Bifurcation: The Emergence of Limit-cycle Dynamics

*Cem Özen*, May 2017._____no_output_____A *Hopf bifurcation* is a critical point in which a periodic orbit appears or disappears through a local change in the stability of a fixed point in a dynamical system as one of the system parameters is varied. Hopf bifurcations occur in many of the well-known dynamical systems such as the Lotka-Volterra model, the Lorenz model, the Selkov model of glycolysis, the Belousov-Zhabotinsky reaction model, and the Hodgkin-Huxley model for nerve membrane.

In this notebook, I will consider a system of chemical reactions known by the name *Brusselator* in literature (see: https://en.wikipedia.org/wiki/Brusselator for more information) as a model for Hopf bifurcations. The Brusselator reactions are given by

$A \longrightarrow X$

$2X + Y\longrightarrow 3X$

$B + X \longrightarrow Y + D$

$X \longrightarrow E$

For the sake of simplicity, we will assume that the reaction constants of all these reactions are unity (i.e. in all the reactions, $k=1$ ). Furthermore let's assume that the reactant concentrations $A$ and $B$ are so large that  they remain constant. Therefore, only $X$ and $Y$ concentrations will be dynamical.

The rate equations for $X$ and $Y$ are then given by

$$
\begin{eqnarray}
\dot{X} & = & A + X^2Y - BX - X, \\
\dot{Y} & = & BX - X^2Y
\end{eqnarray}
$$_____no_output_____The X-nullcline and the Y-nullcline are given by the conditions of  $0 =  A + X^2Y - BX - X$ and $0 =  BX - X^2Y$ respectively. From these equations, we obtain:

$$
\begin{eqnarray}
Y(X) & = & \frac{-A + X(B+1)}{X^2},  & \quad \textrm{(X-nullcline)}  \\
Y(X) & = & \frac{B}{X}, & \quad   \textrm{(Y-nullcline)}
\end{eqnarray}
$$_____no_output_____In this notebook, I will also demonstrate how one can perform symbolical computations using Python's `SymPy` library. We also need extra Jupyter Notebook functionality to render nice display of the resulting equations. (Notice that we are using LaTex in typesetting this document particularly for the purpose of producing nice looking equations). _____no_output_____

import numpy as np
from numpy.linalg import eig
from scipy import integrate
import sympy
from IPython.display import display, Math, Latex
import matplotlib.pyplot as plt
sympy.init_printing(use_latex='mathjax')
%matplotlib inline_____no_output_____

Let's obtain the nullcline equations using `SymPy`: _____no_output_____

X, Y, A, B = sympy.symbols('X Y A B')   # you need to introduce the sysmbols first

# let's get the X-nullcline as a function of X:
sympy.solve(sympy.Eq(A + X**2 * Y - B * X - X, 0), Y)_____no_output_____# let's get the Y-nullcline as a function of X:
sympy.solve(sympy.Eq(B * X - X**2 * Y, 0), Y)_____no_output_____

Now let's find the fixed points ($X^*, Y^*$) of this 2-D system (there is only one, actually). The fixed point is given by the simultaneous solution of the X-nullcline and Y-nullcline equations, therefore

$$ (X^*, Y^*)  =  (A, \frac{B}{A}) $$

For the sake of using `SymPy`, let's obtain this solution once again:_____no_output_____

# Solve the system of equations defined by the X-nullcline and Y-nullcline with respect to X and Y:
sympy.solve([A + X**2 * Y - B * X - X,  B * X - X**2 * Y], [X, Y])_____no_output_____

Now, a bifurcation analysis of the Brusselator model requires us to keep track of the local stability of its fixed point. This can be done, according to the *linearized stability analysis* by evaluating the Jacobian  matrix at the fixed point.

The Jacobian matrix at the fixed point is given by:

$$
\begin{eqnarray}
J & = &  \left\vert\matrix{{\partial f \over \partial x} & {\partial f\over \partial y}  \cr
                           {\partial g \over \partial x} & {\partial g\over \partial y}
                           }\right\vert_{(X^*, Y^*)} \\
  & = & \left\vert\matrix{ -B + 2XY - 1 & X^2  \cr
                            B - 2XY & -X^2
                          }\right\vert_{(X^*, Y^*)}  \\
  & = & \left\vert\matrix{ B - 1 & A^2  \cr
                           -B & -A^2
                          }\right\vert
\end{eqnarray}
$$

This result can also be obtained very easily using `SymPy`:_____no_output_____

# define the Brusselator dynamical system as a SymPy matrix
brusselator = sympy.Matrix([A + X**2 * Y - B * X - X, B * X - X**2 * Y])_____no_output_____# Jacobian matrix with respect to X and Y
J = brusselator.jacobian([X, Y])
J_____no_output_____# Jacobian matrix evaluated at the coordinates of the fixed point
J_at_fp = J.subs({X:A, Y:B/A})   # substitute X with A and Y with B/A
J_at_fp_____no_output_____

A limit-cycle can emerge in a 2-dimensional, attractive dynamical system if the fixed point of the system  goes unstable. In this case, trajectories must be pulled by a limit cycle. (According to the Poincare-Bendixson theorem, a 2-dimensional system cannot have strange attractors). In this case, the Hopf bifurcation is called a *supercritical Hopf bifurcation*, because limit cycle is stable.

In the following, we will see how the stable fixed point (spiral) of the Brusselator goes unstable, giving rise to a limit cycle in turn. Conditions for the stability are determined by the trace and the determinant of the Jacobian. So let's evaluate them:_____no_output_____

Delta = J_at_fp.det()  # determinant of the Jacobian
Delta.simplify()_____no_output_____tau = J_at_fp.trace() # trace of the Jacobian
tau_____no_output_____

To have an unstable spiral we need:

$$
\begin{eqnarray}
\tau & > & 0  \quad \Rightarrow \quad & B > A^2 + 1  \quad  \textrm{required} \\
\Delta & > & 0  \quad {} \quad & \textrm{automatically satisfied} \\
\tau^2 & < & 4 \Delta \quad {} \quad & \textrm{automatically satisfied}
\end{eqnarray}
$$

The second and third conditions were satisfied because of the first condition, automatically. Therefore we need to have:

$$ B > A^2 + 1 $$ for limit cycles._____no_output_____## Birth of A Limit Cycle: Hopf Bifurcation
### Numerical Simulation of the Brusselator System
In the following, I perform a numerical simulation of the (supercritical) Hopf bifurcation in the Brusselator system by varying the parameter $B$ while the value of $A=1$. _____no_output_____

# Brusselator System:
def dX_dt(A, B, X, t):
    x, y = X[0], X[1]
    return np.array([A + x**2 * y - B * x -x,
                     B * x - x**2 * y])

T = 50 * np.pi                      # simulation time
dt = 0.01                           # integration time step

# time steps to be used in integration of the Brusselator system
t=np.arange(0, T, dt)

# create a canvas and 3 subplots..we will use each one for different choice of A and B paraeters
fig, ax = plt.subplots(1, 3)
fig.set_size_inches(13,5)

def plotter(A, B, ax):
    """
    This function draws a phase portrait by assigning a vector characterizing how the concentrations
    change at a given value of X and Y. It also draws a couple of example trajectories.
    """
    # Drow direction fields using matplotlib 's quiver function, similar to what we did
    # in class but qualitatively
    xmin, xmax  = 0,  5                           # min and max values of x axis in the plot
    ymin, ymax  = 0,  5                           # min and max values of y axis in the plot
    x = np.linspace(xmin, xmax, 10)               # divide x axis to intervals
    y = np.linspace(ymin, ymax, 10)               # divide y axis to intervals
    X1 , Y1  = np.meshgrid(x, y)            # from these intervals create a grid
    DX1, DY1 = dX_dt(A, B, [X1, Y1], t)     # compute rate of change of the concentrations on grid points
    M = (np.hypot(DX1, DY1))                # norm of the rate of changes
    M[ M == 0] = 1.                         # prevention against divisions by zero
    DX1 /= M                                # we normalize the direction field vector (each has unit length now)
    DY1 /= M                                # we normalize the direction field vector (each has unit length now)
    Q = ax.quiver(X1, Y1, DX1, DY1, M, pivot='mid', cmap=plt.cm.jet)

    num_traj = 5  # number of trajectories

    # choose several initial points (x_i, y_j),  for i and j chosen as in linspace calls below
    X0  = np.asarray(list(zip(np.linspace(xmin, xmax, num_traj), np.linspace(ymin, ymax, num_traj))))
    vcolors = plt.cm.jet_r(np.linspace(0., 1., num_traj))  # colors for each trajectory

    # integrate the Brusellator ODE's using all initial points to produce corresponding trajectories
    X = np.asarray([integrate.odeint(lambda x, t: dX_dt(A, B, x, t), X0i,t)
                          for X0i in X0])
    # plot the trajectories we obtained above
    for i in range(num_traj):
        x, y = X[i, :, :].T          # x and y histories for trajectory i
        ax.plot(x, y, '-', c=vcolors[i], lw=2)
    # set limits, put labels etc..
    ax.set_xlim(xmin=xmin, xmax=xmax)
    ax.set_ylim(ymin=ymin, ymax=ymax)
    ax.set_xlabel("X", fontsize = 20)
    ax.set_ylabel("Y", fontsize = 20)
    ax.annotate("A={}, B={}".format(A, B), xy = (0.4, 0.9), xycoords = 'axes fraction', fontsize = 20, color = "k")

# Now let's prepare plots for the following choices of A and B:
plotter(A=1, B=1, ax=ax[0])
plotter(A=1, B=2, ax=ax[1])
plotter(A=1, B=3, ax=ax[2])_____no_output_____

Above you see how a limit cycle can be created in a dynamical system, as one of the system parameters is varied.
Here we have kept $A=1$ but varied $B$ from 1 to 2.5. Note that $B=2$ is the borderline value, marking the change in the stability of the fixed point. For $B<1$ the fixed point is stable but as we cross the value $B=2$, it changes its character to unstable and a limit cycle is born. This phenomenon is an example of *Hopf Bifurcation*.

On the leftmost panel we have a stable spiral. Technically, this means that the Jacobian at the fixed point has two complex eigenvalues (a complex conjugate pair). The fact that the eigenvalues are complex is responsible for the spiralling effect. In stable spirals, the real part of the eigenvalues are negative, which is why these spiralling solutions decay, that is, trajectories nearby fall on to the fixed point. As the bifurcation parameter (here $B$) varies, the real part of the complex eigenvalues increase, reach zero at certain value of $B$, and keep growing now on the positive side. If the real part of the eigenvalues are positive, the fixed point is  an unstable spiral; trajectories nearby are pushed out of the fixed point (see the rightmost panel and plot below). Since this 2-D dynamical system is attractive, by the Poincare-Bendixson theorem, the emergence of the unstable spiral accompanies the birth of a limit-cycle. Notice that the panel in the middle is the borderline case between the stable and unstable spirals: in this case the real part of the eigenvalues is exactly zero (see plots below); the linear stabilization theory falsely predicts a neutral oscillation (i.e a center) at $B=2$---due to purely imaginary eigenvalues. However, the fixed point is still a stable spiral then._____no_output_____### Eigenvalues of the Jacobian _____no_output_____

# Eigenvalues of the Jacobian at A=1, B=1  (fixed point is stable spiral)
J_numeric = np.asarray(J_at_fp.evalf(subs={A:1, B:1})).astype(np.float64)
w, _ = eig(J_numeric)
w_____no_output_____# Eigenvalues of the Jacobian at A=1, B=3  (fixed point is unstable spiral)
J_numeric = np.asarray(J_at_fp.evalf(subs={A:1, B:3})).astype(np.float64)
w, _ = eig(J_numeric)
w_____no_output_____

Let's prepare plots showing how the real and imaginary parts of the eigenvalues change as $B$ is varied._____no_output_____

from numpy.linalg import eig

a = 1
eigen_real, eigen_imag = [], []
B_vals = np.linspace(1, 3, 20)
for b in B_vals:
    J_numeric = np.asarray(J_at_fp.evalf(subs={A:a, B:b})).astype(np.float64)
    w, _ = eig(J_numeric)
    eigen_real.append(w[0].real)
    eigen_imag.append(abs(w[0].imag))
eigen_real = np.asanyarray(eigen_real)
eigen_imag = np.asarray(eigen_imag)_____no_output_____fig, ax = plt.subplots(1, 2)
fig.set_size_inches(10,5)
fig.subplots_adjust(wspace=0.5)
ax[0].axhline(y=0, c="k", ls="dashed")
ax[0].plot(B_vals, eigen_real)
ax[0].set_ylabel(r"$\mathfrak{Re}(\lambda)$", fontsize = 20)
ax[0].set_xlabel(r"$B$", fontsize = 20)
ax[1].set_ylabel(r"$|\mathfrak{Im}(\lambda)|$", fontsize = 20)
ax[1].set_xlabel(r"$B$", fontsize = 20)
ax[1].plot(B_vals, eigen_imag)_____no_output_____

Hopf bifurcation, is only one type of bifurcation, albeit a very important one for physical and biological systems. There are other types of bifurcation in which one can create or destroy fixed points or alter their properties in different ways than a Hopf bifurcations does. If you are curious, I suggest you to perform your own numerical experiements by playing with the values of $A$, $B$
or both._____no_output_____### An Animation of the Hopf Bifurcation _____no_output_____

from matplotlib import animation, rc
from IPython.display import HTML

# Brusselator System:
def dX_dt(A, B, X, t):
    x, y = X[0], X[1]
    return np.array([A + x**2 * y - B * x -x, B * x - x**2 * y])

T = 50 * np.pi                      # simulation time
dt = 0.01                           # integration time step

# time steps to be used in integration of the Brusselator system
t=np.arange(0, T, dt)

num_traj = 5                          # number of trajectories
xmin, xmax  = 0,  5                   # min and max values of x axis in the plot
ymin, ymax  = 0,  5                   # min and max values of y axis in the plot
A = 1.                                # we will keep A parameter constant

# vary B parameter
Bmin, Bmax, numB = 1., 3., 100        # min, max, number of steps for varying B
Bvals = np.linspace(Bmin, Bmax, numB)

# set up the figure, the axis, and the plot element we want to animate
fig = plt.figure()
fig.set_size_inches(8,8)
ax = plt.axes(xlim=(xmin, xmax), ylim=(ymin, ymax))
ax.set_ylabel("Y", fontsize = 20)
ax.set_xlabel("X", fontsize = 20)

# choose a set of initial points for our trajectories (in each frame we will use the same set)
X0  = list(zip(np.linspace(xmin, xmax, num_traj), np.linspace(ymin, ymax, num_traj)))

# choose a color set for our trajectories
vcolors = plt.cm.jet_r(np.linspace(0., 1., num_traj))

# prepare the mesh grid
x = np.linspace(xmin, xmax, 15)             # divide x axis to intervals
y = np.linspace(ymin, ymax, 15)             # divide y axis to intervals
X1 , Y1  = np.meshgrid(x, y)                # from these intervals create a grid

# set up the lines, the quiver and the text object
lines = [ax.plot([], [], [], '-', c=c, lw=2)[0] for c in vcolors]
Q = ax.quiver(X1, Y1, [], [], [], pivot='mid', cmap=plt.cm.jet)
text = ax.text(0.02, 0.95, '', fontsize=20, transform=ax.transAxes)

# initialization function: plot the background of each frame. Needs to return each object to be updated
def init():
    for line in lines:
        line.set_data([], [])
    Q.set_UVC([], [], [])
    text.set_text("")
    return Q, lines, text

# animation function.  This is called sequentially
def animate(i):
    B = Bvals[i]
    DX1, DY1 = dX_dt(A, B, [X1, Y1], t)     # compute rate of change of the concentrations on grid points
    M = (np.hypot(DX1, DY1))                # norm of the rate of changes
    M[ M == 0] = 1.                         # prevention against divisions by zero
    DX1 /= M                                # we normalize the direction field vector (each has unit length now)
    DY1 /= M                                # we normalize the direction field vector (each has unit length now)
    Q.set_UVC(DX1, DY1, M)
    # integrate the Brusellator ODE's for the set of trajectories, store them in X
    for line, X0i in zip(lines, X0):
        X = integrate.odeint(lambda x, t: dX_dt(A, B, x, t), X0i,t)
        x, y = X.T        # get x and y for current trajectory
        line.set_data(x, y)
    text.set_text("A={:.2f}, B={:.2f}".format(A, B))
    return Q, lines, text

# call the animator.  blit=True means only re-draw the parts that have changed.
anim = animation.FuncAnimation(fig, animate, init_func=init, frames=100, interval=30, blit=False)
# instantiate the animator.
#anim = animation.FuncAnimation(fig, animate, init_func=init, frames=1000, interval=200, blit=True)
#HTML(anim.to_html5_video())
rc('animation', html='html5')
plt.close()
anim_____no_output_____

In the animation above, we see how the direction field gets modified as $B$ is varied. Also shown several trajectories that are initalized at various points (I have chosen them on the $Y=X$ line here)._____no_output_____## Notes:

Should you encounter difficulty in running the embedded animation, try to launch Jupter Notebook using the command:

`jupyter notebook --NotebookApp.iopub_data_rate_limit=10000000000`_____no_output_____## References:

1) Strogatz, S.H (2015). *Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering, Second Edition*, Boulder, USA: Westview Press.

2) https://en.wikipedia.org/wiki/Brusselator_____no_output_____


# Notebook from ksible/nrpytutorial
Path: in_progress/Tutorial-GiRaFFEfood_NRPy_Aligned_Rotator.ipynb

# `GiRaFFEfood`: Initial data for `GiRaFFE` Aligned Rotator

## Author: Zach Etienne & Patrick Nelson
### Formatting improvements courtesy Brandon Clark

[comment]: <> (Abstract: TODO)

**Notebook Status:**  Validated

**Validation Notes:** This tutorial notebook has been confirmed to be self-consistent with its corresponding NRPy+ module, as documented [below](#code_validation). The initial data has validated against the original `GiRaFFE`, as documented [here](Tutorial-Start_to_Finish_UnitTest-GiRaFFEfood_NRPy.ipynb).

### NRPy+ Source Code for this module: [GiRaFFEfood_NRPy_Aligned_Rotator.py](../edit/GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_Aligned_Rotator.py)

## Introduction:

This module provides another initial data option for $\texttt{GiRaFFE}$. This is a flat-spacetime test with initial data $$A_{\phi} = \frac{\mu \varpi^2}{r^3},$$ where  $\mu = B_p R^3_{\rm NS} / 2$, $R_{\rm NS}$ is the neutron star radius, and $\varpi = \sqrt{x^2+y^2}$ is the cylindrical radius. We let $A_r = A_\theta = 0$.

Additionally, the drift velocity $v^i = \Omega \textbf{e}_z \times \textbf{r} = [ijk] \Omega \textbf{e}^j_z x^k$, where $[ijk]$ is the Levi-Civita permutation symbol, $\Omega$ is the angular velocity of the "neutron star", and $\textbf{e}^i_z = (0,0,1)$. See the [Tutorial-GiRaFFEfood_NRPy](Tutorial-GiRaFFEfood_NRPy.ipynb) tutorial notebook for details on how this is used._____no_output_____

# Table of Contents:
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#initializenrpy): Import core NRPy+ modules and set NRPy+ parameters
1. [Step 2](#set_aphi): Set the vector $A_{\phi}$ Spherical coordinates
1. [Step 3](#jacobian): Use the Jacobian matrix to transform the vectors to Cartesian coordinates
1. [Step 4](#vi): Calculate $v^i$
1. [Step 5](#code_validation): Code Validation against `GiRaFFEfood_NRPy.GiRaFFEfood_NRPy_Aligned_Rotator` NRPy+ Module
1. [Step 6](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file_____no_output_____

# Step 1: Import core NRPy+ modules and set NRPy+ parameters \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

Here, we will import the NRPy+ core modules and set the reference metric to Cartesian, set commonly used NRPy+ parameters, and set C parameters that will be set from outside the code eventually generated from these expressions. We will also set up a parameter to determine what initial data is set up, although it won't do much yet._____no_output_____

# Step 0: Add NRPy's directory to the path
# https://stackoverflow.com/questions/16780014/import-file-from-parent-directory
import os,sys
nrpy_dir_path = os.path.join("..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)

# Step 0.a: Import the NRPy+ core modules and set the reference metric to Cartesian
import sympy as sp               # SymPy: The Python computer algebra package upon which NRPy+ depends
import NRPy_param_funcs as par   # NRPy+: Parameter interface
import indexedexp as ixp         # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support

import reference_metric as rfm
par.set_parval_from_str("reference_metric::CoordSystem","Cartesian")
rfm.reference_metric()

# Step 1a: Set commonly used parameters.
thismodule = "GiRaFFEfood_NRPy_Aligned_Rotator"

# The angular velocity of the "neutron star"
Omega_aligned_rotator = par.Cparameters("REAL",thismodule,"Omega_aligned_rotator",1e3)
B_p_aligned_rotator,R_NS_aligned_rotator = par.Cparameters("REAL",thismodule,
                                                           # B_p_aligned_rotator = the intensity of the magnetic field and
                                                           # R_NS_aligned_rotator= "Neutron star" radius
                                                           ["B_p_aligned_rotator","R_NS_aligned_rotator"],
                                                           [1e-5, 1.0])_____no_output_____

# Step 2: Set the vector $A_{\phi}$ in Spherical coordinates \[Back to [top](#toc)\]
$$\label{set_aphi}$$

We will first build the fundamental vector $A_i$ in spherical coordinates (see [Table 3](https://arxiv.org/pdf/1704.00599.pdf)). Note that we use [reference_metric.py](../edit/reference_metric.py) to set $r$ and $\varpi$ in terms of Cartesian coordinates; this will save us a step later when we convert to Cartesian coordinates. So, we set
\begin{align}
A_{\phi} &= \frac{\mu \varpi^2}{r^3}, \\
\end{align}
with $\mu = B_p R^3_{\rm NS} / 2$, $R_{\rm NS}$ is the neutron star radius, and $\varpi = \sqrt{x^2+y^2}$
_____no_output_____

r     = rfm.xxSph[0]
varpi = sp.sqrt(rfm.xxCart[0]**2 + rfm.xxCart[1]**2)

mu = B_p_aligned_rotator * R_NS_aligned_rotator**3 / 2

ASphD = ixp.zerorank1()

ASphD[2] = mu * varpi**2 / (r**3) # The other components were already declared to be 0._____no_output_____

# Step 3: Use the Jacobian matrix to transform the vector to Cartesian coordinates \[Back to [top](#toc)\]
$$\label{jacobian}$$

Now, we will use the coordinate transformation definitions provided by [reference_metric.py](../edit/reference_metric.py) to build the Jacobian
$$
\frac{\partial x_{\rm Sph}^j}{\partial x_{\rm Cart}^i},
$$
where $x_{\rm Sph}^j \in \{r,\theta,\phi\}$ and $x_{\rm Cart}^i \in \{x,y,z\}$. We would normally compute its inverse, but since none of the quantities we need to transform have upper indices, it is not necessary. Then, since $A_i$ and has one lower index, it will need to be multiplied by the Jacobian:

$$
A_i^{\rm Cart} = A_j^{\rm Sph} \frac{\partial x_{\rm Sph}^j}{\partial x_{\rm Cart}^i},
$$_____no_output_____

# Step 3: Use the Jacobian matrix to transform the vectors to Cartesian coordinates.
drrefmetric__dx_0UDmatrix = sp.Matrix([[sp.diff(rfm.xxSph[0],rfm.xx[0]), sp.diff(rfm.xxSph[0],rfm.xx[1]), sp.diff(rfm.xxSph[0],rfm.xx[2])],
                                       [sp.diff(rfm.xxSph[1],rfm.xx[0]), sp.diff(rfm.xxSph[1],rfm.xx[1]), sp.diff(rfm.xxSph[1],rfm.xx[2])],
                                       [sp.diff(rfm.xxSph[2],rfm.xx[0]), sp.diff(rfm.xxSph[2],rfm.xx[1]), sp.diff(rfm.xxSph[2],rfm.xx[2])]])
#dx__drrefmetric_0UDmatrix = drrefmetric__dx_0UDmatrix.inv() # We don't actually need this in this case.

AD = ixp.zerorank1(DIM=3)
for i in range(3):
    for j in range(3):
        AD[i] = drrefmetric__dx_0UDmatrix[(j,i)]*ASphD[j]_____no_output_____

# Step 4: Calculate $v^i$ \[Back to [top](#toc)\]
$$\label{vi}$$

Here, we will calculate the drift velocity $v^i = \Omega \textbf{e}_z \times \textbf{r} = [ijk] \Omega \textbf{e}^j_z x^k$, where $[ijk]$ is the Levi-Civita permutation symbol, $\Omega$ is the angular velocity of the "neutron star", and $\textbf{e}^i_z = (0,0,1)$. Conveniently, in flat space, the drift velocity reduces to the Valencia velocity because $\alpha = 1$ and $\beta^i = 0$.

However, we need to set the velocity to zero outside the neutron star; this is equivalent to only setting the non-zero velocity inside the star, $r \leq R_{\rm NS}$_____no_output_____

# Step 4: Calculate v^i
LeviCivitaSymbolDDD = ixp.LeviCivitaSymbol_dim3_rank3()

import Min_Max_and_Piecewise_Expressions as noif

unit_zU = ixp.zerorank1()
unit_zU[2] = sp.sympify(1)

global ValenciavU
ValenciavU = ixp.zerorank1()
for i in range(3):
    for j in range(3):
        for k in range(3):
            ValenciavU[i] += noif.coord_leq_bound(r,R_NS_aligned_rotator)*LeviCivitaSymbolDDD[i][j][k] * Omega_aligned_rotator * unit_zU[j] * rfm.xx[k]_____no_output_____

# Step 5: Code Validation against `GiRaFFEfood_NRPy.GiRaFFEfood_NRPy_Aligned_Rotator` NRPy+ Module  \[Back to [top](#toc)\]
$$\label{code_validation}$$

Here, as a code validation check, we verify agreement in the SymPy expressions for the $\texttt{GiRaFFE}$ Aligned Rotator initial data equations we intend to use between

1. this tutorial and
2. the NRPy+ [GiRaFFEfood_NRPy.GiRaFFEfood_NRPy_Aligned_Rotator](../edit/GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_Aligned_Rotator.py) module._____no_output_____

import GiRaFFEfood_NRPy.GiRaFFEfood_NRPy_Aligned_Rotator as gfho
gfho.GiRaFFEfood_NRPy_Aligned_Rotator()

def consistency_check(quantity1,quantity2,string):
    if quantity1-quantity2==0:
        print(string+" is in agreement!")
    else:
        print(string+" does not agree!")
        sys.exit(1)

print("Consistency check between GiRaFFEfood_NRPy tutorial and NRPy+ module:")

for i in range(3):
    consistency_check(ValenciavU[i],gfho.ValenciavU[i],"ValenciavU"+str(i))
    consistency_check(AD[i],gfho.AD[i],"AD"+str(i))Consistency check between GiRaFFEfood_NRPy tutorial and NRPy+ module:
ValenciavU0 is in agreement!
AD0 is in agreement!
ValenciavU1 is in agreement!
AD1 is in agreement!
ValenciavU2 is in agreement!
AD2 is in agreement!

# Step 6: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-GiRaFFEfood_NRPy_Aligned_Rotator.pdf](Tutorial-GiRaFFEfood_NRPy_Aligned_Rotator.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)_____no_output_____

import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-GiRaFFEfood_NRPy_Aligned_Rotator")Notebook output to PDF is only supported on Linux systems, with pdflatex installed.


# Notebook from ulgenklc/Sarcastic-Tweet-Detection
Path: iSarcasm_dataset/isarcasm_LSTM_GRU_biLSTM_Grid_Search.ipynb

!pip install pandas --quiet
!pip install torchtext --quiet_____no_output_____# We import some libraries to load the dataset
import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from collections import Counter
from tqdm.notebook import tqdm

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.utils.data import TensorDataset, DataLoader

import torchtext
from torchtext.data import get_tokenizer

from sklearn.utils import shuffle
from sklearn.metrics import classification_report
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer_____no_output_____#Change the path#
path = '/Users/bengieru/Desktop/NMA-DL/iSarcasm/iSarcasm_dataset/'
header_list = ["text", "label", 'source']
df_train_test = pd.read_csv(path + 'finaldata/train_test.csv', encoding = "utf-8", dtype = {'text':str, 'label':bool, 'source':int})
df_train_test.reset_index(drop=True, inplace=True)
del df_train_test['Unnamed: 0']
df_train_test = df_train_test.loc[df_train_test.text.apply(lambda x: not isinstance(x, (float, int)))]
df_train_test_____no_output_____# let's play with the lemmatized text to see how it's accuracy fairs
X = df_train_test.text.values
print(X.shape)
# Changes values from [0,4] to [0,1]
y = (df_train_test.label.values >= 1).astype(int)

# Split the data into train and test
x_train_text, x_test_text, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42,stratify=y)(3544,)
for s, l in zip(x_train_text[:5], y_train[:5]):
    print('{}: {}'.format(l, s))0: people still care lot leaked file sold public seen 2 leaked file tortured victim still waiting see leak journalist exposing nsa whistleblower karen stewart
0: good
1: magic money tree nurse one mp convenient
1: next train sure identifies train tram 🤦 🏻  ♀
0: prince phillip personally visit woman house apologise accident
tokenizer = get_tokenizer("basic_english")
print('Before Tokenize: ', x_train_text[4])
print('After Tokenize: ', tokenizer(x_train_text[4]))Before Tokenize:  prince phillip personally visit woman house apologise accident
After Tokenize:  ['prince', 'phillip', 'personally', 'visit', 'woman', 'house', 'apologise', 'accident']
x_train_token = [tokenizer(s) for s in tqdm(x_train_text)]
x_test_token = [tokenizer(s) for s in tqdm(x_test_text)]_____no_output_____words = Counter()
for s in x_train_token:
    for w in s:
        words[w] += 1

sorted_words = list(words.keys())
sorted_words.sort(key=lambda w: words[w], reverse=True)
print(f"Number of different Tokens in our Dataset: {len(sorted_words)}")
print(sorted_words[:20])Number of different Tokens in our Dataset: 8822
['n', 'like', 'get', 'im', 'one', 'day', 'time', 'love', 'people', 'year', 'good', 'dont', 'back', 'today', 'really', 'new', 'make', 'go', 'would', 'thing']
count_occurences = sum(words.values())

accumulated = 0
counter = 0

while accumulated < count_occurences * 0.8:
    accumulated += words[sorted_words[counter]]
    counter += 1

print(f"The {counter * 100 / len(words)}% most common words "
      f"account for the {accumulated * 100 / count_occurences}% of the occurrences")The 35.34345953298572% most common words account for the 80.00538122624693% of the occurrences
plt.bar(range(100), [words[w] for w in sorted_words[:100]])
plt.show()_____no_output_____vectorizer = CountVectorizer(binary=True)
x_train_cv = vectorizer.fit_transform(x_train_text)
x_test_cv = vectorizer.transform(x_test_text)_____no_output_____print('Before Vectorize: ', x_train_text[3])Before Vectorize:  next train sure identifies train tram 🤦 🏻  ♀
# Notice that the matriz is sparse
print('After Vectorize: ')
print(x_train_cv[3])After Vectorize:
  (0, 5091)	1
  (0, 7743)	1
  (0, 7356)	1
  (0, 3700)	1
  (0, 7749)	1
def set_device():
    device = "cuda" if torch.cuda.is_available() else "cpu"
    if device != "cuda":
        print("WARNING: For this notebook to perform best, if possible, in the menu under `Runtime` -> `Change runtime type.`  select `GPU` ")
    else:
        print("GPU is enabled in this notebook.")

    return device_____no_output_____# Set the device (check if gpu is available)
device = set_device()
#First we will create a Dictionary (`word_to_idx`).
#This dictionary will map each Token (usually words) to an index (an integer number).
#We want to limit our dictionary to a certain number of tokens (`num_words_dict`), so we will include in our ditionary those with more occurrences.WARNING: For this notebook to perform best, if possible, in the menu under `Runtime` -> `Change runtime type.`  select `GPU`
#Change the path#
glove = pd.read_csv('/Users/bengieru/Downloads/glove/glove.twitter.27B.100d.txt', sep=" ", quoting=3, header=None, index_col=0)
glove_embedding = {key: val.values for key, val in glove.T.items()}_____no_output_____len(sorted_words)_____no_output_____#Let's select only the most used.
num_words_dict = 6000
# We reserve two numbers for special tokens.
most_used_words = sorted_words[:num_words_dict-2]

#We will add two extra Tokens to the dictionary, one for words outside the dictionary (`'UNK'`) and one for padding the sequences (`'PAD'`).

# dictionary to go from words to idx
word_to_idx = {}
# dictionary to go from idx to words (just in case)
idx_to_word = {}

# We include the special tokens first
PAD_token = 0
UNK_token = 1

word_to_idx['PAD'] = PAD_token
word_to_idx['UNK'] = UNK_token

idx_to_word[PAD_token] = 'PAD'
idx_to_word[UNK_token] = 'UNK'

# We popullate our dictionaries with the most used words
for num,word in enumerate(most_used_words):
    word_to_idx[word] = num + 2
    idx_to_word[num+2] = word

def create_embedding_matrix(word_index,embedding_dict,dimension):
    embedding_matrix=np.zeros((len(word_index)+1,dimension))

    for word,index in word_index.items():
        if word in embedding_dict:
            embedding_matrix[index]=embedding_dict[word]
    return embedding_matrix

# My word_to_idx is the same as their word_index
embedding_matrix = create_embedding_matrix(word_to_idx,
                                           embedding_dict = glove_embedding,
                                           dimension = 100)

#Our goal now is to transform each tweet from a sequence of tokens to a sequence of indexes.
# These sequences of indexes will be the input to our pytorch model.

# A function to convert list of tokens to list of indexes
def tokens_to_idx(sentences_tokens,word_to_idx):
    sentences_idx = []
    for sent in sentences_tokens:
        sent_idx = []
        for word in sent:
            if word in word_to_idx:
                sent_idx.append(word_to_idx[word])
            else:
                sent_idx.append(word_to_idx['UNK'])
        sentences_idx.append(sent_idx)
    return sentences_idx

x_train_idx = tokens_to_idx(x_train_token, word_to_idx)
x_test_idx = tokens_to_idx(x_test_token, word_to_idx)

# We need all the sequences to have the same length.
# To select an adequate sequence length, let's explore some statistics about the length of the tweets:
tweet_lens = np.asarray([len(sentence) for sentence in x_train_idx])
print('Max tweet word length: ',tweet_lens.max())
print('Mean tweet word length: ',np.median(tweet_lens))
print('99% percent under: ',np.quantile(tweet_lens,0.99))Max tweet word length:  50
Mean tweet word length:  9.0
99% percent under:  29.0
#We cut the sequences which are larger than our chosen maximum length (`max_lenght`) and fill with zeros the ones that are shorter.
 # We choose the max length
max_length = 30

# A function to make all the sequence have the same lenght
# Note that the output is a Numpy matrix
def padding(sentences, seq_len):
    features = np.zeros((len(sentences), seq_len),dtype=int)
    for ii, tweet in enumerate(sentences):
        len_tweet = len(tweet)
        if len_tweet != 0:
            if len_tweet <= seq_len:
        # If its shorter, we fill with zeros (the padding Token index)
                features[ii, -len(tweet):] = np.array(tweet)[:seq_len]
            if len_tweet > seq_len:
        # If its larger, we take the last 'seq_len' indexes
                features[ii, :] = np.array(tweet)[-seq_len:]
    return features_____no_output_____# We convert our list of tokens into a numpy matrix
# where all instances have the same lenght
x_train_pad = padding(x_train_idx,max_length)
x_test_pad = padding(x_test_idx,max_length)

# We convert our target list a numpy matrix
y_train_np = np.asarray(y_train)
y_test_np = np.asarray(y_test)_____no_output_____some_number = 2
print('Before padding: ', x_train_idx[some_number])
print('After padding: ', x_train_pad[some_number])Before padding:  [1540, 109, 831, 3367, 6, 2102, 3368]
After padding:  [   0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0 1540  109  831 3367    6
 2102 3368]
# create Tensor datasets
train_data = TensorDataset(torch.from_numpy(x_train_pad), torch.from_numpy(y_train_np))
valid_data = TensorDataset(torch.from_numpy(x_test_pad), torch.from_numpy(y_test_np))

# Batch size (this is an important hyperparameter)
batch_size = 100

# dataloaders
# make sure to SHUFFLE your data
train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size,drop_last = True)
valid_loader = DataLoader(valid_data, shuffle=True, batch_size=batch_size,drop_last = True)_____no_output_____x_train_pad.shape, x_test_pad.shape, y_train_np.shape, y_test_np.shape_____no_output_____# Obtain one batch of training data
dataiter = iter(train_loader)
sample_x, sample_y = dataiter.next()

print('Sample input size: ', sample_x.size()) # batch_size, seq_length
print('Sample input: \n', sample_x)
print('Sample input: \n', sample_y)Sample input size:  torch.Size([100, 30])
Sample input:
 tensor([[   0,    0,    0,  ...,  183,    6,  136],
        [   0,    0,    0,  ..., 1515,  102, 1662],
        [   0,    0,    0,  ...,  360,    1, 1096],
        ...,
        [   0,    0,    0,  ...,    1,   89,    1],
        [   0,    0,    0,  ...,  237,  457, 1043],
        [   0,    0,    0,  ..., 1978,    1,  153]])
Sample input:
 tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0,
        0, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0,
        0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0,
        0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,
        0, 1, 0, 0])

Now, we will define the `SentimentRNN` class. Most of the model's class will be familiar to you, but there are two important layers we would like you to pay attention to:

*   Embedding Layer
> This layer is like a linear layer, but it makes it posible to use a sequence of inedexes as inputs (instead of a sequence of one-hot-encoded vectors). During training, the Embedding layer learns a linear transformation from the space of words (a vector space of dimension `num_words_dict`) into the a new, smaller, vector space of dimension `embedding_dim`. We suggest you to read this [thread](https://discuss.pytorch.org/t/how-does-nn-embedding-work/88518/3) and the [pytorch documentation](https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html) if you want to learn more about this particular kind of layers.

*   LSTM layer
> This is one of the most used class of Recurrent Neural Networks. In Pytorch we can add several stacked layers in just one line of code. In our case, the number of layers added are decided with the parameter `no_layers`. If you want to learn more about LSTMs we strongly recommend you this [Colahs thread](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) about them.

_____no_output_____

%%time
# Let us a LSTM RNN using the pre-trianed embedding from GLOVE.
class SentimentLSTM(nn.Module):
    def __init__(self, no_layers, hidden_dim, embedding_matrix, drop_prob=0.1):
        super(SentimentLSTM,self).__init__()

        self.output_dim = output_dim
        self.hidden_dim = hidden_dim
        self.no_layers = no_layers
        self.drop_prob = drop_prob
        self.vocab_size = embedding_matrix.shape[0]
        self.embedding_dim = embedding_matrix.shape[1]

    # Embedding Layer
        self.embedding=nn.Embedding(self.vocab_size,self.embedding_dim)
        self.embedding.weight=nn.Parameter(torch.tensor(embedding_matrix,dtype=torch.float32))
        self.embedding.weight.requires_grad=True

    # LSTM Layers
        self.lstm = nn.LSTM(input_size=self.embedding_dim,hidden_size=self.hidden_dim,
                            num_layers=no_layers, bidirectional=False, batch_first=True,
                            dropout=self.drop_prob)

    # Dropout layer
        self.dropout = nn.Dropout(drop_prob)

    # Linear and Sigmoid layer
        self.fc = nn.Linear(self.hidden_dim, output_dim)
        self.sig = nn.Sigmoid()

    def forward(self,x,hidden):

        batch_size = x.size(0)

        embeds = self.embedding(x)
    #Shape: [batch_size x max_length x embedding_dim]

    # LSTM out
        lstm_out, hidden = self.lstm(embeds, hidden)
    # Shape: [batch_size x max_length x hidden_dim]

    # Select the activation of the last Hidden Layer
        lstm_out = lstm_out[:,-1,:].contiguous()
    # Shape: [batch_size x hidden_dim]

    ## You can instead average the activations across all the times
    # lstm_out = torch.mean(lstm_out, 1).contiguous()

    # Dropout and Fully connected layer
        out = self.dropout(lstm_out)
        out = self.fc(out)

    # Sigmoid function
        sig_out = self.sig(out)

    # return last sigmoid output and hidden state
        return sig_out, hidden

    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
    # Create two new tensors with sizes n_layers x batch_size x hidden_dim,
    # initialized to zero, for hidden state and cell state of LSTM
        h0 = torch.zeros((self.no_layers,batch_size,self.hidden_dim)).to(device)
        c0 = torch.zeros((self.no_layers,batch_size,self.hidden_dim)).to(device)
        hidden = (h0,c0)
        return hidden

#######################################################################################
# Parameters of our network
# Size of our vocabulary
# Number of stacked LSTM layers
no_layerss = [2, 4, 8]
# Dimension of the hidden layer in LSTMs
hidden_dim = 64
# Dropout parameter for regularization
output_dim = 1
# Dropout parameter for regularization
drop_prob = 0.5

# Let's define our model
fig,ax = plt.subplots(3,2, figsize = (18, 18))
fig.suptitle('LSTM model with hidden_dim = %d, output_dim = %d, drop_prob = %.2f'%(hidden_dim, output_dim, drop_prob), fontsize = 24)
for no, no_layers in enumerate(no_layerss):
    model = SentimentLSTM(no_layers,
                          hidden_dim,
                          embedding_matrix,
                          drop_prob = drop_prob)
# Moving to gpu
    model.to(device)
    print(model)

#######################################################################################
# How many trainable parameters does our model have?
    model_parameters = filter(lambda p: p.requires_grad, model.parameters())
    params = sum([np.prod(p.size()) for p in model_parameters])
    print('Total Number of parameters: ',params)

#######################################################################################
# loss and optimization functions
    lrs = np.linspace(0.0003, 0.0007, 5)#define a range of learning rates

# Binary crossentropy is a good loss function for a binary classification problem
    criterion = nn.BCELoss()

# We choose an Adam optimizer
    optimizers = []
    for i,lr in enumerate(lrs):
        optimizer = torch.optim.Adam(model.parameters(), lr = lr)
        optimizers.append(optimizer)

#######################################################################################
# function to predict accuracy
    def acc(pred,label):
        pred = torch.round(pred.squeeze())
        return torch.sum(pred == label.squeeze()).item()

#######################################################################################
# Number of training Epochs
    epochs = 5

# Maximum absolute value accepted for the gradeint
    clip = 5

# Initial Loss value (assumed big)
    valid_loss_min = np.Inf

# Lists to follow the evolution of the loss and accuracy
    epoch_tr_loss, epoch_vl_loss = [[] for i in range(len(lrs))],[[] for i in range(len(lrs))]
    epoch_tr_acc, epoch_vl_acc = [[] for i in range(len(lrs))],[[] for i in range(len(lrs))]

# Train for a number of Epochs
    for i, lr in enumerate(lrs):
        for epoch in range(epochs):
            train_losses = []
            train_acc = 0.0
            model.train()

            for inputs, labels in train_loader:

    # Initialize hidden state
                h = model.init_hidden(batch_size)
    # Creating new variables for the hidden state
                h = tuple([each.data.to(device) for each in h])

    # Move batch inputs and labels to gpu
                inputs, labels = inputs.to(device), labels.to(device)

    # Set gradient to zero
                model.zero_grad()

    # Compute model output
                output,h = model(inputs,h)

    # Calculate the loss and perform backprop
                loss = criterion(output.squeeze(), labels.float())
                loss.backward()
                train_losses.append(loss.item())

    # calculating accuracy
                accuracy = acc(output,labels)
                train_acc += accuracy

    #`clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
                nn.utils.clip_grad_norm_(model.parameters(), clip)
                optimizers[i].step()

  # Evaluate on the validation set for this epoch
            val_losses = []
            val_acc = 0.0
            model.eval()
            for inputs, labels in valid_loader:

    # Initialize hidden state
                val_h = model.init_hidden(batch_size)
                val_h = tuple([each.data.to(device) for each in val_h])

    # Move batch inputs and labels to gpu
                inputs, labels = inputs.to(device), labels.to(device)

    # Compute model output
                output, val_h = model(inputs, val_h)

    # Compute Loss
                val_loss = criterion(output.squeeze(), labels.float())

                val_losses.append(val_loss.item())

                accuracy = acc(output,labels)
                val_acc += accuracy

            epoch_train_loss = np.mean(train_losses)
            epoch_val_loss = np.mean(val_losses)
            epoch_train_acc = train_acc/len(train_loader.dataset)
            epoch_val_acc = val_acc/len(valid_loader.dataset)
            epoch_tr_loss[i].append(epoch_train_loss)
            epoch_vl_loss[i].append(epoch_val_loss)
            epoch_tr_acc[i].append(epoch_train_acc)
            epoch_vl_acc[i].append(epoch_val_acc)
            #print(f'Epoch {epoch+1}')
            #print(f'train_loss : {epoch_train_loss} val_loss : {epoch_val_loss}')
            #print(f'train_accuracy : {epoch_train_acc*100} val_accuracy : {epoch_val_acc*100}')
            if epoch_val_loss <= valid_loss_min:
                #print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(valid_loss_min,epoch_val_loss))
    # torch.save(model.state_dict(), '../working/state_dict.pt')
                valid_loss_min = epoch_val_loss
            #print(25*'==')

#######################################################################################
# plot the results from the training and validation accuracies
#fig,ax = plt.subplots(3,2, figsize = 60,20)
    colors = ['red', 'blue', 'black', 'purple', 'green', 'brown']
    for i,e in enumerate(lrs):
        ax[no][0].plot(epoch_tr_acc[i], linestyle = '--', color = colors[i], label = 'Train Acc, LR = %.4f'%e)
        ax[no][0].plot(epoch_vl_acc[i], color = colors[i], label = 'Validation Acc, LR = %.4f'%e)
# plt.ylim([70, 80])
    ax[no][0].set_title("Accuracy", fontsize = 20)
    ax[no][0].set_xlabel('Epochs', fontsize = 16)
    ax[no][0].set_xticks([0,1,2,3,4])
    ax[no][0].set_ylabel('Number of LSTM Layers = %d'%no_layers, fontsize = 20)
    #ax[no][0].legend(prop = dict(size = 12))

    for i,e in enumerate(lrs):
        ax[no][1].plot(epoch_tr_loss[i], color = colors[i], linestyle = '--', label = 'Train Acc, LR = %.4f'%e)
        ax[no][1].plot(epoch_vl_loss[i], color = colors[i], label = 'Validation Acc, LR = %.4f'%e)
    ax[no][1].set_title("Loss", fontsize = 20)
    ax[no][1].set_xlabel('Epochs', fontsize = 16)
    ax[no][1].set_xticks([0,1,2,3,4])
    #ax[no][1].set_ylabel('Number of Layers = %d'%no_layers, fontsize = 20)
    ax[no][1].legend(prop = dict(size = 10))
#plt.tight_layout()SentimentLSTM(
  (embedding): Embedding(6001, 100)
  (lstm): LSTM(100, 64, num_layers=2, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc): Linear(in_features=64, out_features=1, bias=True)
  (sig): Sigmoid()
)
Total Number of parameters:  675941
SentimentLSTM(
  (embedding): Embedding(6001, 100)
  (lstm): LSTM(100, 64, num_layers=4, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc): Linear(in_features=64, out_features=1, bias=True)
  (sig): Sigmoid()
)
Total Number of parameters:  742501
SentimentLSTM(
  (embedding): Embedding(6001, 100)
  (lstm): LSTM(100, 64, num_layers=8, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc): Linear(in_features=64, out_features=1, bias=True)
  (sig): Sigmoid()
)
Total Number of parameters:  875621
CPU times: user 3min 57s, sys: 1min 7s, total: 5min 5s
Wall time: 3min 46s
%%time
class SentimentGRU(nn.Module):
    def __init__(self, no_layers, hidden_dim, embedding_matrix, drop_prob = 0.01):
        super(SentimentGRU, self).__init__()

        self.output_dim = output_dim
        self.hidden_dim = hidden_dim
        self.no_layers = no_layers
        self.drop_prob = drop_prob
        self.vocab_size = embedding_matrix.shape[0]
        self.embedding_dim = embedding_matrix.shape[1]

    # Embedding layer
        self.embedding=nn.Embedding(self.vocab_size,self.embedding_dim)
        self.embedding.weight=nn.Parameter(torch.tensor(embedding_matrix,dtype=torch.float32))
        self.embedding.weight.requires_grad=True

    # GRU Layers
        self.gru = nn.GRU(input_size= self.embedding_dim,hidden_size=self.hidden_dim,
                            num_layers=no_layers, bidirectional=False, batch_first=True,
                            dropout=self.drop_prob)

    # Dropout layer
        self.dropout = nn.Dropout(drop_prob)

    # Linear and Sigmoid layer
        self.fc = nn.Linear(self.hidden_dim, output_dim)
        self.sig = nn.Sigmoid()

    def forward(self,x,hidden):
        batch_size = x.size(0)
        self.h = self.init_hidden(batch_size)

    # Embedding out
        embeds = self.embedding(x)
    #Shape: [batch_size x max_length x embedding_dim]

    # GRU out
        gru_out, self.h = self.gru(embeds, self.h)
    # Shape: [batch_size x max_length x hidden_dim]

    # Select the activation of the last Hidden Layer
        gru_out = gru_out[:,-1,:].contiguous()
    # Shape: [batch_size x hidden_dim]

    # Dropout and Fully connected layer
        out = self.dropout(gru_out)
        out = self.fc(out)

    # Sigmoid function
        sig_out = self.sig(out)

    # return last sigmoid output and hidden state
        return sig_out

    def init_hidden(self, batch_size):
        hidden = (torch.zeros((self.no_layers,batch_size,self.hidden_dim)).to(device))
        return hidden

#######################################################################################
# Parameters of our network
# Size of our vocabulary
# Number of stacked LSTM layers
no_layerss = [2, 4, 8]
# Dimension of the hidden layer in LSTMs
hidden_dim = 64
# Dropout parameter for regularization
output_dim = 1
# Dropout parameter for regularization
drop_prob = 0.5

# Let's define our model
fig,ax = plt.subplots(3,2, figsize = (18, 18))
fig.suptitle('GRU model with hidden_dim = %d, output_dim = %d, drop_prob = %.2f'%(hidden_dim, output_dim, drop_prob), fontsize = 24)
for no, no_layers in enumerate(no_layerss):
    model = SentimentGRU(no_layers,
                         hidden_dim,
                         embedding_matrix,
                         drop_prob = drop_prob)
# Moving to gpu
    model.to(device)
    print(model)

# How many trainable parameters does our model have?
    model_parameters = filter(lambda p: p.requires_grad, model.parameters())
    params = sum([np.prod(p.size()) for p in model_parameters])
    print('Total Number of parameters: ',params)

#######################################################################################
# loss and optimization functions
    lrs = np.linspace(0.0003, 0.0007, 5) #define a range of learning rates

# Binary crossentropy is a good loss function for a binary classification problem
    criterion = nn.BCELoss()

# We choose an Adam optimizer
    optimizers = []
    for i,lr in enumerate(lrs):
        optimizer = torch.optim.Adam(model.parameters(), lr=lr)
        optimizers.append(optimizer)

#######################################################################################
# function to predict accuracy
    def acc(pred,label):
        pred = torch.round(pred.squeeze())
        return torch.sum(pred == label.squeeze()).item()

#######################################################################################
#######################################################################################
# Number of training Epochs
    epochs = 5

# Maximum absolute value accepted for the gradeint
    clip = 5

# Initial Loss value (assumed big)
    valid_loss_min = np.Inf

# Lists to follow the evolution of the loss and accuracy
    epoch_tr_loss,epoch_vl_loss = [[] for i in range(len(lrs))],[[] for i in range(len(lrs))]
    epoch_tr_acc,epoch_vl_acc = [[] for i in range(len(lrs))],[[] for i in range(len(lrs))]

# Train for a number of Epochs
    for i, lr in enumerate(lrs):
        for epoch in range(epochs):
            train_losses = []
            train_acc = 0.0
            model.train()

            for inputs, labels in train_loader:

    # Move batch inputs and labels to gpu
                inputs, labels = inputs.to(device), labels.to(device)

    # Set gradient to zero
                model.zero_grad()

    # Compute model output
                output = model(inputs,h)

    # Calculate the loss and perform backprop
                loss = criterion(output.squeeze(), labels.float())
                loss.backward()
                train_losses.append(loss.item())

    # calculating accuracy
                accuracy = acc(output,labels)
                train_acc += accuracy

    #`clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
                nn.utils.clip_grad_norm_(model.parameters(), clip)
                optimizer.step()

  # Evaluate on the validation set for this epoch
            val_losses = []
            val_acc = 0.0
            model.eval()
            for inputs, labels in valid_loader:

    # Move batch inputs and labels to gpu
                inputs, labels = inputs.to(device), labels.to(device)

    # Compute model output
                output = model(inputs, val_h)

    # Compute Loss
                val_loss = criterion(output.squeeze(), labels.float())

                val_losses.append(val_loss.item())

                accuracy = acc(output,labels)
                val_acc += accuracy

            epoch_train_loss = np.mean(train_losses)
            epoch_val_loss = np.mean(val_losses)
            epoch_train_acc = train_acc/len(train_loader.dataset)
            epoch_val_acc = val_acc/len(valid_loader.dataset)
            epoch_tr_loss[i].append(epoch_train_loss)
            epoch_vl_loss[i].append(epoch_val_loss)
            epoch_tr_acc[i].append(epoch_train_acc)
            epoch_vl_acc[i].append(epoch_val_acc)
            #print(f'Epoch {epoch+1}')
            #print(f'train_loss : {epoch_train_loss} val_loss : {epoch_val_loss}')
            #print(f'train_accuracy : {epoch_train_acc*100} val_accuracy : {epoch_val_acc*100}')
            if epoch_val_loss <= valid_loss_min:
                #print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(valid_loss_min,epoch_val_loss))
    # torch.save(model.state_dict(), '../working/state_dict.pt')
                valid_loss_min = epoch_val_loss
            #print(25*'==')

#######################################################################################
# plot the results from the training and validation accuracies
#fig,ax = plt.subplots(3,2, figsize = 60,20)
    colors = ['red', 'blue', 'black', 'purple', 'green', 'brown']
    for i,e in enumerate(lrs):
        ax[no][0].plot(epoch_tr_acc[i], linestyle = '--', color = colors[i], label = 'Train Acc, LR = %.4f'%e)
        ax[no][0].plot(epoch_vl_acc[i], color = colors[i], label = 'Validation Acc, LR = %.4f'%e)
# plt.ylim([70, 80])
    ax[no][0].set_title("Accuracy", fontsize = 20)
    ax[no][0].set_xlabel('Epochs', fontsize = 16)
    ax[no][0].set_xticks([0,1,2,3,4])
    ax[no][0].set_ylabel('Number of GRU Layers = %d'%no_layers, fontsize = 20)
    #ax[no][0].legend(prop = dict(size = 12))

    for i,e in enumerate(lrs):
        ax[no][1].plot(epoch_tr_loss[i], color = colors[i], linestyle = '--', label = 'Train Acc, LR = %.4f'%e)
        ax[no][1].plot(epoch_vl_loss[i], color = colors[i], label = 'Validation Acc, LR = %.4f'%e)
    ax[no][1].set_title("Loss", fontsize = 20)
    ax[no][1].set_xlabel('Epochs', fontsize = 16)
    ax[no][1].set_xticks([0,1,2,3,4])
    #ax[no][1].set_ylabel('Number of Layers = %d'%no_layers, fontsize = 20)
    ax[no][1].legend(prop = dict(size = 10))
#plt.tight_layout()SentimentGRU(
  (embedding): Embedding(6001, 100)
  (gru): GRU(100, 64, num_layers=2, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc): Linear(in_features=64, out_features=1, bias=True)
  (sig): Sigmoid()
)
Total Number of parameters:  656997
SentimentGRU(
  (embedding): Embedding(6001, 100)
  (gru): GRU(100, 64, num_layers=4, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc): Linear(in_features=64, out_features=1, bias=True)
  (sig): Sigmoid()
)
Total Number of parameters:  706917
SentimentGRU(
  (embedding): Embedding(6001, 100)
  (gru): GRU(100, 64, num_layers=8, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc): Linear(in_features=64, out_features=1, bias=True)
  (sig): Sigmoid()
)
Total Number of parameters:  806757
CPU times: user 3min 27s, sys: 1min 11s, total: 4min 38s
Wall time: 3min 18s
%%time
class SentimentLSTM_bi(nn.Module):
    def __init__(self,no_layers,hidden_dim,embedding_matrix,drop_prob=0.1):
        super(SentimentLSTM_bi,self).__init__()

        self.output_dim = output_dim
        self.hidden_dim = hidden_dim
        self.no_layers = no_layers
        self.drop_prob = drop_prob
        self.vocab_size = embedding_matrix.shape[0]
        self.embedding_dim = embedding_matrix.shape[1]

    # Embedding Layer
        self.embedding=nn.Embedding(self.vocab_size,self.embedding_dim)
        self.embedding.weight=nn.Parameter(torch.tensor(embedding_matrix,dtype=torch.float32))
        self.embedding.weight.requires_grad=True

    # LSTM Layers
        self.lstm = nn.LSTM(input_size=self.embedding_dim,hidden_size=self.hidden_dim,
                            num_layers=no_layers, bidirectional=True, batch_first=True,
                            dropout=self.drop_prob)

    # Dropout layer
        self.dropout = nn.Dropout(drop_prob)

    # Linear and Sigmoid layer
        self.fc = nn.Linear(self.hidden_dim * 2, output_dim)
        self.sig = nn.Sigmoid()

    def forward(self,x,hidden):
        batch_size = x.size(0)
        embeds = self.embedding(x)
    #Shape: [batch_size x max_length x embedding_dim]

    # LSTM out
        lstm_out, hidden = self.lstm(embeds, hidden)
    # Shape: [batch_size x max_length x hidden_dim]

    # Select the activation of the last Hidden Layer
        lstm_out = lstm_out[:,-1,:].contiguous()
    # Shape: [batch_size x hidden_dim]

    ## You can instead average the activations across all the times
    # lstm_out = torch.mean(lstm_out, 1).contiguous()

    # Dropout and Fully connected layer
        out = self.dropout(lstm_out)
        out = self.fc(out)

    # Sigmoid function
        sig_out = self.sig(out)

    # return last sigmoid output and hidden state
        return sig_out, hidden

    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
    # Create two new tensors with sizes n_layers x batch_size x hidden_dim,
    # initialized to zero, for hidden state and cell state of LSTM
        h0 = torch.zeros((self.no_layers * 2,batch_size, self.hidden_dim)).to(device)
        c0 = torch.zeros((self.no_layers * 2,batch_size, self.hidden_dim)).to(device)

        hidden = (h0,c0)
        return hidden

#######################################################################################
# Parameters of our network
# Size of our vocabulary
# Number of stacked LSTM layers
no_layerss = [2, 4, 8]
# Dimension of the hidden layer in LSTMs
hidden_dim = 64
# Dropout parameter for regularization
output_dim = 1
# Dropout parameter for regularization
drop_prob = 0.5

# Let's define our model
fig,ax = plt.subplots(3,2, figsize = (18, 18))
fig.suptitle('BI-LSTM model with hidden_dim = %d, output_dim = %d, drop_prob = %.2f'%(hidden_dim, output_dim, drop_prob), fontsize = 24)
for no, no_layers in enumerate(no_layerss):
    model = SentimentLSTM_bi(no_layers,
                             hidden_dim,
                             embedding_matrix,
                             drop_prob = drop_prob)
# Moving to gpu
    model.to(device)
    print(model)

#######################################################################################
# How many trainable parameters does our model have?
    model_parameters = filter(lambda p: p.requires_grad, model.parameters())
    params = sum([np.prod(p.size()) for p in model_parameters])
    print('Total Number of parameters: ',params)

#######################################################################################
# loss and optimization functions
    lrs = np.linspace(0.0003, 0.0007, 5)#define a range of learning rates

# Binary crossentropy is a good loss function for a binary classification problem
    criterion = nn.BCELoss()

# We choose an Adam optimizer
    optimizers = []
    for i,lr in enumerate(lrs):
        optimizer = torch.optim.Adam(model.parameters(), lr=lr)
        optimizers.append(optimizer)

#######################################################################################
# function to predict accuracy
    def acc(pred,label):
        pred = torch.round(pred.squeeze())
        return torch.sum(pred == label.squeeze()).item()

#######################################################################################
#######################################################################################
# Number of training Epochs
    epochs = 5

# Maximum absolute value accepted for the gradeint
    clip = 5

# Initial Loss value (assumed big)
    valid_loss_min = np.Inf

# Lists to follow the evolution of the loss and accuracy
    epoch_tr_loss,epoch_vl_loss = [[] for i in range(len(lrs))],[[] for i in range(len(lrs))]
    epoch_tr_acc,epoch_vl_acc = [[] for i in range(len(lrs))],[[] for i in range(len(lrs))]

# Train for a number of Epochs
    for i, lr in enumerate(lrs):
        for epoch in range(epochs):
            train_losses = []
            train_acc = 0.0
            model.train()

            for inputs, labels in train_loader:

    # Initialize hidden state
                h = model.init_hidden(batch_size)
    # Creating new variables for the hidden state
                h = tuple([each.data.to(device) for each in h])

    # Move batch inputs and labels to gpu
                inputs, labels = inputs.to(device), labels.to(device)

    # Set gradient to zero
                model.zero_grad()

    # Compute model output
                output,h = model(inputs,h)

    # Calculate the loss and perform backprop
                loss = criterion(output.squeeze(), labels.float())
                loss.backward()
                train_losses.append(loss.item())

    # calculating accuracy
                accuracy = acc(output,labels)
                train_acc += accuracy

    #`clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
                nn.utils.clip_grad_norm_(model.parameters(), clip)
                optimizer.step()

  # Evaluate on the validation set for this epoch
            val_losses = []
            val_acc = 0.0
            model.eval()
            for inputs, labels in valid_loader:

    # Initialize hidden state
                val_h = model.init_hidden(batch_size)
                val_h = tuple([each.data.to(device) for each in val_h])

    # Move batch inputs and labels to gpu
                inputs, labels = inputs.to(device), labels.to(device)

    # Compute model output
                output, val_h = model(inputs, val_h)

    # Compute Loss
                val_loss = criterion(output.squeeze(), labels.float())

                val_losses.append(val_loss.item())

                accuracy = acc(output,labels)
                val_acc += accuracy

            epoch_train_loss = np.mean(train_losses)
            epoch_val_loss = np.mean(val_losses)
            epoch_train_acc = train_acc/len(train_loader.dataset)
            epoch_val_acc = val_acc/len(valid_loader.dataset)
            epoch_tr_loss[i].append(epoch_train_loss)
            epoch_vl_loss[i].append(epoch_val_loss)
            epoch_tr_acc[i].append(epoch_train_acc)
            epoch_vl_acc[i].append(epoch_val_acc)
            #print(f'Epoch {epoch+1}')
            #print(f'train_loss : {epoch_train_loss} val_loss : {epoch_val_loss}')
            #print(f'train_accuracy : {epoch_train_acc*100} val_accuracy : {epoch_val_acc*100}')
            if epoch_val_loss <= valid_loss_min:
             #   print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(valid_loss_min,epoch_val_loss))
    # torch.save(model.state_dict(), '../working/state_dict.pt')
                valid_loss_min = epoch_val_loss
            #print(25*'==')

#######################################################################################
# plot the results from the training and validation accuracies
#fig,ax = plt.subplots(3,2, figsize = 60,20)
    colors = ['red', 'blue', 'black', 'purple', 'green', 'brown']
    for i,e in enumerate(lrs):
        ax[no][0].plot(epoch_tr_acc[i], linestyle = '--', color = colors[i], label = 'Train Acc, LR = %.4f'%e)
        ax[no][0].plot(epoch_vl_acc[i], color = colors[i], label = 'Validation Acc, LR = %.4f'%e)
# plt.ylim([70, 80])
    ax[no][0].set_title("Accuracy", fontsize = 20)
    ax[no][0].set_xlabel('Epochs', fontsize = 16)
    ax[no][0].set_xticks([0,1,2,3,4])
    ax[no][0].set_ylabel('Number of BI-LSTM Layers = %d'%no_layers, fontsize = 20)
    #ax[no][0].legend(prop = dict(size = 12))

    for i,e in enumerate(lrs):
        ax[no][1].plot(epoch_tr_loss[i], color = colors[i], linestyle = '--', label = 'Train Acc, LR = %.4f'%e)
        ax[no][1].plot(epoch_vl_loss[i], color = colors[i], label = 'Validation Acc, LR = %.4f'%e)
    ax[no][1].set_title("Loss", fontsize = 20)
    ax[no][1].set_xlabel('Epochs', fontsize = 16)
    ax[no][1].set_xticks([0,1,2,3,4])
    #ax[no][1].set_ylabel('Number of Layers = %d'%no_layers, fontsize = 20)
    ax[no][1].legend(prop = dict(size = 10))
#plt.tight_layout()SentimentLSTM_bi(
  (embedding): Embedding(6001, 100)
  (lstm): LSTM(100, 64, num_layers=2, batch_first=True, dropout=0.5, bidirectional=True)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc): Linear(in_features=128, out_features=1, bias=True)
  (sig): Sigmoid()
)
Total Number of parameters:  784549
SentimentLSTM_bi(
  (embedding): Embedding(6001, 100)
  (lstm): LSTM(100, 64, num_layers=4, batch_first=True, dropout=0.5, bidirectional=True)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc): Linear(in_features=128, out_features=1, bias=True)
  (sig): Sigmoid()
)
Total Number of parameters:  983205
SentimentLSTM_bi(
  (embedding): Embedding(6001, 100)
  (lstm): LSTM(100, 64, num_layers=8, batch_first=True, dropout=0.5, bidirectional=True)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc): Linear(in_features=128, out_features=1, bias=True)
  (sig): Sigmoid()
)
Total Number of parameters:  1380517
CPU times: user 8min 44s, sys: 2min 32s, total: 11min 17s
Wall time: 8min 34s