# Notebook from Chiroptera/QCThesis
Path: notebooks/Analyzing results - Mighty4.ipynb

# K-Means: Python vs NumPy vs CUDA [Mighty4]
#### Author: Diogo Silva

This notebook contains an analysis of the results from executing the different implementations (CUDA, NumPy, Python) of the K-Means algorithm.

The machine technical specifications of the machine (Mighty4) are:
- OS: Ubuntu 12.04.5 LTS
- CPU: Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz, 12 cores
- GPU: NVIDIA QUADRO K600, 192 CUDA cores, 1024 MBytes_____no_output_____

# necessary imports
%pylab inline
import seaborn as sns
import pandas as pdPopulating the interactive namespace from numpy and matplotlib
# locations of the results
results_filename="/home/chiroptera/workspace/QCThesis/CUDA/tests/test1v3/results.csv" #local
#results_filename="https://raw.githubusercontent.com/Chiroptera/QCThesis/master/CUDA/tests/test1v3/results.csv" #git repo_____no_output_____results = pd.read_csv(results_filename)_____no_output_____print "Structure of the results"
results.head()Structure of the results
N_labels=[1e3,5e3,1e4,5e4,1e5,5e5,1e6,2e6,4e6]
K_labels=[5,10,20,30,40,50,100,250,500]_____no_output_____

Some of the parameters were don't change in these results, so we can delete them (natural number of clusters, dimensionality and number of iterations). Furthermore, We can delete the rounds column because it becomes useless after averaging the times._____no_output_____

results.drop(['R','NATC','D','iters'], axis=1, inplace=True)

results.head()_____no_output_____

Below is some statistics about the timings for the rounds. The important thing to notice is that there is **low variance** on the data, which suggests that the **results are consistent**._____no_output_____

rounds = results.groupby(['type','N','K'],as_index = True)
results_mean = rounds.mean()
rounds.describe()_____no_output_____

# Time analysis

This section explores some of the results of the runtimes of the algorithms._____no_output_____

times = results_mean.loc["cuda"]
times['cuda']=times['time']
times['numpy']=results_mean.loc["numpy"]
times['python']=results_mean.loc["python"]
times['s_cuda_np']=times['numpy']/times['cuda']
times['s_cuda_py']=times['python']/times['cuda']
times['s_np_py']=times['python']/times['numpy']
times_____no_output_____a=times.groupby(level='K')
#a.get_group(20)['python'].plot(subplots=True,layout=(2,2))
p=a.get_group(20)[['python','numpy','cuda']].plot(title="Time evolution; 20 clusters",logy=True)
plt.xticks(range(len(N_labels)),N_labels)
plt.xlabel("Cardinality")
a.get_group(500)[['python','numpy','cuda']].plot(title="Time evolution; 500 clusters",logy=True)
plt.xticks(range(len(N_labels)),N_labels)
plt.xlabel("Cardinality")_____no_output_____b=times.groupby(level='N')
b.get_group(1e5)[['python','numpy','cuda']].plot(title="Time evolution by number of clusters; 1e5 datapoints",logy=True)
plt.xticks(range(len(K_labels)),K_labels)
plt.xlabel("Number of clusters")_____no_output_____b.get_group(1e5)[['numpy','cuda']].plot(title="Time evolution by number of clusters; 1e5 datapoints",logy=True)
plt.xticks(range(len(K_labels)),K_labels)
plt.xlabel("Number of clusters")

b.get_group(4e6)[['numpy','cuda']].plot(title="Time evolution by number of clusters; 4e6 datapoints",logy=True)
plt.xticks(range(len(K_labels)),K_labels)
plt.xlabel("Number of clusters")_____no_output_____

# Speedup over NumPy_____no_output_____

s_cuda_np = results_mean.loc['numpy'] / results_mean.loc['cuda']
#s_cuda_np['speedup']=s_cuda_np['time']_____no_output_____s_cuda_np.groupby(level=['K']).describe()_____no_output_____for key, grp in s_cuda_np.groupby(level=['K']):
    plt.plot(grp['time'],label=key)#grp.index.levels[0],

plt.legend(loc='best')
plt.title("Speedup by cardinality")
plt.plot([0, 8], [1, 1], 'k-', lw=2)
plt.ylabel("Speedup")
plt.xlabel("Cardinality")

plt.xticks(range(len(N_labels)),N_labels)_____no_output_____s_cuda_np.groupby(level=['N']).describe()_____no_output_____for key, grp in s_cuda_np.groupby(level=['N']):
    plt.plot(grp['time'],label=key)#grp.index.levels[0],

plt.plot([0, 8], [1, 1], 'k-', lw=2) #slowdown/speedup threshold
plt.legend(loc='best')

plt.title("Speedup by cardinality")
plt.ylabel("Speedup")
plt.xlabel("Number of clusters")

plt.xticks(range(len(K_labels)),K_labels)_____no_output_____

# Speedup over Python_____no_output_____

s_cuda_py = results_mean.loc['python'] / results_mean.loc['cuda']_____no_output_____for key, grp in s_cuda_py.groupby(level=['K']):
    plt.plot(grp['time'],label=key)#grp.index.levels[0],

plt.plot([0, 8], [1, 1], 'k-', lw=2) #slowdown/speedup threshold
plt.legend(loc='best')

plt.title("Speedup by cardinality")
plt.ylabel("Speedup")
plt.xlabel("Cardinality")

plt.xticks(range(len(N_labels)),N_labels)_____no_output_____for key, grp in s_cuda_py.groupby(level=['N']):
    plt.plot(grp['time'],label=key)#grp.index.levels[0],

plt.plot([0, 8], [1, 1], 'k-', lw=2) #slowdown/speedup threshold
plt.legend(loc='best')

plt.title("Speedup by cardinality")
plt.ylabel("Speedup")
plt.xlabel("Number of clusters")

plt.xticks(range(len(K_labels)),K_labels)_____no_output_____


# Notebook from Schoyen/FYS4460
Path: project-4/random-walks-on-the-spanning-cluster-large.ipynb

# Random walks on the spanning cluster

In this notebook we'll explore diffusion on the spanning cluster._____no_output_____

import sys
sys.path.append("..")

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.ndimage
import tqdm
import sklearn.linear_model

from generate_spanning_cluster import get_spanning_cluster
from percwalk import percwalk, parallel_percwalk
from log_binning import log_bin_____no_output_____sns.set(color_codes=True)_____no_output_____# Critical percolation probability
p_c = 0.59275_____no_output_____

## Distance moved on the percolating cluster

We'll start by exploring diffusion on the percolating cluster for $p > p_c$. We'll measure the second moment of the average distance, $\langle R^2 \rangle$, as a function of the number of steps performed. Below we'll demonstrate what the diffusion on the percolating cluster looks like._____no_output_____

spanning_cluster = get_spanning_cluster(100, p_c)_____no_output_____

Having created a system with a percolating cluster, we start by visualizing the spanning cluster._____no_output_____

plt.figure(figsize=(14, 10))

plt.imshow(spanning_cluster, origin="lower")
plt.show()_____no_output_____

Here we can see the sites contained in the percolating cluster. Moving on, we'll start a walker on a random site in the percolating cluster._____no_output_____

num_walks = int(1e5)
num_steps = 0_____no_output_____while num_steps <= 1:
    walker_map, displacements, num_steps = percwalk(spanning_cluster, num_walks)_____no_output_____plt.figure(figsize=(14, 10))

plt.imshow(spanning_cluster, origin="lower")
# walker_map is oriented as row-column (ix, iy)
plt.plot(walker_map[1], walker_map[0])
plt.show()_____no_output_____

In the above plot we can see how the walker moves in the spanning cluster. A thick line signifies several walks over a site._____no_output_____## Mean squared distance

Next, we'll look at the mean squared distance, $\langle r^2 \rangle$, as a function of the number of walks, $N$, for varying $p > p_c$.
A free random walker behaves according to
\begin{align}
    \langle r^2 \rangle \propto D t,
\end{align}
as we've seen in the molecular dynamics simulations.
On the percolating cluster we expect that the random walker at times gets stuck in dangling ends or slows down through singly connected bonds.
We introduce the scaling theory
\beging{align}
    \langle r^2 \rangle = t^{2k} f\left[ (p - p_c) t^x \right],
\end{align}
for the mean squared distance on a system with random set sites.
We can categorize the function $f(u)$ for three different situations.
\begin{align}
    f(u) = \begin{cases}
        \text{const} & |u| \ll 1, \\
        u^{\mu} & u \gg 1, \\
        (-u)^{\beta - 2\nu} &  u \ll -1.
    \end{cases}
\end{align}
The first situation arises when $p \approx p_c$ and yields the scaling
\begin{align}
    \langle r^2 \rangle \propto t^{2k},
\end{align}
where $k < 1/2$.
This is known as _subdiffusion_ as opposed to the _superdiffusion_ observed for the initial molecular dynamics simulations.
The second situation $u \gg 1 \implies p > p_c$ and $t > t_0$, where $t_0$ is some threshold time.
This will yield the scaling theory
\begin{align}
    \langle r^2 \rangle \propto (p - p_c)^{\mu} t \propto t^{2k} \left[(p - p_c) t^{x} \right]^{\mu}.
\end{align}
The final situation is found when $p < p_c$ and $t > t_0$ yielding the scaling theory
\begin{align}
    \langle r^2 \rangle \propto (p_c - p)^{\beta - 2\nu} \propto t^{2k} \left[ (p_c - p) t^x \right]^{\beta - 2\nu}.
\end{align}
We will not be exploring the latter situation.
When $p > p_c$ and after a certain time $t_0$ has passed we expect the random walker on the cluster to behave as a normal free random walker.
That is, when $t \gg t_0$ we get
\begin{align}
    \langle r^2 \rangle \propto D t,
\end{align}
when $p > p_c$.
Stated differently, when $\langle r^2 \rangle \gg \xi^2$ the walker behaves as a free random walker.
The interesting fact here is that when $p = p_c$ we know that $\xi \to \infty$ and this condition will never be fullfilled.
This means that the scaling theory described above for the mean squared distance when $p = p_c$ will be true indefinitely.
We can summarize it by the function
\begin{align}
    \langle r^2 \rangle \propto \begin{cases}
        t^{2k} & \langle r^2 \rangle \ll \xi^2, \\
        D t & \langle r^2 \rangle \gg \xi^2.
    \end{cases}
\end{align}_____no_output_____We now simulate a system of size $L$ and compute the mean squared displacement on the spanning cluster several times for every system._____no_output_____

L = 2048
num_walks = 1 * int(1e6)
num_walks_arr = np.arange(num_walks)
p_arr = np.linspace(p_c, 0.8, 12)
num_walkers = 500
num_systems = 20

r_squared = np.zeros((len(p_arr), num_walks))_____no_output_____

It is important to note that the implementation for the percolation walker we use require $2L^2 > N_w$, where $N_w$ is the number of walks (or, time if you like) we let the walkers perform. This is so that the walker won't start backtracking from whence it came thereby flattening the displacement curve._____no_output_____

for i, p in tqdm.tqdm_notebook(enumerate(p_arr), total=len(p_arr)):
    spanning_cluster = get_spanning_cluster(L, p, num_attempts=1000)
    r_squared[i] += parallel_percwalk(spanning_cluster, num_walks, num_walkers, num_systems)_____no_output_____

Having computed the mean squared displacement for several different values of $p \geq p_c$ we look at the diffusion behavior.
For $p = p_c$ we know that $\xi \to \infty$ and we know that
\begin{align}
    \langle r^2(t) \rangle \propto t^{2k},
\end{align}
for all values of $t$ as we will never have $\langle r^2(t) \rangle \gg \xi^2$.
We therefore estimate $k$ by taking the logarithm on both sides of the equation and using linear regression.
\begin{gather}
    \log\left( \langle r^2(t) \rangle \right)
    = \log(C) + 2k \log(t),
\end{gather}
where $C$ is a constant serving as the intercept._____no_output_____

plt.figure(figsize=(14, 10))

plt.loglog(num_walks_arr, r_squared[0])
plt.fill_between(num_walks_arr, r_squared[0], alpha=0.2)
plt.xlabel(r"$t$")
plt.ylabel(r"$\langle r^2(t) \rangle$")
plt.title(r"Plot of the mean squared distance as a function of $t$ for $p = p_c$")
plt.show()_____no_output_____

In this figure we can see how the log-log plot of the mean squared distance as a function of time for $p = p_c$ becomes linear.
We now use linear regression to find an estimate for $k$._____no_output_____

log_walks = np.log(num_walks_arr[1:])
log_r = np.log(r_squared[0, 1:])

clf = sklearn.linear_model.LinearRegression().fit(
    log_walks[:, np.newaxis], log_r[:, np.newaxis]
)

k = clf.coef_[0, 0] / 2
C = clf.intercept_[0]

print(f"k = {k}")k = 0.35000464022009997

Having found the intercept and the coefficient $k$ we plot $\langle r^2(t) \rangle$ as a function of the number of walks (time, in a sense) and compare it to the theoretical estimate._____no_output_____

plt.figure(figsize=(14, 10))

plt.plot(
    num_walks_arr,
    r_squared[0],
    label=rf"$p = p_c = {p_c}$",
)
plt.fill_between(num_walks_arr, r_squared[0], alpha=0.2)

plt.plot(
    num_walks_arr,
    np.exp(C) * num_walks_arr ** (2 * k),
    "--",
    label=r"$\langle r^2(t) \rangle = \exp(C) t^{2k}$",
)

plt.title(r"Plot of the mean squared displacement as a function of time $t$ for $p = p_c$")
plt.xlabel(r"$t$")
plt.ylabel(r"$\langle r^2(t) \rangle$")
plt.legend(loc="best")
plt.show()_____no_output_____

In this figure we have plotted the mean squared displacement as a function of "time", i.e., the number of walks, on the spanning cluster for $p = p_c$.
We observe the expected subdiffusion.
We also see how the theoretical estimate of the subdiffusion follows the data closely.
Below we plot the mean squared distance for $p \geq p_c$._____no_output_____

plt.figure(figsize=(14, 10))

plt.loglog(
    num_walks_arr,
    r_squared[0],
    "--",
    label=rf"$p = p_c = {p_c}$",
)
plt.fill_between(num_walks_arr, r_squared[0], alpha=0.2)

for i, p in enumerate(p_arr[1:]):
    plt.loglog(
        num_walks_arr,
        r_squared[i + 1],
        label=rf"$p = {p:.3f}$"
    )

plt.legend(loc="best")
plt.xlabel(r"$t$")
plt.ylabel(r"$\langle r^2(t) \rangle$")
plt.title(r"Plot of the mean squared distance as a function of time for $p \geq p_c$")
plt.show()_____no_output_____

In the above figure we see how after a time $t_0$ the mean squared distance for $p > p_c$ deviates from the situation when $p = p_c$.
Reformulating the scaling theory for $p > p_c$ to create a data collapse we get
\begin{gather}
    \langle r^2 \rangle \propto t^{2k} (p - p_c) t^{x}
    \implies
    t^{-2k} \langle r^2 \rangle \propto (p - p_c) t^{x}.
\end{gather}
We plot this on a log-log axis when creating the data collapse.
However, the challenge is to find an expression for $x$.
We have previously found that
\begin{align}
    \langle r^2(t) \rangle \propto D(p) t,
\end{align}
where $D(p) = (p - p_c)^{\mu}$ when $p > p_c$ and $r \gg \xi$.
Inserted into the scaling theory we can write
\begin{gather}
    (p - p_c)^{\mu} t \propto t^{2k} \left[(p - p_c) t^{x}\right]^{\mu}
    \implies
    1 = 2k + \mu x
    \implies
    x = \frac{1 - 2k}{\mu}.
\end{gather}
Having already found $k$ we are thus tasked with finding an expression for $\mu$ and we can be on our merry way to plotting a data collapse._____no_output_____We know that when $t < t_0$ the mean squared distance scales as $t^{2k}$ when $p = p_c$ and $p > p_c$.
However, when $t > t_0$ this scaling continues only for $p = p_c$ whereas for $p > p_c$ the mean squared distance will increase faster in time.
We are interested in locating $t_0$ and we do this by finding the point where
\begin{align}
    \langle r^2(t; p_c) \rangle \leq \frac{1}{2} \langle r^2(t; p > p_c) \rangle,
\end{align}
that is, we find the point where the scaling is more than twice as high.
At $t = t_0$ we know that $\langle r^2(t_0) \rangle = \xi^2$.
We are lucky enough to have a scaling ansatz for the characteristic cluster length stating
\begin{align}
    \xi = \xi_0 (p - p_c)^{-\nu}.
\end{align}_____no_output_____

plt.figure(figsize=(14, 10))

plt.loglog(
    num_walks_arr,
    r_squared[0],
    "--",
    label=rf"$p = p_c = {p_c}$",
)
plt.fill_between(num_walks_arr, r_squared[0], alpha=0.2)

xi_squared_list = []
t_list = []
ind_list = []
p_list = []

for i, p in enumerate(p_arr[1:]):
    plt.loglog(
        num_walks_arr,
        r_squared[i + 1],
        label=rf"$p = {p:.3f}$"
    )

    mask = r_squared[0, 1:] <= 0.5 * r_squared[i + 1, 1:]

    if not np.any(mask):
        continue

    index = np.argmax(mask)

    xi_squared_list.append(r_squared[i + 1, index])
    t_list.append(num_walks_arr[index])
    ind_list.append(i + 1)
    p_list.append(p)

    plt.scatter(num_walks_arr[index], r_squared[i + 1, index])

plt.legend(loc="best")
plt.xlabel(r"$t$")
plt.ylabel(r"$\langle r^2(t) \rangle$")
plt.title(r"Plot of the mean squared distance as a function of time for $p \geq p_c$")
plt.show()_____no_output_____

It is interesting to note that for a larger value of $p > p_c$ the cross-over time $t_0$ becomes smaller.
That is, for a higher percolation probability $p$ the time it takes for the system to behave as a "normal" random walker becomes shorter.
This also makes sense as there are fewer sites not contained in the spanning cluster._____no_output_____Having found the points $(t_0, \xi^2)$ we now find an expression for $\nu$.
Taking the logarithm on both sides of the scaling ansatz for the characteristic length yields
\begin{align}
    \log(\xi) = \log(\xi_0) - \nu \log(p - p_c).
\end{align}
We use linear regression to find an expression for the intercept $\log(\xi_0)$ and the exponent $\nu$._____no_output_____

p_min_pc_arr = np.abs(np.array(p_list) - p_c)

clf = sklearn.linear_model.LinearRegression().fit(
    np.log(p_min_pc_arr)[:, np.newaxis],
    np.log(np.sqrt(xi_squared_list))[:, np.newaxis],
)

nu = -clf.coef_[0, 0]
log_xi_0 = clf.intercept_[0]

print(f"nu = {nu}")nu = 1.2067707882240395

The exact value of $\nu$ in two dimensions is $\nu = 4/3$._____no_output_____

nu_exact = 4 / 3
print(f"nu_exact = {nu_exact}")nu_exact = 1.3333333333333333

We therefore see that we are in ballpark range.
We plot the characteristic length as a function of $p$ and compare it to the theoretical scaling using our estimate of $\nu$._____no_output_____

fig = plt.figure(figsize=(14, 10))

plt.plot(
    p_list,
    np.sqrt(xi_squared_list),
    "-o",
    label=r"$Measured$",
)
plt.plot(
    p_list,
    np.exp(log_xi_0) * p_min_pc_arr ** (-nu),
    "-o",
    label=r"$(p - p_c)^{-%.3f}$" % nu
)
#plt.plot(
#    p_list,
#    np.exp(log_xi_0) * p_min_pc_arr ** (-nu_exact),
#    "-o",
#    label=r"$(p - p_c)^{-%.3f}$" % nu_exact
#)
plt.xlabel(r"$p$")
plt.ylabel(r"$\xi$")
plt.title(r"Plot of the characteristic length as a function of the percolation probability $p$")
plt.legend(loc="best")
plt.show()_____no_output_____

In this figure we see how our estimate of $\nu$ follows the measured data quite well.
And, more importantly, they follow the theoretical scaling using the exact value of $\nu$ closely.
Looking at the point where the cross-over time occurs, we see that this scales inversely with increasing $p$.
We know that for $p > p_c$ and $t < t_0$ we have the relation
\begin{align}
    \langle r^2(t) \rangle \propto t^{2k},
\end{align}
that is the system experiences subdiffusion.
When $t > t_0$ the relation takes on the form
\begin{align}
    \langle r^2(t) \rangle \propto t^{2k}\left[(p - p_c) t^x \right]^{\mu}.
\end{align}
However, at the cross-over time, that is when $t = t_0$, we expect these to be proportional.
\begin{gather}
    t_0^{2k} \propto
    t_0^{2k}\left[(p - p_c) t_0^{x} \right]^{\mu}
    \implies
    (p - p_c) t_0^{x} \propto 1
    \implies
    t_0 \propto |p - p_c|^{-1/x}.
\end{gather}_____no_output_____

neg_inv_x, log_C = np.polyfit(np.log(p_min_pc_arr), np.log(t_list), deg=1)

x = -1 / neg_inv_x
print(f"x = {x}")x = 0.29554963750661734
fig = plt.figure(figsize=(14, 10))

plt.plot(
    p_list,
    t_list,
    "-o",
    label=r"Measured",
)

plt.plot(
    p_list,
    np.exp(log_C) * p_min_pc_arr ** (-1 / x),
    "-o",
    label=r"$t_0 \propto |p - p_c|^{-1/x}$"
)

plt.xlabel(r"$p$")
plt.ylabel(r"$t_0$")
plt.title(r"Plot of the scaling of the cross-over time as a function for $p$")
plt.legend(loc="best")
plt.show()_____no_output_____

In order to find an estimate for $x$ we need to find a value for $\mu$.
As we've stated earlier, when $p > p_c$ and $t \gg t_0$ which means that $r \gg \xi$ the random walker moves away from the anomalous diffusion and instead behaves as a free random walker with the relation
\begin{align}
    \langle r^2(t) \rangle = D(p) t.
\end{align}
The Einstein relation for diffusion relates the diffusion constant to the conductance through
\begin{align}
    D(p) \propto \sigma(p) \propto (p - p_c)^{\mu}.
\end{align}_____no_output_____

plt.figure(figsize=(14, 10))

diffusion_list = []
intercept_list = []
p_min_pc_diff_list = []

for i, p_i in enumerate(ind_list):
    t_0 = t_list[i]
    p = p_arr[p_i]

    p_min_pc_diff_list.append(p - p_c)

    cross = num_walks_arr > t_0

    t_cross = num_walks_arr[cross]
    r_cross = r_squared[p_i, cross]

    d, intercept = np.polyfit(t_cross, r_cross, deg=1)
    diffusion_list.append(d)
    intercept_list.append(intercept)

    plt.plot(t_cross, r_cross, label=fr"$p = {p:.3f}$")

plt.title(r"Plot of the mean squared distance as a function of $t > t_0$")
plt.xlabel(r"$t$")
plt.ylabel(r"$\langle r^2(t) \rangle$")
plt.legend(loc="best")
plt.show()_____no_output_____

Ideally we expect the mean squared distance to be completely linear, however, we see that even though $t > t_0$ there are slight deviations to this trend._____no_output_____Taking the logarithm on both sides of the scaling ansatz we are able to estimate $\mu$.
\begin{align}
    \log(D) \propto \mu \log(p - p_c).
\end{align}
Again we use the least squares method to estimate $\mu$._____no_output_____

mu, intercept = np.polyfit(np.log(p_min_pc_diff_list), np.log(diffusion_list), deg=1)_____no_output_____print(mu)0.8923720197088654

From the lecture notes we know that the exact value of $\mu$ is given by $\mu = 1.3$._____no_output_____

mu_exact = 1.3_____no_output_____

This shows that we miss by quite a lot when it comes to estimating $\mu$._____no_output_____

plt.figure(figsize=(14, 10))

plt.plot(
    p_min_pc_diff_list,
    diffusion_list,
    "-o",
    label=r"Measured",
)
plt.plot(
    p_min_pc_diff_list,
    np.exp(intercept) * np.array(p_min_pc_diff_list) ** (mu),
    "-o",
    label=r"$(p - p_c)^{%.3f}$" % mu,
)
plt.xlabel(r"$p - p_c$")
plt.ylabel(r"$D(p)$")
plt.title(r"Plot of the diffusion constant as a function of $p - p_c$")
plt.legend(loc="best")
plt.show()_____no_output_____

In this figure we've plotted the diffusion constant as a function of $p - p_c$.
We compare the measured values with the theoretical scaling using our estimate of $\mu$._____no_output_____

x = (1 - 2 * k) / mu
print(f"x = {x}")x = 0.3361722610460954
p_min_pc = p_arr[1:] - p_c
r_gg_xi = r_squared[1:, 1:]
t_sq = num_walks_arr[1:] ** (-2 * k)_____no_output_____plt.figure(figsize=(14, 10))

for i in range(len(p_min_pc)):
    plt.plot(
        p_min_pc[i] * num_walks_arr[1:] ** (x),
        t_sq * r_gg_xi[i],
        label=fr"$p = {p_arr[i + 1]:.3f}$",
    )
    #plt.plot(
    #    p_min_pc[i] * num_walks_arr[1:] ** (x),
    #    np.exp(intercept) * (p_min_pc[i] * num_walks_arr[1:] ** (x)) ** mu,
    #    label=fr"Theoretical: $p = {p_arr[i + 1]:.3f}$",
    #)

plt.legend(loc="best")
plt.xlabel(r"$(p - p_c) t^{x}$")
plt.ylabel(r"$t^{-2k} \langle r^{2} (t)\rangle$")
plt.title(r"Data-collapse plot for mean squared distance")
plt.show()_____no_output_____

In this figure we can see the data-collapse of the mean squared displacement.
We have compared to the theoretical scaling
\begin{gather}
    \langle r^2 \rangle \propto t^{2k} (p - p_c) t^{x}
    \propto t^{2k}\left[(p - p_c) t^{x}\right]^{\mu},
\end{gather}
where we use the intercept from the diffusion constant as the constant of proportionality.
We note that collapse isn't completely perfect, especially when $p$ is close to $p_c$.
A possible explanation to this is due to the percolation walker being restricted to a finite spanning cluster.
We might be able to alleviate some of this discrepancy if we allow the spanning cluster to have periodic boundaries, however this will require quite a lot of manipulation when labelling the spanning cluster and for the percolation walker and has therefore not been tested._____no_output_____Finally we are interested in estimating the dimension of the random walk, $d_w$.
We have that
\begin{align}
    d_w = \frac{1}{k}.
\end{align}
We can also estimate $d_w$ from the scaling relation
\begin{align}
    \langle r^2(t) \rangle \propto t^{2k} = t^{2/d_w},
\end{align}
when we $t < t_0$ for $p > p_c$._____no_output_____

d_w_list = []
d_w_intercept_list = []
p_min_pc_d_w_list = []

for i, p_i in enumerate(ind_list):
    t_0 = t_list[i]
    p = p_arr[p_i]

    p_min_pc_d_w_list.append(p - p_c)

    pre_cross = num_walks_arr < t_0

    t_pre_cross = num_walks_arr[pre_cross]
    r_pre_cross = r_squared[p_i, pre_cross]

    d_w_2_inv, d_w_intercept = np.polyfit(t_pre_cross, r_pre_cross, deg=1)
    d_w_list.append(2 / d_w_2_inv)
    d_w_intercept_list.append(d_w_intercept)_____no_output_____fig = plt.figure(figsize=(14, 10))

plt.plot(p_list, d_w_list, "-o")
plt.xlabel(r"$p$")
plt.ylabel(r"$d_w$")
plt.title(r"Plot of the dimension of the random walk as a function of $p$")
plt.show()_____no_output_____


# Notebook from shui02/appyter-catalog
Path: appyters/PHGKB_Harmonizome_ETL/PHGKB.ipynb

# Harmonizome ETL: Public Health Genomics Knowledge Base (PHGKB)_____no_output_____Created by: Charles Dai

Credit to: Moshe Silverstein

Data Source: https://phgkb.cdc.gov/PHGKB/downloadCenter.action_____no_output_____

# appyter init
from appyter import magic
magic.init(lambda _=globals: _())_____no_output_____import sys
import os
from datetime import date

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

import harmonizome.utility_functions as uf
import harmonizome.lookup as lookup_____no_output_____%load_ext autoreload
%autoreload 2_____no_output_____

### Notebook Information_____no_output_____

print('This notebook was run on:', date.today(), '\nPython version:', sys.version)_____no_output_____

# Initialization_____no_output_____

%%appyter hide_code

{% do SectionField(
    name='data',
    title='Upload Data',
    img='load_icon.png'
) %}_____no_output_____%%appyter code_eval

{% do DescriptionField(
    name='description',
    text='The original input file was sourced from phgkb.cdc.gov. There is no direct link to download, so it should be downloaded from the source website. The "Genopedia" source is the one that should be downloaded.',
    section='data'
) %}

{% set df_file = FileField(
    constraint='.*\.txt$',
    name='genopedia',
    default='GeneID-Disease.txt',
    label='Genopedia (txt)',
    examples={
        'GeneID-Disease.txt': 'https://phgkb.cdc.gov/PHGKB/downloadCenter.action'
    },
    section='data'
) %}
_____no_output_____

### Load Mapping Dictionaries_____no_output_____

symbol_lookup, geneid_lookup = lookup.get_lookups()_____no_output_____

### Output Path_____no_output_____

output_name = 'phgkb'

path = 'Output/PHGKB'
if not os.path.exists(path):
    os.makedirs(path)_____no_output_____

# Load Data_____no_output_____

%%appyter code_exec

df = pd.read_csv(
    {{df_file}},
    skiprows=4, sep='#', header=None)_____no_output_____df.head()_____no_output_____df.shape_____no_output_____

# Pre-process Data_____no_output_____## Get Relevant Data_____no_output_____

def splitter(s):
    # split attributes from gene, remove parentheses
    lst = list(map(lambda x: x[:x.find('(')], s.split('\t')))
    return lst[0], lst[1:]
df = pd.DataFrame(df[0].map(splitter).tolist(), columns=['Gene Symbol', 'Disease'])
df.head()_____no_output_____

## Split Attribute Lists_____no_output_____

df = df.explode('Disease').dropna().set_index('Gene Symbol')
df.head()_____no_output_____

# Filter Data_____no_output_____## Map Gene Symbols to Up-to-date Approved Gene Symbols_____no_output_____

df = uf.map_symbols(df, symbol_lookup, remove_duplicates=True)
df.shape_____no_output_____

# Analyze Data_____no_output_____## Create Binary Matrix_____no_output_____

binary_matrix = uf.binary_matrix(df)
binary_matrix.head()_____no_output_____binary_matrix.shape_____no_output_____uf.save_data(binary_matrix, path, output_name + '_binary_matrix',
            compression='npz', dtype=np.uint8)_____no_output_____

## Create Gene List_____no_output_____

gene_list = uf.gene_list(binary_matrix, geneid_lookup)
gene_list.head()_____no_output_____gene_list.shape_____no_output_____uf.save_data(gene_list, path, output_name + '_gene_list',
            ext='tsv', compression='gzip', index=False)_____no_output_____

## Create Attribute List_____no_output_____

attribute_list = uf.attribute_list(binary_matrix)
attribute_list.head()_____no_output_____attribute_list.shape_____no_output_____uf.save_data(attribute_list, path, output_name + '_attribute_list',
            ext='tsv', compression='gzip')_____no_output_____

## Create Gene and Attribute Set Libraries_____no_output_____

uf.save_setlib(binary_matrix, 'gene', 'up', path, output_name + '_gene_up_set')_____no_output_____uf.save_setlib(binary_matrix, 'attribute', 'up', path,
                           output_name + '_attribute_up_set')_____no_output_____

## Create Attribute Similarity Matrix_____no_output_____

attribute_similarity_matrix = uf.similarity_matrix(binary_matrix.T, 'jaccard', sparse=True)
attribute_similarity_matrix.head()_____no_output_____uf.save_data(attribute_similarity_matrix, path,
            output_name + '_attribute_similarity_matrix',
            compression='npz', symmetric=True, dtype=np.float32)_____no_output_____

## Create Gene Similarity Matrix_____no_output_____

gene_similarity_matrix = uf.similarity_matrix(binary_matrix, 'jaccard', sparse=True)
gene_similarity_matrix.head()_____no_output_____uf.save_data(gene_similarity_matrix, path,
            output_name + '_gene_similarity_matrix',
            compression='npz', symmetric=True, dtype=np.float32)_____no_output_____

## Create Gene-Attribute Edge List_____no_output_____

edge_list = uf.edge_list(binary_matrix)
uf.save_data(edge_list, path, output_name + '_edge_list',
        ext='tsv', compression='gzip')_____no_output_____

# Create Downloadable Save File_____no_output_____

uf.archive(path)_____no_output_____

### Link to download output files: [click here](./output_archive.zip)_____no_output_____


# Notebook from mathemage/TheMulQuaBio
Path: notebooks/06-Python_II.ipynb

< [Biological Computing in Python I](05-Python_I.ipynb) | [Main Contents](Index.ipynb) | [Biological Computing in R](07-R.ipynb) >_____no_output_____# Biological Computing in Python II

>> ...some things in life are bad. They can really make you mad. Other things just make you swear and curse. When you're chewing on life's gristle, don't grumble; give a whistle, and this'll help things turn out for the best. And... always look on the bright side of life...
— Guess who?_____no_output_____
Contents

1  Numerical computing in Python

1.1  Indexing and accessing arrays

1.2  Manipulating arrays

1.2.1  Replacing, adding or deleting elements

1.2.2  Flattening or reshaping arrays

1.3  Pre-allocating arrays

1.4  numpy matrices

1.4.1  Matrix-vector operations

2  Two particularly useful scipy sub-packages

2.1  sc.stats

2.2  Numerical integration using  scipy

2.2.1  The Lotka-Volterra model

3  Plotting in Python

4  Practicals

5  The need for speed: profiling code

5.1  Profiling in Python

5.2  Quick profiling with timeit

6  Practicals

6.1  Lotka-Volterra model problem

6.2  Extra Credit problems

7  Networks in Python

7.1  Food web network example

8  Practicals

9  Regular expressions in Python

9.1  Metacharacters vs. regular characters

9.2  regex elements

9.3  Regex in python

10  Practicals: Some RegExercises

10.1  Grouping regex patterns

11  Useful re commands

11.1  Finding all matches

11.2  Finding in files

11.3  Groups within multiple matches

11.4  Extracting text from webpages

11.5  Replacing text

12  Practicals

12.1  Blackbirds problem

13  Using Python to build workflows

13.1  Using subprocess

13.1.1  Running processes

13.2  Handling directory and file paths

13.3  Running R

14  Practicals

14.1  Using os problem 1

14.2  Using os problem 2

15  Readings and Resources

_____no_output_____In this chapter, we will build on the [first Python Chapter](05-Python_I.ipynb). We cover some more advanced topics that will round-off your training in Biological Computing in Python.

## Numerical computing in Python

The python package `scipy` allows you to do serious number crunching, including:

* Linear algebra (matrix and vector operations)
* Numerical integration (solving ODEs)
* Fourier transforms
* Interpolation
* Calculating special functions (incomplete Gamma, Bessel, etc.)
* Generation of random numbers
* Using statistical functions and transformations

In the following, we will use the `numpy array` data structure for data manipulations and calculations. These
arrays are similar in some respects to python lists, but are more naturally multidimensional, homogeneous in type (the default is float), and allow efficient (fast) manipulations. Thus numpy arrays are analogous to the R `matrix` data object/structure. We will use the `scipy` package, which includes `numpy`, and lot more.

So let's try `scipy`:_____no_output_____

import scipy as sc_____no_output_____a = sc.array(range(5)) # a one-dimensional array
a_____no_output_____print(type(a))<class 'numpy.ndarray'>
print(type(a[0]))<class 'numpy.int64'>

Thus the last two outputs tell you that firstly, there is a data structure type (and a class) called `numpy.ndarray`, and secondly, that at position `0` (remember, Python indexing starts at 0) it holds an [64 bit integer](https://en.wikipedia.org/wiki/9,223,372,036,854,775,807). All elements in `a` will be of type `int` because that is what `range()` returns (try `?range`).

            (Source: [http://pages.physics.cornell.edu/~myers/teaching/ComputationalMethods/python/arrays.html](http://pages.physics.cornell.edu/~myers/teaching/ComputationalMethods/python/arrays.html))

            A graphical depiction of numpy/numpy arrays, which can have multiple dimensions (even greater than 3).

You can also specify the data type of the array:_____no_output_____

a = sc.array(range(5), float)
a_____no_output_____a.dtype # Check type _____no_output_____

You can also get a 1-D arrays as follows:_____no_output_____

x = sc.arange(5)
x_____no_output_____x = sc.arange(5.) #directly specify float using decimal
x_____no_output_____

As with other Python variables (e.g., created as a list or a dictionary), you can apply methods to variables created as numpy arrays. For example, type `x.` and hit TAB to see all methods you can apply to`x`. To see dimensions of `x`:_____no_output_____

x.shape_____no_output_____

Remember, you can type `:?x.methodname` to get info on a particular method. For example, try `?x.shape`.

You can also convert to and from Python lists:_____no_output_____

b = sc.array([i for i in range(10) if i%2==1]) #odd numbersbetween 1 and `0
b_____no_output_____c = b.tolist() #convert back to list
c_____no_output_____

To make a matrix, you need a 2-D numpy array:_____no_output_____

mat = sc.array([[0, 1], [2, 3]])
mat_____no_output_____mat.shape_____no_output_____

### Indexing and accessing arrays

As with other Python data objects such as lists, numpy array elements can be accessed using square brackets (`[ ]`) with the usual `[row,column]` reference. Indexing of numpy arrays works like that for other data structures, with index values starting at 0. So, you can obtain all the elements of a particular row as:_____no_output_____

mat[1] # accessing whole 2nd row, remember indexing starts at  0_____no_output_____mat[:,1] #accessing whole second column  _____no_output_____

And accessing particular elements:_____no_output_____

mat[0,0] # 1st row, 1st column element_____no_output_____mat[1,0] # 2nd row, 1st column element_____no_output_____

Note that (like all other programming languages) row index always comes before column index. That is, `mat[1]` is always going to mean "whole second row", and `mat[1,1]` means 1st row and 1st column element. Therefore, to access the whole second column, you need:_____no_output_____

mat[:,0] #accessing whole first column  _____no_output_____

Python indexing also accepts negative values for going back to the start
from the end of an array:_____no_output_____

mat[0,1]_____no_output_____mat[0,-1] #interesting!_____no_output_____mat[0,-2] #very interesting, but rather useless for this simple matrix!_____no_output_____

### Manipulating arrays

Manipulating numpy arrays is pretty straightforward.

---
> **Why numpy arrays are computationally efficient:**  The data associated with a numpy array object (its metadata  – number of dimensions, shape, data type, etc – as well as the actual data) are stored in a homogeneous and contiguous block of memory (a "data buffer"), at a particular address in the system's RAM (Random Access Memory). This makes numpy arrays more efficient than a pure Python data structures like lists whose data are scattered across the system memory.

---

#### Replacing, adding or deleting elements

Let's look at how you can replace, add, or delete an array element (a single entry, or whole row(s) or whole column(s)):_____no_output_____

mat[0,0] = -1 #replace a single element
mat_____no_output_____mat[:,0] = [12,12] #replace whole column
mat_____no_output_____sc.append(mat, [[12,12]], axis = 0) #append row, note axis specification_____no_output_____sc.append(mat, [[12],[12]], axis = 1) #append column_____no_output_____newRow = [[12,12]] #create new row_____no_output_____mat = sc.append(mat, newRow, axis = 0) #append that existing row
mat_____no_output_____sc.delete(mat, 2, 0) #Delete 3rd row_____no_output_____

And concatenation:_____no_output_____

mat = sc.array([[0, 1], [2, 3]])
mat0 = sc.array([[0, 10], [-1, 3]])
sc.concatenate((mat, mat0), axis = 0)_____no_output_____

#### Flattening or reshaping arrays

You can also "flatten" or "melt" arrays, that is, change array dimensions (e.g., from a matrix to a vector):_____no_output_____

mat.ravel() # NOTE: ravel is row-priority - happens row by row_____no_output_____mat.reshape((4,1)) # this is different from ravel - check ?sc.reshape_____no_output_____mat.reshape((1,4)) # NOTE: reshaping is also row-priority_____no_output_____mat.reshape((3, 1)) # But total elements must remain the same!_____no_output_____

This is a bit different than how [`R` behaves](07-R.ipynb#Recycling), where you won't get an error (R "recycles" data), which can be dangerous!

### Pre-allocating arrays

As in other computer languages, it is usually more efficient to preallocate an array rather than append / insert / concatenate additional elements, rows, or columns. *why*? – because you might run out of contiguous space in the specific system memory (RAM) address where the current array is stored. Preallocation allocates all the RAM memory you need in one call, while resizing the array (through append,insert,concatenate, resize, etc) may require copying the array to a larger block of memory, slowing things down, and significantly so if the matrix/array is very large.

For example, if you know the size of your matrix or array, you can initialize it with ones or zeros:_____no_output_____

sc.ones((4,2)) #(4,2) are the (row,col) array dimensions_____no_output_____sc.zeros((4,2)) # or zeros_____no_output_____m = sc.identity(4) #create an identity matrix
m_____no_output_____m.fill(16) #fill the matrix with 16
m_____no_output_____

### `numpy` matrices

Scipy/Numpy also has a `matrix` data structure class. Numpy matrices are strictly 2-dimensional, while numpy arrays are N-dimensional. Matrix objects are a subclass of numpy arrays, so they inherit all the attributes and methods of numpy arrays (ndarrays).

The main advantage of scipy matrices is that they provide a convenient notation for matrix multiplication: if `a` and `b` are matrices, then `a * b` is their matrix product.

#### Matrix-vector operations

Now let's perform some common matrix-vector operations on arrays (you can also try the same using matrices instead of arrays):_____no_output_____

mm = sc.arange(16)
mm = mm.reshape(4,4) #Convert to matrix
mm_____no_output_____mm.transpose()_____no_output_____mm + mm.transpose()_____no_output_____mm - mm.transpose()_____no_output_____mm * mm.transpose() ## Note: Elementwise multiplication!_____no_output_____mm // mm.transpose()/usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py:1: RuntimeWarning: divide by zero encountered in floor_divide
  """Entry point for launching an IPython kernel.

Note that we used integer division `//`. Note also the warning you get(because of zero division). So let's avoid the divide by zero:_____no_output_____

mm // (mm+1).transpose()_____no_output_____mm * sc.pi_____no_output_____mm.dot(mm) # MATRIX MULTIPLICATION_____no_output_____mm = sc.matrix(mm) # convert to scipy matrix class
mm_____no_output_____print(type(mm))<class 'numpy.matrixlib.defmatrix.matrix'>
mm * mm # now matrix multiplication is syntactically easier_____no_output_____

We can do a lot more by importing the `linalg` sub-package: `sc.linalg`. Try it._____no_output_____## Two particularly useful `scipy` sub-packages

Two particularly useful `scipy` sub-packages are `sc.integrate` (*what will I need this for?*) and `sc.stats`. *Why not use `R` for this?* — because often you might just want to calculate some summary stats of your simulation results within Python.

### `sc.stats`

Let's take a quick spin in `sc.stats`._____no_output_____

import scipy.stats_____no_output_____scipy.stats.norm.rvs(size = 10) # 10 samples from N(0,1)_____no_output_____scipy.stats.randint.rvs(0, 10, size =7) # Random integers between 0 and 10_____no_output_____

### Numerical integration using  `scipy`

Numerical integration is the approximate computation of an integral using numerical techniques. You need numerical integration whenever you have a complicated function that cannot be integrated analytically using anti-derivatives. For example, calculating the area under a curve A particularly useful application is solving ordinary differential equations (ODEs), commonly used for modelling biological systems.

#### The Lotka-Volterra model

Let's try numerical integration in Python for solving a classical model in biology — the Lotka-Volterra model for a predator-prey system in two-dimensional space (e.g., on land).

The Lotka-Volterra (LV) model is:

\begin{aligned}
    \frac{dR}{dt} &= r R - a C R \\
    \frac{dC}{dt} &= - z C + e a C R
\end{aligned}

where $C$ and $R$ are consumer (e.g., predator) and resource (e.g., prey) population abundance (either number $\times$ area$^{-1}$ ), $r$ is the intrinsic (per-capita) growth rate of the resource population (time$^{-1}$), $a$ is per-capita "search rate" for the resource ($\text{area}\times \text{time}^{-1}$)  multiplied by its attack success probability, which determines the encounter and consumption rate of the consumer on the resource, $z$ is mortality rate ($\text{time}^{-1}$) and $e$ is the consumer's efficiency (a fraction) in converting resource to consumer biomass.

We have already imported scipy above (`import scipy as sc`) so we can proceed to solve the LV model using numerical integration.

First, import `scipy`'s `integrate` submodule:_____no_output_____

import scipy.integrate as integrate_____no_output_____

Now define a function that returns the growth rate of consumer and resource population at any given time step._____no_output_____

def dCR_dt(pops, t=0):

    R = pops[0]
    C = pops[1]
    dRdt = r * R - a * R * C
    dCdt = -z * C + e * a * R * C

    return sc.array([dRdt, dCdt])_____no_output_____type(dCR_dt)_____no_output_____

Assign some parameter values:_____no_output_____

r = 1.
a = 0.1
z = 1.5
e = 0.75_____no_output_____

Define the time vector; let's integrate from time point 0 to 15, using 1000 sub-divisions of time:_____no_output_____

t = sc.linspace(0, 15,  1000)_____no_output_____

Note that the units of time are arbitrary here. _____no_output_____Set the initial conditions for the two populations (10 resources and 5 consumers per unit area), and convert the two into an array (because our `dCR_dt` function take an array as input). _____no_output_____

R0 = 10
C0 = 5
RC0 = sc.array([R0, C0]) _____no_output_____

Now numerically integrate this system forward from those starting conditions: _____no_output_____

pops, infodict = integrate.odeint(dCR_dt, RC0, t, full_output=True)_____no_output_____pops_____no_output_____

So `pops` contains the result (the population trajectories).  Also check what's in infodict (it's a  dictionary with additional information)_____no_output_____

type(infodict)_____no_output_____infodict.keys()_____no_output_____

Check what the `infodict` output is by reading the help documentation with `?scipy.integrate.odeint`. For example, you can return a message to screen about whether the integration was successful: _____no_output_____

infodict['message']_____no_output_____

So it worked, great! But we would like to visualize the results. Let's do it using the `matplotlib` package.

## Plotting in Python

To visualize the results of your numerical simulations in Python (or for data exploration/analyses), you can use `matplotlib`, which uses Matlab like plotting syntax.

First let's import the package:_____no_output_____

import matplotlib.pylab as p_____no_output_____

Now open empty figure object (analogous to an R graphics object)._____no_output_____

f1 = p.figure()_____no_output_____p.plot(t, pops[:,0], 'g-', label='Resource density') # Plot
p.plot(t, pops[:,1]  , 'b-', label='Consumer density')
p.grid()
p.legend(loc='best')
p.xlabel('Time')
p.ylabel('Population density')
p.title('Consumer-Resource population dynamics')
p.show()# To display the figure_____no_output_____

Finally, save the figure as a pdf:_____no_output_____

f1.savefig('../results/LV_model.pdf') #Save figure_____no_output_____

You can use many any other output formats; check the documentation with `p.savefig`. _____no_output_____## Practicals

1. Create a self-standing script using the above example and save it as `LV1.py` in your code directory. In addition to generating the above figure, it should also generate the following figure:

            Generate this figure as part of the `LV1.py` script.

It should save both figures in pdf to the `results` directory, *without displaying them on screen*._____no_output_____## The need for speed: profiling code

Donald Knuth says: *Premature optimization is the root of all evil*.

Indeed, computational speed may not be your initial concern. Also, you should focus on developing clean, reliable, reusable code rather than worrying first about how fast your code runs. However, speed will become an issue when and if your analysis or modeling becomes complex enough (e.g., food web or large network simulations). In that case, knowing which parts of your code take the most time is useful – optimizing those parts may save you lots of time.

To find out what is slowing down your code you need to "profile" your code: locate the sections of your code where  speed bottlenecks exist.

### Profiling in Python

Profiling is easy in `ipython` – simply use the command:

```python
%run -p your_function_name
```

Let's write an illustrative program (name it `profileme.py`):
```python
def my_squares(iters):
    out = []
    for i in range(iters):
        out.append(i ** 2)
    return out

def my_join(iters, string):
    out = ''
    for i in range(iters):
        out += string.join(", ")
    return out

def run_my_funcs(x,y):
    print(x,y)
    my_squares(x)
    my_join(x,y)
    return 0

run_my_funcs(10000000,"My string")
```
Now run it with `run -p profileme.py`, and you should see something like:

```bash
        20000063 function calls (20000062 primitive calls) in 9.026 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    3.335    3.335    3.732    3.732 profileme.py:1(my_squares)
        1    2.698    2.698    5.200    5.200 profileme.py:7(my_join)
 10000001    2.502    0.000    2.502    0.000 {method 'join' of 'str' objects}
 10000008    0.397    0.000    0.397    0.000 {method 'append' of 'list' objects}
        1    0.093    0.093    9.025    9.025 profileme.py:13(run_my_funcs)
[more output]
```_____no_output_____Now you can see that the `my_join` function  is hogging most of the time, followed by `my_squares`. Furthermore, its the string method `join` that is clearly slowing `my_join` down, and the list method `append` that is slowing `my_squares`down. In other words, `.join`ing the string again and again, and `.append`ing values to a list are both not particularly fast.

Can we do better? *Yes!*

Let's try this alternative approach to writing the program (save it as `profileme2.py`):

```python
def my_squares(iters):
    out = [i ** 2 for i in range(iters)]
    return out

def my_join(iters, string):
    out = ''
    for i in range(iters):
        out += ", " + string
    return out

def run_my_funcs(x,y):
    print(x,y)
    my_squares(x)
    my_join(x,y)
    return 0

run_my_funcs(10000000,"My string")
```

We did two things: converted the loop to a list comprehension, and replaced the `.join` with an explicit string concatenation.

Now profile this program (`run -p profileme2.py`), and you should get something like:

```bash
        64 function calls (63 primitive calls) in 4.585 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    2.497    2.497    2.497    2.497 profileme2.py:2(<listcomp>)
        1    1.993    1.993    1.993    1.993 profileme2.py:5(my_join)
        1    0.094    0.094    4.584    4.584 profileme2.py:11(run_my_funcs)
[more output]
```

Woo hoo! So we about halved the time! Not quite enough to grab a pint, but ah well...

Another approach would be to preallocate a `numpy` array instead of using a list for `my_squares`. Try it._____no_output_____### Quick profiling with `timeit`

Alternatively,  you can use the `timeit` module if you want to figure out what the best way to do something specific as part of a larger program (say a particular command or a loop) might be.

Type an run the following code in a python script called `timeitme.py`:

```python
##############################################################################
# loops vs. list comprehensions: which is faster?
##############################################################################

iters = 1000000

import timeit

from profileme import my_squares as my_squares_loops

from profileme2 import my_squares as my_squares_lc

# %timeit my_squares_loops(iters)
# %timeit my_squares_lc(iters)

##############################################################################
# loops vs. the join method for strings: which is faster?
##############################################################################

mystring = "my string"

from profileme import my_join as my_join_join

from profileme2 import my_join as my_join

# %timeit(my_join_join(iters, mystring))
# %timeit(my_join(iters, mystring))
```
Note how we imported the functions using `from profileme import my_squares as my_squares_loops`, etc., which highlights the convenience of Python's elegant object-oriented approach.

Now run the two sets of comparisons using `timeit()` in ipython and make sure every line makes sense. Note that I have commented out the `%timeit()` commands because having a magic command inside a script will not work.

Of course, a simple approach would have been to time the functions like this:

```python
import time
start = time.time()
my_squares_loops(iters)
print("my_squares_loops takes %f s to run." % (time.time() - start))

start = time.time()
my_squares_lc(iters)
print("my_squares_lc takes %f s to run." % (time.time() - start))
```
But you'll notice that if you run it multiple times, the time taken changes each time. So `timeit` takes a sample of runs and returns the average, which is better.

*But remember, don't go crazy with profiling for the sake of shaving a couple of milliseconds, tempting as that may be!*_____no_output_____## Practicals

### Lotka-Volterra model problem

Copy and modify `LV1.py` into another script called `LV2.py` that does the following:

*  Take arguments for the four LV model parameters $r$, $a$, $z$ ,$e$ from the command line:
```
LV2.py arg1 arg2 ... etc
```
*  Runs the Lotka-Volterra model with prey density dependence $r R \left(1 - \frac{R} {K}\right)$, which changes the coupled ODEs to,
\begin{aligned}
    \frac{dR}{dt} &= r R \left(1 - \frac{R} {K}\right) - a C R \\
    \frac{dC}{dt} &= - z C + e a C R
\end{aligned}
*  Saves the plot as `.pdf` in an appropriate location.
*  The chosen parameter values should show in the plot (e.g., $r = 1, a = .5 $, etc) You can change time length $t$ too.
*  Include a script in `code` that will run both `LV1.py` and `LV2.py` with appropriate arguments. This script should also profile the two scripts and print the results to screen for each of the scripts using the `%run -p` approach. Look at and compare the speed bottlenecks in `LV1.py` and `LV2.py`. *Think about how you could further speed up the scripts.*

### Extra Credit problems

*Write every subsequent extra credit script file with a new name such as `LV3.py`,`LV4.py`, etc.*

* **Extra credit**: Choose appropriate values for the parameters such that both predator and prey persist with prey density dependence — the final (non-zero) population values should be printed to screen.

* **Extra-extra credit**: Write a discrete-time version of the LV model called `LV3.py`. The discrete-time model is:

\begin{align}
    R_{t+1} &= R_t (1 + r \left(1 - \frac{R_t}{K}\right) - a C_t)\\
    C_{t+1} &= C_t (1 - z + e a R_t)
\end{align}

Include this script in `run_LV.py`, and profile it as well.

* **Extra-extra-extra credit**: Write a version of the discrete-time model (which you implemented in `LV3.py`) simulation with a random gaussian fluctuation in resource's growth rate at each time-step:

\begin{aligned}
        R_{t+1} &= R_t (1 + (r + \epsilon) \left(1 - \frac{R_t}{K}\right)- a C_t)\\
        C_{t+1} &= C_t (1 - z + e a R_t)
\end{aligned}

where $\epsilon$ is a random fluctuation drawn from a gaussian distribution (use `sc.stats`). Include this
script in ` run_LV.py`, and profile it as well. You can also add fluctuations to both populations simultaneously this way:

\begin{aligned}
    R_{t+1} &= R_t (1 + \epsilon + r +  \left(1 - \frac{R_t}{K}\right) - a C_t)\\
    C_{t+1} &= C_t (1 - z + \epsilon + e a R_t)
\end{aligned}

*As always, test, add, commit and push all your new code and data to your git repository.*_____no_output_____## Networks in Python

ALL biological systems have a network representation, consisting of nodes for the biological entities of interest, and edges or links for the relationships between them. Here are some examples:
* Metabolic networks
* Gene regulatory networks
* Individual-Individual (e.g., social networks)
* Who-eats-whom (Food web) networks
* Mutualistic (e.g., plant-pollinator) networks

*Can you think of a few more examples from biology?*

You can easily simulate, analyze, and visualize biological networks in both `python` and `R` using some nifty packages. A full network analysis tutorial is out of the scope of our Python module's objectives, but let's try a simple visualization using the ` networkx` python package.

For this you need to first install the package, for example, by using:

```bash
sudo apt-get install python3-networkx
```

### Food web network example

As an example, let's plot a food web network.

The best way to store a food web dataset is as an "adjacency list" of who eats whom: a matrix with consumer name/id in 1st column, and resource name/id in 2nd column, and a separate matrix of species names/ids and properties such as biomass (node's abundance), or average body mass. You will see what these data structures look like below.

First, import the necessary modules:_____no_output_____

import networkx as nx
import scipy as sc
import matplotlib.pyplot as p_____no_output_____

Let's generate a "synthetic" food web. We can do this with the following function that generates a random adjacency list of a $N$-species food web with "connectance probability" $C$: the probability of having a link between any pair of species in the food web. _____no_output_____

def GenRdmAdjList(N = 2, C = 0.5):
    """
    """
    Ids = range(N)
    ALst = []
    for i in Ids:
        if sc.random.uniform(0,1,1) < C:
            Lnk = sc.random.choice(Ids,2).tolist()
            if Lnk[0] != Lnk[1]: #avoid self (e.g., cannibalistic) loops
                ALst.append(Lnk)
    return ALst_____no_output_____

Note that we are using a uniform random distribution between `[0,1]` to generate a connectance probability between each species pair.

Now, let's assign a body mass range (in log scale):_____no_output_____Now assign number of species (`MaxN`) and connectance (`C`):_____no_output_____

MaxN = 30
C = 0.75_____no_output_____

Now generate an adjacency list representing a random food web:_____no_output_____

AdjL = sc.array(GenRdmAdjList(MaxN, C))
AdjL_____no_output_____

So that's what an adjacency list looks like. The two columns of numbers correspond to the consumer and resource ids, respectively.

Now generate species (node) data:_____no_output_____

Sps = sc.unique(AdjL) # get species ids_____no_output_____

Now generate body sizes for the species. We will use a log$_{10}$ scale because species body sizes tend to be [log-normally distributed](08-Data_R.ipynb#Histograms)._____no_output_____

SizRan = ([-10,10]) #use log10 scale
Sizs = sc.random.uniform(SizRan[0],SizRan[1],MaxN)
Sizs_____no_output_____

Let's visualize the size distribution we have generated._____no_output_____

p.hist(Sizs) #log10 scale_____no_output_____p.hist(10 ** Sizs) #raw scale_____no_output_____

Now let's plot the network, with node sizes proportional to (log) body size:_____no_output_____

p.close('all') # close all open plot objects_____no_output_____

Let's use a circular configuration. For this, we need to calculate the coordinates, easily done using networkx:_____no_output_____

pos = nx.circular_layout(Sps)_____no_output_____

See `networkx.layout` for inbuilt functions to compute other types of node coordinates.

Now generate a networkx graph object:_____no_output_____

G = nx.Graph()_____no_output_____

Now add the nodes and links (edges) to it:_____no_output_____

G.add_nodes_from(Sps)
G.add_edges_from(tuple(AdjL)) # this function needs a tuple input_____no_output_____

Generate node sizes that are proportional to (log) body sizes:_____no_output_____

NodSizs= 1000 * (Sizs-min(Sizs))/(max(Sizs)-min(Sizs)) _____no_output_____

Now render (plot) the graph:_____no_output_____

nx.draw_networkx(G, pos, node_size = NodSizs)/usr/lib/python3/dist-packages/networkx/drawing/nx_pylab.py:522: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
  if not cb.is_string_like(edge_color) \
/usr/lib/python3/dist-packages/networkx/drawing/nx_pylab.py:543: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
  if cb.is_string_like(edge_color) or len(edge_color) == 1:
/usr/lib/python3/dist-packages/networkx/drawing/nx_pylab.py:724: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
  if not cb.is_string_like(label):

Some of you might get the warning above, or a different one. In that case, just try upgrading the networkx package.

## Practicals

1. Type the above code for plotting a food web network in a program file called `DrawFW.py`. This file should save the plotted network as a pdf.

2. (**Extra Credit**) You can also do nice network visualizations in R. Here you will convert a network visualization script written in `R` using the `igraph` package to a python script that does the same thing.

  * First copy the script file called `Nets.R` and the data files it calls and run it. This script visualizes the [QMEE CDT collaboration network](http://www.imperial.ac.uk/qmee-cdt), coloring the the nodes by the type of node (organization type: "University","Hosting Partner", "Non-hosting Partner").

 * Now, convert this script to a `python` script that does the same thing, including writing to an `.svg` file using the same QMEE CDT link and node data. You can use `networkx` or some other python network visualization package._____no_output_____## Regular expressions in Python

Let's shift gears now, and look at a very important skill that you should learn, or at least be aware of — *Regular expressions*.

Regular expressions (regex) are a tool to find patterns (not just a particular sequence of characters) in strings. For example, `your@email.com` is a specific sequence of characters, but, in fact, all email addresses have such a pattern: alphanumeric characters, a "@", alphanumeric characters, a ".", alphanumeric characters. Using regex, you can search for all email addresses in a text file by searching for this pattern.

There are many uses of regex, such as:

* Parsing (reading) text files and finding and replacing or deleting specific patterns
* Finding DNA motifs in sequence data
* Navigating through files in a directory
* Extracting information from html and xml files

Thus, if you are interested in data mining, need to clean or process data in any other way, or convert a bunch of information into usable data, knowing regex is absolutely necessary.

            (Source: [www.xkcd.com](https://www.xkcd.com/208/))

            Regular expressions could change your life!

Regex packages are available for most programming languages (recall [`grep` in UNIX](01-Unix.ipynb#Using-`grep`); that is how regex first became popular).

### Metacharacters vs. regular characters

A regex may consist of a combination of "metacharacters" (modifiers) and "regular" or literal characters. There are 14 metacharacters:

[ ] { } ( ) \ ^ $ . | ? * +

These metacharacters do special things, for example:

* `[12]` means match target to *1* and if that does not match then match target to *2*
* `[0-9]` means match to any character in range *0* to *9*
* `[^Ff]` means anything except upper or lower case *f* and `[^a-z]` means everything except lower case *a* to *z*

Everything else is interpreted literally (e.g., *a* is matched by entering `a` in the regex).

`[` and `]`, specify a character "class" — the set of characters that you wish to match. Metacharacters are not active inside classes. For example, [a-z$] will match any of the characters `a` to `z`, but also $, because inside a character class it loses its special metacharacter status.

### regex elements

A useful (not exhaustive) list of regex elements is:

|Regex|Description|
|:-|:-|
|\ | inhibit the "specialness" of a (meta)character so that it can be interpreted literally. So, for example, use `\.` to match a period or `\\` to match a slash|
|`aX9`| match the character string *aX9* exactly (case sensitively)|
|`8`| match the number *8*|
|`\n`| match a newline|
|`\t`| match a tab |
|`\s`| match a whitespace |
|`.`| match any character except line break (newline)|
|`\w`|  match a single "word" character: any alphanumeric character (including underscore)|
|`\W`| match any character not covered by `\w`, i.e., match any non-alphanumeric character excluding underscore, such as `?`, `!`, `+`, `<`, etc. |
|`\d`| match a numeric (integer) character|
|`\D`| match any character not covered by ` \d` (i.e., match a non-digit)|
|`[atgc]` | match any character listed: `a`, `t`, `g`, `c`|
| at|gc  | match `at` or `gc`|
|`[^atgc]`| match any character not listed: any character except `a`, `t`, `g`, `c`|
|`?`| match the preceding pattern element zero or one times|
|*|match the preceding pattern element zero or more times|
|`+`| match the preceding pattern element one or more times|
|`{n}`| match the preceding pattern element exactly `n` times|
|`{n,}`| match the preceding pattern element at least `n` times|
|`{n,m}`| match the preceding pattern element at least `n` but not more than `m` times|
|`^`| match the start of a string|
|$| match the end of a string|

### Regex in `python`

Regex functions in python are in the module `re`.

Let's import it:_____no_output_____

import re_____no_output_____

The simplest `python` regex function is `re.search`, which searches the string for match to a given pattern — returns a *match object* if a match is found and `None` if not. Thus, the command `match = re.search(pat, str)` finds matches of the pattern `pat` in the given string `str` and stores the search result in a variable named `match`.

> **Always** put `r` in front of your regex — it tells python to read the regex in its "raw" (literal) form. Without raw string notation (`r"text"`), every backslash (`\`) in a regular expression would have to be prefixed with another one to escape it. Read more about this [here](https://docs.python.org/3.5/library/re.html).

OK, let's try some regexes (type all that follows in `regexs.py`):_____no_output_____

my_string = "a given string"_____no_output_____

Find a space in the string:_____no_output_____

match = re.search(r'\s', my_string)
print(match)<_sre.SRE_Match object; span=(1, 2), match=' '>

That's only telling you that a match was found (the object was created successfully).

To see the match, use:_____no_output_____

match.group()_____no_output_____

Now let's try another pattern:_____no_output_____

match = re.search(r'\d', my_string)_____no_output_____print(match)None

No surprise, because there are no numeric characters in our string!_____no_output_____To know whether a pattern was matched, we can use an `if`:_____no_output_____

MyStr = 'an example'

match = re.search(r'\w*\s', MyStr) # what pattern is this?

if match:
    print('found a match:', match.group())
else:
    print('did not find a match')    found a match: an

Here are some more regexes (add all that follows to `regexs.py`):_____no_output_____

match = re.search(r'2' , "it takes 2 to tango")
match.group()_____no_output_____match = re.search(r'\d' , "it takes 2 to tango")
match.group()_____no_output_____match = re.search(r'\d.*' , "it takes 2 to tango")
match.group()_____no_output_____match = re.search(r'\s\w{1,3}\s', 'once upon a time')
match.group()_____no_output_____match = re.search(r'\s\w*$', 'once upon a time')
match.group()_____no_output_____

Let's switch to a more compact syntax by directly returning the matched group (by directly appending `.group()` to the result)._____no_output_____

re.search(r'\w*\s\d.*\d', 'take 2 grams of H2O').group()_____no_output_____re.search(r'^\w*.*\s', 'once upon a time').group() # 'once upon a '_____no_output_____

 Note that *, `+`, and `{ }` are all "greedy":  They repeat the previous regex token as many times as possible.

As a result, they may match more text than you want. To make it non-greedy and terminate at the first found instance of a pattern, use `?`:_____no_output_____

re.search(r'^\w*.*?\s', 'once upon a time').group()_____no_output_____

To further illustrate greediness in regexes, let's try matching an HTML tag:_____no_output_____

re.search(r'<.+>', 'This is a first test').group()_____no_output_____

But we wanted just `<EM>`!

It's because `+` is greedy. Instead, we can make `+` "lazy":_____no_output_____

re.search(r'<.+?>', 'This is a first test').group()_____no_output_____

OK, moving on from greed and laziness..._____no_output_____

re.search(r'\d*\.?\d*','1432.75+60.22i').group()_____no_output_____

Note `\` before the `.`, to be able to find a literal `.`

Otherwise, `re.search` will consider it to be a regex element (`.` means "match any character except newline").

A couple more examples:_____no_output_____

re.search(r'[AGTC]+', 'the sequence ATTCGT').group()_____no_output_____re.search(r'\s+[A-Z]\w+\s*\w+', "The bird-shit frog's name is Theloderma asper.").group()_____no_output_____

            In case you were wondering what *Theloderma asper*, the "bird-shit frog", looks like. I snapped this one in North-east India ages ago

_____no_output_____How about looking for email addresses in a string? For example, let's try matching a string consisting of an academic's name, email address and research area or interest (no need to type this into any python file):_____no_output_____

MyStr = 'Samraat Pawar, s.pawar@imperial.ac.uk, Systems biology and ecological theory'
match = re.search(r"[\w\s]+,\s[\w\.@]+,\s[\w\s]+",MyStr)
match.group()_____no_output_____

Note the use of `[ ]`'s: for example, `[\w\s]` ensures that any combination of word characters and spaces is found.

Let's see if this regex works on a different pattern of email addresses: _____no_output_____

MyStr = 'Samraat Pawar, s-pawar@imperial.ac.uk, Systems biology and ecological theory'_____no_output_____match = re.search(r"[\w\s]+,\s[\w\.@]+,\s[\w\s&]+",MyStr)
match.group()_____no_output_____

Nope! So let's make the email address part of the regex more robust:_____no_output_____

match = re.search(r"[\w\s]+,\s[\w\.-]+@[\w\.-]+,\s[\w\s&]+",MyStr)
match.group()_____no_output_____

## Practicals: Some RegExercises

The following exercises are not for submission as part of your coursework, but we will discuss them in class on a subsequent day.

1. Try the regex we used above for finding names (`[\w\s]+`) for cases where the person's name has something unexpected, like a `?` or a `+`. Does it work? How can you make it more robust?
* Translate the following regular expressions into regular English:
    * `r'^abc[ab]+\s\t\d'`
    * `r'^\d{1,2}\/\d{1,2}\/\d{4}$'`
    * `r'\s*[a-zA-Z,\s]+\s*'`
* Write a regex to match dates in format YYYYMMDD, making sure that:
    * Only seemingly valid dates match (i.e., year greater than 1900)
    * First digit in month is either 0 or 1
    * First digit in day $\leq 3$ _____no_output_____### Grouping regex patterns

You can group regex patterns into meaningful blocks using parentheses. Let's look again at the example of finding  email addresses._____no_output_____

MyStr = 'Samraat Pawar, s.pawar@imperial.ac.uk, Systems biology and ecological theory'
match = re.search(r"[\w\s]+,\s[\w\.-]+@[\w\.-]+,\s[\w\s&]+",MyStr)
match.group()_____no_output_____

Without grouping the regex:_____no_output_____

match.group(0)_____no_output_____

Now create groups using `( )`:_____no_output_____

match = re.search(r"([\w\s]+),\s([\w\.-]+@[\w\.-]+),\s([\w\s&]+)",MyStr)
if match:
    print(match.group(0))
    print(match.group(1))
    print(match.group(2))
    print(match.group(3))Samraat Pawar, s.pawar@imperial.ac.uk, Systems biology and ecological theory
Samraat Pawar
s.pawar@imperial.ac.uk
Systems biology and ecological theory

Nice! This is very handy for extracting specific patterns from text data. Note that we excluded the `,`'s and the `\s`'s from the grouping parentheses because we don't want them to be returned in the match group list.

Have a look at `re4.py` in the TheMulQuaBio's code repository for more on parsing email addresses using regexes._____no_output_____## Useful `re` commands

Here are some important functions in the `re` module:

|Command|What it does|
|:-|:-|
| `re.search(reg, text)`| Scans the string and finds the first match of the pattern, returning a `match` object if successful and `None` otherwise.|
| `re.match(reg, text)`| Like `re.search`, but only matches the beginning of the string.|
| `re.compile(reg)`| Compiles (stores) a regular expression for repeated use, improving efficiency.|
| `re.split(ref, text)`| Splits the text by the occurrence of the pattern described by the regular expression.|
| `re.findall(ref, text)`| Like `re.search`, but returns a list of all matches. If groups are present, returns a list of groups.|
| `re.finditer(ref, text)`| Like `re.findall`, but returns an iterator containing the match objects over which you can iterate. Useful for "crawling" efficiently through text till you find all necessary number of matches.|
| `re.sub(ref, repl, text)`|  Substitutes each non-overlapping occurrence of the match with the text in `repl`.|
|||

Many of these commands also work on whole contents of files. We will look at an example of this below. Let us try some are particularly useful applications of some of these commands.

### Finding all matches

Above we used re.search() to find the first match for a pattern. In many scenarios, you will need to find *all* the matches of a pattern. The function `re.findall()` does precisely this and returns all matches as a list of strings, with each string representing one match.

Let's try this on an extension of the email example above for some data with multiple addresses: _____no_output_____

MyStr = "Samraat Pawar, s.pawar@imperial.ac.uk, Systems biology and ecological theory; Another academic, a-academic@imperial.ac.uk, Some other stuff thats equally boring; Yet another academic, y.a_academic@imperial.ac.uk, Some other stuff thats even more boring"_____no_output_____

Now `re.findall()` returns a list of all the emails found:_____no_output_____

emails = re.findall(r'[\w\.-]+@[\w\.-]+', MyStr)
for email in emails:
    print(email)s.pawar@imperial.ac.uk
a-academic@imperial.ac.uk
y.a_academic@imperial.ac.uk

Nice! _____no_output_____### Finding in files

You will generally be wanting to apply regex searches to whole files. You might be tempted to write a loop to iterate over the lines of the file, calling `re.findall()` on each line. However, `re.findall()` can return a list of all the matches in a single step.

Let's try finding all species names that correspond to Oaks in a data file:_____no_output_____

f = open('../data/TestOaksData.csv', 'r')
found_oaks = re.findall(r"Q[\w\s].*\s", f.read())

found_oaks_____no_output_____for name in strings:
    print(name.replace(",",""))_____no_output_____

This works because recall that `f.read()` returns the whole text of a file in a single string). Also, the file is closed after reading._____no_output_____### Groups within multiple matches

Grouping pattern matches using `( )` as you learned above, can be combined with `re.findall()`. If the pattern includes *two or more* groups, then instead of returning a list of strings, `re.findall()` returns a list of tuples. Each tuple represents one match of the pattern, and inside the tuple is group(1), group(2), etc.

Let's try it:_____no_output_____

MyStr = "Samraat Pawar, s.pawar@imperial.ac.uk, Systems biology and ecological theory; Another academic, a.academic@imperial.ac.uk, Some other stuff thats equally boring; Yet another academic, y.a.academic@imperial.ac.uk, Some other stuff thats even more boring"

found_matches = re.findall(r"([\w\s]+),\s([\w\.-]+@[\w\.-]+)", MyStr)
found_matches_____no_output_____for item in found_matches:
    print(item)('Samraat Pawar', 's.pawar@imperial.ac.uk')
(' Another academic', 'a.academic@imperial.ac.uk')
(' Yet another academic', 'y.a.academic@imperial.ac.uk')

### Extracting text from webpages

OK, let's step up the ante here. How about extracting text from a web page to create your own data? Let's try extracting data from [this page](https://www.imperial.ac.uk/silwood-park/academic-staff/).

You will need a new package `urllib3`. Install it, and import it (also `import re` if needed). _____no_output_____

import urllib3_____no_output_____conn = urllib3.PoolManager() # open a connection
r = conn.request('GET', 'https://www.imperial.ac.uk/silwood-park/academic-staff/')
webpage_html = r.data #read in the webpage's contents_____no_output_____

This is returned as bytes (not strings). _____no_output_____

type(webpage_html)_____no_output_____

So decode it (remember, the default decoding that this method applies is *utf-8*):_____no_output_____

My_Data  = webpage_html.decode()
#print(My_Data)_____no_output_____

That's a lot of potentially useful information! Let's extract all the names of academics:_____no_output_____

pattern = r"Dr\s+\w+\s+\w+"
regex = re.compile(pattern) # example use of re.compile(); you can also ignore case  with re.IGNORECASE
for match in regex.finditer(My_Data): # example use of re.finditer()
    print(match.group())Dr Arkhat Abzhanov
Dr Arkhat Abzhanov
Dr Cristina Banks
Dr Tom Bell
Dr Martin Bidartondo
Dr Martin Bidartondo
Dr Martin Brazeau
Dr Lauren Cator
Dr Matteo Fumagalli
Dr Matteo Fumagalli
Dr Richard Gill
Dr Richard Gill
Dr Jason Hodgson
Dr Andrew Knight
Dr Andrew Knight
Dr Morena Mills
Dr Morena Mills
Dr Samraat Pawar
Dr Julia Schroeder
Dr Julia Schroeder
Dr Joseph Tobias
Dr Joseph Tobias
Dr Mike Tristem
Dr Mike Tristem
Dr Magda Charalambous
Dr Magda Charalambous
Dr Rebecca Kordas
Dr Rebecca Kordas
Dr Vassiliki Koufopanou
Dr Eoin O
Dr Eoin O
Dr David Orme
Dr James Rosindell
Dr Chris Wilson
Dr Oliver Windram
Dr Colin Clubbe
Dr George McGavin
Dr George McGavin
Dr Michael Themis
Dr Michael Themis

Again, nice! However, its' not perfect. You can improve this by:
* Extracting Prof names as well
* Eliminating the repeated matches
* Grouping to separate title from first and second names
* Extracting names that have unexpected characters (e.g., ``O'Gorman'', which are currently not being matched properly)

*Try making these improvements.*

Of course, you can match and extract other types of patterns as well, such as urls and email addresses (though this example web page does not have email addresses). _____no_output_____### Replacing text

Using the same web page data, let's try using the `re.sub` command on the same web page data (`My_Data`) to replace text:_____no_output_____

New_Data = re.sub(r'\t'," ", My_Data) # replace all tabs with a space
# print(New_Data)_____no_output_____

## Practicals

### Blackbirds problem

Complete the code `blackbirds.py` that you find in the `TheMulQuaBio` (necessary data file is also there).

*As always, test, add, commit and push all your new code and data to your git repository.*_____no_output_____## Using Python to build workflows

You can use python to build an automated data analysis or simulation workflow that involves multiple languages, especially the ones you have already learnt: R, $\LaTeX$, and UNIX bash. For example, you could, in theory, write a single Python script to generate and update your masters dissertation, tables, plots, and all. Python is ideal for building such workflows because it has packages for practically every purpose.

*Thus this topic may be useful for your [Miniproject](Appendix-MiniProj.ipynb), which will involve building a reproducible computational workflow.*

### Using `subprocess`

For building a workflow in Python the `subprocess` module is key. With this module you can run non-Python commands and scripts, obtain their outputs, and also crawl through and manipulate directories.

First, import the module (this is part of the python standard library, so you won't need to install it):_____no_output_____

import subprocess_____no_output_____

#### Running processes

There are two main ways to run commands through subprocess: `run` (available in Python 3.5 onwards) for basic usage, and `Popen` (`P`rocess `open`) for more advanced usage. We will work directly with `popen` because `run()` is a wrapper around `Popen`. Using `Popen` directly gives more control over how the command is run, and how its input and output are processed.

Let's try running some commands in the UNIX bash.

$\star$ In a terminal, first `cd` to your `code` directory,  launch `ipython3`, then and type:_____no_output_____

p = subprocess.Popen(["echo", "I'm talkin' to you, bash!"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)_____no_output_____

This creates an object `p`, from which you can extract the output and other information of the command you ran. Before we do anything more, let's look at our `subprocess.popen` call carefully.

* The command line arguments were passed as a list of strings, which avoids the need for escaping quotes or other special characters that might be interpreted by the shell (for example, in this case, there are apostrophes in the string that is being `echo`ed in bash).
* `stdout` is the output from the process "spawned" by your command. This is bytes sequence (which you will need to decode - more on this below).
* `stderr` is the error code (from which you can capture whether the process ran successfully or not). The method PIPE creates a new "pipe" to the "child process". _____no_output_____

stdout, stderr = p.communicate()_____no_output_____stderr_____no_output_____

Nothing here, because the echo command does no return an any code. the `b` indicates that the output is in bits (unencoded). By default, stdout, stderr (and other outputs of `p.communicate`) are returned as binary (byte) format.

Now check what's in `stdout`:_____no_output_____

stdout_____no_output_____

Let's encode and print it._____no_output_____

print(stdout.decode())I'm talkin' to you, bash!

You can also use a `universal_newlines = True` so that these outputs are returned as encoded text (default being *utf-8* usually), with  line endings converted to '\n'. For more information [see the documentation](https://docs.python.org/3.5/library/subprocess.html).

Let's try something else: _____no_output_____

p = subprocess.Popen(["ls", "-l"], stdout=subprocess.PIPE)
stdout, stderr = p.communicate()
# print(stdout.decode())_____no_output_____

Recall that the `ls -l` command lists all files in a long listing format.

You can also call python itself from bash (!):_____no_output_____

p = subprocess.Popen(["python", "boilerplate.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) # A bit silly!
stdout, stderr = p.communicate()
print(stdout.decode())This is a boilerplate

Similarly, to compile a $\LaTeX$ document (using `pdflatex` in this case), you can do something like:

```python
subprocess.os.system("pdflatex yourlatexdoc.tex")
```_____no_output_____You can also do this instead:_____no_output_____

p = subprocess.Popen(["python", "boilerplate.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) # A bit silly!
stdout, stderr = p.communicate()
print(stdout.decode())This is a boilerplate

### Handling directory and file paths

You can also use `subprocess.os` to make your code OS (Linux, Windows, Mac) independent. For example to assign paths:_____no_output_____

subprocess.os.path.join('directory', 'subdirectory', 'file')_____no_output_____

The result would be appropriately different on Windows (with backslashes instead of forward slashes).

Note that in all cases you can "catch" the output of `subprocess` so that you can then use the output within your
python script. A simple example, where the output is a platform-dependent directory path, is:_____no_output_____

MyPath = subprocess.os.path.join('directory', 'subdirectory', 'file')
MyPath_____no_output_____

Explore what `subprocess` can do by tabbing
`subprocess.`, and also for submodules, e.g., type
`subprocess.os.` and then tab.

### Running `R`

R is likely an important part of your project's analysis and data visualization components in particular — for example for statistical analyses and pretty plotting (`ggplot2`!).

You can run `R` from Python easily. Try the following:

$\star$ Create an R script file called `TestR.R` in your `code` directory with the following content:

```r
print("Hello, this is R!")
```_____no_output_____Now, create a script `TestR.py` with the following content :_____no_output_____

import subprocess
subprocess.Popen("Rscript --verbose TestR.R > ../Results/TestR.Rout 2> ../Results/TestR_errFile.Rout", shell=True).wait()_____no_output_____

Now run `TestR.py` (or `%cpaste`) and check`TestR.Rout` and `TestR_errorFile.Rout`.

Also check what happens if you run (type directly in `ipython` or `python` console):_____no_output_____

subprocess.Popen("Rscript --verbose NonExistScript.R > ../Results/outputFile.Rout 2> ../Results/errorFile.Rout", shell=True).wait()_____no_output_____

It is possible that the location of `RScript` is different in your Ubuntu install. To locate it, try `find /usr -name 'Rscript'` in the linux terminal (not in `python`!). For example, you might need to specify the path to it using `/usr/lib/R/bin/Rscript`.

What do you see on the screen? Now check `outputFile.Rout`and `errorFile.Rout._____no_output_____## Practicals

As always, test, add, commit and push all your new code and data to your git repository.

### Using `os` problem 1

Open `using_os.py` and complete the tasks assigned (hint: you might want to look at `subprocess.os.walk()`)

### Using `os` problem 2

Open `fmr.R` and work out what it does; check that you have `NagyEtAl1999.csv`. Now write python code called
`run_fmr_R.py` that:

Runs `fmr.R` to generate the desired result

`run_fmr_R.py` should also print to the python screen whether the run was successful, and the contents of the R console output

* `git add`, `commit` and `push` all    your week's code and data to your git repository by next Wednesday.*_____no_output_____Readings and Resources
----------------------

* [The matplotlib website](http://matplotlib.org)

* For SciPy, the [official documentation is good](https://docs.scipy.org/doc/); Read about the scipy modules you think will be important to you.

* The "ecosystem" for Scientific computing in python: <http://www.scipy-lectures.org/>

* A Primer on Scientific Programming with Python <http://www.springer.com/us/book/9783642549595>; Multiple copies of this book are available from the central library and can be requested to Silwood from the IC library website. You can also find a pdf - google it

* Many great examples of applications in the [scipy cookbook](https://lagunita.stanford.edu/courses/DB/2014/SelfPaced/about)

* For regex: <https://docs.python.org/2/howto/regex.html>

* Google's short class on regex in python: <https://developers.google.com/edu/python/regular-expressions>
And this exercise: https://developers.google.com/edu/python/exercises/baby-names

* <http://www.regular-expressions.info/> has a good intro, tips and a great array of canned solutions

* Use and abuse of regex: <https://blog.codinghorror.com/regex-use-vs-regex-abuse/>_____no_output_____


# Notebook from UWashington-Astro300/Astro300-A16
Path: FirstLast_HW5.ipynb

## First Last - Homework \#5_____no_output_____

<code>
%matplotlib inline

import numpy as np
import astropy.units as u
import matplotlib.pyplot as plt

from astroplan import Observer, FixedTarget
from astropy.time import Time
from astropy.coordinates import SkyCoord
from astroplan.plots import plot_sky, plot_airmass

import warnings
warnings.filterwarnings('ignore', category=Warning)_____no_output_____
</code>

## Planning observations_____no_output_____### I have given you data on three observatories and three celestial targets. Your job is to determine the best time of year and the best observatory to observe each of the three targets.

- The best time of year should be a two week period of time during the year 2017.
- Pick a single night durning this two week time period.
- A review of astronomical coordinates can be found [here](./Astro_Coordinates.pdf)_____no_output_____## 30 Doradus (NGC 2070)_____no_output_____## HD 209458_____no_output_____## M51_____no_output_____---

## Observatories

Mauna Kea Observatories (Mauna Kea, Hawaii)
- lat = 19.8244 deg
- lon = -155.4733 deg
- elevation = 4159 m

Siding Spring Observatory (Coonabarabran, Australia)
- lat = -31.2733 deg
- lon = 149.0644 deg
- elevation = 1165 m

Roque de los Muchachos Observatory (La Palma, Canary Islands)
- lat = 28.7636 deg
- lon = -17.8947 deg
- elevation = 2396 m

---

## Targets

- 30 Doradus (NGC 2070) - * HII Region *
- HD 209458 - * G-Type star with a transiting exoplanet *
- M51 - * Spiral Galaxy *

You can [SIMBAD](http://simbad.u-strasbg.fr/simbad/sim-fbasic) to find the `ICRS` (International Celestial Reference System) coordinates for the targets, or try the `FixedTarget.from_name()` function

---_____no_output_____### Due Tue Nov 01 - 5pm
- `Make sure to change the filename to your name!`
- `Make sure to change the Title to your name!`
- `File -> Download as -> HTML (.html)`
- `upload your .html and .ipynb file to the class Canvas page`  _____no_output_____