# Notebook from j-m-dean/uravu
Path: docs/source/nested_sampling.ipynb

# Nested sampling

In the [MCMC tutorial](./mcmc.html), Bayes theorem was shown,

$$ p = \frac{P L}{Z}, $$

where where, $p$ is the posterior probability, $P$ is the prior probability, $L$ is the likelihood, and $Z$ is the evidence.
Normally, in the evaluation of the posterior probability, the evidence is removed as a constant of proportionality.
This is due to the fact that it is a single value for a given model and dataset pair.

However, sometimes it is desirable to find the evidence for a model to a given dataset.
In particular, when we want to compare the evidence for a series of models, to determine which best describes the dataset.
In order to achieve this, `uravu` is able to perform [nested sampling](https://doi.org/10.1063/1.1835238) to estimate the Bayesian evidence for a model given a particular dataset.

This tutorial will show how this is achieved, and show the utility of Bayesian model selection.

However, as always we must first create some *synthetic* data. _____no_output_____

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import lognorm
from uravu.distribution import Distribution
from uravu.relationship import Relationship
from uravu import plotting, utils_____no_output_____np.random.seed(2)_____no_output_____

This synthetic data is log-normally distributed in the ordinate.
See [Using distributions](./using_distributions.html) tutorial to find out defining non-normally distributed data. _____no_output_____

x = np.linspace(10, 50, 20)
y = .3 * x ** 2 - 1.4 * x + .2
Y = []
for i in y:
    Y.append(Distribution(lognorm.rvs(s=2, loc=i, scale=1, size=1000)))_____no_output_____

From looking at the code used to synthesize this data, it is clear that the functional form of the model is a second order polynomial.
However, if this was data collected from some measurement and the physical theory suggests it should be analysed with a polynormial of unknown degree then Bayesian model selection would be the ideal tool to find the best model.

Let's quickly write a few models to perform the `n` degree polynomial analysis with. _____no_output_____

def one_degree(x, a, b):
    return b * x + a

def two_degree(x, a, b, c):
    return c * x ** 2 + b * x + a

def three_degree(x, a, b, c, d):
    return d * x ** 3 + c * x ** 2 + b * x + a

def four_degree(x, a, b, c, d, e):
    return e * x ** 4 + d * x ** 3 + c * x ** 2 + b * x + a_____no_output_____

With these functions defined, we can now build the `Relationship` objects for each of the functions. Using the `'diff_evo'` string to perform the global optimisation by [differential evolution](https://en.wikipedia.org/wiki/Differential_evolution). _____no_output_____

one_modeller = Relationship(one_degree, x, Y,
                            bounds=((-300, 0), (-2, 20)))
one_modeller.max_likelihood('diff_evo')_____no_output_____two_modeller = Relationship(two_degree, x, Y,
                            bounds=((-2, 2), (-2, 2), (-1, 1)))
two_modeller.max_likelihood('diff_evo')_____no_output_____three_modeller = Relationship(three_degree, x, Y,
                              bounds=((-2, 2), (-2, 2), (-1, 1), (-0.2, 0.2)))
three_modeller.max_likelihood('diff_evo')_____no_output_____four_modeller = Relationship(four_degree, x, Y,
                             bounds=((-2, 2), (-2, 2), (-1, 1), (-0.2, 0.2), (-0.02, 0.02)))
four_modeller.max_likelihood('diff_evo')_____no_output_____

Having built these, lets see what the maximum likelihood variables are for each. _____no_output_____

print(one_modeller.variable_modes)[-245.49075453   17.69708446]
print(two_modeller.variable_modes)[ 0.92673225 -1.28344677  0.29831624]
print(three_modeller.variable_modes)[ 1.12171456e+00 -1.50994256e+00  3.11896641e-01 -1.85985620e-04]
print(four_modeller.variable_modes)[ 3.50408477e-01 -9.58251997e-01  2.64924917e-01  1.09249940e-03
 -1.10893799e-05]

It is possible to quickly visualise the relationship with the `plotting` function, which shows the log-normal distribution of the ordinate data. _____no_output_____

plotting.plot_relationship(one_modeller)
plt.show()_____no_output_____

Above, in the `variable_modes`, we can see that the highest order terms are quite small for the larger degree of polynomial.
Let's see what effect this has on the evidence.

Note that `uravu` and `dynesty` calculates the natural log of the evidence, $\ln{Z}$ and while this tutorial show how [nested sampling](http://doi.org/10.1063/1.1835238) may be used, `uravu` and `dynesty` also enable the use of [dynamic nested sampling](https://doi.org/10.1007/s11222-018-9844-0) (by setting the keyword argument `dynamic` to `True`)._____no_output_____

one_modeller.nested_sampling()3426it [00:16, 208.10it/s, +500 | bound: 5 | nc: 1 | ncall: 21338 | eff(%): 18.399 | loglstar:   -inf < -123.515 <    inf | logz: -129.507 +/-  0.140 | dlogz:  0.001 >  0.509]
two_modeller.nested_sampling()3790it [00:24, 152.12it/s, +500 | bound: 7 | nc: 1 | ncall: 22819 | eff(%): 18.800 | loglstar:   -inf < -82.248 <    inf | logz: -88.982 +/-  0.150 | dlogz:  0.001 >  0.509]
three_modeller.nested_sampling()6147it [00:44, 138.76it/s, +500 | bound: 16 | nc: 1 | ncall: 30273 | eff(%): 21.957 | loglstar:   -inf < -84.756 <    inf | logz: -96.245 +/-  0.207 | dlogz:  0.001 >  0.509]
four_modeller.nested_sampling()9083it [01:08, 132.08it/s, +500 | bound: 29 | nc: 1 | ncall: 39979 | eff(%): 23.970 | loglstar:   -inf < -81.820 <    inf | logz: -99.205 +/-  0.261 | dlogz:  0.001 >  0.509]

Having estimated $\ln{Z}$ for each `Relationship`, lets plot them as a function of the number of variables in the model._____no_output_____

variables = [len(one_modeller.variables), len(two_modeller.variables),
             len(three_modeller.variables), len(four_modeller.variables)]
ln_evidence = [one_modeller.ln_evidence.n, two_modeller.ln_evidence.n,
               three_modeller.ln_evidence.n, four_modeller.ln_evidence.n]
ln_evidence_err = [one_modeller.ln_evidence.s, two_modeller.ln_evidence.s,
                   three_modeller.ln_evidence.s, four_modeller.ln_evidence.s]_____no_output_____plt.errorbar(variables, ln_evidence, ln_evidence_err, marker='o', ls='')
plt.xlabel('Number of variables')
plt.ylabel(r'$\ln{Z}$')
plt.show()_____no_output_____

We can see that the evidence reaches a maxiumum at 3 variables (the `two_degree` model), this indicates that this is the most probable model for analysis of this dataset.

Finally, we can use some built in functionality of `uravu` to compare different evidence values, in a value known as the [Bayes factor](https://doi.org/10.2307/2291091).
Let's compare the `one_degree` and `two_degree` models._____no_output_____

print(utils.bayes_factor(two_modeller.ln_evidence, one_modeller.ln_evidence))81.0+/-0.4

The Bayes factor between the two and one degree models, $2\ln{B_{21}}$, has a value of $\sim81$.
The Table in [Kass and Raftery](https://doi.org/10.2307/2291091) suggests that this shows "Strong" evidence for the model with more variables (the `two_degree` model).

This means that it is sensible to go ahead and use the `two_degree` model in the further analysis of our data.
Below the plot of the second order polynomial is shown over the data. _____no_output_____

plotting.plot_relationship(two_modeller)
plt.show()_____no_output_____


# Notebook from razofz/nabo
Path: docs/analysis/mll_enl/notebooks/1_preprocessing.ipynb

# Preprocessing_____no_output_____In this notebook we will pre-process the data. The main steps here are:

- Cell and gene filtering
- Cell-wise count normalization
- Variance correction and identification of highly variable genes (HVGs)
- Dimensionality reduction using PCA

We will use the [Dataset](../../../api.rst#dataset) class for these purposes._____no_output_____If you haven't done so already, download the [zip file](https://files.osf.io/v1/resources/7t3rc/providers/osfstorage/5b452fcd4e95590012affb23?action=download&version=1&view_only=338f2b3896fe48ab99f52cc54cd01653&direct), extract it and then change the working directory to the *notebooks* directory extracted from the zip file._____no_output_____

import nabo_____no_output_____

The `mtx_to_h5` function can easily convert [MTX format](https://math.nist.gov/MatrixMarket/formats.html) data (an output of 10x's [Cell Ranger](https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/what-is-cell-ranger) pipeline) into the [HDF5](https://www.hdfgroup.org/) format. The commands below read and convert the data for two sample visualizations, **WT** and **MLL-ENL**, into HDF5 format. Each command takes three parameters:

- path to the directory containing [three files](https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/output/matrices) (*matrix.mtx*, *genes.tsv* and *barcodes.tsv*)
- name of the output file
- `batch_size`. Larger values of `batch_size` will allow the data to be written to HDF5 format faster, but will also lead to larger memory consumption.

*Note: once the data is saved into HDF5 format, you don't have to run this command again, even if you want to re-run the rest of the notebook.*_____no_output_____

nabo.mtx_to_h5('../raw_data/WT', '../analysis_data/WT.h5', batch_size=1000)
nabo.mtx_to_h5('../raw_data/MLL_ENL', '../analysis_data/MLL_ENL.h5', batch_size=1000)Saving cell-wise data          : 100%| 00:00
Saving gene-wise data          : 100%| 00:00
Saving cell-wise data          : 100%| 00:00
Saving gene-wise data          : 100%| 00:00

In the cell below, the **WT** sample data is visualized using the [Dataset](../../../api.rst#dataset) class. The sample data is already saved in the HDF5 format. Basic statistics are visualized using the [plot_raw](../../../api.rst#nabo.Dataset.plot_raw) method of the returned instance of Dataset._____no_output_____

# use force_recalc=True to ensure that previously saved information is overwritten.
wt_data = nabo.Dataset('../analysis_data/WT.h5', force_recalc=True)
wt_data.plot_raw()

Next, poor quality cells and genes are removed by calling [filter_data](../../../api.rst#nabo.Dataset.filter_data) method with minimum and maximum threshold values for the following parameters:

- total expression values (can be total UMIs/counts per cell)
- number genes expressed per cell
- percentage of mitochondrial gene expression (ex. % UMI from mito genes) in each cell
- percentage of ribosomal gene expression (ex. % UMI from ribo genes) in each cell
- minimum number of cells where a gene should be expressed i.e. have non zero expression value.

The filtered data is visualized using [plot_filtered](../../../api.rst#nabo.Dataset.plot_filtered) method. The number of cells and genes filtered for each cutoff are also displayed._____no_output_____

wt_data.filter_data(min_exp=6000, max_exp=60000, min_ngenes=2000, max_ngenes=6500,
                    min_mito=0, max_mito=5, min_ribo=20, max_ribo=40)
wt_data.plot_filtered()

Other samples can be be loaded and filtered in a similar way. At this stage no distinction is made over which sample would be a reference and which would be a target sample._____no_output_____

me_data = nabo.Dataset('../analysis_data/MLL_ENL.h5', force_recalc=True)
me_data.plot_raw()
me_data.filter_data(min_exp=6000, max_exp=60000, min_ngenes=2000, max_ngenes=6500,
                        min_mito=0, max_mito=5, min_ribo=28, max_ribo=40)
me_data.plot_filtered()

The next step is normalization, which accounts for sequencing depth of each cell such that cross cell comparisons can be made. Normalization is performed over a dataset using the [set_sf](../../../api.rst#nabo.Dataset.set_sf) method. During this step Nabo simply saves the size factors for each cell, and will later use these size factors for normalization. By default, the size factor for each cell is calculated by summing the expression values of genes in a cell. However, users can override this behaviour by providing their own size factors. By default, the [set_sf](../../../api.rst#nabo.Dataset.set_sf) method will take only the filtered genes into account. This behaviour can be controlled by toggling the `all_genes` parameter to either `False` (default) or `True`.

This step is recommended even if data is pre-normalized._____no_output_____

wt_data.set_sf(size_scale=10000)
me_data.set_sf(size_scale=10000)

Next, we correct variance to remove any mean-variance trend. This step is essential before the HVGs (highly variable genes) can be found. This step can be skipped if you already have a list of HVGs. Please refer to the documentation on [correct_var](../../../api.rst#nabo.Dataset.correct_var) to learn more about this step._____no_output_____

wt_data.correct_var(n_bins=100, lowess_frac=0.5)

To identify HVGs, [find_hvgs](../../../api.rst#nabo.Dataset.find_hvgs) method is called with a cutoff for variance (`var_min_thresh`) and non-zero mean (`nzm_min_thresh`) in log scale. This method will also produce three subplots which are useful in order to to study variance correction and genes selected as HVGs. In all three plots the blue dots are the HVGs and grey dots are other genes.

- The third subplot shows the trend between log mean and log variance. The red dots (may look like a line because dots overlaps) show the Lowess fitted baseline.
- The middle subplot has corrected variance on the y axis and one may see that the mean-variance trend no longer exists or has diminished.
- The first plot on the left has log of corrected variance on the y axis and log non-zero mean on x axis; the vertical and horizontal lines represent the cutoffs used for marking HVGs._____no_output_____

wt_data.find_hvgs(var_min_thresh=1.5, nzm_min_thresh=-1, use_corrected_var=True)_____no_output_____

It is critical to observe here that variance correction and HVG identification was done only on one sample (`wt_data` in this case). This is because we aim to use cells from  this sample as [reference cells](../../../glossary.rst), while cells from the other sample will be regarded as [target cells](../../../glossary.rst) for downstream analysis.

So, **WT** are the *reference cells* and **MLL-ENL** are the *target cells*.

Next, we do PCA using the [fit_ipca](../../../api.rst#nabo.Dataset.fit_ipca) method._____no_output_____

wt_data.fit_ipca(wt_data.hvgList, n_comps=100)

Before we transform the values from reference and target into PCA space, it can be useful to visualize the variance explained by the PC components. The [plot_scree](../../../api.rst#nabo.Dataset.transform_pca) method generates two subplots. The topmost one is the usual scree plot, while the one at the bottom shows cumulative explained variance on the y-axis. [plot_scree](../../../api.rst#nabo.Dataset.transform_pca) takes three parameters:

- PCA object (`ipca` attribute of reference sample).
- Target variance: PC component where cumulative variance reached target variance is marked by a vertical line.
- Number of components to show._____no_output_____

nabo.plot_scree(wt_data.ipca, 0.5, 50)Variance target closest at PC15

Now we transform reference and target cells into PCA space using the [transform_pca](../../../api.rst#nabo.Dataset.transform_pca) method. Before we transform them into PCA space, we save mean and variance of each of the genes used for fitting PCA (i. e. the HVGs) into a dataframe (called `scaling_params` below). These means and variances are calculated from the reference sample using the [get_scaling_params](../../../api.rst#nabo.Dataset.get_scaling_params) method, and is used to z-scale genes. It is important to note that the same scaling parameters obtained from the reference sample are also used for the target sample(s). Scaling is performed within the [transform_pca](../../../api.rst#nabo.Dataset.transform_pca) method._____no_output_____

scaling_params = wt_data.get_scaling_params(wt_data.ipca.genes)
wt_data.transform_pca('../analysis_data/hvg_pca_WT.h5',
                      'data',
                      wt_data.ipca,
                      scaling_params)
me_data.transform_pca('../analysis_data/hvg_pca_ME.h5',
                      'data',
                      wt_data.ipca,
                      scaling_params)

The last preprocessing step is to generate a control dataset. The implicit idea behind using HVGs is to segregate those reference cells that imply that genes with very low variability will have the least capability to differentiate the reference cells from one another. Using such genes as control for performing cross-sample cell mapping can prove to be a useful control, as we will see in later stages of analysis. The [get_lvgs](../../../api.rst#nabo.Dataset.get_lvgs) method allows for easy obtaining of low variance genes (LVGs). The genes with (log) non-zero mean expression below the value of parameter `nzm_cutoff` (or `log_nzm_cutoff`) are  not included in LVG list.

Next, we use these LVGs to fit PCA._____no_output_____

lvgs = wt_data.get_lvgs(log_nzm_cutoff=-1, use_corrected_var=True)
wt_data.fit_ipca(lvgs)

As before, we transform the reference and target cells using LVG-fitted PCA and save them to a different file (one can save them to the same file as well using a different group name)._____no_output_____

scaling_params = wt_data.get_scaling_params(wt_data.ipca.genes)
wt_data.transform_pca('../analysis_data/lvg_pca_WT.h5',
                      'data',
                      wt_data.ipca,
                      scaling_params)
me_data.transform_pca('../analysis_data/lvg_pca_ME.h5',
                      'data',
                      wt_data.ipca,
                      scaling_params)


# Notebook from sasalatart/open-sir
Path: SIR-X.ipynb

## SIR-X

Coronavirus COVID-19 is a pandemic that is spreading quickly worlwide. Up to the 29th of March, there are 666,211 cases confirmed, 30,864 deaths and 141,789 recovered people worldwide. Governments and citizens are taking quick decisions to limit the spread of the virus and minimize the number of infected and deaths. These decisions are taken based on the experts opinion, which justify their claims based in the results of predictive models.

This Jupyter Notebook is an effort to decrease the access barriers to state of the art yet simple models that can be used to take public policy decisions to limit disease spread and save lives.

### SIR model

Most epidemic models share a common approach on modelling the spread of a disease. The SIR model is a simple deterministic compartmental model to predict disease spread. An objective population is divided in three groups: the susceptible ($S$), the infected ($I$) and the recovered or removed ($R$). These quantities enter the model as fractions of the total population $P$:

$$ S = \frac{\text{Number of susceptible individuals}}{\text{Population size}}$$

$$ I = \frac{\text{Number of infected individuals}}{\text{Population size}}$$

$$ R = \frac{\text{Number of recovered or removed individuals}}{\text{Population size}}$$

As a pandemics infects and kills much more quickly than human natural rates of birth and death, the population size is assumed constant except for the individuals that recover or die. Hence, $S+I+R=P/P=1$. The pandemics dynamics is modelled as a system of ordinary differential equations which governs the rate of change at which the percentage of susceptible, infected and recovered/removed individuals in a population evolve.

The number of possible transmissions is proportional to the number of interactions between the susceptible and infected populations, $S \times I $:

$$\frac{dS}{dt} = -\alpha SI.$$

Where $\alpha$ is the reproduction rate of the process which quantifies how many of the interactions between susceptible and infected populations yield to new infections per day.

The population of infected individuals will increase with new infections and decrease with recovered or removed people.

$$\frac{dI}{dt} = \alpha S I  - \beta I, $$
$$\frac{dR}{dt} = \beta I. $$

Where $ \beta $ is the percentage of the infected population that is removed from the transmission process per day.

In early stages of the infection, the number of infected people is much lower than the susceptible populations. Hence, $S \approx 1$ making $dI/dt$ linear and the system has the analytical solution $I(t) = I_0 \exp (\alpha - \beta)t$.

_____no_output_____#### Numerical implementation - SIR model

Three python packages are imported: numpy for numerical computing, matplotlib.pyplot for visualization and the numerical integration routine odeint from scipy.integrate:_____no_output_____

import numpy as np # Numerical computing
import matplotlib.pyplot as plt # Visualization
from scipy.integrate import odeint # ODE system numerical integrator
from scipy.optimize import curve_fit # Minimize squared errors using LM method_____no_output_____

Implementing systems of ordinary differential equations (ODEs) in python is straightforward. First, a function is created to represent the system inputs and outputs. The inputs of the function are a vector of state variables ${\vec{w}}$, the independent variable $t$ and a vector of parameters $\vec{p}$. The output of the function must be the right hand side of the ODE system as a list.

Following this approach, the SIR model can be implemented as it follows:

$$ \vec{w} = [S,I,R]$$
$$ \vec{p} = [\alpha, \beta] $$

And $t$ enters directly. The function return will be the list of ODEs.

$$\vec{f} = \left[ \frac{dS}{dt}, \frac{dI}{dt}, \frac{dR}{dt} \right]$$

So $\vec{f} = \text{sir}(\vec{w}, t, \vec{p})$.

The solution of this system is a vector field $\vec{w} = [S(t),I(t),R(t)]$. In day to day words, it gives the percentage of the population who are susceptible (S), infected (I) and recovered or removed R(t) as a function of time. There is no analytical solution for this system. However, a numerical solution can be obtained using a numerical integrator. In this implementation, the function scipy.odeint is used to integrate the differential system. The ODE system of the SIR model was implemented in the function sirx(t,w,p) on the module model.py. The solver is implemented in the function _solve on the module model.py.
_____no_output_____### SIR-X model

A new epidemic model based in SIR, SIRX, was developed by the [Robert Koch Institut](http://rocs.hu-berlin.de/corona/docs/forecast/model/#sir-x-dynamics-outbreaks-with-temporally-increasing-interventions) and is implemented in what follows. A full description of the model is available in the [Robert Koch Institut SIRX model webiste](http://rocs.hu-berlin.de/corona/docs/forecast/model/#sir-x-dynamics-outbreaks-with-temporally-increasing-interventions).

The ODE system of the SIR-X model was implemented in the function sirx(t,w,p) on the module model.py_____no_output_____### Usage example

#### Case study

The borough of Ealing, in London, is selected arbitrarly as one of the authors is living there at the moment. According to the UK office for National Statistics, the population of Ealing by mid-year 2018 is [342,000](https://www.ealing.gov.uk/info/201048/ealing_facts_and_figures/2184/population_and_households/1). The number of reported infections at 29/03/2020 is 241.

#### Model parameters
As an implementation examples, the parameter $\beta$ is estimated from the methodology followed by the [Robert Koch Institut SIRX model webiste](http://rocs.hu-berlin.de/corona/docs/forecast/model/#sir-x-dynamics-outbreaks-with-temporally-increasing-interventions). The institute estimated the a removal rate value $\beta = 0.38/d$ (mean infections time $T_I = 1/\beta = 2.6d)$ based on one third of the reported average infections preioud of moderate cases in Mainland China.

The reproduction number is fixed $R_0 = \alpha / \beta = 2.5$ as a first approximation.

Please note that the predictions of this model shouldn't be taken in consideratin, as the SIR model doesn't consider dynamic variation of model parameters, which is observed in reality._____no_output_____### Solution and implementation
The aim of this API is to provide an user friendly approach to build a SIR model and fit it to a target dataset in order to make predictions in few lines of code.
_____no_output_____

# Use Ealing as an example to determine model initial conditions
Ealing_data = [8, 18, 20, 18, 21, 42, 53, 54, 80, 97, 106, 123, 136, 165, 209, 241] # N_of infected

P_Ealing = 342000 # Ealing population ONS 2018 mid year
I_Ealing = 8      # Infected people at 14/03/2020
R_Ealing = 0      # Recovered people at 29/03/2020
n_days = len(Ealing_data)

# Input parameters
beta = 0.38 # Per day
alpha = 2.5 * beta # WHO estimate_____no_output_____

#### Calculate model parameters and initial conditions_____no_output_____

# Calculate initial conditions in terms of total number of individuals
S0 = (P_Ealing-I_Ealing)
I0 = I_Ealing
R0 = R_Ealing    # Recovered people

# Construct vector of parameters
params = [alpha, beta]

# Construct vector of initial conditions
w0 = [S0, I0, R0]_____no_output_____

### Build the model with the default parameters and predict the number of susceptible, infected and recovered people in the Ealing borough._____no_output_____

from models import SIR, SIRX_____no_output_____# Initialize an emplty SIR model
my_SIR = SIR()
# Set model parameters
my_SIR._set_params(p=params,initial_conds=w0)

# Call model.solve functions with the time in days and the number of points
# as the number of days
my_SIR.solve(n_days-1, n_days)
# Unpack the numerical solution using the model.fetch() method
sol = my_SIR.fetch()
# Unpack the numerical solution for the susceptible (S), infected (I) and recovered or removed (R)
S_sir = sol[:,1]
I_sir = sol[:,2]
R_sir = sol[:,3]
# Plot the results
# Define array of days. Note that the initial day is the day "zero", so
# the final day is the number of days minus one. This is divided in n_days
# intervals to be consistent with the input
days_list = np.linspace(0, n_days-1, n_days)
plt.plot(days_list, I_sir)
plt.plot(days_list, Ealing_data,'bo')
plt.show()
my_SIR.r0_____no_output_____

If the default parameters are used, the results are completely unreliable. Indeed, the model predicts more than 150 times more people infected. This is why a model shouldn't be used blindly, and always the parameters must be checked. In UK, Social distancing started voluntarily on the 16th of February, and the lockdown started on the 23rd of February. The effect of this policy change in terms of our model, is a decrease in the reproduction rate $R_0 = \alpha / \beta$. As the national health system (NHS) of UK didn't reach full capacity on the period between the 15th and the 29th of March, it is reasonable to assume that the main change occured owing to a decrease in the transmission rate $\alpha$.

To obtain a more realistic approximation, the parameter can be modified to better reproduce the observed data. This process is named **parameter fitting** and it is widely used not only on epidemiology, but in any discipline which uses mathematical models to make prediction.

The function model.fit() enables to fit the desired parameters to a certain dataset. The parameter fitting is straightforward using open-sir:_____no_output_____### Parameter Fitting
Fitting $R_0$ through $\alpha$ keeping $\beta$ constant

In the following case study, $R_0$ will be fitted to minimize the mean squared error between the model predictions and UK historical data on the Ealing borough in the time period between the 15th and the 29th of March of 2020._____no_output_____

Ealing_data_____no_output_____# Create SIR with default parameters
my_SIR_fitted = SIR()
my_SIR_fitted.set_params(params,w0)

# Fit parameters
w = my_SIR_fitted.fit(days_list, Ealing_data, P_Ealing, fit_index=[True,False])
# Print the fitted reproduction rate
print("Fitted reproduction rate R_0 = %.2f" % my_SIR_fitted.r0)
# Build the new solution
my_SIR_fitted.solve(n_days-1, n_days)
# Extract solution
sol = my_SIR_fitted.fetch()
# Plot the results

plt.plot(days_list,sol[:,2])
plt.plot(days_list,Ealing_data,'bo')
plt.show()_____no_output_____my_SIR_fitted.r0_____no_output_____my_SIR_fitted.p[[False,True]]
my_SIR_____no_output_____

## DANGER ZONE
### Example: predict the total number of infections and the time where the number of infected people is maximum

This is extremely dangerous as $R_0$ is extremely likely to change with time. However we have seen many people taking decisions in this kind of analysis. Use it at your own risk and with a metric ton of salt._____no_output_____

long_term_days = 90
# Convert into seconds
tf_long = long_term_days-1
sol_long = my_SIR_fitted.solve(tf_long, long_term_days).fetch()
N_S_long = sol_long[:,1]*P_Ealing
N_I_long = sol_long[:,2]*P_Ealing
N_R_long = sol_long[:,3]*P_Ealing_____no_output_____# Plot the number of susceptible, infected and recovered in a two months period
tspan_long = np.linspace(0,tf_long,long_term_days)
plt.figure(figsize=[15,5])
plt.subplot(1,3,1)
plt.plot(tspan_long,N_S_long)
plt.xlabel('Days')
plt.ylabel("Number of people")
plt.title("Susceptible")
plt.subplot(1,3,2)
plt.plot(tspan_long,N_I_long)
plt.xlabel('Days')
plt.title("Infected")
plt.subplot(1,3,3)
plt.plot(tspan_long,N_R_long)
plt.title("Recovered or removed")
plt.xlabel('Days')
plt.show()_____no_output_____

It can be observed that the SIR model reproduces the all familiar infection bell, as well as the evolution of susceptible and recovered population. It is interesting to observe that if no measures are taken in a $R_0 = 1.47$ scenario, 65% of the Ealing population would be infected in three months._____no_output_____### Sensitivity to $R_0$

A known weakness of all pandemics prediction model is the sensitivity to their parameters. In the following case study, $R_0$ will be fitted to minimize the mean squared error between the model predictions and UK historical data on the Ealing borough in the time period between the 15th and the 29th of March of 2020._____no_output_____

def compare_infections(model, tf, numpoints, alpha_list=2.5, abserr=1e-8, relerr=1e-6):
    """ compare_infections compare SIR model predictions against
    a list of alpha values

    Inputs:
    w0: Initial conditions
    t: Time vector /
    alpha_list: list or numpy array of values of alpha to be tested

    Outputs:
    S_list: List of predictions for the fraction of susceptible population for each alpha
    I_list: List of predictions for the fraction of infected population for each alpha
    R_list: List of predictions for the fraction of recovered/removed population for each alpha
    """
    S_list = []
    I_list = []
    R_list = []

    for i in alpha_list:
        # Update parameter list
        model.p[0] = i
        wsol=model.solve(tf,numpoints).fetch()
        S_list.append(wsol[:,1])
        I_list.append(wsol[:,2])
        R_list.append(wsol[:,3])
    return S_list, I_list, R_list_____no_output_____

### Generate predictions for each alpha_____no_output_____

alpha_list = beta*np.array([1.5,1.6,1.7])
S_list, I_list, R_list = compare_infections(my_SIR, tf_long, long_term_days, alpha_list)_____no_output_____col = ['r','b','k']
plt.figure(figsize=[15,5])
for i in range(len(S_list)):
    plt.subplot(1,3,1)
    plt.plot(tspan_long,S_list[i], col[i]+"--")
    plt.legend(['R_0 = 1.5','R_0 = 1.6','R_0 = 1.7'])
    plt.xlabel('Days')
    plt.ylabel('Fraction of population')
    plt.title('S')
    plt.subplot(1,3,2)
    plt.plot(tspan_long,I_list[i], col[i])
    plt.legend(['R_0 = 1.5','R_0 = 1.6','R_0 = 1.7'])
    plt.xlabel('Days')
    plt.title('I')
    plt.subplot(1,3,3)
    plt.plot(tspan_long,R_list[i], col[i]+"-.")
    plt.legend(['R_0 = 1.5','R_0 = 1.6','R_0 = 1.7'])
    plt.xlabel('Days')
    plt.title('R')_____no_output_____

We observe that a change as little as 6% in the reproduction rate, can change dramatically the dynamic of the pandemic_____no_output_____### Example 4: Fit R_0 for UK values
[sourced from UK Arcgis](https://www.arcgis.com/apps/opsdashboard/index.html#/f94c3c90da5b4e9f9a0b19484dd4bb14)_____no_output_____

P_UK = 67886011
# Data up to 28th of March
I_UK= [3269, 3983, 5018, 5683, 6650, 8077, 9529, 11658, 14543, 17089]
n_days = len(I_UK) # Final day
t_d = np.linspace(0,n_days-1,n_days)

n_S0 = P_UK-I_UK[0]
n_I0 = I_UK[0]
n_R0 = 0
n0_UK = [n_S0, n_I0, n_R0]
p = [alpha,beta]

# Create empty model
SIR_UK = SIR()
SIR_UK.set_params(p,n0_UK)
# Train model
SIR_UK.fit(t_d, I_UK, P_UK)
# Build numerical solution
I_opt = SIR_UK.solve(n_days-1,n_days).fetch()[:,2]
# lag = 6

R_opt = SIR_UK.r0 #

plt.figure(figsize=[6,6])
plt.plot(t_d,I_UK,'o')
plt.plot(t_d, I_opt)
plt.legend(["UK Data","SIR, $R_{0,opt}$ = %.2f"%R_opt])
plt.title("Fitting of the SIR model against 15 days of UK data")
plt.ylabel("Number of people infected")
plt.xlabel("Day")
plt.xlim([min(t_d),max(t_d)])
plt.show()
_____no_output_____MSE = sum(np.sqrt((I_opt-I_UK)**2))/len(I_UK)
print("Mean squared error on the model in the train dataset %.2f" % MSE)_____no_output_____

### Calculate confidence intervals

_____no_output_____

from models import ci_bootstrap, ci_block_cv_____no_output_____# Get the confidence interval through random bootstrap
# Define bootstrap options
options = {"alpha":0.95, "n_iter":1000, "r0_ci":True}
# Call bootstrap
par_ci, par_list = ci_bootstrap(SIR_UK, t_d, I_UK, P_UK, options)_____no_output_____par_ci_____no_output_____alpha_min = par_ci[0][0]
alpha_max = par_ci[0][1]
# Explore the confidence intervals
print("IC 95% for alpha:", par_ci[0])
print("IC 95% for beta:", par_ci[1])
print("IC 95% for r0:", par_ci[2])_____no_output_____

Now we can visualize_____no_output_____

# Build numerical solution
I_opt = SIR_UK.solve(n_days-1, n_days).fetch()[:,2]
beta_0 = SIR_UK.p[1]
SIR_minus = SIR().set_params([alpha_min, beta_0],n0_UK)
SIR_plus = SIR().set_params([alpha_max, beta_0],n0_UK)
I_minus = SIR_minus.solve(n_days-1, n_days).fetch()[:,2]
I_plus = SIR_plus.solve(n_days-1, n_days).fetch()[:,2]

# lag = 6

R_opt = SIR_UK.r0 #

plt.figure(figsize=[6,6])
plt.plot(t_d,I_UK,'o')
plt.plot(t_d, I_opt)
plt.plot(t_d, I_minus)
plt.plot(t_d, I_plus)
plt.legend(["UK Data","SIR, $R_{0,opt}$ = %.2f"%R_opt,"IC_95-","IC_95+"])
plt.title("Fitting of the SIR model against 15 days of UK data")
plt.ylabel("Number of people infected")
plt.xlabel("Day")
plt.xlim([min(t_d),max(t_d)])
plt.show()_____no_output_____

An extremely asymmetrical confidence interval for $R_0$ using simple random bootstrap is observed. This occurs most likely because of
neglecting the temporal structure of the exponential.

To further investigate this phenomena, we can observe the distribution of the $R_0$ parameter on the parameter list_____no_output_____

plt.hist(par_list[:,0]/par_list[:,1], bins=50, density=True, stacked=True)
plt.xlabel('$R_0$')
plt.ylabel('Frequency')
plt.title("Probability density diagram of $R_0$ bootstrapping")
plt.show()_____no_output_____

It is interesting to observe that the spread is assymetrical towards lower $R_0$ values. This asymmetry is expected owing to the effect of lockdowns and public policies to promote social distancing. A strong assumption of the SIR model is that the spread rate $\alpha$ and removal rate $\beta$ are constant, which is not the case in reality specially when strong public policies to limit the spread of a virus take place._____no_output_____

# We previously imported ci_block_cv which provides a better prediction of the mean squared error of the predictions
n_lags = 1
options={"lags":n_lags,"min_sample":3}
MSE_avg, MSE_list, p_list = ci_block_cv(SIR_UK, t_d, I_UK, P_UK, options)_____no_output_____p_list_____no_output_____print("The average mean squared error on the time cross validation bootstrapping is: %.3f" % MSE_avg)_____no_output_____

We can visualize the results of the block bootstrapping in terms of the variation of the reproduction rate $R_0$ and the mean squared error when a subset of the days is taken. By default, ci_block_cv starts with the data of the three days, fit the model on that data, predicts the number of infected in the next period, calculate the mean squared error between the prediction and the test dataset, and stores it into two arrays. It repeats this until it uses the data of $(n-1)$ intervals to predict the $n-th$ latest observation of infections._____no_output_____

r0_roll = p_list[:,0]/p_list[:,1]

plt.figure(figsize=[10,5])
plt.subplot(1,2,1)
plt.plot(t_d[(2+n_lags):], r0_roll,'ro')
plt.xlabel("Days used in the block to fit parameters")
plt.ylabel("Rolling $R_0$")
plt.title("Block bootstrapping change in $R_0$")
plt.subplot(1,2,2)
plt.plot(MSE_list,'bo')
plt.xlabel("Days used in the block to fit parameters")
plt.ylabel("Mean squared error in number of infected")
plt.title("Block bootstrapping change in MSE")
plt.show()_____no_output_____


# Notebook from mm5110/PIC16A
Path: live_lectures/14-live-lecture.ipynb

## Midterm Review Questions

### Likely Topics

- Dictionaries
- Functions
- Object-oriented programming
- Iteration
- Numpy
- lambda functions
- Exceptions + error handling
    - Raising exceptions
    - `try-except`

_____no_output_____## `lambda` Functions

- How do they work?
- Do we need docstrings?_____no_output_____

def doubler(x):
    return 2*x

doubler(5)

doubler = lambda x: 2*x

L = ["Picard", "Janeway", "Sisko"]

# sort by LAST letter
L.sort(key = lambda x: x[-1])
L_____no_output_____# function to give a list of squares up to k
squares = lambda k: [j**2 for j in range(1, k+1)]

squares(5)_____no_output_____import numpy as np
squares = lambda k: list(np.arange(1, k+1)**2)
squares(5)_____no_output_____

## "Magic" Methods

What's magic about "magic" methods is simply that they are called implicitly through other operations. _____no_output_____

class totoro:

    def yell(self):
        print("AAAAAAHHHHHHH")

    def __str__(self):
        return "An object of class totoro. A magic method was called."_____no_output_____my_neighbor = totoro()
my_neighbor.yell()     # explicitly calling a method of the classAAAAAAHHHHHHH
print(my_neighbor)     # example of magic method being calledAn object of class totoro. A magic method was called.
# printing an object: print(str(obj))_____no_output_____2 + 2 # calling the __add__() magic method of integers
(2).__add__(2)_____no_output_____

## `try-except` and `if-else`_____no_output_____

# this code errors

d = {
    "Sisko" : "DS9",
    "Janeway" : "VOY"
}

key = input("please choose a Star Trek Captain")

print(d[key])please choose a Star Trek Captain Picard
# prevent error scenario from ever occurring
# using if-else

d = {
    "Sisko" : "DS9",
    "Janeway" : "VOY"
}

key = input("please choose a Star Trek Captain")

# check if the key is in the dictionary
if key in d.keys():
    # print if so
    print(d[key])
else:
    print("Sorry, your input wasn't found in my data.")please choose a Star Trek Captain Picard
# allow the scenario to occur
# define alternate behavior for the exception

d = {
    "Sisko" : "DS9",
    "Janeway" : "VOY"
}

key = input("please choose a Star Trek Captain")

try:
    print(d[key])
except KeyError:
    print("Sorry, I couldn't find that value.")please choose a Star Trek Captain Picard


# Notebook from cedrickchee/fastai
Path: courses/dl2/imdb.ipynb

## IMDb_____no_output_____At Fast.ai we have introduced a new module called fastai.text which replaces the torchtext library that was used in our 2018 dl1 course. The fastai.text module also supersedes the fastai.nlp library but retains many of the key functions._____no_output_____

from fastai.text import *
import html_____no_output_____

The Fastai.text module introduces several custom tokens.

We need to download the IMDB large movie reviews from this site: http://ai.stanford.edu/~amaas/data/sentiment/
Direct link : [Link](http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz) and untar it into the PATH location. We use pathlib which makes directory traveral a breeze._____no_output_____**===================================== (START) Download IMDb data =====================================**_____no_output_____

%mkdir data/aclImdb
%cd data/aclImdb/home/ubuntu/data/aclImdb
!aria2c --file-allocation=none -c -x 5 -s 5 http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz[#6acf06 79MiB/80MiB(99%) CN:1 DL:14MiB]
06/26 15:59:49 [NOTICE] Download complete: /home/ubuntu/data/aclImdb/aclImdb_v1.tar.gz

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
6acf06|OK  |    14MiB/s|/home/ubuntu/data/aclImdb/aclImdb_v1.tar.gz

Status Legend:
(OK):download completed.
!tar -zxf aclImdb_v1.tar.gz -C ._____no_output_____%cd ../..
%rm data/aclImdb/aclImdb_v1.tar.gz/home/ubuntu
%mv data/aclImdb/ data/aclImdb2_____no_output_____%mv data/aclImdb2/aclImdb data/_____no_output_____%rm -rf data/aclImdb2_____no_output_____PATH = Path('data/aclImdb/')_____no_output_____!ls -lah {PATH}total 1.7M
drwxr-xr-x 4 ubuntu ubuntu 4.0K Jun 26  2011 .
drwxrwxr-x 8 ubuntu ubuntu 4.0K Jun 26 16:17 ..
-rw-r--r-- 1 ubuntu ubuntu 882K Jun 11  2011 imdbEr.txt
-rw-r--r-- 1 ubuntu ubuntu 827K Apr 12  2011 imdb.vocab
-rw-r--r-- 1 ubuntu ubuntu 4.0K Jun 26  2011 README
drwxr-xr-x 4 ubuntu ubuntu 4.0K Jun 26 16:02 test
drwxr-xr-x 5 ubuntu ubuntu 4.0K Jun 26 16:02 train

**===================================== (END) Download IMDb data =====================================**_____no_output_____

BOS = 'xbos'  # beginning-of-sentence tag
FLD = 'xfld'  # data field tag_____no_output_____

## Standardize format_____no_output_____

CLAS_PATH = Path('data/imdb_clas/')
CLAS_PATH.mkdir(exist_ok=True)_____no_output_____!ls dataaclImdb  dogscats  dogscats.zip  imdb_clas  pascal  spellbee
LM_PATH = Path('data/imdb_lm/')
LM_PATH.mkdir(exist_ok=True)_____no_output_____!ls dataaclImdb  dogscats  dogscats.zip  imdb_clas  imdb_lm  pascal  spellbee

The IMDb dataset has 3 classes; positive, negative and unsupervised(sentiment is unknown).
There are 75k training reviews(12.5k pos, 12.5k neg, 50k unsup)
There are 25k validation reviews(12.5k pos, 12.5k neg & no unsup)

Refer to the README file in the IMDb corpus for further information about the dataset._____no_output_____

!cat data/aclImdb/READMELarge Movie Review Dataset v1.0

Overview

This dataset contains movie reviews along with their associated binary
sentiment polarity labels. It is intended to serve as a benchmark for
sentiment classification. This document outlines how the dataset was
gathered, and how to use the files provided.

Dataset

The core dataset contains 50,000 reviews split evenly into 25k train
and 25k test sets. The overall distribution of labels is balanced (25k
pos and 25k neg). We also include an additional 50,000 unlabeled
documents for unsupervised learning.

In the entire collection, no more than 30 reviews are allowed for any
given movie because reviews for the same movie tend to have correlated
ratings. Further, the train and test sets contain a disjoint set of
movies, so no significant performance is obtained by memorizing
movie-unique terms and their associated with observed labels.  In the
labeled train/test sets, a negative review has a score <= 4 out of 10,
and a positive review has a score >= 7 out of 10. Thus reviews with
more neutral ratings are not included in the train/test sets. In the
unsupervised set, reviews of any rating are included and there are an
even number of reviews > 5 and <= 5.

Files

There are two top-level directories [train/, test/] corresponding to
the training and test sets. Each contains [pos/, neg/] directories for
the reviews with binary labels positive and negative. Within these
directories, reviews are stored in text files named following the
convention [[id]_[rating].txt] where [id] is a unique id and [rating] is
the star rating for that review on a 1-10 scale. For example, the file
[test/pos/200_8.txt] is the text for a positive-labeled test set
example with unique id 200 and star rating 8/10 from IMDb. The
[train/unsup/] directory has 0 for all ratings because the ratings are
omitted for this portion of the dataset.

We also include the IMDb URLs for each review in a separate
[urls_[pos, neg, unsup].txt] file. A review with unique id 200 will
have its URL on line 200 of this file. Due the ever-changing IMDb, we
are unable to link directly to the review, but only to the movie's
review page.

In addition to the review text files, we include already-tokenized bag
of words (BoW) features that were used in our experiments. These
are stored in .feat files in the train/test directories. Each .feat
file is in LIBSVM format, an ascii sparse-vector format for labeled
data.  The feature indices in these files start from 0, and the text
tokens corresponding to a feature index is found in [imdb.vocab]. So a
line with 0:7 in a .feat file means the first word in [imdb.vocab]
(the) appears 7 times in that review.

LIBSVM page for details on .feat file format:
http://www.csie.ntu.edu.tw/~cjlin/libsvm/

We also include [imdbEr.txt] which contains the expected rating for
each token in [imdb.vocab] as computed by (Potts, 2011). The expected
rating is a good way to get a sense for the average polarity of a word
in the dataset.

Citing the dataset

When using this dataset please cite our ACL 2011 paper which
introduces it. This paper also contains classification results which
you may want to compare against.

@InProceedings{maas-EtAl:2011:ACL-HLT2011,
  author    = {Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher},
  title     = {Learning Word Vectors for Sentiment Analysis},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  month     = {June},
  year      = {2011},
  address   = {Portland, Oregon, USA},
  publisher = {Association for Computational Linguistics},
  pages     = {142--150},
  url       = {http://www.aclweb.org/anthology/P11-1015}
}

References

Potts, Christopher. 2011. On the negativity of negation. In Nan Li and
David Lutz, eds., Proceedings of Semantics and Linguistic Theory 20,
636-659.

Contact

For questions/comments/corrections please contact Andrew Maas
amaas@cs.stanford.edu
CLASSES = ['neg', 'pos', 'unsup']_____no_output_____def get_texts(path):
    texts, labels = [], []

    for idx, label in enumerate(CLASSES):
        for fname in (path/label).glob('*.*'):
            texts.append(fname.open('r').read())
            labels.append(idx)
    return np.array(texts), np.array(labels)_____no_output_____trn_texts, trn_labels = get_texts(PATH / 'train')
val_texts, val_labels = get_texts(PATH / 'test')_____no_output_____len(trn_texts), len(val_texts)_____no_output_____col_names = ['labels', 'text']_____no_output_____

We use a random permutation numpy array to shuffle the text reviews._____no_output_____

np.random.seed(42)
trn_idx = np.random.permutation(len(trn_texts))
val_idx = np.random.permutation(len(val_texts))_____no_output_____trn_texts = trn_texts[trn_idx]
val_texts = val_texts[val_idx]

trn_labels = trn_labels[trn_idx]
val_labels = val_labels[val_idx]_____no_output_____df_trn = pd.DataFrame({ 'text': trn_texts, 'labels': trn_labels }, columns=col_names)
df_val = pd.DataFrame({ 'text': val_texts, 'labels': val_labels }, columns=col_names)_____no_output_____# DEBUG
# View train df
df_trn.head()_____no_output_____# DEBUG
# View validation df
df_val.head()_____no_output_____

The pandas dataframe is used to store text data in a newly evolving standard format of label followed by text columns. This was influenced by a paper by Yann LeCun (LINK REQUIRED). Fastai adopts this new format for NLP datasets. In the case of IMDB, there is only one text column._____no_output_____

# we remove everything that has a label of 2 `df_trn['labels'] != 2` because label of 2 is "unsupervised" and we can’t use it.
df_trn[df_trn['labels'] != 2].to_csv(CLAS_PATH / 'train.csv', header=False, index=False)

df_val.to_csv(CLAS_PATH / 'test.csv', header=False, index=False)

(CLAS_PATH / 'classes.txt').open('w').writelines(f'{o}\n' for o in CLASSES)_____no_output_____

We start by creating the data for the Language Model(LM). The LM's goal is to learn the structure of the English language. It learns language by trying to predict the next word given a set of previous words(ngrams). Since the LM does not classify reviews, the labels can be ignored.

The LM can benefit from all the textual data and there is no need to exclude the unsup/unclassified movie reviews.

We first concat all the train(pos/neg/unsup = **75k**) and test(pos/neg=**25k**) reviews into a big chunk of **100k** reviews. And then we use sklearn splitter to divide up the 100k texts into 90% training and 10% validation sets._____no_output_____

trn_texts, val_texts = sklearn.model_selection.train_test_split(
    np.concatenate([trn_texts, val_texts]), test_size=0.1)_____no_output_____len(trn_texts), len(val_texts)_____no_output_____df_trn = pd.DataFrame({ 'text': trn_texts, 'labels': [0] * len(trn_texts) }, columns=col_names)
df_val = pd.DataFrame({ 'text': val_texts, 'labels': [0] * len(val_texts) }, columns=col_names)

df_trn.to_csv(LM_PATH / 'train.csv', header=False, index=False)
df_val.to_csv(LM_PATH / 'test.csv', header=False, index=False)_____no_output_____

## Language model tokens_____no_output_____In this section, we start cleaning up the messy text. There are 2 main activities we need to perform:

1. Clean up extra spaces, tab chars, new line chars and other characters and replace them with standard ones.
2. Use the awesome [spaCy](http://spacy.io) library to tokenize the data. Since spaCy does not provide a parallel/multicore version of the tokenizer, the fastai library adds this functionality. This parallel version uses all the cores of your CPUs and runs much faster than the serial version of the spacy tokenizer.

Tokenization is the process of splitting the text into separate tokens so that each token can be assigned a unique index. This means we can convert the text into integer indexes our models can use.

We use an appropriate `chunksize` as the tokenization process is memory intensive._____no_output_____### Tokenize_____no_output_____

chunksize = 24000_____no_output_____

Before we pass it to spaCy, we will write a simple fixup function which is each time we have looked at different datasets (about a dozen in building this), every one had different weird things that needed to be replaced. So here are all the ones we have come up with so far, and hopefully this will help you out as well. All the entities are HTML unescaped and there are bunch more things we replace. Have a look at the result of running this on text that you put in and make sure there's no more weird tokens in there._____no_output_____

re1 = re.compile(r'  +')

def fixup(x):
    x = x.replace('#39;', "'").replace('amp;', '&').replace('#146;', "'").replace(
        'nbsp;', ' ').replace('#36;', '$').replace('\\n', "\n").replace('quot;', "'").replace(
        '
', "\n").replace('\\"', '"').replace('<unk>', 'u_n').replace(' @.@ ', '.').replace(
        ' @-@ ', '-').replace('\\', ' \\ ')
    return re1.sub(' ', html.unescape(x))_____no_output_____def get_texts(df, n_lbls=1):
    labels = df.iloc[:, range(n_lbls)].values.astype(np.int64)
    texts = f'\n{BOS} {FLD} 1 ' + df[n_lbls].astype(str)
    for i in range(n_lbls + 1, len(df.columns)):
        texts += f' {FLD} {i - n_lbls} ' + df[i].astype(str)
    texts = texts.apply(fixup).values.astype(str)

    tok = Tokenizer().proc_all_mp(partition_by_cores(texts))
    return tok, list(labels)_____no_output_____def get_all(df, n_lbls):
    tok, labels = [], []

    for i, r in enumerate(df):
        print(i)
        tok_, labels_ = get_texts(r, n_lbls)
        tok += tok_
        labels += labels_
    return tok, labels_____no_output_____df_trn = pd.read_csv(LM_PATH / 'train.csv', header=None, chunksize=chunksize)
df_val = pd.read_csv(LM_PATH / 'test.csv', header=None, chunksize=chunksize)_____no_output_____# tok_trn, trn_labels = get_all(df_trn, 1)
# tok_val, val_labels = get_all(df_val, 1)0

Fix spaCy issue above:_____no_output_____

!python -m spacy download enCollecting https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz
  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz (37.4MB)
    100% |████████████████████████████████| 37.4MB 81.7MB/s ta 0:00:01  7% |██▎                             | 2.6MB 611kB/s eta 0:00:57
Installing collected packages: en-core-web-sm
  Running setup.py install for en-core-web-sm ... done
Successfully installed en-core-web-sm-2.0.0
You are using pip version 9.0.3, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

    Linking successful
    /home/ubuntu/anaconda3/envs/fastai/lib/python3.6/site-packages/en_core_web_sm
    -->
    /home/ubuntu/anaconda3/envs/fastai/lib/python3.6/site-packages/spacy/data/en

    You can now load the model via spacy.load('en')

# Re-run these 2 lines of codes.
tok_trn, trn_labels = get_all(df_trn, 1)
tok_val, val_labels = get_all(df_val, 1)0
1
2
3
0
(LM_PATH / 'tmp').mkdir(exist_ok=True)_____no_output_____

**Testing**_____no_output_____

type(tok_trn), len(tok_trn)_____no_output_____tok_trn[0]_____no_output_____type(trn_labels), len(trn_labels)_____no_output_____

Tokenization Result

Beginning of the stream token (`xbos`), beginning of field number 1 token (`xfld 1`), and tokenized text. You'll see that the punctuation is on whole now a separate token._____no_output_____

' '.join(tok_trn[0])_____no_output_____' '.join(tok_trn[1])_____no_output_____

**Save work**_____no_output_____

np.save(LM_PATH / 'tmp' / 'tok_trn.npy', tok_trn)
np.save(LM_PATH / 'tmp' / 'tok_val.npy', tok_val)_____no_output_____# Try to load it back up later
tok_trn = np.load(LM_PATH / 'tmp' / 'tok_trn.npy')
tok_val = np.load(LM_PATH / 'tmp' / 'tok_val.npy')_____no_output_____# DEBUG - check serialized numpy files are created
!ls -lh {LM_PATH}/tmptotal 381M
-rw-rw-r-- 1 ubuntu ubuntu 1010K Jun 27 15:42 itos.pkl
-rw-rw-r-- 1 ubuntu ubuntu  299M Jun 28 07:30 tok_trn.npy
-rw-rw-r-- 1 ubuntu ubuntu   34M Jun 28 07:30 tok_val.npy
-rw-rw-r-- 1 ubuntu ubuntu   42M Jun 27 15:42 trn_ids.npy
-rw-rw-r-- 1 ubuntu ubuntu  6.3M Jun 27 15:42 val_ids.npy

### Numericalize_____no_output_____Now that we got it tokenized, the next thing we need to do is to turn it into numbers which we call numericalizing it._____no_output_____

freq = Counter(p for o in tok_trn for p in o)
freq.most_common(25)_____no_output_____

The *vocab* is the **unique set of all tokens** in our dataset. The vocab provides us a way for us to simply replace each word in our datasets with a unique integer called an index.

In a large corpus of data one might find some rare words which are only used a few times in the whole dataset. We discard such rare words and avoid trying to learn meaningful patterns out of them.

Here we have set a minimum frequency of occurence to 2 times. It has been observed by NLP practicioners that a maximum vocab of 60k usually yields good results for classification tasks. So we set `max_vocab` to 60000._____no_output_____

max_vocab = 60000
min_freq = 1_____no_output_____itos = [o for o, c in freq.most_common(max_vocab) if c > min_freq]
itos.insert(0, '_pad_')
itos.insert(0, '_unk_')_____no_output_____len(itos)_____no_output_____

We create a reverse mapping called `stoi` which is useful to lookup the index of a given token. `stoi` also has the same number of elements as `itos`. We use a high performance container called [collections.defaultdict](https://docs.python.org/2/library/collections.html#collections.defaultdict) to store our `stoi` mapping._____no_output_____

stoi = collections.defaultdict(lambda: 0, { v: k for k, v in enumerate(itos) })
len(itos)_____no_output_____trn_lm = np.array([ [stoi[o] for o in p] for p in tok_trn ])
val_lm = np.array([ [stoi[o] for o in p] for p in tok_val ])_____no_output_____

**Testing**_____no_output_____

' '.join(str(o) for o in trn_lm[0])_____no_output_____

**Save work**_____no_output_____

np.save(LM_PATH / 'tmp' / 'trn_ids.npy', trn_lm)
np.save(LM_PATH / 'tmp' / 'val_ids.npy', val_lm)
pickle.dump(itos, open(LM_PATH / 'tmp' / 'itos.pkl', 'wb'))_____no_output_____trn_lm = np.load(LM_PATH / 'tmp' / 'trn_ids.npy')
val_lm = np.load(LM_PATH / 'tmp' / 'val_ids.npy')
itos = pickle.load(open(LM_PATH / 'tmp' / 'itos.pkl', 'rb'))_____no_output_____vs = len(itos)
vs, len(trn_lm)_____no_output_____

## Pre-training

### WikiText-103 conversion_____no_output_____We are now going to build an English language model (LM) for the IMDb corpus. We could start from scratch and try to learn the structure of the English language. But we use a technique called transfer learning to make this process easier. In transfer learning (a fairly recent idea for NLP) a pre-trained LM that has been trained on a large generic corpus(_like wikipedia articles_) can be used to transfer it's knowledge to a target LM and the weights can be fine-tuned.

Our source LM is the WikiText-103 LM created by Stephen Merity @ Salesforce research. [Link to dataset](https://www.salesforce.com/products/einstein/ai-research/the-wikitext-dependency-language-modeling-dataset/)
The language model for wikitext103 (AWD LSTM) has been pre-trained and the weights can be downloaded here: [Link](http://files.fast.ai/models/wt103/). Our target LM is the IMDb LM._____no_output_____

# wget options:
# -nH don't create host directories
# -r specify recursive download
# -np don't ascend to the parent directory
# -P get all images, etc. needed to display HTML page
!wget -nH -r -np -P {PATH} http://files.fast.ai/models/wt103/--2018-06-28 04:49:15--  http://files.fast.ai/models/wt103/
Resolving files.fast.ai (files.fast.ai)... 67.205.15.147
Connecting to files.fast.ai (files.fast.ai)|67.205.15.147|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 857 [text/html]
Saving to: ‘data/aclImdb/models/wt103/index.html’

models/wt103/index. 100%[===================>]     857  --.-KB/s    in 0s

2018-06-28 04:49:16 (140 MB/s) - ‘data/aclImdb/models/wt103/index.html’ saved [857/857]

Loading robots.txt; please ignore errors.
--2018-06-28 04:49:16--  http://files.fast.ai/robots.txt
Reusing existing connection to files.fast.ai:80.
HTTP request sent, awaiting response... 404 Not Found
2018-06-28 04:49:16 ERROR 404: Not Found.

--2018-06-28 04:49:16--  http://files.fast.ai/models/wt103/?C=N;O=D
Reusing existing connection to files.fast.ai:80.
HTTP request sent, awaiting response... 200 OK
Length: 857 [text/html]
Saving to: ‘data/aclImdb/models/wt103/index.html?C=N;O=D’

models/wt103/index. 100%[===================>]     857  --.-KB/s    in 0s

2018-06-28 04:49:16 (160 MB/s) - ‘data/aclImdb/models/wt103/index.html?C=N;O=D’ saved [857/857]

--2018-06-28 04:49:16--  http://files.fast.ai/models/wt103/?C=M;O=A
Reusing existing connection to files.fast.ai:80.
HTTP request sent, awaiting response... 200 OK
Length: 857 [text/html]
Saving to: ‘data/aclImdb/models/wt103/index.html?C=M;O=A’

models/wt103/index. 100%[===================>]     857  --.-KB/s    in 0s

2018-06-28 04:49:16 (138 MB/s) - ‘data/aclImdb/models/wt103/index.html?C=M;O=A’ saved [857/857]

--2018-06-28 04:49:16--  http://files.fast.ai/models/wt103/?C=S;O=A
Reusing existing connection to files.fast.ai:80.
HTTP request sent, awaiting response... 200 OK
Length: 857 [text/html]
Saving to: ‘data/aclImdb/models/wt103/index.html?C=S;O=A’

models/wt103/index. 100%[===================>]     857  --.-KB/s    in 0s

2018-06-28 04:49:16 (134 MB/s) - ‘data/aclImdb/models/wt103/index.html?C=S;O=A’ saved [857/857]

--2018-06-28 04:49:16--  http://files.fast.ai/models/wt103/?C=D;O=A
Reusing existing connection to files.fast.ai:80.
HTTP request sent, awaiting response... 200 OK
Length: 857 [text/html]
Saving to: ‘data/aclImdb/models/wt103/index.html?C=D;O=A’

models/wt103/index. 100%[===================>]     857  --.-KB/s    in 0s

2018-06-28 04:49:17 (123 MB/s) - ‘data/aclImdb/models/wt103/index.html?C=D;O=A’ saved [857/857]

--2018-06-28 04:49:17--  http://files.fast.ai/models/wt103/bwd_wt103.h5
Reusing existing connection to files.fast.ai:80.
HTTP request sent, awaiting response... 200 OK
Length: 462387687 (441M) [text/plain]
Saving to: ‘data/aclImdb/models/wt103/bwd_wt103.h5’

models/wt103/bwd_wt 100%[===================>] 440.97M  7.69MB/s    in 59s

2018-06-28 04:50:16 (7.45 MB/s) - ‘data/aclImdb/models/wt103/bwd_wt103.h5’ saved [462387687/462387687]

--2018-06-28 04:50:16--  http://files.fast.ai/models/wt103/bwd_wt103_enc.h5
Reusing existing connection to files.fast.ai:80.
HTTP request sent, awaiting response... 200 OK
Length: 462387634 (441M) [text/plain]
Saving to: ‘data/aclImdb/models/wt103/bwd_wt103_enc.h5’

models/wt103/bwd_wt 100%[===================>] 440.97M  6.76MB/s    in 59s

2018-06-28 04:51:16 (7.43 MB/s) - ‘data/aclImdb/models/wt103/bwd_wt103_enc.h5’ saved [462387634/462387634]

--2018-06-28 04:51:16--  http://files.fast.ai/models/wt103/fwd_wt103.h5
Reusing existing connection to files.fast.ai:80.
HTTP request sent, awaiting response... 200 OK
Length: 462387687 (441M) [text/plain]
Saving to: ‘data/aclImdb/models/wt103/fwd_wt103.h5’

models/wt103/fwd_wt 100%[===================>] 440.97M  7.66MB/s    in 59s

2018-06-28 04:52:15 (7.48 MB/s) - ‘data/aclImdb/models/wt103/fwd_wt103.h5’ saved [462387687/462387687]

--2018-06-28 04:52:15--  http://files.fast.ai/models/wt103/fwd_wt103_enc.h5
Reusing existing connection to files.fast.ai:80.
HTTP request sent, awaiting response... 200 OK
Length: 462387634 (441M) [text/plain]
Saving to: ‘data/aclImdb/models/wt103/fwd_wt103_enc.h5’

models/wt103/fwd_wt 100%[===================>] 440.97M  7.76MB/s    in 58s

2018-06-28 04:53:13 (7.61 MB/s) - ‘data/aclImdb/models/wt103/fwd_wt103_enc.h5’ saved [462387634/462387634]

--2018-06-28 04:53:13--  http://files.fast.ai/models/wt103/itos_wt103.pkl
Reusing existing connection to files.fast.ai:80.
HTTP request sent, awaiting response... 200 OK
Length: 4161252 (4.0M) [text/plain]
Saving to: ‘data/aclImdb/models/wt103/itos_wt103.pkl’

models/wt103/itos_w 100%[===================>]   3.97M  9.86MB/s    in 0.4s

2018-06-28 04:53:13 (9.86 MB/s) - ‘data/aclImdb/models/wt103/itos_wt103.pkl’ saved [4161252/4161252]

--2018-06-28 04:53:13--  http://files.fast.ai/models/wt103/?C=N;O=A
Reusing existing connection to files.fast.ai:80.
HTTP request sent, awaiting response... 200 OK
Length: 857 [text/html]
Saving to: ‘data/aclImdb/models/wt103/index.html?C=N;O=A’

models/wt103/index. 100%[===================>]     857  --.-KB/s    in 0s

2018-06-28 04:53:14 (146 MB/s) - ‘data/aclImdb/models/wt103/index.html?C=N;O=A’ saved [857/857]

--2018-06-28 04:53:14--  http://files.fast.ai/models/wt103/?C=M;O=D
Reusing existing connection to files.fast.ai:80.
HTTP request sent, awaiting response... 200 OK
Length: 857 [text/html]
Saving to: ‘data/aclImdb/models/wt103/index.html?C=M;O=D’

models/wt103/index. 100%[===================>]     857  --.-KB/s    in 0s

2018-06-28 04:53:14 (186 MB/s) - ‘data/aclImdb/models/wt103/index.html?C=M;O=D’ saved [857/857]

--2018-06-28 04:53:14--  http://files.fast.ai/models/wt103/?C=S;O=D
Reusing existing connection to files.fast.ai:80.
HTTP request sent, awaiting response... 200 OK
Length: 857 [text/html]
Saving to: ‘data/aclImdb/models/wt103/index.html?C=S;O=D’

models/wt103/index. 100%[===================>]     857  --.-KB/s    in 0s

2018-06-28 04:53:14 (169 MB/s) - ‘data/aclImdb/models/wt103/index.html?C=S;O=D’ saved [857/857]

--2018-06-28 04:53:14--  http://files.fast.ai/models/wt103/?C=D;O=D
Reusing existing connection to files.fast.ai:80.
HTTP request sent, awaiting response... 200 OK
Length: 857 [text/html]
Saving to: ‘data/aclImdb/models/wt103/index.html?C=D;O=D’

models/wt103/index. 100%[===================>]     857  --.-KB/s    in 0s

2018-06-28 04:53:14 (153 MB/s) - ‘data/aclImdb/models/wt103/index.html?C=D;O=D’ saved [857/857]

FINISHED --2018-06-28 04:53:14--
Total wall clock time: 4m 0s
Downloaded: 14 files, 1.7G in 3m 56s (7.50 MB/s)
!ls -lh {PATH}/models/wt103total 1.8G
-rw-rw-r-- 1 ubuntu ubuntu 441M Mar 29 00:31 bwd_wt103_enc.h5
-rw-rw-r-- 1 ubuntu ubuntu 441M Mar 29 00:34 bwd_wt103.h5
-rw-rw-r-- 1 ubuntu ubuntu 441M Mar 29 00:36 fwd_wt103_enc.h5
-rw-rw-r-- 1 ubuntu ubuntu 441M Mar 29 00:39 fwd_wt103.h5
-rw-rw-r-- 1 ubuntu ubuntu  857 Jun 28 04:49 index.html
-rw-rw-r-- 1 ubuntu ubuntu  857 Jun 28 04:49 index.html?C=D;O=A
-rw-rw-r-- 1 ubuntu ubuntu  857 Jun 28 04:53 index.html?C=D;O=D
-rw-rw-r-- 1 ubuntu ubuntu  857 Jun 28 04:49 index.html?C=M;O=A
-rw-rw-r-- 1 ubuntu ubuntu  857 Jun 28 04:53 index.html?C=M;O=D
-rw-rw-r-- 1 ubuntu ubuntu  857 Jun 28 04:53 index.html?C=N;O=A
-rw-rw-r-- 1 ubuntu ubuntu  857 Jun 28 04:49 index.html?C=N;O=D
-rw-rw-r-- 1 ubuntu ubuntu  857 Jun 28 04:49 index.html?C=S;O=A
-rw-rw-r-- 1 ubuntu ubuntu  857 Jun 28 04:53 index.html?C=S;O=D
-rw-rw-r-- 1 ubuntu ubuntu 4.0M Mar 29 00:30 itos_wt103.pkl

The pre-trained LM weights have an embedding size of 400, 1150 hidden units and just 3 layers. We need to match these values  with the target IMDB LM so that the weights can be loaded up._____no_output_____

em_sz, nh, nl = 400, 1150, 3_____no_output_____

Here are our pre-trained path and our pre-trained language model path._____no_output_____

PRE_PATH = PATH / 'models' / 'wt103'
PRE_LM_PATH = PRE_PATH / 'fwd_wt103.h5'_____no_output_____wgts = torch.load(PRE_LM_PATH, map_location=lambda storage, loc: storage)_____no_output_____

**Map IMDb vocab to WikiText vocab**_____no_output_____We calculate the mean of the layer0 encoder weights. This can be used to assign weights to unknown tokens when we transfer to target IMDB LM._____no_output_____

# ==================================== START DEBUG ====================================

# specialized container datatypes providing alternatives to Python’s general purpose built-in containers, dict.
# dict subclass that remembers the order entries were added.

# In other words,
# a regular dict does not track the insertion order, and iterating over it produces the values in an arbitrary order.
# In an OrderedDict, by contrast, the order the items are inserted is remembered and used when creating an iterator.
print(type(wgts))

print(len(wgts))
for k in wgts.keys():
    print(k)

tmp_wgts = wgts['0.encoder.weight']
print(f'\n{type(tmp_wgts)}' )

tmp_enc_wgts = to_np(tmp_wgts)
print( type(to_np(tmp_wgts)) )

# pre-trained LM weights have an embedding size of 400
print( tmp_enc_wgts.shape )

tmp_row_m = tmp_enc_wgts.mean(0) # returns the average of the array elements along axis 0
print( type(tmp_row_m) ) # numpy.ndarray
print( tmp_row_m.shape ) # shape: (400,)

# ==================================== END DEBUG ====================================<class 'collections.OrderedDict'>
15
0.encoder.weight
0.encoder_with_dropout.embed.weight
0.rnns.0.module.weight_ih_l0
0.rnns.0.module.bias_ih_l0
0.rnns.0.module.bias_hh_l0
0.rnns.0.module.weight_hh_l0_raw
0.rnns.1.module.weight_ih_l0
0.rnns.1.module.bias_ih_l0
0.rnns.1.module.bias_hh_l0
0.rnns.1.module.weight_hh_l0_raw
0.rnns.2.module.weight_ih_l0
0.rnns.2.module.bias_ih_l0
0.rnns.2.module.bias_hh_l0
0.rnns.2.module.weight_hh_l0_raw
1.decoder.weight

<class 'torch.FloatTensor'>
<class 'numpy.ndarray'>
(238462, 400)
<class 'numpy.ndarray'>
(400,)
enc_wgts = to_np(wgts['0.encoder.weight']) # converts np.ndarray from torch.FloatTensor.output shape: (238462, 400)
row_m = enc_wgts.mean(0) # returns the average of the array elements along axis 0. output shape: (400,)_____no_output_____itos2 = pickle.load( (PRE_PATH / 'itos_wt103.pkl').open('rb') )
stoi2 = collections.defaultdict(lambda: -1, { v: k for k, v in enumerate(itos2) })_____no_output_____# ==================================== START DEBUG ====================================

print( type(itos2) )
print(len(itos2))

i = 0
for k, v in enumerate(itos2):
    i = i + 1
    if i <= 10:
        print(f'{k}: {v}')
    else:
        break

print( type(stoi2) )
print( len(stoi2) )
print( stoi2['On'] ) # returns -1 (because the vocab is not found)
print( stoi2['the'] ) # returns 2

# ==================================== END DEBUG ====================================<class 'list'>
238462
0: _unk_
1: _pad_
2: the
3: ,
4: .
5: of
6: and
7: in
8: to
9: a
<class 'collections.defaultdict'>
238462
-1
2

Before we try to transfer the knowledge from WikiText to the IMDb LM, we match up the vocab words and their indexes. We use the `defaultdict` container once again, to assign mean weights to unknown IMDb tokens that do not exist in WikiText-103._____no_output_____

new_w = np.zeros((vs, em_sz), dtype=np.float32) # shape: (60002, 400)

for i, w in enumerate(itos):
    r = stoi2[w]
    new_w[i] = enc_wgts[r] if r >= 0 else row_m_____no_output_____

We now overwrite the weights into the `wgts` OrderedDict.
The decoder module, which we will explore in detail is also loaded with the same weights due to an idea called weight tying._____no_output_____

wgts['0.encoder.weight'] = T(new_w)
wgts['0.encoder_with_dropout.embed.weight'] = T(np.copy(new_w)) # weird thing with how we do embedding dropout
wgts['1.decoder.weight'] = T(np.copy(new_w))_____no_output_____

Now that we have the weights prepared, we are ready to create and start training our new IMDb language PyTorch model!_____no_output_____## Language model_____no_output_____It is fairly straightforward to create a new language model using the fastai library. Like every other lesson, our model will have a backbone and a custom head. The backbone in our case is the IMDb LM pre-trained with WikiText and the custom head is a linear classifier. In this section we will focus on the backbone LM and the next section will talk about the classifier custom head.

bptt (*also known traditionally in NLP LM as ngrams*) in fastai LMs is approximated to a std. deviation around 70, by perturbing the sequence length on a per-batch basis. This is akin to shuffling our data in computer vision, only that in NLP we cannot shuffle inputs and we have to maintain statefulness.

Since we are predicting words using ngrams, we want our next batch to line up with the end-points of the previous mini-batch's items. batch size is constant but the fastai library expands and contracts bptt each mini-batch using a clever stochastic implementation of a batch. (original credits attributed to [Smerity](https://twitter.com/jeremyphoward/status/980227258395770882))_____no_output_____

wd = 1e-7
bptt = 70
bs = 52
opt_fn = partial(optim.Adam, betas=(0.8, 0.99))_____no_output_____

The goal of the LM is to learn to predict a word/token given a preceeding set of words(tokens). We take all the movie reviews in both the 90k training set and 10k validation set and concatenate them to form long strings of tokens. In fastai, we use the `LanguageModelLoader` to create a data loader which makes it easy to create and use bptt sized mini batches. The  `LanguageModelLoader` takes a concatenated string of tokens and returns a loader.

We have a special modeldata object class for LMs called `LanguageModelData` to which we can pass the training and validation loaders and get in return the model itself._____no_output_____

trn_dl = LanguageModelLoader(np.concatenate(trn_lm), bs, bptt)
val_dl = LanguageModelLoader(np.concatenate(val_lm), bs, bptt)
md = LanguageModelData(PATH, 1, vs, trn_dl, val_dl, bs=bs, bptt=bptt)_____no_output_____

**Choosing dropout**_____no_output_____We setup the dropouts for the model - these values have been chosen after experimentation. If you need to update them for custom LMs, you can change the weighting factor (`0.7` here) based on the amount of data you have. For more data, you can reduce dropout factor and for small datasets, you can reduce overfitting by choosing a higher dropout factor. *No other dropout value requires tuning*._____no_output_____

drops = np.array([0.25, 0.1, 0.2, 0.02, 0.15]) * 0.7_____no_output_____

We first tune the last embedding layer so that the missing tokens initialized with mean weights get tuned properly. So we freeze everything except the last layer.

We also keep track of the *accuracy* metric._____no_output_____

learner = md.get_model(opt_fn, em_sz, nh, nl,
                       dropouti=drops[0], dropout=drops[1], wdrop=drops[2], dropoute=drops[3], dropouth=drops[4])

learner.metrics = [accuracy]
learner.freeze_to(-1)_____no_output_____

**Measuring accuracy**_____no_output_____

learner.model.load_state_dict(wgts)_____no_output_____

We set learning rates and fit our IMDb LM. We first run one epoch to tune the last layer which contains the embedding weights. This should help the missing tokens in the WikiText-103 learn better weights._____no_output_____

lr = 1e-3
lrs = lr_____no_output_____learner.fit(lrs / 2, 1, wds=wd, use_clr=(32, 2), cycle_len=1)_____no_output_____

Note that we print out accuracy and keep track of how often we end up predicting the target word correctly. While this is a good metric to check, it is not part of our loss function as it can get quite bumpy. We only minimize cross-entropy loss in the LM.

The exponent of the cross-entropy loss is called the perplexity of the LM. (low perplexity is better)._____no_output_____

learner.save('lm_last_ft')_____no_output_____learner.load('lm_last_ft')_____no_output_____learner.unfreeze()_____no_output_____learner.lr_find(start_lr=lrs / 10, end_lr=lrs * 10, linear=True)_____no_output_____learner.sched.plot()_____no_output_____learner.fit(lrs, 1, wds=wd, use_clr=(20, 10), cycle_len=15)_____no_output_____learner.sched.plot_loss()_____no_output_____

We save the trained model weights and separately save the encoder part of the LM model as well. This will serve as our backbone in the classification task model._____no_output_____

learner.save('lm1')_____no_output_____learner.save_encoder('lm1_enc')_____no_output_____# My extra codes
# Resume training
learner.load('lm1')_____no_output_____# My extra codes
# Resume training
learner.load_encoder('lm1_enc')_____no_output_____# My extra codes
# Resume training
learner.fit(lrs, 1, wds=wd, use_clr=(20, 10), cycle_len=13)_____no_output_____learner.sched.plot_loss()_____no_output_____learner.save('lm2')
learner.save_encoder('lm2_enc')_____no_output_____

## Classifier tokens_____no_output_____The classifier model is basically a linear layer custom head on top of the LM backbone. Setting up the classifier data is similar to the LM data setup except that we cannot use the unsup movie reviews this time.

_Note: we are using the CSV files created earlier which have already filtered out the unsup movie reviews.______no_output_____

df_trn = pd.read_csv(CLAS_PATH / 'train.csv', header=None, chunksize=chunksize)
df_val = pd.read_csv(CLAS_PATH /'test.csv', header=None, chunksize=chunksize)_____no_output_____tok_trn, trn_labels = get_all(df_trn, 1)
tok_val, val_labels = get_all(df_val, 1)0
1
0
1
(CLAS_PATH / 'tmp').mkdir(exist_ok=True)

np.save(CLAS_PATH / 'tmp' / 'tok_trn.npy', tok_trn)
np.save(CLAS_PATH /'tmp' / 'tok_val.npy', tok_val)

np.save(CLAS_PATH /'tmp' / 'trn_labels.npy', trn_labels)
np.save(CLAS_PATH /'tmp' / 'val_labels.npy', val_labels)_____no_output_____tok_trn = np.load(CLAS_PATH / 'tmp' / 'tok_trn.npy')
tok_val = np.load(CLAS_PATH / 'tmp' / 'tok_val.npy')_____no_output_____itos = pickle.load((LM_PATH / 'tmp' / 'itos.pkl').open('rb'))
stoi = collections.defaultdict(lambda: 0, { v: k for k, v in enumerate(itos) })
len(itos)_____no_output_____trn_clas = np.array([[stoi[o] for o in p] for p in tok_trn])
val_clas = np.array([[stoi[o] for o in p] for p in tok_val])_____no_output_____np.save(CLAS_PATH / 'tmp' / 'trn_ids.npy', trn_clas)
np.save(CLAS_PATH / 'tmp' / 'val_ids.npy', val_clas)_____no_output_____

## Classifier_____no_output_____Now we can create our final model, a classifier which is really a custom linear head over our trained IMDb backbone. The steps to create the classifier model are similar to the ones for the LM._____no_output_____

trn_clas = np.load(CLAS_PATH / 'tmp' / 'trn_ids.npy')
val_clas = np.load(CLAS_PATH / 'tmp' / 'val_ids.npy')_____no_output_____trn_labels = np.squeeze(np.load(CLAS_PATH / 'tmp' / 'trn_labels.npy'))
val_labels = np.squeeze(np.load(CLAS_PATH / 'tmp' / 'val_labels.npy'))_____no_output_____bptt, em_sz, nh, nl = 70, 400, 1150, 3
vs = len(itos) # num of tokens (vocab size)
opt_fn = partial(optim.Adam, betas=(0.8, 0.99))
bs = 48_____no_output_____min_lbl = trn_labels.min()
trn_labels -= min_lbl
val_labels -= min_lbl
c = int(trn_labels.max()) + 1 # num of class_____no_output_____

**Shuffle documents; Sort-ish to save computation**_____no_output_____In the classifier, unlike LM, we need to read a movie review at a time and learn to predict its sentiment as pos/neg. We do not deal with equal bptt size batches, so we have to pad the sequences to the same length in each batch. To create batches of similar sized movie reviews, we use a sortish sampler method invented by [@Smerity](https://twitter.com/Smerity) and [@jekbradbury](https://twitter.com/jekbradbury)

The `SortishSampler` cuts down the overall number of padding tokens the classifier ends up seeing._____no_output_____

trn_ds = TextDataset(trn_clas, trn_labels)
val_ds = TextDataset(val_clas, val_labels)_____no_output_____

**Turning it to DataLoader**_____no_output_____

trn_samp = SortishSampler(trn_clas, key=lambda x: len(trn_clas[x]), bs=bs // 2)
val_samp = SortSampler(val_clas, key=lambda x: len(val_clas[x]))_____no_output_____trn_dl = DataLoader(trn_ds, bs // 2, transpose=True, num_workers=1, pad_idx=1, sampler=trn_samp)
val_dl = DataLoader(val_ds, bs, transpose=True, num_workers=1, pad_idx=1, sampler=val_samp)_____no_output_____md = ModelData(PATH, trn_dl, val_dl)_____no_output_____

**Create RNN Encoder**_____no_output_____

# part 1
dps = np.array([0.4, 0.5, 0.05, 0.3, 0.1])_____no_output_____dps = np.array([0.4, 0.5, 0.05, 0.3, 0.4]) * 0.5_____no_output_____

`get_rnn_classifer` going to create exactly the same encoder more or less, and we are going to pass in the same architectural details as before. But this time, with the head we add on, you have a few more things you can do. One is you can add more than one hidden layer. In `layers = [em_sz * 3, 50, c]`:

- `em_sz * 3`: this is what the input to my head (i.e. classifier section) is going to be.
- `50`: this is the output of the first layer
- `c`: this is the output of the second layer_____no_output_____And you can add as many as you like. So you can basically create a little multi-layer neural net classifier at the end. Similarly, for `drops=[dps[4], 0.1]`, these are the dropouts to go after each of these layers._____no_output_____

m = get_rnn_classifer(bptt, 20 * 70, c, vs, emb_sz=em_sz, n_hid=nh, n_layers=nl, pad_token=1,
            layers=[em_sz * 3, 50, c], drops=[dps[4], 0.1],
            dropouti=dps[0], wdrop=dps[1], dropoute=dps[2], dropouth=dps[3])_____no_output_____opt_fn = partial(optim.Adam, betas=(0.7, 0.99))_____no_output_____learn = RNN_Learner(md, TextModel(to_gpu(m)), opt_fn=opt_fn)
learn.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)
learn.clip = 25.
learn.metrics = [accuracy]_____no_output_____lr = 3e-3
lrm = 2.6
lrs = np.array([lr / (lrm**4), lr / (lrm**3), lr / (lrm**2), lr / lrm, lr])_____no_output_____lrs = np.array([1e-4, 1e-4, 1e-4, 1e-3, 1e-2])_____no_output_____wd = 1e-7
wd = 0
learn.load_encoder('lm2_enc')_____no_output_____

We start out just training the last layer and we get 92.9% accuracy:_____no_output_____

learn.freeze_to(-1)_____no_output_____learn.lr_find(lrs / 1000)
learn.sched.plot()_____no_output_____learn.fit(lrs, 1, wds=wd, use_clr=(8, 3), cycle_len=1)_____no_output_____learn.save('clas_0')_____no_output_____learn.load('clas_0')_____no_output_____

Then we unfreeze one more layer, get 93.3% accuracy:_____no_output_____

learn.freeze_to(-2)_____no_output_____learn.fit(lrs, 1, wds=wd, use_clr=(8, 3), cycle_len=1)_____no_output_____learn.save('clas_1')_____no_output_____learn.load('clas_1')_____no_output_____

Then we fine-tune the whole thing. This was the main attempt before our paper came along at using a pre-trained model:_____no_output_____

learn.unfreeze()_____no_output_____learn.fit(lrs, 1, wds=wd, use_clr=(32, 10), cycle_len=14)_____no_output_____learn.sched.plot_loss()_____no_output_____learn.save('clas_2')_____no_output_____

The previous state of the art result was 94.1% accuracy (5.9% error). With bidir we get 95.4% accuracy (4.6% error)._____no_output_____## Fin_____no_output_____


# Notebook from eotp/python-FU-class
Path: 03-2021-05-07/notebooks/03-01_Python_101.ipynb

# Python 101 - An Introduction to Python_____no_output_____## 1. Objects

The most basic component of any programming language are "things", also called variables or
(in special cases) objects.

The most common basic "things" in Python are integers, floats, strings, booleans, and
some special objects of various types. We'll meet many of these as we go through the lesson._____no_output_______TIP:__ To run the code in a cell quickly, press Ctrl-Enter._____no_output_______TIP:__ To run the code in a cell and go to next cell, press Shift-Enter._____no_output_____

# A thing
2_____no_output_____# Use print to show multiple things in the same cell
# Note that you can use single or double quotes for strings
print(2)
print(5)
print("Hello")_____no_output_____a = 2
b = "Hello"
c = True
d = 2.0
e = [a,c,d]
print(a,b,c,d,e)_____no_output_____print(
    type(a),
    type(b),
    type(c),
    type(d),
    type(e),
     )_____no_output_____

## 2. Commands that operate on things

Just storing data in variables isn't much use to us. Right away, we'd like to start performing
operations and manipulations on data and variables.

There are three very common means of performing an operation on a thing._____no_output_____### 2.1 Use an operator

All of the basic math operators work like you think they should for numbers. They can also
do some useful operations on other things, like strings. There are also boolean operators that
compare quantities and give back a `bool` variable as a result._____no_output_____

a = 2
b = 3_____no_output_____# this is a comment
print(
    a+b,
    a*b,
    a**b,
    a/b,
    a//b
)_____no_output_____"Hello" + " World"_____no_output_____"Hello " * 4_____no_output_____# "Hello "**4_____no_output_____# Boolean operators compare two things
a = (1 > 3)
a_____no_output_____b = (3 == 3)
b_____no_output_____

### 2.2 Use a function

These will be very familiar to anyone who has programmed in any language, and work like you
would expect._____no_output_____

a = "hello world"_____no_output_____type(a)_____no_output_____

Only very few functions are available by default in the Python interpreter (``print()``, ``len()``, ``type()``, ``round()``, ...).
All other functions must be imported from modules. _____no_output_____

len('hello')_____no_output_____round(3.3)_____no_output_____#?round_____no_output_____round(3.14159,3)_____no_output_____import math_____no_output_____math.pi_____no_output_____import math as m_____no_output_____m.pi_____no_output_____from math import pi_____no_output_____pi_____no_output_____from math import *_____no_output_____sqrt(4)_____no_output_____math.sqrt(4)_____no_output_____m.sqrt(4)_____no_output_____

### 2.3 Use a method

Before we get any farther into the Python language, we have to say a word about "objects". We
will not be teaching object oriented programming in this workshop, but you will encounter objects
throughout Python (in fact, even seemingly simple things like ints and strings are actually
objects in Python).

In the simplest terms, you can think of an object as a small bundled "thing" that contains within
itself both data and functions that operate on that data. For example, strings in Python are
objects that contain a set of characters and also various functions that operate on the set of
characters. When bundled in an object, these functions are called "methods".

Instead of the "normal" `function(arguments)` syntax, methods are called using the
syntax `variable.method(arguments)`._____no_output_____

a_____no_output_____a.capitalize()_____no_output_____a.title()_____no_output_____a.upper()_____no_output_____a.replace("o", "--")_____no_output_____

__TIP:__ To find out what a function does, you can type it's name and then a question mark to
get a pop up help window. Shift-Tab for a tool-tip._____no_output_____## 3. Indexing and Slicing_____no_output_____

a = a.upper()
a_____no_output_____a[0:2]_____no_output_____a[2:]_____no_output_____a[:4]_____no_output_____a[::3]_____no_output_____a[-5:]_____no_output_____a[-1]_____no_output_____a[::-1]_____no_output_____

## 4. Collections of things

While it is interesting to explore your own height, in science we work with larger  slightly more complex datasets. In this example, we are interested in the characteristics and distribution of heights. Python provides us with a number of objects to handle collections of things.

Probably 99% of your work in scientific Python will use one of four types of collections:
`lists`, `tuples`, `dictionaries`, and `numpy arrays`. We'll look quickly at each of these and what
they can do for you._____no_output_____### 4.1 Lists

Lists are probably the handiest and most flexible type of container.

Lists are declared with square brackets [].

Individual elements of a list can be selected using the syntax `a[ind]`._____no_output_____

a = ["blueberry", "strawberry", "pineapple", 1, True]
type(a)_____no_output_____a[1]_____no_output_____a[-1]_____no_output_____a[::-1]_____no_output_____a_____no_output_____a[0] = "new fruit"_____no_output_____print(a)_____no_output_____a.append("a new thing")_____no_output_____a_____no_output_____a.pop()
a.pop()
a.pop()_____no_output_____a_____no_output_____a.sort()_____no_output_____a_____no_output_____a.reverse()_____no_output_____a_____no_output_____

> **Challenge:** Store a bunch of heights (in metres) in a list
1. Ask five people around you for their heights (in metres).
2. Store these in a list called `heights`.
3. Append your own height to the list.
4. Get the first height from the list and print it.
5. Compute the mean height and print it. _Hint: the function mean is available from the `statistics` module_
_____no_output_____

## your code here
_____no_output_____# %load ../src/_solutions/height.py_____no_output_____

### 4.2 Tuples

We won't say a whole lot about tuples except to mention that they basically work just like lists, with
two major exceptions:

1. You declare tuples using () instead of []
1. Once you make a tuple, you can't change what's in it (referred to as immutable)

You'll see tuples come up throughout the Python language, and over time you'll develop a feel for when
to use them.

In general, they're often used instead of lists:

1. to group items when the position in the collection is critical, such as coord = (x, y)
1. when you want to make prevent accidental modification of the items, e.g. shape = (12, 23)
1. when we need a *hashable* object (as key in a mapping/dict) (explained later)_____no_output_____

b = (1,2,3,4,5)
type(b)_____no_output_____b1 = [1,2,3,4,5]
type(b1)_____no_output_____b1[0] = 2
b1_____no_output_____#b[0] = 2_____no_output_____

### 4.3 Dictionaries

Dictionaries are the collection to use when you want to store and retrieve things by their names
(or some other kind of key) instead of by their position in the collection. A good example is a set
of model parameters, each of which has a name and a value. Dictionaries are declared using {}.

`key` $\to$ `value` pairs_____no_output_____

my_dict = {"Marry" : 22 , "Frank" : 33  }_____no_output_____my_dict_____no_output_____my_dict["Marry"]_____no_output_____my_dict["Frank"]_____no_output_____my_dict["Anne"] = 13_____no_output_____my_dict_____no_output_____my_dict["Anne"]_____no_output_____#my_dict["Heidi"]_____no_output_____my_dict.get("Heidi", "Danger no entry found!")_____no_output_____my_dict.items()_____no_output_____my_dict.keys()_____no_output_____my_dict.values()_____no_output_____

## 5. Repeating yourself

So far, everything that we've done could, in principle, be done by hand calculation. In this section
and the next, we really start to take advantage of the power of programming languages to do things
for us automatically.

We start here with ways to repeat yourself. The two most common ways of doing this are known as for
loops and while loops. For loops in Python are useful when you want to cycle over all of the items
in a collection (such as all of the elements of an array), and while loops are useful when you want to
cycle for an indefinite amount of time until some condition is met.

The basic examples below will work for looping over lists, tuples, and arrays. Looping over dictionaries
is a bit different, since there is a key and a value for each item in a dictionary. Have a look at the
Python docs for more information._____no_output_____### 5.1 __`for` Loops_______no_output_____

wordlist = ["hi", "hello", "by"]_____no_output_____import time
for word in wordlist:
    print(word + "!")
    time.sleep(1)
    print("-----------")
print("Done")_____no_output_____

**Note on indentation**: Notice the indentation once we enter the for loop.  Every idented statement after the for loop declaration is part of the for loop.  This rule holds true for while loops, if statements, functions, etc. Required identation is one of the reasons Python is such a beautiful language to read.

If you do not have consistent indentation you will get an `IndentationError`.  Fortunately, most code editors will ensure your indentation is correction.

__NOTE__ In Python the default is to use four (4) spaces for each indentation, most editros can be configured to follow this guide._____no_output_____> **Challenge**
* Sum all of the values in a collection using a for loop
`numlist = [1, 4, 77, 3]`_____no_output_____

## your code here ...
_____no_output_____# %load ../src/_solutions/sumlist.py_____no_output_____

> **Challenge**
* Combine items from two lists and print them as one string to the console
`name = ["John", "Ringo", "Paul", "George"]`
`surname = ["Lennon", "Star", "McCartney", "Harrison"]`_____no_output_____

name = ["John", "Ringo", "Paul", "George"]
surname = ["Lennon", "Star", "McCartney", "Harrison"]_____no_output_____## your code here ...
_____no_output_____# %load ../src/_solutions/list_combine1.py_____no_output_____# %load ../src/_solutions/list_combine2.py_____no_output_____# %load ../src/_solutions/list_combine3.py_____no_output_____

### 5.2 **`while` loop**_____no_output_____

# and want to stop once a certain condition is met.
step = 0
prod = 1
while prod < 100:
    step = step + 1
    prod = prod * 2
    print(step, prod)
    time.sleep(0.3)

print('Reached a product of', prod, 'at step number', step)_____no_output_____

## 6. Making choices

Often we want to check if a condition is True and take one action if it is, and another action if the
condition is False. We can achieve this in Python with an if statement.

__TIP:__ You can use any expression that returns a boolean value (True or False) in an if statement.
Common boolean operators are ==, !=, <, <=, >, >=. You can also use `is` and `is not` if you want to
check if two variables are identical in the sense that they are stored in the same location in memory._____no_output_____

x = 0

if x > 0:
    print("x is positive")
elif x < 0:
    print("x is negative")
else:
    print("x is zero")_____no_output_____

> **Challenge**
* Implement [FizzBuzz](https://en.wikipedia.org/wiki/Fizz_buzz)
* Game for children to learn about division
* Children sit in a circle and count upwards
* If number is divisible by 3
    * Say "Fizz"
* If number is divisible by 5
    * Say "Buzz"
* If neither
    * Say the number_____no_output_____> _**Hint**: the [modulo operator](https://en.wikipedia.org/wiki/Modulo_operation) `%` can tell you if a number is divisible by another, e.g.:______no_output_____

print(10 % 2)
print(10 % 3)
print(10 % 4)
print(10 % 5)_____no_output_____### your code here ...
_____no_output_____# %load ../src/_solutions/fizzbuzz.py_____no_output_____

## 7. IO_____no_output_____Write a text file_____no_output_____

f = open("../data/interim/my_file.txt", "w")
for i in range(5):
    f.write("Line {}\n".format(i))
f.close()_____no_output_____# using a context manager
with open("../data/interim/my_file.txt", "a") as f:
    for i in range(100,105):
        f.write("LINE {}\n".format(i))_____no_output_____

Read a file_____no_output_____

with open ("../data/interim/my_file.txt", "r") as f:
    print(f.read())_____no_output_____

>**Challenge**
* Extract the numerical values of the file `my_file.txt` into a list of floating point values._____no_output_____

my_storage = [] # list
with open ("../data/interim/my_file.txt", "r") as f:
    ## your code here ...
    pass

my_storage_____no_output_____# %load ../src/_solutions/read_numbers_from_file.py_____no_output_____

## 8 Functions (UDFs)_____no_output_____

def my_func(a,b,c=10):
    rv = (a-b)*c
    return rv_____no_output_____my_result = my_func(a=1, b=2)
my_result_____no_output_____

> **Challenge**
* Write a function that computes Kelvin from Fahrenheit (`fahrenheit_to_kelvin`)
* Write a function that computes Celsius from Kelvin (`kelvin_to_celsius`)
* Write a function that computes Celsius form Fahrenheit (`fahrenheit_to_celsius`); Resue the two functions from above.

`fahrenheit_to_kelvin` $$K = (°F - 32.0) \times \frac{5}{9} + 273.15  $$

`kelvin_to_celsius` $$°C = K - 273.15  $$_____no_output_____#### Fahrenheit from Kelvin_____no_output_____

def fahrenheit_to_kelvin(F):
    """
    Function to compute Fahrenheit from Kelvin
    """
    ## your code here ...
    pass_____no_output_____

> __Test your function__:
`fahrenheit_to_kelvin(-50)` should yield approx 228 K_____no_output_____

#fahrenheit_to_kelvin(-50)_____no_output_____# %load ../src/_solutions/fahrenheit_to_kelvin.py_____no_output_____

####  Celsius from Kelvin_____no_output_____

def kelvin_to_celsius(K):
    '''
    Function to compute Celsius from Kelvin
    '''
    ## your code here ...
    pass_____no_output_____

> __Test your function__:
`kelvin_to_celsius(0)` should yield approx -273.15 degC_____no_output_____

#kelvin_to_celsius(0)_____no_output_____# %load ../src/_solutions/kelvin_to_celsius.py_____no_output_____

#### Celsius from Fahrenheit_____no_output_____

def fahrenheit_to_celsius(F):
    '''
    Function to compute Celsius from Fahrenheit
    '''
    ## your code here ...
    pass_____no_output_____

> __Test your function__:
`fahrenheit_to_celsius(451)` should yield approx 232 degC_____no_output_____

#fahrenheit_to_celsius(451)_____no_output_____# %load ../src/_solutions/fahrenheit_to_celsius.py_____no_output_____

### Code refactoring_____no_output_____

# %load ../src/temperature_module.py_____no_output_____

#### Add path to look for modules_____no_output_____

import sys
sys.path.append("../src")_____no_output_____

#### Load own module_____no_output_____

import temperature_module as tm_____no_output_____tm.kelvin_to_celsius(100)_____no_output_____tm.fahrenheit_to_celsius(100)_____no_output_____tm.fahrenheit_to_kelvin(100)_____no_output_____

## 9. Final challenge

### Build rock scissors paper

> __Task__
Implement the classic children's game Rock-paper-scissors, as well as a simple predictive AI (artificial intelligence) player.

> Rock Paper Scissors is a two player game.
> Each player chooses one of rock, paper or scissors, without knowing the other player's choice.

> The winner is decided by a set of rules:
>  * Rock beats scissors
>  * Scissors beat paper
>  * Paper beats rock

> If both players choose the same thing, there is no winner for that round. If you don't konw the rules you may find them [here](https://en.wikipedia.org/wiki/Rock%E2%80%93paper%E2%80%93scissors).
> For this task, the computer will be one of the players.

> The operator will select Rock, Paper or Scissors and the computer will keep a record of the choice frequency, and use that information to make a weighted random choice in an attempt to defeat its opponent.
> Consider the function `input()` to ask for user input. Try to implement an exit rule as well.
_____no_output_____

## your code here ...
_____no_output_____# %load ../src/_solutions/rock_paper_scissors.py_____no_output_____

***_____no_output_____