# Notebook from Guillaume-Garrigos/inverse_problems
Path: notebook/lasso_1-regularisation_path.ipynb

$\newcommand{\RR}{\mathbb{R}}$
$\newcommand{\Nn}{\mathcal{N}}$
$\newcommand{\EE}{\mathbb{E}}$_____no_output_____

%matplotlib inline
from ipywidgets import interactive
import numpy as np
from numpy import linalg as la
import matplotlib.pyplot as plt_____no_output_____# Uncoment the pip command below if you are using Google Colab.
# In case of errors, use the option --upgrade --force-reinstall

#!pip install git+https://github.com/Guillaume-Garrigos/invprob.git  # For Google Colab
import sys  # For local launch
sys.path.append('..')  # For local launch
import invprob.sparse as sparse_____no_output_____np.random.seed(seed=78) # Seed for np.random
dpi = 100 # Resolution used for plotting (230 for small screen, 100 for large one)_____no_output_____

# Notes on Inverse Problems
(c) Guillaume Garrigos. This notebook is rendered as a static page on [nbviewer](https://nbviewer.jupyter.org/github/Guillaume-Garrigos/invprob/blob/master/notebook/lasso_1-regularisation_path.ipynb), and as an interactive notebook on [Google Colab](https://colab.research.google.com/github/Guillaume-Garrigos/invprob/blob/master/notebook/lasso_1-regularisation_path.ipynb)_____no_output_____## I. Sparse Regularisation with the L1 norm (LASSO) : the regularization path

We try here to recover a ground truth signal $x_0\in \RR^N$, being given a finite number of linear measurments:

\begin{equation}
y = \Phi x
\end{equation}

where $\Phi \in \RR^{M\times N}$ and $y \in \RR^M$.

To do so, we are going to exploit a prior assumption on $x_0$, which is that it is a *sparse* vector, in the sense that it has numerous zero entries.
More precisely, we assume that $x_0$ is the solution of the following optimisation problem

\begin{equation}
\min\limits_{y = \Phi x} \ \Vert x \Vert_1 , \hspace{1cm} \text{ where } \Vert x \Vert_1 := \sum_{i=1}^N \vert x_i \vert.
\tag{$P_0$}
\end{equation}_____no_output_____### 1. Presentation of the problem_____no_output_____We define $\Phi$, $x_0$ and $y$._____no_output_____

# We start by defining the characteristics of the problem: dimensions, sparsity level, etc.
data_size = 100
data_number = round(data_size/2)
sparsity_level = 10

# We define the main components of our problem
Phi = np.random.randn(data_number,data_size)
x0 = np.sign(sparse.randn(data_size,1,sparsity_level))
y = Phi@x0_____no_output_____

Let's compare the ground truth $x_0$ with the pseudo inverse solution of the linear system $\Phi x = y$, that is, $x^\dagger := \Phi^\dagger y$._____no_output_____

x_pinv = la.lstsq(Phi, y, rcond=None)[0]
_ = plt.figure(dpi=dpi)
sparse.stem(x0, "C0", "ground truth")
sparse.stem(x_pinv, "C1", "pinv solution")
plt.show()_____no_output_____

We see that even if $x^\dagger$ is a solution of the linear system, it is far from $x_0$, and doesn't share at all the same sparse structure.
One could argue here that $x^\dagger$ has high intensities on the support of $x_0$, and so that it would be sufficient to pass it trough a high-pass filter.
But this is not true everywhere, and $x^\dagger$ might have quite high intensities elsewhere, implying that setting up the right threshold would be a hard problem.

To recover $x_0$ from $y$ we would need to solve ($P_0$), but this is a hard optimisation problem: it consists in minimising a nonsmooth function over a constraint onto which it is expensive to project on.

Instead, we are going to look for an approximation of $x_0$, considering solutions coming from a *regularisation method*. _____no_output_____### 2. The penalised problem

In this example, we are going to consider a variational regularisation method, based on *penalisation*, like the Tikhonov regularisation method.
The idea is quite simple: instead of solving ($P_0$), we are going to solve an other problem which can be seen as a perturbation of ($P_0$).
More concretely, we are going to relax the constraint $\Phi x= y$ and replace it by a penalisation term:

\begin{equation}
\min\limits_{x \in \RR^N} f_\lambda(x):= \Vert x \Vert_1 + \frac{1}{2 \lambda}\Vert \Phi x - y \Vert^2.
\tag{$P_\lambda$}
\end{equation}

The rationale behind this is that, when $\lambda \sim 0$, the solutions of ($P_\lambda$) must verify $\Phi x_\lambda \sim y$  "first", and also having a low $\ell^1$ norm, which makes them a good approximation of $x_0$.
This problem is called the [LASSO](https://en.wikipedia.org/wiki/Lasso_(statistics)) in certain comunities, in particular in statistics and compressed sensing._____no_output_____What we need now is to find a solution of ($P_\lambda$), for a given (small) parameter $\lambda$.
For this, we are going to use the [Forward-Backward algorithm](https://en.wikipedia.org/wiki/Forward%E2%80%93backward_algorithm).
Given an initialization $x^0 \in \RR^N$ and a stepsize $0 < \tau < \frac{2}{\Vert \Phi^* \Phi \Vert}$, the Forward-Backward algorithm writes as follows:

\begin{equation}
x^{k+1} = \text{Soft}_{\lambda \tau}(x^k - \tau \Phi^*(\Phi x^k - y))
\end{equation}

where $\text{Soft}_{\alpha} : \RR^N \rightarrow \RR^N$ is the Soft-Thresholding operator, defined for all $\alpha > 0$ by

\begin{equation}
\text{Soft}_{\alpha}(x) := \left(\text{soft}_{\alpha}(x_i)\right)_{i=1,..,N}, \quad \text{ where } \text{soft}_{\alpha}(x_i) =
\begin{cases}
0 & \text{ if } \vert x_i \vert \leq \alpha \\
x - \text{sgn}(x_i) \lambda & \text{ if } \vert x_i \vert \geq \lambda
\end{cases}
\end{equation}

_____no_output_____

def fb_lasso(A, y, reg_param, iter_nb, x_ini=None, verbose=False):
    ''' Use the Forward-Backward algorithm to find a minimizer of:
             reg_param*norm(x,1) + 0.5*norm(Ax-y,2)**2
        Eventually outputs the functional values and support of the iterates
        while running the method
    '''
    # Manage optional input/output
    if verbose:  # Optional output
        regret = np.zeros(iter_nb)
        support = np.zeros(iter_nb)
        details = {
            "function_value": regret,
            "iterate_support": support
        }
    if x_ini is not None:
        x = x_ini
    else:
        x = np.zeros((A.shape[1], 1))

    # The core of the algorithm
    stepsize = 0.8 * 2/la.norm(A, 2)**2
    for k in range(iter_nb):
        x = x - stepsize * A.T@(A@x - y)
        x = sparse.soft_thresholding(x, reg_param*stepsize)
        if verbose:
            regret[k] = 0.5*la.norm(A@x-y, 2)**2 + reg_param*la.norm(x, 1)
            support[k] = sparse.norm0(x)

    # Output
    if verbose:
        return x, details
    else:
        return x_____no_output_____reg_param = 0.1  # We take lambda close to zero. Try to change it!
iter_nb = 4000
x_reg, details = fb_lasso(Phi, y, reg_param, iter_nb, verbose=True)

_ = plt.figure(dpi=dpi)
sparse.stem(x0, "C0", "ground truth")
sparse.stem(x_reg, "C1", "reg solution")
plt.show()_____no_output_____

We hardly see the difference between $x_\lambda$ and $x_0$ when $\lambda =$0.1.
The only noticeable difference is the apparition of spikes having a negligeable amplitude.

We can look more into details to iterates generated by the algorithm.
In particular, we see that the support of the iterates is essentially decreasing (not monotonically though), until a point where it reaches a stable support._____no_output_____

plt.figure(dpi=dpi, figsize=(12, 4))

plt.subplot(1, 2, 1)
plt.title(r"Evolution of $f_\lambda(x^k)$")
plt.semilogy(details.get("function_value"))

plt.subplot(1, 2, 2)
plt.title(r"Evolution of #supp$(x^k)$")
plt.plot(details.get("iterate_support"))
plt.show()_____no_output_____

### 3. The regularisation path

The set

$$\{x_\lambda \ | \ x_\lambda \text{ is a solution of } (P_\lambda), \lambda > 0 \}$$

is called the *regularization path*, and $\lambda$ is generally referred to as the *regularisation parameter*.
As we saw before, $x_\lambda \to x_0$ when $\lambda \to 0$.
Here we will try to visualise the set of regularised solutions.

First, here is a precalculated animation showing $x_\lambda$ for varying values of $\lambda$.
See how $x_\lambda$ starts from zero (when $\lambda$ is large), and then converges to $x_0$ by progressively getting spikes "growing".
See also how additional spikes appear and then tend to vanish along time.
![reg_path_noiseless](../output/L1_reg/reg_path_noiseless.gif "reg_path_noiseless")

Now, if you want to compute yourself these solutions, you just need to use the algorithm introduced above.
Here is some code to do so._____no_output_____

def compute_solution(reg_param):
    ''' Given a regularisation parameter, we compute the corresponding
        regularized solution with the FB algorithm. The number of
        iterations to use is customly chosen.
    '''
    reg_param = round(reg_param,1)  # For display purposes

    # Choice of the stopping criterion
    if reg_param > 10:
        iter_nb = 200
    else:
        if reg_param > 1:
            iter_nb = 1000
        else:
            iter_nb = 3000
    # We get the solution
    x_reg = fb_lasso(Phi, y, reg_param, iter_nb)
    # We plot it
    _ = plt.cla()
    _ = plt.title(r"Regularisation parameter $\lambda=$" + str(reg_param))
    sparse.stem(x0, "C0","ground truth")
    sparse.stem(x_reg, "C1", "reg solution")
    _ = plt.show()_____no_output______ = plt.figure(dpi=dpi, figsize=(12, 6))
interactive(compute_solution,reg_param=(1,100,1))_____no_output_____


# Notebook from inferential/BIO-BTE-06-L-7
Path: Notebooks/NB06b_Data_Access_And_Quality_Control_BN.ipynb

#r "nuget: FSharp.Stats, 0.4.0"
#r "nuget: BioFSharp, 2.0.0-beta5"
#r "nuget: BioFSharp.IO, 2.0.0-beta5"
#r "nuget: Plotly.NET, 2.0.0-beta8"
#r "nuget: BIO-BTE-06-L-7_Aux, 0.0.8"
#r "nuget: Deedle, 2.3.0"
#r "nuget: ISADotNet, 0.2.4"
#r "nuget: ISADotNet.XLSX, 0.2.4"

#r "nuget: Plotly.NET.Interactive, 2.0.0-beta8"

open System.IO
open ISADotNet
open ISADotNet.API
open Deedle
open BioFSharp
open FSharpAux
open FSharp.Stats
open Plotly.NET
open FSharp.Stats.Fitting.LinearRegression.OrdinaryLeastSquares.Linear
open System.IO
open BIO_BTE_06_L_7_Aux.FS3_Aux
open BIO_BTE_06_L_7_Aux.Deedle_Aux_____no_output_____

# NB06b Data Access and Quality Control

[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/CSBiology/BIO-BTE-06-L-7/gh-pages?filepath=NB06b_Data_Access_And_Quality_Control_BN.ipynb)

[Download Notebook](https://github.com/CSBiology/BIO-BTE-06-L-7/releases/download/NB06b_NB06b_NB06c_NB06c_NB06d_NB06d/NB06b_Data_Access_And_Quality_Control_BN.ipynb)

With this notebook, we want to converge the threads of computational and experimental proteomics by analyzing the data measured by you during the practical course.
Behind the scenes there was already a lot going on! While you were going through the hands-on tutorials addressing single steps of the computation proteomics pipeline, we executed
a pipeline combining all the steps. In this tutorial, we will provide you with the output of the pipeline, your sample description and a lot of code, which should help you explore the
data set generated by you! Since the experiment is of a non neglectable depth - so is the code needed to analyze it. Do not hesitate if you do not understand every step right away, this is no
trivial task and nontrivial are the means to achieve it! We included some questions along the way. The questions aim to check your understanding, but they can also serve as food for thought.

For the explorative data analysis, we are using [Deedle](http://bluemountaincapital.github.io/Deedle/tutorial.html).
Deedle is an easy to use library for data and time series manipulation and for scientific programming.
It supports working with structured data frames, ordered and unordered data, as well as time series. Deedle is designed to work well for exploratory programming using F#.

## I. Reading the sample description

Before we analyze our data, we will download and read the sample description provided by the experimentalist._____no_output_____

let directory = __SOURCE_DIRECTORY__
let path2 = Path.Combine[|directory;"downloads/alle_Gruppen_V11_SWATE.xlsx"|]
downloadFile path2 "alle_Gruppen_V11_SWATE.xlsx" "bio-bte-06-l-7"

let _,_,_,myAssayFile = XLSX.AssayFile.AssayFile.fromFile path2
let inOutMap = BIO_BTE_06_L_7_Aux.ISA_Aux.createInOutMap myAssayFile_____no_output_____

Next, we will prepare functions to look up parameters, which might be needed for further calculations. _____no_output_____

let normalizeFileName (f:string) = if Path.HasExtension f then f else Path.ChangeExtension(f, "wiff")

//
let getStrain (fileName:string) =
    let fN = fileName |> normalizeFileName
    BIO_BTE_06_L_7_Aux.ISA_Aux.tryGetCharacteristic inOutMap "Cultivation -Sample preparation" "strain" fN myAssayFile
    |> Option.defaultValue ""

//
let getExpressionLevel (fileName:string) =
    let fN = fileName |> normalizeFileName
    BIO_BTE_06_L_7_Aux.ISA_Aux.tryGetCharacteristic inOutMap "Cultivation -Sample preparation" "gene expression" fN myAssayFile
    |> Option.defaultValue "Wt-Like"

//
let get15N_PS_Amount (fileName:string) =
    let fN = fileName |> normalizeFileName
    BIO_BTE_06_L_7_Aux.ISA_Aux.tryGetCharacteristic inOutMap "Extraction" "gram #2" fN myAssayFile |> Option.defaultValue ""
    |> String.split ' '
    |> Array.head
    |> float
//
let getGroupID (fileName:string) =
    let fN = fileName |> normalizeFileName
    BIO_BTE_06_L_7_Aux.ISA_Aux.tryGetParameter inOutMap "Extraction" "Group name" fN myAssayFile |> Option.defaultValue ""
    |> int_____no_output_____

A quick execution to test the retrieval of data from the isa sample table:_____no_output_____

getStrain "20210312BN2_U1.wiff"
getExpressionLevel "20210312BN2_U1.wiff"
get15N_PS_Amount "20210312BN2_U1.wiff"
getGroupID "20210312BN2_U1.wiff"_____no_output_____

Now that we have the sample sheet, all that is missing is the data to be analyzed:_____no_output_____

let path = Path.Combine[|directory; "downloads/Quantifications_bn_annotated_replaced.txt"|]
downloadFile path "Quantifications_bn_annotated_replaced.txt" "bio-bte-06-l-7"_____no_output_____

## II. Raw data access using Deedle:
As teasered in the primer, we want to work with our tabular data using Deedle. Luckily, Deedle does not only deliver data frame and series
manipulation, but also allows us to quickly read the recently downloaded data into the memory:_____no_output_____

let rawData = Frame.ReadCsv(path, separators = "\t")_____no_output_____

To visualize the data, we can call the "formatAsTable" function. The preview of visual studio code does not allow
for the charts to be scrollable, so we pipe the output into "Chart.Show", to visualize the data in your browser._____no_output_____

rawData
|> Frame.take 10
|> formatAsTable 1500.
|> Chart.Show
_____no_output_____

Looking at the raw data, we can see that each row contains a different quantification of a peptide ion, with the columns containing
a single ion feature each, such as peptide ion charge, sequence or a quantification value reported for a file (e.g. light, heavy or ratio).
Since the columns ProteinGroup, StringSequence, PepSequenceID and Charge uniquely identify a row, we can use these to index the rows.
For this, we use a language feature called ["anonymous record type"](https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/anonymous-records).
Here we create a tuple like structure, with the additional feature that each element of the tuple is named (e.g.: Proteingroup)._____no_output_____

let indexedData =
    rawData
    // StringSequence is the peptide sequence
    |> Frame.indexRowsUsing (fun os ->
        {|
            ProteinGroup    = os.GetAs<string>("ProteinGroup");
            Synonyms        = os.GetAs<string>("Synonyms")
            StringSequence  = os.GetAs<string>("StringSequence");
            PepSequenceID   = os.GetAs<int>("PepSequenceID");
            Charge          = os.GetAs<int>("Charge")
        |}
    )
    _____no_output_____// The effect of our frame manipulation can be observed:
indexedData
|> Frame.take 10
|> formatAsTable 1500.
|> Chart.Show
_____no_output_____

## III. Augmenting and filtering the data frame
The data frame already contains all information needed to perform the analysis, but it could still benefit from
some quality-of-life upgrades. Say, we want to encode the specific qConcat protein as a separate feature:_____no_output_____

// Why does it make sense to model Qprots using this type, why do we not simply use a string?

let finalRaw =
    indexedData
    // What does this line filter for? Why does this make sense for our analysis?
    // How many peptide ions did the filter remove?
    |> Frame.filterRows (fun k s -> k.Synonyms <> "" && k.ProteinGroup |> String.contains "QProt_newPS")

finalRaw
|> Frame.filterRows (fun x s -> x.StringSequence = "DTDILAAFR")
|> Frame.getCol "20210312BN2_U1.Ratio"
|> Series.mapValues string_____no_output_____finalRaw
|> Frame.take 10
|> formatAsTable 1500.
|> Chart.Show
_____no_output_____

## IV. Global quality control.

With our data frame prepared, we want to see on a global scale if our experiment worked.
We plot the overall mean of the 14N and 15N quantifications and observe if we can recover our dilution series (15N),
while keeping the analyte to be quantified at a constant level (14N).

Since it comes in handy to simplify the data frame, we will only keep columns that contain a specific identifier,
such as, "Ratio", "Light" or "Heavy". _____no_output_____

let sliceQuantColumns quantColID frame =
    frame
    |> Frame.filterCols (fun ck os -> ck |> String.contains ("." + quantColID))
    |> Frame.mapColKeys (fun ck -> ck.Split('.') |> Array.item 0)

// How did the data frame change, how did the column headers change?
let ratios = sliceQuantColumns "Ratio" finalRaw
let light = sliceQuantColumns "Light" finalRaw
let heavy = sliceQuantColumns "Heavy" finalRaw

ratios
|> Frame.filterRows (fun x s -> x.StringSequence = "DTDILAAFR")
|> Frame.getCol "20210312BN2_U1"
|> Series.mapValues string_____no_output_____ratios
|> Frame.take 10
|> formatAsTable 1500.
|> Chart.Show
_____no_output_____

A nice tool to explore and compare distributions of different populations is the representation as a boxplot!
To use this tool, we will define a function which creates a boxplot for every column (file) of our data set:_____no_output_____

let createBoxPlot f =
    f
    |> Frame.getNumericCols
    |> Series.map (fun k s ->
         let x,y =
            s
            |> Series.values
            |> Seq.map (fun values -> k,values)
            |> Seq.unzip
         Chart.BoxPlot(x, y, Orientation = StyleParam.Orientation.Vertical)
         )
    |> Series.values
    |> Chart.Combine
    |> Chart.withY_AxisStyle "Ion intensity"_____no_output_____

The function applied to the n14 values: _____no_output_____

// How is the data distributed?
light
|> createBoxPlot_____no_output_____

The function applied to the n15 values:_____no_output_____

// Can you recover the dilution series?
heavy
|> createBoxPlot_____no_output_____

The following function performs a normalization which accounts for a specific effect. Can you
determine what the function accounts for?_____no_output_____

let normalizePeptides f =
    f
    |> Frame.transpose
    |> Frame.getNumericCols
    |> Series.mapValues (fun s ->
        let m = Stats.median s
        s / m
        )
    |> Frame.ofColumns
    |> Frame.transpose_____no_output_____// How does the distribution of the date change, when the normalization is applied?
light
|> normalizePeptides
|> createBoxPlot_____no_output_____heavy
|> normalizePeptides
|> createBoxPlot _____no_output_____

Finally we have a look at the ratios. _____no_output_____

// Does it make sense to normalize the ratios the same way?
ratios
|> createBoxPlot _____no_output_____

## Local quality control

Now that we know on a global scale how our experiment worked, it might be time to have a look at the details.
First, we want to write a function that allows us to plot all peptides of a protein vs. the dilution used. This way we can identify peptides that
we want to use and those that seem to be prone to error and should thus be discarded.
To keep things simple, we apply a filter step at the beginning, which only keeps peptides belonging to one protein and samples measured by one group
in the data frame. What are the sources of error? Which peptides do you think should be discarded and why? Which proteins need to be analyzed with extra care?
Hint: you can hover over the data points to get insight into the file name and gene expression pattern of the corresponding strain._____no_output_____

// With this type we create an alias to our row key, this allows us to write functions, which operate on data frames such as 'plotPeptidesOf','discardPeptideIonInFile' and 'discardPeptideIon'
type PeptideIon =
    {|
        ProteinGroup    : string
        Synonyms        : string
        StringSequence  : string
        PepSequenceID   : int
        Charge          : int
    |}

// Given a frame, a prot-ID and a group-ID this function creates an xy plot for every peptide ion belonging to the protein/proteingroup.
// The parameter 'prot' can either be given a valid Cre-ID or a synonym.
// What is the unit of the x-Axis? How is the ratio calculated?
let plotPeptidesOf (ratios : Frame<PeptideIon,string>) (prot : string) (groupID : int) =
    try
        ratios
        |> Frame.filterRows (fun k s -> k.Synonyms.Contains prot || k.ProteinGroup.Contains prot)
        |> Frame.filterCols (fun k s -> getGroupID k = groupID)
        |> Frame.transpose
        |> Frame.getNumericCols
        |> Series.map (fun pep (values) ->
            let fileNames,ratios,fileLabel =
                values
                |> Series.map (fun fileName (ratio) ->
                        let expressionLevel = getExpressionLevel fileName
                        fileName, ratio, (sprintf "%s %s" fileName expressionLevel)
                    )
                |> Series.values
                |> Seq.unzip3
            Chart.Point(fileNames, ratios, Labels = fileLabel)
            |> Chart.withTraceName (sprintf "S:%s_C:%i" pep.StringSequence pep.Charge)
            |> Chart.withX_AxisStyle("File name")
            |> Chart.withY_AxisStyle("Ratio")
        )
        |> Series.values
        |> Chart.Combine
        |> Chart.withTitle "BN-PAGE extraction"
        |> Chart.withSize (600.,700.)
        |> Chart.withMarginSize (Bottom = 175.)
    with :? System.ArgumentException -> failwith "ERROR: Input protein was not found."_____no_output_____

First we get an overview of available protein ids._____no_output_____

ratios.RowKeys
|> Array.ofSeq
|> Array.map (fun k -> k.Synonyms)
|> Array.distinct_____no_output_____

Then we can start to visualizes our results:_____no_output_____

plotPeptidesOf ratios "rbcL" 1_____no_output_____plotPeptidesOf ratios "RBCS2;RBCS1" 2_____no_output_____plotPeptidesOf ratios "SEBP1" 4_____no_output_____plotPeptidesOf ratios "TRK1" 1 _____no_output_____plotPeptidesOf ratios "PRK1" 1_____no_output_____// Describe what happened with the last 3 plots._____no_output_____

Since we want to save our result and use it for the next notebook, where we will have a look at the isotopic labeling efficiency and finally calculate absolute protein amounts, we
need to save the filtered frame. Additionally, we want to keep information which was dropped along the way: isotopic patterns. In order to do so, we perform a join operation, which keeps only those rows
present in both files:
_____no_output_____

//  Are there redundant columns in the result frame? Why?
let frameComplete =
    Frame.join JoinKind.Inner finalRaw ratios_____no_output_____

With the plots at hand, we can use the following functions to manipulate the data frame and discard peptides and/or whole files which we do not want to use for
an absolute protein quantification e.g.:_____no_output_____

let discardPeptideIonInFile stringsequence charge filename (ratios:Frame<PeptideIon,string>) =
    ratios
    |> Frame.map (fun r c value ->
        let cFileName = String.split '.' c |> Array.head
        if r.StringSequence = stringsequence && r.Charge = charge && cFileName = filename then nan else value
    )

let discardPeptideIon stringsequence charge (ratios:Frame<PeptideIon,string>) =
    ratios
    |> Frame.filterRows (fun r s -> (r.StringSequence = stringsequence && r.Charge = charge) |> not)_____no_output_____

These functions can then be used to create an updated version of the frame, containing only the values we want to use for quantification e.g.:_____no_output_____

let filtered =
    frameComplete
    |> discardPeptideIonInFile "IYSFNEGNYGLWDDSVK" 3 "Gr2rbcL2_5"
    |> discardPeptideIon "IYSFNEGNYGLWDDSVK" 2_____no_output_____

Of course, it is possible to apply very strict additional filters onto the previously filtered frame:_____no_output_____

let ratiosFiltered =
    filtered
    |> Frame.filterCols (fun k s ->
        let kFileName = String.split '.' k |> Array.head
        try
            get15N_PS_Amount kFileName > 0.1
        with
        | _ -> true
    )_____no_output_____

This frame can then be saved locally using the following pattern:    _____no_output_____

let frameToSave =
    ratiosFiltered
    |> Frame.indexRowsOrdinally

// frameToSave.SaveCsv(@"C:\YourPath\testOut.txt", separator = '\t', includeRowKeys = false)
frameToSave.SaveCsv(System.IO.Path.Combine [|__SOURCE_DIRECTORY__; "downloads"; "qualityControlResult_BN.txt"|], separator = '\t', includeRowKeys = false)_____no_output_____


# Notebook from camponogaraviera/qiskit-metal
Path: docs/circuit-examples/A.Qubits/11-Star_shaped_qubit.ipynb

# Star shaped qubit

We'll be creating a 2D design and adding a star shaped qubit QComponent.

There's a central star shaped component. This component is essentially a circle with trpezoids subtracted from it. A T-junction qubit is attached to this central shape. Then there are four coupling resonators and one readout resonator present in the default setting. The shape of the central component and the number coupling resonators are customizable. Let's try out a few diferent ways to create and render this component._____no_output_____

# So, let us dive right in. For convenience, let's begin by enabling
# automatic reloading of modules when they change.
%load_ext autoreload
%autoreload 2_____no_output_____

Next, we import the relevant modules. You can add new modules as and when needed. _____no_output_____

import qiskit_metal as metal
import numpy as np
from math import *
from qiskit_metal import designs, draw, Dict
from qiskit_metal.qlibrary.core import BaseQubit, QComponent_____no_output_____

Next we import the GUI_____no_output_____

from qiskit_metal import MetalGUI_____no_output_____

A QDesign class must be instantiated each time a new quantum circuit design is created. The design class `DesignPlanar` is best for 2D circuit designs._____no_output_____

design = designs.DesignPlanar()
gui = MetalGUI(design)
gui.rebuild() _____no_output_____

# A Star Qubit

You can create a ready-made star qubit from the QComponent Library, qiskit_metal.qlibrary.qubits. star_qubit.py is the file containing our qubit so StarQubit is the module we import.
_____no_output_____

from qiskit_metal.qlibrary.qubits.star_qubit import StarQubit_____no_output_____# The following default_options can be overridden by user.
StarQubit.get_template_options(design)_____no_output_____# To force overwrite a QComponent with an existing name.
# This is useful when re-running cells in a notebook.
design.overwrite_enabled = True_____no_output_____

Let us place the qubit at (x,y) =(5,5), and change the default rotation values. As this design is meant to have up to 5 contacts including one readout and up to 4 coupling resonators, these angles should be 72 degrees apart. However, the number of connectors may be changed to any value between 0 and 4._____no_output_____

qubit_options = dict(
    pos_x = '5um',
    pos_y = '5um',
    number_of_connectors = 4 # Change the number of connectors
)

# Create a new Concentric Transmon object with name 'Q1'
q1 = StarQubit(design, 'Star', options=qubit_options)

gui.rebuild()  # rebuild the design and plot
gui.autoscale() #resize GUI to see QComponent
gui.zoom_on_components(['Star']) #Can also gui.zoom_on_components([q1.name])_____no_output_____#Let's see what the Q1 object looks like

q1 #print Q1 information_____no_output_____#Save screenshot as a .png formatted file.
gui.screenshot()_____no_output_____# Screenshot the canvas only as a .png formatted file.
gui.figure.savefig('shot.png')

from IPython.display import Image, display
_disp_ops = dict(width=500)
display(Image('shot.png', **_disp_ops))_____no_output_____

# Closing the Qiskit Metal GUI_____no_output_____

gui.main_window.close()_____no_output_____


# Notebook from srom/rna_learn
Path: notebook/Condensed Traits DB.ipynb

import os
import datetime
from pathlib import Path

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

if os.getcwd().endswith('notebook'):
    os.chdir('..')_____no_output_____sns.set(palette='colorblind', font_scale=1.3)_____no_output_____species_df = pd.read_csv(os.path.join(os.getcwd(), 'data/condensed_traits/condensed_species_NCBI.csv'))_____no_output_____len(species_df)_____no_output_____species_df.columns_____no_output_____species_df.head()_____no_output_____len(species_df[species_df['range_tmp'].notnull()])_____no_output_____len(species_df[species_df['growth_tmp'].notnull()])_____no_output_____len(species_df['phylum'].unique())_____no_output_____ax = species_df['growth_tmp'].hist(figsize=(10, 5), bins=50, log=True);
ax.set_title('Distribution of Growth Temperature')
ax.set_xlabel('Growth Temperature');
ax.set_ylabel('Count (log-scale)');_____no_output_____ax = species_df['range_tmp'].hist(figsize=(16, 6), log=True)_____no_output_____

Species selection:

- Mesophilic, thermophilic, extreme thermophilic, psychrophilic
- At least 5 species from the same Phylum_____no_output_____

phylum_count = species_df[species_df['range_tmp'].isin([
    'thermophilic', 'mesophilic', 'psychrophilic', 'extreme thermophilic',
])][
    ['phylum', 'species_tax_id']
].groupby(
    ['phylum']
).count().sort_values('species_tax_id', ascending=False)_____no_output_____100 * len(phylum_count[phylum_count['species_tax_id'] > 10]) / len(phylum_count)_____no_output_____relevant_phylums = phylum_count[phylum_count['species_tax_id'] > 5].index_____no_output_____n_phylum = lambda v: 100 * len(phylum_count[phylum_count['species_tax_id'] >= v]) / len(phylum_count)
n_total = lambda v: len(phylum_count[phylum_count['species_tax_id'] >= v]) * v_____no_output_____x = list(range(1, 21))
y = [n_total(v) for v in x]

f, axes = plt.subplots(1, 2, figsize=(20, 6))
ax, ax2 = axes

ax.plot(x, y);

best_idx = np.argmax(y)
best_value = x[best_idx]
ax.axvline(best_value, color='green');
ax.set_title(f'Best value: {best_value}, totaling {y[best_idx]:,} species');
ax.set_xlabel('Number of species per phylum');
ax.set_ylabel('Total number of species');

y2 = [n_phylum(v) for v in x]
ax2.plot(x, y2);
ax2.axvline(best_value, color='green');
ax2.set_title(f'Phyla considered at best value: {y2[best_idx]:.0f}%');
ax2.set_xlabel('Number of species per phylum');
ax2.set_ylabel('% of all phyla');_____no_output_____x = list(range(1, 600))
y = [n_total(v) for v in x]

f, axes = plt.subplots(1, 2, figsize=(20, 6))
ax, ax2 = axes

ax.plot(x, y);

best_idx = np.argmax(y)
best_value = x[best_idx]
ax.axvline(best_value, color='green');
ax.set_title(f'Best value: {best_value}, totaling {y[best_idx]:,} species');
ax.set_xlabel('Number of species per phylum');
ax.set_ylabel('Total number of species');

y2 = [n_phylum(v) for v in x]
ax2.plot(x, y2);
ax2.axvline(best_value, color='green');
ax2.set_title(f'Phyla considered at best value: {y2[best_idx]:.0f}%');
ax2.set_xlabel('Number of species per phylum');
ax2.set_ylabel('% of all phyla');_____no_output_____0.35 * len(phylum_count)_____no_output_____selected_phyla = phylum_count[phylum_count['species_tax_id'] >= 11].index_____no_output_____selected_phyla_____no_output_____species_df[
    (species_df['phylum'].isin(selected_phyla)) &
    (species_df['range_tmp'].isin([
        'thermophilic', 'mesophilic', 'psychrophilic', 'extreme thermophilic',
    ]))
][['range_tmp', 'species_tax_id']].groupby(['range_tmp']).count()_____no_output_____avg = species_df['genome_size'].mean()
ax = species_df['genome_size'].hist(figsize=(12, 6));
ax.axvline(avg, color='red');
ax.set_title(f'Distribution of genome Size (mean: {avg:,.0f} bases)');_____no_output_____seed = 10
n_species_per_phylum = 11
rs = np.random.RandomState(seed)
selected_species = []
for phylum in selected_phyla:
    df = species_df[
        (species_df['phylum'] == phylum) &
        (species_df['range_tmp'].isin([
            'thermophilic', 'mesophilic', 'psychrophilic', 'extreme thermophilic',
        ]))
    ]
    ix = df.index.tolist()
    selected_indices = rs.choice(ix, size=n_species_per_phylum, replace=False)
    selected_species += species_df.loc[selected_indices]['species_tax_id'].values.tolist()

print(len(selected_species))209
species_df[
    species_df['species_tax_id'].isin(selected_species)
][['range_tmp', 'species_tax_id']].groupby(['range_tmp']).count()_____no_output_____

## Stratification?_____no_output_____

count_per_phylum = species_df[species_df['range_tmp'].notnull()][['phylum', 'species_tax_id']].groupby(['phylum']).count().sort_values(
    'species_tax_id', ascending=False,
)
len(count_per_phylum)_____no_output_____count_per_phylum_____no_output_____

## Load list of NCBI species_____no_output_____

ncbi_full_list_path = os.path.join(os.getcwd(), 'data/condensed_traits/ncbi_available_sequences.txt')
ncbi_genomes_list = pd.read_csv(ncbi_full_list_path, sep='\t', skiprows=1)
ncbi_genomes_list.head()/Users/srom/workspace/rna_learn/env/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3051: DtypeWarning: Columns (21) have mixed types.Specify dtype option on import or set low_memory=False.
  interactivity=interactivity, compiler=compiler, result=result)
species_taxids = ncbi_genomes_list[
    (ncbi_genomes_list['assembly_level'] == 'Complete Genome') &
    (ncbi_genomes_list['genome_rep'] == 'Full') &
    (ncbi_genomes_list['excluded_from_refseq'].isnull())
]['species_taxid'].unique()
len(species_taxids)_____no_output_____species_in_dataset = species_df[species_df['growth_tmp'].notnull()]['species_tax_id'].unique()
len(species_in_dataset)_____no_output_____candidate_species_taxids = sorted(set(species_taxids) & set(species_in_dataset))
len(candidate_species_taxids)_____no_output_____candidate_species = species_df[species_df['species_tax_id'].isin(candidate_species_taxids)].reset_index(drop=True)_____no_output_____ax = candidate_species['growth_tmp'].hist(figsize=(10, 5), bins=50, log=False);
ax.set_title('Distribution of Growth Temperature')
ax.set_xlabel('Growth Temperature');
ax.set_ylabel('Count');_____no_output_____ax = candidate_species['growth_tmp'].hist(figsize=(10, 5), bins=50, log=True);
ax.set_title('Distribution of Growth Temperature')
ax.set_xlabel('Growth Temperature');
ax.set_ylabel('Count (log-scale)');_____no_output_____ncbi_candidate_records = ncbi_genomes_list[
    (ncbi_genomes_list['assembly_level'] == 'Complete Genome') &
    (ncbi_genomes_list['genome_rep'] == 'Full') &
    (ncbi_genomes_list['species_taxid'].isin(candidate_species_taxids))
].reset_index(drop=True)

len(ncbi_candidate_records)_____no_output_____ncbi_candidate_records['seq_rel_date_processed'] = ncbi_candidate_records['seq_rel_date'].apply(
    lambda d_str: datetime.datetime.strptime(d_str, '%d/%m/%Y')
)_____no_output_____def select_representative_genome(ncbi_candidate_records):
    species_taxid = ncbi_candidate_records['species_taxid'].unique()

    stats = {
        'reference genome': 0,
        'representative genome': 0,
        'most recent': 0,
        'most recent paired with GenBank': 0,
        'most recent not paired with GenBank': 0,
    }
    stain_indices = []
    for specie_taxid in species_taxid:
        df = ncbi_candidate_records[ncbi_candidate_records['species_taxid'] == specie_taxid]
        ix = df.index
        for row_ix in ix:
            row = ncbi_candidate_records.loc[row_ix]
            refseq_category = row['refseq_category']
            stain_idx = None
            if refseq_category == 'reference genome':
                stats['reference genome'] += 1
                stain_idx = row_ix
                break
            elif refseq_category == 'representative genome':
                stats['representative genome'] += 1
                stain_idx = row_ix
                break

        if stain_idx is None:
            stats['most recent'] += 1
            sorted_df = df.sort_values('seq_rel_date_processed', ascending=False)

            for row_ix in sorted_df.index:
                row = ncbi_candidate_records.loc[row_ix]
                gbrs_paired_asm = row['gbrs_paired_asm']
                paired_asm_comp = row['paired_asm_comp']
                if (
                    not pd.isnull(gbrs_paired_asm) and
                    not pd.isnull(paired_asm_comp) and
                    paired_asm_comp == 'identical'
                ):
                    stain_idx = row_ix
                    stats['most recent paired with GenBank'] += 1
                    break

            if stain_idx is None:
                stain_idx = sorted_df.index[0]
                stats['most recent not paired with GenBank'] += 1

        stain_indices.append(stain_idx)

    return ncbi_candidate_records.loc[stain_indices].reset_index(drop=True), stats_____no_output_____ncbi_candidate_records_filtered, stats = select_representative_genome(ncbi_candidate_records)_____no_output_____len(ncbi_candidate_records_filtered)_____no_output_____stats_____no_output_____ncbi_candidate_records_filtered.head()_____no_output_____def make_download_url(row):
    ftp_path = row['ftp_path'].strip().replace('(', '_').replace(')', '_')
    assembly_accession = row['assembly_accession'].strip().replace('(', '_').replace(')', '_')
    asm_name = row['asm_name'].strip().replace('/', '_').replace('#', '_').replace('(', '_').replace(')', '_')
    download_url_base = f"{ftp_path}/{assembly_accession}_{asm_name}".replace(' ', '_')
    return download_url_base

ncbi_candidate_records_filtered['download_url_base'] = ncbi_candidate_records_filtered.apply(
    make_download_url,
    axis=1,
)
ncbi_candidate_records_filtered['download_url_base'].iloc[0]_____no_output_____#ncbi_candidate_records_filtered.to_csv(
#    os.path.join(os.getcwd(), 'data/condensed_traits/ncbi_species_to_fetch.csv'),
#    index=False,
#)_____no_output_____

## Check data quality of downloaded sequences_____no_output_____

data_folder = os.path.join(os.getcwd(), 'data/condensed_traits/sequences')_____no_output_____def check_for_missing_files(ncbi_records, data_folder):
    stats_missing = {
        'dna (fna)': 0,
        'annotations (gff)': 0,
        'protein (faa)': 0,
        'CDS (fna)': 0,
        'RNA (fna)': 0,
    }

    details = []
    for row_ix in ncbi_records.index:
        row = ncbi_records.loc[row_ix]
        specie_taxid = row['species_taxid']
        files = [
            ('dna (fna)', os.path.join(data_folder, f'{specie_taxid}', f'{specie_taxid}_genomic.fna.gz')),
            ('annotations (gff)', os.path.join(data_folder, f'{specie_taxid}', f'{specie_taxid}_genomic.gff.gz')),
            ('protein (faa)', os.path.join(data_folder, f'{specie_taxid}', f'{specie_taxid}_protein.faa.gz')),
            ('CDS (fna)', os.path.join(data_folder, f'{specie_taxid}', f'{specie_taxid}_cds_from_genomic.fna.gz')),
            ('RNA (fna)', os.path.join(data_folder, f'{specie_taxid}', f'{specie_taxid}_rna_from_genomic.fna.gz')),
        ]
        for category, filepath in files:
            is_missing = False
            if not Path(filepath).is_file():
                stats_missing[category] += 1
                is_missing = True

            details.append([specie_taxid, category, is_missing])

    details_df = pd.DataFrame(details, columns=['species_taxid', 'content', 'is_missing'])

    return stats_missing, details_df_____no_output_____stats_missing, details_df = check_for_missing_files(ncbi_candidate_records_filtered, data_folder)_____no_output_____stats_missing_____no_output_____species_to_discard = details_df[details_df['is_missing']]['species_taxid'].unique()
len(species_to_discard)_____no_output_____species_with_all_sequences = ncbi_candidate_records_filtered[
    ~ncbi_candidate_records_filtered['species_taxid'].isin(species_to_discard)
].reset_index(drop=True)
len(species_with_all_sequences)_____no_output_____#species_with_all_sequences.to_csv(
#    os.path.join(os.getcwd(), 'data/condensed_traits/ncbi_species_final.csv'),
#    index=False,
#)_____no_output_____species_taxid = species_with_all_sequences['species_taxid'].values.tolist()_____no_output_____df_ = species_df[species_df['species_tax_id'].isin(species_taxid)]
ax = df_['growth_tmp'].hist(figsize=(10, 5), bins=50, log=True);
ax.set_title('Distribution of Growth Temperature')
ax.set_xlabel('Growth Temperature');
ax.set_ylabel('Count (log scale)');_____no_output_____species_with_all_sequences.iloc[0]['download_url_base']_____no_output_____stats_missing_2, details_df_2 = check_for_missing_files(species_with_all_sequences, data_folder)
stats_missing_2_____no_output_____species_to_discard_2 = details_df_2[details_df_2['is_missing']]['species_taxid'].unique()
species_with_cds_and_rna = species_with_all_sequences[
    ~species_with_all_sequences['species_taxid'].isin(species_to_discard_2)
].reset_index(drop=True)
len(species_with_cds_and_rna)_____no_output_____#species_with_cds_and_rna.to_csv(
#    os.path.join(os.getcwd(), 'data/condensed_traits/ncbi_species_final.csv'),
#    index=False,
#)_____no_output_____species_taxid = species_with_cds_and_rna['species_taxid'].values.tolist()
df_ = species_df[species_df['species_tax_id'].isin(species_taxid)]
ax = df_['growth_tmp'].hist(figsize=(10, 5), bins=50, log=True);
ax.set_title('Distribution of Growth Temperature')
ax.set_xlabel('Growth Temperature');
ax.set_ylabel('Count (log scale)');_____no_output_____len(df_[['phylum', 'species_tax_id']].groupby(['phylum']).count().sort_values(
    'species_tax_id', ascending=False,
))_____no_output_____df_[['superkingdom', 'species_tax_id']].groupby(['superkingdom']).count().sort_values(
    'species_tax_id', ascending=False,
)_____no_output_____species_taxid = species_with_cds_and_rna['species_taxid'].values.tolist()
df__ = species_df[
    (species_df['species_tax_id'].isin(species_taxid)) &
    (species_df['superkingdom'] == 'Archaea')
]
ax = df__['growth_tmp'].hist(figsize=(10, 5), bins=50, log=True);
ax.set_title('Distribution of Growth Temperature for Archaea')
ax.set_xlabel('Growth Temperature');
ax.set_ylabel('Count (log scale)');_____no_output_____


# Notebook from Benedict-Carling/BioCRNPyler
Path: examples/5. Parameters.ipynb

## Parameters in BioCRNpyler
BioCRNpyler has a flexible parameter system allowing for models to be rapidly produced using very few parameters common across many reactions and then refined so that each mechanisms and components have unique reaction parameters derived from the literature or experiments. This parameter system is designed to work automatically, so all users have to do is specify a parameter file of the proper format or a parameter dictionary which can be loaded into Mixtures and Components using the parameter_file and/or parameters (for dictionaries) keywords. Parameters from multiple sources can be used together.

Parameters passed into a Component supercede mixture level parameters in reactions produced by that Component.

#### Parameter File Overview:
Parameter files can be .tsv (or .txt) or .csv. The first line of the file should contain column headings. The following headings are required (in any order): mechanism_id, part_id, param_name, param_val (spaces can be substituted for underscores and headings are not case sensitive). mechanism_id is the name of the Mechanism or the kind of mechanism that will use this parameter, for example "transcription" or  "transcription_mm" for Mechalis-Menten transcription would go in this column. part_id refers to the name supplied by the Component that will use this mechanism, for example "ptet" for a tet repressed promoter and "ptet_leak" for leak reactions of that promoter. param_name refers to the name of the model parameter, for example "ktx", "kb", or "ku". The value of these columns is case sensitive and underscores are different from spaces. All these parameter headings (if they are not empty) are required to be alpha-numeric (plus single internal underscores) and start with a letter.

#### Viewing Parameters Used in a Model
The easiest way to examine which parameter values are used in a Model and where BioCRNpyler found them is with the prett_print function of a ChemicalReactionNetwork or a Reaction with show_rates and show_keys both set to True.

    CRN.prett_print(show_rates = True, show_keys = True)

show_rates toggles whether reaction rates are shown. The parameter values used in the rates will be shown below the rate function. show_keys toggles whether the ParameterKey used to search and find these rates is displayed. search_key is the key the Mechanism and component searched for in order to find the rate. found_key is the key the parameter was found in after defaulting.

#### Parameter Dictionary Overview:
A parameter dictionary should have the following forms (note that part_id and mechanism can be None):

Using ParameterKeys (named tuples):

    parameters = {ParameterKey(mechanism = "mechanism_name", part_id = "part_id", name = "parameter_name"):param_val ...} #using ParameterKeys

Using tuples or even just strings (for default parameters) shorthand is also valid:

    parameters = {("mechanism_name", "part_id", "parameter_name"):val1, "k":default_val ...}

#### Parameter Value Defaulting:
Parameters can be passed into Components and Mixtures as both files and/or dictionaries. Parameters into Components will be used by reactions created by that Component:

    Component(... parameters = parameters, parameter_file = "file_name" ...)

If a Component cannot find its own parameters, it will use parameters passed into the Mixture:

    Mixture(... parameters = parameters, parameter_file = "file_name" ...)

At both the Mixture and Component levels, BioCRNpyler does nto require an exact keyword match. When Mechanisms are called by Components to produce reactions, they will initially look for

    ParameterKey(mechanism = "mechanism_name", part_id = "part_id", name = "parameter_name") --> param_val.

If that particular parameter key cannot be found, the software will default to the following keys in this order:

    ParameterKey(mechanism = "mechanism_type", part_id = "part_id" , name = "parameter_name")
    ParameterKey(mechanism = None, part_id = "part_id" , name = "parameter_name")
    ParameterKey(mechanism = "mechanism_name", part_id = None , name = "parameter_name")
    ParameterKey(mechanism = "mechanism_type", part_id = None , name = "parameter_name")
    ParameterKey(mechanism =  None, part_id = None , name = "parameter_name")

As a note, mechanism_name refers to the .name variable of a Mechanism. mechanism_type refers to the .type variable of a Mechanism. Either of these can be used as a mechanism_id. This allows for models to be constructed easily using default parameter values and for parameters to be shared between different Mechanisms and/or Components.

#### The ParameterDatabase and ParameterEntries
Each Component and Mixture has its own ParameterDatabase full of ParameterEntries. In general, users are not expected to make these objects by hand. Instead, ParameterDatabases will be automatically populated with ParameterEntries using either a parameter_file or parameter_dictionary. However, they can be accessed and view by looking at Component.parameter_database and Mixture.parameter_database

#### Multiple Parameter Files:
Components and Mixtures can both have one more multiple parameter files by passing in a list of filenames instead of a single filename to the parameter_file keyword. Components use parameters loaded from their file(s) before defaulting to the file(s) supplied to a Mixture. The last file in any list will take precedent and overwrite parameter files which were written earlier.

Below is an example csv with all the parameters for a tetR promoter undergoing Michaelis Menten transcription and translation._____no_output_____## 1. The Parameter File
In the following cell we look at an example parameter file that will run with no parameter defaulting._____no_output_____

from biocrnpyler import *
perfect_param_file_name = "Perfect Param File Example.tsv"

#Open and print the parameter file
param_file = open(perfect_param_file_name)
print("****Parameter File****")
print(param_file.read())
param_file.close()****Parameter File****
mechanism_id	part_id	param_name	param_val	comments
transcription_mm	ptet_tetR	kb	10.	extra columns are okay!
transcription_mm	ptet_tetR	ku	.1	These are the parameters for transcription
transcription_mm	ptet_tetR	ktx	1.
transcription_mm	ptet_leak	ktx	.1	These are the parameters for transcription leak
transcription_mm	ptet_leak	ku	.1
transcription_mm	ptet_leak	kb	.1
one_step_cooperative_binding	ptet_tetR	ku	.1	These are parameters for tetR dimerization and binding with the promoter
one_step_cooperative_binding	ptet_tetR	kb	.1
one_step_cooperative_binding	ptet_tetR	cooperativity	2
translation_mm	BCD	ktl	2.0	These are parameters for translation
translation_mm	BCD	ku	.25
translation_mm	BCD	kb	10
rna_degredation_mm		kb	10	These are parameters for RNA degredation.
rna_degredation_mm		ku	.5	They will be the same for all RNAs because of parameter defaulting.
rna_degredation_mm		kdeg	1.5

### Use this parameter file to create a CRN_____no_output_____

#Create a Regulated Promoter
Ptet = RegulatedPromoter("ptet", regulators=["tetR"], leak=True)
reg_rep_assembly = DNAassembly(name="reporter", promoter=Ptet, rbs="BCD")
tet = Protein("tetR")
components = [reg_rep_assembly, tet]
myMixture = TxTlExtract(name="txtl", parameter_file = perfect_param_file_name, components=components)

#Print the parameter database created from the file
print("\n****Loaded Parameters****")
for param in myMixture.parameter_database:
    print(param.parameter_key, "==>", param.value)

print("\n****Resulting CRN****\nNotice that the found_key perfectly matches the search_key.\n")
print(myMixture.compile_crn().pretty_print(show_rates = True, show_keys = True))
****Loaded Parameters****
ParameterKey(mechanism='transcription_mm', part_id='ptet_tetR', name='kb') ==> 10.0
ParameterKey(mechanism='transcription_mm', part_id='ptet_tetR', name='ku') ==> 0.1
ParameterKey(mechanism='transcription_mm', part_id='ptet_tetR', name='ktx') ==> 1.0
ParameterKey(mechanism='transcription_mm', part_id='ptet_leak', name='ktx') ==> 0.1
ParameterKey(mechanism='transcription_mm', part_id='ptet_leak', name='ku') ==> 0.1
ParameterKey(mechanism='transcription_mm', part_id='ptet_leak', name='kb') ==> 0.1
ParameterKey(mechanism='one_step_cooperative_binding', part_id='ptet_tetR', name='ku') ==> 0.1
ParameterKey(mechanism='one_step_cooperative_binding', part_id='ptet_tetR', name='kb') ==> 0.1
ParameterKey(mechanism='one_step_cooperative_binding', part_id='ptet_tetR', name='cooperativity') ==> 2.0
ParameterKey(mechanism='translation_mm', part_id='BCD', name='ktl') ==> 2.0
ParameterKey(mechanism='translation_mm', part_id='BCD', name='ku') ==> 0.25
ParameterKey(mechanism='translation_mm', part_id='BCD', name='kb') ==> 10.0
ParameterKey(mechanism='rna_degredation_mm', part_id=None, name='kb') ==> 10.0
ParameterKey(mechanism='rna_degredation_mm', part_id=None, name='ku') ==> 0.5
ParameterKey(mechanism='rna_degredation_mm', part_id=None, name='kdeg') ==> 1.5

****Resulting CRN****
Notice that the found_key perfectly matches the search_key.

Species (12) = {0. [dna[reporter]:2x_protein[tetR]], 1. complex[[dna[reporter]:2x_protein[tetR]]:protein[RNAP]], 2. protein[tetR], 3. protein[RNAP], 4. protein[reporter], 5. complex[protein[RNAase]:rna[reporter]], 6. complex[dna[reporter]:protein[RNAP]], 7. protein[RNAase], 8. rna[reporter], 9. complex[protein[Ribo]:rna[reporter]], 10. dna[reporter], 11. protein[Ribo]}
Reactions (9) = [
0. dna[reporter]+protein[RNAP] <--> complex[dna[reporter]:protein[RNAP]]
 Kf=k_forward * dna_reporter * protein_RNAP
 Kr=k_reverse * complex_dna_reporter_protein_RNAP
  k_forward=0.1
  found_key=(mech=transcription_mm, partid=ptet_leak, name=kb).
  search_key=(mech=transcription_mm, partid=ptet_leak, name=kb.
  k_reverse=0.1
  found_key=(mech=transcription_mm, partid=ptet_leak, name=ku).
  search_key=(mech=transcription_mm, partid=ptet_leak, name=ku.

1. complex[dna[reporter]:protein[RNAP]] --> dna[reporter]+rna[reporter]+protein[RNAP]
 Kf=k_forward * complex_dna_reporter_protein_RNAP
  k_forward=0.1
  found_key=(mech=transcription_mm, partid=ptet_leak, name=ktx).
  search_key=(mech=transcription_mm, partid=ptet_leak, name=ktx.
  k_reverse=None

2. 2protein[tetR]+dna[reporter] <--> [dna[reporter]:2x_protein[tetR]]
 Kf=k_forward * protein_tetR^2 * dna_reporter
 Kr=k_reverse * dna_reporter_2x_protein_tetR
  k_forward=0.1
  found_key=(mech=one_step_cooperative_binding, partid=ptet_tetR, name=kb).
  search_key=(mech=one_step_cooperative_binding, partid=ptet_tetR, name=kb.
  k_reverse=0.1
  found_key=(mech=one_step_cooperative_binding, partid=ptet_tetR, name=ku).
  search_key=(mech=one_step_cooperative_binding, partid=ptet_tetR, name=ku.

3. [dna[reporter]:2x_protein[tetR]]+protein[RNAP] <--> complex[[dna[reporter]:2x_protein[tetR]]:protein[RNAP]]
 Kf=k_forward * dna_reporter_2x_protein_tetR * protein_RNAP
 Kr=k_reverse * complex_dna_reporter_2x_protein_tetR_protein_RNAP
  k_forward=10.0
  found_key=(mech=transcription_mm, partid=ptet_tetR, name=kb).
  search_key=(mech=transcription_mm, partid=ptet_tetR, name=kb.
  k_reverse=0.1
  found_key=(mech=transcription_mm, partid=ptet_tetR, name=ku).
  search_key=(mech=transcription_mm, partid=ptet_tetR, name=ku.

4. complex[[dna[reporter]:2x_protein[tetR]]:protein[RNAP]] --> [dna[reporter]:2x_protein[tetR]]+rna[reporter]+protein[RNAP]
 Kf=k_forward * complex_dna_reporter_2x_protein_tetR_protein_RNAP
  k_forward=1.0
  found_key=(mech=transcription_mm, partid=ptet_tetR, name=ktx).
  search_key=(mech=transcription_mm, partid=ptet_tetR, name=ktx.
  k_reverse=None

5. rna[reporter]+protein[Ribo] <--> complex[protein[Ribo]:rna[reporter]]
 Kf=k_forward * rna_reporter * protein_Ribo
 Kr=k_reverse * complex_protein_Ribo_rna_reporter
  k_forward=10.0
  found_key=(mech=translation_mm, partid=BCD, name=kb).
  search_key=(mech=translation_mm, partid=BCD, name=kb.
  k_reverse=0.25
  found_key=(mech=translation_mm, partid=BCD, name=ku).
  search_key=(mech=translation_mm, partid=BCD, name=ku.

6. complex[protein[Ribo]:rna[reporter]] --> rna[reporter]+protein[reporter]+protein[Ribo]
 Kf=k_forward * complex_protein_Ribo_rna_reporter
  k_forward=2.0
  found_key=(mech=translation_mm, partid=BCD, name=ktl).
  search_key=(mech=translation_mm, partid=BCD, name=ktl.
  k_reverse=None

7. rna[reporter]+protein[RNAase] <--> complex[protein[RNAase]:rna[reporter]]
 Kf=k_forward * rna_reporter * protein_RNAase
 Kr=k_reverse * complex_protein_RNAase_rna_reporter
  k_forward=10.0
  found_key=(mech=rna_degredation_mm, partid=None, name=kb).
  search_key=(mech=rna_degredation_mm, partid=reporter, name=kb.
  k_reverse=0.5
  found_key=(mech=rna_degredation_mm, partid=None, name=ku).
  search_key=(mech=rna_degredation_mm, partid=reporter, name=ku.

8. complex[protein[RNAase]:rna[reporter]] --> protein[RNAase]
 Kf=k_forward * complex_protein_RNAase_rna_reporter
  k_forward=1.5
  found_key=(mech=rna_degredation_mm, partid=None, name=kdeg).
  search_key=(mech=rna_degredation_mm, partid=reporter, name=kdeg.
  k_reverse=None

]

We will now look at an example of a parameter file that uses defaulting. If you were to fill in this file with full parameter signatures (mechanism_id, part_id, param_name, value), the errors at the bottom of the readout would slowly diminish. However, even without full values the file loads and runs. Although this example uses only the key "param_name" for default values, there exists a heirarchy of keys to allow for shared parameters between different Components and Mechanisms.

The parameter key heirarchy (top takes priority):
1) (mechanism_name, part_id, param_name)
2) (mechanism_type, part_id, param_name)
3) (part_id, param_name)
4) (mechanism_name, param_name)
5) (mechanism_type, param_name)
6) (param_name)

here the column "mechanism_type" can either be a Mechanism's type string Mechanism.type (eg "transcription") or its name string Mechanism.name (eg "transcription_mm"). _____no_output_____

from biocrnpyler import *
default_param_file_name = "Default Param File Example.tsv"

#Open and print the parameter file
param_file = open(default_param_file_name)
print("****Parameter File****")
print(param_file.read())
param_file.close()****Parameter File****
param_name	param_val	comments	mechanism_id	part_id
ku	1.0	Any parameter called ku will default to this
kb	1.0	Default kb
ktx	2.0	Default ktx
ktl	3.0	Default ktl
cooperativity	2.0	Default cooperativity
kdeg	.5	Default degredation

### And use these parameters in a CRN - not all reactions of the same name share the same parameters!_____no_output_____

#Create a Regulated Promoter
Ptet = RegulatedPromoter("ptet", regulators=["tetR"], leak=True)
reg_rep_assembly = DNAassembly(name="reporter", promoter=Ptet, rbs="BCD")
tet = Protein("tetR")
components = [reg_rep_assembly, tet]

myMixture = TxTlExtract(name="txtl", parameter_file = default_param_file_name, components=components)

#Print the parameter dictionary created from the file
print("\n****Loaded Parameters****")
for param in myMixture.parameter_database:
    print(param.parameter_key, "==>", param.value)

print("\n****Resulting CRN****\nNotice that only the name portion of parameter keys match.\n")
print(myMixture.compile_crn().pretty_print(show_rates = True, show_keys = True))
****Loaded Parameters****
ParameterKey(mechanism=None, part_id=None, name='ku') ==> 1.0
ParameterKey(mechanism=None, part_id=None, name='kb') ==> 1.0
ParameterKey(mechanism=None, part_id=None, name='ktx') ==> 2.0
ParameterKey(mechanism=None, part_id=None, name='ktl') ==> 3.0
ParameterKey(mechanism=None, part_id=None, name='cooperativity') ==> 2.0
ParameterKey(mechanism=None, part_id=None, name='kdeg') ==> 0.5

****Resulting CRN****
Notice that only the name portion of parameter keys match.

Species (12) = {0. [dna[reporter]:2x_protein[tetR]], 1. complex[[dna[reporter]:2x_protein[tetR]]:protein[RNAP]], 2. protein[tetR], 3. protein[RNAP], 4. protein[reporter], 5. complex[protein[RNAase]:rna[reporter]], 6. complex[dna[reporter]:protein[RNAP]], 7. protein[RNAase], 8. rna[reporter], 9. complex[protein[Ribo]:rna[reporter]], 10. dna[reporter], 11. protein[Ribo]}
Reactions (9) = [
0. dna[reporter]+protein[RNAP] <--> complex[dna[reporter]:protein[RNAP]]
 Kf=k_forward * dna_reporter * protein_RNAP
 Kr=k_reverse * complex_dna_reporter_protein_RNAP
  k_forward=1.0
  found_key=(mech=None, partid=None, name=kb).
  search_key=(mech=transcription_mm, partid=ptet_leak, name=kb.
  k_reverse=1.0
  found_key=(mech=None, partid=None, name=ku).
  search_key=(mech=transcription_mm, partid=ptet_leak, name=ku.

1. complex[dna[reporter]:protein[RNAP]] --> dna[reporter]+rna[reporter]+protein[RNAP]
 Kf=k_forward * complex_dna_reporter_protein_RNAP
  k_forward=2.0
  found_key=(mech=None, partid=None, name=ktx).
  search_key=(mech=transcription_mm, partid=ptet_leak, name=ktx.
  k_reverse=None

2. 2protein[tetR]+dna[reporter] <--> [dna[reporter]:2x_protein[tetR]]
 Kf=k_forward * protein_tetR^2 * dna_reporter
 Kr=k_reverse * dna_reporter_2x_protein_tetR
  k_forward=1.0
  found_key=(mech=None, partid=None, name=kb).
  search_key=(mech=one_step_cooperative_binding, partid=ptet_tetR, name=kb.
  k_reverse=1.0
  found_key=(mech=None, partid=None, name=ku).
  search_key=(mech=one_step_cooperative_binding, partid=ptet_tetR, name=ku.

3. [dna[reporter]:2x_protein[tetR]]+protein[RNAP] <--> complex[[dna[reporter]:2x_protein[tetR]]:protein[RNAP]]
 Kf=k_forward * dna_reporter_2x_protein_tetR * protein_RNAP
 Kr=k_reverse * complex_dna_reporter_2x_protein_tetR_protein_RNAP
  k_forward=1.0
  found_key=(mech=None, partid=None, name=kb).
  search_key=(mech=transcription_mm, partid=ptet_tetR, name=kb.
  k_reverse=1.0
  found_key=(mech=None, partid=None, name=ku).
  search_key=(mech=transcription_mm, partid=ptet_tetR, name=ku.

4. complex[[dna[reporter]:2x_protein[tetR]]:protein[RNAP]] --> [dna[reporter]:2x_protein[tetR]]+rna[reporter]+protein[RNAP]
 Kf=k_forward * complex_dna_reporter_2x_protein_tetR_protein_RNAP
  k_forward=2.0
  found_key=(mech=None, partid=None, name=ktx).
  search_key=(mech=transcription_mm, partid=ptet_tetR, name=ktx.
  k_reverse=None

5. rna[reporter]+protein[Ribo] <--> complex[protein[Ribo]:rna[reporter]]
 Kf=k_forward * rna_reporter * protein_Ribo
 Kr=k_reverse * complex_protein_Ribo_rna_reporter
  k_forward=1.0
  found_key=(mech=None, partid=None, name=kb).
  search_key=(mech=translation_mm, partid=BCD, name=kb.
  k_reverse=1.0
  found_key=(mech=None, partid=None, name=ku).
  search_key=(mech=translation_mm, partid=BCD, name=ku.

6. complex[protein[Ribo]:rna[reporter]] --> rna[reporter]+protein[reporter]+protein[Ribo]
 Kf=k_forward * complex_protein_Ribo_rna_reporter
  k_forward=3.0
  found_key=(mech=None, partid=None, name=ktl).
  search_key=(mech=translation_mm, partid=BCD, name=ktl.
  k_reverse=None

7. rna[reporter]+protein[RNAase] <--> complex[protein[RNAase]:rna[reporter]]
 Kf=k_forward * rna_reporter * protein_RNAase
 Kr=k_reverse * complex_protein_RNAase_rna_reporter
  k_forward=1.0
  found_key=(mech=None, partid=None, name=kb).
  search_key=(mech=rna_degredation_mm, partid=reporter, name=kb.
  k_reverse=1.0
  found_key=(mech=None, partid=None, name=ku).
  search_key=(mech=rna_degredation_mm, partid=reporter, name=ku.

8. complex[protein[RNAase]:rna[reporter]] --> protein[RNAase]
 Kf=k_forward * complex_protein_RNAase_rna_reporter
  k_forward=0.5
  found_key=(mech=None, partid=None, name=kdeg).
  search_key=(mech=rna_degredation_mm, partid=reporter, name=kdeg.
  k_reverse=None

]

### Setting Parameters at the component level
Components can have their own parameter files instead of relying on the parameter files passed into a mixture. Components can also have the mixtures default parameters overwritten with a parameter dictionary. Below, we will create a DNAassembly which has custom parameters loaded in as a dictionary (this works the same as loading them with a file). We will put this in a Mixture with the default parameters from the above example. This example also helps illustrate how the parameter loading heirarchy works if you examine the final CRNs._____no_output_____

#Create custom parameter dictionary for the ptet promoter. In this case, we will add specific leak parameters and
ra_param_dict = {
    ParameterKey(mechanism = "transcription_mm", part_id = None, name = "ku"):33.33, #These parameters will take priority over single key parameters
    ParameterKey(mechanism = "transcription_mm", part_id = None, name = "kb"):.3333,
    ParameterKey(mechanism = "transcription_mm", part_id = None, name = "ktx"):3.333,
    ParameterKey(mechanism = "transcription_mm", part_id = "ptet_leak", name = "ku"):111.1, #these parameters will take priority over the ones above
    ParameterKey(mechanism = "transcription_mm", part_id = "ptet_leak", name = "kb"):.1111,
}

#Use the parameter_file keyword to update the parameters with a file.
#Use the parameters keyword to update the parameters with a dictionary
#If Use both: the dictionary takes precedent to the file if there are conflicts.
Ptet = RegulatedPromoter("ptet", regulators=["tetR"], leak=True, parameters = ra_param_dict)
reg_rep_assembly = DNAassembly(name="reporter", promoter=Ptet, rbs="BCD")
tet = Protein("tetR")
components = [reg_rep_assembly, tet]
myMixture = TxTlExtract(name="txtl", parameter_file = default_param_file_name, components=components)_____no_output_____

### Mixture parameters come from the file_____no_output_____

print("\n****Mixture Parameters****")
for param in myMixture.parameter_database:
    print(param.parameter_key, "==>", param.value)
****Mixture Parameters****
ParameterKey(mechanism=None, part_id=None, name='ku') ==> 1.0
ParameterKey(mechanism=None, part_id=None, name='kb') ==> 1.0
ParameterKey(mechanism=None, part_id=None, name='ktx') ==> 2.0
ParameterKey(mechanism=None, part_id=None, name='ktl') ==> 3.0
ParameterKey(mechanism=None, part_id=None, name='cooperativity') ==> 2.0
ParameterKey(mechanism=None, part_id=None, name='kdeg') ==> 0.5

### Component level parameters come from the dictionary_____no_output_____

### Component Parameters are different!
print("\n****Ptet Parameters****")
for param in Ptet.parameter_database:
    print(param.parameter_key, "==>", param.value)
****Ptet Parameters****
ParameterKey(mechanism='transcription_mm', part_id=None, name='ku') ==> 33.33
ParameterKey(mechanism='transcription_mm', part_id=None, name='kb') ==> 0.3333
ParameterKey(mechanism='transcription_mm', part_id=None, name='ktx') ==> 3.333
ParameterKey(mechanism='transcription_mm', part_id='ptet_leak', name='ku') ==> 111.1
ParameterKey(mechanism='transcription_mm', part_id='ptet_leak', name='kb') ==> 0.1111

### The Compiled CRN uses Component level parameters if it can find them before defaulting to Mixture level Parameters_____no_output_____

myCRN = myMixture.compile_crn()
print(myCRN.pretty_print(show_rates = True, show_keys = True))Species (12) = {0. [dna[reporter]:2x_protein[tetR]], 1. complex[[dna[reporter]:2x_protein[tetR]]:protein[RNAP]], 2. protein[tetR], 3. protein[RNAP], 4. protein[reporter], 5. complex[protein[RNAase]:rna[reporter]], 6. complex[dna[reporter]:protein[RNAP]], 7. protein[RNAase], 8. rna[reporter], 9. complex[protein[Ribo]:rna[reporter]], 10. dna[reporter], 11. protein[Ribo]}
Reactions (9) = [
0. dna[reporter]+protein[RNAP] <--> complex[dna[reporter]:protein[RNAP]]
 Kf=k_forward * dna_reporter * protein_RNAP
 Kr=k_reverse * complex_dna_reporter_protein_RNAP
  k_forward=1.0
  found_key=(mech=None, partid=None, name=kb).
  search_key=(mech=transcription_mm, partid=ptet_leak, name=kb.
  k_reverse=1.0
  found_key=(mech=None, partid=None, name=ku).
  search_key=(mech=transcription_mm, partid=ptet_leak, name=ku.

1. complex[dna[reporter]:protein[RNAP]] --> dna[reporter]+rna[reporter]+protein[RNAP]
 Kf=k_forward * complex_dna_reporter_protein_RNAP
  k_forward=2.0
  found_key=(mech=None, partid=None, name=ktx).
  search_key=(mech=transcription_mm, partid=ptet_leak, name=ktx.
  k_reverse=None

2. 2protein[tetR]+dna[reporter] <--> [dna[reporter]:2x_protein[tetR]]
 Kf=k_forward * protein_tetR^2 * dna_reporter
 Kr=k_reverse * dna_reporter_2x_protein_tetR
  k_forward=1.0
  found_key=(mech=None, partid=None, name=kb).
  search_key=(mech=one_step_cooperative_binding, partid=ptet_tetR, name=kb.
  k_reverse=1.0
  found_key=(mech=None, partid=None, name=ku).
  search_key=(mech=one_step_cooperative_binding, partid=ptet_tetR, name=ku.

3. [dna[reporter]:2x_protein[tetR]]+protein[RNAP] <--> complex[[dna[reporter]:2x_protein[tetR]]:protein[RNAP]]
 Kf=k_forward * dna_reporter_2x_protein_tetR * protein_RNAP
 Kr=k_reverse * complex_dna_reporter_2x_protein_tetR_protein_RNAP
  k_forward=1.0
  found_key=(mech=None, partid=None, name=kb).
  search_key=(mech=transcription_mm, partid=ptet_tetR, name=kb.
  k_reverse=1.0
  found_key=(mech=None, partid=None, name=ku).
  search_key=(mech=transcription_mm, partid=ptet_tetR, name=ku.

4. complex[[dna[reporter]:2x_protein[tetR]]:protein[RNAP]] --> [dna[reporter]:2x_protein[tetR]]+rna[reporter]+protein[RNAP]
 Kf=k_forward * complex_dna_reporter_2x_protein_tetR_protein_RNAP
  k_forward=2.0
  found_key=(mech=None, partid=None, name=ktx).
  search_key=(mech=transcription_mm, partid=ptet_tetR, name=ktx.
  k_reverse=None

5. rna[reporter]+protein[Ribo] <--> complex[protein[Ribo]:rna[reporter]]
 Kf=k_forward * rna_reporter * protein_Ribo
 Kr=k_reverse * complex_protein_Ribo_rna_reporter
  k_forward=1.0
  found_key=(mech=None, partid=None, name=kb).
  search_key=(mech=translation_mm, partid=BCD, name=kb.
  k_reverse=1.0
  found_key=(mech=None, partid=None, name=ku).
  search_key=(mech=translation_mm, partid=BCD, name=ku.

6. complex[protein[Ribo]:rna[reporter]] --> rna[reporter]+protein[reporter]+protein[Ribo]
 Kf=k_forward * complex_protein_Ribo_rna_reporter
  k_forward=3.0
  found_key=(mech=None, partid=None, name=ktl).
  search_key=(mech=translation_mm, partid=BCD, name=ktl.
  k_reverse=None

7. rna[reporter]+protein[RNAase] <--> complex[protein[RNAase]:rna[reporter]]
 Kf=k_forward * rna_reporter * protein_RNAase
 Kr=k_reverse * complex_protein_RNAase_rna_reporter
  k_forward=1.0
  found_key=(mech=None, partid=None, name=kb).
  search_key=(mech=rna_degredation_mm, partid=reporter, name=kb.
  k_reverse=1.0
  found_key=(mech=None, partid=None, name=ku).
  search_key=(mech=rna_degredation_mm, partid=reporter, name=ku.

8. complex[protein[RNAase]:rna[reporter]] --> protein[RNAase]
 Kf=k_forward * complex_protein_RNAase_rna_reporter
  k_forward=0.5
  found_key=(mech=None, partid=None, name=kdeg).
  search_key=(mech=rna_degredation_mm, partid=reporter, name=kdeg.
  k_reverse=None

]


# Notebook from abjer/sds2019
Path: material/session_15/exercise_15.ipynb

# Exercise Set 15: Text Classification and Sentiment Analysis

*Morning, August 21, 2019*

In this Exercise Set you will practice using two basic text classification methods: rule- and machine learning-based. The exercise has XX parts:

1. Implement a lexical look-up method.
2. Apply pre-packaged rulebased dictionaries.
3. Train a simple baseline machine learning classifier.

In the end, you will then compare the results of these approaches.

First, load our standard stuff and import the following modules:_____no_output_____

## Standard stuff:
import numpy as np, seaborn as sns, pandas as pd
## For text classification:
import nltk, nltk.sentiment, sklearn
%matplotlib inline_____no_output_____

## Exercise 15 Part 1: Implementing your own Lexical Lookup method
There are many curated dictionaries and lexicons online for all sorts of topics (see for instance this project: https://hedonometer.org/index.html where the lexicons behind it can be downloaded here: https://github.com/andyreagan/hedonometer/blob/master/hedonometer/static/hedonometer/labMT1.txt). For this exercise we will use the following list of positive and negative words (positive:http://ptrckprry.com/course/ssd/data/negative-words.txt ; negative: http://ptrckprry.com/course/ssd/data/positive-words.txt) compilled by Hu and Liu.

We will use the following dataset (a random sample of the trustpilot review data we collected in exercise 8) to practice on.

> **Ex. 15.1.1:**  Load it like this (remember to import the `request` package):

```python
# download data
path2data = 'https://raw.githubusercontent.com/snorreralund/scraping_seminar/master/english_review_sample.csv'
df = pd.read_csv(path2data)```

The important columns are ***reviewBody*** containing the text, and the ***reviewRating_ratingValue*** containing the rating / stars._____no_output_____

# [Answer to Ex. 15.1.1]_____no_output_____

> **Ex. 15.1.2:** Next, we should download and prepare the dictionaries.
1. Download the lists using python's `requests.get()`. The lists are documents of words separated by new line (which is the '\n' character).
2. Make sure to remove the comment section in the top by splitting at the right place.
3. `.split()` these documents into words.
4. Convert them into sets (using the `set()`-command) and assign these to two variables (e.g. ***positive*** and ***negative***).

*Hint*: You can do all of the above in one line of code per list._____no_output_____

# [Answer to Ex. 15.1.2]_____no_output_____

Now we need to prepare the trustpilot reviews, our documents to be analyzed. This means lowercasing and tokenizing them to match the format that our dictionary comes in.

> **Ex. 15.1.3:** Define a function `preprocessing(string)`, that takes in a string and returns a list of words. The function should do the following:
1. lowercases the string using the `.lower()` command.
2. tokenize the words using the `nltk.tokenize.TweetTokenizer()` which is good for social media type user content (i.e. emojiies and more free use of punctuation and commas.
3. return tokenized documents.

*Hint*: You can first initialize the tokenizer (outside the preprocessing function), and then you use the `.tokenize()` method._____no_output_____

# [Answer to Ex 15.1.3]_____no_output_____

>**Ex 15.1.4:** Apply the preprocessing function to all the documents (i.e. our review texts in the column ***reviewBody***).

*Hint:* Use the `.apply()` method._____no_output_____

# [Answer to Ex 15.1.4]_____no_output_____

Now, we need to match the words in our dictionaries to the tokenized documents.
>**Ex 15.1.5:** Define a function `count_dictionary(document,dictionary)` that takes a tokenized document and a set of words (i.e. the dictionaries we loaded in Ex 15.1.2) and counts the number of matches. The function should do the following:
1. Filter the words not in the dictionary.
*Hint*: you can use a list comprehension or for loop and `if word in` condition.
2. Return the length of the filtered document.			_____no_output_____

# [Answer Ex 15.1.5]_____no_output_____

>**Ex 15.1.6:** Define two new columns (***positive_liu***, ***negative_liu***) in the dataframe applying the count_dictionary function to all tokenized documents with the positive and negative set as input.

*Hint:* the `.apply()` method allows you to input named arguments matching your `count_dictionary()` function to input the sentiment dictionary. _____no_output_____

# [Answer Ex 15.1.6]_____no_output_____

> **Ex 15.1.7 (BONUS):** Define your own set of words and use the count_dictionary function to match them (remember that they need to be lowercased to match)._____no_output_____

# [Answer Ex 15.1.7]_____no_output_____

### Exercise 15 Part 2: Applying prepackaged rulebased dictionaries and comparison of results
In the following exercise you will apply two prepackaged dictionaries which are also rulebased. One is MIT's VADER method (https://github.com/cjhutto/vaderSentiment) which is builtin to the NLTK package, and the Finn Aarups (DTU) AFINN sentiment analyzer, which has the advantage of working on danish text.
_____no_output_____>**Ex 15.2.1:** Apply VADER: Define a four new columns using output from the VADER sentiment analyzer
1. Import the `nltk`, `nltk.sentiment` package and download the vader_lexicon using `nltk.download('vader_lexicon')`.
2. Initialize the the vader sentiment analyzer: nltk.sentiment.vader.SentimentIntensityAnalyzer()
3. Apply the `.polarity_scores` function to all documents. This function should be provided with a string. So, you do not need to tokenize the documents, but instead you can just use the ***reviewBody*** column of your dataframe.
4. Convert the resulting Series of dictionaries (Vader outputs dictionaries of 4 scores) to a dataframe by first converting the Series into a list using the `list()` function, and then the `pd.DataFrame()` function.
5. Rename the columns adding vader_ as prefix.
6. Merge with original Dataframe.

*Hint:* Use `pd.concat([df,df2],axis=1)`
_____no_output_____

# [Answer Ex 15.2.1]_____no_output_____

>**Ex 15.2.2:** Install the AFINN package: `pip install afinn`. Read how to apply the afinn package here (https://github.com/fnielsen/afinn) and define a column called afinn.

*Hint:* As VADER's `polarity_score`, AFINN should be provided with a string and not the tokenized documents._____no_output_____

# [Answer Ex 15.2.2]_____no_output_____

>**Ex 15.2.3:** Plot all the different dictionary and rulebased sentiment analysis columns against each other using the seaborn plotting function `sns.pairplot`. Set the color (i.e. hue argument as the number of stars in the `reviewRating_ratingValue` column).

*Hint:* To plot, work with a smaller dataframe. You shoul select only the relevant columns, i.e. the ones with the sentiment analysis results and the star rating column. Moreover, you can downsample the dataframe to 1000 rows using the `.sample()` method.
_____no_output_____

# [Answer Ex 15.2.3]_____no_output_____

>**Ex 15.2.4:** Inspect a document where the LIU and VADER dictionaries disagree. More specifically, look at documents where the LIU lexicon is positive and the VADER is negative. For that, you should filter on positive expressions by LIU, and sample in the most negative according to the VADER. Try to explain the disagreement in terms of simple matches versus negations.

*Hint*: Pick a document and see which words the LIU lexicon has matched using python set notation: e.g.

```python
# overlap between two sets
set(documents[0])&positive
```_____no_output_____

# [Answer Ex 15.2.4]_____no_output_____

## Exercise 15 Part 3: Train a classifier to score the reviews and compare results
In following exercise you will implement a simple text classifier based on the logistic regression trained on the ratings. We will walk through the standard procedures of Supervised Learning - i.e. parameter search and crossvalidation - but first we need to transform our documents into feature vectors. Here, we will use the sklearn function `sklearn.feature_extraction.text.CountVectorizer`. This module will transform the documents into BoWs (bag of words), using words counts and combinations of words (i.e. ngram) as columns.

> **Ex 15.3.1:** Make a train-test (50-50) split before applying any transformation to avoid *data leakage* and to set up a clean hyperparameter search using the train set, and saving the test set for crossvalidation.

*Hint:* Use `sklearn.model_selection.train_test_split`.
_____no_output_____

#[Answer Ex 15.3.1]_____no_output_____

> **Ex 15.3.2:** Preproce the documents as in Ex 15.1.3. Then, initialize the `sklearn.feature_extraction.text.CountVectorizer`.
The countvectorizer has many possible inputs: The important ones here are the tokenization function (use the one defined above, i.e. `tokenizer=tokenizer.tokenize`) and `ngram_range=(1,2)`. Finally, run the `.fit()` function with the train documents as input._____no_output_____

#[Answer Ex 15.3.2]_____no_output_____

> **Ex 15.3.3:** Transform train and test documents to a sparse vector using the `transform` command on both the test and the train documents. Assign these to two variables, e.g. `X_train` and `X_test`. Afterwards, inspect the resulting dimensions of each input vector using the `.shape` function._____no_output_____

#[Answer Ex 15.3.3]_____no_output_____

> **Ex 15.3.4:** Convert ratings to binary outcome. Here we make a decision to count ratings as positive if above 3 and otherwise negative. Define ***y_train*** and ***y_test*** as a binary variables expressing whether it is higher than 3 or not._____no_output_____

#[Answer Ex 15.3.4]_____no_output_____

Now we are ready to train our model. We will use the penalized logistic regression which is widely used for training baseline in within natural language processing (NLP) - especially with a little twist see [Wang and Manning 2012](https://www.aclweb.org/anthology/P12-2018).
As always we should we remember to do our parameter search using the training data only.

> **Ex 15.3.5:** In this case we will do only a search over the penalizationterm C and, therefore, we can use the `sklearn.linear_model.LogisticRegressionCV()` which handles kfold-based parameter search directly. The function takes as arguments the range of values to be searched and the number of folds used for cross-validation.

*Hint:* You can use a `LogisticRegressionCV()` with the arguments `Cs=np.linspace(0.1,2,10)`,`cv=10` and `verbose=2`. You can vary these arguments if you want._____no_output_____

# [Answer Ex 15.3.5]_____no_output_____

> **Ex 15.3.6:** Report the performance of the classifier. You should report the accuracy. Additionally, you might want to report precision, recall, f1, AUC and roc-auc-curve.

*Hint:* These measures can be computed using the `sklearn.metrics` package. You might want to define a function that takes your classifier and data as input and returns the metrics._____no_output_____

# [Answer Ex 15.3.6]_____no_output_____

> **Ex 15.3.7:** Create a column in the dataframe applying the classifier to all documents.
1. First transform documents using vectorizer from before.
2. Get the predicted probability of a positive review: `clf.predict_proba(X)`. Note that you only need the second column - it predicts probabilities for both classes 0 and 1._____no_output_____

# [Answer Ex 15.3.7]_____no_output_____

>**Ex 15.3.8 (BONUS):** Compute the correlation of the computed sentiment with the given rating for all three methods. Would you expect a positive or a negative correlation? Note that there are different categories describing the ratings' subjects (see the ***categories*** column). You can compute the correlation averaging the ratings and the sentiment scores over these categories._____no_output_____

# [Answer Ex 15.3.8 (BONUS)]_____no_output_____


# Notebook from rafaelscnunes/mab786
Path: experiments/vectorizing_wisard.ipynb

<code>
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score

df = {
    "cooking": pd.read_csv('../dataset/processed/cooking.csv'),
    "crypto": pd.read_csv('../dataset/processed/crypto.csv'),
    "robotics": pd.read_csv('../dataset/processed/robotics.csv'),
    "biology": pd.read_csv('../dataset/processed/biology.csv'),
    "travel": pd.read_csv('../dataset/processed/travel.csv'),
    "diy": pd.read_csv('../dataset/processed/diy.csv'),
    #"physics": pd.read_csv('physics.csv'),
}

from WiSARD import WiSARD
import numpy as np
import pandas as pd
import math
import sys
import random
import matplotlib.pyplot as plt
import itertools
def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, cm[i, j],
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('Classe verdadeira')
    plt.xlabel('Classe prevista')_____no_output_____data = []
for file in df:
    for i in range(df[file].shape[0]):
        if type(df[file].iloc[i]['title']) is str:
            data.append(df[file].iloc[i]['title'])
        if type(df[file].iloc[i]['content']) is str:
            data.append(df[file].iloc[i]['content'])_____no_output_____X = []
y = []
for file in df:
    for i in range(2000):
        if type(df[file].iloc[i]['content']) is str:
            X.append(df[file].iloc[i]['content'])
            y.append(file)

X_test = []
y_test = []
for file in df:
    for i in range(2001,2501):#range(df[file].shape[0]):
        if type(df[file].iloc[i]['content']) is str:
            X_test.append(df[file].iloc[i]['content'])
            y_test.append(file)
            _____no_output_____sv = CountVectorizer(binary=True, max_df=0.025, min_df=0.005)
d = sv.fit(data)

X_ = sv.transform(X)
X_test_ = sv.transform(X_test)

print('Vocabulary size: ' + str(len(sv.vocabulary_)))

w = WiSARD(8, seed=random.randint(0, 2**32-1), ignore_zero_addr=True)
w.fit(X_.toarray(), y)
result = w.predict(X_test_.toarray())

# Making the Confusion Matrix
cm = confusion_matrix(y_test, result, labels=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'])
#class_names = ['A','B','C','D','E','F','G','H']
# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cm, classes=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'], title='Matriz de Confusao')
plt.show()

print('Accuracy: ' + str(accuracy_score(y_test, result)))Vocabulary size: 832
sv = CountVectorizer(binary=True, max_df=0.4, min_df=0.01)
d = sv.fit(data)

X_ = sv.transform(X)
X_test_ = sv.transform(X_test)

print('Vocabulary size: ' + str(len(sv.vocabulary_)))

w = WiSARD(8, seed=random.randint(0, 2**32-1), ignore_zero_addr=True)
w.fit(X_.toarray(), y)
result = w.predict(X_test_.toarray())

# Making the Confusion Matrix
cm = confusion_matrix(y_test, result, labels=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'])
#class_names = ['A','B','C','D','E','F','G','H']
# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cm, classes=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'], title='Matriz de Confusao')
plt.show()

print('Accuracy: ' + str(accuracy_score(y_test, result)))Vocabulary size: 488
Confusion matrix, without normalization
[[238  20  69   9  29 135]
 [  1 434  35   8   5  17]
 [  9  62 309   9  24  87]
 [ 21  78 119 140  51  91]
 [  7  22  48  10 360  53]
 [ 11   7  35   9  15 423]]
sv = CountVectorizer(binary=True, max_df=0.9, min_df=0.005)
d = sv.fit(data)

X_ = sv.transform(X)
X_test_ = sv.transform(X_test)

print('Vocabulary size: ' + str(len(sv.vocabulary_)))

w = WiSARD(8, seed=random.randint(0, 2**32-1), ignore_zero_addr=True)
w.fit(X_.toarray(), y)
result = w.predict(X_test_.toarray())

# Making the Confusion Matrix
cm = confusion_matrix(y_test, result, labels=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'])
#class_names = ['A','B','C','D','E','F','G','H']
# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cm, classes=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'], title='Matriz de Confusao')
plt.show()

print('Accuracy: ' + str(accuracy_score(y_test, result)))Vocabulary size: 980
Confusion matrix, without normalization
[[299  24  44   8  20 105]
 [  2 455  29   4   2   8]
 [  4  34 407   3  11  41]
 [  9  76 130 179  26  80]
 [  1  11  26   5 429  28]
 [  5   8  26   3   7 451]]
sv = CountVectorizer(binary=True, max_df=0.3, min_df=50)
d = sv.fit(data)

X_ = sv.transform(X)
X_test_ = sv.transform(X_test)

print('Vocabulary size: ' + str(len(sv.vocabulary_)))

w = WiSARD(8, seed=random.randint(0, 2**32-1), ignore_zero_addr=True)
w.fit(X_.toarray(), y)
result = w.predict(X_test_.toarray())

# Making the Confusion Matrix
cm = confusion_matrix(y_test, result, labels=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'])
#class_names = ['A','B','C','D','E','F','G','H']
# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cm, classes=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'], title='Matriz de Confusao')
plt.show()

print('Accuracy: ' + str(accuracy_score(y_test, result)))Vocabulary size: 7277
Confusion matrix, without normalization
[[403  16  19   3   9  50]
 [  0 485  13   0   1   1]
 [  0   5 482   2   3   8]
 [  5  24  49 367  12  43]
 [  1   6  13   0 472   8]
 [  1   3  18   3   1 474]]
sv = CountVectorizer(binary=True, max_df=0.4, min_df=30)
d = sv.fit(data)

X_ = sv.transform(X)
X_test_ = sv.transform(X_test)

print('Vocabulary size: ' + str(len(sv.vocabulary_)))

w = WiSARD(8, seed=random.randint(0, 2**32-1), ignore_zero_addr=True)
w.fit(X_.toarray(), y)
result = w.predict(X_test_.toarray())

# Making the Confusion Matrix
cm = confusion_matrix(y_test, result, labels=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'])
#class_names = ['A','B','C','D','E','F','G','H']
# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cm, classes=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'], title='Matriz de Confusao')
plt.show()

print('Accuracy: ' + str(accuracy_score(y_test, result)))Vocabulary size: 9541
Confusion matrix, without normalization
[[400  19  25   1   7  48]
 [  0 482  16   0   1   1]
 [  0   6 479   1   1  13]
 [  5  28  56 366  12  33]
 [  1   7  17   0 465  10]
 [  1   4  21   4   2 468]]
sv = CountVectorizer(binary=True, max_df=0.5, min_df=20)
d = sv.fit(data)

X_ = sv.transform(X)
X_test_ = sv.transform(X_test)

print('Vocabulary size: ' + str(len(sv.vocabulary_)))

w = WiSARD(8, seed=random.randint(0, 2**32-1), ignore_zero_addr=True)
w.fit(X_.toarray(), y)
result = w.predict(X_test_.toarray())

# Making the Confusion Matrix
cm = confusion_matrix(y_test, result, labels=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'])
#class_names = ['A','B','C','D','E','F','G','H']
# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cm, classes=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'], title='Matriz de Confusao')
plt.show()

print('Accuracy: ' + str(accuracy_score(y_test, result)))Vocabulary size: 11733
Confusion matrix, without normalization
[[398  16  20   2   8  56]
 [  0 485   9   1   2   3]
 [  0   4 482   1   2  11]
 [  2  30  54 356  15  43]
 [  1   8  15   0 466  10]
 [  1   5  22   4   2 466]]
sv = CountVectorizer(binary=True, max_df=0.3, min_df=50)
d = sv.fit(data)

X_ = sv.transform(X)
X_test_ = sv.transform(X_test)

print('Vocabulary size: ' + str(len(sv.vocabulary_)))

w = WiSARD(2, seed=random.randint(0, 2**32-1), ignore_zero_addr=True)
w.fit(X_.toarray(), y)
result = w.predict(X_test_.toarray())

# Making the Confusion Matrix
cm = confusion_matrix(y_test, result, labels=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'])
#class_names = ['A','B','C','D','E','F','G','H']
# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cm, classes=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'], title='Matriz de Confusao')
plt.show()

print('Accuracy: ' + str(accuracy_score(y_test, result)))Vocabulary size: 7277
Confusion matrix, without normalization
[[405  15  23   3   7  47]
 [  0 484  14   0   1   1]
 [  0   5 482   2   1  10]
 [  4  28  57 360  15  36]
 [  0   5  18   1 466  10]
 [  1   5  19   3   3 469]]
sv = CountVectorizer(binary=True, max_df=0.3, min_df=50)
d = sv.fit(data)

X_ = sv.transform(X)
X_test_ = sv.transform(X_test)

print('Vocabulary size: ' + str(len(sv.vocabulary_)))

w = WiSARD(16, seed=random.randint(0, 2**32-1), ignore_zero_addr=True)
w.fit(X_.toarray(), y)
result = w.predict(X_test_.toarray())

# Making the Confusion Matrix
cm = confusion_matrix(y_test, result, labels=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'])
#class_names = ['A','B','C','D','E','F','G','H']
# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cm, classes=['cooking', 'crypto', 'robotics', 'biology', 'travel', 'diy'], title='Matriz de Confusao')
plt.show()

print('Accuracy: ' + str(accuracy_score(y_test, result)))Vocabulary size: 7277
Confusion matrix, without normalization
[[399  10  22   2   8  59]
 [  0 482  13   1   3   1]
 [  1   8 478   1   1  11]
 [  3  23  51 370  17  36]
 [  0   3  20   0 470   7]
 [  1   5  20   4   2 468]]

</code>


# Notebook from pachterlab/BP_2020
Path: notebooks/fix_lung_markers.ipynb

<code>
import glob
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
import matplotlib.patches as mpatches
import scanpy as sc
from scipy import stats

from collections import OrderedDict
from sklearn.decomposition import TruncatedSVD
from sklearn.manifold import TSNE
from sklearn.preprocessing import scale

from sklearn.cluster import KMeans
from sklearn.preprocessing import normalize
from sklearn.preprocessing import LabelEncoder
from sklearn.neighbors import NeighborhoodComponentsAnalysis
from matplotlib import cm
from matplotlib.lines import Line2D

def nd(arr):
    return np.asarray(arr).reshape(-1)
def yex(ax):
    lims = [np.min([ax.get_xlim(), ax.get_ylim()]),
            np.max([ax.get_xlim(), ax.get_ylim()])]

    # now plot both limits against eachother
    ax.plot(lims, lims, 'k-', alpha=0.75, zorder=0)
    ax.set_aspect('equal')
    ax.set_xlim(lims)
    ax.set_ylim(lims)
    return ax

def trim_axs(axs, N):
    """little helper to massage the axs list to have correct length..."""
    axs = axs.flat
    for ax in axs[N:]:
        ax.remove()
    return axs[:N]

import warnings
warnings.filterwarnings('ignore')

fsize=20

plt.rcParams.update({'font.size': fsize})
%config InlineBackend.figure_format = 'retina'/home/sina/bin/miniconda3/lib/python3.7/site-packages/anndata/_core/anndata.py:21: FutureWarning: pandas.core.index is deprecated and will be removed in a future version.  The public classes are available in the top-level namespace.
  from pandas.core.index import RangeIndex
df = pd.read_excel("../reference/Table_S4.10X.xlsx", sheet_name=None)_____no_output_____dfs = []
for k,v in df.items():

    tmp = df[k]

    celltype = tmp.columns[0]
    tmp.columns = tmp.iloc[0]
    tmp = tmp.drop(0)
    tmp["celltype"] = celltype

    dfs.append(tmp)_____no_output_____df = pd.concat(dfs, ignore_index=True)_____no_output_____df.to_csv("write/quake_lung_markers.csv")_____no_output_____
</code>