# Notebook from LaurestinusV/CSCI-E-96
Path: notebooks/01_PitfallsInDataMining.ipynb

# Assignment 01 - Pitfalls in Data Mining
## CSCI E-96

The goal of data mining is to find important relationships in large complex datasets. These dataset typically contain a large number of variables. The **high-dimensional** nature of the data leads to some commonly encountered pitfalls which lead to incorrect inferences.

In this assignment you will gain a bit of experience with three important concepts in data mining:

1. **False Discovery Rate Control:** The goal of data mining is to find important relationships in large complex datasets. These dataset typically contain a large number of variables. The **high-dimensional** nature of the data leads to some commonly encountered pitfalls which lead to incorrect inferences. A related problem is cutting off a large-scale analysis when a desired relationship is 'found'. This practice of **p-value mining** often leads to unwarranted inferences. You will apply false discovery rate (FDR) control methods to address this problem.
2. **Key-Value Pairs:** Large scale data is typically managed using key-value (KV) pairs. The exercises in this assignment give you some experience working with KV pair data management.
3. **Map and Reduce Processes:** Much of large scale data mining requires use of a split-apply-combine approach. The data is split into manageable chunks, analytic transformations are applied, and the result combined or aggregated. A commonly used class of a split-apply-combine algorithm is MapReduce.

In order to keep the scope of this assignment manageable, you will use limited versions of KV pair management and MapReduce. Specifically, you will use common Python tools to implement these concepts rather than dedicated large scale analytic platforms. _____no_output_____## Multiple Hypothesis Tesing

Testing multiple hypothesis in high-dimensional data can be problematic. Exhaustively testing all pairwise relationships between variables in a data set is a commonly used, but generally misleading from of **multiple comparisons**. The chance of finding false significance, using such a **data dredging** approach, can be surprisingly high.

In this exercise you will perform multiple comparisons on only 20 **identically distributed independent (iid)** variables. Ideally, such tests should not find significant relationships, but the actual result is quite different.

To get started, execute the code in the cell below to load the required packages. _____no_output_____

import pandas as pd
import numpy as np
import numpy.random as nr
from scipy.stats import ttest_ind, f_oneway
from itertools import product, combinations
import matplotlib.pyplot as plt
import seaborn as sns

pd.set_option("display.max_rows", None, "display.max_columns", None)_____no_output_____

In this exercise you will apply a t-test to all pairwise combinations of identical Normally distributed variables. In this case, we will create a data set with 20 iid Normal distributions of 1000 samples each. Execute the code in the cell below to find this data and display the mean and variance of each variable.  _____no_output_____

ncolumns = 20
nr.seed(234)
normal_vars = nr.normal(size=(1000,ncolumns))
print('The means of the columns are\n', np.mean(normal_vars, axis = 0))
print('\nThe variances of the columns are\n', np.var(normal_vars, axis = 0))The means of the columns are
 [-1.16191649e-01  2.80829317e-02 -1.78516419e-02 -1.44691489e-02
  3.03718152e-02  1.20007442e-02 -9.58845606e-05  1.98662580e-03
  4.94154934e-02 -4.11640866e-02 -6.32977862e-03 -5.93868192e-02
 -2.56373595e-02  1.43568791e-02 -1.44725765e-02 -1.37023955e-02
  1.80622439e-02  5.87029691e-02 -2.02650514e-02 -1.56346106e-02]

The variances of the columns are
 [0.94834508 1.04744241 1.0258018  0.96977571 1.0089001  1.04113864
 1.00657222 0.99192594 1.04713487 1.04329434 1.04023108 0.96791346
 1.03706907 1.07179865 1.01431404 1.05060289 1.02054329 0.9686211
 1.02810287 0.99521555]

Notice that means and variances are close to 0.0 and 1.0 respectively. As expected, there is not much difference between these variables.

How many of these t-tests will show **significance** at the 0.05 cut-off level? There are 380 pairwise combinations, so we expect to find a number of falsely significant test results at this level. To find out, complete and execute the code in the cell below to filter the test results and print those that show significance. _____no_output_____### Creating a hash

The goal of this exercise is to compute pairwise hypothesis tests of the differences in means for each of the iid Normal vectors. As an intermediate step you will create a Pandas data frame using a hash of the keys of the vectors. The data frame will contain the **key-value**, $(K,V)$, pairs. Each key must represent an index for the two vectors used to compute the test statistic. The keys will then be used to index the results of these hypothesis tests.

The question is, how can we create a hash from the keys for the pair of vectors? In this case to we will use a simple, but far from optimal hash. For the two vector indicies $i, j$, for some key and modulo, $m$, we will compute the hash as:

$$h(i,j) = (i + key*j) mod m$$

> **Computational Note:** The Pandas data frame is an efficient and reasonably scalable **hash table**. The hash function used depends on the type of the key; integer, string, etc. The resulting dictionary of key-value pairs, $(K,V)$, can therefore be access in far less than linear time, often about $O(log(N))$.

If you are not familiar with Python dictionaries you can find a short tutorial [here](https://www.tutorialspoint.com/python_data_structure/python_hash_table.htm), as well as many other places on the web._____no_output_____> **Exercise 1-1:** Given that our space of vectors is actually quite small, just 20, we do not need a sophisticated and scalable hash function. This hashed key will then be used to store and retrieve the values using a Python dictionary, in about $O(log(N))$ time.

> In this exercise you will test a simple hash function and its inverse. Examine the code below and notice that the hash function encodes the two indexes into a single integer by simple additional and multiplication. The modulo operation limits the size of the hash table. However, to keep things simple you will not need to implement any hash collision resolution mechanism. As a result, the size of the table is set much larger than required.

> To test this hash, do the following:
> 1. Create a function called hash function to compute the hash. The arguments to the function are $i$ and $j$, the `hash\_key` and the `modulo\_multiplier`. The defaults of the arguments are $hash\_key=1024$ and $modulo\_multiplier=32$. The modulo number is $hash\_key * modulo\_multiplier$, e.g. $modulo = 32,768$. The multiplier is the ratio of expected values stored, $n$, to the number of unique hash keys, $m$, e.g. the ratio $m/n$.
> 2. Using the Python [ittertools.combinations](https://docs.python.org/3/library/itertools.html#itertools.combinations) function create all unique pairwise combinations of indexes i and j. The arguments to this function are the indexes to the iid Normal vectors. The iterator is `range(ncolumns)` choose 2, since these comparisons are pairwise.
> 3. Within this loop call the hash with the values of $i$ and $j$ as arguments.
> 3. On a single line print the following; the values of i and j, the hash key value, but only if $i \le 6$. The restriction is to keep the printed output shorter.   _____no_output_____

def hash_function(i, j, hash_key=1024, modulo_multiplier=32):
    ## Put your code below.

for i,j in combinations(range(ncolumns), 2):
    if i < 6:
        hash = hash_function(i,j)
        print('i = ' + str(i) + '  j = ' + str(j) + '   hash = ' + str(hash))_____no_output_____

> Examine the key pairs and the hash values. The question is, are there any hash collisions? This can be done as follows:
> 5. Compute a list of the hash values for all combinations of $i$ and $j$.
> 6. Print the length of the list.
> 7. Print the length of the unique values of the hash. You can find the unique values in a list with the [numpy.unique](https://numpy.org/doc/stable/reference/generated/numpy.unique.html) function. _____no_output_____

## Put your code below.

_____no_output_____

> Examine the results you have printed. Is there any evidence of hash key collisions?
> The ratio of $m/n$ is deliberately kept high since the simple hash function has no collision resolution mechanism. Optionally, you can try reducing this ration (the multiplier) to 16 and 8, noting the increase in hash collisions.
> **End of exercise.**_____no_output_____### The map process

We are constructing this example a map and a reduce process. The processes are intended to compute the hypothesis test for differences of means between all the pairs of vectors. The first step is the map process, which creates the keys, or values of $i$ and $j$ for these pairs.

> **Exercise 1-2:** You will now create the code for the map task which build a data frame with $i, j$ key pairs indexed by the hash. By the following steps you will create code that represents a map task.
> 1. Create a data frame with two columns $i$ and $j$ with rows $= hash_key * modulo_multiplier $ and set all values to $= numpy.nan$.
> 2. Create a loop over all combinations of the pairs of i and j.
> 2. Compute the hash key value for the indexes, i and j.
> 4. Add the $i$ and $j$ values to the row indexed by the hash key.
> 5. Return the hash table.
> 6. Execute the function to create the hash table.
> 7. Compute and print the length of the hash table. _____no_output_____

def map_hypothesis(vars, hash_key=1024, modulo_multiplier=32):
    ## Put your code below.

    for i,j in combinations(range(ncolumns), 2):
        ## Put your code below.
        ## Compute the hash key and added the values to the
        ## row of the data frame.

    return hash_table

hash_table = map_hypothesis(normal_vars)_____no_output_____

> **End of exercise.**_____no_output_____### The shuffle and reduce task

Now that you have the keys for the pairwise combinations of the vectors it is time to perform the reduce process. The reduce process computes the pair-wise t-statistics and p-values. These statistical values are indexed by the keys of the pair of vectors. This process reduces the full vectors of values down to just two numbers for each pair of vectors.

> **Exercise 1-3:** You will now create and apply the following code for the reduce process:
> 1. Create an empty data frame with columns, `i`, `j`, `t_statistic`, and `p_value`.
> 2. Using a for loop iterate over all possible (hashed) keys of the data frame. An if statement is used to test if these are valid values of the key, i. Use the [numpy.isnan](https://numpy.org/doc/stable/reference/generated/numpy.isnan.html) function for this test.
> 3. Extract the values of i and j from the input data frame.
> 4. Using keys, compute the t-statistic and p-value using [scipy.stats import ttest_ind](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html).
> 5. Append a row to the output data frame.
> 6. Return the data frame, sorted in ascending order, using the [Pandas.DataFrame.sort_values](https://turned.pydata.org/docs/reference/api/pandas.DataFrame.sort_values.html) method and re-indexed using the [Pandas.DataFrame.reset_index](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reset_index.html) method.
> 7. Execute your function and save the returned data frame._____no_output_____

def reduce_significance(hash_table, values):
    ## Create a data framreturn the results of the
    ## the reduce process. The results are grouped by the first
    ## index i.
    test_results = pd.DataFrame(columns=['i','j','t_statistic','p_value'])

    ## As a substitute for a shuffle we will use a simple search
    ## through the data frame
    for hash in range(hash_table.shape[0]):
    ## Put your code below.
        if not np.isnan(hash_table.iloc[hash,0]):

    ## Given the i,j pairs we need to compute the t-statistic and the p-value.
    ## This is the reduce step, since for each i,j pair there is only
    ## a t-statistic and a p-value.
    ## Put your code below.

    ## Sort and return the results
    #test_results.sort_values('p_value', axis=0, ascending=True# , inplace=True)
    return test_results.sort_values('p_value', axis=0, ascending=True).reset_index(drop=True)

test_stats = reduce_significance(hash_table, normal_vars)       _____no_output_____

> 8. In the cell below, create a filter for pair test cases which are significant and save these cases in a data frame.
> 9. Print the number (len) of significant results.
> 10. Print the rows with the significant test results._____no_output_____

significance_level = 0.05
## Put your code below.

_____no_output_____

> Notice the large number of apparently significant tests. Answer the following questions:
> 1. Is the number of false positive cases higher than expected?
> 2. Examine which of the iid Normal vectors contribute to the false positive results. Are there vectors which contribute multiple times?
> **End of exercise.**_____no_output_____### Bonferroni correction

Several adjustments to the multiple comparisons problem have been proposed. In Dunn published a method know as the **Bonferroni correction** in 1961. The Bonferroni correction is a widely used method to reduce the false positive rate of hypothesis tests.  The adjustment is simple:
$$\alpha_b = \frac{\alpha}{m}\\
with\\
m =\ number\ of\ groups$$

Can the Bonferroni correction help? Yes, by greatly increasing the confidence level required for a statistically significant result. The problem with the Bonferroni correction is the reduction in power as the  grows smaller. For big data problems with large numbers of groups, this issue can be especially serious.

**Exercise 1-4:** You will now apply the Bonferroni correction to the iid Normal vectors. To do so, you will compute the Bonferroni threshold and the apply it to the p-values:   _____no_output_____

## Put your code below

_____no_output_____

> Even with the Bonferroni correction we have some false significance tests, if only just barely!
> **End of exercise.**

But, can we detect small effect with Bonferroni correction, as this method significantly reduces power of tests? Execute the code in the cell below, which compares a standard Normal to a Normal with a small mean (effect size), to find out. _____no_output_____

nr.seed(567)
print(significance_bonferroni)
ttest_ind(normal_vars[:,0], nr.normal(loc = 0.01, size=(1000,1)))_____no_output_____

Given the Bonferroni correction, this difference in means would not be found significant. This illustrates the downside of the correction, which may prevent detection of significant effects, while still finding false significance. _____no_output_____## False Discovery Rate Control Methods

We have seen the potential pitfalls of multiple hypothesis testing. Further, we have seen that a simple approach to **false discovery rate (FDR) control** is not effective. You will now apply more sophisticated FDR control methods to control the FDR.

Inflammatory bowel disease is an auto immune disease that is characterized by chronic inflammation in the digestive tract. In 2020, there were around 2.5 million people with inflammatory bowel disease in the United States. It is estimated that the prevalence of IBD among U.S. population will rise to around 3.5 million by 2030.There are two forms of IBD: Ulcerative Colitis (UC) and Crohn’s disease (CD).

The specific problem we will explore is to determine which genes lead to expression of a certain disease. In this example, there are gene expression data for 97 patients. Some of these patients have ulcerative colitis and others have Crohn's disease, which are believed to be genetically inherited.

One approach to this problem is to perform hypothesis tests on the expression of the genes between patients with the two conditions. Since there are over 10,000 genes there is considerable chance for false discovery. Therefore, careful application of FDR control is required.

To continue with the example, execute the code in the cell below to load the data and print the dimensionality of the data frame. _____no_output_____

gene_data = pd.read_csv('../data/ColonDiseaseGeneData-Cleaned.csv')
print(gene_data.shape)_____no_output_____

There are data from 97 patients for 10,497 genes. A large number of hypothesis tests are required!

Execute the code in the cell below to view the first 5 columns of the data frame, which includes the expression of the first 4 genes.  _____no_output_____

 print(gene_data.iloc[:,:5])      _____no_output_____

### Holm's method

You will apply two FDR control methods to these data.These methods attempt to conod trol the FDR while not being overly conservative like the Bonferronic correction. The first of these Holm's method.

The Holm's method operates on the ordered set of p-values, $D = \{ p_{(1)}, p_{(2)}, p_{(3)}, \ldots, p_{(n)} \}$. The threshold for the $ith$ p-value, $p(i) is:

$$p(i) \le Threshold(Holm's) = \frac{\alpha}{N - i + 1}$$

For example: for the 10th ordered p-value with 1,000 total tests (genes) and significance level of 0.05, the cutoff is:

$$p(10) \le \frac{0.05}{1000 - 10 + 1} = 0.00005045$$_____no_output_____### Map process

> **Exercise 01-4:** To start the processing of these data you will first create and execute code for a map process. The map process groups the data by the patient's disease into data frame, ulcerative, crohns. The keys for each of these key-value pairs are the gene identifier. Notice that one key is all that is needed in this case. Now do the following to create and execute a function, `map_gene`:
> 1. Create a logical mask and group the values by `Disease State` into two data frames.
> 2. Return the transpose of the two data frames, removing the `Disease State` values. The result of this operation should be data frames with gene expressions in the columns and the gene identifier as the row index. _____no_output_____

def map_gene(gene_data):
    ## First, separate the columns by disease type
    ## Put your code below.

ulcerative, crohns = map_gene(gene_data)    _____no_output_____

> 3. Execute the code in the cells below to display the heads of these data frames and examine the results. _____no_output_____

ulcerative.head()_____no_output_____crohns.head()_____no_output_____

### Reduce process

> **Exercise 01-5:** With the key-value pairs organized by disease state, it is time to create and execute code of a reduce process. The reduce process will compute the pairwise t-statistics and p-values for each gene and return the sorted results. Specifically, your `gene_test` with arguments of the two mapped data frames will do the following:
> 1. Create an empty data frame with columns gene, t_statistics, and p-value.
> 2. A for loop iterates over the keys of either of the data frames.
> 3. Compute the t-statistic and p-value for the gene (key).
> 4. Append the results to the data frame.
> 5. Sort the results data frame, inplace, into ascending order.
> 6. Return the resulting data frame._____no_output_____

def gene_test(ulcerative, crohns):
    test_results = pd.DataFrame(columns=['gene','t_statistic','p_value'])
    ## Put your code below.

    return test_results.set_index('gene')

gene_statistics = gene_test(ulcerative, crohns)    _____no_output_____

### Significance of results

With the gene data reduced to the t-test statistics, you will now determine the significance of these tests. It is important to understand that scientists believe that expression of a disease, like Corhn's, is only in a small number of genes.

> **Exercise 01-6:** As a first step in understanding the gene expression significance complete and execute the code in the cell below to find the number of 'significant' genes using the simple single hypothesis test cutoff criteria.  _____no_output_____

significance_level =0.05
## Put your code below.

_____no_output_____

> Does this large number of 'statistically significant' results appear credible, given that only a few genes are thought to have significant expression for this disease?
> **End of exercise.**

> **Exercise 01-7:** We have already seen that the Bonferroni correction is a rather conservative approach to testing the significance of large numbers of hypotheses. You will now use the Bonferroni correction to test the significance of the gene expression, by completing the code in the cell below.  _____no_output_____

## Put your code below.
_____no_output_____

> The foregoing result seems reasonable, but is it too conservative?_____no_output_____> **Exercise 01-08:** You will now apply the Holms method to determining significance of the gene expression test results. In the cell below complete the `holms_significance` function with arguments of the results data frame and the significance level. This function does the following:
> 1. Find the number of test results and compute the numerator used for the cutoff calculation.
> 2. Compute the vector of thresholds using the Holms formula. Use the Python `range`function to get the values of the index i. But, keep in mind that range produces a zero-indexed iterator, and the algorithm needs a one-indexed list.  Use the [numpy.divide](https://numpy.org/doc/stable/reference/generated/numpy.divide.html) function to perform the vector divide. Save these threshold values in a data frame in a 'holms_threshold' column.
> 3. Using the threshold values compute a logical vector and save it in a column names 'significance' in the data frame.
> 4. Return the data frame.
> Finally, execute the function and save the results in a data frame. Then find the length of the subset where the 'significance' value is True. _____no_output_____

def holms_significance(test_results, significance):
    ## First compute the thresholds for each of the ordered tests
    ## Put your code below.

    ## Now we test the significance of the ordered p-values

    return test_results

holms_results = holms_significance(gene_statistics, significance_level)
len(holms_results.loc[holms_results.loc[:,'significant'],:])_____no_output_____

> Despite the general properties that the Holm's method is considered less conservative than the Bonferroni correction the results agree in this case. Does this agreement give you some confidence in the result and why?
> **End of exercise.**_____no_output_____You can visualize the results of the Holm's method test. The plot has two key elements:
1. Plot the curve of the p-values vs. the order number, i. The line is color coded by significance or not.
2. Plot the threshold line. This line is straight since the threshold is a linear function of i._____no_output_____

def plot_significance(results, threshold):
    results['number'] = range(len(results))
    fig, ax = plt.subplots(figsize=(8, 6))
    sns.lineplot(x='number',y=threshold, data=results, ax=ax, color='black', linewidth=0.5)
    sns.scatterplot(x='number',y='p_value', hue='significant', data=results, s=3, ax=ax)
    ax.set_title('Significance of gene expression')
    ax.set_xlabel('Gene number')
    ax.set_ylabel('p-value')

plot_significance(holms_results.iloc[:500,:], 'holms_threshold')    _____no_output_____

Notice the following about this plot:
1. The p-value significance line crosses the threshold point at an apparent break point.
2. The significant p-values are all very small since there are so many tests._____no_output_____### Benamini-Hochberg FDR Control

The Benamini-Hochberg FDR control algorithm is another way to control false discoveries. Stat with an ordered set of $n$ p-values, $D = \{ p_{(1)}, p_{(2)}, p_{(3)}, \ldots, p_{(n)} \}$ we define a false discovery rate, $q$:

$$FDR(D) \le q$$

The cutoff threshold for the ith p-value is then:
$$p_{(i)} \le Threshold(D_q) = \frac{q}{n} i$$_____no_output_____> **Exercise 01-9:** In this exercise you will apply the Benamini-Hochberg FDR control algorithm for testing the significance of the gene expressions. The `BH_significance` function is quite similar to the Holm's method function you have already created. Given the large number of genes you must use a low false discovery rate, $0.001$, or 1 out of 1,000.
> Execute your function, saving the result. Then print the number of significant cases. _____no_output_____

def BH_significance(test_results, false_discovery_tollerance):
    ## First compute the thresholds for each of the ordered tests
    ## Put your code below.

    ## Now we test the significance of the ordered p-values

    return test_results

BH_results = BH_significance(gene_statistics, 0.001)
len(BH_results.loc[BH_results.loc[:,'significant'],:])_____no_output_____

> The result is similar to the first two FDR control methods. Given the false discovery parameter of 0.0001 do you think this is a reasonable result? _____no_output_____Finally, execute the code in the cell below and examine the resulting plot._____no_output_____

plot_significance(BH_results.iloc[:500,:], 'bh_threshold')    _____no_output_____

> **Exercise 01-10**: Bonus question. Compare the plot above to the foregoing plot for Holm's method. Are the breaks in slope of the p-value curve at the crossing point with the threshold values reasonable in both cases? _____no_output_____##### Copyright 2021, Stephen F. Elston. All rights reserved. _____no_output_____


# Notebook from alexfmsu/pyquantum
Path: ipynb/.ipynb_checkpoints/TC-checkpoint.ipynb

# TC
from PyQuantum.TC.Cavity import Cavity
from PyQuantum.TC.Hamiltonian import Hamiltonian

from PyQuantum.TC.WaveFunction import WaveFunction

from PyQuantum.TC.Evolution import run_wf_____no_output_____# Common
from PyQuantum.Common.LoadPackage import *
from PyQuantum.Common.STR import *

from PyQuantum.Common.Tools import mkdir
from PyQuantum.Common.Print import *

# from PyQuantum.Common.PyPlot import PyPlot3D
from PyQuantum.TC.PlotBuilder3D import PlotBuilder3D

# from shutil import copyfile
# from numpy.random import rand_____no_output_____config = load_pkg("config", "../PyQuantum/TC/config.py")

mkdir(config.path)
# copyfile("PyQuantum/TC/config.py", config.path + '/config.py')_____no_output_____cavity = Cavity(
    wc=config.wc,
    wa=config.wa,
    g=config.g,
    n_atoms=config.n_atoms,
    n_levels=config.n_levels
)

cavity.print()Cavity:
wc: 21.506 GHz

wa: 21.506 GHz

 g: 2.151 GHz

 n_atoms: 2

print("T:", config.T)
print("nt:", config.nt)
print("dt:", config.dt)

print()T: 4.9999999999999996e-05
nt: 5000
dt: 9.999999999999999e-09

H = Hamiltonian(
    capacity=config.capacity,
    cavity=cavity,
    RWA=True,
    reduced=True
)

H.iprint()
H.df_____no_output_____w_0 = WaveFunction(states=H.states, init_state=config.init_state)

w_0.print()[0, [0, 2]] (1+0j)
[0, [1, 1]] 0j
[0, [2, 0]] 0j
[1, [0, 1]] 0j
[1, [1, 0]] 0j
[2, [0, 0]] 0j
run_wf(
    w_0=w_0,
    H=H,
    dt=config.dt,
    nt=config.nt,
    config=config,
    fidelity_mode=True
){0, 2}
{0: [0, [0, 2]], 1: [0, [1, 1]], 2: [0, [2, 0]], 3: [1, [0, 1]], 4: [1, [1, 0]], 5: [2, [0, 0]]}
plt = PlotBuilder3D()

plt.set_width(950)
plt.set_height(800)

title = ""

if config.capacity - config.n_atoms > 0:
    title += "" + str(config.capacity - config.n) + \
        " фотонов в полости" + "" + "

"
else:
    title += "" + "empty cavity" + "" + "

"

title += ""
title += "n_atoms = " + str(config.n_atoms)

title += "
"
title += "
wc = " + wc_str(config.wc)
title += "
wa = " + wa_str(config.wa)
title += "

 g/hwc = " + str(config.g/config.wc)
title += "
"
title += "
"
title += ""

plt.set_title(title)

y_scale = 1

if config.T < 0.25 * config.mks:
    y_scale = 0.1
elif config.T <= 0.5 * config.mks:
    y_scale = 0.025
elif config.T == 0.5 * config.mks:
    y_scale = 0.01
elif config.T == 1 * config.mks:
    y_scale = 7.5
    # y_scale = 10
elif config.T == 5 * config.mks:
    y_scale = 1

plt.set_yscale(y_scale)

plt.set_xaxis("states")
plt.set_yaxis("time, " + T_str_mark(config.T))
plt.set_zaxis("prob.\t\t\t\t\t\t.")

plt.iplot(
    x_csv=config.x_csv,
    y_csv=config.y_csv,
    z_csv=config.z_csv,
    filename="tc_wf"
)_____no_output_____


# Notebook from Team-Schwessinger/Pgt210Methylation
Path: multi_to_single_fast5_mapped_ids/Infected_leaves_3_multi_to_single_fast5s (only_mapped_fast5s).ipynb

We need to get from mapped bam file to single fast5s:

Input is:
* mapped bam file.
* multifast5 of fastqs mapped to the reference

Output:
* folder with single fast5s that are mapped

What we need to do:
* get ID of all mapped fastqs
* extract them in batches
* make them single fast5s_____no_output_____

import os_____no_output_____

### INPUTS_____no_output_____### *****Define directories before running on different sampes*****_____no_output_____

###define input directories here
BAMIN_DIR = '../../analyses/mapping/infected_leaves/infected_leaves_3/'
FAST5IN_DIR = '../../data/genomic_data/infected_leaves/workspace_fast5/infected_leaves_3_fast5_out/'
OUT_DIR = '../../analyses/single_fast5s/infected_leaves/infected_leaves_3_mapped_single_fast5/'
_____no_output_____

### *****Define directories before running on different sampes*****_____no_output_____

BAMIN_DIR = os.path.abspath(BAMIN_DIR)
FAST5IN_DIR = os.path.abspath(FAST5IN_DIR)
OUT_DIR = os.path.abspath(OUT_DIR)_____no_output_____##catch the bam based on naming convention
inbam_fn = [os.path.join(BAMIN_DIR, x) for x in os.listdir(BAMIN_DIR) if x.endswith('sorted.bam')][0]_____no_output_____##generated the read ID list
mappedids_fn = inbam_fn.replace('.bam', '.mappedids.txt')
#!samtools  view -F 4  {inbam_fn} | cut -f 1 | sort | uniq > {mappedids_fn}_____no_output_____!head {mappedids_fn}00002a25-e616-429e-ab4f-b9f18e222dea
0000ac65-2942-4a3f-94c0-d1d784d0de1a
0000c55c-ea37-48e2-8bb0-84472ea6446d
0000d5dd-cd6c-4941-9928-865eac457b98
00017ecf-bb06-4bbd-81ea-4826683a64f0
00018e95-35cf-4f6e-9193-07962768a6a5
0001dbb8-3b02-49d2-b88b-0df1293088d7
0001de9a-5c62-4f31-b0ba-da3c7e5d266b
0002328a-d643-4563-b624-da289675b07a
0002dcae-475d-46a6-b8be-b79010ec5f4b
### generate a temporary folder that holds the batch fast5s
TMPOUT_DIR = os.path.join(OUT_DIR, 'tmp')
if not os.path.exists(TMPOUT_DIR):
    os.makedirs(TMPOUT_DIR)_____no_output_____#read in the mapped reads ids
mapped_reads = []
with open(mappedids_fn) as fh:
    for line in fh:
        mapped_reads.append(line.rstrip())_____no_output_____mapped_reads = set(mapped_reads)_____no_output_____###go from multies fast5 of mapped reads to single fast5s
!multi_to_single_fast5 -i {FAST5IN_DIR} -s {TMPOUT_DIR} -t 10 --recursive
!touch {OUT_DIR}/single_fast5.done^C25 of 1942|                                         |  1% ETA:  1 day, 4:27:17
Traceback (most recent call last):
  File "/home/jamila/anaconda3/bin/multi_to_single_fast5", line 10, in <module>
    sys.exit(main())
  File "/home/jamila/anaconda3/lib/python3.7/site-packages/ont_fast5_api/conversion_tools/multi_to_single_fast5.py", line 104, in main
    args.recursive, follow_symlinks=not args.ignore_symlinks)
  File "/home/jamila/anaconda3/lib/python3.7/site-packages/ont_fast5_api/conversion_tools/multi_to_single_fast5.py", line 40, in batch_convert_multi_files_to_single
    pool.join()
  File "/home/jamila/anaconda3/lib/python3.7/multiprocessing/pool.py", line 556, in join
    self._worker_handler.join()
  File "/home/jamila/anaconda3/lib/python3.7/threading.py", line 1044, in join
    self._wait_for_tstate_lock()
  File "/home/jamila/anaconda3/lib/python3.7/threading.py", line 1060, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt
#move fast5s you want from tmp to out dir
count = 0
for directory in os.listdir(TMPOUT_DIR):
    directory = os.path.join(TMPOUT_DIR, directory)
    #check if path is directory
    if os.path.isdir(directory):
        print(directory)
        fast5s = [fn for fn in os.listdir(directory)]
        for fast5_file in fast5s:
            if fast5_file.rstrip('.fast5') in mapped_reads:
                count = count + 1
                #move the files by renaming absolute path
                old_fn = os.path.join(directory, fast5_file)
                new_fn = os.path.join(OUT_DIR, fast5_file)
                os.replace(old_fn, new_fn)
        #count = count + len(fast5s)
        print(count)

    #print(os.path.isfile(x))/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/20
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/6
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/36
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/61
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/38
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/90
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/10
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/37
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/79
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/83
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/78
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/45
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/80
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/1
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/62
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/67
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/54
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/27
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/43
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/72
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/58
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/84
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/13
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/7
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/21
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/82
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/15
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/39
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/65
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/34
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/47
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/5
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/30
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/76
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/25
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/24
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/17
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/85
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/63
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/29
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/51
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/31
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/68
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/60
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/70
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/16
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/28
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/86
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/33
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/0
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/52
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/14
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/40
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/23
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/9
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/55
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/87
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/75
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/57
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/49
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/89
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/44
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/19
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/64
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/94
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/59
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/53
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/71
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/50
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/92
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/11
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/35
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/4
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/48
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/93
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/77
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/12
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/73
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/18
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/66
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/22
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/46
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/91
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/41
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/42
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/88
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/8
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/69
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/3
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/32
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/2
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/26
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/81
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/56
0
/home/jamila/jamila_Storage/analyses/single_fast5s/germinated_spores/rep1/tmp/74
0
count == len(mapped_reads)_____no_output_____for x in os.listdir(OUT_DIR):
    if not x.endswith('.fast5'):
        print(x)_____no_output_____#remove tmp folder
!rm -r {TMPOUT_DIR}
_____no_output_____len(mapped_reads)_____no_output_____


# Notebook from AfroditiMariaZaki/OxCompBio
Path: tutorials/Python/09_Libraries/Libraries_solutions.ipynb

# 9. Python libraries_____no_output_____## 9.1 Using libraries_____no_output_____While creating your own functions can be incredibly useful, others may have done so already (and these might actually be faster).

Often, these will come packaged as part of a wider python library._____no_output_____A simple example of this is converting degree angles to radians.

The basic formalism for this is:

radians = degrees * pi / 180.0_____no_output_____

# Exercise 9.1.1: Write a function that converts degrees to radians and convert 25 degrees to radians
def deg2rad(angle):
    pi = 3.141593
    return angle * (pi / 180)

deg2rad(65)_____no_output_____

However, the `math` python library already offers a function to do this, allowing you to perform the same operation without having to write all the above lines._____no_output_____

import math
math.radians(65)_____no_output_____

## 9.2 Importing libraries_____no_output_____In the example above, we used the `math` library through the use of the `import` statement.

In order to use any library you first must `import` it. For example to use the `string` library we do:_____no_output_____

import string_____no_output_____

Sometimes you might want to import a library but call it something else than its standard name. For example, you might consider that the `random` library name it too long for you, and you want to instead call it as `rand`. To do this we import the library as an *alias* using the `import libraryname as newname` construct: _____no_output_____

import random as rand_____no_output_____

We can now use `rand` just like we would have used `random` if we had plainly imported the module._____no_output_____## 9.3 Using library functions and objects_____no_output_____Python libraries often contain multiple functions and objects that can be accessed and used.
To access a given function you only need to call it using the construct `libraryname.functioname()`.

For example to use the `pi` variable in math we would just do:_____no_output_____

import math
math.pi_____no_output_____

Libraries usually contain lots of modules that themselves contain several functions.
For example the commonly used `argarse` library (which is used to process arguments passed to python scripts), has an `ArgumentParser` object which is used for most argument parsing work.

We can use it in the following way:_____no_output_____

import argparse

parser = argparse.ArgumentParser_____no_output_____

However, if we wanted to solely use the ArgumentParser object we could also selectively import the object using the `from` construct:_____no_output_____

from argparse import ArgumentParser

parser = ArgumentParser_____no_output_____

The exact means by which a given library's functions/objects/modules are used will differ from library to library. Whenever using a new library, we recommend that you spend time reading the library's Application Program Interface (API) documentation.

For example, you can see the documentation for argparse here: https://docs.python.org/3/library/argparse.html_____no_output_____## 9.4 A whole world of libraries_____no_output_____One of the main advantages of python is that there are many different libraries which exist that can help you achieve any goal.

The standard python distribution offers a limited about of libraries, termed the "standard library", details about them can be found here: https://docs.python.org/3/library/

These include the above mentioned `math`, `random`, and `argparse` libraries.

Beyond the "standard library", there are also many amazing community driver packages, here is a non-exhaustive list of them:

- NumPy (http://www.numpy.org/) and SciPy (https://www.scipy.org/) are excellent libraries
  that do a lot more that simple mathematical operations (like the Euclidean distance calculator above).
  These also interface very well with plotting libraries.

- Matplotlib (https://matplotlib.org/) is the most commonly used plotting library available. It is simple,
  and can do a lot. Other plotting libraries, such as GGPlot (http://ggplot.yhathq.com/), SeaBorn   (https://seaborn.pydata.org/), and Bokeh (https://bokeh.pydata.org/en/latest/).

-  Pandas (https://pandas.pydata.org/) is a great library to manipulate data structures.
   Its ease of use makes it ideal to work with large data sets.

-  MDAnalysis (https://www.mdanalysis.org/), MDTraj (http://mdtraj.org/1.9.0/), and
   PyTraj (https://github.com/Amber-MD/pytraj), are some of the libraries used to process
   Molecular Dynamics trajcetories and other files. These interface very well with
   NumPy and Pandas and Matplotlib.

-  SciKit-Learn (http://scikit-learn.org/stable/) for machine learning in Python.

-  MPI4Py (http://mpi4py.readthedocs.io/en/stable/) allows Python scripts to be parallelised
   (run over multiple processors).

-  Cython (http://cython.org/) allows you to write parts of your code in C/C++, making it very fast.

Each library will have its means of directly installing them, however keeping track of various installs and avoiding clashes between libraries can be quite tedious. We instead recommend that a package manager like `conda` be used instead.

See here for more details about conda: https://www.anaconda.com/distribution/_____no_output_____