# Notebook from taldatech/cs236756-intro-to-ml
Path: cs236756_tutorial_06_decision_trees.ipynb

#  CS 236756 - Technion - Intro to Machine Learning
---
#### Tal Daniel

## Tutorial 06 - Decision Trees
---_____no_output_____###  Agenda
---

* [Motivation](#-Motivation)
* [How Decision Trees Work](#-Considerations-for-Building-a-Tree)
* [Decision Boundries](#-Decision-Boundries-(Also-for-Regression-Tasks) )
* [Impurity Metrics](#-Impurity-&-Impurity-Metrics)
    * [Entropy](#Entropy)
    * [Gini](#Gini-Index)
* [Relation Mutual Information (Information Gain)](#Relation-Mutual-Information-(Information-Gain) )
* [Prunning & Regularization](#-Prunning-&-Regularization-Hyperparameters)
* [Example on the Titanic Dataset](#-Example---Titanic-Dataset)
* [Random Forest](#-Random-Forest)
* [Recommended Videos](#-Recommended-Videos)
* [Credits](#-Credits)_____no_output_____

# imports for the tutorial
import numpy as np
import pandas as pd
import re  # regex
from sklearn.tree import DecisionTreeClassifier
import matplotlib.pyplot as plt
%matplotlib notebook_____no_output_____

##  Motivation
---
* Decision Trees are ML algorithms that can perform **both** classification and regression tasks, and even multioutput tasks.
* They are very powerful algorithms, capable of fitting complex datasets.
* They are a fundamental component of **Random Forests**, which are amongst the most powerful ML algorithms today.
* Decision Trees require **very little data** preparation - they do not require **feature scaling or centering**._____no_output_____###  Examples
---
**Akinator** - uses some implementation of *Decision Trees*. _____no_output_____**20 Questions** - one player is chosen to be the answerer. That person chooses a subject (object) but does not reveal this to the others. All other players are questioners. They each take turns asking a question which can be answered with a simple "Yes" or "No."
* *Features* - different properties of a character we ask about
* *Label* - the identity of the character
* *Training Set* - the characters you have encountered in your life
* *Test/Inference* - play "20 Questions"
* A good strategy:
    * Eliminate as many options as possible
    * Simple questions
    * Consider more likely characters (Leibniz isn't who most people think of, but in the Technion, this might not be the case...)
* Note: this game is a special case of classification where *each sample is its own class*_____no_output_____###  Decision Tree Visualization
---
_____no_output_____###  Considerations for Building a Tree
---
#### **Binary** or **Multi-Valued**
* Each split can be into 2 nodes or more than 2 nodes. Binary trees are easier to work with.
* Every *Polythetic* tree can be represented by an equivalent *Monothetic* tree.
* For example, the following tree can be transformed to the tree in the previous section:

* Most algorithms use *binary* trees, mainly, we will focus on the **CART** algorithm which is implemented in the Scikit-Learn library.
    * The CART algorithm produces only binary trees: nonleaf nodes **always** have 2 children (i.e. questions only have yes/no answers)._____no_output_____#### Choosing the **Property** (=Feature) to Test

* For a given point of splitting, what is the next feature the descision should build upon?
* *Simple* questions: each branch should consider **few features** (1 if it is binary)
* The first features to be chosen should be the ones that *eliminate* the most options
    * Which is to say, **increase the certainty of the decision**
    * Or, lead us to **nodes that are homogenous (pure)**_____no_output_____#### **Stopping Criteria**
* When should a node be declared a *leaf*?

#### **Pruning**
* If the tree becomes too large (and thus, overfitting to the train dataset), how, and to what length, it can be made smaller and simpler?_____no_output_____###  Decision Boundaries (Also for Regression Tasks)
---
* In regression problems, the data is continuous, and boundaries have to be set such that the input falls within a certain range.
* These decision boundaries can easily drawn on a graph as in the following example: _____no_output_____####  Decision Trees for Regression Tasks
---
* In this tutorial we deal with classification tasks, where the input can be continuous features, but what if we want to complete a regression task, that is, output a value and not a class?

* Decision Trees are also capable of performing these tasks. We will not get into it, but the idea is the same.
* Instead of predicting a class, the tree predicts a value which is determined by the mean of the values in that branch.
* The CART algorithm is almost the same except for instead of minimizing the impurity, the MSE is minimized.
* You can read more about it on Scikit-Learn's documentation, under `DecisionTreeRegressor`._____no_output_____###  Estimating Class Probabilities (Class Conditional Probabality)
---
* A Decision Tree can also estimate the probability that an instance belongs to a particular class $k$
    * First, it traverses the tree to find the leaf node for this instance
    * Then, it returns the ratio of *training* instances of class $k$ in this node
    * In Scikit-Learn (after training) - `tree_clf.predict_proba([list of samples])`
* The class-conditional probabilities $P(c|D)$ are estimated by fitting a categorical MLE: $$ p_i = P(c|D) = \frac{1}{|D|} \sum_{i \in D} \mathbb{1}(y_i =c) $$
    * $c$ - class
    * $D$ - Data
    * Reminder - **Categorical Distribution** is a generalization of Bernoulli to multiple classes_____no_output_____##  Impurity & Impurity Metrics
---
* **Simplicity** is the leading principle in the tree building process
* Therefore, we seek an attribute $A$ at each node $S$ that makes the immediate descendents as "pure" as possible.
* At each node $S$, the vector $P=(p_1, p_2, ..., p_k)$, represents the probability (ratio) of samples belonging to class $i$
    * **Impurity** of node $S$ - $\phi(P) \geq 0$
    * $\phi(P)=0$ if all samples have the same label
        * That is, a node is "pure" if all training instances it applies to belong to the same class
    * $\phi(P)$ is *maximized* for **uniform distribution**
    * $\phi(P)$ is **symmetric** w.r.t. $P$
    * $\phi(P)$ is *smooth*, that is, differentiable everywhere_____no_output_____###  Impurity Metrics
---
#### Entropy
---
* The concept of entropy originates in thermodynamics as a measure of *molecular disorder*. Entropy approaches 0 when molecules are still and well ordered.
* Later, it was introduced in the *information theory* domain, where the average information content of a message is measured (a reduction in entropy is called **information gain**). When all messages are identical, the entropy is zero.
* In Machine Learning, it is used as an impurity measure - a set's entropy is 0 when it contains instances of only one class.
* Given a system whose exact description is *unknown*, its entropy is defined as **the amount of information needed to exactly specify the state of the system**
    * If the system has $k$ states, we would need $\log k$ bits to represent them_____no_output_____* Denote $p_{i, k}$ as the ratio of class $k$ instances among the training instances in the $i^{th}$ node, the entropy is: $$ H_i = \sum_{k=1}^n p_{i,k} \log\frac{1}{p_{i,k}} = - \sum_{k=1}^n p_{i,k} \log p_{i,k} $$
* A friendly introduction to Information Theory
* _____no_output_____#### Gini Index
---
* Gini is another common measure for impurity which can be intepreted as a *variance impurity*
* Denote $p_{i, k}$ as the ratio of class $k$ instances among the training instances in the $i^{th}$ node, the GiniIndex is: $$ G_i(P) = \sum_{k=1}^n p_{i,k} (1-p_{i,k}) = 1 - \sum_{k=1}^n p_{i,k}^2 $$
    * Reminder: given $X \sim Bern(p) \rightarrow Var(X) = p \cdot (1-p)$
* This is the expected error rate at node $N$ if the category label is selected randomly from class distribution present at $N$

_____no_output_____#### Gini vs. Entropy
---
* Scikit-Learn's default is **Gini**, but by changing the call to `criterion="entropy"`, the Entropy will be used to build the tree
* Most of the time, **both** lead to similar trees.
* **Gini** is (slightly) faster to compute
* However, sometimes they are different as Gini tends to isloate the most frequent class in its own branch of the tree, while Entropy tends to produce slightly more *balanced* trees_____no_output_____##  Choosing the Features/Attributes for the Splits
---
* Choose the one that most decreases impurity (and thus, increases purity)
* Given node $S$ with probabilities $P$ and discrete attribute $A$, the **impurity drop** is defined as: $$ \Delta \phi(S,A) = \phi(S) - \sum_{v \in \textit{Values}(A)} p_v \phi(S_v) $$
    * For the **Binary** case: $$ \Delta \phi(S,A) = \phi(S) - p_L \phi(S_L) - (1 - p_L) \phi(S_R) $$
* For the *Entropy* measure, this measure is called **Information Gain (IG)** (or Mutual Information - MI): $$ IG(P) = H(P) - \sum_{v \in \textit{Values}(A)} p_aH(p_a)$$
* For *Gini*, the measure is called **Gini Gain**_____no_output_____* We wish to **maximize** the information gain on the impurity drop, which is equivalent to **minimizing** the right expression.
* Denote a single feature as $k$ and a threshold value as $t_k$ (for example, height $\leq 30 $ cm) ,**the CART algorithm cost function** (which we wish to minimize): $$ J(k, t_k) = \frac{m_{left}}{m}G_{left} + \frac{m_{right}}{m}G_{right} = p_LG_{left} + p_R G_{right} $$ where $G_{left/right}$ is the measure of impurity of the left/right subset and $m_{left/right}$ is the number of instances in the left/right subset.
* The tree building algorithm (CART) is *greedy*, it greedily searches for an optimum split at the top level, then repeats the process at each level. It does not check whether or not the split will lead to the lowest possible impurity several levels down. A greedy algorithm usually produces reasonably good solutions, but they are not guaranteed to be optimal. _____no_output_____###  Relation to Mutual Information (Information Gain)
---

* **Mutual Information (MI)** calculates the statistical dependence between two variables and is the name given to information gain when applied to variable selection.
* MI is calculated between two variables and measures the reduction in uncertainty for one variable given a known value of the other variable.
* The mutual information between two random variables X and Y can be stated formally as follows: $$ I(X;Y) = H(X) - H(X \mid Y) $$
    * $I(X ; Y)$ is the mutual information for $X$ and $Y$, $H(X)$ is the entropy for $X$ and $H(X \mid Y)$ is the conditional entropy for $X$ given $Y$.
    * The result has the units of *bits*.
* MI is always **larger than or equal to zero**, where the larger the value, the greater the relationship between the two variables. If the calculated result is zero, then the variables are **independent**._____no_output_____* Mutual Information and Information Gain are the same thing, although the context or usage of the measure often gives rise to the different names. For example:
    * Effect of Transforms to a Dataset (Decision Trees): Information Gain.
    * Dependence Between Variables (Feature Selection): Mutual Information.
* Notice the similarity in the way that the mutual information is calculated and the way that information gain is calculated, they are equivalent: $$I(X;Y) = H(X) - H(X \mid Y) $$ $$ IG(S, a) =  H(S) - H(S \mid a)$$_____no_output_____####  Exercise 1 - Decsion Tree by Hand
---
As of September 2012, 800 extrasolar planets have been identified in our galaxy. Supersecret surveying spaceships sent to all these planets have established whether they are habitable for humans or not, but sending a spaceship to each planet is expensive. In this problem, you will come up with decision trees to predict if a planet is habitable
based only on features observable using telescopes.

In the following table you are given the data from all 800 planets surveyed so far. The features observed by telescope are Size (“Big” or “Small”), and Orbit (“Near” or “Far”). Each row indicates the values of the features and habitability, and how many times that set of values was observed. So, for example, there were 20 “Big” planets “Near” their star that
were habitable._____no_output_____| Size (Big / Small) | Orbit (Near / Far) | Habitable (Yes / No) |  Count
| --- | --- | --- | --- |
| Big |  Near | Yes|  20
| Big |  Far | Yes|  170
| Small |  Near | Yes|  139
| Small |  Far | Yes|  45
| Big |  Near | No|  130
| Big |  Far | No|  30
| Small |  Near | No|  11
| Small |  Far | No|  255
| 190+, 160- / 184+, 266- |  159+, 141- / 215+, 285- |  374 / 426 |  800

**Derive and draw the decision tree on this data (use the maximum information gain (entropy) criterion for splits, don’t do any pruning).**_____no_output_____####  Solution 1
Let's first calculate the entropy of starting state: $$ H(Habitable) = -\frac{20 + 170 + 139+45}{800} \log_2 \frac{20 + 170 + 139+45}{800} -\frac{130 + 30 + 11+255}{800} \log_2 \frac{130 + 30 + 11+255}{800}$$
 $$ = -\frac{374}{800} \log_2 \frac{374}{800} -\frac{426}{800} \log_2 \frac{426}{800} = 0.4841 + 0.5183 \approx 1 $$

Let's calculate the **Information Gain** from each split:
$$ H(Habitable|Size) = \frac{190+160}{800}H(Size=Big) + \frac{184+266}{800}H(Size=Small) $$
 $$ = \frac{350}{800} \cdot (-\frac{190}{350}\log_2 (\frac{190}{350}) -\frac{160}{350}\log_2 (\frac{160}{350})) + \frac{450}{800}\cdot(-\frac{184}{450}\log_2 (\frac{184}{450}) -\frac{266}{450}\log_2 (\frac{266}{450}))$$
$$ = 0.984 \rightarrow IG(Habitable|Size) = 1 - 0.984 = 0.016 $$_____no_output_____$$ H(Habitable|Orbit) = \frac{159+141}{800}H(Orbit=Near) + \frac{215 + 285}{800}H(Orbit=Far) $$
 $$ = \frac{300}{800} \cdot (-\frac{159}{300}\log_2 (\frac{159}{300}) -\frac{141}{300}\log_2 (\frac{141}{300})) + \frac{500}{800}\cdot(-\frac{215}{500}\log_2 (\frac{215}{500}) -\frac{285}{500}\log_2 (\frac{282}{500})) $$
 $$ = 0.9901 \rightarrow IG(Habitable|Orbit) = 1 - 0.9901 = 0.009 $$_____no_output_____So the Information Gain from the size is larger, thus, the first split will be based on the Size feature.
_____no_output_____The final tree looks like this:

####  More Decision Trees Practice Excercises _____no_output_____###  Computational Complexity
---
This section's material goes beyond the scope of this course, but it is important to understand why we can't really build an **optimal** tree and what is the computational cost of training and making predicitons using the built tree.
* Denote $m$ as the number of instances (the training set size) and $n$ as the number of features.
* Finding the **optimal tree** is known to be an *NP-Complete* problem, as it requires $O(exp(m))$ time, making the problem intractable even for small training sets.
    * *P* is the set of problems that can be solved in polynomial time. *NP* is the set of problems whose solutions can be verfied in polynomial time. An *NP-Hard* problem is a problem which any *NP* problem can be reduced in polynimial time. An *NP-Complete* is both *NP* and *NP-Hard*._____no_output_____* **Making predicitons** requires traversing the Decision Tree from root to a leaf. Decision Trees are generally approximately balanced, so traversing a Decision Tree requires going through roughly $O(\log_2(m))$ nodes. Since each node only requires checking the value of one feature, the overall prediciton complexity is just $O(\log_2(m))$, independent of the number of features (so prediciton is quite **fast**, even with large training sets).
* **Training or building the tree** requires comparing all features on all samples at each node. This results in a training complexity of $O(n \times m \log (m))$. For small training sets (less than a few thousands instances), Scikit-Learn can speed up training by presorting the data (set `presort=True`), but this slows down training considerably for larger training sets._____no_output_____###  Highly-Branchig Attributes
---
* **Problem**: attributes with a large number of values
    * Extreme case: each example has its own value. E.g. example ID, Date/Day attribute in weather data
* Information gain is **biased** towards choosing attributes with a large number of values
* This may cause:
    * **Overfitting** - selection of an attribute that is non-optimal for prediction
    * **Fragmentation** - data is fragmented into (too) many small sets_____no_output_____###  Possible Solution - Normalization of the Inforamtion Gain of Features
---
* Define **Split Information** of feature $A$ as the intrinsic information of a split:
    * Entropy of distribution of instances into branches, or, how much information do we need to tell which branch an instance belongs to.
    * Formally: $$\textit{SplitInformation}(S,A) = -\sum_{v \in \textit{Values}(A)} p_vlog(p_v) $$
    * Observation: attributes with higher intrinsic information are less useful
    * Another way of defining it is $\textit{SplitInformation}(S,A) = \log n(A)$, where $n(A)$ is the number of different values feature $A$ can split over the group of samples $S$_____no_output_____* **Gain Ratio**
    * Modification of the information gain (IG) that reduces its bias towards multi-valued atributes.
    * Takes number of and size of branches into account when chossing an attribute
        * Corrects the IG by taking the intrinsic information of a split into account
    * Formally: $$ GR(S,A) =  \frac{IG(S,A)}{\textit{SplitInformation}(S,A)}$$_____no_output_____##  Pruning & Regularization Hyperparameters
---
* **Underfitting & Overfitting** - if we allow Decision Trees to grow until *absolute purity*, we will reach overfitting
    *
    * Such model is often called a *nonparametric model*, not because it does not have many parameters (it often has a lot!), but because the number of parameters is not determined prior to learning (the tree keeps on growing).
* Decision Trees make very few assumptions about the training data (unlike linear models, which assume that the data is linear)._____no_output_____* To avoid **overfitting** the training data, we need to restrict the Decision Tree's freedom during training - this is called **regularization**.
* The methods:
    * **Stop Splitting** - stop growing a tree once a stopping criteria is reached
        * May suffer from the *horizon effect* - such a method will be biased towards trees in which greatest impurity reduction is near the root
    * **Prunning** - grow a full tree, then merge nodes until a stopping criteria is reached or deleting (=prunning) unneccessary nodes if the purity improvement it provides is not *statistically signifacnt*
        * Use Hypothesis Testing - going back to Tutorial 02, standard statistical tests, such as chi-square,  $\chi^2 \textit{test}$, are used to estimate the probability (*p-value*) that the improvement is purely the result of chance (the *null hypothesis*, $H_0$). If the *p-value* is higher than a given threshold (usually 5%), then the node is considered unneccessary and its children are deleted). The prunning continues until all unnccessary nodes have been pruned._____no_output_____* **Common Stopping Criteria** (in practice):
    * *Minimum number of samples* in leaf nodes - **increasing** will regularize the model
        * In Scikit-Learn, set the input parameter `min_samples_leaf`
    * *Minimum samples split* - the minimum number of samples a node must have before it can split, **increasing** will regularize the model
        * In Scikit-Learn, set the input parameter `min_samples_split`
    * *Limit the depth of the tree* - **reducing** the max depth will regularize the model (less chance of overfitting)
       * In Scikit-Learn, set the input parameter `max_depth` (the default is `None` which means unlimited)
    * *Maximum number of leaves* - maximum number of leaf nodes, **reducing** will regularize the model
       * In Scikit-Learn, set the input parameter `max_leaf_nodes`
    * Using the validation accuracy, stop if no significant improvement
    * Also consider `min_weight_fraction_leaf` and `max_features` (more in the Scikit-Learn docs)_____no_output_____##  Example - Titanic Dataset
---
The sinking of the RMS Titanic is one of the most infamous shipwrecks in history.  On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 passengers and crew. This sensational tragedy shocked the international community and led to better safety regulations for ships.

One of the reasons that the shipwreck led to such loss of life was that there were not enough lifeboats for the passengers and crew. Although there was some element of luck involved in surviving the sinking, some groups of people were more likely to survive than others, such as women, children, and the upper-class.

We will build a decision tree to decide what sorts of people were likely to survive.

To read more about the dataset and the fields - Click Here_____no_output_____

# let's load the titanic dataset and speratre it into train and test set
dataset = pd.read_csv('./datasets/titanic_dataset.csv')
# print the number of rows in the data set
number_of_rows = len(dataset)
num_train = int(0.8 * number_of_rows)
# reminder, the data looks like this
dataset.sample(10)_____no_output_____# preprocessing
original_ds = dataset.copy() # Using 'copy()' allows to clone the dataset, creating a different object with the same values
dataset['Has_Cabin'] = dataset["Cabin"].apply(lambda x: 0 if type(x) == float else 1)
# Create new feature FamilySize as a combination of SibSp and Parch
dataset['FamilySize'] = dataset['SibSp'] + dataset['Parch'] + 1
# Create new feature IsAlone from FamilySize
dataset['IsAlone'] = 0
dataset.loc[dataset['FamilySize'] == 1, 'IsAlone'] = 1
# Remove all NULLS in the Embarked column
dataset['Embarked'] = dataset['Embarked'].fillna('S')
# Remove all NULLS in the Fare column
dataset['Fare'] = dataset['Fare'].fillna(dataset['Fare'].median())

# Remove all NULLS in the Age column
age_avg = dataset['Age'].mean()
age_std = dataset['Age'].std()
age_null_count = dataset['Age'].isnull().sum()
age_null_random_list = np.random.randint(age_avg - age_std, age_avg + age_std, size=age_null_count)
# Next line has been improved to avoid warning
dataset.loc[np.isnan(dataset['Age']), 'Age'] = age_null_random_list
dataset['Age'] = dataset['Age'].astype(int)

# Define function to extract titles from passenger names
def get_title(name):
    title_search = re.search(' ([A-Za-z]+)\.', name)
    # If the title exists, extract and return it.
    if title_search:
        return title_search.group(1)
    return ""

dataset['Title'] = dataset['Name'].apply(get_title)

# Group all non-common titles into one single grouping "Rare"
dataset['Title'] = dataset['Title'].replace(['Lady', 'Countess','Capt', 'Col','Don', 'Dr', 'Major',
                                             'Rev', 'Sir', 'Jonkheer', 'Dona'], 'Rare')
dataset['Title'] = dataset['Title'].replace('Mlle', 'Miss')
dataset['Title'] = dataset['Title'].replace('Ms', 'Miss')
dataset['Title'] = dataset['Title'].replace('Mme', 'Mrs')

# mapping
# Mapping Sex
dataset['Sex'] = dataset['Sex'].map( {'female': 0, 'male': 1} ).astype(int)
# Mapping titles
title_mapping = {"Mr": 1, "Master": 2, "Mrs": 3, "Miss": 4, "Rare": 5}
dataset['Title'] = dataset['Title'].map(title_mapping)
dataset['Title'] = dataset['Title'].fillna(0)
# Mapping Embarked
dataset['Embarked'] = dataset['Embarked'].map( {'S': 0, 'C': 1, 'Q': 2} ).astype(int)
# Mapping Fare
dataset.loc[ dataset['Fare'] <= 7.91, 'Fare'] = 0
dataset.loc[(dataset['Fare'] > 7.91) & (dataset['Fare'] <= 14.454), 'Fare'] = 1
dataset.loc[(dataset['Fare'] > 14.454) & (dataset['Fare'] <= 31), 'Fare']  = 2
dataset.loc[ dataset['Fare'] > 31, 'Fare'] = 3
dataset['Fare'] = dataset['Fare'].astype(int)

# Mapping Age
dataset.loc[ dataset['Age'] <= 16, 'Age'] = 0
dataset.loc[(dataset['Age'] > 16) & (dataset['Age'] <= 32), 'Age'] = 1
dataset.loc[(dataset['Age'] > 32) & (dataset['Age'] <= 48), 'Age'] = 2
dataset.loc[(dataset['Age'] > 48) & (dataset['Age'] <= 64), 'Age'] = 3
dataset.loc[ dataset['Age'] > 64, 'Age']

# Feature selection: remove variables no longer containing relevant information
drop_elements = ['PassengerId', 'Name', 'Ticket', 'Cabin', 'SibSp']
dataset = dataset.drop(drop_elements, axis=1)_____no_output_____# show a sample of the preprocessed dataset
dataset.sample(10)_____no_output_____# Take the relevant training data and labels
x = dataset.drop('Survived', axis=1).values
y = dataset['Survived'].values  # 1 for Survived, 0 for Dead
# shuffle
rand_gen = np.random.RandomState(0)
shuffled_indices = rand_gen.permutation(np.arange(len(x)))

x_train = x[shuffled_indices[:num_train]]
y_train = y[shuffled_indices[:num_train]]
x_test = x[shuffled_indices[num_train:]]
y_test = y[shuffled_indices[num_train:]]

print("total training samples: {}, total test samples: {}".format(num_train, number_of_rows - num_train))total training samples: 712, total test samples: 179
# let's build a tree with no limitations
tree_clf = DecisionTreeClassifier(criterion='gini')
tree_clf.fit(x_train, y_train)_____no_output_____# let's check the accuracy on the test set
y_pred = tree_clf.predict(x_test)
accuracy = np.sum(y_pred == y_test) / len(y_test)
print("accuracy: {:.3f} %".format(accuracy * 100))accuracy: 75.419 %
# let's add regularization
tree_clf = DecisionTreeClassifier(criterion='gini', max_depth=3)
tree_clf.fit(x_train, y_train)
# let's check the accuracy on the test set
y_pred = tree_clf.predict(x_test)
accuracy = np.sum(y_pred == y_test) / len(y_test)
print("accuracy: {:.3f} %".format(accuracy * 100))accuracy: 83.799 %
# predicting probabilities
# for the first passenger in the test set, the probability of survival:
prob = tree_clf.predict_proba([x_train[0]])
print("passenger 0 chances of survival: {:.3f} %".format(prob[0][0] * 100))passenger 0 chances of survival: 89.318 %
# generate an image graph
from sklearn.tree import export_graphviz
export_graphviz(tree_clf, out_file="./titanic_tree.dot", feature_names=dataset.columns.values[1:].tolist(),
               class_names=['Dead', 'Survived'], rounded=True, filled=True)
# open the file with your favourite text editor and copy-pase it into http://www.webgraphviz.com/_____no_output_____

_____no_output_____##  Random Forest
---
* Random forests (or random decision forests) are an **ensemble learning** method for classification, regression and other tasks.
* They operate by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees.
    * We will talk on ensemble learning later in the course.
* Random decision forests try tend to decision trees' habit of overfitting to their training set._____no_output_____

* Image Source_____no_output_____#### Random Forest Algorithm
---
* The training algorithm for random forests applies **bootstrap aggregating, or bagging** (tutorial 11), to tree learners.
* Given a training set $X = x_1, ..., x_n$ with targets $Y = y_1, ..., y_n$, bagging repeatedly ($B$ times) selects a **random sample with replacement** of the training set and fits trees to these samples:
    * For $b = 1,...,B$:
        * Sample, with replacement, $n$ training examples from $X, Y$; call these $X_b, Y_b$.
        * Train a classification or regression tree $f_b$ on $X_b, Y_b$.
    * After training, predictions for unseen samples $x'$ can be made by averaging the predictions from all the individual regression trees on $x'$: $$ \hat{f} = \frac{1}{B} \sum_{i=1}^B f_b(x') $$
        * Or by taking the **majority vote** in the case of *classification* trees._____no_output_____

from sklearn.ensemble import RandomForestClassifier
random_state= 42
rnd_clf = RandomForestClassifier(random_state=random_state, n_estimators=50, max_depth=3)
rnd_clf.fit(x_train, y_train)
# let's check the accuracy on the test set
y_pred = rnd_clf.predict(x_test)
accuracy = np.sum(y_pred == y_test) / len(y_test)
print("accuracy: {:.3f} %".format(accuracy * 100))accuracy: 84.916 %

* This bootstrapping procedure leads to better model performance because it decreases the *variance* of the model, without increasing the *bias*.
* This means that while the predictions of a single tree are highly sensitive to noise in its training set, the average of many trees is not, as long as the trees are not correlated.
* Simply training many trees on a single training set would give strongly correlated trees (or even the same tree many times, if the training algorithm is deterministic).
    * Bootstrap sampling is a way of de-correlating the trees by showing them different training sets._____no_output_____###  Recommended Videos
---
####  Warning!
* These videos do not replace the lectures and tutorials.
* Please use these to get a better understanding of the material, and not as an alternative to the written material.

#### Video By Subject

* Decision Trees - Machine Learning Lecture 29 "Decision Trees / Regression Trees" -Cornell CS4780
* Decision Trees Example with Gini - StatQuest: Decision Trees
* Information Gain, Mutual Information - Decision Tree 4: Information Gain
* Random Forests - Random Forest - Fun and Easy Machine Learning
    * Random Forests / Bagging - Machine Learning Lecture 31 "Random Forests / Bagging" -Cornell CS4780_____no_output_____##  Credits
---
* Based upon *Pattern Classification*, chapter 8
* Information Gain and Mutual Information for Machine Learning
* Icons from Icon8.com - https://icons8.com
* Datasets from Kaggle - https://www.kaggle.com/
* Examples and code snippets were taken from "Hands-On Machine Learning with Scikit-Learn and TensorFlow"_____no_output_____


# Notebook from ajlee21/ponyo
Path: human_tests/Human_latent_transform_simulation.ipynb

# Test simulating compendia by latent transformation_____no_output_____

%load_ext autoreload
%autoreload 2

import os
import sys
import pandas as pd
import umap
import glob
from keras.models import load_model
from sklearn.decomposition import PCA
from plotnine import (ggplot,
                      labs,
                      geom_point,
                      aes,
                      ggsave,
                      theme_bw,
                      theme,
                      facet_wrap,
                      scale_color_manual,
                      guides,
                      guide_legend,
                      element_blank,
                      element_text,
                      element_rect,
                      element_line,
                      coords)

import warnings
warnings.filterwarnings(action='ignore')

from ponyo import utils, train_vae_modules, simulate_expression_dataUsing TensorFlow backend.
# Set seeds to get reproducible VAE trained models
train_vae_modules.set_all_seeds()WARNING:tensorflow:From /home/alexandra/Documents/Repos/ponyo/ponyo/train_vae_modules.py:56: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

# Read in config variables
base_dir = os.path.abspath(os.path.join(os.getcwd(),"../"))
config_filename = os.path.abspath(os.path.join(base_dir,
                                           "human_tests",
                                           "config_test_human.tsv"))
params = utils.read_config(config_filename)_____no_output_____# Load parameters
local_dir = params["local_dir"]
dataset_name = params['dataset_name']
analysis_name = params["simulation_type"]
rpkm_data_filename = params["raw_data_filename"]
normalized_data_filename = params["normalized_data_filename"]
metadata_filename = params["metadata_filename"]
experiment_id_filename = params["experiment_ids_filename"]
NN_architecture = params['NN_architecture']
latent_dim = params['latent_dim']
num_simulated_experiments = params['num_simulated_experiments']
metadata_delimiter = params["metadata_delimiter"]
experiment_id_colname = params['metadata_experiment_colname']
sample_id_colname = params['metadata_sample_colname']

NN_dir = os.path.join(
    base_dir,
    dataset_name,
    "models",
    NN_architecture)_____no_output_____assert os.path.exists(rpkm_data_filename)_____no_output_____

## Setup directories_____no_output_____

utils.setup_dir(config_filename)_____no_output_____

## Pre-process data_____no_output_____

train_vae_modules.normalize_expression_data(base_dir,
                                            config_filename,
                                            rpkm_data_filename,
                                            normalized_data_filename)input: dataset contains 50 samples and 5000 genes
Output: normalized dataset contains 50 samples and 5000 genes
utils.create_experiment_id_file(metadata_filename,
                                normalized_data_filename,
                                experiment_id_filename,
                                config_filename)There are 3219 experiments in the compendium
There are 6 experiments with gene expression data
6 experiment ids saved to file

## Train VAE_____no_output_____

# Directory containing log information from VAE training
vae_log_dir = os.path.join(
    base_dir,
    dataset_name,
    "logs",
    NN_architecture)_____no_output_____# Train VAE
train_vae_modules.train_vae(config_filename,
                            normalized_data_filename)input dataset contains 50 samples and 5000 genes
WARNING:tensorflow:From /home/alexandra/anaconda3/envs/test_ponyo/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
tracking <tf.Variable 'Variable:0' shape=() dtype=float32> beta
WARNING:tensorflow:From /home/alexandra/anaconda3/envs/test_ponyo/lib/python3.7/site-packages/tensorflow_core/python/ops/nn_impl.py:183: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/alexandra/anaconda3/envs/test_ponyo/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

Train on 45 samples, validate on 5 samples
Epoch 1/10
45/45 [==============================] - 4s 94ms/step - loss: 2510.3052 - val_loss: 2202.2642
Epoch 2/10
45/45 [==============================] - 4s 82ms/step - loss: 1668.0350 - val_loss: 1887.4274
Epoch 3/10
45/45 [==============================] - 4s 80ms/step - loss: 1555.4765 - val_loss: 1699.6106
Epoch 4/10
45/45 [==============================] - 4s 83ms/step - loss: 1546.7784 - val_loss: 1721.3652
Epoch 5/10
45/45 [==============================] - 4s 88ms/step - loss: 1464.9972 - val_loss: 1669.0635
Epoch 6/10
45/45 [==============================] - 4s 91ms/step - loss: 1448.0171 - val_loss: 1412.8141
Epoch 7/10
45/45 [==============================] - 4s 85ms/step - loss: 1491.7861 - val_loss: 1403.7571
Epoch 8/10
45/45 [==============================] - 4s 81ms/step - loss: 1428.9717 - val_loss: 1322.2233
Epoch 9/10
45/45 [==============================] - 4s 86ms/step - loss: 1405.8313 - val_loss: 1424.0236
Epoch 10/10
45/45 [==============================] - 4s 88ms/step - loss: 1325.3066 - val_loss: 1101.1403

## Simulate data by latent transformation_____no_output_____

# Run simulation
simulated_data = simulate_expression_data.simulate_by_latent_transformation(
    num_simulated_experiments,
    normalized_data_filename,
    NN_architecture,
    latent_dim,
    dataset_name,
    analysis_name,
    metadata_filename,
    metadata_delimiter,
    experiment_id_colname,
    sample_id_colname,
    experiment_id_filename,
    local_dir,
    base_dir)Normalized gene expression data contains 50 samples and 5000 genes
Return: simulated gene expression data containing 382 samples and 5001 genes

## Visualize latent transform compendium_____no_output_____

# Load VAE models
model_encoder_filename = glob.glob(os.path.join(
        NN_dir,
        "*_encoder_model.h5"))[0]

weights_encoder_filename = glob.glob(os.path.join(
    NN_dir,
    "*_encoder_weights.h5"))[0]

model_decoder_filename = glob.glob(os.path.join(
    NN_dir,
    "*_decoder_model.h5"))[0]

weights_decoder_filename = glob.glob(os.path.join(
    NN_dir,
    "*_decoder_weights.h5"))[0]

# Load saved models
loaded_model = load_model(model_encoder_filename)
loaded_decode_model = load_model(model_decoder_filename)

loaded_model.load_weights(weights_encoder_filename)
loaded_decode_model.load_weights(weights_decoder_filename)_____no_output_____pca = PCA(n_components=2)_____no_output_____# Read data
normalized_compendium = pd.read_csv(normalized_data_filename, header=0, sep="\t", index_col=0)_____no_output_____# Encode normalized compendium into latent space
compendium_encoded = loaded_model.predict_on_batch(normalized_compendium)

compendium_encoded_df = pd.DataFrame(data=compendium_encoded,
                                     index=normalized_compendium.index)

# Get and save PCA model
model = pca.fit(compendium_encoded_df)

compendium_PCAencoded = model.transform(compendium_encoded_df)

compendium_PCAencoded_df = pd.DataFrame(data=compendium_PCAencoded,
                                         index=compendium_encoded_df.index,
                                         columns=['1','2'])

# Add label
compendium_PCAencoded_df['experiment_id'] = 'background'_____no_output_____# Embedding of real template experiment (encoded)
before_encoded_filename = os.path.join(local_dir, "simulated_before_encoded.txt")

template_encoded_df = pd.read_csv(before_encoded_filename, header=0, sep='\t', index_col=0)

template_PCAencoded = model.transform(template_encoded_df)

template_PCAencoded_df = pd.DataFrame(data=template_PCAencoded,
                                         index=template_encoded_df.index,
                                         columns=['1','2'])

# Add back label column
template_PCAencoded_df['experiment_id'] = 'before_experiment'_____no_output_____# Embedding of simulated experiment (encoded)
after_encoded_filename = os.path.join(local_dir, "simulated_after_encoded.txt")

simulated_encoded_df = pd.read_csv(after_encoded_filename,header=0, sep='\t', index_col=0)

simulated_PCAencoded = model.transform(simulated_encoded_df)

simulated_PCAencoded_df = pd.DataFrame(data=simulated_PCAencoded,
                                         index=simulated_encoded_df.index,
                                         columns=['1','2'])

# Add back label column
simulated_PCAencoded_df['experiment_id'] = 'after_experiment'_____no_output_____# Concatenate dataframes
combined_PCAencoded_df = pd.concat([compendium_PCAencoded_df,
                                    template_PCAencoded_df,
                                    simulated_PCAencoded_df])

print(combined_PCAencoded_df.shape)
combined_PCAencoded_df.head()(62, 3)
# Plot
fig = ggplot(combined_PCAencoded_df, aes(x='1', y='2'))
fig += geom_point(aes(color='experiment_id'), alpha=0.5)
fig += labs(x ='PCA 1',
            y = 'PCA 2',
            title = 'PCA original data with experiments (latent space)')
fig += theme_bw()
fig += theme(
    legend_title_align = "center",
    plot_background=element_rect(fill='white'),
    legend_key=element_rect(fill='white', colour='white'),
    legend_title=element_text(family='sans-serif', size=15),
    legend_text=element_text(family='sans-serif', size=12),
    plot_title=element_text(family='sans-serif', size=15),
    axis_text=element_text(family='sans-serif', size=12),
    axis_title=element_text(family='sans-serif', size=15)
    )
fig += guides(colour=guide_legend(override_aes={'alpha': 1}))
fig += scale_color_manual(['red', '#bdbdbd', 'blue'])
fig += geom_point(data=combined_PCAencoded_df[combined_PCAencoded_df['experiment_id'] == 'after_experiment'],
                  alpha=0.2,
                  color='red')
fig += geom_point(data=combined_PCAencoded_df[combined_PCAencoded_df['experiment_id'] == 'before_experiment'],
                  alpha=0.2,
                  color='blue')

print(fig)_____no_output_____


# Notebook from iRONStoolbox/iRONStoolbox
Path: iRONS/Notebooks/A - Knowledge transfer/.ipynb_checkpoints/3.c. Reservoir operating policy-checkpoint.ipynb

# Optimisation of the reservoir operating policy
In this Notebook we will see why and how to optimise the operating policy of a water reservoir system. An **operating policy** is a function that returns the operational decision to be made at any given time (such as the release volume for the next 24 hours) based on the conditions of the reservoir system (for instance, the reservoir storage, the demand forecast, the time of year, etc.) at that time. Differently from the release/pumping schedulings discussed in the previous Notebooks, which are optimised every time an operational decision must be made, the operating policy is optimised once and then applied forever after (or at least, until a revision of the policy is needed). In other words, the optimisation of the operating policy does not return a set of optimal operational decisions but rather a strategy for making optimal decisions ([Dobson et al, 2019](https://doi.org/10.1016/j.advwatres.2019.04.012)). It follows that, while the optimisation of release/pumping schedulings aims at maximising the benefits against the short- or mid-term forecasts of inflow and demand, the optimisation of the operating policy maximises the long-term benefits. These can be estimated by using (sufficiently long) historical time series or model projections.

<left><left>_____no_output_____Once again we consider a simple illustrative system where a reservoir is operated to supply water to a domestic consumption node, while ensuring a minimum environmental flow in the downstream river (also called “environmental compensation flow”) and maintaining the water level in the reservoir within prescribed limits. We use a mathematical model to link all the key variables that represent the reservoir dynamics (inflow, storage and outflows) and use model simulation/optimisation to determine the reservoir operating policy that optimizes the **long-term** (several years) system performance. We use the historical time series of inflows and water demand to estimate such long-term performance. The underpinning assumption here is that the system forcings observed over the past years are representative of the forcings that will drive the system in the future (if this assumption is not sensible, for instance because of ongoing changes that will likely impact the hydrological regime or demand pattern, then one may use model projections of inflow and demand in place of historical observations)
<left> <left>

We will use a simple form of operating policy where the reservoir release is only determined by the storage value, as in the Figure below. Higher storage values are associated to higher releases, which is useful for flood control purposes, whereas at low storage values less water is released to reduce the risk of future water shortages [(Loucks et al., 1981)](https://link.springer.com/book/10.1007/978-3-319-44234-1).
<left><left>_____no_output_____## Import libraries
To run this notebook we need to import some libraries: (🚨 in order to run the code like in the box below, place the mouse pointer in the cell, then click on “▶ Run” button above or press shift + enter)_____no_output_____

from bqplot import pyplot as plt
from bqplot import *
from bqplot.traits import *
import numpy as np
import ipywidgets as widgets
from IPython.display import display
from platypus import NSGAII, Problem, Real, Integer # Import the optimizer_____no_output_____

## Defining the shape of the operating policy
As anticipated, in this notebook we will use an operating policy that determines the reservoir release based on the storage value only. In particular, we will use a piece-wise linear function of the storage. The function is implemented in a series of submodules, that we can import with the following code:_____no_output_____

from Modules.Interactive_release_policy import Interactive_piecewiselin_manual, Interactive_piecewiselin_auto
from irons.Software.read_data import read_csv_data
from irons.Software.day2week2month import day2week
from irons.Software.operating_policy import op_piecewiselin_1res_____no_output_____

In our code, the operating policy uses release and rescaled storage values. Specifically, the storage is scaled by the reservoir active capacity, so that in the operating policy function it varies between 0 (dead storage) and 1 (full storage).
The piece-wise linear function representing the operating policy is delineated by 4 points (x0, x1, x2 and x3): the minimimum and maximum (storage,release) points (x0 and x3 respectively), and two inflection points (x1, x2) where the slope of the function changes. The function returns a constant release (for instance, the target demand) when the storages stay between points x1 and x2; the release is reduced if the storage is below x1, or increased if it goes above x2. Let's now attribute a (tentative) value to the coordinates of these points, so that we can visualise the operating policy. For example:_____no_output_____

# System characteristics
d       = 15 # ML/week - water demand (we assume as constant)
env_min = 4 # ML/week - environmental compensation flow
Qreg_rel_mean = d # ML/week - the (long-term) mean release = demand

# System constraints
Qreg_rel_min = env_min # ML/week - the release at minimum storage
Qreg_rel_max = 40 # ML/week - the maximum release capacity

s_min = 0 # ML - minimum storage (set to zero for now)
s_max = 150 #  ML - maximum storage (=reservoir capacity)

### Operating policy defining points ###
s_0 = s_min/s_max
s_1 = 0.2 # storage fraction at which 1st change in slope occurs
s_2 = 0.8 # storage fraction at which 2nd change in slope occurs
s_3 = s_max/s_max

Qreg_rel_ref = Qreg_rel_mean # ML/week - the target demand
u_0 = Qreg_rel_min
u_1 = Qreg_rel_ref
u_2 = Qreg_rel_ref
u_3 = Qreg_rel_max

x0 = [s_0, u_0]
x1 = [s_1, u_1]
x2 = [s_2, u_2]
x3 = [s_3, u_3]

param = [x0, x1, x2, x3]
# Release policy #
policy_rel = op_piecewiselin_1res(param)
### Storage fraction ###
s_frac = np.arange(0,1.01,0.01)_____no_output_____

We can now create and plot the operating policy with the following code:_____no_output_____

### Axis characteristics ###
x_sc_0 = LinearScale(min=0,max=1.03);y_sc_0 = LinearScale(min=0,max=u_3)
x_ax_0 = Axis(label='Storage fraction', scale=x_sc_0)
y_ax_0 = Axis(label='Release (ML/week)', scale=y_sc_0, orientation='vertical')
### Plot ###
policy_function_points = Scatter(x = [x0[0], x1[0], x2[0], x3[0]],
                                 y = [x0[1], x1[1], x2[1], x3[1]],
                                 colors=['red'],stroke = 'lightgray',
                                 scales={'x': x_sc_0, 'y': y_sc_0},
                                 names = ['x0','x1','x2','x3'])
policy_function_0 = Lines(x = s_frac, y = policy_rel,
                          colors=['blue'],stroke = 'lightgray',
                          scales={'x': x_sc_0, 'y': y_sc_0})
### Figure characteristics ###
fig_0 = plt.Figure(marks = [policy_function_0,policy_function_points],title = 'Operating policy', axes=[x_ax_0, y_ax_0],
                   layout={'width': '450px', 'height': '400px'}, legend_style = {'fill': 'white', 'opacity': 0.5},
                           fig_margin={'top':0, 'bottom':40, 'left':60, 'right':0})
widgets.VBox([fig_0])_____no_output_____

## Optimising the operating policy by trial and error (manual optimisation)
In this section we will refine the parameters of the operating policy (that is, in our example, the coordinates of points x0,x1,x2,x3) by trying to increase the system performance when simulated against the historical inflows, evaporation and water demand data.
### Loading historical inflows and evaporation data
Let's assume we want to look at 100 weeks from 2014 to 2015, and load the inflow and evaporation observations for this period from a file._____no_output_____

### Load evaporation data ###
inputs_folder_path                  = 'Inputs/'
clim_data_file                      = 'clim_data_2014_15.csv'
date_day, clim_data_day            = read_csv_data(inputs_folder_path, clim_data_file)
date, e_data_week, e_data_week_cum = day2week(date_day,clim_data_day[:,0:1],date_end=pd.Timestamp('2015-12-02 00:00:00'))
### Load inflow data ###
I_data_file                         = 'inflow_data_2014_15.csv'
date_day, I_data_day               = read_csv_data(inputs_folder_path, I_data_file)
date, I_data_week, I_data_week_cum = day2week(date_day,I_data_day[:,0:1],date_end=pd.Timestamp('2015-12-02 00:00:00'))_____no_output_____

Plot the evaporation and inflow time series:_____no_output_____

# Axis characteristics
x_sc_1 = DateScale();y_sc_1 = LinearScale(min=0,max=40)
x_ax_1 = Axis(scale=x_sc_1);y_ax_1 = Axis(label='ML/week', scale=y_sc_1, orientation='vertical')
# Bar plot
evap_plot = plt.bar(date,e_data_week[:,0],colors=['green'],stroke = 'lightgray',scales={'x': x_sc_1, 'y': y_sc_1},
                      labels = ['evaporation'], display_legend = True)
inflow_plot = plt.bar(date,I_data_week[:,0],colors=['blue'],stroke = 'lightgray',scales={'x': x_sc_1, 'y': y_sc_1},
                      labels = ['inflow'], display_legend = True)
# Figure characteristics
fig_1 = plt.Figure(marks = [inflow_plot,evap_plot],title = 'Inflow', axes=[x_ax_1, y_ax_1],
                    layout={'min_width': '900px', 'max_height': '200px'}, legend_style = {'fill': 'white', 'opacity': 0.5})
widgets.VBox([fig_1])_____no_output_____

### Definition of reservoir simulation function and initial storage
Next, we need to import the iRONs function that implements the reservoir simulation (this function iteratively applies the mass balance equation so to reconstruct the temporal evolution of the reservoir variables over the simulation period). We also need to specify the initial storage volume to start the simulation._____no_output_____

### Import the reservoir simulation function ###
from irons.Software.res_sys_sim import res_sys_sim
### Define initial storage for simulation ###
s_ini = 140 # ML - initial storage volume_____no_output_____

### Definition of the system objectives
Last, we need to define the objectives that we want to be maximised by the operating policy. As anticipated in the Introduction of this Notebook, our illustrative reservoir is operated to support domestic supply while maintaining the reservoir level above a prescribed target (this could be, for example, because the quality of the water deteriorates when levels are low, requiring more costly treatment).
We will pursue the first objective by minimising the following Total Squared Deficit (TDC) with respect to the historical water demand:

$$TSD = \sum_{t=1}^{N} [ \ max( \ 0, \ d(t)-Qreg\_rel(t) \ ) \ ]^2 $$

where N is the length of the simulation period and d(t) is the water demand for each time-interval in that period. Notice that the function $max(0,...)$ enables us to only count the difference between demand d and release u when this is positive, that is, when the release u is smaller than the demand d, and a water shortage is indeed produced. Also, the squaring is a 'mathematical trick' to make sure that larger deficit amounts are given more weight than smaller ones. This translates the fact that small deficit amounts are easier to mitigate and hence more acceptable, while larger ones can cause disproportionately severe impacts and should be avoided as much as possible.

We are also interested in minimising the chances that the reservoir level goes below a critical threshold. We measure how well this criterion is satisfied by the following Critical Storage Violation (CSV) function:

$$CSV = \sum_{t=1}^{N} [ \ max ( \ cs - s(t) , \ 0) \ ] $$

where, again, N is the length of the simulation period, s is the reservoir storage, and cs is the critical storage threshold that should preferably not be transpassed. For our case, let's set this threshold to 40 ML._____no_output_____

N = len(date)
cs = np.zeros((N+1,1))+40  # (ML)  critical storage threshold_____no_output_____

### Determining the optimal operating policy via interactive visualisation

Now use the sliders to modify the parameters of the operating policy in a way that minimises the Total Squared Deficit (TSD) and the Critical Storage Violation (CSV)._____no_output_____

u_ini = u_0, u_1, u_2, u_3
d = d + np.zeros((N,1))
fig_1a,fig_1b,fig_1c, u_ref,s_ref_1,s_ref_2 = Interactive_piecewiselin_manual(res_sys_sim, op_piecewiselin_1res,
                                                                        date,
                                                                        I_data_week, e_data_week,
                                                                        s_ini, s_min, s_max,
                                                                        u_ini, Qreg_rel_min, Qreg_rel_max,
                                                                        cs, d)

Box_layout = widgets.Layout(justify_content='center')
widgets.VBox([widgets.HBox(
    [widgets.HBox([u_ref, widgets.VBox([s_ref_1,s_ref_2])],layout=Box_layout), fig_1a],layout=Box_layout),fig_1b,fig_1c],layout=Box_layout)_____no_output_____

## From manual to automatic optimization
As we have seen, when we deal with two conflicting objectives, we cannot find a solution that optimise both simoultaneously. If we prioritize one objective, the other one is deteriorated: there is a trade-off between the two. It would then be interesting to explore this tradeoff, and find a set of operating policies that produce different optimal combinations of the two objectives. However, this is too cumbersome to do manually, so we can use a multi-objective optimisation algorithm to do that for us. The algorithm will automatically test a very large number of combinations of the policy parameters u_ref, s_ref_1, s_ref_2, until it finds a set of combinations that realise (approximately) optimal tradeoffs.

Here we use a multi-objective optimisation algorithm called NSGAII, which is implemented in the Python Platypus package. For more information about these methods and tools, see [Deb et al, 2002](https://ieeexplore.ieee.org/document/996017) and the [Platypus webpage](https://platypus.readthedocs.io). The code to run the optimisation is the following:_____no_output_____

def auto_optim(vars):

    u_ref  = vars[0]
    s_1    = vars[1]
    s_2    = vars[2]

    x0 = [s_0, u_0]
    x1 = [s_1, u_ref]
    x2 = [s_2, u_ref]
    x3 = [s_3, u_3]
    param = x0, x1, x2, x3
    # Release policy #
    policy_rel = op_piecewiselin_1res(param)

    Qreg = {'releases' : {'type'  : 'operating policy',
                          'input' : policy_rel},
            'inflows' : [],
            'rel_inf' : []}

    Qenv, Qspill, Qreg_rel, Qreg_inf, s, E = res_sys_sim(
                                               I_data_week, e_data_week,
                                               s_ini, s_min, s_max,
                                               env_min, d,
                                               Qreg)

    TSD = (np.sum((np.maximum(d-Qreg_rel,np.zeros((N,1))))**2)).astype('int')
    CSV = (np.sum((np.maximum(cs-s,np.zeros((N+1,1)))))).astype('int')

    constraints = [s_2-s_1]

    return [TSD, CSV], constraints

problem = Problem(3,2,1)
Real0   = Real(0,u_3); Real1 = Real(0, 1); Real2 = Real(0, 1)

problem.types[:]       = [Real0] + [Real1] + [Real2]
problem.constraints[:] = ">=0"
problem.function       = auto_optim

population_size = 50
algorithm       = NSGAII(problem,population_size)
algorithm.run(1000) # Number of iterations

results1_optim = np.array([algorithm.result[i].objectives[0] for i in range(population_size)])
results2_optim = np.array([algorithm.result[i].objectives[1] for i in range(population_size)])

sol_optim = [algorithm.result[i].variables for i in range(population_size)]_____no_output_____

#### Plot the optimisation results
We can visualise the tradeoffs between the two objectives in one plot, called Pareto front, which displays the combination of the two objective values in correspondence to a set of optimised solutions. Click on one point in the Pareto front to visualise the operating policy that generates that performance, and associated storage time series.  What do you think would be a balanced solution? Why most of the optimal solutions have the same water deficit every week?_____no_output_____

fig_pf, fig_2a,fig_2b,fig_2c = Interactive_piecewiselin_auto(res_sys_sim, op_piecewiselin_1res,
                                                       date,
                                                       I_data_week, e_data_week,
                                                       s_ini, s_min, s_max,
                                                       u_ini, Qreg_rel_min, Qreg_rel_max,
                                                       cs, d,
                                                       results1_optim,results2_optim,sol_optim)_____no_output_____

Plot_____no_output_____

Box_layout = widgets.Layout(justify_content='center')
widgets.VBox([widgets.HBox(
    [fig_pf, fig_2a],layout=Box_layout),fig_2b,fig_2c],layout=Box_layout)_____no_output_____

### References

Deb K. et al (2002) A fast and elitist multiobjective genetic algorithm: NSGA-II, IEEE Transactions on Evolutionary Computation, 6(2), 182-197, doi:10.1109/4235.996017.

Dobson B. et al (2019) An argument-driven classification and comparison of reservoir operation optimization methods, Advances in Water Resources, 128, 74-86.

Loucks D. P. et al (1981) Water resource systems planning and analysis, Prentice-Hall._____no_output_____


# Notebook from stuffstuffstuf1/The-Python-Workshop
Path: Chapter04/Exercise64/Exercise64.ipynb

<code>
grades = ['A', 'B', 'C', 'D', 'E', 'F']
students_count = [20, 30, 10, 5, 8, 2]_____no_output_____import matplotlib.pyplot as plt
plt.bar(grades, students_count, color=['green', 'gray', 'gray', 'gray', 'gray', 'red'])_____no_output_____plt.title('Grades Bar Plot for Biology Class')
plt.xlabel('Grade')
plt.ylabel('Num Students')
plt.bar(grades, students_count, color=['green', 'gray', 'gray', 'gray', 'gray', 'red'])
plt.show()_____no_output_____plt.barh(grades, students_count, color=['green', 'gray', 'gray', 'gray', 'gray', 'red'])_____no_output_____
</code>


# Notebook from nalomran/Bioinformatics-with-Python-Cookbook-Second-Edition
Path: Chapter04/Genepop_Format.ipynb

from collections import defaultdict

from Bio.PopGen.GenePop import read
from Bio.PopGen.GenePop.LargeFileParser import read as read_large

from genomics.popgen.plink.convert import to_genepop_____no_output_____f = open('relationships_w_pops_121708.txt')
pop_ind = defaultdict(list)
f.readline()  # header
for line in f:
    toks = line.rstrip().split('\t')
    fam_id = toks[0]
    ind_id = toks[1]
    pop = toks[-1]
    pop_ind[pop].append((fam_id, ind_id))_____no_output_____

## Check for consistency issues_____no_output_____

all_inds = []
for inds in pop_ind.values():
    all_inds.extend(inds)
for line in open('hapmap1.ped'):
    toks = line.rstrip().replace(' ', '\t').split('\t')
    fam = toks[0]
    ind = toks[1]
    if (fam, ind) not in all_inds:
        print('Problems with %s/%s' % (fam, ind))Problems with 2469/NA20281

## Convert from PLINK to Genepop_____no_output_____

to_genepop('hapmap1_auto', 'hapmap1_auto', pop_ind)
to_genepop('hapmap10', 'hapmap10', pop_ind)
to_genepop('hapmap10_auto', 'hapmap10_auto', pop_ind)
to_genepop('hapmap10_auto_noofs_ld', 'hapmap10_auto_noofs_ld', pop_ind)
to_genepop('hapmap10_auto_noofs_2', 'hapmap10_auto_noofs_2', pop_ind)
#slow
#talk about coding: ACGT - 1234
#talk about pop Pop names_____no_output_____

## Using the in-memory parser_____no_output_____Careful: this will severely increase your memory usage, consider it optional_____no_output_____

rec = read(open('hapmap1_auto.gp'))
print('Header: %s' % len(rec.comment_line))
print('Number of loci %d' % len(rec.loci_list))
print('Number of populations %d' % len(rec.pop_list))
print('Population names: %s' % ', '.join(rec.pop_list))
print('Individuals per population %s' % ', '.join([str(len(inds)) for inds in rec.populations]))
ind = rec.populations[1][0]
print('Individual %s, SNP %s, alleles: %d %d' % (ind[0], rec.loci_list[0], ind[1][0][0], ind[1][0][1]))
del rec
#talk about population namesHeader: 8
Number of loci 13892
Number of populations 11
Population names: 2436/NA19983, 1459/NA12865, NA18594/NA18594, NA18140/NA18140, NA20881/NA20881, NA19007/NA19007, NA19372/NA19372, M005/NA19652, 2581/NA21371, NA20757/NA20757, Y105/NA19099
Individuals per population 82, 165, 84, 85, 88, 86, 90, 77, 171, 88, 167
Individual 1328/NA06989, SNP 1/rs873927/1356693, alleles: 1 1

## Using the Large File Parser_____no_output_____

def count_individuals(fname):
    rec = read_large(open(fname))
    pop_sizes = []
    for line in rec.data_generator():
        if line == ():
            pop_sizes.append(0)
        else:
            pop_sizes[-1] += 1
    return pop_sizes_____no_output_____print('Individuals per population %s' % ', '.join([str(ninds) for ninds in count_individuals('hapmap1_auto.gp')]))Individuals per population 82, 165, 84, 85, 88, 86, 90, 77, 171, 88, 167
print(len(read_large(open('hapmap10.gp')).loci_list))
print(len(read_large(open('hapmap10_auto.gp')).loci_list))
print(len(read_large(open('hapmap10_auto_noofs_ld.gp')).loci_list))143993
138760
55345


# Notebook from HPI-Information-Systems/TimeEval
Path: notebooks/data-prep/GHL.ipynb

# Gasoil heating loop (GHL)

Use the `DANGER` annotation for anomalies as described in the paper:

> In the test set the additional variables labeled as ATTACK, DANGER and FAULT are introduced.
> They determine different parts of attack evolution.
> We will use the DANGER series to compare results with the fault-detection algorithm._____no_output_____

import pandas as pd
import os
from typing import List
from pathlib import Path
import matplotlib.pyplot as plt
from config import data_raw_folder, data_processed_folder
from timeeval import Datasets_____no_output_____plt.rcParams["figure.figsize"] = (20, 10)_____no_output_____dataset_collection_name = "GHL"
source_folder = Path(data_raw_folder) / "Gasoil Heating Loop" / "data"
target_folder = data_processed_folder

print(f"Looking for source datasets in {Path(source_folder).absolute()} and\nsaving processed datasets in {Path(target_folder).absolute()}")Looking for source datasets in /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data and
saving processed datasets in /home/projects/akita/data/benchmark-data/data-processed

file handling and transformations_____no_output_____

def list_regular_files(path: Path) -> List[Path]:
    return [f for f in path.iterdir() if f.is_file()]

def process(source: Path, target: Path, is_train: bool = False) -> int:
    df = pd.read_csv(source)
    # fix to constant samling frequency (floor timestamp and remove duplicates taking first value)
    df.insert(0, "timestamp", df["Time"].astype(int))
    df = df.drop(["Time"], axis=1)
    df = df.drop_duplicates(subset="timestamp", keep="first", ignore_index=True)
    # remove "distrubances" random value columns
    rand_column_names = [c for c in df.columns if "rand" in c]
    df = df.drop(rand_column_names, axis=1)
    # add anomaly label
    if is_train:
        df["is_anomaly"] = 0
    else:
        df["is_anomaly"] = (df["DANGER"] == 1).astype(int)
        df = df.drop(["DANGER", "FAULT", "ATTACK"], axis=1)
    df.to_csv(target, index=False)
    return len(df)_____no_output_____# shared by all datasets
dataset_type = "synthetic"
train_is_normal = True
train_type = "semi-supervised"
input_type = "multivariate"
datetime_index = False

# create target directory
dataset_subfolder = Path(input_type) / dataset_collection_name
target_subfolder = target_folder / dataset_subfolder
try:
    os.makedirs(target_subfolder)
    print(f"Created directories {target_subfolder}")
except FileExistsError:
    print(f"Directories {target_subfolder} already exist")
    pass

dm = Datasets(target_folder)Directories /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL already exist
files = list_regular_files(source_folder)
train_file = [f for f in files if "train" in str(f)]
if train_file:
    train_file = train_file[0]
else:
    raise ValueError("Training file not found!")
files = [f for f in files if "train" not in str(f)]

# Process training file
filename = f"{train_file.stem}.train.csv"
train_path = dataset_subfolder / filename
train_target_filepath = target_subfolder / filename

process(train_file, train_target_filepath, is_train=True)
print(f"Processed training dataset {train_file} -> {train_target_filepath}")

for f in files:
    dataset_name = f.stem
    filename = f"{dataset_name}.test.csv"
    path = dataset_subfolder / filename
    target_filepath = target_subfolder / filename

    dataset_length = process(f, target_filepath)

    # save metadata
    dm.add_dataset((dataset_collection_name, dataset_name),
        train_path = train_path,
        test_path = path,
        dataset_type = dataset_type,
        datetime_index = datetime_index,
        split_at = None,
        train_type = train_type,
        train_is_normal = train_is_normal,
        input_type = input_type,
        dataset_length = dataset_length
    )
    print(f"Processed source dataset {f} -> {target_filepath}")

# save metadata of benchmark
dm.save()Processed training dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/train_1500000_seed_11_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/train_1500000_seed_11_vars_23.train.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/45_Lev_corr_Temp_fault_seed_193_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/45_Lev_corr_Temp_fault_seed_193_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/34_Lev_corr_Temp_fault_seed_151_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/34_Lev_corr_Temp_fault_seed_151_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/23_Lev_fault_Temp_corr_seed_79_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/23_Lev_fault_Temp_corr_seed_79_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/12_Lev_fault_Temp_corr_seed_43_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/12_Lev_fault_Temp_corr_seed_43_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/28_Lev_corr_Temp_fault_seed_119_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/28_Lev_corr_Temp_fault_seed_119_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/40_Lev_corr_Temp_fault_seed_180_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/40_Lev_corr_Temp_fault_seed_180_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/17_Lev_fault_Temp_corr_seed_57_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/17_Lev_fault_Temp_corr_seed_57_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/35_Lev_corr_Temp_fault_seed_153_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/35_Lev_corr_Temp_fault_seed_153_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/46_Lev_corr_Temp_fault_seed_194_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/46_Lev_corr_Temp_fault_seed_194_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/41_Lev_corr_Temp_fault_seed_181_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/41_Lev_corr_Temp_fault_seed_181_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/29_Lev_corr_Temp_fault_seed_120_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/29_Lev_corr_Temp_fault_seed_120_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/21_Lev_fault_Temp_corr_seed_73_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/21_Lev_fault_Temp_corr_seed_73_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/04_Lev_fault_Temp_corr_seed_23_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/04_Lev_fault_Temp_corr_seed_23_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/01_Lev_fault_Temp_corr_seed_11_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/01_Lev_fault_Temp_corr_seed_11_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/15_Lev_fault_Temp_corr_seed_49_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/15_Lev_fault_Temp_corr_seed_49_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/36_Lev_corr_Temp_fault_seed_159_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/36_Lev_corr_Temp_fault_seed_159_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/20_Lev_fault_Temp_corr_seed_67_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/20_Lev_fault_Temp_corr_seed_67_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/37_Lev_corr_Temp_fault_seed_163_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/37_Lev_corr_Temp_fault_seed_163_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/02_Lev_fault_Temp_corr_seed_17_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/02_Lev_fault_Temp_corr_seed_17_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/08_Lev_fault_Temp_corr_seed_33_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/08_Lev_fault_Temp_corr_seed_33_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/39_Lev_corr_Temp_fault_seed_170_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/39_Lev_corr_Temp_fault_seed_170_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/43_Lev_corr_Temp_fault_seed_189_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/43_Lev_corr_Temp_fault_seed_189_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/14_Lev_fault_Temp_corr_seed_47_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/14_Lev_fault_Temp_corr_seed_47_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/10_Lev_fault_Temp_corr_seed_39_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/10_Lev_fault_Temp_corr_seed_39_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/30_Lev_corr_Temp_fault_seed_130_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/30_Lev_corr_Temp_fault_seed_130_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/32_Lev_corr_Temp_fault_seed_137_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/32_Lev_corr_Temp_fault_seed_137_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/42_Lev_corr_Temp_fault_seed_185_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/42_Lev_corr_Temp_fault_seed_185_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/26_Lev_corr_Temp_fault_seed_113_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/26_Lev_corr_Temp_fault_seed_113_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/05_Lev_fault_Temp_corr_seed_27_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/05_Lev_fault_Temp_corr_seed_27_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/19_Lev_fault_Temp_corr_seed_62_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/19_Lev_fault_Temp_corr_seed_62_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/06_Lev_fault_Temp_corr_seed_29_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/06_Lev_fault_Temp_corr_seed_29_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/11_Lev_fault_Temp_corr_seed_41_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/11_Lev_fault_Temp_corr_seed_41_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/33_Lev_corr_Temp_fault_seed_139_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/33_Lev_corr_Temp_fault_seed_139_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/47_Lev_corr_Temp_fault_seed_197_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/47_Lev_corr_Temp_fault_seed_197_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/09_Lev_fault_Temp_corr_seed_37_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/09_Lev_fault_Temp_corr_seed_37_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/22_Lev_fault_Temp_corr_seed_777_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/22_Lev_fault_Temp_corr_seed_777_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/24_Lev_fault_Temp_corr_seed_83_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/24_Lev_fault_Temp_corr_seed_83_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/25_Lev_corr_Temp_fault_seed_111_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/25_Lev_corr_Temp_fault_seed_111_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/38_Lev_corr_Temp_fault_seed_166_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/38_Lev_corr_Temp_fault_seed_166_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/07_Lev_fault_Temp_corr_seed_31_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/07_Lev_fault_Temp_corr_seed_31_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/13_Lev_fault_Temp_corr_seed_666_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/13_Lev_fault_Temp_corr_seed_666_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/48_Lev_corr_Temp_fault_seed_199_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/48_Lev_corr_Temp_fault_seed_199_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/27_Lev_corr_Temp_fault_seed_115_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/27_Lev_corr_Temp_fault_seed_115_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/03_Lev_fault_Temp_corr_seed_19_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/03_Lev_fault_Temp_corr_seed_19_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/16_Lev_fault_Temp_corr_seed_53_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/16_Lev_fault_Temp_corr_seed_53_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/31_Lev_corr_Temp_fault_seed_132_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/31_Lev_corr_Temp_fault_seed_132_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/44_Lev_corr_Temp_fault_seed_190_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/44_Lev_corr_Temp_fault_seed_190_vars_23.test.csv
Processed source dataset /home/projects/akita/data/benchmark-data/data-raw/Gasoil Heating Loop/data/18_Lev_fault_Temp_corr_seed_59_vars_23.csv -> /home/projects/akita/data/benchmark-data/data-processed/multivariate/GHL/18_Lev_fault_Temp_corr_seed_59_vars_23.test.csv
dm.refresh()
dm.df().loc[(slice(dataset_collection_name,dataset_collection_name), slice(None))]_____no_output_____

## Experimentation_____no_output_____

files = [f for f in list_regular_files(source_folder)]
files_____no_output_____df = pd.read_csv(files[0])
# fix to constant samling frequency (floor timestamp and remove duplicates taking first value)
df.insert(0, "timestamp", df["Time"].astype(int))
df = df.drop(["Time"], axis=1)
df = df.drop_duplicates(subset="timestamp", keep="first", ignore_index=True)
# remove "disturbances" random value columns
rand_column_names = [c for c in df.columns if "rand" in c]
df = df.drop(rand_column_names, axis=1)
# add anomaly label
df["is_anomaly"] = (df["DANGER"] == 1).astype(int)
df = df.drop(["DANGER", "FAULT", "ATTACK"], axis=1)
df_____no_output_____plt.Figure()
pd.read_csv(files[0]).iloc[135000:150000].plot(x="Time", y=["DANGER", "FAULT", "ATTACK"])#, "RT_level", "RT_temperature.T", "HT_temperature.T", "inj_valve_act", "heater_act"])
#plt.yscale("log")
plt.show()_____no_output_____train_file = [f for f in files if "train" in str(f)]
if train_file:
    train_file = train_file[0]
else:
    raise ValueError("Training file not found!")
train_file_____no_output_____df_train = pd.read_csv(train_file)
# fix to constant samling frequency (floor timestamp and remove duplicates taking first value)
df_train.insert(0, "timestamp", df_train["Time"].astype(int))
df_train = df_train.drop(["Time"], axis=1)
df_train = df_train.drop_duplicates(subset="timestamp", keep="first", ignore_index=True)
# remove "disturbances" random value columns
rand_column_names = [c for c in df_train.columns if "rand" in c]
df_train = df_train.drop(rand_column_names, axis=1)
df_train["is_anomaly"] = 0
df_train_____no_output_____


# Notebook from mas-dse-ringhilt/DSE-American-Gut-Project
Path: Notebooks/alpha_diversity/observed_otu_calc.ipynb

## author: Ryan Inghilterra_____no_output_____basic observed_otu alpha diversity calculation from cleaned biom dataset_____no_output_____

<code>
import pandas as pd
from skbio.diversity import alpha
import numpy as np_____no_output_____# gather microbiome data
biom = pd.read_pickle('../join_wflow_output_data/2.21.rar1000_clean_biom.pkl')
biom = biom.fillna(0)_____no_output_____biom.shape_____no_output_____# calculate alpha diversity for each observation

sample_ids = pd.DataFrame(biom['sample_id'])
biom = biom.drop(columns = ['sample_name', 'sample_id'])
biom = biom.values

alpha_diversities = []

for sample in range(len(biom)):
    alpha_diversities.append(alpha.observed_otus(biom[sample, :]))_____no_output_____sample_id_list = list(sample_ids.sample_id.values)

final_df = pd.DataFrame()

final_df['sample_id'] = sample_id_list

final_df['otu_count'] = alpha_diversities

print(final_df.shape)
final_df.head()_____no_output_____final_df.to_csv('2.21.rar1000.obs_otus_alpha.csv', index=False)_____no_output_____
</code>


# Notebook from ahalfpen727/-Downloads-Genomic-Analyses-w-Python
Path: jupyter-notebooks/.ipynb_checkpoints/RNA-Seq Analysis using pandas-checkpoint.ipynb

## A quick introduction to pandas using RNA-Seq Data

(*Not* to be taken as any kind of RNA-seq tutorial!)
_____no_output_____Objectives:

* Read in a csv file to pandas
* Basic concept of a dataframe
* Summary data
* Operations on columns
* Plotting columns
* Row, column slicing
* Selecting rows based on a criterion
* New columns from calculations on others
* Mapping rows to generate a new column_____no_output_____First we'll import some needed packages - matplotlib for plotting, and the pandas package for analyzing tables of data.  We often use shorthands for these..._____no_output_____

import pandas as pd
import numpy as np
import matplotlib.pylab as plt
%matplotlib inline_____no_output_____

We're going to follow [an actual RNA-Seq tutorial by Stephen Turner](http://www.gettinggeneticsdone.com/2015/12/tutorial-rna-seq-differential.html) and use data from the paper ["Differential analysis of gene regulation at transcript resolution with RNA-seq."](http://www.ncbi.nlm.nih.gov/pubmed/23222703) which has been conveniently put on figshare at https://figshare.com/articles/GSE37704_sample_data/1601975:_____no_output_____

remote_gene_count_file = "https://ndownloader.figshare.com/files/3560183"
remote_metadata_file = "https://ndownloader.figshare.com/files/3560180"_____no_output_____

Not all packages let you do this, but pandas lets you "load" files directly from a URL:_____no_output_____

gene_data = pd.read_csv(remote_gene_count_file)
metadata = pd.read_csv(remote_metadata_file)_____no_output_____

Let's take a look at the data:_____no_output_____

gene_data.head()_____no_output_____metadata_____no_output_____gene_data.describe()_____no_output_____gene_data.plot()_____no_output_____

It's actually going to be a bit of a pain having that ensemble gene name be a data column of our data,
since everything else is numeric: let's set it to be the rowname (or in pandas, the index, which is a
more general concept) rather than a data row:
_____no_output_____

gene_data = gene_data.set_index('ensgene')
gene_data.head()_____no_output_____

We can already see that a number of genes only have counts of zero across all samples, and that having the lengths in with the gene counts will make some steps more difficult; let's figure out how to:

* Get rid of rows with all counts zero
* Get rid of a column

First, let's understand how to operate with individual columns in a data frame:_____no_output_____Pandas data frames are just collections of same-length columns, each of which has a given type (float, int, etc).  Selecting a few of them is quite easy:_____no_output_____

print(gene_data.columns)
print(list(gene_data.columns))Index(['length', 'SRR493366', 'SRR493367', 'SRR493368', 'SRR493369',
       'SRR493370', 'SRR493371'],
      dtype='object')
['length', 'SRR493366', 'SRR493367', 'SRR493368', 'SRR493369', 'SRR493370', 'SRR493371']
gene_data['length'].head()_____no_output_____print(gene_data[[1]].describe())
print(gene_data['length'].describe())
print(gene_data.length.describe())
print(gene_data[gene_data.columns[1]].describe())
           SRR493366
count   19808.000000
mean      645.977585
std      4636.727936
min         0.000000
25%         0.000000
50%        68.000000
75%       455.000000
max    389976.000000
count     19808.000000
mean       4960.883128
std        3711.270948
min          33.000000
25%        2393.750000
50%        4143.000000
75%        6619.000000
max      118327.000000
Name: length, dtype: float64
count     19808.000000
mean       4960.883128
std        3711.270948
min          33.000000
25%        2393.750000
50%        4143.000000
75%        6619.000000
max      118327.000000
Name: length, dtype: float64
count     19808.000000
mean        645.977585
std        4636.727936
min           0.000000
25%           0.000000
50%          68.000000
75%         455.000000
max      389976.000000
Name: SRR493366, dtype: float64

When would the above ways of selecting a column be more or less convenient?_____no_output_____

gene_data['length'].max()_____no_output_____gene_data[['length','SRR493366']].head()_____no_output_____

Let's create a new data frame, `gene_count_data` that omits the length column. _____no_output______Hands on.  There's a few ways to do this - the manual way...______no_output_____

gene_count_data = gene_data[['SRR493366', 'SRR493367', 'SRR493368', 'SRR493369', 'SRR493370', 'SRR493371']]
gene_count_data.head()_____no_output_____gene_count_data = gene_data[[1,2,3,4,5,6]]
gene_count_data.head()_____no_output_____names = list(gene_data.columns)
gene_count_data = gene_data[names[1:]]
gene_count_data.head()_____no_output_____gene_count_data = gene_data[gene_data.columns[1:]]
gene_count_data.head()_____no_output_____

Now that we've got the counts, we can look at plots a little more sensibly:_____no_output_____

gene_count_data.plot()_____no_output_____gene_count_data.hist()_____no_output_____gene_count_data.hist(log=True, bins=50)_____no_output_____gene_count_data['SRR493366'].hist(log=True, bins=50)_____no_output_____gene_count_data['SRR493366'].hist(log=True, bins=50, range=[0,50000])_____no_output_____

Ok, now we want to skip rows that have all-zero (or maybe all-below-some-threshold) number of counts.  Let's play around with data frame rows.

When you index a data frame with a list or with a name, it selects columns; if you just use numbers, it selects rows:_____no_output_____

gene_count_data[3:7]_____no_output_____gene_count_data[-5:]_____no_output_____gene_count_data['SRR493366'][-5:]_____no_output_____

You can select rows not just by numbers, but by whether a condition is true or false_____no_output_____

awesome_gene_count=124
awesome_rows = (gene_count_data.SRR493366 == awesome_gene_count)
print(awesome_rows[:10])ensgene
ENSG00000186092    False
ENSG00000279928    False
ENSG00000279457    False
ENSG00000278566    False
ENSG00000273547    False
ENSG00000187634     True
ENSG00000188976    False
ENSG00000187961    False
ENSG00000187583    False
ENSG00000187642    False
Name: SRR493366, dtype: bool
gene_count_data[awesome_rows]_____no_output_____

So this gives us a way to select our rows... we just need a way to figure out if all of the relevant columns pass some criterion.  This would work:_____no_output_____

genes_with_counts = gene_count_data[(gene_count_data["SRR493366"] > 0) & (gene_count_data["SRR493367"] > 0) &
                                    (gene_count_data["SRR493368"] > 0) & (gene_count_data["SRR493369"] > 0) &
                                    (gene_count_data["SRR493370"] > 0) & (gene_count_data["SRR493371"] > 0)]
genes_with_counts.head()_____no_output_____

But it's sort of too terrible to contemplate (what if we had 30 samples?)_____no_output_____

(gene_count_data > 0).head()_____no_output_____(gene_count_data > 0).all()_____no_output_____(gene_count_data > 0).all(axis=1)_____no_output_____gene_count_data[(gene_count_data > 0).all(axis=1)]

_____no_output_____genes_have_counts_data = gene_count_data[(gene_count_data > 0).all(axis=1)]
genes_have_counts_data.head()_____no_output_____genes_have_counts_data.describe()_____no_output_____genes_have_counts_data.hist(log=True, bins=50)_____no_output_____

_Hands on - create a function which takes such a raw gene dataframe, applies a count threshold, and gets rid of the lengths______no_output_____

def gene_counts_above_threshold(dataframe, threshold=0):
    """
    Returns a data frame from a given dataframe without any column named 'length', and
    with rows such that every column is above threshold.

    dataframe: an input pandas dataframe
    threshold: an threshold that all int64 columns must _exceed_ for the row to be kept
    returns a pandas dataframe with the column named 'length' stripped,
             and with rows such that all int64 columns are strictly above threshold
    """
    no_length_column = dataframe[dataframe.columns.difference(["length"])]

    above_threshold = no_length_column[(no_length_column > threshold).all(axis=1)]
    return above_threshold_____no_output_____help(gene_counts_above_threshold)Help on function gene_counts_above_threshold in module __main__:

gene_counts_above_threshold(dataframe, threshold=0)
    Returns a data frame from a given dataframe without any column named 'length', and
    with rows such that every column is above threshold.

    dataframe: an input pandas dataframe
    threshold: an threshold that all int64 columns must _exceed_ for the row to be kept
    returns a pandas dataframe with the column named 'length' stripped,
             and with rows such that all int64 columns are strictly above threshold

genes_have_counts_data = gene_counts_above_threshold(gene_data, 5)_____no_output_____genes_have_counts_data.head()_____no_output_____genes_have_counts_data.hist(log=True, bins=50)_____no_output_____

Ok, so now we have the rows and columns we want, and we're ready to start doing some very crude analyses.
However, we need to scale the data across samples; we might just have more data overall from one sample
so that the counts would seem elevated compared to another.  A simple but crude way to do that scaling is
to scale by some average number of counts:_____no_output_____

genes_have_counts_data.mean()_____no_output_____scaled_data = genes_have_counts_data / genes_have_counts_data.mean()_____no_output_____scaled_data.head()_____no_output_____scaled_data.describe()_____no_output_____

Scaling by mean is actually a pretty bad idea, because a few highly-expressed genes can badly skew the mean.
A more robust method is to scale by the median - do that now_____no_output_____

scaled_data = genes_have_counts_data / genes_have_counts_data.median()
print(scaled_data.head())
print(scaled_data.describe())                 SRR493366  SRR493367  SRR493368  SRR493369  SRR493370  \
ensgene
ENSG00000279457   0.068862   0.073684   0.060860   0.075718   0.065116
ENSG00000187634   0.371257   0.323684   0.430220   0.540470   0.493023
ENSG00000188976   4.901198   4.818421   5.001049   3.201044   3.083721
ENSG00000187961   0.359281   0.402632   0.377754   0.616188   0.593023
ENSG00000187583   0.071856   0.126316   0.136411   0.114883   0.111628

                 SRR493371
ensgene
ENSG00000279457   0.091816
ENSG00000187634   0.514970
ENSG00000188976   3.001996
ENSG00000187961   0.712575
ENSG00000187583   0.127745
          SRR493366     SRR493367     SRR493368     SRR493369     SRR493370  \
count  11970.000000  11970.000000  11970.000000  11970.000000  11970.000000
mean       3.198304      3.197885      3.226698      3.298093      3.284102
std       17.745064     17.011737     17.510860     18.267737     18.489697
min        0.017964      0.015789      0.012592      0.015666      0.013953
25%        0.341317      0.336842      0.339979      0.321149      0.323256
50%        1.000000      1.000000      1.000000      1.000000      1.000000
75%        2.484281      2.518421      2.507870      2.655352      2.650581
max     1167.592814   1119.742105   1137.850997   1074.261097   1107.951163

          SRR493371
count  11970.000000
mean       3.266213
std       17.362200
min        0.011976
25%        0.321357
50%        1.000000
75%        2.658683
max     1011.423154

Let's do a little sanity check, and use a PCA to see if the samples we expect to be most similar in gene counts
in fact are - we expect the control samples (first three) to cluster together, and the treatment samples (last three)
to cluster together_____no_output_____

from sklearn.decomposition import PCA
from sklearn.preprocessing import scale

pca = PCA(n_components=2).fit_transform(scale(scaled_data).T)
print(pca)
plt.scatter(pca[:,0], pca[:,1])
plt.xlabel("1st PCA")
plt.ylabel("2nd PCA")[[-20.21252387   1.72118593]
 [-21.36436765  -1.61000477]
 [-21.12586582  -0.7095232 ]
 [ 20.76081705   0.50899454]
 [ 17.32916166   3.75675258]
 [ 24.61277862  -3.66740508]]

So this seems promising.  Now let's generate an MA plot - mean expression of the gene across all samples,
_vs_ the change in expression, all log2 scaled_____no_output_____

np.log2(scaled_data.mean(axis=1))[:5]_____no_output_____A_untreated = np.log2(scaled_data[scaled_data.columns[0:3]].mean(axis=1))
A_treated = np.log2(scaled_data[scaled_data.columns[3:7]].mean(axis=1))
A = 1./2. *(A_untreated + A_treated)
M = A_treated - A_untreated

print(A_untreated[:5])
print(A_treated[:5])
print(M[:5])ensgene
ENSG00000279457   -3.882522
ENSG00000187634   -1.414830
ENSG00000188976    2.294809
ENSG00000187961   -1.396350
ENSG00000187583   -3.164525
dtype: float64
ensgene
ENSG00000279457   -3.688725
ENSG00000187634   -0.954125
ENSG00000188976    1.630213
ENSG00000187961   -0.642515
ENSG00000187583   -3.082103
dtype: float64
ensgene
ENSG00000279457    0.193796
ENSG00000187634    0.460704
ENSG00000188976   -0.664596
ENSG00000187961    0.753835
ENSG00000187583    0.082422
dtype: float64
scaled_data['M'] = M
scaled_data['A'] = A_____no_output_____plt.scatter(scaled_data.A, scaled_data.M)
plt.xlabel('A')
plt.ylabel('M')_____no_output_____

Now we're going to make a volcano plot: p-value of change in expression vs change in expression

We'll calculate the p-values _very_ crudely by doing a 2-sided independent t-test of the scaled counts in the
control and treatment samples.   This will require looking at the data a little differently - we're going to
be processing subsets of columns for each row._____no_output_____

from scipy.stats import ttest_ind_____no_output_____onerow = scaled_data[0:1]
onerow.values[0]_____no_output_____onerow.values[0][0:3]_____no_output_____ttest_ind(onerow.values[0][0:3], onerow.values[0][3:6])_____no_output_____ttest_ind(onerow.values[0][0:3], onerow.values[0][3:7]).pvalue_____no_output_____def gene_pvalue(dataframe_row):
    control = dataframe_row[0:3]
    treatment = dataframe_row[3:7]
    return ttest_ind(control, treatment).pvalue_____no_output_____gene_pvalue(onerow.values[0])_____no_output_____foo = scaled_data.apply(gene_pvalue, axis=1)_____no_output_____

Hands on: make a "p_value" column in the scaled_data dataframe, and plot M along x axis and -log10 p_value along y axis_____no_output_____

scaled_data['p_value'] = scaled_data.apply(gene_pvalue, axis=1)
plt.scatter(scaled_data['M'], -np.log10(scaled_data['p_value']))
plt.xlabel('Log2 change in expression')
plt.ylabel('-Log10 p_value')_____no_output_____sorted_data = scaled_data.sort_values('p_value')_____no_output_____sorted_data.head()_____no_output_____

Now you could start applying the appropriate multiple-hypothesis testing corrections to see if any of these changes were significant.  A Bonferroni correction would look like_____no_output_____

sorted_data['bonferroni'] = sorted_data['p_value'] * sorted_data.shape[0]_____no_output_____sorted_data.head()_____no_output_____

## Advanced - simple aggregation _____no_output_____Let's say there was information about more than just individual genes: there was also families of genes we
were interested in:_____no_output_____

genes_with_families = genes_have_counts_data

def phony_family_from_name(gene_name):
    return int(gene_name[-1])

families = list(map(phony_family_from_name, genes_have_counts_data.index))
genes_with_families['family'] = families_____no_output_____genes_with_families.head()_____no_output_____

We can do simple things like find out how many are in each family:_____no_output_____

genes_with_families['family'].value_counts()_____no_output_____

Now we can start aggregating data by family:_____no_output_____

family_data = genes_with_families.groupby('family').aggregate('mean')_____no_output_____family_data.head()_____no_output_____family_data.plot(kind='bar')_____no_output_____