# Notebook from ne1s0n/bioinformateachers
Path: python_exercises/logistic_regression.ipynb

# Assignment

Do [logistic regression](https://en.wikipedia.org/wiki/Logistic_regression) on the famous [Iris dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set) and plot the decision boundary. Then exted to multilass using [softmax](https://en.wikipedia.org/wiki/Softmax_function)._____no_output_____# Setup
The topics of this exercise are very famous and very well known, so a lot of educational material is available online. E.g., for logistic regression:

* [Logistic regression @towardsdatascience](https://towardsdatascience.com/understanding-logistic-regression-9b02c2aec102)
* [Logistic regression @Biochemia Medica](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3936971/)
* [Logistic regression @datacamp](https://www.datacamp.com/community/tutorials/understanding-logistic-regression-python)

Let's just frame the terms of the first part of the problem:
* we are talking about **binary** classification: all samples belong to two classes (represented by values 0 and 1). We want to build a statistical machine able to predict, for a new sample, the class
* for each sample we have a set numbers, called **features**. The easiest-to-understand case is when these features are actual phyisical measurements, but any set of data will do
* in **standard regression** we weight each feature with a number (positive or negative, big or small). We then multiply each feature times its weight and add everything...
* ...but we cannot do that now, since the resulting number could be very big (positive big or negative big). But our desired result is a class! In other words our machine needs to produce either a zero or a one. Or maybe **all the numbers between zero and one**, so that we have an idea about how certain our prediction is
* in fact, a 0.99 will show more confidence than a 0.8, which will be more confident than a 0.65, even if all three outputs at the end can be considered as "class 1" (since those numbers are closer to 1 than to 0)
* to do so we feed what is produced by a regular regression into a **sigmoid function**, which looks like this:

This function takes in input any number from -infinity to +infinity and returns a value between zero and one. This will be our prediction. Let's start building the solution._____no_output_____## Setting the random seed

First of all, we setup the random seed to ensure reproducibility of results. Since tensorflow uses an internal random generator we need to fix both the general seed (via numpy `seed()`) and tensorflow seed (via `set_seet()`)_____no_output_____

#general random seed
from numpy.random import seed
seed(777)

#tensorflow-specific seed
import tensorflow
tensorflow.random.set_seed(777)_____no_output_____

# The Iris dataset

The dataset we are going to use is very famous. It was published by Robert Fisher in 1936 together with the paper [The use of multiple measurements in taxonomic problems](https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1469-1809.1936.tb02137.x). Data are public and nowadays this dataset is shipped with many statistical software and packages. We are going to use the copy coming with [sci-kit learn](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html#sklearn.datasets.load_iris)._____no_output_____First of all, let's verify the sci-kit version available:_____no_output_____

import sklearn.datasets
print (sklearn.__version__)0.22.2.post1

If we had version 0.23+ we could simply do:

> `iris = sklearn.datasets.load_iris(return_X_y = False, as_frame = True)`

But the above instruction would generate an error with the current installation. The main difference is in the format used to store data: numpy array (version 0.22 and below) or pandas dataframe/series (version 0.23 and above). Since we want to use the pandas data structure the returned data require a little conversion:_____no_output_____

import pandas as pd

iris = sklearn.datasets.load_iris()
iris.data = pd.DataFrame(iris.data, columns=iris.feature_names) #converting numpy array -> pandas DataFrame
iris.target = pd.Series(iris.target) #converting numpy array -> pandas Series_____no_output_____

The variable `iris` is now a [bunch object](https://scikit-learn.org/stable/modules/generated/sklearn.utils.Bunch.html#sklearn.utils.Bunch) and contains all the required data._____no_output_____

#uncomment the following instruction to print a detailed description
#of the Iris dataset, here omitted for compactness
#print(iris.DESCR)_____no_output_____

The dataset describes 150 flower samples, belonging to three species of Iris (Iris setosa, Iris virginica and Iris versicolor). Four features were measured from each sample: the length and the width of the sepals and of the petals, in centimeters. Features are found in the attribute `.data` of the returned bunch object:_____no_output_____

print('Shape of the feature table: ' + str(iris.data.shape))Shape of the feature table: (150, 4)

We can take a look at the actual numbers:_____no_output_____

print(iris.data)     sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)
0                  5.1               3.5                1.4               0.2
1                  4.9               3.0                1.4               0.2
2                  4.7               3.2                1.3               0.2
3                  4.6               3.1                1.5               0.2
4                  5.0               3.6                1.4               0.2
..                 ...               ...                ...               ...
145                6.7               3.0                5.2               2.3
146                6.3               2.5                5.0               1.9
147                6.5               3.0                5.2               2.0
148                6.2               3.4                5.4               2.3
149                5.9               3.0                5.1               1.8

[150 rows x 4 columns]

As said above, each of the 150 lines represents a different flower, each belonging to one of the three Iris species. The species will be our target, the class that we are trying to predict. Let's take a look into the dataset:_____no_output_____

print('Shape of the target variable: ' + str(iris.target.shape))
print('Names for each class: ' + str(iris.target_names))

#using Counter object to print a tally of the classes
from collections import Counter
print('Numerosity for each class: ' + str(Counter(iris.target)))Shape of the target variable: (150,)
Names for each class: ['setosa' 'versicolor' 'virginica']
Numerosity for each class: Counter({0: 50, 1: 50, 2: 50})

Classes are represented via a numeric index: 0 for *setosa*, 1 for *versicolor*, 2 for *virginica*. The samples are presented in order, with the first 50 samples being *setosa*, then 50 *versicolor* and the last 50 being *virginica*.

Always when working with a new datasets it is importat to plot the data if possible. We are unfortunately talking about a 5-dimensional dataset (the four features + the target class) which is not easily representable. One solution in these cases is to slice a subset of the whole dataset.

In the following code we'll plot two features at a time, plus the class._____no_output_____

#change these two values to plot different features, remembering the numbering:
# 0 : sepal length (cm)
# 1 : sepal width (cm)
# 2 : petal length (cm)
# 3 : petal width (cm)
feature_x = 0
feature_y = 1

#old reliable pyplot!
import matplotlib.pyplot as plt

#starting a new plot
fig, ax = plt.subplots()

#adding data in three bunches of 50, once per class
ax.scatter(x=iris.data.iloc[0:50,feature_x],    y=iris.data.iloc[0:50,feature_y],    c='red',   label=iris.target_names[0])
ax.scatter(x=iris.data.iloc[50:100,feature_x],  y=iris.data.iloc[50:100,feature_y],  c='green', label=iris.target_names[1])
ax.scatter(x=iris.data.iloc[100:150,feature_x], y=iris.data.iloc[100:150,feature_y], c='blue',  label=iris.target_names[2])

#the axis names are taken from feature names
ax.set_xlabel(iris.feature_names[feature_x])
ax.set_ylabel(iris.feature_names[feature_y])

#adding the legend and printing the plot
ax.legend()
plt.show()_____no_output_____

The plot shows clearly that setosa is quite separate from the other two classes. Even chosing other features for the plot the general result is similar.

To be totally frank, this dataset is quite simple. In fact even if it's not possible to easily plot everything, using the four features most classifier can reach very close to 100% accuracy when trying to separate Setosa from the other species.

To make things a little more interesting we decide to renounce to half of our features, using only the first two columns. Moreover, we join together Setosa and Versicolor. In other words, we want a classifier able to discriminate virginica (which becomes the new class "1") from the other irises (which all together become the new class "0"):_____no_output_____

#simplifly the problem: less classes, less features
features = iris.data.iloc[:, 0:2]
target = iris.target

#updating class labels. To makes things difficult we put together old classes 0 and 1
#in a new class (non virginica) and keep old class 2 (virginica) as new class 1.
#For an easier problems put together versicolor and virginica and keep setosa by itself
target[0:100] = 0
target[100:150] = 1_____no_output_____

Let's take a look at the new dataset:_____no_output_____

#starting a new plot
fig, ax = plt.subplots()

#adding data in two bunches
ax.scatter(x=features.iloc[0:100,0],   y=features.iloc[0:100,1],   c='red',  label='Not virginica')
ax.scatter(x=features.iloc[100:150,0], y=features.iloc[100:150,1], c='blue', label='virginica')

#the axis names are taken from feature names
ax.set_xlabel(iris.feature_names[feature_x])
ax.set_ylabel(iris.feature_names[feature_y])

#adding the legend and printing the plot
ax.legend()
plt.show()_____no_output_____

Things are getting interesting! This is now a difficult problem and there is no clear cut solution. Let's proceed._____no_output_____# Training and validation sets

Each time there is some kind of "learning" involved we need to split our data. A subset will be used for training, and a subset will be used for validation. (there may be room for another subset, the "test set", but we are not talking about it now).

In our current dataset the samples are sorted by class: the first 100 are from "Not virginica" class, and the remaining 50 are from virginica. We want to keep this 2:1 proportion (roughly) the same in both train and validation set._____no_output_____

#selecting the first 100 samples for training would be a bad choice...
print(Counter(iris.target[0:100]))Counter({0: 100})

To do so we are going to use what is called a [stratified approach](https://machinelearningmastery.com/cross-validation-for-imbalanced-classification/) using a [StratifiedShuffleSplit](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html) object from Sci-kit learn:_____no_output_____

#we want to have the same proportion of classes in both train and validation sets
from sklearn.model_selection import StratifiedShuffleSplit

#building a StratifiedShuffleSplit object (sss among friends) with 20% data
#assigned to validation set (here called "test")
sss = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=0)

#the .split() method returns (an iterable over) two lists which can be
#used to index the samples that go into train and validation sets
for train_index, val_index in sss.split(features, target):
    features_train = features.iloc[train_index, :]
    features_val   = features.iloc[val_index, :]
    target_train   = target[train_index]
    target_val     = target[val_index]

#let's print some shapes to get an idea of the resulting data structure
print(features_train.shape)
print(features_val.shape)
print(target_train.shape)
print(target_val.shape)

print(type(features_train))
print(type(target_train))(120, 2)
(30, 2)
(120,)
(30,)
<class 'pandas.core.frame.DataFrame'>
<class 'pandas.core.series.Series'>

It appears that we are using 20% of our data (30 out of 150) for the validation set and the other 80% for the train set._____no_output_____# Logistic regression using Keras

While most statistical packets implement some form of logistic regression in this exercise we are interested in using Keras, which is a library aimed to (Deep) Neural Networks.
Actually logistic regression plays an important role in neural networks and it's typically used in the last (or second to last) layer of a classifier. For more details on how to use keras a good starting point is the [documentation on training and evaluation](https://www.tensorflow.org/guide/keras/train_and_evaluate).

Our neural network will be very easy and very minimal, and will be comprised of only one node (neuron) implementing both regression and sigmoid function.

We are now ready to build the model! _____no_output_____

#we are building a "sequential" model, meaning that the data will
#flow like INPUT -> ELABORATION -> OUTPUT. In particular, we will
#not have any loops, i.e. our output will never be recycled as
#input for the first layer
from keras.models import Sequential

#a "dense" layer is a layer were all the data coming in are connected
#to all nodes. In our case there is only one node in the layer, and
#it receives all the features
from keras.layers import Dense

# 2-class logistic regression in Keras
model = Sequential()
model.add(Dense(1, activation='sigmoid', input_dim=features_train.shape[1]))

#the model is declared, but we still need to compile it to actually
#build all the data structures
model.compile(optimizer='rmsprop', loss='binary_crossentropy')_____no_output_____

Let's take a look inside the model:_____no_output_____

print(model.summary())Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense (Dense)                (None, 1)                 3
=================================================================
Total params: 3
Trainable params: 3
Non-trainable params: 0
_________________________________________________________________
None

Keras informs us that there are three trainable parameters (W1, W2, B), and a single node. The output is a single number. Excellent._____no_output_____We have now prepared everything we need and are ready to train the model on our data. It's an iterative process that cycles many times through what are called `epochs`. We'll start with ten:_____no_output_____

history = model.fit(features_train, target_train, epochs=10, validation_data=(features_val, target_val))Epoch 1/10
4/4 [==============================] - 0s 36ms/step - loss: 0.6551 - val_loss: 0.6609
Epoch 2/10
4/4 [==============================] - 0s 5ms/step - loss: 0.6397 - val_loss: 0.6504
Epoch 3/10
4/4 [==============================] - 0s 6ms/step - loss: 0.6295 - val_loss: 0.6419
Epoch 4/10
4/4 [==============================] - 0s 5ms/step - loss: 0.6210 - val_loss: 0.6340
Epoch 5/10
4/4 [==============================] - 0s 6ms/step - loss: 0.6131 - val_loss: 0.6272
Epoch 6/10
4/4 [==============================] - 0s 6ms/step - loss: 0.6065 - val_loss: 0.6214
Epoch 7/10
4/4 [==============================] - 0s 7ms/step - loss: 0.6006 - val_loss: 0.6158
Epoch 8/10
4/4 [==============================] - 0s 7ms/step - loss: 0.5947 - val_loss: 0.6103
Epoch 9/10
4/4 [==============================] - 0s 7ms/step - loss: 0.5889 - val_loss: 0.6047
Epoch 10/10
4/4 [==============================] - 0s 6ms/step - loss: 0.5836 - val_loss: 0.6003

We asked for ten epochs and the network did just that. At each iteration the network is trying really hard to minimize a [value called "loss"](https://keras.io/api/losses/). The specifics are defined by our choice of loss function (we selected `binary_crossentropy`). The basic idea is that the smaller the loss the better the fit.

Note that the network minimizes the loss on the training set and does not use the validation set during the learning process. It can however measure the loss on the validation set to give us an idea on how well it can generalize on new data.

It's handy at this point to define a function that takes in the `history` object returned by `.fit()` and plots it:_____no_output_____

#function to take a look at losses evolution
def plot_loss_history(h, title):
    plt.plot(h.history['loss'], label = "Train loss")
    plt.plot(h.history['val_loss'], label = "Validation loss")
    plt.xlabel('Epochs')
    plt.title(title)
    plt.legend()
    plt.show()_____no_output_____plot_loss_history(history, 'Logistic (10 epochs)')_____no_output_____

The good news is that loss just goes down, both in train and validation set. We can keep training - without recompiling, we just add new epochs to our network._____no_output_____

#putting verbose to 0 to avoid filling the screen
history2 = model.fit(features_train, target_train, epochs=490,
                     validation_data=(features_val, target_val), verbose=0)_____no_output_____

Let's see if we improved:_____no_output_____

#putting together the whole history
history.history['loss'] += history2.history['loss']
history.history['val_loss'] += history2.history['val_loss']

#and plotting again
plot_loss_history(history, 'Logistic (500 epochs)')_____no_output_____

This is very informative: losses keep shrinking, meaning that the network keeps improving. However after a first phase of steep improvement the gain for each epoch slows down considerably.

Moreover, we now see a clear difference between train and validation set. This means that, while the network keeps improving, its performances on new data are expected to be worse than those on the training data.

Now, we could ask: what happens if we keep training for a long time? We have prepared the code for 10000 epochs, but it takes a long time to run, and it's faster if we simply show the saved results (but you can try to run it putting the following flag to `True`):_____no_output_____

do_10000_epochs = False_____no_output_____#what happens if we keep going for a (very) long time?
if (do_10000_epochs):
    #train for 10000 epochs, just to show how the model evolves
    history3 = model.fit(features_train, target_train, epochs=9500,
                         validation_data=(features_val, target_val), verbose=0)

    #putting together the whole history
    history.history['loss'] += history3.history['loss']
    history.history['val_loss'] += history3.history['val_loss']

    #and plotting again
    plot_loss_history(history, 'Logistic (10000 epochs)')_____no_output_____

Our pre-recorded results look like this:

![regression_10000_epochs](https://raw.githubusercontent.com/ne1s0n/coding_excercises/master/data/regression_loss10000.png)_____no_output_____It appears that there is constant, slow improvement on training set. Improvement on validation set is slower, and if we had the patience to go for a veeeery long time the orange curve would become completely flat. _____no_output_____# Decision Boundary

The assignment asks us to plot the [decision boundary](https://en.wikipedia.org/wiki/Decision_boundary), i.e. a representation in the feature space of the criterions the model is using to classify your data. For this task we'll use the [mlxtend module](http://rasbt.github.io/mlxtend/), which unfortunately does not come with the standard installation. Let's add it!_____no_output_____

!pip install mlxtendRequirement already satisfied: mlxtend in /usr/local/lib/python3.6/dist-packages (0.14.0)
Requirement already satisfied: scikit-learn>=0.18 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (0.22.2.post1)
Requirement already satisfied: numpy>=1.10.4 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (1.18.5)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from mlxtend) (49.6.0)
Requirement already satisfied: matplotlib>=1.5.1 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (3.2.2)
Requirement already satisfied: scipy>=0.17 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (1.4.1)
Requirement already satisfied: pandas>=0.17.1 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (1.0.5)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.18->mlxtend) (0.16.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.5.1->mlxtend) (1.2.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.5.1->mlxtend) (2.4.7)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.5.1->mlxtend) (0.10.0)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.5.1->mlxtend) (2.8.1)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.17.1->mlxtend) (2018.9)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from cycler>=0.10->matplotlib>=1.5.1->mlxtend) (1.15.0)

We can now import `mlxtend` package safely._____no_output_____

from mlxtend.plotting import plot_decision_regions

#we'll print the training set
plot_decision_regions(features_train.to_numpy(), target_train.to_numpy(), clf=model)
plt.title('Decision boundary for 0 (non virginica) vs 1 (virginica)')
plt.xlabel(iris.feature_names[feature_x])
plt.ylabel(iris.feature_names[feature_y])
plt.show()/usr/local/lib/python3.6/dist-packages/mlxtend/plotting/decision_regions.py:244: MatplotlibDeprecationWarning: Passing unsupported keyword arguments to axis() will raise a TypeError in 3.3.
  ax.axis(xmin=xx.min(), xmax=xx.max(), y_min=yy.min(), y_max=yy.max())

The decision boundary is linear, as expected by logistic regression. This means that all samples in the pink area will be classified as 1 (virginica) and all points in the blue area be considered 0 (non virginica).

Note that relatively many virginica samples are in the blue area compared to the numer of non-virginica present in the pink area. Also note that, roughly speaking, the regressor assigned a wider area to non-virginica. This is a direct consequence of having an unbalanced dataset: two-thirds of the samples are non-virginica (blue squares) and one-third are virginica (red triangles). **The resulting regressor is polarised** toward the more numerous class._____no_output_____# Actual predictions

Any model is only useful when it's used to predict new, unknown data. In fact the whole validation set was put apart and not really used for training for this specific reason.

Luckily, it's very easy to apply a trained model to new values via the [predict() method](https://keras.io/api/models/model_training_apis/#predict-method)._____no_output_____

predictions = model.predict(features_val)
print(predictions)[[0.22289124]
 [0.42798153]
 [0.2892147 ]
 [0.56683457]
 [0.15156415]
 [0.18946213]
 [0.42798153]
 [0.3026018 ]
 [0.49765265]
 [0.64505094]
 [0.27618527]
 [0.34743673]
 [0.17050767]
 [0.3626923 ]
 [0.46204674]
 [0.47805318]
 [0.48159555]
 [0.1062865 ]
 [0.09407479]
 [0.44728738]
 [0.477463  ]
 [0.36707753]
 [0.34797326]
 [0.44611812]
 [0.10606205]
 [0.33196336]
 [0.14232534]
 [0.51312387]
 [0.19990602]
 [0.09407479]]

We can now compare the list of prediction (sometimes called Ŷ) with the true classes from our validation set (sometimes called Y)._____no_output_____

#starting a new plot
fig, ax = plt.subplots()

#adding data in two bunches
ax.scatter(x=range(30), y=predictions, c='red',  label='predictions')
ax.scatter(x=range(30), y=target_val,  c='blue', label='true classes')

#adding a horizontal line at quote 0.5, to represent the decision boundary
ax.hlines(y=0.5, xmin=0, xmax=29, colors='black')

#adding the legend and printing the plot
ax.legend(loc='upper right')
plt.show()_____no_output_____

Note that the vast majority of predictions is below the 0.5 line (recall that the closer a prediction is to 0 or to 1 the higher the prediction confidence).

This confirms our suspicions: the regressor prefers to assign new data to the 0 (non virginica) class._____no_output_____# Multiclass classification using Softmax

The assignment ask us to do actual three-classes classification using a Softmax function, which can be easily considered as an extension of logistic regression over three (or more) classes.

Luckily, Keras provides a [softmax activation function](https://keras.io/api/layers/activations/#softmax-function), which we will use instead of the logistic we previously used.

The structure of our network will be similar, but the output goes from a single number to **three** numbers, one per class, and we thus need three nodes:

As a result, the loss function will need to change. Remember, loss represents a measure of how good the predictions are. Previously we used binary_crossentropy, but since now predictions are multiclass we need to change function. Luckily Keras provides a natural extension for the multiclass case with [CategoricalCrossentropy](https://keras.io/api/losses/probabilistic_losses/#categoricalcrossentropy-class)_____no_output_____## A multiclass model

We are now ready to declare our new model:_____no_output_____

# 3-class softmax regression in Keras
model_multi = Sequential()
model_multi.add(Dense(3, activation='softmax', input_dim=features_train.shape[1]))

#compile the model specifying the new multiclass loss
model_multi.compile(optimizer='rmsprop', loss='categorical_crossentropy')_____no_output_____

Let's take a look under the hood:_____no_output_____

print(model_multi.summary())Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense_1 (Dense)              (None, 3)                 9
=================================================================
Total params: 9
Trainable params: 9
Non-trainable params: 0
_________________________________________________________________
None

We now have to train nine parameters: three coefficients (W1, W2, B) times tree nodes._____no_output_____## Categorical data

We are now *almost* ready to train. First, we need to go back to having three classes:_____no_output_____

#a new array for multi class
target_multi = target
target_multi[0:50] = 0      #setosa
target_multi[50:100] = 1    #versicolor
target_multi[100:150] = 2   #virginica

#or we could have used iris.target ..._____no_output_____

The problem is that our `target_multi` variable is a numeric array. But those numbers we used (0, 1, and 2) do not represent real values. In other words, "virginica" is not twice "versicolor". Numbers here are used as labels, not as quantities.

In fact, to properly train a model the structure of the target array must change to [one-hot encoding](https://en.wikipedia.org/wiki/One-hot). In simple terms, it needs to become a table with one row per sample (150 in total) and one column per class (three in total). Something like:

| Setosa | Versicolor | Virginica |
|------|------|------|
|   0  |   1  |   0  |
|   1  |   0  |   0  |
|   1  |   0  |   0  |
|   0  |   0  |   1  |

As you can see the first sample is Versicolor, the second and third are Setosa, the last one is Virginica. Note that there is only a single "one" per row.

Luckily, it's easy to pass to one-hot encoding using keras function [to_categorical](https://keras.io/api/utils/python_utils/#to_categorical-function):

_____no_output_____

#the "utils" subpackage is very useful, take a look to it when you have time
from keras.utils import to_categorical

#converting to categorical
target_multi_cat = to_categorical(target_multi)

#since everything else is a Pandas dataframe, let's stick to the format
#for consistency
target_multi_cat = pd.DataFrame(target_multi_cat)

#let's take a look
print(target_multi_cat)       0    1    2
0    1.0  0.0  0.0
1    1.0  0.0  0.0
2    1.0  0.0  0.0
3    1.0  0.0  0.0
4    1.0  0.0  0.0
..   ...  ...  ...
145  0.0  0.0  1.0
146  0.0  0.0  1.0
147  0.0  0.0  1.0
148  0.0  0.0  1.0
149  0.0  0.0  1.0

[150 rows x 3 columns]

## Training and validation sets, redux

We are now ready to create our training and validation sets, as done above:_____no_output_____

#we can actually reuse the StratifiedShuffleSplit object ("sss") declared above,
#thus keeping the same split ratio (80/20). We are going to overwrite the
#previous variables for simplicity, but using the new target table:
for train_index, val_index in sss.split(features, target_multi_cat):
    features_train = features.iloc[train_index, :]
    features_val   = features.iloc[val_index, :]
    target_train   = target_multi_cat.iloc[train_index, :]
    target_val     = target_multi_cat.iloc[val_index, :]_____no_output_____

Just a little check:_____no_output_____

#shapes
print(features_train.shape)
print(features_val.shape)
print(target_train.shape)
print(target_val.shape)

#number of classes per split
print('\nClasses in train set:')
print(target_train.sum())
print('\nClasses in validation set:')
print(target_val.sum())(120, 2)
(30, 2)
(120, 3)
(30, 3)

Classes in train set:
0    40.0
1    40.0
2    40.0
dtype: float32

Classes in validation set:
0    10.0
1    10.0
2    10.0
dtype: float32

We have now a balanced dataset, with 40 instances for each class in the training set and 10 in the validation set._____no_output_____## Training_____no_output_____We are ready to train. This time we go directly to 500 epochs, trained in silent mode. We then plot the loss function evolution. _____no_output_____

history_multi = model_multi.fit(features_train, target_train, epochs=500,
                     validation_data=(features_val, target_val), verbose=0)

plot_loss_history(history_multi, 'Softmax multiclass (500 epochs)')_____no_output_____

## Decision boundary

We want now to plot again the decision boundary. Unfortunately `plot_decision_regions` function from [mlxtend](http://rasbt.github.io/mlxtend/) module does not support one-hot encoded multiclasses natively. Luckily [there's a quick workaround](https://www.machinecurve.com/index.php/2019/10/17/how-to-use-categorical-multiclass-hinge-with-keras/#visualizing-the-decision-boundary), but if you get lost in the code don't worry and just look at the plot :) _____no_output_____

import numpy as np

#we define a class to take the Keras model and convert its predictions
#from "one probability per iris type" to "just the iris type with the highest probability"
class Onehot2Int(object):
    def __init__(self, model):
        self.model = model

    def predict(self, X):
        y_pred = self.model.predict(X)
        return np.argmax(y_pred, axis=1)

#we wrap our trained model, instantiating a new object
keras_model_no_ohe = Onehot2Int(model_multi)

#and we can now plot the decision boundary safely (we still need to convert
#the target one-hot-encoded matrix to int, though)
plot_decision_regions(features_train.to_numpy(), np.argmax(target_train.to_numpy(), axis=1),
                      clf=keras_model_no_ohe)
plt.title('Decision boundary for 0 (setosa) vs 1 (versicolor) vs 2 (virginica)')
plt.xlabel(iris.feature_names[feature_x])
plt.ylabel(iris.feature_names[feature_y])
plt.show()/usr/local/lib/python3.6/dist-packages/mlxtend/plotting/decision_regions.py:244: MatplotlibDeprecationWarning: Passing unsupported keyword arguments to axis() will raise a TypeError in 3.3.
  ax.axis(xmin=xx.min(), xmax=xx.max(), y_min=yy.min(), y_max=yy.max())

Please notice how the decision boundaries are still linear. This is expected since Softmax is the natural extension of Logistic Regression to more than two classes._____no_output_____# Next steps...

In this exercise we used a keras node with sigmoid activation function to perform logistic regression on one of the classes from the famous Iris dataset. Several more steps could be implemented:

* to improve our understanding of logistic regression we could implement it from scratch, without using keras, following the [steps detailed here](https://towardsdatascience.com/a-logistic-regression-from-scratch-3824468b1f88)
* the original dataset contains three classes. We could take the chance to build three different regressors, one per class, and then link them together, maybe via a [softmax node](https://keras.io/api/layers/activations/#softmax-function), and train the resulting two-layers, four-nodes network
* what is the effect of including the other two features, which we excluded when solving this exercise? Do performance improve? What happens to decision boundary?
* when implementing the newtork we used [rmsprop optimizer](https://keras.io/api/optimizers/rmsprop/) and [binary_crossentropy](https://keras.io/api/losses/probabilistic_losses/#binarycrossentropy-class) loss function, but other [optimizers](https://keras.io/api/optimizers/) and [loss functions](https://keras.io/api/losses/) are available and their effect could be explored_____no_output_____


# Notebook from gordian-biotechnology/CellOracle
Path: docs/notebooks/01_ATAC-seq_data_processing/option1_scATAC-seq_data_analysis_with_cicero/.ipynb_checkpoints/01_atacdata_analysis_with_cicero_and_monocle3 -checkpoint.ipynb

# 0. Import library_____no_output_____

library(cicero)_____no_output_____

# 1. Prepare data
In this tutorial we'll use acATAC-seq data from the 10x genomics database.
You do not need to download these data if you analyze your own scATAC-seq data. _____no_output_____

# Create folder to store data
dir.create("data")

# Download demo dataset from 10x genomics
system("wget -O data/matrix.tar.gz http://cf.10xgenomics.com/samples/cell-atac/1.1.0/atac_v1_E18_brain_fresh_5k/atac_v1_E18_brain_fresh_5k_filtered_peak_bc_matrix.tar.gz")

# Unzip data
system("tar -xvf data/matrix.tar.gz -C data")_____no_output_____# You can substitute the data path below with the data path of your scATAC data.
data_folder <- "data/filtered_peak_bc_matrix"

# Create a folder to save results
output_folder <- "cicero_output"
dir.create(output_folder)_____no_output_____

# 2. Load data and make Cell Data Set (CDS) object
## 2.1. Process data to make CDS object_____no_output_____

# read in matrix data using the Matrix package
indata <- Matrix::readMM(paste0(data_folder, "/matrix.mtx"))
# binarize the matrix
indata@x[indata@x > 0] <- 1

# format cell info
cellinfo <- read.table(paste0(data_folder, "/barcodes.tsv"))
row.names(cellinfo) <- cellinfo$V1
names(cellinfo) <- "cells"

# format peak info
peakinfo <- read.table(paste0(data_folder, "/peaks.bed"))
names(peakinfo) <- c("chr", "bp1", "bp2")
peakinfo$site_name <- paste(peakinfo$chr, peakinfo$bp1, peakinfo$bp2, sep="_")
row.names(peakinfo) <- peakinfo$site_name

row.names(indata) <- row.names(peakinfo)
colnames(indata) <- row.names(cellinfo)

# make CDS
input_cds <-  suppressWarnings(new_cell_data_set(indata,
cell_metadata = cellinfo,
gene_metadata = peakinfo))

input_cds <- monocle3::detect_genes(input_cds)

#Ensure there are no peaks included with zero reads
input_cds <- input_cds[Matrix::rowSums(exprs(input_cds)) != 0,] _____no_output_____

# 3. Qauality check and Filtering_____no_output_____

# Visualize peak_count_per_cell
hist(Matrix::colSums(exprs(input_cds)))_____no_output_____# filter cells by peak_count
# PLEASE SET APPROPRIATE THRESHOLD VALUES according to your data
max_count <-  15000
min_count <- 2000
input_cds <- input_cds[,Matrix::colSums(exprs(input_cds)) >= min_count]
input_cds <- input_cds[,Matrix::colSums(exprs(input_cds)) <= max_count]
_____no_output_____

# 4. Process cicero-CDS object_____no_output_____

# Data preprocessing
set.seed(2017)

input_cds <- detect_genes(input_cds)
input_cds <- estimate_size_factors(input_cds)
input_cds <- preprocess_cds(input_cds, method = "LSI")

# Dimensional reduction with umap
input_cds <- reduce_dimension(input_cds, reduction_method = 'UMAP',
                              preprocess_method = "LSI")
umap_coords <- reducedDims(input_cds)$UMAP

cicero_cds <- make_cicero_cds(input_cds, reduced_coordinates = umap_coords)

# Save cds object if you want
saveRDS(cicero_cds, paste0(output_folder, "/cicero_cds.Rds"))

_____no_output_____

# 5. Load reference genome information

To run cicero, we need to get a genome coordinates files, which contains the lengths of each chromosomes.
You can read mm10 genome information with the following command.
The text file, mm10_chromosome_length.txt is included in the celloracle notebook folder.https://github.com/morris-lab/CellOracle/tree/master/docs/notebooks/01_ATAC-seq_data_processing/option1_scATAC-seq_data_analysis_with_cicero

If your scATAC-seq data use different reference genome, you need to get a genome coordinates files for your reference genome. Please see the Cicero documentation for more information.https://cole-trapnell-lab.github.io/cicero-release/docs_m3/#installing-cicero_____no_output_____

# !!Please make sure that the reference genome information below match the reference genome of your scATAC-seq data.

# If your scATAC-seq uses mm10 reference genome, you can read chromosome length file with the following command.
chromosome_length <- read.table("./mm10_chromosome_length.txt")

# For mm9 genome, you can use the following command.
#data("mouse.mm9.genome")
#chromosome_length <- mouse.mm9.genome

# For hg19 genome, you can use the following command.
#data("human.hg19.genome")
#chromosome_length <- mhuman.hg19.genome

_____no_output_____

# 6. Run Cicero_____no_output_____

# runc the main function
conns <- run_cicero(cicero_cds, chromosome_length) # Takes a few minutes to run

# save results
saveRDS(conns, paste0(output_folder, "/cicero_connections.Rds"))

# check results
head(conns)[1] "Starting Cicero"
[1] "Calculating distance_parameter value"
[1] "Running models"
[1] "Assembling connections"
[1] "Done"

# 6. Save results for the next step_____no_output_____

all_peaks <- row.names(exprs(input_cds))
write.csv(x = all_peaks, file = paste0(output_folder, "/all_peaks.csv"))
write.csv(x = conns, file = paste0(output_folder, "/cicero_connections.csv"))
_____no_output_____


# Notebook from JeepGangPoulet/JeepGangPoulet
Path: exercises/01-python/intro_to_python.ipynb

# Introduction to Python_____no_output_____

This notebook was initially made for the CS-233 Introduction to Machine Learning course at EPFL, and adapted for the CIVIL-226 Introduction to Machine Learning for Engineers course. Many thanks to Sena Kiciroglu and the Computer Vision Lab ([CVLab](https://www.epfl.ch/labs/cvlab)) for allowing us to use this notebook.

**Author(s):** [Sena Kiciroglu](mailto:sena.kiciroglu@epfl.ch), minor changes by [Tom Winandy](mailto:tom.winandy@epfl.ch) and [David Mizrahi](mailto:david.mizrahi@epfl.ch)

Welcome to the first exercise of Introduction to Machine Learning. Today we will get familiar with Python, the language we will use for all the exercises of this course.

This week we will introduce some important concepts in the basics of Python. Next week, you will learn how to work with NumPy, a popular Python library used for scientific computing.

Python is a popular language to use for machine learning tasks. This is especially true because of the selection of **libraries and frameworks**, developed specifically for machine learning and scientific computing. To name a few, you have Keras, TensorFlow and PyTorch for developing neural networks, SciPy and NumPy used for scientific computing, Pandas for data analysis, etc. (You might also get to dabble in PyTorch in the upcoming weeks.)

Python also allows you to write quick, readable, high-level code. It's great for fast prototyping.

You can find a useful Python cheatsheet at: https://www.pythoncheatsheet.org/

Let's get into it!
_____no_output_____## 1. Anaconda and Jupyter Notebook

If you're reading this Jupyter notebook as it is intended, chances are you already installed Anaconda, a Python distribution that comes with its own package management system, `conda`. Using `conda`, you can install and upgrade software packages and libraries. It will make managing the versions of the libraries you use very convenient.

In these exercises we will use Jupyter Notebooks, which contain Python code, text explanations and visuals.

The Jupyter Notebook document (such as the one you are looking at right now) consists of cells containing Python code, text or other content. You can run each cell by clicking on the button `Run` in the top toolbar, or you can use a keyboard shortcut `Ctrl` + `Enter` (run current cell) or `Shift` + `Enter` (run current cell and move to the cell below)._____no_output_____## 2. Indentation and Control Flow_____no_output_____Finally we get to start doing some coding!

First thing to know: Python does not separate different lines of code with a semicolon `;`. So just RUN the following cell with no worries._____no_output_____

# This is a Python comment. Start the line with `#` for a comment
print("First line of code. I will declare some variables")
a = 1 # second line!!
b = 2
c = "Fish"
print(f"My variables are: a = {a}, b = {b}, c = {c}")First line of code. I will declare some variables
My variables are: a = 1, b = 2, c = Fish

Easy! However, in Python you have to be careful and have perfect indentation (a reason why Python code is so readable). The reason is, Python uses indentation to keep track of what is part of the if statement, the loops and the functions. This is different from Java (this is assuming you know Java) where you would have curly brackets `{ }` for this purpose.

Let's start with the if statement.

### 2.1. If Statement

The rule is, all indented parts after the `if condition :` belong to that branch of the if statement.

```python
if condition :
    inside the statement
    still inside the statement
elif condition:
    inside the else-if part of the statement
else:
     inside the else part of the statement
outside the statement
 ```

Let's see it in action:_____no_output_____

if a + b == 3:
    print("It's me again! We are inside the first if statement")
    print("It's optional to use parentheses for the condition a + b ==3")
    print("Don't forget to put a `:` at the end of the condition!!")
    if (c == "Fish"):
        print("This is a second if statement inside the first one")
    print("I'm out of the second if statement, but still inside the first one")
else:
    print("This is the else part of the first if statement.")
    print("These lines will never be printed!")
print("I'm not inside any of the if statements")_____no_output_____

#### Exercise

Let's see another if statement example. Try to figure out what the output will be **BEFORE** running the cell below.

Reminder, we declared

```python
a = 1
b = 2
c = "Fish"
 ```_____no_output_____

# Don't run me until you find the output first!
if a == 5:
    print ("1")
    if b == 1:
        print("2")
# here comes an else-if
elif a == 2 or c == "Fish":
    print("3")

    if b == 1:
        print("4")
        if b == 2:
            print("5")
    if b == 2:
        print("6")
    if c == "Fish":
        if a == 1:
            if b == 100:
                print("7")
            else:
                print("8")
    elif a == 1:
        print("9")
print ("10")_____no_output_____

### 2.2. Loops

Let's talk about loops. The syntax for a while-loop is:

```python
while condition:
    inside the loop
    inside the loop
    inside the loop
outside the loop
 ```

 A small example:_____no_output_____

count = 0
while count < 3:
    count += 1 # this is the same as count = count +1
    print(f"Count is {count}")
print("Left the loop!")_____no_output_____

For-loops iterate through sequences, in this way:

```python
for x in sequence:
    inside the loop
    inside the loop
    inside the loop
outside the loop
```

 An example is shown below:_____no_output_____

#Here is a basic list of strings
fish_list = ["salmon", "trout", "parrot", "clown", "dory"]

#The for loop:
for fish in fish_list:
    print(fish)
    print("*")
print("fish list over!")_____no_output_____

An incredibly useful built-in function to use in for loops is `range()`. Range allows you to create a sequence of integers from the start (default is 0), to the stop, with a given step size (default is 1). We can use `range()` in for loops as shown in the example below._____no_output_____

# "default start is 0, default step size is 1"
for number in range(7):
    print (number)
print("**")

# now we also provide the start as 2.
# Default step size 1 is still used.
for number in range(2,7):
    print(number)
print("**")

# now we also provide the step size as 2.
for number in range(2,7,2):
    print(number)
print("**")

# what happens if step size is -1?
for number in range(6,-1,-1):
    print(number)_____no_output_____

One more useful built-in function will be `enumerate()`. Let's go back to the fish list.
_____no_output_____

for fish in fish_list:
    print(fish)_____no_output_____

What if I also want to keep track of the index of the list element? You can use `enumerate()` which creates a sequence of 2-tuples, where each tuple contains an integer index and an actual element of the original list. Here is how it looks like:_____no_output_____

for item_index, fish in enumerate(fish_list):
    print(f"{item_index}: {fish}")_____no_output_____

## 3. Data Types and Basic Operations

Python is a **dynamically typed** language. This means that the data type is inferred at run-time and can be changed during run-time. To check the type of a variable you can use the function `type()`._____no_output_____

# var_1 is first defined as an integer
var_1 = 1
print(f"{var_1} is {type(var_1)}")

# var_1's type is changed to string
var_1 = "hi!"
print(f"{var_1} is {type(var_1)}")

# more types
var_1 = 0.312
print(f"{var_1} is {type(var_1)}")
var_1 = 3.
print(f"{var_1} is {type(var_1)}")
var_1 = 3+2j
print(f"{var_1} is {type(var_1)}")
var_1 = True
print(f"{var_1} is {type(var_1)}")_____no_output_____

### 3.1. Type Casting

Some examples of type casting in Python:_____no_output_____

# From int to float
var_1 = 42
print(f"{var_1} is {type(var_1)}")
var_1 = float(var_1)
print(f"{var_1} is {type(var_1)}")
print ("**")

# From float to int
var_2 = 3.14
print(f"{var_2} is {type(var_2)}")
var_2 = int(var_2)
# This operations does FLOOR, not round!
print(f"{var_2} is {type(var_2)}")
print ("**")

# From string to int
var_3 = "100"
print(f"{var_3} is {type(var_3)}")
var_3 = int(var_3)
print(f"{var_3} is {type(var_3)}")
print("**")

# From float to string
var_4 = 1.23
print(f"{var_4} is {type(var_4)}")
var_4 = str(var_4)
print(f"{var_4} is {type(var_4)}")
print("**")
_____no_output_____

### 3.2. Basic Operations

Arithmetic operations are fairly standard. There are some examples below.
* Look out for the difference between `/` division and `//` integer division.
* `**` is used for power.
* `%` is modulo._____no_output_____

a = 50
b = 7

print(f"a + b = {a + b}")
print(f"a - b = {a - b}")
print(f"a * b = {a * b}")
print(f"a / b = {a / b}")
print(f"a // b = {a // b}") # integer divison
print(f"a ** b = {a ** b}") # power
print(f"a % b = {a % b}") # modulo_____no_output_____

Boolean operations are also fairly standard:_____no_output_____

print(f"(True and False) = {True and False}")
print(f"(True or False) = {True or False}")
print(f"((True and False) or True) = {(True and False) or True}")_____no_output_____

You can declare strings with a single quote `'`, a double quote `"` or a three double quotes `"""`. The string declared with `"""` is known as a *docstring*, it can span multiple lines and is usually used to comment functions and classes.

**Note:** Throughout the exercises, we will be using f-strings to format our strings nicely. You can learn more about them [here](https://realpython.com/python-f-strings/)._____no_output_____

a = 'Life\'s but a walking shadow, a poor player,'
print(a)
a = "That struts and frets his hour upon the stage,"
print(a)
a = """And then is heard no more. It is a tale
Told by an idiot, full of sound and fury,
Signifying nothing."""
print(a)_____no_output_____# The types of quotes do not change anything!
a = "fish" # double quote
b = 'fish' # single quote
print(a == b) # the string is the same!_____no_output_____

## 4. Lists

Lists are data types containing a sequence of values. The size of the list can change during run-time, as you add and remove elements from the list.

Here is how you can create lists:_____no_output_____

list_a = [] # empty
print(f"list_a {list_a}")

list_b = [1, 2, 3, 4] # 4 elements
print(f"list_b {list_b}")

list_c = [1, 'cat', 0.23] # mixed types
print(f"list_c {list_c}")

list_d = [1, ['cat', 'dog'], 2, 3] # list in list
print(f"list_d {list_d}")

list_e = [1] * 10 # a list of 1s of length 10
print(f"list_e {list_e}")

list_f = list(range(5)) # turns range object into a list
print(f"list_f {list_f}")_____no_output_____

Below we introduce some common operations with lists.
* Use `len(list1)` to find the length of the list.
* `list1.append(element)` to add an element to the end of the list.
* `list1.insert(index, element)` to add an element to an index in the list
* `list1.extend(list2)` to extend the elements of list1 with the elements of list2
* `list1.pop()` removes last element from the list
* `list1.pop(index)` removes the element at the given index
* `list1.remove(element)` removes the first instance of the given element_____no_output_____

# Some common operations
b = ["great", "minds", "think", "alike"]
print(f"b: {b}")

# finding the length
print(f"length of b is {len(b)}")

# append element to list
b.append("sometimes")
print(f"b.append(\"sometimes\")= {b}")

# extend list
c = ["-", "Abraham", "Lincoln"]
b.extend(c)
print(f"c: {c}")
print(f"b.extend(c) = {b}")

# removes element and specific index
b.pop(6)
print(f"b.pop(6) = {b}")

# remove specific element
b.remove("Lincoln")
b.remove("-")
print(f"b.remove(\"Lincoln\"); b.remove(\"-\") = {b}")
_____no_output_____

You  can also check whether an element is in a list in the following way:_____no_output_____

list_1 = ["a", "b", "c"]
if "b" in list_1:
    print("\"b\" is in list")
else:
    print("\"b\" is not in list")_____no_output_____

### 4.1. List Indexing and Slicing:

You can extract a single element from a list in the following way:
`list1[index]`

In lists, the indices start from 0. You can also index elements from the end of the list to the beginning by $-1, -2, -3...$. Check out the image below for the example list:

`list_1 = ["a", "b", "c", "d", "e"]`_____no_output__________no_output_____* You can extract multiple elements by slicing. This will give you elements from the start up to **(but not including)** the end index.

  `list1[start_index:end_index]`

* If you do not specify the `start_index`, you will retrieve the elements from index $0$ up to the `end_index`.

  `list1[:end_index]` is the same as `list1[0:end_index]`

* If you do not specify the `end_index`, you will retrieve the elements from the `start_index` up to (and **including**) the end of the list.

  `list1[start_index:]`

* You can provide a step size.
  `list1[start_index:end_index:step_size]`
  _____no_output_____#### Exercise

Try to write the output of the following code **BEFORE** running the cell._____no_output_____

# Don't run BEFORE you solve it!
list_1 = ["a", "b", "c", "d", "e"]

print(f"list_1[-3] = {list_1[-3]}")
print(f"list_1[0:2] = {list_1[0:2]}")
print(f"list_1[:4:2] = {list_1[:4:2]}")
print(f"list_1[::-1] = {list_1[::-1]}")
print(f"list_1[-4:-1] = {list_1[-4:-1]}")_____no_output_____

You can also assign new values to indices using slicing. Here is an example:_____no_output_____

list_1 = ["a", "b", "c", "d", "e"]

list_1[-1]= "<3"
print(list_1)

list_1[0:2] = ["x", "y"]
print(list_1)

list_1[::2] = [":)",":(", ":O"]
print(list_1)_____no_output_____

### 4.2. Copying

We have one last thing to say about lists. Observe the behaviour of the following code:_____no_output_____

# Case 1:

list_1 = ["a", "b", "c", "d", "e"]
print(f"list_1 before {list_1}")

list_2 = list_1
list_2.append("Z")

print(f"list_1 after {list_1}")_____no_output_____# Case 2:

list_1 = ["a", "b", "c", "d", "e"]
print(f"list_1 before function {list_1}")

def function_that_changes_list(input_list):
    input_list.append("Z")

function_that_changes_list(list_1)

print(f"list_1 after function {list_1}")_____no_output_____

We never changed list_1 explicitly, but the values changed anyway. What's going on?

Well, in Python, when you say `list_2 = list_1`, you are not actually creating a new list, you are only copying the **reference** to the same list. This means that they are actually two variables pointing to the same list! So when you change the values of `list_2`, the values of `list_1` also change (since they are referring to the same list). Something similar is at play when you pass this list to a function. So be careful!

If you do not want this to happen, you can use the function `.copy()` to create a new object with the same values.

#### Exercise

Change the code below and fix the two cases given above using the `.copy()` function. Make sure the contents of `list_1` do not change._____no_output_____

# Case 1
list_1 = ["a", "b", "c", "d", "e"]
print(f"list_1 before {list_1}")

list_2 = list_1
list_2.append("Z")

print(f"list_1 after {list_1}")
print("**")

# Case 2
list_1 = ["a", "b", "c", "d", "e"]
print(f"list_1 before function {list_1}")

def function_that_changes_list(input_list):
    input_list.append("Z")

list_2 = list_1.copy()
function_that_changes_list(list_2)

print(f"list_1 after function {list_1}")_____no_output_____

#### Exercise

Now that we know how lists work, here is a quick exercise for you. Fill in the function below that takes a list and returns True if it is a palindrome, False if it is not. Palindromes are defined as sequences that read the same forwards and backwards.
Examples of palindrome lists:
* ["cat", "dog", "fish", "dog", "cat"]
* [0, 1, 2, 3, 3, 2, 1, 0]
* [1]
* []

You may use a for-loop in this exercise. However, if you're feeling ambitious try to do it in 1 line, without using a for-loop (hint: use slicing)_____no_output_____

def function_is_palindrome(input_list):
    is_palindrome = True
    # Your code here
    return is_palindrome_____no_output_____test_list_1 = ["cat", "dog", "fish", "dog", "cat"]
res_1 = function_is_palindrome(test_list_1)

test_list_2 = ["cat", "dog", "fish",  "bird", "dog", "cat"]
res_2 = function_is_palindrome(test_list_2)

test_list_3 = ["cat"]
res_3 = function_is_palindrome(test_list_3)

test_list_4 = ["cat", "cat"]
res_4 = function_is_palindrome(test_list_4)

if not (res_1 and not res_2 and res_3 and res_4):
    print("Test failed")
else:
    print("Correct! :)")
_____no_output_____

## 5. Tuples

Tuples are similar to lists but they are fixed in size and **immutable**, which means that change is not allowed.
We declare tuples in the following way using parentheses`()`:_____no_output_____

tuple_1 = ("wash", "your", "hands", "with", "soap")

print(f"tuple_1 = {tuple_1}")_____no_output_____

Since change is not allowed, observe the result of the following piece of code._____no_output_____

tuple_1[2] = ("face")_____no_output_____

You can typecast from list to tuple and vice versa! _____no_output_____

sequence_1 = ["here", "comes", "the", "sun"]
print(f"{sequence_1} is {type(sequence_1)}")

# from list to tuple
sequence_1 = tuple(sequence_1)
print(f"{sequence_1} is {type(sequence_1)}")

#from tuple to list
sequence_1 = list(sequence_1)
print(f"{sequence_1} is {type(sequence_1)}")_____no_output_____

## 6. Dictionaries

An incredibly useful data type to know, you might also know dictionaries as "hash maps". Dictionaries are collections of "key: value" pairs. You can access the values using the keys in $O(1)$ time.

The keys of a dictionary must be **immutable** and **unique**. Below we show how to define a dictionary.
_____no_output_____

shopping_list = {"apples": 3, "pears":2, "eggs":6, "bread":1, "yogurt":1}
print(f"shopping_list = {shopping_list}")
print("**")

book_dict = {}
print(f"book_dict = {book_dict}")
#add key value pairs
book_dict["vonnegut"] = "cat\'s cradle"
book_dict["ishiguro"] = "never let me go"
print(f"book_dict = {book_dict}")
print("**")

# we can retrieve the dict keys:
print(book_dict.keys())
# and the dict values:
print(book_dict.values())
print("**")

#we can also iterate through the dict keys and values with a for-loop
for key, value in book_dict.items():
    print(f"{key} : {value}")

print("**")
#we can modify the value of a key
book_dict["ishiguro"] = "a pale view of hills"
print(f"modified book_dict = {book_dict}")
print("**")

#and we can remove a key completely
removed_value = book_dict.pop("ishiguro")
print(f"book_dict with removed value = {book_dict}")
print(f"removed_value = {removed_value}")_____no_output_____

## 7. Functions

You can define a function in Python in the following way:_____no_output_____

def multiply(a, b):
    return a * b

print(f"multiply(100, 2) = {multiply(100, 2)}")_____no_output_____

You can have default arguments by specifying their default value in the parameters._____no_output_____

def add(a, b, c=0, d=1):
    return a + b + c + d

# use no default arguments
print(f"add(1, 2, 100, 1000) = {add(1, 2, 100, 1000)}")

# use the default value of d
print(f"add(1, 2, 100) = {add(1, 2, 100)}")

# use the default value of c and d
print(f"add(1, 2) = {add(1, 2)}")

# use the default value of c
print(f"add(1, 2, d=1000) = {add(1, 2, d=1000)}")_____no_output_____

A function can return multiple values in a tuple. You can assign the values of the tuple to separate variables. This is called **tuple unpacking**._____no_output_____

def min_max(input_list):
    return min(input_list), max(input_list)

test_list = [1,2,3,4]
min_val, max_val = min_max(test_list)
print(f"min_val: {min_val}, max_val: {max_val}")_____no_output_____

Note: You have seen tuple unpacking when using function `enumerate` in for-loop._____no_output_____### 7.1. Common Built-in Functions

Here we introduce some nifty commonly used built-in functions.

* You already learned `range()`, `enumerate()`!
* We have also seen `type()` to return the type of the object. We use `str()`, `int()`, `float()`, `list()`, `tuple()` for typecasting.
* The functions `len()`, `sum()`, `min()`, `max()`, `any()`, `all()`, `sorted()`, `zip()` are useful for lists and tuples.

Let's see them in action below_____no_output_____

list_1 = list(range(5))
print(f"list_1 = {list_1}")

print(f"len(list_1) = {len(list_1)}")
print(f"sum(list_1) = {sum(list_1)}")
print(f"min(list_1) = {min(list_1)}")
print(f"max(list_1) = {max(list_1)}")
print("**")

list_2 = [5,3,1,2,0,6]
print(f"list_2 = {list_2}")
print(f"sorted(list_2) = {sorted(list_2)}")
print("**")

# any checks whether there are any 1s in the list (OR)
# all checks whether all elements are 1s. (AND)
# in Python: 1 = True, 0 = False
list_3 = [1, 1, 1]
print(f"list_3 = {list_3}")
print(f"any(list_3) = {any(list_3)}")
print(f"all(list_3) = {all(list_3)}")

list_4 = [0, 1, 1]
print(f"list_4 = {list_4}")
print(f"any(list_4) = {any(list_4)}")
print(f"all(list_4) = {all(list_4)}")

list_5 = [0, 0, 0]
print(f"list_5 = {list_5}")
print(f"any(list_5) = {any(list_5)}")
print(f"all(list_5) = {all(list_5)}")
print("**")

# zip function:
x = [1,2,3]
y = [4,5,6]
zipped = zip(x,y)
print(f"x {x}")
print(f"y {y}")
print(f"zipped {list(zipped)}")_____no_output_____

### 7.2. List Comprehensions

One of the most practical things about Python is that you can do many things on just a single line. One popular example is so called *list comprehensions*, a specific syntax to create and initalize lists of objects. Here are some examples.

A syntax for list comprehension is shown below:
`[thing for thing in list]`

Let's make it more concrete with an example._____no_output_____

list_of_numbers = [1, 2, 3, 101, 102, 103]
print(f"list_of_numbers = {list_of_numbers}")

#I want to create a new list with all these items doubled.
doubled_list = [2 * elem for elem in list_of_numbers]
print(f"doubled_list = {doubled_list}")

#A new list with all these items as floats
float_list = [float(elem) for elem in list_of_numbers]
print(f"float_list = {float_list}")_____no_output_____

Let's make it more interesting by adding an if in there:

`[thing for thing in list if condition]`
_____no_output_____

#I want to create a new list with all these items doubled
#IF the element is above 100
conditional_doubled_list = [2 * elem for elem in list_of_numbers if elem > 100]
print(f"conditional_doubled_list = {conditional_doubled_list}")_____no_output_____

#### Exercise

You will be given a list of vocabulary words. Your task is to use list comprehensions to iterate through a document and create a new list including the words that are included in the vocabulary. You don't need to worry about duplicates.

Example:
```python
vocabulary = ["a" "c", "e"]
document = ["a", "b", "c", "d"]
new_list = ["a", "c"]
```_____no_output_____

vocabulary = ['epfl', 'europe', 'swiss', 'switzerland', 'best', 'education', 'high', 'higher', 'research', 'school', 'science', 'students', 'technology', 'top-tier', 'university']

document = """The École polytechnique fédérale de Lausanne (EPFL) is a research institute
and university in Lausanne, Switzerland, that specializes in natural sciences and engineering.
It is one of the two Swiss Federal Institutes of Technology, and it has three main missions:
education, research and technology transfer at the highest international level. EPFL is widely regarded
as a world leading university. The QS World University Rankings ranks EPFL 12th in the world
across all fields in their 2017/2018 ranking, whilst Times Higher Education World
University Rankings ranks EPFL as the world's 11th best school for Engineering and Technology."""
document_parsed = document.split()
document_parsed = [word.lower() for word in document_parsed]
new_list = []

#your code here
new_list = ...

#We convert the list to a set and then back to a list. We do this because converting it to a set automatically
#removes duplicates (since sets are sequences that do not contain duplicates). afterwards we sort it.
new_list = sorted(list(set(new_list)))
_____no_output_____correct_result = ['best', 'education', 'epfl', 'higher', 'research', 'school', 'swiss', 'technology', 'university']

if new_list == correct_result:
    print ("Correct! :)")
else:
    print ("Incorrect :(")
_____no_output_____

## 8. Object-Oriented Programming_____no_output_____Object-oriented programming is a programming paradigm that provides a means of structuring programs so that properties and behaviors are bundled into individual objects.

For this end, we use classes. Classes are used to create user-defined data structures. Classes define functions called methods, which identify the behaviors and actions that an object created from the class can perform with its data._____no_output_____All class definitions start with the class keyword, which is followed by the name of the class and a colon. Any code that is indented below the class definition is considered part of the class’s body. To start, let's declare an `EPFL_faculty` class._____no_output_____

class EPFL_faculty:

    def __init__(self, name, number_of_students):
        self.name = name
        self.number_of_students = number_of_students

    # Instance method
    def description(self):
        return f"The faculty {self.name} has {self.number_of_students} students"_____no_output_____

While the class is the blueprint, an instance is an object that is built from a class and contains real data. `.__init__()` sets the initial state of the object by assigning the values of the object’s properties. That is, `.__init__()` initializes each new instance of the class._____no_output_____

enac = EPFL_faculty("ENAC", 500)_____no_output_____

Now that the instance `enac` has been created, we can call the method description._____no_output_____

enac.description()_____no_output_____

Inheritance is the process by which one class takes on the attributes and methods of another. Newly formed classes are called child classes, and the classes that child classes are derived from are called parent classes. Child classes inherit from the parent's attributs and methods but it can overwrite methods. Let's define a class `EPFL_section` that inherits from `EPFL_faculty` and that overwrites the method `description(self)`._____no_output_____

class EPFL_section(EPFL_faculty):

    def description(self):
        return f"The section {self.name} has {self.number_of_students} students"_____no_output_____genie_civil = EPFL_section("GC", 200)_____no_output_____genie_civil.description()_____no_output_____

__Disclaimer__: This part of the tutorial and its text was inspired by and taken from "Object-Oriented Programming (OOP) in Python 3" by David Amos. _____no_output_____### Additional OOP resources_____no_output_____For further information on object-oriented programming and classes in Python, here are two useful resources:
* Object-Oriented Programming (OOP) in Python 3: https://realpython.com/python3-object-oriented-programming/
* Classes from the official Python Tutorial: https://docs.python.org/3/tutorial/classes.html_____no_output_____## 9. (OPTIONAL) Matplotlib_____no_output_____Perhaps the most widely used plotting library in Python is Matplotlib. If you've ever used MATLAB, you'll find that the functions look pretty similar.

In the following exercise sessions, we won't ask you to do any plotting. So this part is optional for those who are interested in having a short introduction.

First, we will import Matplotlib._____no_output_____### Importing in Python

* A short note on importing: to be able to use modules in our code, we import them.

  example: `import numpy`

* We can also select a name for the imported module.

  example: `import numpy as np`. Now when we call numpy functions, we will always use `np.` as a prefix, i.e. `np.zeros()`

* You can also choose to only import selected functions/variables/classes from the module.

  example: `from numpy import arange`. Now you can use this function as `arange(5)`. You cannot use any other functions from the numpy module as you did not import them._____no_output_____

# To import Matplotlib we do:

import matplotlib.pyplot as plt_____no_output_____

Let's do some plotting!

Let's start with the simplest of plots, the good old line-plot. The function we will use is `plot()`_____no_output_____

#Let's create some data first to plot
x = list(range(10))
y = [2,3,5,1,0,2,3,0,0,1]

#first create a figure
fig = plt.figure()

#now do the plotting
#specifying a color and marker are optional.
#check out the documentation to see what else you can do with the plot function
plt.plot(x, y, marker="*", color="r")

#axis labels and title
plt.xlabel("x")
plt.ylabel("y")
plt.title("just a random plot")

#so that we see the plot
plt.show()

#close the plot
plt.close(fig)_____no_output_____

#### Exercise

You can plot two lines on top of one another by calling the `plt.plot()` function consecutively. Try to implement this! Also, specify the parameter `label` of the `plt.plot()` function and call the function `plt.legend()` to create a legend for your graph. It should look like the figure shown below._____no_output_____![result](images/two_lines_plot.png)_____no_output_____

#Let's create some data first to plot
x = list(range(10))
y1 = [2,3,5,1,0,2,3,0,0,1]
y2 = [1,2,3,5,1,0,2,3,0,0]

#first create a figure
fig = plt.figure()
#your code here
_____no_output_____

You can create scatter plots (line plots without lines) with `scatter()`_____no_output_____

#Let's create some data first to plot
x = list(range(10))
y1 = [2,3,5,1,0,2,3,0,0,1]

#first create a figure
fig = plt.figure()

#now do the plotting
p1 = plt.scatter(x, y1, marker="*", color="r")

#axis labels and title
plt.xlabel("x")
plt.ylabel("y")
plt.title("just a random scatter plot")

#so that we see the plot
plt.show()

#close the plot
plt.close(fig)_____no_output_____

And you can read and display images with `imread()` and `imshow()`_____no_output_____

#first create a figure
fig = plt.figure()

#now do the plotting
im = plt.imread("images/krabby_patty.jpg")
plt.imshow(im)

#axis labels and title
plt.xlabel("x")
plt.ylabel("y")
plt.title("krabby patty")

#so that we see the plot
plt.show()

#close the plot
plt.close(fig)_____no_output_____

And that's all for this exercise! If you have any problems, just ask (or even Google) them. You can check out the official Python tutorials for further learning.

https://docs.python.org/3/tutorial/_____no_output_____


# Notebook from katiehuang1221/onl_ds5_project_4
Path: notebook/network.ipynb

"""
Explore network analysis

Include notes from DataCramp.

"""_____no_output_____import matplotlib.pyplot as plt
import networkx as nx_____no_output_____# Draw the graph to screen
nx.draw(T_sub)
plt.show()_____no_output_____

The .nodes() method returns a list of nodes,
while the .edges() method returns a list of tuples.
Recall that passing in the keyword argument data=True in these methods retrieves the corresponding metadata associated with the nodes and edges as well._____no_output_____

T.nodes(data=True)
>> NodeDataView({1: {'category': 'I', 'occupation': 'politician'}, 3: {'category': 'D', 'occupation': 'celebrity'},...})_____no_output_____T.edges(data=True)
>> OutEdgeDataView([(1, 3, {'date': datetime.date(2012, 11, 16)}),...])_____no_output_____

* Undirected (Graph). Ex: Facebook

* Directed (DiGraph). Ex: twitter (follow)

* MultiGraph. Ex: trip records between bike sharing stations. Collapsed with "weight" for each edge

    - `network_name.edges[node1, node2]['attribute'] = value`
    - `'attribute'` = `'weight'`

* Self-loop. Nodes that are connected to themselves

    - `.number_of_selfloops()`_____no_output_____## Visualization

### 1. Matrix plots

Undirected (symmetric)
Directed (may not be symmetric)

### 2. Arc plots

Ordered axis (Ex: transportation)

### 3. Circos plots

**nxviz API** (Originally for genomics. All nodes form a circle).

A corresponding `nx.from_numpy_matrix(A)` allows one to quickly create a graph from a NumPy matrix. The default graph type is `Graph()`; if you want to make it a `DiGraph()`, that has to be specified using the `create_using` keyword argument, e.g. (`nx.from_numpy_matrix(A, create_using=nx.DiGraph`))._____no_output_____

# Matrix Plot

# Import nxviz
import nxviz as nv

# Create the MatrixPlot object: m
m = nv.MatrixPlot(T)

# Draw m to the screen
m.draw()

# Display the plot
plt.show()

# Convert T to a matrix format: A
A = nx.to_numpy_matrix(T)

# Convert A back to the NetworkX form as a directed graph: T_conv
T_conv = nx.from_numpy_matrix(A, create_using=nx.DiGraph())

# Check that the `category` metadata field is lost from each node
for n, d in T_conv.nodes(data=True):
    assert 'category' not in d.keys()_____no_output_____# Circos Plot

# Import necessary modules
import matplotlib.pyplot as plt
from nxviz import CircosPlot

# Create the CircosPlot object: c
c = CircosPlot(T)

# Draw c to the screen
c.draw()

# Display the plot
plt.show()_____no_output_____# Arc Plot

# Import necessary modules
import matplotlib.pyplot as plt
from nxviz import ArcPlot

# Create the un-customized ArcPlot object: a
a = ArcPlot(T)

# Draw a to the screen
a.draw()

# Display the plot
plt.show()

# Create the customized ArcPlot object: a2
a2 = ArcPlot(T,node_order='category',node_color='category')

# Draw a2 to the screen
a2.draw()

# Display the plot
plt.show()_____no_output_____

## Degree centrality

### 1. Degree centrality

Important nodes (more edges).

* The number of neighbors that a node has is called its "degree"
* The degree of a node is the number of neighbors that it has.

* Degree centrality =  of neighbors I have / \# of neighbors I could possibly have
Ex: twitter broadcasters, airport transportation hubs, disease super-spreaders

`G.neighbors(node)`
`nx.degree_centrality(G)`: {node: degree centrality}_____no_output_____

# Import matplotlib.pyplot
import matplotlib.pyplot as plt

# Compute the degree centrality of the Twitter network: deg_cent
deg_cent = nx.degree_centrality(T)

# Plot a histogram of the degree centrality distribution of the graph.
plt.figure()
plt.hist(list(deg_cent.values()))
plt.show()

# Plot a histogram of the degree distribution of the graph
plt.figure()
plt.hist(degrees)
plt.show()

# Plot a scatter plot of the centrality distribution and the degree distribution
plt.figure()
plt.scatter(x=degrees, y=list(deg_cent.values()))
plt.show()_____no_output_____

### Graph algorrithms

#### 1. Finding paths
* Optimization: shortest transport paths
* Modeling: disease spread, information passing

#### Breadth-first search (BFS)
Shorest path between two nodes

```
G
len(G.edges())
len(G.nodes())

G.neighbors(node)
G.neighbors(next neighbor)
```_____no_output_____### 2. Betweenness centrality

All shortest paths: a set of path where each path is the shortest path between a given pair of nodes.
Betweenness centrality =
\# of shortest paths through a node / all possible shortest paths existing for every pair of nodes
*Captures bottleneck node in a graph*
    - critical information transfer links
    - Barbell example:

```
G = nx.barbell_graph(m1=5, m2=1)
nx.betweenness_centrality(G)

```

Nodes located at the end of a barbell are fully connected to each other --> betweenness centrality =0
(There is no shortest path that has to run through these nodes in order to get to the other nodes)._____no_output_____## Comunities & cliques

* clique: a set of nodes that are completely connected by an edge to every other node \[completly connected graph\]
* simplest clique: an edge
* simplest complex clique: a triangle
    - NetworkX provides an API for counting the number of triangles that every node is involved in: `nx.triangles(G)`

Application: e.g. friend recommendation systems

```
G

from itertools import combinations
for n1, n2 comninations(G.nodes(),2):
    print(n1,n2)

```

### Maximal cliques
Definition: a clique that, when extended by one node is no longer a clique
Application: community finding (finding union of cliques)

```
nx.find_cliques(G)
list(nx.find_cliques(G))

```

### Subgraphs
Visualize portions of a large graph (path, communities/cliques, degrees of separation from a node)

```
G = nx.erdos_renyi_graph(n=20, p=0.2) #n=number of nodes, p=probability that an edge exists between a given pair of nodes

nodes = G.neighbors(8)
nodes.append(8)

G_eight = G.subgraph(nodes)
nx.draw(G_eight, with_labels=True)

```

#### Connected component subgraphs
A set of nodes connected to some nodes but not conneted to other nodes in the larger graph.
```
G = nx.errdos_renyi_graph(N=100, p=0.03)
list((nx.connected_component_subgraphs(G))

```_____no_output_____

# Extract the nodes of interest: nodes
nodes = [n for n, d in T.nodes(data=True) if d['occupation'] == 'celebrity']

# Create the set of nodes: nodeset
nodeset = set(nodes)

# Iterate over nodes
for n in nodes:

    # Compute the neighbors of n: nbrs
    nbrs = T.neighbors(n)

    # Compute the union of nodeset and nbrs: nodeset
    nodeset = nodeset.union(nbrs)

# Compute the subgraph using nodeset: T_sub
T_sub = T.subgraph(nodeset)

# Draw T_sub to the screen
nx.draw(T_sub)
plt.show()_____no_output_____