# Notebook from KyleHaggin/DS-Unit-4-Sprint-3-Deep-Learning
Path: module2-convolutional-neural-networks/LS_DS_432_Convolutional_Neural_Networks_Lecture.ipynb

Lambda School Data Science

*Unit 4, Sprint 3, Module 2*

---_____no_output_____# Convolutional Neural Networks (Prepare)

> Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers. *Goodfellow, et al.*_____no_output_____## Learning Objectives
- Part 1: Describe convolution and pooling
- Part 2: Apply a convolutional neural network to a classification task
- Part 3: Use a pre-trained convolution neural network for object detection

Modern __computer vision__ approaches rely heavily on convolutions as both a dimensinoality reduction and feature extraction method. Before we dive into convolutions, let's talk about some of the common computer vision applications:
* Classification [(Hot Dog or Not Dog)](https://www.youtube.com/watch?v=ACmydtFDTGs)
* Object Detection [(YOLO)](https://www.youtube.com/watch?v=MPU2HistivI)
* Pose Estimation [(PoseNet)](https://ai.googleblog.com/2019/08/on-device-real-time-hand-tracking-with.html)
* Facial Recognition [Emotion Detection](https://www.cbronline.com/wp-content/uploads/2018/05/Mona-lIsa-test-570x300.jpg)
* and *countless* more

We are going to focus on classification and pre-trained object detection today. What are some of the applications of object detection?_____no_output_____

from IPython.display import YouTubeVideo
YouTubeVideo('MPU2HistivI', width=600, height=400)_____no_output_____

# Convolution & Pooling (Learn)
_____no_output_____## Overview

Like neural networks themselves, CNNs are inspired by biology - specifically, the receptive fields of the visual cortex.

Put roughly, in a real brain the neurons in the visual cortex *specialize* to be receptive to certain regions, shapes, colors, orientations, and other common visual features. In a sense, the very structure of our cognitive system transforms raw visual input, and sends it to neurons that specialize in handling particular subsets of it.

CNNs imitate this approach by applying a convolution. A convolution is an operation on two functions that produces a third function, showing how one function modifies another. Convolutions have a [variety of nice mathematical properties](https://en.wikipedia.org/wiki/Convolution#Properties) - commutativity, associativity, distributivity, and more. Applying a convolution effectively transforms the "shape" of the input.

One common confusion - the term "convolution" is used to refer to both the process of computing the third (joint) function and the process of applying it. In our context, it's more useful to think of it as an application, again loosely analogous to the mapping from visual field to receptive areas of the cortex in a real animal._____no_output_____

from IPython.display import YouTubeVideo
YouTubeVideo('IOHayh06LJ4', width=600, height=400)_____no_output_____

## Follow Along

Let's try to do some convolutions in `Keras`._____no_output_____### Convolution - an example

Consider blurring an image - assume the image is represented as a matrix of numbers, where each number corresponds to the color value of a pixel._____no_output_____

import imageio
import matplotlib.pyplot as plt
from skimage import color, io
from skimage.exposure import rescale_intensity

austen = io.imread('https://dl.airtable.com/S1InFmIhQBypHBL0BICi_austen.jpg')
austen_grayscale = rescale_intensity(color.rgb2gray(austen))
austen_grayscale_____no_output_____plt.imshow(austen_grayscale, cmap="gray");_____no_output_____austen_grayscale.shape_____no_output_____import numpy as np
import scipy.ndimage as nd

horizontal_edge_convolution = np.array([[1,1,1,1,1],
                                     [0,0,0,0,0],
                                     [0,0,0,0,0],
                                     [0,0,0,0,0],
                                     [-1,-1,-1,-1,-1]])

vertical_edge_convolution = np.array([[1, 0, 0, 0, -1],
                                     [1, 0, 0, 0, -1],
                                     [1, 0, 0, 0, -1],
                                     [1, 0, 0, 0, -1],
                                     [1, 0, 0, 0, -1]])

austen_edges = nd.convolve(austen_grayscale, horizontal_edge_convolution)
austen_edges.shape_____no_output_____plt.imshow(austen_edges, cmap="gray");_____no_output_____

## Challenge

You will be expected to be able to describe convolution. _____no_output_____# CNNs for Classification (Learn)_____no_output_____## Overview_____no_output_____### Typical CNN Architecture

![A Typical CNN](https://upload.wikimedia.org/wikipedia/commons/thumb/6/63/Typical_cnn.png/800px-Typical_cnn.png)

The first stage of a CNN is, unsurprisingly, a convolution - specifically, a transformation that maps regions of the input image to neurons responsible for receiving them. The convolutional layer can be visualized as follows:

![Convolutional layer](https://upload.wikimedia.org/wikipedia/commons/6/68/Conv_layer.png)

The red represents the original input image, and the blue the neurons that correspond.

As shown in the first image, a CNN can have multiple rounds of convolutions, [downsampling](https://en.wikipedia.org/wiki/Downsampling_(signal_processing)) (a digital signal processing technique that effectively reduces the information by passing through a filter), and then eventually a fully connected neural network and output layer. Typical output layers for a CNN would be oriented towards classification or detection problems - e.g. "does this picture contain a cat, a dog, or some other animal?"

Why are CNNs so popular?
1. Compared to prior image learning techniques, they require relatively little image preprocessing (cropping/centering, normalizing, etc.)
2. Relatedly, they are *robust* to all sorts of common problems in images (shifts, lighting, etc.)

Actually training a cutting edge image classification CNN is nontrivial computationally - the good news is, with transfer learning, we can get one "off-the-shelf"!_____no_output_____## Follow Along_____no_output_____

from tensorflow.keras import datasets
from tensorflow.keras.models import Sequential, Model # <- May Use
from tensorflow.keras.layers import Dense, Conv2D, MaxPooling2D, Flatten_____no_output_____(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()

# Normalize pixel values to be between 0 and 1
train_images, test_images = train_images / 255.0, test_images / 255.0_____no_output_____class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck']

plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(train_images[i], cmap=plt.cm.binary)
    # The CIFAR labels happen to be arrays,
    # which is why you need the extra index
    plt.xlabel(class_names[train_labels[i][0]])
plt.show()_____no_output_____train_images[0].shape_____no_output_____32*32*3_____no_output_____# Setup Architecture

model = Sequential()
model.add(Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(MaxPooling2D((2,2)))
model.add(Conv2D(64, (3,3), activation='relu'))
model.add(MaxPooling2D((2,2)))
model.add(Conv2D(64, (3,3), activation='relu'))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dense(10, activation='softmax'))

model.summary()_____no_output_____# Compile Model

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])_____no_output_____# Fit Model
model.fit(train_images, train_labels, epochs=10, validation_data=(test_images, test_labels))_____no_output_____# Evaluate Model

test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)_____no_output_____

## Challenge

You will apply CNNs to a classification task in the module project._____no_output_____# CNNs for Object Detection (Learn)_____no_output_____## Overview_____no_output_____### Transfer Learning - TensorFlow Hub

"A library for reusable machine learning modules"

This lets you quickly take advantage of a model that was trained with thousands of GPU hours. It also enables transfer learning - reusing a part of a trained model (called a module) that includes weights and assets, but also training the overall model some yourself with your own data. The advantages are fairly clear - you can use less training data, have faster training, and have a model that generalizes better.

https://www.tensorflow.org/hub/

**WARNING** - Dragons ahead!

![Dragon](https://upload.wikimedia.org/wikipedia/commons/thumb/d/d8/Friedrich-Johann-Justin-Bertuch_Mythical-Creature-Dragon_1806.jpg/637px-Friedrich-Johann-Justin-Bertuch_Mythical-Creature-Dragon_1806.jpg)

TensorFlow Hub is very bleeding edge, and while there's a good amount of documentation out there, it's not always updated or consistent. You'll have to use your problem-solving skills if you want to use it!_____no_output_____## Follow Along_____no_output_____

import numpy as np

from tensorflow.keras.applications.resnet50 import ResNet50
from tensorflow.keras.preprocessing import image
from tensorflow.keras.applications.resnet50 import preprocess_input, decode_predictions

def process_img_path(img_path):
  return image.load_img(img_path, target_size=(224, 224))

def img_contains_banana(img):
  x = image.img_to_array(img)
  x = np.expand_dims(x, axis=0)
  x = preprocess_input(x)
  model = ResNet50(weights='imagenet')
  features = model.predict(x)
  results = decode_predictions(features, top=3)[0]
  print(results)
  for entry in results:
    if entry[1] == 'banana':
      return entry[2]
  return 0.0_____no_output_____import requests

image_urls = ["https://github.com/LambdaSchool/ML-YouOnlyLookOnce/raw/master/sample_data/negative_examples/example11.jpeg",
              "https://github.com/LambdaSchool/ML-YouOnlyLookOnce/raw/master/sample_data/positive_examples/example0.jpeg"]

for _id,img in enumerate(image_urls):
    r = requests.get(img)
    with open(f'example{_id}.jpg', 'wb') as f:
        f.write(r.content)_____no_output_____from IPython.display import Image
Image(filename='./example0.jpg', width=600)_____no_output_____img_contains_banana(process_img_path('example0.jpg'))_____no_output_____Image(filename='example1.jpg', width=600)_____no_output_____img_contains_banana(process_img_path('example1.jpg'))_____no_output_____

Notice that, while it gets it right, the confidence for the banana image is fairly low. That's because so much of the image is "not-banana"! How can this be improved? Bounding boxes to center on items of interest._____no_output_____## Challenge

You will be expected to apply a pretrained model to a classificaiton problem today. _____no_output_____# Review

- Part 1: Describe convolution and pooling
    * A Convolution is a function applied to another function to produce a third function
    * Convolutional Kernels are typically 'learned' during the process of training a Convolution Neural Network
    * Pooling is a dimensionality reduction technique that uses either Max or Average of a feature map region to downsample data
- Part 2: Apply a convolutional neural network to a classification task
    * Keras has layers for convolutions :)
- Part 3: Use a pre-trained convolution neural network for object detection
    * Check out both pretinaed models available in Keras & TensorFlow Hub_____no_output_____# Sources

- *_Deep Learning_*. Goodfellow *et al.*
- [Keras CNN Tutorial](https://www.tensorflow.org/tutorials/images/cnn)
- [Tensorflow + Keras](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D)
- [Convolution Wiki](https://en.wikipedia.org/wiki/Convolution)
- [Keras Conv2D: Working with CNN 2D Convolutions in Keras](https://missinglink.ai/guides/keras/keras-conv2d-working-cnn-2d-convolutions-keras/)_____no_output_____


# Notebook from sashakarimi/ML-Hult-Summer-2019
Path: Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb

# A Systematic Approach to Visualizing Data
## Exploring a Telecom Customer Churn Dataset_____no_output_____TO DO
- Nothing so far._____no_output_____Acknowlegements
- Thanks to David Wihl for fixing a plotting error._____no_output_____## Introduction

In this notebook we'll explore a dataset containing information about a telecom company's customers. It comes from an [IBM Watson repository](https://www.ibm.com/communities/analytics/watson-analytics-blog/predictive-insights-in-the-telco-customer-churn-data-set/).

We've chosen this dataset because it is complicated enough to teach us things but not so complicated that it sidetracks us. It also gives us an opportunity to reason about a type of business problem that you will encounter (if you haven't already).

The dataset comes to us in the form of an Excel file. The name of the file is "WA_FN-UseC_-Telcom-Customer-Churn.xlsx". It's quite a cumbersome name, but we'll stick with it so we'll always know where it came from (you can quickly google the name in a pinch). And in general datasets come to us with various names and it's better to get to used to that right from the start.

We get data in two ways -- by creating it or by getting it from somewhere else. Once we get data, data scientists spend a surprising amount of time getting their heads wrapped around the data, cleaning it, and preparing it to be analyzed. We'll illustrate the main steps of this process here.

The objective of this notebook is to provide a template for data exploration -- the first and perhaps most critical step in any kind of data analysis project including machine learning.

There are two main reasons for visualizing data:
- To know what you have
- To get ideas for how to investigate the data

Missing values, data attributes with values that have different orders of magnitude, or skewed/unrepresentative data can all affect the quality of what you can infer from the data.

So how should data be visualized? There is no set recipe. Here are some guidelines._____no_output_____## 1. Get the Data into Manipulable Form

First thing to do is to load up the data. This depends on the source of the input file -- it could come from a file on the local computer system or from a remote location like an Amazon AWS S3 bucket. For us, the file is already stored locally in the Data folder.

**NOTE:** We'll use some standard Python packages to load, manipulate, and visualize the data. These packages are tools that make our lives as data scientists a lot easier and less tedious. Packages are loaded using the "import" keyword or the "from A import B" or the "from A import B as C" locutions._____no_output_____

# We keep plotting simple and use common packages and defaults
import matplotlib.pyplot as plt
import seaborn as sns
# Set the aesthetics for Seaborn visuals
sns.set(context='notebook',
        style='whitegrid',
        palette='deep',
        font='sans-serif',
        font_scale=1.3,
        color_codes=True,
        rc=None)
%matplotlib inline_____no_output_____import os # to navigate the file system
import numpy as np # for number crunching
import pandas as pd # for data loading and manipulation

# OS-independent way to navigate the file system
# One directory up in relation to directory of this notebook
new_dir = os.path.normpath(os.getcwd() + os.sep + os.pardir)
# Where the file is
file_url = new_dir + os.sep + "Data" + os.sep + "WA_Fn-UseC_-Telco-Customer-Churn.xlsx"
file_url_____no_output_____# Read the excel sheet into a pandas dataframe
df_churn = pd.read_excel(file_url, sheetname=0)_____no_output_____

The data is not in a form that can be manipulated for exploration and visualization._____no_output_____## 2. Get a Handle on the Structure of the Data_____no_output_____### 2a. Number of Rows and Columns_____no_output_____

# Look at the first few lines of the data -- scroll to the right to see more columns
df_churn.head()_____no_output_____

How many rows (# of customers) and how many columns (# attritbutes of each customer) do we have?_____no_output_____

# Number of rows and columns
num_rows, num_cols = df_churn.shape
num_rows, num_cols_____no_output_____

This meams we have 7,043 customers with each customer tagged with 21 individual attributes such as customerID, gender, SeniorCitizen, etc. Let's get a complete list of the attributes._____no_output_____### 2b. An Overview of Customer Features (or Attributes)

Here is a complete list of features (also known as attributes) that describe each of the 7,043 customers. We don't know yet if all customers are tagged with all 21 features -- we'll find out soon._____no_output_____

# Here is a list of the features with the first 5 values of each feature.
feature_list = list(df_churn)
# First 5 values of each feature in the list
first_5 = [list(df_churn[attribute][0:5]) for attribute in feature_list]
list(zip(feature_list, first_5))_____no_output_____

Some of the features have a discrete set of possible values (e.g., gender, PaymentMethod) while some others can take a range of values that need not be discrete (e.g., tenure, MonthlyCharges, TotalCharges)._____no_output_____### 2c. Categorical Feature and Their Possible Values
Some of the features in our dataset are *categorical* -- their values come from a small handful of discrete possibilities. Features like gender and payment method fit are categorical. Categorical features are also known as *discrete* features. Let's separate the discrete/categorical features from the rest -- we'll get a better grip if we look at them separately first._____no_output_____

# Identify all the features that are categorical

# Feature index numbers that are *not* categorical.
# Just count from the dataset starting at customerID's index = 0
not_categorical = [0,5,18,19] # CustomerID is not a feature but a unique identifier
# the categorical features are then the complement of the above list
categorical = list(set(range(df_churn.shape[1])) - set(not_categorical))
# get the unique values of the categorical features
[[feature_list[feature_index], list(df_churn.iloc[:, feature_index].unique())] \
    for feature_index in categorical]_____no_output_____

### 2d. Numerical Features

Three of the features in this dataset are numerical:
- tenure
- MonthlyCharges
- TotalCharges _____no_output_____

# Put the numerical features into a list for subsquent use
numeric_features = [feature_list[n] for n in not_categorical][1:] # CustomerID is not a numeric feature
numeric_features_____no_output_____

Notice that the SeniorCitizen attribute or feature is respresented numerically as a 1 or 0 -- but these numbers actually represent "Yes" or "No". In othere words, SeniorCitizen is a categorical attribute._____no_output_____## 3. Visualize the Numerical Features_____no_output_____### SIDEBAR

Let's build ourselves a handy way to look at any set of attributes we choose. We can use this to isolate and explore various groups of attributes._____no_output_____

# View a selection of rows and columns in the df_churn dataframe
df_map = {'telco churn data': df_churn}
# We're calling our dataset 'telco churn data'
def table_view(data_frame_name, feature_list, start_row=3, end_row=5):
    '''
    Displays selected columns and rows of a data frame.
    '''
    # Verify the inputs are sane
    # get the size of the dataframe
    num_rows, num_cols = df_map[data_frame_name].shape
    if (start_row < 0) | (start_row > num_rows) :
        return print("Please use a valid Start Row number. It can be any number from 0 to {}".format(num_rows))
    if (end_row < 0) | (end_row > num_rows) | (end_row < start_row + 1):
        return print("Please use a valid End Row number. \
                     It can be any number from 0 to {} \
                     and must be greater than your start row number".format(num_rows))

    view = df_map[data_frame_name][feature_list].iloc[start_row:end_row]

    return view_____no_output_____# Using the table_view function defined above
# The name of our dataset (as defined above) is 'telco churn data'
# You can select any and any number of attributes in selected_columns
selected_columns = ['tenure', 'SeniorCitizen', 'MonthlyCharges', 'TotalCharges']
table_view('telco churn data', selected_columns, 10, 14)_____no_output_____# We can  explore views in an interactive way
from __future__ import print_function
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
from ipywidgets import Button, HBox, VBox
from IPython.display import display
from IPython.display import clear_output_____no_output_____# Layout the interactive view widgets

# Data Frame Chooser Dropdown
dataFrame = widgets.Select(
             options=['telco churn data'],
             value='telco churn data',
             description='Data Source:',
             disabled=False
            )

# Start Row Text Field
startRow = widgets.IntText(
            value=7,
            description='Start Row:',
            disabled=False
           )

# End Row Text Field
endRow = widgets.IntText(
           value=12,
           description='End Row:',
           disabled=False
          )

# Attribute Selector (Multiple Select)
allFeatures = widgets.SelectMultiple(
               options = feature_list,
               rows = 20,
               description = 'Select Mulitple Features:',
               value = ['gender', 'SeniorCitizen', 'tenure', 'MonthlyCharges']
              )

# Button
button = widgets.Button(
          description='Show View',
          disabled=False,
          button_style='info', # 'success', 'info', 'warning', 'danger' or ''
          tooltip='Go!',
          icon=''
         )

def on_button_clicked(b):
    # Pass the values of the widgets to the table_view function
    clear_output()
    return print(table_view(dataFrame.value, list(allFeatures.value), startRow.value, endRow.value))

button.on_click(on_button_clicked)_____no_output_____

### A Simple Display of Slices of the Dataset_____no_output_____

# Display the elements
HBox([VBox([dataFrame, allFeatures, button]), VBox([startRow, endRow])])_____no_output_____

### Summary Statistics_____no_output_____

# Here are summary statistics - rough format but still useful
#df_churn['TotalCharges'].astype(float)
df_churn[['tenure', 'MonthlyCharges', 'TotalCharges']].describe(include='all')_____no_output_____

### Box Plots_____no_output_____

fig, (ax1,ax2,ax3) = plt.subplots(nrows=1, ncols=3, figsize=(12,4))
sns.boxplot(x=df_churn['tenure'], ax = ax1, palette='Set1')
#ax1.set_title("Tenure")
sns.boxplot(x=df_churn['MonthlyCharges'], ax = ax2, palette='Set2')
#ax2.set_title("Monthly Charge")
# There are some monthly charges missing -- supress for now because we haven't yet handled missing values
#sns.boxplot(x=df_churn['TotalCharges'], ax = ax3, palette='Set3')
#ax3.set_title("Total Charge")
plt.tight_layout()_____no_output_____

### Histogram_____no_output_____

# Set up the plot
def plotFeatureHist(feature_name):
    fig, ax = plt.subplots(figsize=(12,7))
    sns.distplot(df_churn[feature_name], kde=False)
    return plt.show()_____no_output_____# How are the monthly charges distributed?
plotFeatureHist('MonthlyCharges')_____no_output_____

## 4. Visualize Relationships Between Numerical Features_____no_output_____

# How are tenure and monthly charges related?
g = sns.JointGrid(x="MonthlyCharges", y="tenure", data=df_churn)
g = g.plot(sns.regplot, sns.distplot)_____no_output_____# Pairwise scatter plots of the numerical attributes
cols_numeric = ['tenure', 'MonthlyCharges', 'TotalCharges']
sns.set(style='whitegrid', context='notebook')
sns.pairplot(df_churn[cols_numeric], size=3.5)_____no_output_____

### How are the numerical attributes correlated?_____no_output_____

# Calculate the correlation table
# Not sure why SeniorCitizen appears but TotalCharges doesn't appear.
corr = df_churn.corr()
corr_____no_output_____# Correlation Density Plot
feature_display_names = ['Tenure', 'Monthly Charges', 'Total Charges']
cm = df_churn.corr()
sns.set(font_scale=1)
# NOTE: fmt directive controls number of decimal points displayed
hm = sns.heatmap(cm,
                cbar=True,
                annot=True,
                square=False,
                fmt='.2f',
                annot_kws={'size':14},
                yticklabels=feature_display_names,
                xticklabels=feature_display_names)
plt.title('Correlation of Numerical Features')_____no_output_____

## 5. Visualize the Categorical Features_____no_output_____

# Remind ourselves of the categorical features in the dataset
[feature_list[feature_index] for feature_index in categorical]_____no_output_____

### How Balanced are the Categorical Features?

How are the categorical features of the dataset distributed? For example, are there very few senior citizens? Is it the case that an overwhelming number of customers in the dataset have no dependents? Understanding how the features are balanced will give us a sense of how *generalizable* the results obtained from the dataset will be._____no_output_____### SIDEBAR_____no_output_____

# Set up the plot
def plotFeatureCount(feature_name, count_flag):
    fig, ax = plt.subplots(figsize=(12,7))

    if count_flag == 'Count':
        ax = sns.countplot(x=feature_name, data=df_churn)
    elif count_flag == 'Percentage':
        x = df_churn[feature_name].unique()
        y = [len([val for val in df_churn[feature_name] if val == x_val])/len(df_churn[feature_name]) * 100 \
             for x_val in x]
        ax = sns.barplot(x,y)

    plt.ylabel(count_flag)
    return plt.show()_____no_output_____# set up the plot for interactivity
# Dropdown
w_features = widgets.Dropdown(
               options = [feature_list[feature_index] for feature_index in categorical],
               description = 'Select Feature:',
               value = 'gender',
               button_style='info'
              )

w_radio = widgets.RadioButtons(
    options=['Count', 'Percentage'],
    value='Count',
    description='Display:',
    disabled=False
)

def on_value_change(change):
    # Pass the value of the dropdown to the plotFeatureCount function
    clear_output()
    return plotFeatureCount(w_features.value, w_radio.value)

w_features.observe(on_value_change)
w_radio.observe(on_value_change)_____no_output_____

### Display Categorical Features_____no_output_____

# Show a default plot
plotFeatureCount(w_features.value, w_radio.value)
# Show the widgets
HBox([w_features, w_radio])_____no_output_____

## EXERCISE 1

How balanced are the categorical features? Do you anticipate any problems using this dataset to predict if a customer will switch (be subject to churn) or not?_____no_output_____## 6. Visualize Relationships Between Categorical Features_____no_output_____### Does gender make a difference for churn?_____no_output_____

# One way to visualize the effect of gender on churn -- i.e. interdependence between variables
from plotnine import *
(ggplot(df_churn, aes(x='Churn', fill='gender')) + geom_bar(position='fill'))/anaconda/lib/python3.6/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
  from pandas.core import datetools

### Does a factor in addition to gender affect churn?_____no_output_____

# Does the streaming movies along with gender affect churn?
from plotnine import *
(ggplot(df_churn, aes(x='Churn', fill='gender')) + geom_bar(position='fill') + facet_wrap('~StreamingMovies'))_____no_output_____

## 7. Visualize Relationships Between Numerical and Categorical Attributes_____no_output_____### Does tenure affect churn?_____no_output_____

# Density Plot
sns.kdeplot(df_churn.query("Churn == 'No'").tenure, shade=True, alpha=0.2, label='No', color='salmon')
sns.kdeplot(df_churn.query("Churn == 'Yes'").tenure, shade=True, alpha=0.2, label='Yes', color='dodgerblue')
plt.title('The first 20 months are critical')
plt.xlabel('Tenure')_____no_output_____

### Does tenure in addition to gender affect churn?_____no_output_____

# Facet Plots
g = sns.FacetGrid(df_churn, row='Churn', col='gender', margin_titles=True)
g.map(sns.distplot, 'tenure')_____no_output_____

### Do senior citizens pay more per month?_____no_output_____

# Pivots
# Do monthly charges depend on gender? They don't seem to; but they do seem to depend on whether
# or not the person is a senior citizen -- 0 = Not a senior citizen, 1 = senior citizen
fig, ax = plt.subplots(figsize=(10,7))
sns.boxplot(x='gender', y='MonthlyCharges', hue="SeniorCitizen", data=df_churn, palette='Set3')
plt.legend(loc='upper right')_____no_output_____

### Do monthly charges depend on the lenght of the contract?_____no_output_____

# Jitter plot
# Do monthly charges depend on the length of contract?
from plotnine import *
(ggplot(df_churn, aes(x='Contract', y='MonthlyCharges')) + geom_jitter(position=position_jitter(0.4)))_____no_output_____

### Does having online backup service increase monthly charges?_____no_output_____

# Jitter plot
# How does online backup service affect monthly charges?
from plotnine import *
(ggplot(df_churn, aes(x='OnlineBackup', y='MonthlyCharges')) + geom_jitter(position=position_jitter(0.4)))_____no_output_____

## 8. Find and Handle Missing Values_____no_output_____

# For each column in the dataset, add up the rows in which the column data is missing
# You can do this across the entire dataset for a quick look at what's missing
df_churn.isnull().sum()_____no_output_____

The zeros mean there are no missing data values. This is very nice, but unusual in most datasets, so we've lucked out. When there are missing values it takes effort and judgement to decide how to handle them. Sometimes it's not clear how to handle missing data even though there are a number of standard techniques to choose from.

One thing to watch out for is a value that seems to be missing, except that it really is an empty string like '' or a string with some spaces such as '   '. These usually trip up the plotting functions and that's one (stressful) way to identify them. _____no_output_____

# For each of the features, find rows where they might be empty -- we'll have to handle these appropriately
def isEmpty(feature):
    empty_rows = []
    for i in range(len(df_churn)):
        if isinstance(df_churn[feature][i], str):
            empty_rows.append(i)

    return empty_rows_____no_output_____# Which of the numerical features have no numerical values?
empty = [[feature,isEmpty(feature)] for feature in numeric_features]
empty_____no_output_____# Let's have a look at these rows where one or more numberical features are empty.
df_churn.iloc[empty[2][1]]_____no_output_____

It turns out that TotalCharges are empty when tenure = 0. These rows do have a monthly charge. We'll just make the total charge equal the monthly charge in these cases._____no_output_____

df_churn.iloc[488]['MonthlyCharges']_____no_output_____# Get the monthly charges for these rows
monthly_charges = [df_churn.iloc[loc]['MonthlyCharges'] for loc in empty[2][1]]
monthly_charges
#df_churn.set_value(488, 'TotalCharges', 52.555)
#df_churn.iloc[488]_____no_output_____# For all customers whose tenure is 0 months, set the TotalCharges equal to the MonthlyCharges
# This is the new dataset
[df_churn.set_value(loc, 'TotalCharges', df_churn.iloc[loc]['MonthlyCharges']) for loc in empty[2][1]]
df_churn.shape_____no_output_____# Let's check if the values of TotalCharges are as they should be.
df_churn.iloc[empty[2][1]][['MonthlyCharges','TotalCharges']]_____no_output_____

## Exercise 2

What other checks on the data would you do to catch weird values appearing in the data?_____no_output_____## Summary

Data exploration and visualization are useful first steps in machine learning because they help give us ideas for how to investigate the data as well as prepare the data so that its suitable for machine learning. Some important activities include:

- Finding and handling missing data
- Identifying skewed features that require appropriate transformation
- Ensuring that the data is representative (which has implications for how should use the data and the kinds of inferences we can plausibly make from it)
- Seeing if features should be combined to improve prediction
- Normalizing or scaling features if feature values range widely (e.g., one features has values in the range 1 to 10 while another feature has values in the range 100,000 to 1,000,000).

To systematicaly explore the dataset, take the following steps:

1. Get the data into manipulable form.
2. Get a handle on the structure of the data.
3. Visualize the numerical features.
4. Visualize the relationships between numerical features.
5. Visualize the categorical features.
6. Visualize the relationships between categorical features.
7. Visualize the relationships between numerical and categorical features.
8. Find and handle missing values.

When there are too many features (columns) in the dataset to apply the methods above, there are other techiques that come into play both for visualization and for analysis. These techniques allow us to rank the importance of features so we can prioritize the ones to scrutinize. We'll get to these techiques when we study feature engineering later in the course._____no_output_____


# Notebook from ccnss/ccnss2017_students
Path: module4/Tutorial_2_SSMs_Kalman.ipynb

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm_____no_output_____

# Tutorial 2: State space models and filtering

## Introduction

In this tutorial we will generate data from a linear-Gaussian state space model (LG-SSM), and solve it by different exact and approximate methods. Then we'll introduce a hierarchical model (a "Double Kalman Filter"), a model that was used a.o. in Behrens et al. 2007 to model how people assess volatility in their environment and adjust reward-based decisions accordingly. You've seen this paper referenced in several talks during the summer school (some of you even collected data using a similar model), and this tutorial will give you an opportunity to learn to fit the data using the actual (or assumed) generative model.

SSMs are a continuous version of Hidden Markov Models, sharing the same graphical model structure.

This class of models has many practical applications in modeling sequential temporal events (e.g. the location of a moving object in tracking applications such as GPU), as well as modeling other sequential data e.g. sequences of words in language modeling, or genes in models for computational biology. In neuroscience it has been used to model navigation, motor control, reward learning, and a range of other phenomena.

We will try to make inferences about the hidden state variables $z_{1:t}$, given the noisy observations $x_{i:t}$, a process called state estimation. We will assume that both the transition model $p(z_t |z_{t-1})$ and the observation model $p(x_t |z_{t})$ are known, and Gaussian.

\begin{eqnarray}
z_t=A \cdot z_{t-1}+\epsilon \\
x_t=B \cdot z_{t}+\delta\\
z_1 \sim \mathcal{N}(0, \sigma_1^{2}) \\
\epsilon \sim \mathcal{N}(0, \sigma_z^{2}). \\
\delta \sim \mathcal{N}(0, \sigma_x^{2}).
\end{eqnarray}

_____no_output_____## Exercise 1
Write a function to generate data from a 1D LG-SSM (where both z and x are scalars). The function should take the standard deviations $\sigma_z$ and $\sigma_x$ as inputs, as well as the number of steps,and a prior mean for $z_1$. It should return a vector of $z$-s and $x$-s. For simplicity, we will assume that the prior and transition variances are the same, $\sigma_1^{2}=\sigma_z^{2}$. Throughout the tutorial we will take A and B to be just 1.

Test the function by generating data from a SSM with $\sigma_z=0.3$, $\sigma_x=1.2$  and  $E(Z)=0$ for 100 timesteps, and make a plot showing the evolution of z and x over time, with the observations $x_{1:t}$ scattered around the line $z_{1:t}$.

_____no_output_____
We will now try to estimate the true hidden sequence z, given the noisy observations x, using Bayesian inference. We will do this online, estimating $z_t$ for every time step t, given the observations available at that time, $x_{1:t}$. a process called filtering. It is also possible to estimate the most likely sequence $z_t$ given all the observations (a method called smoothing), but we will leave this for now.

In our first approach, we will proceed by discretizing the domains of Z and X, and try to represent the posterior probability distribution over z by a series of probability masses over these discrete values.

## Exercise 2
First, derive an update rule for the probability distribution $p(z_t)$, given the posterior from the previous step, $p(z_{t-1})$, and the observation at time t, $x_t$.

Then write a function that performs one step of the filtering, taking as input the previous posterior $p(z_{t-1})$, the observation $x_t$, as well as $\sigma_z$ and $\sigma_x$, and returning the posterior $p(z)$, and a maximum likelihood estimate for $z_t$ (such as the MAP estimate, or posterior mean). Probability distributions should be represented by a vector of point masses over the discretized range. The upper and lower bounds of the descritization can be declared  as global variables with values 10 and -10 respectively._____no_output_____Finally write a function that iterates over all the observations, performing the whole filtering process. Then call this function, and try to estimate the hidden states for the process you generated before, using the observations. Be careful to match the prior for $z_1$ in the estimation process to the one used in the generative model. Experiment with different discretizations, to see how many points you have to use in the interval to get a good result and produce some plots comparing your estimates for $z_t$ with the ground truth. Compare your state estimates to that obtained if you simply averaged over the observations with a sliding time window.

_____no_output_____The above approach is not feasible in general, in particular where z and x are high dimensional variables. The number of points needed to tile the respective domains grows exponentially with the dimensionality of these variables, making the computational cost of this approach prohibitive, and the accuracy insufficient. This is an issue we already encountered with PPCs.

Luckily, in the linear-Gaussian case it is possible to calculate the exact posterior for $z_t$, since the Gaussian distribution is self-conjugate. This means that the posterior over $z_t$ will always be Gaussian, as long as the prior for z and the generative model are given by Gaussians. We will also implement this using our 1d example, with $A=B=1$, in which case the equations for the solution are:
\begin{eqnarray}
\\
\mu_{z_t}=\mu_{z_{t-1}}+K_t*(x_t-\mu_{z_{t-1}})\\
\\
\sigma_{z_t}^2=\sigma_x^2*K_t\\
\\
K_t=\frac{\sigma_z^2+\sigma_{z_{t-1}}^2}{\sigma_z^2+\sigma_{z_{t-1}}^2+\sigma_y^2}
\end{eqnarray}

where $K_t$ is the Kalman gain at time t. This first equation is similar to error driven learning rules you've already seen, except instead of a learning rate we now use the Kalman gain, to trade off uncertainty about the measurement and the observation in an optimal way. This method is known as the Kalman filter.

## Exercise 3

Write a function that computes one step of the analytical solution to the Kalman filter. The inputs should be the prior mean and variance of z, an observation, and the known standard deviations of the generative process as before, and the function should output the posterior mean and variance.

Then write a function that performs the filtering over the whole sequence of observations, and finally estimate the sequence $z_t$ for the process you used in the previous exercises. Compare your solutions to those from Ex.2

_____no_output_____As mentioned above, the exact approach requires all the distributions specifying the generative model to be Gaussian, in which case the posterior is also Gaussian, and fully specified by a mean and variance. When this is not the case, alternative approximate approaches may still be used. We implemented a naive, brute force approach in Ex.1. This time around we'll do something a bit smarter, and implement a sampling based method called particle filtering. Instead of specifying whole distributions, we will take a sample (particle) from the prior, follow it around as it evolves during using the generative model, and specify the new posterior using the resulting particles and weights.

## Exercise 4

Write a function that implements one step of the particle filter, with proposal distribution $q(z_t\rvert z^s_{t-1},y_t)=p(z_t\rvert z_{t-1})$, the prior over $z_t$ given $z_{t-1}$. Apart from the usual stuff, it should take as input the vector of particles, and their weights, and output the new particles, transitioned under the dynamics model, and the new weights, as weighted by the observation model. (You could also just pass the particles if you perform the resampling at the end of the step, as in the second figure and the pseudocode below).

Then, as before, write a function that loops through the whole sequence of observations, and finally test it on the SSMs you generated. Test how many particles you need to get good results.

| Pass weights, sample at the beginning of iteration| or | resample at the end of each iteration            |
|:------------------------------------:|  |:-----------------------------:|
|  |  |  |

_____no_output_____## Exercise 5
### Double Kalman Filter.

One possible extension to our original generative model is to account for the case when the variance in the transition dynamics is not constant. Many variables in the environment, and in the brain, can go through different phases of volatility. A simple way to model this is to have the variance of the transition model itself vary from timestep to timestep. In the double KF we implement this variance itself is the output of a hidden LG-SSM. This hierarchical model is no longer analytically solvable, however we can still implement our two approximations from above. As mentioned in the intro, a similar model was used by Behrens and colleagues to model how people make inferences about the reward structure in the environment, to adjust their learning rate (the equivalent of the Kalman Gain in Ex.3).
The new generative model is given by:

\begin{eqnarray}
z_t=z_{t-1}+\epsilon \\
y_t=y_{t-1}+\mathcal{N}(0,max[z_t,0])\\
x_t=y_{t}+\delta\\
z_1 \sim \mathcal{N}(0, \sigma_1^{2}) \\
\epsilon \sim \mathcal{N}(0, \sigma_z^{2}). \\
\delta \sim \mathcal{N}(0, \sigma_x^{2}).
\end{eqnarray}

Change your solutions to the previous exercises to generate a process form this model, and solve it using the approximate methods we used before. Use $sigma_z=0.5$, $sigma_x=10$, and $E(Z)=5$. It should be mentioned that neither of these solution methods is particularly elegant or effective once the dimensionality of that state space starts increasing, as in this example. For an easier problem (for the algorithm to solve, not necessarily to implement), you could go directly to implementing a solution to the Behrens et al. example, which is an optional exercise below.

To implement the discretization solution, you will have to calculate an approximation to the joint distribution of y and z, calculating probability masses for a matrix. Use a small discretization parameter(e.g. 100), as otherwise the algorithm will take too long to run. Plot the joint posterior over y and z as the process evolves using a heatmap. Us 0.1 and 10 as bounds for z, and -50 and 50 as bounds for y.
_____no_output_____

computational biology
evolution
biology
neuroscience


# Notebook from Hoseung/pyRamAn
Path: scripts/Rotation/Untitled.ipynb

<code>
"""
1.
This script plots co-evolution history of galaxy stellar mass growth and
lambda_r evolution. Considering abrupt stellar mass growth as a consequence of
galaxy merger, the role of merger history on lambda_r might be understood.

2.
This script loads lambda_mp.py output catalogs (.pickle).

3.
For the moment, only galaxies with full halo merger history are stored in catalogs.

2015/08/18

Multiple yaxis

@author: hoseung
"""

import pickle
import numpy as np

wdir = '/home/hoseung/Work/data/05427/'

nout_fi = 187
nout_ini = 37
nouts = np.arange(nout_ini, nout_fi + 1)
nnouts = len(nouts)

def load_pickle(fname):
    with open(fname, 'rb') as f:
        return pickle.load(f)_____no_output_____wdir = '/home/hoseung/Work/data/36413/'_____no_output_____# Final galaxies
# Why set..?
#final_gals = set()
for inout, nout in enumerate(reversed(np.arange(nout_ini, nout_fi + 1))):
    #cat = load_pickle(wdir + 'catalog_GM/' + 'catalog' + str(nout) + '.pickle')
    cat = pickle.load(open(wdir + 'catalog_GM/' + 'catalog' + str(nout) + '.pickle', 'rb'))
    #final_gals.update(cat['final'])
    #print(cat)
    final_gals = cat['idx']

ngals = len(final_gals)
mstar = np.zeros((ngals, nnouts))
l_r = np.zeros((ngals, nnouts))
reff = np.zeros((ngals, nnouts))
fg = np.zeros((ngals, nnouts), dtype=int)
print(ngals)
final_gals = np.sort(list(final_gals))

#%%#######################################################################
# Read catalogs and extract mstar and lambda-r of galaxies at all nouts.
for inout, nout in enumerate(reversed(np.arange(nout_ini, nout_fi + 1))):
    cat = load_pickle(wdir + 'catalog_GM/' + 'catalog' + str(nout) + '.pickle')
    for igal, idgal in enumerate(cat['idx']):
        #ind = which galaxy..? -> need tree. (save the ID of final halo too)
        ind = np.where(idgal == final_gals)[0]
#        print(ind, final_gals[ind], inout, nout, idgal, cat['id'][igal])
        if len(ind) > 0 :
            fg[ind,inout] = final_gals[ind]
            mstar[ind,inout] = cat['mstar'][igal]
            l_r[ind,inout] = cat['lambda_r'][igal]
            reff[ind,inout] = cat['rgal'][igal]

#%%
zreds=[]
aexps=[]
import load
for nout in nouts:
    info = load.info.Info(nout=nout, base=wdir, load=True)
    aexps.append(info.aexp)
    zreds.append(info.zred)
aexps = np.array(aexps)
zreds = np.array(zreds)

#%%
def aexp2zred(aexp):
    return [1.0/a - 1.0 for a in aexp]

def zred2aexp(zred):
    return [1.0/(1.0 + z) for z in zred]

def lbt2aexp(lts):
    import astropy.units as u
    from astropy.cosmology import WMAP7, z_at_value
    zreds = [z_at_value(WMAP7.lookback_time, ll * u.Gyr) for ll in lts]
    return [1.0/(1+z) for z in zreds]

# For a given list of nouts,
# calculate a nice-looking set of zreds.
# AND lookback times
z_targets=[0, 0.2, 0.5, 1, 2, 3]
z_target_str=["{:.2f}".format(z) for z in z_targets]
a_targets_z = zred2aexp(z_targets)
z_pos =  [nout_ini + (1 - (max(aexps) - a)/aexps.ptp()) * nnouts for a in a_targets_z]

lbt_targets=[0.00001,1,3,5,8,12]
lbt_target_str=["{:.0f}".format(l) for l in lbt_targets]
a_targets_lbt = lbt2aexp(lbt_targets)
lbt_pos = [nout_ini + (1 - (max(aexps) - a)/aexps.ptp()) * nnouts for a in a_targets_lbt]
#from astropy.cosmology import WMAP7 as cosmo
#lookback_t=[cosmo.lookback_time(i).value for i in zreds]

#%%
import matplotlib.pyplot as plt
# plot each galaxy.
# stellar mass growth and lambda_r as a function of time.

# The exponent (also called ass "offset") in the figure (1e11)
# overlaps with lookback time tick labels.
# And moving the offset around is not easy.
# So, manually divide the values.
mm = mstar/1e10

plt.close()
plt.ioff()139
def make_patch_spines_invisible(ax):
    """
        Useful for plotting multiple variables (more than two twinx())
    """
    ax.set_frame_on(True)
    ax.patch.set_visible(False)
    #for sp in ax.spines.itervalues():
    # Changed in Python3
    for sp in ax.spines.values():
        sp.set_visible(False)_____no_output_____for i, idgal in enumerate(cat['idx']):
#for i, idgal in enumerate([1618]):
    print(idgal, "!!!!")
    if mm[i][0] < 0.2 :
        print(idgal, mm[i][0], mm[i][-1])
        continue

    plt.rcParams["figure.figsize"] = [12,10]
    fig, axes = plt.subplots(3)
#    plt.figure(num=1, figsize=[10,20])
    fig.suptitle("ID: " + str(idgal).zfill(5), fontsize=18)#, y=1.01)
    lns1 = axes[0].plot(nouts[::-1], l_r[i], label=r"$\lambda_{R}$")
    axes[0].set_xticks(z_pos)
    axes[0].set_xticklabels(z_target_str)
    plt.subplots_adjust(left = 0.1, right = 0.9, \
                        wspace = 0.1, hspace = 0.0, \
                        bottom = 0.1, top = 0.85)

    axes[0].set_xlim([37,187])
    axes[0].set_ylim([0,1.0])
    axes[0].set_ylabel(r"$\lambda_{R}$")
    axes[0].set_xlabel("redshift")

#    ax2 = axes[0].twinx()
    lns2 = axes[1].plot(nouts[::-1], mm[i], 'r-', label="stellar mass")
    axes[1].set_ylim([0, 1.3*max(mm[i])])
    axes[1].set_xlim([37,187])
    axes[1].set_ylabel(r"Stellar mass $[10^{10}M_{\odot}]$")
    axes[1].get_yaxis().get_offset_text().set_y(1)

#    ax3 = ax1.twinx() # Reff
#    ax3.spines["right"].set_position(("axes", 1.2))
#    make_patch_spines_invisible(ax3)
    # Second, show the right spine.
#    ax3.spines["right"].set_visible(True)
    axes[2].set_ylabel("Reff [kpc]")
    axes[2].set_xlim([37,187])
    lns3 = axes[2].plot(nouts[::-1], reff[i], 'g-', label='Reff')

    # hide x axes so that subplots stick together.
    plt.setp([a.get_xticklabels() for a in fig.axes[:-1]], visible=False)

    ax4 = axes[0].twiny()
    ax4.set_xlabel("Lookback time", labelpad=10)
    ax4.set_xticks(lbt_pos)
    ax4.set_xticklabels(lbt_target_str)
    lns = lns1+lns2+lns3
    labs = [l.get_label() for l in lns]
    axes[0].legend(lns, labs, loc=0)
    # logend location codes:
    # 0 ~ 10
    # best, ur, ul, lr, ll, r, cl, cr, lower c, upper c, center
    #

#    plt.show()
    plt.savefig(wdir + 'catalog/' + str(idgal).zfill(5) + '.png')
    plt.close()15135 !!!!
15135 0.0 0.0
15162 !!!!
15162 0.0 0.0
15201 !!!!
15201 0.0 0.0
15226 !!!!
15226 0.0 0.0
15241 !!!!
15241 0.0 0.0
15287 !!!!
15287 0.0 0.0
15332 !!!!
15332 0.0 0.0
15353 !!!!
15353 0.0 0.0
15433 !!!!
15433 0.0 0.0
15468 !!!!
15468 0.0 0.0
15484 !!!!
15484 0.0 0.0
15501 !!!!
15501 0.0 0.0
15567 !!!!
15567 0.0 0.0
15584 !!!!
15584 0.0 0.0
15615 !!!!
15615 0.0 0.0
15646 !!!!
15646 0.0 0.0
15706 !!!!
15706 0.0 0.0
15747 !!!!
15747 0.0 0.0
15834 !!!!
15834 0.0 0.0
15852 !!!!
15852 0.0 0.0
15861 !!!!
15861 0.0 0.0
15876 !!!!
15876 0.0 0.0
15209 !!!!
15209 0.0 0.0
15232 !!!!
15232 0.0 0.0
15248 !!!!
15248 0.0 0.0
15307 !!!!
15307 0.0 0.0
15321 !!!!
15321 0.0 0.0
15336 !!!!
15336 0.0 0.0
15360 !!!!
15360 0.0 0.0
15394 !!!!
15394 0.0 0.0
15416 !!!!
15416 0.0 0.0
15438 !!!!
15438 0.0 0.0
15456 !!!!
15456 0.0 0.0
15492 !!!!
15492 0.0 0.0
15507 !!!!
15507 0.0 0.0
15557 !!!!
15557 0.0 0.0
15575 !!!!
15575 0.0 0.0
15588 !!!!
15588 0.0 0.0
15604 !!!!
15604 0.0 0.0
15634 !!!!
15634 0.0 0.0
15713 !!!!
15713 0.0 0.0
15729 !!!!
15729 0.0 0.0
15816 !!!!
15816 0.0 0.0
15866 !!!!
15866 0.0 0.0
15881 !!!!
15881 0.0 0.0
15154 !!!!
15154 0.0 0.0
15179 !!!!
15179 0.0 0.0
15217 !!!!
15217 0.0 0.0
15252 !!!!
15252 0.0 0.0
15278 !!!!
15278 0.0 0.0
15296 !!!!
15296 0.0 0.0
15310 !!!!
15310 0.0 0.0
15344 !!!!
15344 0.0 0.0
15378 !!!!
15378 0.0 0.0
15400 !!!!
15400 0.0 0.0
15427 !!!!
15427 0.0 0.0
15443 !!!!
15443 0.0 0.0
15462 !!!!
15462 0.0 0.0
15475 !!!!
15475 0.0 0.0
15496 !!!!
15496 0.0 0.0
15511 !!!!
15511 0.0 0.0
15525 !!!!
15525 0.0 0.0
15561 !!!!
15561 0.0 0.0
15578 !!!!
15578 0.0 0.0
15608 !!!!
15608 0.0 0.0
15622 !!!!
15622 0.0 0.0
15640 !!!!
15640 0.0 0.0
15680 !!!!
15680 0.0 0.0
15717 !!!!
15717 0.0 0.0
15733 !!!!
15733 0.0 0.0
15826 !!!!
15826 0.0 0.0
15886 !!!!
15886 0.0 0.0
15158 !!!!
15158 0.0 0.0
15194 !!!!
15194 0.0 0.0
15298 !!!!
15298 0.0 0.0
15312 !!!!
15312 0.0 0.0
15326 !!!!
15326 0.0 0.0
15349 !!!!
15349 0.0 0.0
15380 !!!!
15380 0.0 0.0
15465 !!!!
15465 0.0 0.0
15478 !!!!
15478 0.0 0.0
15498 !!!!
15498 0.0 0.0
15527 !!!!
15527 0.0 0.0
15563 !!!!
15563 0.0 0.0
15580 !!!!
15580 0.0 0.0
15593 !!!!
15593 0.0 0.0
15624 !!!!
15624 0.0 0.0
15642 !!!!
15642 0.0 0.0
15656 !!!!
15656 0.0 0.0
15687 !!!!
15687 0.0 0.0
15703 !!!!
15703 0.0 0.0
15719 !!!!
15719 0.0 0.0
15831 !!!!
15831 0.0 0.0
15873 !!!!
15873 0.0 0.0
15137 !!!!
15137 0.0 0.0
15165 !!!!
15165 0.0 0.0
15207 !!!!
15207 0.0 0.0
15229 !!!!
15229 0.0 0.0
15243 !!!!
15243 0.0 0.0
15305 !!!!
15305 0.0 0.0
15318 !!!!
15318 0.0 0.0
15414 !!!!
15414 0.0 0.0
15435 !!!!
15435 0.0 0.0
15471 !!!!
15471 0.0 0.0
15504 !!!!
15504 0.0 0.0
15519 !!!!
15519 0.0 0.0
15551 !!!!
15551 0.0 0.0
15570 !!!!
15570 0.0 0.0
15586 !!!!
15586 0.0 0.0
15602 !!!!
15602 0.0 0.0
15617 !!!!
15617 0.0 0.0
15633 !!!!
15633 0.0 0.0
15648 !!!!
15648 0.0 0.0
15665 !!!!
15665 0.0 0.0
15696 !!!!
15696 0.0 0.0
15711 !!!!
15711 0.0 0.0
15727 !!!!
15727 0.0 0.0
15839 !!!!
15839 0.0 0.0
15855 !!!!
15855 0.0 0.0
15863 !!!!
15863 0.0 0.0
15160 !!!!
15160 0.0 0.0
15220 !!!!
15220 0.0 0.0
15239 !!!!
15239 0.0 0.0
15285 !!!!
15285 0.0 0.0
15300 !!!!
15300 0.0 0.0
15313 !!!!
15313 0.0 0.0
15327 !!!!
15327 0.0 0.0
15350 !!!!
15350 0.0 0.0
15385 !!!!
15385 0.0 0.0
15430 !!!!
15430 0.0 0.0
15448 !!!!
15448 0.0 0.0
15466 !!!!
15466 0.0 0.0
15479 !!!!
15479 0.0 0.0
15499 !!!!
15499 0.0 0.0
15514 !!!!
15514 0.0 0.0
15528 !!!!
15528 0.0 0.0
15547 !!!!
15547 0.0 0.0
15565 !!!!
15565 0.0 0.0
15581 !!!!
15581 0.0 0.0
15595 !!!!

</code>


# Notebook from empet/Mathematical-Physics
Path: Eigenfunctions-Laplace-snowflake.ipynb

## Eigenfunctions of the Laplace operator on the Koch snowflake_____no_output_____We generate and plot a few eigenfunctions of the Laplace operator on a snowflake. _____no_output_____The numerical computation of eigenvalues of differential operators is
usually performed via
 a discretization of the  operator to get
a finite dimensional matrix approximation through finite differences,
finite elements, or any other numerical method. The spectrum of the
approximation is then taken as an approximation to the spectrum of the
original differential operator._____no_output_____The properties of Laplacian eigenvalues and eigenfunctions have been investigated in many scientific
disciplines,  theory of acoustical, optical, and quantum waveguides, condensed
matter physics and quantum mechanics, quantum graphs, image processing, computer
graphics,  biology and more.

Nowadays,  eigenfunctions of the Laplace operator have applications in Data Science and Machine Learning,
for face recognition, clustering,  nonlinear image denoising and segmentation,  mapping of protein energy landscapes, and more.
A google search reveals more examples._____no_output_____To get an idea of their properties,  we illustrate here how to compute the spectrum of the Laplace operator, $\Delta u=-\lambda u,$
$$\Delta=\displaystyle\frac{\partial^2}{\partial x^2}+\displaystyle\frac{\partial^2}{\partial y^2},$$
on a bounded planar domain, with Dirichlet (bc = 'D') or Neumann boundary condition (bc='N').._____no_output_____The bounded domain is represented by a binary np.array, with 0 inside the domain and 1 outside it.

In this example  we are starting with a black and white image of a Koch snowflake, i.e. the bounded domain is the planar domain whose boundary is a closed Koch curve. First we read this image and process it to get a binary array representing the snowflake domain, to be passed to the algorithm that computes the first K eigenvalues and eigenfunctions of the discretized Laplace operator._____no_output_____

%matplotlib inline_____no_output_____import numpy as np
import scipy.sparse
from scipy.sparse import linalg
from skimage import  (filters, io)
import plotly.graph_objs as go_____no_output_____img=io.imread('images/kochSnowflake.jpg', as_gray=True)
io.imshow(img)_____no_output_____threshold = filters.threshold_otsu(img)
binary = img < threshold_____no_output_____

Read the binary array representing a compact 2D region, and  perform the numerical discretization of the Helmholz equation $\Delta u=-\lambda u$ with Dirichlet boundary condition
 on this special domain.

For a better performance we are working with `scipy.sparse` matrices._____no_output_____

def discrete_Laplace(bimg, bc = 'N'):

    if bimg.shape[0] != bimg.shape[1]:
        raise ValueError('The code below works with arrays of shape (n,n)')
    vals = np.unique(bimg)
    if len(vals) != 2 or 0 or 1 not in vals:
        raise ValueError('Your array must be a non-constant binary array')
    bimg = np.flipud(bimg) #flip the image because Plotly works in a right reference system
    newimg = bimg + 1e-05 # a slight perturbation of the binary array
    n = newimg.shape[0]
    e = np.ones(n)
    if bc == 'N':
        diagonal_vals = np.array([-e, e, -e])
        partial_x = scipy.sparse.spdiags(diagonal_vals, np.array([-1, 0, n-1]), n, n)
    elif bc == 'D':
        diagonal_vals = np.array([-e, e])
        partial_x = scipy.sparse.spdiags(diagonal_vals, np.array([-1, 0]), n, n)
    else:
        raise ValueError('Only Neumann and Dirichlet boundary conditions are implemented')

    gradient = scipy.sparse.vstack([scipy.sparse.kron(partial_x, scipy.sparse.eye(n)),
                                    scipy.sparse.kron(scipy.sparse.eye(n), partial_x)])

    img_as_row = np.concatenate((newimg.flatten(), newimg.flatten()))
    Laplace = gradient.transpose()*scipy.sparse.spdiags(img_as_row, [0], 2*n**2, 2*n**2)*gradient
    return Laplace_____no_output_____

Compute the K smalllest eigenvalues and eigenfunctions of the Laplace operator, [https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.eigs.html](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.eigs.html),  with shift-invert mode corresponding to `sigma`=0.
In this case the transformed eigenvalues are `eigvals'=1/(eigvals-sigma)=1/eigvals`.
With the option `which='LM'` we get the k largest `eigvals'`, i.e.  the smallest
eigenvalues of the original problem._____no_output_____`Laplace.todense()` is a symmetric positive matrix of dimension $n^2 \times n^2$. The eigenvalues of the Laplace matrix  are ordered as follows:

$$0\leq\lambda_0\leq \lambda_1 \leq \cdots \lambda_{n^2-1}$$_____no_output_____with $\lambda_0>0$ for Dirichlet boundary condition._____no_output_____

def Laplace_spectrum(Laplace, K):
    return  scipy.sparse.linalg.eigsh(Laplace, K, which='LM', sigma=0) #eigvals, u    _____no_output_____def get_plotly_contour(u, bimg,   I, size=0.1, colorscale = 'RdBu', reversescale=False):
    # u: eigenfunction to be plotted
    # bimg: binary array representing the bounded domain on which the Laplace spectrum is computed image
    # I: index of the eigenvalue whose eigenfunction is plotted
    # size:  step for  contour lines
    if I >= u.shape[1]:
        raise ValueError(f'You must compute first at least {I+1} eigenvalues and eigenfunctions')
    n = bimg.shape[0]
    uu = u[:, I].reshape((n, n)) #reshape the I^th eigenvector to create a contour trace from it
    uu = uu / np.max(np.fabs(uu)) # normalize the image representing the eigenfunction u[:,I]
    uu[bimg==1] = np.nan
    comin, comax = np.nanmin(uu), np.nanmax(uu)
    lim = max([abs(comin), comax])  #-lim, lim are the  start, and end values for the contour lines
    return go.Contour(
                    z=uu.tolist(),
                    colorscale=colorscale,
                    reversescale=reversescale,
                    colorbar=dict(thickness=20),
                    contours=dict(start=-lim, end=lim, size=size),
                    line=dict(width=0.5, color='rgb(200,200,200)'))_____no_output_____pl_RdYlBu = [[0.0, 'rgb(49, 54, 149)'],
            [0.1, 'rgb(68, 115, 179)'],
            [0.2, 'rgb(116, 173, 209)'],
            [0.3, 'rgb(169, 216, 232)'],
            [0.4, 'rgb(224, 243, 248)'],
            [0.5, 'rgb(254, 254, 190)'],
            [0.6, 'rgb(254, 224, 144)'],
            [0.7, 'rgb(252, 172, 96)'],
            [0.8, 'rgb(244, 109, 67)'],
            [0.9, 'rgb(214, 47, 38)'],
            [1.0, 'rgb(165, 0, 38)']]
_____no_output_____layout = go.Layout(title='',
                   font=dict(family='Balto',
                             size=12),
                   width=600, height=600,
                   xaxis=dict(visible=False),
                   yaxis=dict(visible=False),
                   plot_bgcolor='rgb(0, 101, 93)')
_____no_output_____

Let us compute 26 eigenvalues and the corresponding eigenfunctions of the Laplace operator defined on the snowflake._____no_output_____

K = 26
I = 25

Laplace = discrete_Laplace(binary)
n = binary.shape[0]
eigvals, u = Laplace_spectrum(Laplace, K)
contour = get_plotly_contour(u, binary, I, size =0.07, colorscale = pl_RdYlBu)
fw = go.FigureWidget(data=[contour], layout=layout)
title = f"Eigenfunction of the Laplace operator defined on the snowflake
"+\
        f"Corresponding  eigenvalue: eig[{I}]={eigvals[I]:0.11f}"
fw.layout.update(title=title);_____no_output_____from IPython.display import IFrame
IFrame('https://plot.ly/~empet/15205',  width=600, height=600)_____no_output_____


# Notebook from ashishpatel26/1d-network-for-Audio
Path: 1dVGG_for_audio.ipynb

import numpy as np
import pandas as pd

##thanks to https://www.kaggle.com/toregil/mystery-planet-99-8-cnn and https://www.kaggle.com/aleksod/0-75-precision-0-60-recall-linear-svc
##very useful kernels pls have a look at their implementations too!_____no_output_____train = pd.read_csv("/kaggle/input/kepler-labelled-time-series-data/exoTrain.csv")_____no_output_____test = pd.read_csv("/kaggle/input/kepler-labelled-time-series-data/exoTest.csv")_____no_output_____train.head()_____no_output_____train.groupby('LABEL').size()_____no_output_____train.describe()_____no_output_____test.groupby('LABEL').size()_____no_output_____

THE FOLLOWING ARE STARS WITH EXOPLANETS..._____no_output_____

train[0:37]_____no_output_____import matplotlib.pyplot as plt_____no_output_____for i in range(0,10): #random numbers :/
    flux = train[train.LABEL == 1].drop('LABEL', axis=1).iloc[i,:]
    time = np.arange(len(flux)) * (36.0/60.0) # time in units of hours
    plt.figure(figsize=(15,5))
    plt.title('Flux of star {} with  confirmed exoplanets'.format(i+1),color='r')
    plt.ylabel('Flux, e-/s',color='r')
    plt.xlabel('Time, hours',color='r')
    plt.plot(time, flux)_____no_output_____for i in range(10,20): #random numbers :/
    flux = train[train.LABEL == 1].drop('LABEL', axis=1).iloc[i,:]
    time = np.arange(len(flux)) * (36.0/60.0) # time in units of hours
    plt.figure(figsize=(15,5))
    plt.title('Flux of star {} with  confirmed exoplanets'.format(i+1),color='r')
    plt.ylabel('Flux, e-/s',color='r')
    plt.xlabel('Time, hours',color='r')
    plt.plot(time, flux)_____no_output_____import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Conv1D, MaxPool1D, Flatten
from keras import backend as K
from keras.preprocessing import image
from keras.applications.mobilenet import MobileNet
from keras.applications.vgg16 import preprocess_input, decode_predictions
from keras.models import Model
import timeit
import warnings
warnings.filterwarnings('ignore')Using TensorFlow backend.
INPUT_LIB = '/kaggle/input/kepler-labelled-time-series-data/'
raw_data = np.loadtxt(INPUT_LIB + 'exoTrain.csv', skiprows=1, delimiter=',')
x_train = raw_data[:, 1:]
y_train = raw_data[:, 0, np.newaxis] - 1.
raw_data = np.loadtxt(INPUT_LIB + 'exoTest.csv', skiprows=1, delimiter=',')
x_test = raw_data[:, 1:]
y_test = raw_data[:, 0, np.newaxis] - 1.
del raw_data_____no_output_____x_train = ((x_train - np.mean(x_train, axis=1).reshape(-1,1)) /
           np.std(x_train, axis=1).reshape(-1,1))
x_test = ((x_test - np.mean(x_test, axis=1).reshape(-1,1)) /
          np.std(x_test, axis=1).reshape(-1,1))_____no_output_____from scipy.ndimage.filters import uniform_filter1d
x_train = np.stack([x_train, uniform_filter1d(x_train, axis=1, size=200)], axis=2)
x_test = np.stack([x_test, uniform_filter1d(x_test, axis=1, size=200)], axis=2)_____no_output_____x_train = x_train - x_train.mean()
x_train = x_train / x_train.max()_____no_output_____x_test = x_test - x_test.mean()
x_test = x_test / x_test.max()_____no_output_____model = Sequential()
model.add(Conv1D(input_shape=x_train.shape[1:],filters=64,kernel_size=9,padding="same", activation="relu"))
model.add(Conv1D(filters=64,kernel_size=9,padding="same", activation="relu"))
model.add(MaxPool1D(pool_size=4,strides=4))
model.add(Conv1D(filters=128, kernel_size=9, padding="same", activation="relu"))
model.add(Conv1D(filters=128, kernel_size=9, padding="same", activation="relu"))
model.add(MaxPool1D(pool_size=4,strides=4))
model.add(Conv1D(filters=256, kernel_size=9, padding="same", activation="relu"))
model.add(Conv1D(filters=256, kernel_size=9, padding="same", activation="relu"))
model.add(Conv1D(filters=256, kernel_size=9, padding="same", activation="relu"))
model.add(MaxPool1D(pool_size=4,strides=4))
model.add(Conv1D(filters=512, kernel_size=9, padding="same", activation="relu"))
model.add(Conv1D(filters=512, kernel_size=9, padding="same", activation="relu"))
model.add(Conv1D(filters=512, kernel_size=9, padding="same", activation="relu"))
model.add(MaxPool1D(pool_size=4,strides=4))
model.add(Conv1D(filters=512, kernel_size=9, padding="same", activation="relu"))
model.add(Conv1D(filters=512, kernel_size=9, padding="same", activation="relu"))
model.add(Conv1D(filters=512, kernel_size=9, padding="same", activation="relu"))
model.add(MaxPool1D(pool_size=4,strides=4))

model.add(Flatten())
model.add(Dense(units=4096,activation="relu"))
model.add(Dense(units=4096,activation="relu"))
model.add(Dense(units=1, activation='sigmoid'))_____no_output_____

The next cell picks data for us. Since our data is highly imbalanced we need to use this function to pick equal amounts of data from both the classes._____no_output_____

def batch_generator(x_train, y_train, batch_size=32):
    """
    Gives equal number of positive and negative samples, and rotates them randomly in time
    """
    half_batch = batch_size // 2
    x_batch = np.empty((batch_size, x_train.shape[1], x_train.shape[2]), dtype='float32')
    y_batch = np.empty((batch_size, y_train.shape[1]), dtype='float32')

    yes_idx = np.where(y_train[:,0] == 1.)[0]
    non_idx = np.where(y_train[:,0] == 0.)[0]

    while True:
        np.random.shuffle(yes_idx)
        np.random.shuffle(non_idx)

        x_batch[:half_batch] = x_train[yes_idx[:half_batch]]
        x_batch[half_batch:] = x_train[non_idx[half_batch:batch_size]]
        y_batch[:half_batch] = y_train[yes_idx[:half_batch]]
        y_batch[half_batch:] = y_train[non_idx[half_batch:batch_size]]

        for i in range(batch_size):
            sz = np.random.randint(x_batch.shape[1])
            x_batch[i] = np.roll(x_batch[i], sz, axis = 0)

        yield x_batch, y_batch_____no_output_____from keras.optimizers import Adam
opt = Adam(lr=4e-5)
model.compile(optimizer=opt, loss=keras.losses.binary_crossentropy, metrics=['binary_accuracy'])
hist = model.fit_generator(batch_generator(x_train, y_train, 32),
                           validation_data=(x_test, y_test),
                           verbose=2, epochs=50,
                           steps_per_epoch=x_train.shape[1]//32)Epoch 1/50
 - 6s - loss: 0.6342 - binary_accuracy: 0.6638 - val_loss: 0.4602 - val_binary_accuracy: 0.8035
Epoch 2/50
 - 5s - loss: 0.5404 - binary_accuracy: 0.7449 - val_loss: 0.5685 - val_binary_accuracy: 0.7737
Epoch 3/50
 - 5s - loss: 0.5226 - binary_accuracy: 0.7497 - val_loss: 0.2521 - val_binary_accuracy: 0.8860
Epoch 4/50
 - 5s - loss: 0.4981 - binary_accuracy: 0.7705 - val_loss: 0.3642 - val_binary_accuracy: 0.8474
Epoch 5/50
 - 5s - loss: 0.3945 - binary_accuracy: 0.8438 - val_loss: 0.1723 - val_binary_accuracy: 0.9070
Epoch 6/50
 - 5s - loss: 0.3431 - binary_accuracy: 0.8677 - val_loss: 0.2658 - val_binary_accuracy: 0.9000
Epoch 7/50
 - 5s - loss: 0.2286 - binary_accuracy: 0.9208 - val_loss: 0.1621 - val_binary_accuracy: 0.9491
Epoch 8/50
 - 5s - loss: 0.2006 - binary_accuracy: 0.9302 - val_loss: 0.1509 - val_binary_accuracy: 0.9281
Epoch 9/50
 - 5s - loss: 0.1594 - binary_accuracy: 0.9504 - val_loss: 0.1088 - val_binary_accuracy: 0.9491
Epoch 10/50
 - 5s - loss: 0.1434 - binary_accuracy: 0.9492 - val_loss: 0.0946 - val_binary_accuracy: 0.9579
Epoch 11/50
 - 5s - loss: 0.1100 - binary_accuracy: 0.9697 - val_loss: 0.3258 - val_binary_accuracy: 0.8667
Epoch 12/50
 - 5s - loss: 0.1270 - binary_accuracy: 0.9628 - val_loss: 0.1180 - val_binary_accuracy: 0.9509
Epoch 13/50
 - 5s - loss: 0.1124 - binary_accuracy: 0.9631 - val_loss: 0.1103 - val_binary_accuracy: 0.9491
Epoch 14/50
 - 5s - loss: 0.1228 - binary_accuracy: 0.9621 - val_loss: 0.0340 - val_binary_accuracy: 0.9860
Epoch 15/50
 - 5s - loss: 0.0859 - binary_accuracy: 0.9760 - val_loss: 0.0649 - val_binary_accuracy: 0.9772
Epoch 16/50
 - 5s - loss: 0.0656 - binary_accuracy: 0.9795 - val_loss: 0.0478 - val_binary_accuracy: 0.9860
Epoch 17/50
 - 5s - loss: 0.1147 - binary_accuracy: 0.9637 - val_loss: 0.0395 - val_binary_accuracy: 0.9895
Epoch 18/50
 - 5s - loss: 0.0645 - binary_accuracy: 0.9811 - val_loss: 0.1559 - val_binary_accuracy: 0.9509
Epoch 19/50
 - 5s - loss: 0.0663 - binary_accuracy: 0.9801 - val_loss: 0.0549 - val_binary_accuracy: 0.9807
Epoch 20/50
 - 5s - loss: 0.0437 - binary_accuracy: 0.9893 - val_loss: 0.0309 - val_binary_accuracy: 0.9895
Epoch 21/50
 - 5s - loss: 0.0561 - binary_accuracy: 0.9826 - val_loss: 0.0397 - val_binary_accuracy: 0.9877
Epoch 22/50
 - 5s - loss: 0.0912 - binary_accuracy: 0.9738 - val_loss: 0.0789 - val_binary_accuracy: 0.9737
Epoch 23/50
 - 5s - loss: 0.0530 - binary_accuracy: 0.9852 - val_loss: 0.0825 - val_binary_accuracy: 0.9754
Epoch 24/50
 - 5s - loss: 0.0626 - binary_accuracy: 0.9807 - val_loss: 0.0250 - val_binary_accuracy: 0.9912
Epoch 25/50
 - 6s - loss: 0.0729 - binary_accuracy: 0.9763 - val_loss: 0.0379 - val_binary_accuracy: 0.9895
Epoch 26/50
 - 5s - loss: 0.0237 - binary_accuracy: 0.9943 - val_loss: 0.0588 - val_binary_accuracy: 0.9895
Epoch 27/50
 - 5s - loss: 0.0457 - binary_accuracy: 0.9880 - val_loss: 0.0462 - val_binary_accuracy: 0.9912
Epoch 28/50
 - 5s - loss: 0.0509 - binary_accuracy: 0.9858 - val_loss: 0.0446 - val_binary_accuracy: 0.9912
Epoch 29/50
 - 5s - loss: 0.0248 - binary_accuracy: 0.9934 - val_loss: 0.0714 - val_binary_accuracy: 0.9772
Epoch 30/50
 - 5s - loss: 0.0304 - binary_accuracy: 0.9924 - val_loss: 0.0448 - val_binary_accuracy: 0.9912
Epoch 31/50
 - 5s - loss: 0.1585 - binary_accuracy: 0.9470 - val_loss: 0.1578 - val_binary_accuracy: 0.9561
Epoch 32/50
 - 5s - loss: 0.0425 - binary_accuracy: 0.9886 - val_loss: 0.0710 - val_binary_accuracy: 0.9754
Epoch 33/50
 - 5s - loss: 0.0445 - binary_accuracy: 0.9874 - val_loss: 0.0220 - val_binary_accuracy: 0.9930
Epoch 34/50
 - 5s - loss: 0.0206 - binary_accuracy: 0.9949 - val_loss: 0.0397 - val_binary_accuracy: 0.9930
Epoch 35/50
 - 5s - loss: 0.0380 - binary_accuracy: 0.9890 - val_loss: 0.0325 - val_binary_accuracy: 0.9912
Epoch 36/50
 - 6s - loss: 0.0373 - binary_accuracy: 0.9883 - val_loss: 0.0354 - val_binary_accuracy: 0.9895
Epoch 37/50
 - 5s - loss: 0.0241 - binary_accuracy: 0.9931 - val_loss: 0.0514 - val_binary_accuracy: 0.9842
Epoch 38/50
 - 5s - loss: 0.0488 - binary_accuracy: 0.9880 - val_loss: 0.0499 - val_binary_accuracy: 0.9860
Epoch 39/50
 - 5s - loss: 0.0200 - binary_accuracy: 0.9949 - val_loss: 0.0507 - val_binary_accuracy: 0.9895
Epoch 40/50
 - 5s - loss: 0.0185 - binary_accuracy: 0.9946 - val_loss: 0.0616 - val_binary_accuracy: 0.9930
Epoch 41/50
 - 5s - loss: 0.0414 - binary_accuracy: 0.9877 - val_loss: 0.0553 - val_binary_accuracy: 0.9825
Epoch 42/50
 - 5s - loss: 0.0128 - binary_accuracy: 0.9968 - val_loss: 0.0512 - val_binary_accuracy: 0.9895
Epoch 43/50
 - 5s - loss: 0.0590 - binary_accuracy: 0.9845 - val_loss: 0.0579 - val_binary_accuracy: 0.9860
Epoch 44/50
 - 5s - loss: 0.0223 - binary_accuracy: 0.9949 - val_loss: 0.0680 - val_binary_accuracy: 0.9877
Epoch 45/50
 - 5s - loss: 0.0228 - binary_accuracy: 0.9931 - val_loss: 0.0558 - val_binary_accuracy: 0.9877
Epoch 46/50
 - 5s - loss: 0.0101 - binary_accuracy: 0.9975 - val_loss: 0.0343 - val_binary_accuracy: 0.9912
Epoch 47/50
 - 5s - loss: 0.0317 - binary_accuracy: 0.9902 - val_loss: 0.0374 - val_binary_accuracy: 0.9912
Epoch 48/50
 - 5s - loss: 0.0288 - binary_accuracy: 0.9921 - val_loss: 0.0326 - val_binary_accuracy: 0.9930
Epoch 49/50
 - 5s - loss: 0.0100 - binary_accuracy: 0.9978 - val_loss: 0.0254 - val_binary_accuracy: 0.9930
Epoch 50/50
 - 5s - loss: 0.0202 - binary_accuracy: 0.9956 - val_loss: 0.0382 - val_binary_accuracy: 0.9895
plt.plot(hist.history['loss'], color='b')
plt.plot(hist.history['val_loss'], color='r')
plt.show()
plt.plot(hist.history['binary_accuracy'], color='b')
plt.plot(hist.history['val_binary_accuracy'], color='r')
plt.show()_____no_output_____non_idx = np.where(y_test[:,0] == 0.)[0]
yes_idx = np.where(y_test[:,0] == 1.)[0]
y_hat = model.predict(x_test)[:,0]_____no_output_____X = range(len(y_hat))_____no_output_____

APPLYING A THRESHOLD OF 0.7 on my predictions_____no_output_____

y_test_pred=[]
for i in y_hat:
    if i>=0.7:
        y_test_pred.append(1)
    else:
        y_test_pred.append(0)_____no_output_____plt.scatter(X,y_test_pred,color='b')
plt.show()
plt.scatter(X,y_test,color='r')
plt.show()_____no_output_____y_test_pred=np.asarray(y_test_pred)_____no_output_____

We use the following metrics to evaluate our model_____no_output_____

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
print("accuracy ",accuracy_score(y_test, y_test_pred))
print("precision ",precision_score(y_test, y_test_pred))
print("recall-score ",recall_score(y_test, y_test_pred))
print("f1-score ",f1_score(y_test, y_test_pred))        _____no_output_____

Precision is the fraction of relevant instances among the retrieved instances, while recall is the fraction of the total amount of relevant instances that were actually retrieved. So our model is performing pretty well using vgg16 architecture. I havent tried using the pre trained weights of vgg but that might give some interesting results too!_____no_output_____


# Notebook from valentinissot/corona
Path: notebooks/Corona_visualisation.ipynb

import os
import datetime
import yaml
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt_____no_output_____# Lecture du fichier d'environnement
ENV_FILE = '/home/albane/Cours/Projet Corona/env.yaml'

with open(ENV_FILE) as f:
    params = yaml.load(f) #, Loader=yaml.FullLoader)

# Initialisation des chemins vers les fichiers
ROOT_DIR = os.path.dirname(os.path.abspath(ENV_FILE))
DATA_FILE = os.path.join(ROOT_DIR,
                         params['directories']['processed'],
                         params['files']['all_data'])

# Lecture du fichier de données
epidemie_df = (pd.read_csv(DATA_FILE, parse_dates=['Last Update'])
               .assign(day=lambda _df: _df['Last Update'].dt.date)
               .drop_duplicates(subset=['Country/Region', 'Province/State', 'day'])
               [lambda df: df['day'] <= datetime.date(2020, 3, 20)]
              )_____no_output_____def get_country(country_or_region):
    epidemie_agg = epidemie_df[epidemie_df['Country/Region'] == country_or_region].groupby(['Country/Region', 'day']).agg({'Confirmed': 'sum', 'Deaths': 'sum', 'Recovered': 'sum'}).reset_index()
    epidemie_agg=epidemie_agg.rename(columns = {'day':'Date', 'Confirmed':'Total confirmed'})
    epidemie_agg["New cases"]=epidemie_agg["Total confirmed"].diff()
    epidemie_agg["Active"]=epidemie_agg["Total confirmed"]-epidemie_agg["Deaths"]-epidemie_agg["Recovered"]
    return(epidemie_agg)_____no_output_____get_country(country_or_region="France").tail()_____no_output_____

## Plot COVID-19 comparison between countries_____no_output_____

countries_list = ['France','Italy','Spain','Germany','South Korea','Iran','UK','US']

def evol_by_country(countries_to_plot, variable_to_plot):
    fig = plt.gcf()
    fig.set_size_inches(15, 8, forward=True)
    for c in countries_list :
        plt.plot(get_country(c)['Date'], get_country(c)[variable_to_plot], label=c)
    plt.grid(True)
    plt.title("Comparison of the evolution of COVID-19 : %s" % variable_to_plot)
    plt.legend()
    plt.show()
_____no_output_____evol_by_country(countries_list,'Total confirmed')_____no_output_____evol_by_country(countries_list,'Deaths')_____no_output_____evol_by_country(countries_list,'Recovered')_____no_output_____

## Plot COVID-19 evolution for a specific country_____no_output_____

def one_country_evol(country_to_plot):
    get_country(country_to_plot).set_index('Date')[['Total confirmed','Deaths','Recovered']].plot(secondary_y=['Deaths','Recovered'],figsize=(15, 10))
    plt.title("Detailed evolution of COVID-19 in %s" % country_to_plot)
    plt.grid(True)
    plt.show()_____no_output_____one_country_evol('France')_____no_output_____one_country_evol('Mainland China')_____no_output_____one_country_evol('US')_____no_output_____


# Notebook from alcome1614/ModifiedInteractionNetworks
Path: demo.ipynb

from files.scene import *
from files.multiscene import *
from files.IN import *_____no_output_____import tikzplotlib_____no_output_____

# Data Creation

## Scene creation_____no_output_____

#No interaction scene
escena_no_interaction = Scene(n_body=3, R_init=polygon_init, V_init=polygon_init, r=0.1, v=0.01, dt=1)
escena_no_interaction.simulate(interaction=zero_interaction, t_steps=100)

#Central force scene
escena_central_force = Scene(n_body=3, R_init=random_init, V_init=random_init, r=0.1, v=8e-3, dt=1)
escena_central_force.simulate(interaction=central_force, t_steps=100, alpha=1e-4, beta=0)

#Vicsek model scene
escena_vicsek = Scene(n_body=6,  R_init=random_init, V_init=random_init, r=0.6, v=1e-1, dt=1)
escena_vicsek.simulate(interaction=vicsek, t_steps=100, radius=0.4, angle_vision=np.pi)_____no_output_____fig, ax = plt.subplots(1,3,figsize=(15,5))

for axi, scene, title in zip(ax,[escena_no_interaction, escena_central_force, escena_vicsek], ['no interaction', 'central force', 'Vicsek model']):
    axi.set_xlim(-1,1)
    axi.set_ylim(-1,1)
    axi.set_title(title)
    axi.tick_params(labelbottom=False, labelleft=False, right=False, top=False, bottom=False, left=False)
    scene.plot_trajectory(who='truth', ax=axi)
    axi.legend()
plt.show()
_____no_output_____

## Multiscene creation_____no_output_____

n_scenes = 3
t_steps = 100
ms = MultiScene()
ms.add_bunch(n_scenes=n_scenes, n_body=5, t_steps=100, R_init=random_init, V_init=random_init, dt=1, r=2e-1, v=5e-2, interaction=zero_interaction)
ms.add_bunch(n_scenes=n_scenes, n_body=5, t_steps=t_steps, R_init=random_init, V_init=random_init, dt=1, r=7.5e-1, v=8e-3, interaction=central_force, alpha=1e-4, beta=0)
ms.add_bunch(n_scenes=n_scenes, n_body=5, t_steps=t_steps, R_init=random_init, V_init=polygon_init, r=6e-1, v=1e-2, interaction=vicsek, radius=2.5e-1)
ms.save('data/demo_multiscene')

ms = MultiScene(load='data/demo_multiscene')_____no_output_____fig, ax = plt.subplots(3,3,figsize=(15,15))
scene_i = 0
for axi,title in zip(ax, ['no interaction', 'central force', 'Vicsek model']):
    for axij in axi:
        axij.set_xlim(-1,1)
        axij.set_ylim(-1,1)
        axij.set_title(title+ f' scene:{scene_i}')
        axij.tick_params(labelbottom=False, labelleft=False, right=False, top=False, bottom=False, left=False)
        ms(scene_i).plot_trajectory(who='truth', ax=axij)
        axij.legend()
        scene_i +=1
plt.show()
_____no_output_____

## Creating Training and Validation Datasets_____no_output_____

n_scenes = 100
t_steps = 100
ms = MultiScene()
ms.add_bunch(n_scenes=n_scenes, n_body=5, t_steps=100, R_init=random_init, V_init=random_init, dt=1, r=4e-1, v=5e-2, interaction=zero_interaction)
ms.save('data/demo_train')

ms = MultiScene()
ms.add_bunch(n_scenes=n_scenes, n_body=5, t_steps=100, R_init=random_init, V_init=random_init, dt=1, r=2e-1, v=5e-2, interaction=zero_interaction)
ms.save('data/demo_val')_____no_output_____

### Extracting the features and targets_____no_output_____

ms_train = MultiScene(load='data/demo_train')
X_train = ms_train.get_features(joint=True) #features
V_train = ms_train.get_target(var='V', joint=True) #target velocities
A_train = ms_train.get_target(var='A', joint=True) #target accelerations

ms_val = MultiScene(load='data/demo_val')
X_val = ms_val.get_features(joint=True) #features
V_val = ms_val.get_target(var='V', joint=True) #target velocities
A_val = ms_val.get_target(var='A', joint=True) #target accelerations_____no_output_____

# CREATING A MODEL

## Training_____no_output_____

IN = InteractionNetwork(D_E=2, R_hidden_layers=[], O_hidden_layers=[], output_var='A', self_edges=False,  aggregator='sum', include_v=False, name='demo_model')
IN.fit(X_train, A_train, validation=(X_val, A_val), learning_rate=1e-1, epochs=10000, print_every=100)
IN.save('saved_models/'+IN._name)
IN = InteractionNetwork(load='saved_models/demo_model')_____no_output_____

### Visualizing the loss evolution for training and validation_____no_output_____

plt.plot([np.log10(x) for x in IN.history['loss']], label='training loss')
plt.plot([np.log10(x) for x in IN.history['validation']], label='validation loss')
plt.legend()
plt.show()_____no_output_____

### Computing position MSE for different length mulstistep rollout predictions_____no_output_____

ms_val.predict_trajectory(IN)
ms_val.compute_mse()
ms_val.save('data/demo_val')_____no_output_____fig, ax = plt.subplots( figsize=(12,6.))
ms_val.plot_mse(ax=ax)
ax.legend(loc="lower center", bbox_to_anchor=(0.6, 0.1), ncol=2)
ax.set_ylim(top=-0, bottom=-9)
ax.set_xlim(right=20, left=0)
plt.show()_____no_output_____

### Visualizing multistep rollout predictions_____no_output_____

n_scenes = 3
hspace = 0.0
wspace = 0.0

fig, ax = plt.subplots(n_scenes,2, figsize=(10,15))
for i, axi in enumerate(ax):
  for axii,model in zip(axi,['truth', 'demo_model']):
    ms_val(i+10).plot_trajectory_tex(who=model, ax=axii)
    if i==0: axii.set_title(f'{model}')
    if model=='truth': axii.set_ylabel(f'Scene {i+1}')
    axii.set_xlim(-1,1)
    axii.set_ylim(-1,1)
    axii.tick_params(labelbottom=False, labelleft=False, right=False, top=False, bottom=False, left=False)
    axii.legend()
  plt.subplots_adjust(wspace=wspace, hspace=hspace)
plt.show()_____no_output_____


# Notebook from sytseng/CS109b_Final_Project_Spring_2019
Path: Microbiome Data Augmentation.ipynb

Microbiome Data Augmentation Using Dynamic Time Warping (DTW) and  Barycentric Averaging (DBA)
_____no_output_____

from utils.utils import transform_labels

from augment import augment_train_set

import numpy as np
import pandas as pd
import random
import matplotlib.pyplot as plt

import pickle
from sklearn.manifold import MDS
from sklearn.manifold import LocallyLinearEmbedding
import seaborn as sns
import json_____no_output_____import numpy as np
import pandas as pd
_____no_output_____%matplotlib inline_____no_output_____

# Motivation

In machine learning, data augmentation is the process of creating synthetic examples in order to augment a dataset used to learn a model. Microbiome data is characterized with high dimensionality with very few samples obtained in a longitudinal study. Typical data augmentation for time series via spline interpolation combined with a Guassian noise would increase the resoltution within a microbiome sample. However this approach would not allow us to simulate new subjects. In our project, we have data from 9 mice at hand, each sampled ca. 75 times. Our data augmentation is to simulate additional mice.

Lately, applying Dynamic Time Warping (DTW) combined with non-parametric nearest neighbor (NN) has shown good results in time series classification. Especially the weighted version of time series by Barycentric Averaging (DBA) allows to create synthetic time series from a given set of time series. This method has been proposed by [G. Forestier a.l.](https://germain-forestier.info/publis/icdm2017.pdf) for augmenting sparse datasets. This technique of data augmentation has proven to drastically increase deep CNN’s accuracy. [H. Fawaz et.al.](https://germain-forestier.info/publis/aaltd2018.pdf) recently proposed a Python implementation of the DTW-DBA algorithm, wich we adapted for our project on microbiome timeseries.

The algorithm starts by choosing randomly a time series and gives it a weight of 1. Then, it finds its nearest neighbor and gives it a weight of 0.5. Then it defines an exponential decay function that maps a distance to a weight. The function is used to decide upon the weights for all time series. _____no_output_____# Data Augmentation by DTW-DBA

We cloned the [GitHub](https://github.com/hfawaz/aaltd18) repository of DTW-DBA implementation. On JupyterHub, we have to run the command `./utils/build-cython.sh` to compile the optimized DTW code written in Cython. _____no_output_____

from augment import augment_train_set_____no_output_____

The function below is a helper function takes a multivariate time series `X`, the corresponding labels vector `Y` and produces `2^nb_multiply` times additional time series.

In our microbiome context, `X` is a 3D matrix, where dimension 1 represents a subject (mouse), dimension 2 represents the time (sample) and finally dimension 3 represents an OTU. The matrix contains the abundance for the given subject, sample, time. The vector `Y` has a binary value, 0 for healthy subjects, and 1 for IBD subjects.

Assuming that `X` contains data for 4 healthy mice, and `nb_multiply` is 2, the function will return simulated samples for 12 'virtual' mice, for each time point and for each OTU._____no_output_____

Y, Y = transform_labels(ts_abundance_y_healthy, ts_abundance_y_healthy)
Y.shape, ts_abundance_y_healthy.shape_____no_output_____
def augment_ts(X, Y, nb_multiply=2):

    print("Remaining: ", nb_multiply)

    if nb_multiply==0:
        return X, Y
    #Y, Y = transform_labels(Y, Y)
    classes, classes_counts = np.unique(Y, return_counts=True)
    nb_classes = len(np.unique(Y, axis=0))
    _, classes_counts = np.unique(Y, return_counts=True)
    nb_prototypes = classes_counts.max()
    if len(X.shape) == 2:  # if univariate
            # add a dimension to make it multivariate with one dimension
            X = X.reshape((X.shape[0], X.shape[1], 1))
    syn_train_set = augment_train_set(X, Y, classes,nb_prototypes,limit_N=False,
                                 weights_method_name='as', distance_algorithm='dtw')
    # get the synthetic train and labels
    syn_X, syn_Y = syn_train_set
    # concat the synthetic with the reduced random train and labels
    aug_X = np.array(X.tolist() + syn_X.tolist())
    aug_Y = np.array(Y.tolist() + syn_Y.tolist())

    np.save('aug_X_{}.npy'.format(Y[0]), aug_X)
    np.save('aug_Y_{}.npy'.format(Y[0]), aug_Y)

    aug_X, aug_Y = augment_ts(aug_X, aug_Y, nb_multiply-1)

    return aug_X, aug_Y_____no_output_____

Below we import our mouse dataset._____no_output_____

data_folder = '../Microbiome_Module_Material/Part c full processed data/'
with open(data_folder + 'mouse_set_data_merged_20190508.pkl', 'rb') as handle:
    df_all = pickle.load(handle)
df_all.head()_____no_output_____

The function below takes the mice data frame and returns the matrix `X` and `Y`, which we need to augment._____no_output_____

def prepare_data(df, y):
    otus = df.otu.unique()
    mice = df.mouse.unique()
    time_i = df.time_i.unique()
    ts_abundance_x = np.empty([len(mice), len(time_i), len(otus)])
    ts_abundance_y = np.empty([len(mice)])
    ts_abundance_y[:] = y
    df = df.fillna(0)
    for otu_i, otu in enumerate(otus):
        df2 = df[df.otu==otu]
        df_ts = df2[['mouse','time_i','abundance']] \
                    .groupby(['mouse', 'time_i']).max().reset_index() \
                    .pivot(index='mouse', columns='time_i', values='abundance').reset_index().astype(float)
        ts_abundance_x[:,:,otu_i] = df_ts.values[:,1:]
    return ts_abundance_x, ts_abundance_y, otus, time_i_____no_output_____

### Data Augmentation for Healthy mice
_____no_output_____

df_all_healthy = df_all[df_all.group==1]
ts_abundance_x_healthy, ts_abundance_y_healthy, otus_healthy, time_i_healthy = prepare_data(df_all_healthy, y=0)
print('The original dataset for healthy mice is available for {} subjects, {} data points and {} OTUs'.format(ts_abundance_x_healthy.shape[0],
                                                                                ts_abundance_x_healthy.shape[1],
                                                                                ts_abundance_x_healthy.shape[2]))_____no_output_____

Now, we apply the DTW-DBA method to augment the time series. _____no_output_____

ts_abundance_x_healthy_aug, ts_abundance_y_healthy_aug = augment_ts(ts_abundance_x_healthy,
                                                                    ts_abundance_y_healthy,
                                                                    nb_multiply=10)
print('The augmented dataset for healthy mice contains {} subjects, {} data points and {} OTUs'.format(ts_abundance_x_healthy_aug.shape[0],
                                                                                ts_abundance_x_healthy_aug.shape[1],
                                                                                ts_abundance_x_healthy_aug.shape[2]))The augmented dataset for healthy mice contains 1024 subjects, 75 data points and 202 OTUs
with open('otus_healthy.txt', 'w') as f:
    f.write(json.dumps(list(df_all_healthy.otu.unique())))
with open('time_i_healthy.txt', 'w') as f:
    f.write(json.dumps(list(df_all_healthy.time_i.unique())))_____no_output_____

### Data Augmentation for IBD mice
_____no_output_____

df_all_ibd = df_all[df_all.group==2]
ts_abundance_x_ibd, ts_abundance_y_ibd, otus_ibd, time_i_ibd = prepare_data(df_all_ibd, y=1)
print('The original dataset for IBD mice is available for {} subjects, {} data points and {} OTUs'.format(ts_abundance_x_ibd.shape[0],
                                                                                ts_abundance_x_ibd.shape[1],
                                                                                ts_abundance_x_ibd.shape[2]))The original dataset for IBD mice is available for 5 subjects, 73 data points and 176 OTUs
ts_abundance_x_ibd_aug, ts_abundance_y_ibd_aug = augment_ts(ts_abundance_x_ibd,
                                                            ts_abundance_y_ibd,
                                                            nb_multiply=10)
print('The augmented dataset for IBD mice contains {} subjects, {} data points and {} OTUs'.format(ts_abundance_x_ibd_aug.shape[0],
                                                                                ts_abundance_x_ibd_aug.shape[1],
                                                                                ts_abundance_x_ibd_aug.shape[2]))The augmented dataset for IBD mice contains 1280 subjects, 73 data points and 176 OTUs
with open('otus_ibd.txt', 'w') as f:
    f.write(json.dumps(list(df_all_ibd.otu.unique())))
with open('time_i_ibd.txt', 'w') as f:
    f.write(json.dumps(list(df_all_ibd.time_i.unique())))_____no_output_____

# Visualization of Augmented Data

Below we plot the abundnance from a real mice versus synthetic mice for visual comparison. _____no_output_____

ts_abundance_x_healthy_aug = np.load('aug_X_0.0.npy')
ts_abundance_y_healthy_aug = np.load('aug_Y_0.0.npy')
ts_abundance_x_ibd_aug = np.load('aug_X_1.0.npy')
ts_abundance_y_ibd_aug = np.load('aug_Y_1.0.npy')_____no_output_____def plot_augmented_data(df_all, X, mice_dict, title):
    df = pd.DataFrame()
    otus = df_all.otu.unique()
    times = df_all.time_i.unique()
    f, ax = plt.subplots(1,1,figsize=(10,5))
    nb=1
    for label, mice_list in mice_dict.items():
        for i in mice_dict[label]:
            for j in range(len(times)):
                for k in [1, 5]:
                    f = pd.DataFrame({'i':nb, 'group': 1,  'mouse': i, 'time_i': times[j], 'abundance':X[i,j,k], 'otu': otus[k]},index=['i'])
                    df = df.append(f)
                    nb = nb + 1
            sns.lineplot(x="time_i", y="abundance",
                         data=df[df.mouse==i], ax=ax, label=label+' - mouse {}'.format(i))
    plt.title(title)
    plt.show()_____no_output_____plot_augmented_data(df_all_healthy,
                    ts_abundance_x_healthy_aug,
                    {'original':[0,2], 'simulated':[20, 25]},
                    title='OTU Abundance on Healthy Mice - Real vs Simulation')_____no_output_____plot_augmented_data(df_all_ibd,
                    ts_abundance_x_ibd_aug,
                    {'original':[6,9], 'simulated':[18, 30]},
                    title='OTU Abundance on IBD Mice - Real vs Simulation')
_____no_output_____

**Multidimensional Scaling (MDS)**

In order to have a spacial interpretation of the synthetic data generation, we visualize the result using  manifold learning — a class of unsupervised estimators that seeks to describe datasets as low-dimensional manifolds embedded in high-dimensional spaces [source](https://jakevdp.github.io/PythonDataScienceHandbook/05.10-manifold-learning.html).

Given high-dimensional microbiome time series, MDS seeks a low-dimensional representation of the data that preserves the distance between every pair of points._____no_output_____

def plot_augmented_data_2D(model, X_healthy=ts_abundance_x_healthy_aug, X_ibd=ts_abundance_x_ibd_aug):
    fig, ax = plt.subplots(1,2,figsize=(15,5))
    for i in range(X_healthy.shape[0]):
        out = model.fit_transform(X_healthy[i].T)
        if i<4:
            ax[0].scatter(out[:, 0], out[:, 1], color='blue', label='original - healthy')
        else:
            ax[1].scatter(out[:, 0], out[:, 1], color='green', label='simulated - healthy')
    for i in range(X_ibd.shape[0]):
        out = model.fit_transform(X_ibd[i].T)
        if i<5:
            ax[0].scatter(out[:, 0], out[:, 1], color='blue', label='original - IBD')
        else:
            ax[1].scatter(out[:, 0], out[:, 1], color='green', label='simulated - IBD')
    ax[0].set_title('MSD - Original Mouse Data \n blue - healthy, black - IBD')
    ax[1].set_title('MSD - Simulated Mouse Data \n blue - healthy, black - IBD')
    plt.axis('equal');_____no_output_____model_mds = MDS(n_components=2, random_state=1)
plot_augmented_data_2D(model_mds)_____no_output_____

**Locally linear embedding (LLE)**

Linear embeddings, which essentially consist of rotations, translations, and scalings of data into higher-dimensional spaces, breaks down when the embedding is nonlinear, what is the case for microbiome time series. Instead of preserving the distances between each pair of points, LLE preserves only the distances between neighboring points _____no_output_____

model_lle = LocallyLinearEmbedding(n_neighbors=10, n_components=2, method='modified',
                               eigen_solver='dense')
plot_augmented_data_2D(model_lle)_____no_output_____