# Notebook from carlomarxdk/algorithmic-fairness
Path: Week 10/exercises.ipynb

# Data analysis in the Amazon audit
Most of the materials in this notebook were kindly provided by Piotr Sapiezynski (www.sapiezynski.com)_____no_output_____### The motivation behind the analysis
Read it here
https://themarkup.org/amazons-advantage/2021/10/14/amazon-puts-its-own-brands-first-above-better-rated-products_____no_output_____

import pandas as pd
from sklearn.model_selection import train_test_split
import statsmodels.api as sm
from statsmodels.tools.tools import add_constant_____no_output_____# this is the data from the markup audit, downloaded from their github:
# https://github.com/the-markup/investigation-amazon-brands
data = pd.read_csv('https://www.sapiezynski.com/cs4910/markup/data.csv')
data.head()_____no_output_____

![features](https://mrkp-static-production.themarkup.org/graphics/amazon-methodology-product-comparison/1634143723511/assets/product-pairs-subtraction-desktop.png)_____no_output_____

columns = ['stars_delta', 'reviews_delta', 'is_amazon',
        'is_shipped_by_amazon', 'is_sold_by_amazon',
        'is_top_clicked', 'placed_higher']_____no_output_____

Exercise 1

Use a random forest classifier to predict `placed_higher` using the features `is_amazon`,`is_shipped_by_amazon`, `is_sold_by_amazon`,`is_top_clicked`, `stars_delta`, `reviews_delta`. What is the most important feature?

_____no_output_____You should get that `is_amazon` is the most important feature. How does it translate into the probability of being the top result? RandomForests won't tell us... But logistic regression can.

Quick reminder on regression and explainability - here applied to the amazon dataset.

**Linear** regression has the following formula:

$$ y = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... $$

This means that:
* when all variables are equal to 0, the outcome variable $y$ is equal to $\beta_0$, the intercept.
* a **unit** change in $x_1$ corresponds to $y$ changing by $\beta_1$

It might not be a powerful model, but it offers a clear interpretation/explanation in regression problems.

**Logistic** regression is mostly used for binary classification and it looks similar to the linear regression, but now the outcome variable is the logarithm of odds of success.

$$ ln(\frac{\pi}{1-\pi}) = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... $$

where $pi$ is the probability of success, so $1-\pi$ is the probability of failure.

Then, a unit change in $x_1$ corresponds to $\beta_1$ change in log odd ratios... but that is not intuitive at all. Let's rewrite it to calculate the probability directly:

$$\begin{align}
ln(\frac{\pi}{1-\pi}) &= \beta_0 + \beta_1x_1 + \beta_2x_2 + ... \\
\frac{\pi}{1-\pi} &= e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...} \\
{\pi} &= ({1-\pi})e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...} \\
{\pi} &= e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...}-\pi e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...}\\
{\pi}(1+e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...}) &= e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...} \\
{\pi} &= \frac{e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...}}{1+e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ...}}
\end{align}$$

So, while we cannot directly say from that a unit change of $x_1$ corresponds to a certain change in the probability of the outcome, we can just plug zeros and ones into the equation above and compute the changes._____no_output_____Let's load the data from scratch and do some cleaning (I'm dividing by two, because The Markup left the values of -2, 0, 2 so they never have a "unit" change. If we divide them by two, the unit change actually is meaningful)._____no_output_____

data = pd.read_csv('https://www.sapiezynski.com/cs4910/markup/data.csv')
data['is_shipped_by_amazon'] /= 2
data['is_sold_by_amazon'] /= 2
data['is_amazon'] /= 2
data['is_top_clicked'] /= 2
dataset = data[columns]
train, test = train_test_split(dataset, test_size=0.2, stratify=dataset['placed_higher'])_____no_output_____

Let's start with a model that only has the intercept (constant) and `is_amazon` as features._____no_output_____

# adding the intercept as a variable (add column of ones to the dataframe)
train = add_constant(train)
# selecting which features to train on
features = ['const', 'is_amazon']
# training a logistic regression model. First the dependent (outcome variable), then the features
log_reg = sm.Logit(train['placed_higher'], train[features]).fit()Optimization terminated successfully.
         Current function value: 0.490868
         Iterations 7
log_reg.summary()_____no_output_____

We will mostly be looking at the bottom table.

`const` (the intercept) translates to the probability if all other variables are equal to 0

$$ p = \frac{e^{\beta_{const}}}{1+e^{\beta_{const}}} $$_____no_output_____

import numpy as np
# the beta coefficients are stored in the .params list, with the constant being first
# get the constant out of the fit model:
const = log_reg.params[0]

print("P (placed higher) = %.3f" %(np.e**const/(1+np.e**const)))P (placed higher) = 0.516

This should be very close to 50%. In our case if all variables are zero, it means that either:
1. The products are both from amazon
2. The products are both not from amazon

We don't have a way to differentiate them, so it's a coin toss (50%)_____no_output_____The other coefficients describe the change in odds that corresponds to a unit change in the variable.
* if the coefficient is positive, it means that an **increase** of variable corresponds to an **increase** in the odds of the positive outcome
* if the coefficient is negative, it means that an **increase** of variable corresponds to an **decrease** in the odds of the positive outcome

So what happens when `is_amazon` is equal to 1, i.e. one product is an amazon product and the other isn't?
By looking at the `coef` we already know it's going to go up (because the coefficient is positive). But by How much? Let's calculate:

$$ p = \frac{e^{\beta_{const} + \beta_{is\_amazon}}}{1+e^{\beta_{const} + \beta_{is\_amazon}}} $$_____no_output_____

numerator = np.e**(log_reg.params[0] + log_reg.params[1])
print("P (placed higher) = %.3f" %(numerator/(1+numerator)))P (placed higher) = 0.937

That means that in this dataset 93\% of the time we have two products where one is from amazon and the other isn't, it's the amazon product that will be on the first place.

Solved? Well no, because maybe this is just because amazon products are better. Let's include the star difference in the analysis to **control** for this:_____no_output_____

features = ['const', 'is_amazon','stars_delta', ]
log_reg = sm.Logit(train['placed_higher'], train[features]).fit()
log_reg.summary()Optimization terminated successfully.
         Current function value: 0.494329
         Iterations 6

Now notice the last two columns - that's the 95% confidence interval. If one value is positive, and the other negative, the interval includes 0, which means we can't really tell if the effect is positive, or negative - it's not significant. Turns out, controling for the star rating doesn't change anything.

Let's add the review delta:

* **Write code to include `reviews_delta`**
* Is the `reviews_delta` *important*?_____no_output_____Let's include the rest of our variables:
* **Write code to include the other variables**
* Which variable is *important* among those included?
* How do you interpret its value? (*Hint*: look at the scenarios when this variable is equal to 0, and to 1)_____no_output_____What about the predictive performance? The Markup uses RandomForests with hyperparameter tuning to get the best prediction and ends up with 73.2% accuracy.
* **Calculate accuracy for your model**
* Which model would you give preference and why? _____no_output_____The problem that we looked at so far is defined as follows: given the top two items, which one is going to be placed in the first position?

Now, let's solve a slightly different problem formulation (not covered in the Markup write up): given characteristics of the product, how likely is it the top product on its search page?_____no_output_____

Excercise 2: Interpreting the meaning of logistic regression coefficients

In this exercise you will interpret the values of the coefficients of a logistic regression model that tries to predict whether a product was the top product on its search page.

In particular, please answer these questions:
1. What is the probability that a hypothetical product with the lowest rating, no reviews, not sold/shipped/produced by amazon, and not among the top clicked products will be placed at the top?
1. Increase in which variables corresponds to a higher probability of being placed at the top?
1. How do you interpret the `stars` coefficient and its statistical significance?
1. If a product receives one more review without changing any other of its characteristics, how does it affect its chances to placed at the top?
    * How about 1000000 more reviews?
1. What is the probability that a five-star product with 1000 reviews, produced, sold, and shipped by amazon that was among the top clicked results will be placed on the top of the result list?
    * Is this number higher, or lower than you expected? Does it change how you interpret the results of the model for only the top two positions? Why?

_____no_output_____

# let's get the data first:
dataset = pd.read_csv('https://www.sapiezynski.com/cs4910/markup/dataset_full.csv')
# the lowest star rating is 1, not 0, let's adjust the rating such that it starts at 0:
dataset['stars'] -= 1
dataset.head()_____no_output_____

This dataset is an aggregation of all first search result pages from the Markup audit that had any amazon products on them. The sponsored results are removed.

In this dataset each row is a product with its star rating, the number of reviews, the binary indicators of whether it's an amazon product, it's sold or shipped by amazon, whether it was among the most clicked products, and the outcome variable - whether it was the top product in its search results page.

We're looking at at most 24 first results, so on average only one in 24 rows is the top placed product:
_____no_output_____

print("Fraction of top placed products (is_top = 1): %.2f %%" %(dataset['is_top'].mean()* 100))Fraction of top placed products (is_top = 1): 4.55 %

**You code here**_____no_output_____


# Notebook from PyTrans/Artificial-Intelligence
Path: A Model Based on Deep Learning for Predicting Travel Mode Choice.ipynb

# A Model Based on Deep Learning for Predicting Travel Mode Choice_____no_output_____## ABSTARCT
Recognizing the limitations of previous travel mode choice models such as random utility models and multi-layer perceptron neural network models, this study develops a framework using a deep neural network with deep learning schemes, to predict travelers’ mode choice behavior. Deep neural networks and deep learning are relatively newer models, applied mostly so far to pattern recognition, image/voice processing, and for big data analytics. We develop such a choice model with a structure that is appropriate for the travel mode choice problem, and demonstrate the success of the model using an available dataset. The research also develops an important component of the model that takes into account the inherent characteristics of choice models that all individuals have different choice alternatives, an aspect not considered in the neural network models of the past that led to poorer performance. The proposed model is compared against existing mode choice models. The results prove that the new model clearly outperforms the previous mode choice models.

_____no_output_____## Prerequisite
Scikit-learn
pip install -U sklearn

Keras
pip install keras

If you use GPU
install tensorflow, Cuda, Cuda DNN_____no_output_____

import pandas as pd
import itertools
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
import numpy as np
import gzip
import pickle
import datetime
import os
from keras.models import Sequential
from keras.layers import Dense, Dropout,Add
import keras
from keras.optimizers import SGDUsing TensorFlow backend.

## Introduction

The advantage of Artificial Intelligence (AI) is recognized in various fields. This paper aims to implement the concept of deep learning (DL) algorithms, one branch of the AI family, for a prediction model of travel mode choice. We will compare the proposed model to existing travel choice models. Random utility models (RUMs) are traditionally used to predict travelers’ choice, but such models typically have strong assumptions along with limitations in their accuracy. RUMs assume that individuals select the alternative giving maximum utility and that an individual’s utility can be calculated with linear combinations of deterministic elements and unseen errors. Based on the random utility model, researchers proposed various types of discrete choice models (logit model, nested logit model, cross–nested logit model, and probit model) in order to better capture individuals’ choice behaviors (1,2,3,4,5,6 and 7). The random utility models however have inherent limitations since an individual’s choice is not as simple as in the assumptions. Mode choice models play an active and pivotal role in a transportation planning process. In transportation planning models, a mode choice model is necessary to predict the demands for each mode of travel (such as autos, bus, rail, and walk). In addition, the market share ratio (modal split ratio), which can be inferred from the choice model, is used as a key index by policy makers. Thus, the quality of the behavior representation affects not only the reliability of the transportation models, but also the success of the transportation policies.

With various large datasets currently becoming available in the transportation domain, developing novel algorithms is necessary for improving prediction accuracy. Several researchers and companies are extending their efforts towards the machine learning area, which has shown success in various tasks related to big data (8). We might be able to obtain information from various sources such as digital maps, real-time transportation information, users’ profiles, and users’ activity logs from smart-phone based applications. Several companies, such as Google, Microsoft, and Amazon, are now able to acquire their users’ personal preferences. Furthermore, with the development of transportation network services such as by Uber, Lyft, Zip Car, etc., and applications such as Google directions, predicting travel choices at the individual level becomes even more important. Traditional transportation models, however, cannot benefit from such large datasets because they have mainly focused on inferring population behaviors from canonical sample data sets.

Deep learning is one promising area in Big Data Analytics. It is commonly applied to computer vision, pedestrian detection, language modeling, picture classification, and speech recognition (9,10, and 11). However, in the transportation field, to the best of the authors’ knowledge, a deep learning-based mode choice model is yet to be developed.

RUMs such as NLM and Cross Nested Logit model (CNLM) in predicting travel mode choice (16 ,17 and 18). In summary, the results from past research are somewhat mixed in comparing the AI approaches have been used to analyze travel behaviors or predict traffic conditions since the late 1990s, but the performance of such approaches was often disputed. Some researchers argued that AI models do not guarantee improvements compared to traditional models (12, 13, 14, and 15). Multi-layer perceptron models (MLP) were the most commonly used AI scheme for mode choice models. From several experiments, Carvalho et al. (12) concluded that it is hard to say that MLP captures travelers’ choice behavior better than a logit model, and that AI brings no advantages in computing times for optimization either. Hensher and Ton (13) compared the performance of Nested Logit Models (NLM) against MLP in modeling commuter mode choice. They found that although ANN forecasts individuals’ choices better than NLM, NLM predicts market share ratios slightly better. Cantarella and Luca (15) examined a multilayer feed-forward neural network model by comparing it with random utility models (RUMs), but they could not determine which model was better, as two case studies showed different results. There are also some recent studies that show AI models to outperform performance of traditional discrete choice models and neural network models.

Since 2006, when Hinton proposed several techniques for deep learning structures, configuring deep networks with more than three layers have shown widespread success in training neural networks. In the past, using more than 2 layers and a large number of perceptrons in neural networks generally proved to be unsuccessful (9). We find that previous AI based travel mode choice models can also be regarded as using shallow network structures with a single layer and a small number of perceptrons. This was however mainly because a high number of layers would result in an over-fitting problem, as many realized, in addition to the computing time being considerable. As is known, overfitting would make the estimated model fit closely to the sample training set, but perform poorly on a real dataset. With the evolution of deep learning theory, these problems can be tackled with several newer techniques: efficient initialization, stochastic gradient descent method, regularization, dropout, etc. In addition, parallel processors such as the Graphics Processing Unit (GPU) can boost the computing speed, which has resulted in significant increase the application of DL significantly.

This study proposes Deep Neural Network (DNN), one among a family of deep learning algorithms, to predict travelers’ mode choice behavior. The next section explains the experimental data used in the study. Section 3 briefly describes previous mode choice models such as random utility models and MLPs, and introduces DNN. In the following section, we describe how we construct our mode choice models. The performance of each model is evaluated in the fifth section. Finally, we will close our paper with conclusions and future research.
_____no_output_____## 2. Experimental Data

To explore the performance of the Deep Neural Network model, we utilized data from a mode choice survey of long-distance travel. Abay (19) conducted the Revealed Preference (RP) and Stated Preference (SP) surveys to estimate the hypothetical demand for SWISS Metro, a new innovative intercity passenger transport, in Switzerland. Several studies utilized this data for evaluating their proposed model. (6, 19, 20, 21, and 22) The SP survey data is available on the Biogeme website with discrete choice estimation packages (20). There are three alternative travel modes in the choice set: car (only for car owner), rail, and Swiss metro. Car and rail are existing modes and Swiss Metro (SM) is a hypothetical mode. The dataset also contains various attributes such as travel time, travel cost, age, luggage, currently available mode, annual season pass, number of seats, and frequency. (6)

The total number of individual observations is 6,768. Prior machine learning research partitioned their observations into two subsets: learning set and test set (12,15,17, and 18) or three subsets: learning set, validation set, and test set (16). Our research separates the observations into two subsets. The total number of the sample is equal to a sum of the learning set and the test set. The learning set is independent from the test set, and they have no common observations. The learning set is used to train a model by pairing the input with the selected alternatives, which can be regarded as “supervised learning”. The degree to which the trained model explains the travelers’ choice behavior is evaluated by applying the model to the test set. A small learning set could induce the over-fitting problem, since a small number of data is unlikely to represent the behavior of all participants. Inferring the behavior of entire participants from a high portion of learning set could have high explanatory power for the observation set. But allocating high learning rate is undesirable in that a small test set could also be biased. This research will examine the effect of learning rate to these stated problems by changing the rate from zero to one. Furthermore, we will also delve into the prediction power relationship between learning set and test set.
_____no_output_____

filename = "./data/swissmetro.csv"
df = pd.read_csv(filename)
df.describe()_____no_output_____

We only choose the selected trip purpose(1,3,5,7) as Bierlaire's analysis._____no_output_____

purpose = ["1","3","5","7",1,3,5,7]
df = df[df['PURPOSE'].isin(purpose) == True]_____no_output_____

Our utility functions are configured as the utility functions in Bierlaire et al. (6). They found that both travel time and travel cost generally affect all travel modes’ choice probability. The relevance of other variables depended on the modes. Luggage only influences the car choice. Frequency, annual season pass, and age are selected as additional explanatory variables for rail. The probability of selecting the Swiss Metro (SM) alternative is dependent on frequency, annual pass, and number of seats. In our work, the multiple training sets are imported to the Biogeme software (20) and the estimated models are tested on both the test sets, and the total observation set. In all trials in both models, the input variables, except for “number of seats” and “constant for car”, are statistically significant since the p-value of each variable is lower than 0.5. The two variables just mentioned are insignificant over half the training sets, which make us drop them from the input variable set. All estimated models have a pseudo ρ^2 less than 0.260 except for one case. Thus, we concluded that the models are indicative of a good model fit.

Instead of including all possible explanatory variables in MLPs and DNN, identifying significant variables from traditional discrete choice models, such as a nested logit model, could increase computational efficiency, since a large number of input variables to a neural network could make it too complex a network to train. DNN has a considerably complex network structure with many layers. All perceptrons in the first hidden layer are directionally linked from input variables and this affects the results. The concept of selecting input variables from a traditional model is very significant, in that we can avoid unnecessary computing processes._____no_output_____

variables = ["TRAIN_TT", "TRAIN_CO", "CAR_TT", "CAR_CO", "SM_TT", "SM_CO","AGE","GA","LUGGAGE","TRAIN_HE","SM_HE"]
choice = "CHOICE"
choicesetkey = {1:"Train", 2:"SM", 3:"Car"}
_____no_output_____

### Building data for Deeplearing_____no_output_____

df["choice"] = df[choice]
df.loc[df.GA ==0,"TRAIN_CO"]=0
df.loc[df.GA ==0,"SM_CO"]=0
purposevector = df["PURPOSE"]
#To make a discrete choice array from choice column
for i in choicesetkey:
    df[choicesetkey[i]] = 0
    df.loc[df[choice] == i, choicesetkey[i]] = 1.0
newvars = []
newvars.extend(variables)
newvars.extend(choicesetkey.values())
newvars.append("choice")
inputvars = ["TRAIN_AV", "SM_AV","CAR_AV"]
inputvars.extend(newvars)
data = df[inputvars]
data["PURPOSE"] = df["PURPOSE"]
data["CHOICE"] = df["choice"]
data.loc[:,["SM_AV"]]=df["SM_AV"]
data.loc[:,["CAR_AV"]] = df["CAR_AV"]
data.loc[:,["TRAIN_AV"]] = df["TRAIN_AV"]
avset = pd.DataFrame()

data["SP"] = df["SP"]
population = df[newvars]
pop_train = df.loc[df[choice] == 1]
pop_sm = df.loc[df[choice] == 2]
pop_car = df.loc[df[choice] == 3]
starttime = datetime.datetime.now()C:\Users\Daisik\anaconda3\lib\site-packages\ipykernel_launcher.py:16: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  app.launch_new_instance()
C:\Users\Daisik\anaconda3\lib\site-packages\ipykernel_launcher.py:17: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
C:\Users\Daisik\anaconda3\lib\site-packages\ipykernel_launcher.py:23: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
def createdataset(data, variables, choicesetkey, df):
    normalized_data=(data-df.min())/(df.max()-df.min())
    revealed = data["choice"].values
    datax = normalized_data[variables].values
    datay = data[choicesetkey.values()].values
    X = datax
    Y = datay
    X = np.asarray(X).astype("float32")
    Y = np.asarray(Y)
    #X=X.reshape(X.shape[0], 11, 1)
    return X, Y, revealed_____no_output_____

### Training Strategy - training (learning) set ratio_____no_output_____

samplerate = 0.7
if samplerate < 1:
    train, test = train_test_split(data, test_size=1 - samplerate)

else:
    train, test = data.copy(), data.copy()_____no_output_____train.loc[:,["CHOICE"]] = train["choice"]
test.loc[:,["CHOICE"]] = test["choice"]_____no_output_____

### Availability of Alternatives Function (AAF)
The next item to discuss in the DNN structure in Figure 3 refers to the alternatives in the choice set of individuals. In a choice prediction model, alternatives that are unavailable to certain individuals should be addressed. Without this, an unavailable alternative can be predicted as their choice. For instance, for individuals who do not own a vehicle, that mode should be eliminated from the choice set. In conventional discrete choice models, unavailable alternatives are controlled by increasing the alternative’s utility, for a neural network, we need to develop a method to handle the issue.
Thus, we propose the Availability of Alternatives Function (AAF) to add to the DNN, shown as a vertical box in Figure 3. From the feedforward process in Eq (5) below, the dimension of the perceptron in the last hidden layer becomes same as the number of alternatives. Previous DNN models have not dealt with unavailable alternatives. Thus these alternatives could also have a value in those models, since the softmax function in the last step (Eq 7) calculates each alternative’s choice probability based on the last layer’s output. Thus, it is important to force the probability of unavailable alternatives to be zero, as in Eq (6). Otherwise, the model might predict unreasonable results to certain individuals. AAF controls the value of the output hidden layer (z_j). If the alternative j is unavailable for individual n, the z_j is altered to negative infinity, so the probability of j becomes to zero.
_____no_output_____

def makeavset(dt):
    avset = pd.DataFrame()
    modes = ['TRAIN_AV', 'SM_AV', "CAR_AV"]
    avset = dt[modes]
    avset.loc[avset.CAR_AV == 0, "CAR_AV"] = -99999.0
    avset.loc[avset.TRAIN_AV == 1, "TRAIN_AV"] = 0.0
    avset.loc[avset.SM_AV == 1, "SM_AV"] = 0.0
    avset.loc[avset.CAR_AV == 1, "CAR_AV"] = 0.0
    return avset_____no_output_____AAF=True
trainav = makeavset(train)
testav = makeavset(test)
popav = makeavset(data)

pop_train_av = makeavset(pop_train)
pop_sm_av = makeavset(pop_sm)
pop_car_av = makeavset(pop_car)
print ("1-3", datetime.datetime.now())
train.drop(['PURPOSE','TRAIN_AV','SM_AV',"CAR_AV","CHOICE","SP"], axis=1, inplace=True)
test.drop(['PURPOSE','TRAIN_AV','SM_AV',"CAR_AV","CHOICE","SP"], axis=1, inplace=True)
print ("1-4", datetime.datetime.now())
dataset = (train, test, population, pop_train, pop_sm, pop_car, trainav, testav, popav,pop_train_av, pop_sm_av, pop_car_av)
print ("2", datetime.datetime.now())1-3 2020-05-07 01:14:25.705519
1-4 2020-05-07 01:14:25.706516
2 2020-05-07 01:14:25.706516
(trainx, trainy, trainrevealed) = createdataset(train, variables, choicesetkey,df)
(testx, testy, validrevealed) = createdataset(test, variables, choicesetkey,df)
(popx, popy, poprevealied) = createdataset(population, variables, choicesetkey,df)
(pop_train_x, pop_train_y,pop_train_revealed) = createdataset(pop_train, variables, choicesetkey,df)
(pop_sm_x, pop_sm_y, pop_sm_revealed) = createdataset(pop_sm, variables, choicesetkey,df)
(pop_car_x, pop_car_y, pop_car_revealed) = createdataset(pop_car, variables, choicesetkey,df)_____no_output_____

### Deep Learning Model Design

The mechanisms of both MLP and DNN are similar. Both structures have hidden layers, neurons, activation functions for each hidden layer, a number of epochs for training, and a learning rate. An epoch, as is well-known, is a single pass through the whole training set. We set the learning rate at 0.001. The cost function we set for these models is a cross-entropy function (Eq 8), which can be derived from the maximum likelihood principle (27). The last hidden layer of both types of neural networks is connected to the sigmoid activation function for our travel choice model. This enables us to attain each alternative’s choice possibility, which is a similar output as from RUMs. The ReLU function inherently generates a zero value when an input value is less than zero. From our preliminary experience, we found that ReLU could not provide each alternative’s choice probability. This is a serious drawback since calculating a market share ratio from the model is an important purpose of a choice model. It does not mean that applying ReLU function is not suitable to the travel choice model because ReLU function is experimentally known to increase the prediction accuracy (28 and 29), and is efficient in optimization process (9). Details of our neural network structures are discussed in the next paragraphs.

The proposed model is a deep neural network (DNN) that uses an RRRS structure, based on the MLP results that showed the RRRS hidden layer structure to outperform the SSSS structure. This DNN model is called DNN(RRRS). Finally, we constructed our proposed model (DNN-A) with AAF (Availability of Alternatives Function) as in Eq (6), and as shown above in Figure 3. A 2000-epoch training scheme is selected after we observed that too many epochs induce the over-fitting problem, as indicated in Figure 4 for DNN-A. At a certain epoch number, the log likelihood of the test set becomes maximum, whereas that of training set keeps increasing as the iterations keeps processing, which is over-fitting.

As part of the deep-learning used in our work, we initialize the neural networks with Xavier’s initialization technique as in Glorot and Bengio (26).

A certain ratio of neurons in each network is randomly dropped out to prevent overfitting. This ratio, and the learning set ratio are important factors in the model’s performance. To determine the appropriate dropout ratio and learning set ratio, a sensitivity analysis of the network’s performance is conducted with the results shown in Figure 5 (a). The percentage of correct prediction is calculated by applying the trained model to test data set. In each combination, the average value of the performance index is calculated from 10 replica sets. When the dropout ratio is below 0.3, there are critical gaps between the trained model performance and test set performance. When the dropout ratio is too high, the accuracy is too low. At a dropout ratio of 0.5, the model is very stable, with a learning set ratio of 0.5 (Figure 5 (b)), which means that a small size of learning data set could also explain overall observations without an over-fitting issue. Finally, a deep neural network with a dropout ratio of 0.5 and a learning set ratio of 0.7 is implemented to the dataset and the results are compared with previously discussed models in the next section._____no_output_____

input1 = keras.layers.Input(shape=(11,))
h1=keras.layers.Dense(200, activation='relu',use_bias=True, kernel_initializer=keras.initializers.glorot_normal(seed=None),bias_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.05, seed=None))(input1)
d1=keras.layers.Dropout(0.5)(h1)
h2=keras.layers.Dense(200, activation='relu',use_bias=True, kernel_initializer=keras.initializers.glorot_normal(seed=None),bias_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.05, seed=None))(d1)
d2=keras.layers.Dropout(0.5)(h2)
h3=keras.layers.Dense(200, activation='relu',use_bias=True, kernel_initializer=keras.initializers.glorot_normal(seed=None),bias_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.05, seed=None))(d2)
d3=keras.layers.Dropout(0.5)(h3)
# h4=keras.layers.Dense(200, activation='sigmoid',use_bias=True, kernel_initializer=keras.initializers.glorot_normal(seed=None),bias_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.05, seed=None))(d3)
# d4=keras.layers.Dropout(0.5)(h4)
h5=keras.layers.Dense(200, activation='sigmoid',use_bias=True, kernel_initializer=keras.initializers.glorot_normal(seed=None),bias_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.05, seed=None))(d3)
d5=keras.layers.Dropout(0.5)(h5)
h6=keras.layers.Dense(3)(d5)
if AAF==True:
    input2 = keras.layers.Input(shape=(3,))
    #k21 = keras.layers.Dense(3)(input2)
    added = keras.layers.Add()([h6, input2])
    out = keras.layers.Softmax()(added)
    model = keras.models.Model(inputs=[input1, input2], output=out)
else:
    out = keras.layers.Softmax()(h6)
    model = keras.models.Model(inputs=input1, output=out)

C:\Users\Daisik\anaconda3\lib\site-packages\ipykernel_launcher.py:18: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=[<tf.Tenso..., outputs=Tensor("so...)`

To estimate the weighted matrix of each layer, we applied a stochastic gradient descent algorithm (SGD), which is known to be effective for parallelizing over multiple processors such as GPUs. TensorFlow (8) with python 2.7 is used to analyze both MLP and DNN by fully utilizing a Parallel GPU computing environment. GPU has 2048 processors boosting the optimization speeds. The tests are conducted under Ubuntu 14.04 with Intel I5 3.30 GHz quad core CPU and 16GB memory. Average training time of 0.7 learning set rate with this condition is 46.24 seconds. When GPU is not employed, training the model takes 451.13 seconds, on average._____no_output_____

opt = SGD(lr=0.01, momentum=0.9)
model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])_____no_output_____

### Training a model
_____no_output_____

if AAF==True:
    model.fit([trainx, trainav],trainy, epochs=2000, batch_size=100)

else:
    model.fit(trainx, trainy, epochs=2000, batch_size=100)
    Epoch 1/2000
4939/4939 [==============================] - 0s 37us/step - loss: 1.0200 - accuracy: 0.5331
Epoch 2/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.9222 - accuracy: 0.5772
Epoch 3/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.9030 - accuracy: 0.5951
Epoch 4/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8894 - accuracy: 0.5999
Epoch 5/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8881 - accuracy: 0.6003
Epoch 6/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8850 - accuracy: 0.6036
Epoch 7/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8743 - accuracy: 0.6062
Epoch 8/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8734 - accuracy: 0.6121
Epoch 9/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8800 - accuracy: 0.6084
Epoch 10/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8803 - accuracy: 0.6048
Epoch 11/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8739 - accuracy: 0.6133
Epoch 12/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8699 - accuracy: 0.6133
Epoch 13/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8727 - accuracy: 0.6080
Epoch 14/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8624 - accuracy: 0.6147
Epoch 15/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8686 - accuracy: 0.6019
Epoch 16/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8580 - accuracy: 0.6131
Epoch 17/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8599 - accuracy: 0.6100
Epoch 18/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8473 - accuracy: 0.6129
Epoch 19/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8436 - accuracy: 0.6115
Epoch 20/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8409 - accuracy: 0.6119
Epoch 21/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8413 - accuracy: 0.6094
Epoch 22/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8342 - accuracy: 0.6121
Epoch 23/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8308 - accuracy: 0.6092
Epoch 24/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8291 - accuracy: 0.6096
Epoch 25/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8359 - accuracy: 0.6003
Epoch 26/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8308 - accuracy: 0.6058
Epoch 27/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8346 - accuracy: 0.6028
Epoch 28/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8272 - accuracy: 0.6106
Epoch 29/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8330 - accuracy: 0.5934
Epoch 30/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8342 - accuracy: 0.5969
Epoch 31/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8230 - accuracy: 0.6106
Epoch 32/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8252 - accuracy: 0.6165
Epoch 33/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8213 - accuracy: 0.6104
Epoch 34/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8172 - accuracy: 0.6109
Epoch 35/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8284 - accuracy: 0.6066
Epoch 36/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8187 - accuracy: 0.6068
Epoch 37/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8204 - accuracy: 0.6111
Epoch 38/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8191 - accuracy: 0.6076
Epoch 39/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8181 - accuracy: 0.6133
Epoch 40/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8124 - accuracy: 0.6070
Epoch 41/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8161 - accuracy: 0.6084
Epoch 42/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8152 - accuracy: 0.6113
Epoch 43/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8109 - accuracy: 0.6131
Epoch 44/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8263 - accuracy: 0.6005
Epoch 45/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8113 - accuracy: 0.6109
Epoch 46/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8136 - accuracy: 0.6104
Epoch 47/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8101 - accuracy: 0.6098
Epoch 48/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8134 - accuracy: 0.6129
Epoch 49/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8175 - accuracy: 0.6054
Epoch 50/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8073 - accuracy: 0.6106
Epoch 51/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8063 - accuracy: 0.6109
Epoch 52/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8078 - accuracy: 0.6161
Epoch 53/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8054 - accuracy: 0.6074
Epoch 54/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8072 - accuracy: 0.6137
Epoch 55/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8090 - accuracy: 0.6052
Epoch 56/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8060 - accuracy: 0.6082
Epoch 57/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8065 - accuracy: 0.6048
Epoch 58/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8033 - accuracy: 0.6151
Epoch 59/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8109 - accuracy: 0.5983
Epoch 60/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8007 - accuracy: 0.6086
Epoch 61/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8035 - accuracy: 0.6106
Epoch 62/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8080 - accuracy: 0.6009
Epoch 63/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8096 - accuracy: 0.6084
Epoch 64/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8048 - accuracy: 0.6072
Epoch 65/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8033 - accuracy: 0.6109
Epoch 66/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8021 - accuracy: 0.6104
Epoch 67/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8029 - accuracy: 0.6074
Epoch 68/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8022 - accuracy: 0.6040
Epoch 69/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8010 - accuracy: 0.6115
Epoch 70/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8016 - accuracy: 0.6106
Epoch 71/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.8047 - accuracy: 0.6102
Epoch 72/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8011 - accuracy: 0.6080
Epoch 73/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.7993 - accuracy: 0.6129
Epoch 74/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.8026 - accuracy: 0.6082
Epoch 75/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.7982 - accuracy: 0.6133
Epoch 76/2000
4939/4939 [==============================] - 0s 18us/step - loss: 0.7995 - accuracy: 0.6090
Epoch 77/2000
4939/4939 [==============================] - 0s 17us/step - loss: 0.7985 - accuracy: 0.6054
Epoch 78/2000

### Evaluation_____no_output_____

if AAF==True:
    _, acc = model.evaluate([testx, testav], testy)
    predicted=model.predict([testx, testav])
else:
    _, acc = model.evaluate(testx, testy)
    predicted = model.predict(testx)2117/2117 [==============================] - 0s 26us/step
print("accuracy:", acc)accuracy: 0.6854038834571838


# Notebook from akbernamazi/AQI
Path: Progress_AIQ.ipynb

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import os
import matplotlib.pyplot as plt
import seaborn as sns

# Warnings
import warnings
warnings.filterwarnings('ignore')_____no_output_____print(os.getcwd())C:\Users\Neha\Desktop\3-2\Project4
data=pd.read_csv('C:/Users/Neha/Desktop/3-2/MP4//data.csv',encoding="ISO-8859-15")_____no_output_____data.shape_____no_output_____##data.fillna(0, inplace=True)
data.head()_____no_output_____

## preprocessing the data
Here we are going to keep only the needed columns
i.e removing stn_code,agency,location_monitoring_station,pm2_5_____no_output_____

data = data[['sampling_date','state','location','so2','no2','date']]_____no_output_____#removing any rows with NaN type od date
data = data.dropna()_____no_output_____data.head()_____no_output_____

From here we start our calculation of Air Quality Index for the gases:
1. SO2
2. NO2_____no_output_____### What is sulfur dioxide?

Sulfur dioxide is a gas. It is invisible and has a nasty, sharp smell. It reacts easily with other substances to form harmful compounds, such as sulfuric acid, sulfurous acid and sulfate particles.

About 99% of the sulfur dioxide in air comes from human sources. The main source of sulfur dioxide in the air is industrial activity that processes materials that contain sulfur, eg the generation of electricity from coal, oil or gas that contains sulfur. Some mineral ores also contain sulfur, and sulfur dioxide is released when they are processed. In addition, industrial activities that burn fossil fuels containing sulfur can be important sources of sulfur dioxide.

Sulfur dioxide is also present in motor vehicle emissions, as the result of fuel combustion. In the past, motor vehicle exhaust was an important, but not the main, source of sulfur dioxide in air. However, this is no longer the case._____no_output_____

data[['so2','state']].groupby(["state"]).median().sort_values(by='so2',ascending=False).head(10).plot.bar(color='r' , title = ' TOP 10 Cities ')
plt.show()_____no_output_____#function for calculating the AQI base on the concentration of SO2
def AQIForSO2(so2):
    if(0<=so2 and so2<=40):
        I = ((50*so2)/(40))
    elif(41<=so2 and so2<=80):
        I = (((100-51)*(so2-41))/(80-41))+51
    elif(81<=so2 and so2<=380):
        I = (((200-101)*(so2-81))/(380-81))+101
    elif(381<=so2 and so2<=800):
        I = (((300-201)*(so2-381))/(800-381))+201
    elif(801<=so2 and so2<=1600):
        I = (((400-301)*(so2-801))/(1600-801))+301
    else:
        I = (((500-401)*(so2-1600))/(1600))+401
    return I_____no_output_____#including SI column in dataset
data['si']=data['so2'].apply(AQIForSO2)
df= data[['so2','si']]
df.head()
_____no_output_____data[['date','si']].groupby(["date"]).median().sort_values(by='si',ascending=False).plot.line(color='r')
plt.show()_____no_output_____

### What is Nitrogen Dioxide (NO2)

As for PM and O3, the evidence on NO2 and health comes from different sources of information, including observational epidemiology, controlled human exposures to pollutants and animal toxicology. The observational data are derived from studies outdoors where NO2 is one component of the complex mixture of different pollutants found in ambient air and from studies of NO2 exposure indoors where its sources include unvented combustion appliances._____no_output_____

data[['no2','state']].groupby(["state"]).median().sort_values(by='no2',ascending=False).head(10).plot.bar(color='g',title="Top 10 Cities")
plt.show()_____no_output_____#function for calculating the AQI base on the concentration of SO2
def AQIForNO2(no2):
    if(0<=no2 and no2<=40):
        NI = ((50*no2)/(40))
    elif(41<=no2 and no2<=80):
        NI = (((100-51)*(no2-41))/(80-41))+51
    elif(81<=no2 and no2<=180):
        NI = (((200-101)*(no2-81))/(180-81))+101
    elif(181<=no2 and no2<=280):
        NI = (((300-201)*(no2-181))/(280-181))+201
    elif(281<=no2 and no2<=400):
        NI = (((400-301)*(no2-281))/(400-281))+301
    else:
        NI = (((500-401)*(no2-400))/(400))+401
    return NI_____no_output_____#including NI in dataset
data['ni']=data['no2'].apply(AQIForNO2)
df= data[['no2','ni']]
df.head()_____no_output_____#keeping the highest (NI or SI) for AQI of the place
def chooseAQI(si,ni):
    aqi=ni
    if(si>ni):
        aqi=si
    return aqi_____no_output_____data['AQI']=data.apply(lambda x:chooseAQI(x['si'],x['ni']),axis=1)
df= data[['sampling_date','state','si','ni','AQI']]
df.head()_____no_output_____df.tail()_____no_output_____
_____no_output_____


# Notebook from deepkick/pydata-book
Path: ch07.ipynb

# Data Cleaning and Preparation_____no_output_____

import numpy as np
import pandas as pd
PREVIOUS_MAX_ROWS = pd.options.display.max_rows
pd.options.display.max_rows = 20
np.random.seed(12345)
import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))
np.set_printoptions(precision=4, suppress=True)_____no_output_____

## Handling Missing Data_____no_output_____

string_data = pd.Series(['aardvark', 'artichoke', np.nan, 'avocado'])
string_data
string_data.isnull()_____no_output_____string_data[0] = None
string_data.isnull()_____no_output_____

### Filtering Out Missing Data_____no_output_____

from numpy import nan as NA
data = pd.Series([1, NA, 3.5, NA, 7])
data.dropna()_____no_output_____data[data.notnull()]_____no_output_____data = pd.DataFrame([[1., 6.5, 3.], [1., NA, NA],
                     [NA, NA, NA], [NA, 6.5, 3.]])
cleaned = data.dropna()
data_____no_output_____cleaned_____no_output_____data.dropna(how='all')_____no_output_____data[4] = NA
data
data.dropna(axis=1, how='all')_____no_output_____df = pd.DataFrame(np.random.randn(7, 3))
df.iloc[:4, 1] = NA
df.iloc[:2, 2] = NA
df_____no_output_____df.dropna()_____no_output_____df.dropna(thresh=2)_____no_output_____

### Filling In Missing Data_____no_output_____

df.fillna(0)_____no_output_____df.fillna({1: 0.5, 2: 0})_____no_output______ = df.fillna(0, inplace=True)
df_____no_output_____df = pd.DataFrame(np.random.randn(6, 3))
df.iloc[2:, 1] = NA
df.iloc[4:, 2] = NA
df
df.fillna(method='ffill')
df.fillna(method='ffill', limit=2)_____no_output_____data = pd.Series([1., NA, 3.5, NA, 7])
data.fillna(data.mean())_____no_output_____

## Data Transformation_____no_output_____### Removing Duplicates_____no_output_____

data = pd.DataFrame({'k1': ['one', 'two'] * 3 + ['two'],
                     'k2': [1, 1, 2, 3, 3, 4, 4]})
data_____no_output_____data.duplicated()_____no_output_____data.drop_duplicates()_____no_output_____data['v1'] = range(7)
data.drop_duplicates(['k1'])_____no_output_____data.drop_duplicates(['k1', 'k2'], keep='last')_____no_output_____

### Transforming Data Using a Function or Mapping_____no_output_____

data = pd.DataFrame({'food': ['bacon', 'pulled pork', 'bacon',
                              'Pastrami', 'corned beef', 'Bacon',
                              'pastrami', 'honey ham', 'nova lox'],
                     'ounces': [4, 3, 12, 6, 7.5, 8, 3, 5, 6]})
data_____no_output_____meat_to_animal = {
  'bacon': 'pig',
  'pulled pork': 'pig',
  'pastrami': 'cow',
  'corned beef': 'cow',
  'honey ham': 'pig',
  'nova lox': 'salmon'
}_____no_output_____lowercased = data['food'].str.lower()
lowercased
data['animal'] = lowercased.map(meat_to_animal)
data_____no_output_____data['food'].map(lambda x: meat_to_animal[x.lower()])_____no_output_____

### Replacing Values_____no_output_____

data = pd.Series([1., -999., 2., -999., -1000., 3.])
data_____no_output_____data.replace(-999, np.nan)_____no_output_____data.replace([-999, -1000], np.nan)_____no_output_____data.replace([-999, -1000], [np.nan, 0])_____no_output_____data.replace({-999: np.nan, -1000: 0})_____no_output_____

### Renaming Axis Indexes_____no_output_____

data = pd.DataFrame(np.arange(12).reshape((3, 4)),
                    index=['Ohio', 'Colorado', 'New York'],
                    columns=['one', 'two', 'three', 'four'])_____no_output_____transform = lambda x: x[:4].upper()
data.index.map(transform)_____no_output_____data.index = data.index.map(transform)
data_____no_output_____data.rename(index=str.title, columns=str.upper)_____no_output_____data.rename(index={'OHIO': 'INDIANA'},
            columns={'three': 'peekaboo'})_____no_output_____data.rename(index={'OHIO': 'INDIANA'}, inplace=True)
data_____no_output_____

### Discretization and Binning_____no_output_____

ages = [20, 22, 25, 27, 21, 23, 37, 31, 61, 45, 41, 32]_____no_output_____bins = [18, 25, 35, 60, 100]
cats = pd.cut(ages, bins)
cats_____no_output_____cats.codes
cats.categories
pd.value_counts(cats)_____no_output_____pd.cut(ages, [18, 26, 36, 61, 100], right=False)_____no_output_____group_names = ['Youth', 'YoungAdult', 'MiddleAged', 'Senior']
pd.cut(ages, bins, labels=group_names)_____no_output_____data = np.random.rand(20)
pd.cut(data, 4, precision=2)_____no_output_____data = np.random.randn(1000)  # Normally distributed
cats = pd.qcut(data, 4)  # Cut into quartiles
cats
pd.value_counts(cats)_____no_output_____pd.qcut(data, [0, 0.1, 0.5, 0.9, 1.])_____no_output_____

### Detecting and Filtering Outliers_____no_output_____

data = pd.DataFrame(np.random.randn(1000, 4))
data.describe()_____no_output_____col = data[2]
col[np.abs(col) > 3]_____no_output_____data[(np.abs(data) > 3).any(1)]_____no_output_____data[np.abs(data) > 3] = np.sign(data) * 3
data.describe()_____no_output_____np.sign(data).head()_____no_output_____

### Permutation and Random Sampling_____no_output_____

df = pd.DataFrame(np.arange(5 * 4).reshape((5, 4)))
sampler = np.random.permutation(5)
sampler_____no_output_____df
df.take(sampler)_____no_output_____df.sample(n=3)_____no_output_____choices = pd.Series([5, 7, -1, 6, 4])
draws = choices.sample(n=10, replace=True)
draws_____no_output_____

### Computing Indicator/Dummy Variables_____no_output_____

df = pd.DataFrame({'key': ['b', 'b', 'a', 'c', 'a', 'b'],
                   'data1': range(6)})
pd.get_dummies(df['key'])_____no_output_____dummies = pd.get_dummies(df['key'], prefix='key')
df_with_dummy = df[['data1']].join(dummies)
df_with_dummy_____no_output_____mnames = ['movie_id', 'title', 'genres']
movies = pd.read_table('datasets/movielens/movies.dat', sep='::',
                       header=None, names=mnames)
movies[:10]_____no_output_____all_genres = []
for x in movies.genres:
    all_genres.extend(x.split('|'))
genres = pd.unique(all_genres)_____no_output_____genres_____no_output_____zero_matrix = np.zeros((len(movies), len(genres)))
dummies = pd.DataFrame(zero_matrix, columns=genres)_____no_output_____gen = movies.genres[0]
gen.split('|')
dummies.columns.get_indexer(gen.split('|'))_____no_output_____for i, gen in enumerate(movies.genres):
    indices = dummies.columns.get_indexer(gen.split('|'))
    dummies.iloc[i, indices] = 1_____no_output_____movies_windic = movies.join(dummies.add_prefix('Genre_'))
movies_windic.iloc[0]_____no_output_____np.random.seed(12345)
values = np.random.rand(10)
values
bins = [0, 0.2, 0.4, 0.6, 0.8, 1]
pd.get_dummies(pd.cut(values, bins))_____no_output_____

## String Manipulation_____no_output_____### String Object Methods_____no_output_____

val = 'a,b,  guido'
val.split(',')_____no_output_____pieces = [x.strip() for x in val.split(',')]
pieces_____no_output_____first, second, third = pieces
first + '::' + second + '::' + third_____no_output_____'::'.join(pieces)_____no_output_____'guido' in val
val.index(',')
val.find(':')_____no_output_____val.index(':')_____no_output_____val.count(',')_____no_output_____val.replace(',', '::')
val.replace(',', '')_____no_output_____

### Regular Expressions_____no_output_____

import re
text = "foo    bar\t baz  \tqux"
re.split('\s+', text)_____no_output_____regex = re.compile('\s+')
regex.split(text)_____no_output_____regex.findall(text)_____no_output_____text = """Dave dave@google.com
Steve steve@gmail.com
Rob rob@gmail.com
Ryan ryan@yahoo.com
"""
pattern = r'[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}'

# re.IGNORECASE makes the regex case-insensitive
regex = re.compile(pattern, flags=re.IGNORECASE)_____no_output_____regex.findall(text)_____no_output_____m = regex.search(text)
m
text[m.start():m.end()]_____no_output_____print(regex.match(text))_____no_output_____print(regex.sub('REDACTED', text))_____no_output_____pattern = r'([A-Z0-9._%+-]+)@([A-Z0-9.-]+)\.([A-Z]{2,4})'
regex = re.compile(pattern, flags=re.IGNORECASE)_____no_output_____m = regex.match('wesm@bright.net')
m.groups()_____no_output_____regex.findall(text)_____no_output_____print(regex.sub(r'Username: \1, Domain: \2, Suffix: \3', text))_____no_output_____

### Vectorized String Functions in pandas_____no_output_____

data = {'Dave': 'dave@google.com', 'Steve': 'steve@gmail.com',
        'Rob': 'rob@gmail.com', 'Wes': np.nan}
data = pd.Series(data)
data
data.isnull()_____no_output_____data.str.contains('gmail')_____no_output_____pattern
data.str.findall(pattern, flags=re.IGNORECASE)_____no_output_____matches = data.str.match(pattern, flags=re.IGNORECASE)
matches_____no_output_____matches.str.get(1)
matches.str[0]_____no_output_____data.str[:5]_____no_output_____pd.options.display.max_rows = PREVIOUS_MAX_ROWS_____no_output_____

## Conclusion_____no_output_____


# Notebook from C24IO/amazon-sagemaker-stock-prediction
Path: notebooks/dbg-data-preperation.ipynb

import os
import io
import boto3
import sagemaker
import datetime
import pandas as pd
from pandas.tools.plotting import autocorrelation_plot
from pandas.tools.plotting import scatter_matrix
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

# Define IAM role and session
role = sagemaker.get_execution_role()
session = sagemaker.Session()

s3_data_key = 'dbg-stockdata/source'
s3_bucket = session.default_bucket()
processed_filename = 'dbg_processed'_____no_output_____

# Stock Price Prediction Using Deutsche Börse Dataset on SageMaker DeepAR

## Overview
In this project we use Deutsche Börse Public Dataset consisting of trade data aggregated to one minute intervals from the Xetra trading systems

We obtain the data from public S3 bucket, hosted by Deutsche Börse under AWS Open Data Registry, aggregate the data within a given period, and prepare training, test and validation sets.
We then upload the prepared dat set to our own S3 bucket. This data location is then used with SageMaker provided DeepAR algorithm, to predict future movement of stock prices.

Following diagrams provides an high level overview of the project architecture:
![ML Architecture](../images/architecture.png)_____no_output_____

source_bucket = "s3://deutsche-boerse-xetra-pds"_____no_output_____

## 1. Obtaining Data

Since the DBG dataset is partitioned into separate folders by date, we generate a series of scripts to download the data files, in parallel.

DBG dataset contains trading data beginning June-17, 2017. However we chose to download last 6 months of data, so that the processing time is manageable within the scope of this workshop session.
_____no_output_____

start_date = datetime.datetime.strptime('2017-07-01','%Y-%m-%d')

end_date = datetime.datetime.strptime('2018-10-31','%Y-%m-%d')

end_date_str = end_date.strftime('%Y-%m-%d')
start_date_str = start_date.strftime('%Y-%m-%d')

print("Download date range: {} - {}".format(start_date_str, end_date_str))
_____no_output_____

We store all files in a local folder, prior to pre-processing the data.

After processing, we store the processed data set in another folder, which we'll then upload to an our SageMaker S3 bucket.
_____no_output_____

download_folder = '../data/download'
processed_folder = '../data/processed'
! mkdir -p {download_folder}
! mkdir -p {processed_folder}_____no_output_____

We then generate the download script in script folder and execute it to have the files downloaded.

Running bash script makes the download more reliable than running loops from Pythin runtime._____no_output_____

download_script = download_folder + '/download.sh'
dates = list(pd.date_range(start_date_str, end_date_str, freq='D').strftime('%Y-%m-%d'))
with open(download_script, 'w') as f:
    f.write("#!/bin/bash\n")
    f.write("\nset -euo pipefail\n")
    for date in dates:
        success_file =  os.path.join(download_folder, date, 'success')

        f.write("""
if [ ! -f {success_file} ]; then
    echo "Getting DBG dataset for date {date}"
    mkdir -p {download_folder}/{date}
    aws s3 sync {source_bucket}/{date} {download_folder}/{date} --no-sign-request
    touch {success_file}
else
    echo "DBG dataset for date {date} already exists"
fi\n""".format(success_file=success_file, date=date, source_bucket=source_bucket, download_folder=download_folder))

! chmod +x {download_script}     _____no_output_____

After creating the script, we run the script from shell, using Jupyter magic command.

Downloading data should take about half an hour to an hour to complete depending on the dat range chosen.

In order to avoid this process, you can alternative use the last cell in this notebook to obtain the prepared data from an S3 bucket accompanying this workshop, and load directly to your own S3 bucket._____no_output_____

%%time
!  {download_script}_____no_output_____

## 2. Data Preparation

To prepare the data, we collect day wise tading data from separate files, filter out any data falling out of trading window, and not pertaining to common stock trading.

### 2.1. Load Data
As first step we load all csv files in to a single data frame.

Data loading should take about 1-2 hours to complete depending on the instance type you have chosen for your notebook instance._____no_output_____

%%time
def convert_datetime(df):
    try:
        df["CalcTime"] = pd.to_datetime("1900-01-01 " + df["Time"])
        df["CalcDateTime"] = pd.to_datetime(df["Date"] + " " + df["Time"])
    except:
        pass
    return df

def load_csvs(data_folder):
    df = None
    total_loaded = 0
    df_initialized = False
    data_dir = data_folder + '/'
    data_subdirs = list(map(lambda date: data_dir + date, dates))
    for d in data_subdirs:
        trading_files = sorted(list(filter(lambda x:os.path.getsize(x)>2000, [os.path.join(d, x) for x in os.listdir(d) if x.endswith('.csv')])))
        if len(trading_files) > 0:
            print("Loading {} files from {}".format(len(trading_files), d))
            if df_initialized == False:
                frame = [pd.read_csv(f, engine='python', error_bad_lines=False, warn_bad_lines=False) for f in trading_files]
                df = convert_datetime(pd.concat(frame, ignore_index = True))
                df_initialized = True
            else:
                dfa = convert_datetime(pd.concat([pd.read_csv(f, engine='python', error_bad_lines=False) for f in trading_files], ignore_index = True))
                dft = [df, dfa]
                df = pd.concat(dft, ignore_index = True)
            total_loaded = total_loaded + len(trading_files)
            print(df.shape)
    print("Total of {} files loaded".format(total_loaded))
    return df
unprocessed_df = load_csvs(download_folder)_____no_output_____unprocessed_df.head()_____no_output_____

### 2.2. Filter records
Next we filter the records pertaining to common stack with trading volumes greater than zero, and within the regular trading hours.
Within the scope of this exercise, we focus our attention to top 100 stocks, by trading volume.

This filtering should take about 1-2 minute to complete depending on the instance type you have chosen for your notebook instance._____no_output_____

%%time
# Filter common stock between trading hours 08:00 and 20:00
# Exclude auctions (those are with TradeVolume == 0)
#Number of stocks to keep : Top 100, by trading volume
num_stocks=100
time_fmt = "%H:%M"
opening_hours_str = "08:00"
closing_hours_str = "20:00"
opening_hours = datetime.datetime.strptime(opening_hours_str, time_fmt)
closing_hours = datetime.datetime.strptime(closing_hours_str, time_fmt)

common_stocks = unprocessed_df[(unprocessed_df.SecurityType == 'Common stock') & \
                    (unprocessed_df.TradedVolume > 0) & \
                    (unprocessed_df.CalcTime >= opening_hours) & \
                    (unprocessed_df.CalcTime <= closing_hours)]

# Sort the stocks in descending order by trading volume
sort_by_volume = common_stocks[['Mnemonic', 'TradedVolume']].groupby(['Mnemonic']).sum().sort_values(['TradedVolume'], ascending=[0]).head(num_stocks)
stock_symbols = list(sort_by_volume.index.values)
sorted_stocks = common_stocks[common_stocks.Mnemonic.isin(stock_symbols)]
sorted_stocks = sorted_stocks.set_index(['Mnemonic', 'CalcDateTime']).sort_index()
stock_symbols = list(sort_by_volume.index.values)
print(sorted_stocks.shape)_____no_output_____

Check to see how many days' records we imported. If you import all records starting from July-2017, to October-2018, it should contain 326 days' woth of records._____no_output_____

len(sorted(list(sorted_stocks['Date'].unique())))_____no_output_____

### 2.3. Select features by minute

Next we build a clean data frame containing minute by minute transaction records with the following data points:
- Mnemonic (Stock Ticker Symbol)
- Minimum Price (During the interval)
- Maximum Prixe (During the interval)
- Start Price (At star of the interval)
- End Price (At end of the interval)
- Trading Volume (During the interval)
- Number of Trades (During the interval)_____no_output_____

%%time
#Build minute by minute index for trading hours
non_empty_days = sorted(list(sorted_stocks['Date'].unique()))
def build_index(non_empty_days, from_time, to_time):
    date_ranges = []
    for date in non_empty_days:
        yyyy, mm, dd = date.split('-')
        from_hour, from_min = from_time.split(':')
        to_hour, to_min = to_time.split(':')
        t1 = datetime.datetime(int(yyyy), int(mm), int(dd), int(from_hour),int(from_min),0)
        t2 = datetime.datetime(int(yyyy), int(mm), int(dd), int(to_hour),int(to_min),0)
        date_ranges.append(pd.DataFrame({"OrganizedDateTime": pd.date_range(t1, t2, freq='1Min').values}))
    agg = pd.concat(date_ranges, axis=0)
    agg.index = agg["OrganizedDateTime"]
    return agg

# Prepared data would contain numeric features for all stocks,
# for all days in the interval, for which there were trades (that means excluding weekends and holidays)
# for all minutes from 08:00 until 20:00
# in minutes without trades the prices from the last available minute are carried forward
# trades are filled with zero for such minutes

def basic_stock_features(input_df, mnemonic, time_index):
    stock = input_df.loc[mnemonic].copy()
    stock['HasTrade'] = 1.0
    stock = stock.reindex(time_index)
    features = ['MinPrice', 'MaxPrice', 'EndPrice', 'StartPrice']
    for f in features:
        stock[f] = stock[f].fillna(method='ffill')
    features = ['HasTrade', 'TradedVolume', 'NumberOfTrades']
    for f in features:
        stock[f] = stock[f].fillna(0.0)
    stock['Mnemonic'] = mnemonic
    selected_features = ['Mnemonic', 'MinPrice', 'MaxPrice', 'StartPrice', 'EndPrice', 'HasTrade', 'TradedVolume', 'NumberOfTrades']
    return stock[selected_features]

datetime_index = build_index(non_empty_days, opening_hours_str, closing_hours_str)["OrganizedDateTime"].values

stocks = []
for stock in stock_symbols:
    stock = basic_stock_features(sorted_stocks, stock, datetime_index)
    stocks.append(stock)

prepared = pd.concat(stocks, axis=0)
prepared.insert(0, 'Id', range(0, 0 + len(prepared)))
prepared = prepared.reset_index().set_index('Id')_____no_output_____prepared.head()_____no_output_____

### 2.4. Save processed data
Finally we save the processed data and upload to our SageMaker S3 Bucket.

Source data, as downloaded from DBG dataset includes minute by minute details of price movement and trading numbers and volumes for the stocks. However, in practice, we might be interested in time series at various aggregation level, in order to do more effective hourly, daily or weekly predictions.

Therefore we resample the data at various interval levels. First we define a resampling function that is able to resample various metrices for the aggregation levels.

For example, at a certain interval level, we are interested in:

- Last record for the End Price
- First record for the Start Price
- Minimum of all records for Min Price
- Maximum of all records for Max Price
- Total of all records for Number of Trades and Traded Volume_____no_output_____

intervals = ['M', 'W', 'D', 'H']

def resample_data(df, interval = None, mnemonics=None, metrics=None):
    if mnemonics is None:
        mnemonics = list(df.Mnemonic.unique())
    if metrics is None:
        metrics = list(df.columns.values)
    if 'Mnemonic' in metrics:
        metrics.remove('Mnemonic')
    if 'HasTrade' in metrics:
        metrics.remove('HasTrade')

    columns = list(df.columns.values)
    if 'CalcDateTime' not in columns:
        df["CalcDateTime"] = pd.to_datetime(df["CalcDateTime"])

    if interval is None or not isinstance(interval, str) or interval not in intervals:
        raise ValueError('Interval not supported, must be one of : {}'.format(intervals))

    resampeled_frames = []
    for mnemonic in mnemonics:
        print("Resampling {} records for interval - {}".format(mnemonic, interval))
        selected = df[df.Mnemonic == mnemonic].copy()
        selected.index = selected['CalcDateTime']
        selected = selected.sort_index()
        resampled = pd.DataFrame()
        for metric in metrics:
            if metric == 'EndPrice':
                resampled[metric] = selected[metric].resample(interval).last()
            elif metric == 'StartPrice':
                resampled[metric] = selected[metric].resample(interval).first()
            elif metric == 'MinPrice':
                resampled[metric] = selected[metric].resample(interval).min()
            elif metric == 'MaxPrice':
                resampled[metric] = selected[metric].resample(interval).max()
            elif metric == 'TradedVolume' or metric == "NumberOfTrades" :
                resampled[metric] = selected[metric].resample(interval).sum()
            else:
                pass
        resampled_columns = ['Mnemonic']
        for col in list(resampled.columns.values):
            resampled_columns.append(col)
        resampled['Mnemonic'] = mnemonic
        resampled = resampled[resampled_columns]
        resampled.dropna(inplace = True)
        resampeled_frames.append(resampled)
    resampeleds = pd.concat(resampeled_frames)
    return resampeleds_____no_output_____

Next we do a resampling for the following intervals and upload the file to an appropriate S3 location directly, saving local disk space:
- Month
- Week
- Day
- Hour

Resampling of the time series data at four different interval levels should take about 10 to 20 minutes to complete, depending on the instance type you have chosen for your notebook instance.

Note that this causes gaps in time series. However SageMaker DeepAR algorithm is designed to work with time series with gaps._____no_output_____

%%time
for interval in intervals:
    rescaled_series = resample_data(prepared, interval)
    csv_buffer = io.StringIO()
    rescaled_series.to_csv(csv_buffer)
    s3_resource = boto3.resource('s3')
    s3_resource.Object(s3_bucket, '{}/{}/resampled_stockdata.csv'.format(s3_data_key, interval, interval)).put(Body=csv_buffer.getvalue())
    _____no_output_____

Having the prepared data saved to an S3 bucket not only allows us to save some space in lcoal machine, but also helps when starting training job using SageMaker.

Since reading from S3 is optimized for high speed, when you are using an EC2 instance, which is what a SageMaker Notebook instance runs on, it makes sense to just load the data directly from S3, as and when needed.

Also, since SageMaker notebook instances come with only 5GB of Elastic Block Storage (to remain cost efficient), it is advisable to keep the device storage space clean, when not used.

Therefore, we delete the raw data files locally, to conserve storage space on notebook instance.
_____no_output_____

#Remove the download folder
!rm -rf {download_folder}
#Reset the data frames loaded in memory
rescaled_series = None
prepared = None
unprocessed_df = None_____no_output_____

### 2.5. Alternatively - Load resampled data for workshop

Since obtaining raw data from source S3 buckets maintained by Deutsche Börse, filtering, resampling and uploading to SageMaker S3 bucket can take upward of two hours, within the time limit of this workshop you could obtain cleaned data as hosted directly from S3 buckets specifically maintained for this workshop.
_____no_output_____

workshop_bucket = "fsv-309"
intervals = ['M', 'W', 'D', 'H']
datafilename = "resampled_stockdata.csv "

for interval in intervals:
    !aws s3 cp s3://{workshop_bucket}/{s3_data_key}/{interval}/{datafilename} s3://{s3_bucket}/{s3_data_key}/{interval}/{datafilename}
_____no_output_____

### 2.6. Validate Data Load from S3

Before moving on, we want to validate data by downloading from the location on S3 that we just saved, and run some quick introspection to develop an intuition about the nature of metrices and their possible influence on stock price movement.

For our preliminary analysis, we choose to load and investigate data with daily interval._____no_output_____

def load_resampled_from_s3(interval, bucket, s3_data_key):
    s3 = boto3.client('s3')
    obj = s3.get_object(Bucket=bucket, Key="{}/{}/resampled_stockdata.csv".format(s3_data_key, interval))
    loaded = pd.read_csv(io.BytesIO(obj['Body'].read()), parse_dates=True)
    mnemonics = list(loaded.Mnemonic.unique())

    return loaded, mnemonics_____no_output_____%%time
interval = "H"
stockdata, stocksymbols = load_resampled_from_s3(interval, s3_bucket, s3_data_key)
_____no_output_____stockdata.head()_____no_output_____

### Data exploration
At the end, we do a quick check on the data, and plot some series to develop an intuition of what the data looks like._____no_output_____

matplotlib.rcParams['figure.figsize'] = (25, 17) # use bigger graphs
def timeseries_plot(mnemonics, metrics, data=None, interval = None, bucket = None, s3_key = None):
    if data is None and interval is not None and bucket is not None and s3_key is not None:
        data, symbols = load_resampled_from_s3(interval, bucket, s3_key)
    columns = list(data.columns)
    ax = None
    for mnemonic in mnemonics:
        selected = data[data.Mnemonic == mnemonic].copy()
        selected.index = selected['CalcDateTime']
        selected = selected.sort_index()
        for column in columns:
            if column != 'CalcDateTime' and column != 'Mnemonic' and column not in metrics:
                selected = selected.drop(column, axis=1)
        selected_columns = list(selected.columns)
        for i, column in enumerate(selected_columns):
            selected_columns[i] = "{}-{}".format(mnemonic, column)
        selected.columns = selected_columns
        ax = selected.plot( ax = ax )_____no_output_____

Let's plot ending price for 3 automobile companies - BMW, Daimler, Porsche, Volkwagen_____no_output_____

timeseries_plot(['BMW', 'CON', 'DAI', 'PAH3', 'VOW3'], ['EndPrice'], stockdata)_____no_output_____

Observe here since the four stocks we chose to display are all automobile companies, their movements show very similar pattern.

We can also choose to display these stocks at lower resolution, such as at weekly interval, and still be able to observe the same patter._____no_output_____

timeseries_plot(['BMW', 'CON', 'DAI', 'PAH3', 'VOW3'], ['EndPrice'], interval = 'D', bucket = s3_bucket, s3_key = s3_data_key)
_____no_output_____timeseries_plot(['BMW','CON', 'DAI', 'PAH3', 'VOW3'], ['EndPrice'], interval = 'W', bucket = s3_bucket, s3_key = s3_data_key)
_____no_output_____

Now, in order to develop an intuition on how different time series for a single stock might influence each other,  we plot some metrices alongside each other, that might be somehow related, for a sample stock, in this case we just used "BMW".

Since the data set gives minute by minute observation of values, it might clutter the visual plot when it comes to understanding the patterns in the data, use data resampled at higher interval rate, such as daily or weekly.

_____no_output_____

timeseries_plot(['BMW'], ['EndPrice', 'MinPrice', 'MaxPrice'], interval = 'W', bucket = s3_bucket, s3_key = s3_data_key)_____no_output_____

### Observation 1

We can observed from the graph above:

- When there is a downward trend, the `EndPrice` is closer to the `MinPrice`, than to the `MaxPrice`
- When there is an upward trend, the `EndPrice` is closer to `MaxPrice` than to the `MinPrice`_____no_output_____

timeseries_plot(['BMW'], ['StartPrice', 'MinPrice', 'MaxPrice'], interval = 'W', bucket = s3_bucket, s3_key = s3_data_key)_____no_output_____

### Observation 2
Similar (and correlated) behavior is true for the `StartPrice`:

- When there is a downward trend, the `StartPrice` is closer to the `MaxPrice`, than to the `MinPrice`
- When there is an upward trend, the `StartPrice` is closer to `MinPrice` than to the `MaxPrice`_____no_output_____

timeseries_plot(['BMW'], ['StartPrice', 'EndPrice'], interval = 'W', bucket = s3_bucket, s3_key = s3_data_key)_____no_output_____

### Observation 3

Another observation that can be made is that:
- when the trend is upwards, `EndPrice` is above `StartPrice`
- when the trend is downwars, `EndPrice` is below `StartPrice`

Therefore: if the lines of `EndPrice` and `StartPrice` cross, one could expect trend reversal.
(Of course one needs to account for the variance, and false positives)_____no_output_____With these observations, we develop intuition that behaviour of the various metrices present in the data might have a correlation with future movement of the time series.

In the next step we apply Deep Neural Networks based Machine Learning techniques to try and find the correlation and hopefully be able to predict future movement, with certain confidence level._____no_output_____


# Notebook from JaniniRami/finding-exoplanets
Path: lightkurve/lightkurve.ipynb

<code>
from lightkurve import search_targetpixelfile_____no_output_____#targetpixelfiles refers to the pixels produced by kepler or tess telescopes.
# Note: most pixels will be 5x5
#search_targetpixelfile(star_designation_number, quarter 16 == year 4)
pixelfile = search_targetpixelfile('KIC 8462852', quarter=16).download(quality_bitmask='hardest')Warning: 25% (1042/4203) of the cadences will be ignored due to the quality mask (quality_bitmask=2096639).
#Looking at the pizel file
# Note: these frames by themselves are useless for the analyzing process.
pixelfile.plot(frame=1000)_____no_output_____# To analyze the pixels, we will need to convert them into a light curve file
lightcurve = pixelfile.to_lightcurve(aperture_mask='all')
lightcurve_____no_output_____lc.plot()
#Plot of Tabby's Star approcimately 1,470 light-years away._____no_output_____pixelFile = search_targetpixelfile('KIC 6922244', author="Kepler", quarter=4, cadence="long").download()_____no_output_____lc = pixelFile.to_lightcurve(aperture_mask=pixelFile.pipeline_mask)_____no_output_____lc.plot()_____no_output_____flat_lc = lc.flatten(window_length=401)
flat_lc.plot()_____no_output_____# period is the time this planet takes to orbit the star (3= 3-years)
folded_lc = flat_lc.fold(period=3.5225)
folded_lc.plot()_____no_output_____#find
import numpy as np
periodogram = flat_lc.to_periodogram(method='bls', period=np.arange(0.3, 10, 0.001))
periodogram.plot()_____no_output_____
</code>


# Notebook from joohiang/notebooks
Path: notebooks/wildfire-prediction/notebooks/3-classification-with-OPLS.ipynb

# Classification of the Fire Using OPLS-DA

#### In this notebook, we use statistical learning for modeling the evolution of the wildfires and predict when are going to become critical.
#### To achieve this, we use the Orthogonal Partial Least Square Discriminant Analysis (OPLS-DA) technique.

#### We consider a binary classification problem with the following classes:

#### - Class -1: Non-critical fire (burns less than 10 000 acres)
#### - Class 1: Critical fire (burns more than 10 000 acres)

#### Also, we consider the scenario where we want to predict if the fire is going to grow critically within the next 6 hours.

#### This horizon of prediction of 6 hours is arbitrary (you could change it), and corresponds to the delay that could be to mobilise the appropriate resources to deal with the fire before it becomes critical._____no_output_____### Install some required packages_____no_output_____

!pip install pyopls_____no_output_____

### Import some packages_____no_output_____

import os, sys
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.cross_decomposition import PLSRegression
from sklearn.model_selection import cross_val_predict, LeaveOneOut
from sklearn.metrics import r2_score, accuracy_score, classification_report

from pyopls import OPLS
from pyopls import OPLSValidator, OPLSDAValidator

import ast
import time
from collections import Counter_____no_output_____

### Global variable_____no_output_____

RESULTS = "../results"_____no_output_____

# STEP 1: Prediction at 6 hours horizon_____no_output_____## STEP 1.0: Load the data_____no_output_____

%%time
train_df_6h = pd.read_csv(
    os.path.join(
        RESULTS,
        "extracted-features-ComprehensiveFCParameters-filtered-target-horizon-6h-train.csv",
    ),
    index_col=0,
)
train_df_6h.shape_____no_output_____train_df_6h.head(2)_____no_output_____%%time
val_df_6h = pd.read_csv(
    os.path.join(
        RESULTS,
        "extracted-features-ComprehensiveFCParameters-filtered-target-horizon-6h-val.csv",
    ),
    index_col=0,
)
val_df_6h.shape_____no_output_____val_df_6h.head(2)_____no_output_____%%time
test_df_6h = pd.read_csv(
    os.path.join(
        RESULTS,
        "extracted-features-ComprehensiveFCParameters-filtered-target-horizon-6h-test.csv",
    ),
    index_col=0,
)
test_df_6h.shape_____no_output_____test_df_6h.head(2)_____no_output_____

## STEP 1.1: Modeling_____no_output_____### Project orthogonally the data into the latent space (O-PLS)_____no_output_____

X_cols = [c for c in train_df_6h.columns if c != "target"]
y_col = "target"

X_train_6h = train_df_6h[X_cols]
y_train_6h = train_df_6h[y_col]

# because we will use cross-validation here,
# we concatenate the validation and test datasets
# we consider the resulting dataset to test the model
X_test_6h = pd.concat([test_df_6h, val_df_6h])
X_test_6h = X_test_6h.sample(frac=1)
y_test_6h = X_test_6h[y_col]
X_test_6h = X_test_6h[X_cols]_____no_output_____X_train_6h.shape_____no_output_____

### Distribution of the classes_____no_output_____#### Train_____no_output_____

y_train_6h.value_counts(normalize=True)_____no_output_____

#### Validation_____no_output_____

#### Test_____no_output_____

y_test_6h.value_counts(normalize=True)_____no_output_____

## O-PLS Model_____no_output_____#### Running this cell could take a few minutes..._____no_output_____

%%time
ncomp = 50  # using cross-validation, we determined this value as the optimum. But, this could be changed.
opls_6h = OPLS(ncomp)
Z_train_6h = opls_6h.fit_transform(X_train_6h, y_train_6h)

print(Z_train_6h.shape)_____no_output_____

#### Check the goodness of the transformation
#### Lower values mean that most variance is removed_____no_output_____#### Running this cell could take a few minutes..._____no_output_____

%%time
opls_6h.score(X_train_6h)_____no_output_____%%time
pls = PLSRegression(1)

y_pred = cross_val_predict(pls, X_train_6h, y_train_6h, cv=100, n_jobs=-1)
q_squared = r2_score(y_train_6h, y_pred)
dq_squared = r2_score(y_train_6h, np.clip(y_pred, -1, 1))
accuracy = accuracy_score(y_train_6h, np.sign(y_pred))

processed_y_pred = cross_val_predict(pls, Z_train_6h, y_train_6h, cv=100, n_jobs=-1)
processed_q_squared = r2_score(y_train_6h, processed_y_pred)  # 0.981
processed_dq_squared = r2_score(y_train_6h, np.clip(processed_y_pred, -1, 1))
processed_accuracy = accuracy_score(y_train_6h, np.sign(processed_y_pred))

fpr, tpr, thresholds = roc_curve(y_train_6h, y_pred)
roc_auc = roc_auc_score(y_train_6h, y_pred)
proc_fpr, proc_tpr, proc_thresholds = roc_curve(y_train_6h, processed_y_pred)
proc_roc_auc = roc_auc_score(y_train_6h, processed_y_pred)

plt.figure(figsize=(20, 10))
plt.plot(fpr, tpr, lw=2, color="blue", label=f"Unprocessed (AUC={roc_auc:.4f})")
plt.plot(
    proc_fpr,
    proc_tpr,
    lw=2,
    color="red",
    label=f"{ncomp}-component OPLS (AUC={proc_roc_auc:.4f})",
)
plt.plot([0, 1], [0, 1], color="gray", lw=2, linestyle="--")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("ROC Curve")
plt.legend(loc="lower right")
plt.show()_____no_output_____

### Fit the model_____no_output_____

%%time
pls = PLSRegression(5)
pls.fit(Z_train_6h, y_train_6h)_____no_output_____

### Plot the transformed data into the latent space_____no_output_____

%%time
plt.figure(figsize=(20, 10))
df = pd.DataFrame(
    np.column_stack([pls.x_scores_, opls_6h.T_ortho_[:, 0]]),
    index=X_train_6h.index,
    columns=["t", "t_ortho"],
)
pos_df = df[y_train_6h == 1]
neg_df = df[y_train_6h == -1]
plt.scatter(neg_df["t"], neg_df["t_ortho"], c="blue", label="Non-critical fire")
plt.scatter(pos_df["t"], pos_df["t_ortho"], c="red", label="Critical fire")
plt.title("PLS Scores")
plt.xlabel("t_ortho")
plt.ylabel("t")
plt.legend(loc="upper right")
plt.show()_____no_output_____

### Make predictions on the test dataset with the fitted model_____no_output_____

%%time
Z_test_6h = opls_6h.transform(X_test_6h)
y_pred_test_6h = pls.predict(Z_test_6h)
fpr_test_6h, tpr_test_6h, thresholds_test_6h = roc_curve(y_test_6h, y_pred_test_6h)
roc_auc_test_6h = roc_auc_score(y_test_6h, y_pred_test_6h)

plt.figure(figsize=(20, 10))
plt.plot(
    fpr_test_6h,
    tpr_test_6h,
    lw=2,
    color="red",
    label=f"Test (AUC={roc_auc_test_6h:.4f})",
)
plt.plot([0, 1], [0, 1], color="gray", lw=2, linestyle="--")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("ROC Curve")
plt.legend(loc="lower right")
plt.show()_____no_output_____

## Conclusion

### We can see that the AUC of 0.66 is relatively low.
### This means that there is little information in the data. The pedictors are not correlated enough to the target.
### This results in a low predictive power of the model.
### Finally, we could improve the model by collecting ground truth historical data on the evolution of fires, for example, the number of acres burnt each hour._____no_output_____


# Notebook from oscarpimentel/el4106-variable-stars-alerce-ztf
Path: experiments/others/clean_original_data_v7.1.ipynb

# ALeRCE classes

https://github.com/ZwickyTransientFacility/ztf-avro-alert

1. **AGN:** Active Galactic Nuclei
1. **Blazar:** Blazar
1. **CV/Nova:** Cataclysmic Variable Star/Nova
1. **Ceph:** Cepheid Variable Star
1. **DSCT:** Delta Scuti Star
1. **EA:** Eclipsing Algol
1. **EB/EW:** Eclipsing Binaries/Eclipsing W Ursa Majoris
1. **LPV:** Long Period Variable
1. **Periodic-Other:** Periodic-Other
1. **QSO:** Quasi-Stellar Object
1. **RRL:** RRLyrae Variable Star
1. **RSCVn:** RS Canum Venaticorum
1. **SLSN:** Super Luminous Supernova
1. **SNII:** Supernova II
1. **SNIIb:** Supernova IIb
1. **SNIIn:** Supernova IIn
1. **SNIa:** Supernova Ia
1. **SNIbc:** Supernova Ibc
1. **TDE:** Tidal disruption event (to remove)
1. **YSO:** Young Stellar Object
1. **ZZ:** ZZ Ceti Stars (to remove)_____no_output_____

<code>
import sys
sys.path.append('../../')_____no_output_____%load_ext autoreload
%autoreload 2
from vstars.alerce_utils import process_df_labels, process_df_detections, keep_only_valid_objs
import numpy as np
import pandas as pd

load_rootdir = '../../../../tesis/surveys_data'
survey_name = 'alerceZTFv7.1'
uses_corr = False # uses_corr=False only can be used with SNe objects
df_index_names = {
    'oid':'oid', # object id
    'oid_det':'objectId', # object id
    'label':'classALeRCE', # object class name
    'ra':'ra',
    'dec':'dec',
    'band':'fid', # band
    'obs_day':'mjd', # days
    'obs':'magpsf_corr' if uses_corr else 'magpsf', # observations
    'obs_error':'sigmapsf_corr' if uses_corr else 'sigmapsf', # observation errors
}
detections_cols = ['objectId', 'fid', 'mjd', df_index_names['obs'], df_index_names['obs_error']]

### load files and processing
detections_df = pd.read_parquet(f'{load_rootdir}/{survey_name}/detections_with_xmatch')
detections_df, det_objs = process_df_detections(detections_df, df_index_names['oid_det'], df_index_names['oid'], detections_cols, uses_corr=uses_corr)
print(f'detections_df - columns: {list(detections_df.columns)} - id: {detections_df.index.name}')

labels_df = pd.read_csv(f'{load_rootdir}/{survey_name}/dfcrossmatches_prioritized_v7.0.1.csv')
labels_df, label_objs = process_df_labels(labels_df, df_index_names['oid'], det_objs)
print(f'labels - columns: {list(labels_df.columns)} - id: {labels_df.index.name}')

### filter
valid_objs = list(set(det_objs) & set(label_objs))
labels_df = keep_only_valid_objs(labels_df, valid_objs)
detections_df = keep_only_valid_objs(detections_df, valid_objs)

### print info
classes = set(labels_df[df_index_names['label']].values)
print('classes:', classes)../../vstars/alerce_utils.py:80: UserWarning: only use uses_corr=False with SNe objects
  warnings.warn('only use uses_corr=False with SNe objects')
%load_ext autoreload
%autoreload 2
from vstars.level_bars import LevelBar

classes, counts = np.unique(labels_df[df_index_names['label']].values, return_counts=True)
population_cdict = {c:counts[kc] for kc,c in enumerate(classes)}
print(LevelBar(population_cdict, ncols=60))The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
|█▌                                                        | AGN - 3,177/122,825 (2.59%)
|▍                                                         | Blazar - 1,019/122,825 (0.83%)
|▍                                                         | CV/Nova - 981/122,825 (0.80%)
|▎                                                         | Ceph - 716/122,825 (0.58%)
|▍                                                         | DSCT - 946/122,825 (0.77%)
|███▍                                                      | EA - 7,304/122,825 (5.95%)
|██████████████████▌                                       | EB/EW - 39,353/122,825 (32.04%)
|██████▋                                                   | LPV - 14,207/122,825 (11.57%)
|                                                          | NLAGN - 6/122,825 (0.00%)
|                                                          | NLQSO - 77/122,825 (0.06%)
|▏                                                         | Periodic-Other - 521/122,825 (0.42%)
|██████▌                                                   | QSO - 14,000/122,825 (11.40%)
|████████████████▋                                         | RRL - 35,259/122,825 (28.71%)
|▌                                                         | RSCVn - 1,253/122,825 (1.02%)
|                                                          | SLSN - 33/122,825 (0.03%)
|▏                                                         | SNII - 325/122,825 (0.26%)
|                                                          | SNIIb - 16/122,825 (0.01%)
|                                                          | SNIIn - 37/122,825 (0.03%)
|▋                                                         | SNIa - 1,535/122,825 (1.25%)
|                                                          | SNIbc - 103/122,825 (0.08%)
|                                                          | TDE - 5/122,825 (0.00%)
|▉                                                         | YSO - 1,947/122,825 (1.59%)
|                                                          | ZZ - 5/122,825 (0.00%)

detections_df.info()
detections_df[-20:]<class 'pandas.core.frame.DataFrame'>
Index: 6210462 entries, ZTF17aaaemke to ZTF20abefeou
Data columns (total 4 columns):
fid         int64
mjd         float64
magpsf      float32
sigmapsf    float32
dtypes: float32(2), float64(1), int64(1)
memory usage: 189.5+ MB
labels_df.info()
labels_df[:20]<class 'pandas.core.frame.DataFrame'>
Index: 122825 entries, ZTF19aapcxhy to ZTF18abgqxlw
Data columns (total 8 columns):
classALeRCE          122825 non-null object
ra                   122825 non-null float64
dec                  122825 non-null float64
period               77759 non-null object
source               122825 non-null object
id_source            122825 non-null object
class_source         122825 non-null object
separation_arcsec    122825 non-null float64
dtypes: float64(3), object(5)
memory usage: 8.4+ MB
%load_ext autoreload
%autoreload 2
from vstars.alerce_utils import get_valid_classes_objs, keep_only_valid_objs

target_classes = [
    #'EA', # Eclipsing Binaries
    'EB/EW', # Eclipsing Binaries
    'Ceph', # Cefeidas
    'RRL', # RR Lyrae
    'DSCT', # Delta Scuti
    'LPV', # Long Period Variables
]
valid_objs = get_valid_classes_objs(labels_df, df_index_names, target_classes)
print(valid_objs[:10])The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
['ZTF18abwwdsc', 'ZTF18aaakigd', 'ZTF18aaavkyj', 'ZTF18abwwdxw', 'ZTF19aaocniv', 'ZTF17aacemqz', 'ZTF18aaiyfjx', 'ZTF19aaczymt', 'ZTF18abvpirg', 'ZTF18abccnft']
new_detections_df = keep_only_valid_objs(detections_df, valid_objs)
print(new_detections_df.info())
new_detections_df[:50]<class 'pandas.core.frame.DataFrame'>
Index: 4656626 entries, ZTF17aaaemke to ZTF20abefeou
Data columns (total 4 columns):
fid         int64
mjd         float64
magpsf      float32
sigmapsf    float32
dtypes: float32(2), float64(1), int64(1)
memory usage: 142.1+ MB
None
new_labels_df = keep_only_valid_objs(labels_df, valid_objs)
print(new_labels_df.info())
new_labels_df[:50]<class 'pandas.core.frame.DataFrame'>
Index: 90481 entries, ZTF18abwwdsc to ZTF18abgqxlw
Data columns (total 8 columns):
classALeRCE          90481 non-null object
ra                   90481 non-null float64
dec                  90481 non-null float64
period               67464 non-null object
source               90481 non-null object
id_source            90481 non-null object
class_source         90481 non-null object
separation_arcsec    90481 non-null float64
dtypes: float64(3), object(5)
memory usage: 6.2+ MB
None
print(new_labels_df.loc['ZTF18aaavkyj'])
print(new_detections_df.loc['ZTF18aaavkyj'])classALeRCE                     Ceph
ra                           192.885
dec                          24.1226
period                       1.09572
source                     CRTSnorth
id_source            1123064031941.0
class_source                    ACEP
separation_arcsec            0.29111
Name: ZTF18aaavkyj, dtype: object
              fid           mjd     magpsf  sigmapsf
oid
ZTF18aaavkyj    1  58567.446736  16.525101  0.070550
ZTF18aaavkyj    1  58588.213599  15.957800  0.069776
ZTF18aaavkyj    2  58589.241782  15.788184  0.034676
import pandas as pd

### save files
extra_name = '' if uses_corr else '_noncorr'
save_root_dir = f'../../data/{survey_name}'
new_labels_df.to_parquet(f'{save_root_dir}/labels_vs{extra_name}.parquet')
new_detections_df.to_parquet(f'{save_root_dir}/detections_vs{extra_name}.parquet')_____no_output_____
</code>