# Notebook from lotusxai/join-titanic-project
Path: join-tatanic-project/Notebooks/covid_global_eda_prediction_ml_sir_regression.ipynb

# COVID Global Forecast: SIR model + ML regressions

In the context of the global COVID-19 pandemic, Kaggle has launched several challenges in order to provide useful insights that may answer some of the open scientific questions about the virus. This is the case of the [COVID19 Global Forecasting](https://www.kaggle.com/c/covid19-global-forecasting-week-1), in which participants are encouraged to fit worldwide data in order to predict the pandemic evolution, hopefully helping to determine which factors impact the transmission behavior of COVID-19.

**TABLE OF CONTENTS**

1. [Exploratory data analysis (EDA)](#section1)

    1.1. [COVID-19 global tendency excluding China](#section11)

    1.2. [COVID-19 tendency in China](#section12)

    1.3. [Italy, Spain, UK and US](#section13)

2. [SIR model](#section2)

    2.1. [Implementing the SIR model](#section21)

3. [Data enrichment](#section3)

    3.1. [Join data, filter dates and clean missings](#section31)

    3.2. [Compute lags and trends](#section32)

    3.3. [Add country details](#section33)

4. [Predictions with machine learning](#section4)

    4.1. [Linear Regression for one country](#section41)

    4.2. [Logistic regression with lags](#section42)

**Disclaimer 1**: this notebook is still under construction.

**Disclaimer 2**: the training dataset is being updated on a daily basis, in order to include the most recent cases. I'll try to keep the notebook updated in order to prevent data leaking and other potential problems._____no_output_____

!pip install lightgbm
!pip install xgboost
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn import preprocessing
import time
from datetime import datetime
import warnings
warnings.filterwarnings('ignore')

# ML libraries
import lightgbm as lgb
import xgboost as xgb

from xgboost import plot_importance, plot_tree
from sklearn.model_selection import RandomizedSearchCV, GridSearchCV
from sklearn import linear_model
from sklearn.metrics import mean_squared_errorRequirement already satisfied: lightgbm in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (2.3.1)
Requirement already satisfied: scikit-learn in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from lightgbm) (0.20.3)
Requirement already satisfied: numpy in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from lightgbm) (1.14.3)
Requirement already satisfied: scipy in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from lightgbm) (1.1.0)
You are using pip version 10.0.1, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Requirement already satisfied: xgboost in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (1.0.2)
Requirement already satisfied: scipy in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from xgboost) (1.1.0)
Requirement already satisfied: numpy in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (from xgboost) (1.14.3)
You are using pip version 10.0.1, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

# 1. Exploratory data analysis (EDA)

First of all, let's take a look on the data structure:_____no_output_____The dataset covers 163 countries and almost 2 full months from 2020, which is enough data to get some clues about the pandemic. Let's see a few plots of the worldwide tendency to see if we can extract some insights:_____no_output_____**Observations**: The global curve shows a rich fine structure, but these numbers are strongly affected by the vector zero country, China. Given that COVID-19 started there, during the initial expansion of the virus there was no reliable information about the real infected cases. In fact,  the criteria to consider infection cases was modified around 2020-02-11, which strongly perturbed the curve as you can see from the figure. _____no_output_____## 1.1. COVID-19 global tendency excluding China and in China

Since details of the initial breakthrough strongly interfere with the results, it's recomended to analyze China independently. Let's first see the results without China: _____no_output_____

test = pd.read_csv("../input/test.csv")
train = pd.read_csv("../input/train.csv")

print(train.head(5))

pop_italy = 60486683.
pop_spain = 46749696.
pop_UK = 67784927.
pop_US = 331002651.
confirmed_total_date_noChina = train[train['Country/Region']!='China'].groupby(['Date']).agg({'ConfirmedCases':['sum']})
fatalities_total_date_noChina = train[train['Country/Region']!='China'].groupby(['Date']).agg({'Fatalities':['sum']})
total_date_noChina = confirmed_total_date_noChina.join(fatalities_total_date_noChina)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5))

total_date_noChina.plot(ax=ax1)
ax1.set_title("Global confirmed cases excluding China", size=13)
ax1.set_ylabel("Number of cases", size=13)
ax1.set_xlabel("Date", size=13)

fatalities_total_date_noChina.plot(ax=ax2, color='orange')
ax2.set_title("Global deceased cases excluding China", size=13)
ax2.set_ylabel("Number of cases", size=13)
ax2.set_xlabel("Date", size=13)

confirmed_total_date_China = train[train['Country/Region']=='China'].groupby(['Date']).agg({'ConfirmedCases':['sum']})
fatalities_total_date_China = train[train['Country/Region']=='China'].groupby(['Date']).agg({'Fatalities':['sum']})
total_date_China = confirmed_total_date_China.join(fatalities_total_date_China)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5))
total_date_China.plot(ax=ax1)
ax1.set_title("China confirmed cases", size=13)
ax1.set_ylabel("Number of cases", size=13)
ax1.set_xlabel("Date", size=13)

fatalities_total_date_China.plot(ax=ax2, color='orange')
ax2.set_title("Deceased cases in China", size=13)
ax2.set_ylabel("Number of cases in China", size=13)
ax2.set_xlabel("Date", size=13)   Id Province/State Country/Region   Lat  Long        Date  ConfirmedCases  \
0   1            NaN    Afghanistan  33.0  65.0  2020-01-22             0.0
1   2            NaN    Afghanistan  33.0  65.0  2020-01-23             0.0
2   3            NaN    Afghanistan  33.0  65.0  2020-01-24             0.0
3   4            NaN    Afghanistan  33.0  65.0  2020-01-25             0.0
4   5            NaN    Afghanistan  33.0  65.0  2020-01-26             0.0

   Fatalities
0         0.0
1         0.0
2         0.0
3         0.0
4         0.0

**Observations**: In this case the general behavior looks cleaner, and in fact the curve resembles a typical epidemiology model like [SIR](http://mat.uab.cat/matmat/PDFv2013/v2013n03.pdf). SIR models present a large increasing in the number of infections that, once it reaches the maximum of the contagion, decreases with a lower slope. For comparison, a SIR simulation from section [2. SIR model]_____no_output_____**Observations**:

* **Smoothness**. Both plots are less smooth than theoretical simulations or the curve from the rest of the world cumulative
* **Infected criteria**. The moment in which the criteria to consider an infected case was changed is direclty spotted
* **Irregularities**. There are some iregularities. I should check the literature in depth to look for evidences, but the reasons may be that both the resources spent to monitor the epidemy and the security measures to stop ot have been changing over time
* **Plateaux**. It looks like the curve has reached a plateaux, which would imply that China is on their maximum of contagion _____no_output_____## 1.3. Tendency in Italy, Spain, UK and US

Both Italy and Spain are experiencing the larger increase in COVID-19 positives in Europe. At the same time, UK is a unique case given that it's one of the most important countries in Europe but recently has left the European Union, which has create an effective barrier to human mobility from other countries. _____no_output_____

confirmed_total_date_Italy = train[train['Country/Region']=='Italy'].groupby(['Date']).agg({'ConfirmedCases':['sum']})
fatalities_total_date_Italy = train[train['Country/Region']=='Italy'].groupby(['Date']).agg({'Fatalities':['sum']})
total_date_Italy = confirmed_total_date_Italy.join(fatalities_total_date_Italy)

confirmed_total_date_Spain = train[train['Country/Region']=='Spain'].groupby(['Date']).agg({'ConfirmedCases':['sum']})
fatalities_total_date_Spain = train[train['Country/Region']=='Spain'].groupby(['Date']).agg({'Fatalities':['sum']})
total_date_Spain = confirmed_total_date_Spain.join(fatalities_total_date_Spain)

confirmed_total_date_UK = train[train['Country/Region']=='United Kingdom'].groupby(['Date']).agg({'ConfirmedCases':['sum']})
fatalities_total_date_UK = train[train['Country/Region']=='United Kingdom'].groupby(['Date']).agg({'Fatalities':['sum']})
total_date_UK = confirmed_total_date_UK.join(fatalities_total_date_UK)

confirmed_total_date_US = train[train['Country/Region']=='US'].groupby(['Date']).agg({'ConfirmedCases':['sum']})
fatalities_total_date_US  = train[train['Country/Region']=='US'].groupby(['Date']).agg({'Fatalities':['sum']})
total_date_US = confirmed_total_date_US.join(fatalities_total_date_US)

plt.figure(figsize=(15,10))
plt.subplot(2, 2, 1)
total_date_Italy.plot(ax=plt.gca(), title='Italy')
plt.ylabel("Confirmed infection cases", size=13)

plt.subplot(2, 2, 2)
total_date_Spain.plot(ax=plt.gca(), title='Spain')
plt.ylabel("Confirmed infection cases", size=13)

plt.subplot(2, 2, 3)
total_date_UK.plot(ax=plt.gca(), title='United Kingdom')
plt.ylabel("Confirmed infection cases", size=13)

plt.subplot(2, 2, 4)
total_date_US.plot(ax=plt.gca(), title='United States')
plt.ylabel("Confirmed infection cases", size=13)
_____no_output_____

As a fraction of the total population of each country:_____no_output_____

total_date_Italy.ConfirmedCases = total_date_Italy.ConfirmedCases/pop_italy*100.
total_date_Italy.Fatalities = total_date_Italy.ConfirmedCases/pop_italy*100.
total_date_Spain.ConfirmedCases = total_date_Spain.ConfirmedCases/pop_spain*100.
total_date_Spain.Fatalities = total_date_Spain.ConfirmedCases/pop_spain*100.
total_date_UK.ConfirmedCases = total_date_UK.ConfirmedCases/pop_UK*100.
total_date_UK.Fatalities = total_date_UK.ConfirmedCases/pop_UK*100.
total_date_US.ConfirmedCases = total_date_US.ConfirmedCases/pop_US*100.
total_date_US.Fatalities = total_date_US.ConfirmedCases/pop_US*100.

plt.figure(figsize=(15,10))
plt.subplot(2, 2, 1)
total_date_Italy.ConfirmedCases.plot(ax=plt.gca(), title='Italy')
plt.ylabel("Fraction of population infected")
plt.ylim(0, 0.06)

plt.subplot(2, 2, 2)
total_date_Spain.ConfirmedCases.plot(ax=plt.gca(), title='Spain')
plt.ylabel("Fraction of population infected")
plt.ylim(0, 0.06)

plt.subplot(2, 2, 3)
total_date_UK.ConfirmedCases.plot(ax=plt.gca(), title='United Kingdom')
plt.ylabel("Fraction of population infected")
plt.ylim(0, 0.005)

plt.subplot(2, 2, 4)
total_date_US.ConfirmedCases.plot(ax=plt.gca(), title='United States')
plt.ylabel("Fraction of population infected")
plt.ylim(0, 0.005)

confirmed_total_date_Italy = train[(train['Country/Region']=='Italy') & train['ConfirmedCases']!=0].groupby(['Date']).agg({'ConfirmedCases':['sum']})
fatalities_total_date_Italy = train[(train['Country/Region']=='Italy') & train['ConfirmedCases']!=0].groupby(['Date']).agg({'Fatalities':['sum']})
total_date_Italy = confirmed_total_date_Italy.join(fatalities_total_date_Italy)

confirmed_total_date_Spain = train[(train['Country/Region']=='Spain') & (train['ConfirmedCases']!=0)].groupby(['Date']).agg({'ConfirmedCases':['sum']})
fatalities_total_date_Spain = train[(train['Country/Region']=='Spain') & (train['ConfirmedCases']!=0)].groupby(['Date']).agg({'Fatalities':['sum']})
total_date_Spain = confirmed_total_date_Spain.join(fatalities_total_date_Spain)

confirmed_total_date_UK = train[(train['Country/Region']=='United Kingdom') & (train['ConfirmedCases']!=0)].groupby(['Date']).agg({'ConfirmedCases':['sum']})
fatalities_total_date_UK = train[(train['Country/Region']=='United Kingdom') & (train['ConfirmedCases']!=0)].groupby(['Date']).agg({'Fatalities':['sum']})
total_date_UK = confirmed_total_date_UK.join(fatalities_total_date_UK)

confirmed_total_date_US = train[(train['Country/Region']=='US') & (train['ConfirmedCases']!=0)].groupby(['Date']).agg({'ConfirmedCases':['sum']})
fatalities_total_date_US = train[(train['Country/Region']=='US') & (train['ConfirmedCases']!=0)].groupby(['Date']).agg({'Fatalities':['sum']})
total_date_US = confirmed_total_date_US.join(fatalities_total_date_US)

italy = [i for i in total_date_Italy.ConfirmedCases['sum'].values]
italy_30 = italy[0:50]
spain = [i for i in total_date_Spain.ConfirmedCases['sum'].values]
spain_30 = spain[0:50]
UK = [i for i in total_date_UK.ConfirmedCases['sum'].values]
UK_30 = UK[0:50]
US = [i for i in total_date_US.ConfirmedCases['sum'].values]
US_30 = US[0:50]

# Plots
plt.figure(figsize=(12,6))
plt.plot(italy_30)
plt.plot(spain_30)
plt.plot(UK_30)
plt.plot(US_30)
plt.legend(["Italy", "Spain", "UK", "US"], loc='upper left')
plt.title("COVID-19 infections from the first confirmed case", size=15)
plt.xlabel("Days", size=13)
plt.ylabel("Infected cases", size=13)
plt.ylim(0, 60000)
plt.show()
_____no_output_____

**Observations**:
* **Italy**. With more than 35.000 confirmed cases, Italy shows one of the most alarming scenarios of COVID-19. The infections curve is very steep, and the percentage of population infected is near the 0.1%
* **Spain**. Spain is the second country from the EU with more confirmed COVID-19 cases, slightly below 15.000 cases. Around 0.03% of the population has currently (2020-03-19) contracted the virus
* **United Kingdom**. Despite not being very far from them, the UK shows less cases. There are several theories for this; @codingdon pointed out in the direction of a lower number of COVID-19 tests, and I think that maybe the Brexit process had an impact into international travels that delayed the arrival of the COVID-19 (baed on [this](https://www.ons.gov.uk/peoplepopulationandcommunity/leisureandtourism/articles/traveltrends/2018). Anyway, these are just theories and we can't take anything for granted. Only a 0.004% of UK citizens have been reported by COVID-19 infection
* **United States**. Despite the virus spreaded in the US at last, the speed curve of infection is the steep. _____no_output_____# 2. SIR model

We have seen some general behavior of the virus in agregated data, for the country where the coronavirus was originated and for four other interesting countries. There's a lot of information to be extracted from this data; for example, we haven't analyzed the effects of long/lat of countries. However, since our main purpose is to develop a predective model in order to understand the key factors that impact the COVID-19 transmission, I'll move on to one of the most famous epidemiologic models: SIR.

SIR is a simple model that considers a population that belongs to one of the following states:
1. **Susceptible (S)**. The individual hasn't contracted the disease, but she can be infected due to transmisison from infected people
2. **Infected (I)**. This person has contracted the disease
3. **Recovered/Deceased (R)**. The disease may lead to one of two destinies: either the person survives, hence developing inmunity to the disease, or the person is deceased.

Image by Kai Sasaki from [lewuathe.com](https://www.lewuathe.com/covid-19-dynamics-with-sir-model.html)

There are many versions of this model, considering birth and death (SIRD with demography), with intermediate states, etc. However, since we are in the early stages of the COVID-19 expansion and our interest is focused in the short term, we will consider that people develops immunity (in the long term, immunity may be lost and the COVID-19 may come back within a certain seasonality like the common flu) and there is no transition from recovered to the remaining two states. With this, the differential equations that govern the system are:

$$ {dS \over dt} = - {\beta S I \over N} $$

$$ {dI \over dt} = {\beta S I \over N} - \gamma I$$

$$ {dR \over dt} = \gamma I$$

Where $\beta$ is the contagion rate of the pathogen and $\gamma$ is the recovery rate._____no_output_____## 2.1. Implementing the SIR model

SIR model can be implemented in many ways: from the differential equations governing the system, within a mean field approximation or running the dynamics in a social network (graph). For the sake of simplicity, I'vem chosen the first option, and we will simply run a numerical method (Runge-Kutta) to solve the differential equations system.

The functions governing the dif.eqs. are:_____no_output_____

# Susceptible equation
def fa(N, a, b, beta):
    fa = -beta*a*b
    return fa

# Infected equation
def fb(N, a, b, beta, gamma):
    fb = beta*a*b - gamma*b
    return fb

# Recovered/deceased equation
def fc(N, b, gamma):
    fc = gamma*b
    return fc
# In order to solve the differential equations system, we develop a 4rth order Runge-Kutta method:
# Runge-Kutta method of 4rth order for 3 dimensions (susceptible a, infected b and recovered r)
def rK4(N, a, b, c, fa, fb, fc, beta, gamma, hs):
    a1 = fa(N, a, b, beta)*hs
    b1 = fb(N, a, b, beta, gamma)*hs
    c1 = fc(N, b, gamma)*hs
    ak = a + a1*0.5
    bk = b + b1*0.5
    ck = c + c1*0.5
    a2 = fa(N, ak, bk, beta)*hs
    b2 = fb(N, ak, bk, beta, gamma)*hs
    c2 = fc(N, bk, gamma)*hs
    ak = a + a2*0.5
    bk = b + b2*0.5
    ck = c + c2*0.5
    a3 = fa(N, ak, bk, beta)*hs
    b3 = fb(N, ak, bk, beta, gamma)*hs
    c3 = fc(N, bk, gamma)*hs
    ak = a + a3
    bk = b + b3
    ck = c + c3
    a4 = fa(N, ak, bk, beta)*hs
    b4 = fb(N, ak, bk, beta, gamma)*hs
    c4 = fc(N, bk, gamma)*hs
    a = a + (a1 + 2*(a2 + a3) + a4)/6
    b = b + (b1 + 2*(b2 + b3) + b4)/6
    c = c + (c1 + 2*(c2 + c3) + c4)/6
    return a, b, c_____no_output_____

And finally, to obtain the evolution of the disease we simply define the initial conditions and call the rk4 method:_____no_output_____

def SIR(N, b0, beta, gamma, hs):

    """
    N = total number of population
    beta = transition rate S->I
    gamma = transition rate I->R
    k =  denotes the constant degree distribution of the network (average value for networks in which
    the probability of finding a node with a different connectivity decays exponentially fast
    hs = jump step of the numerical integration
    """

    # Initial condition
    a = float(N-1)/N -b0
    b = float(1)/N +b0
    c = 0.

    sus, inf, rec= [],[],[]
    for i in range(10000): # Run for a certain number of time-steps
        sus.append(a)
        inf.append(b)
        rec.append(c)
        a,b,c = rK4(N, a, b, c, fa, fb, fc, beta, gamma, hs)

    return sus, inf, rec

_____no_output_____

Results obtained for N=world population, only one initial infected case, $\beta=0.3$, $\gamma=0.5$ and a leap pass $h_s = 0.1$ are shown below:_____no_output_____

# Parameters of the model
N = 7800*(10**6)
b0 = 0
beta = 0.7
gamma = 0.2
hs = 0.1

sus, inf, rec = SIR(N, b0, beta, gamma, hs)

f = plt.figure(figsize=(8,5))
plt.plot(sus, 'b.', label='susceptible');
plt.plot(inf, 'r.', label='infected');
plt.plot(rec, 'c.', label='recovered/deceased');
plt.title("SIR model")
plt.xlabel("time", fontsize=10);
plt.ylabel("Fraction of population", fontsize=10);
plt.legend(loc='best')
plt.xlim(0,1000)
plt.savefig('SIR_example.png')
plt.show()_____no_output_____

**Observations**:
* The number of infected cases increases for a certain time period, and then eventually decreases given that individuals recover/decease from the disease
* The susceptible fraction of population decreases as the virus is transmited, to eventually drop to the absorbent state 0
* The oposite happens for the recovered/deceased case

Notice that different initial conditions and parameter values will lead to other scenarios, feel free to play with these numbers to study the system._____no_output_____# 3. Data enrichment

Analyzing SIR simulations was meant to understand a model that approximately resembles the transmission mechanism of many virus, including the COVID-19. However, there are alternative methods that may prove being equally useful both to predict and to understand the pandemic evolution. Many of these methods rely on having rich data to extract conclusions and allow algorithms to extrapolate patterns in data, and that is exactly what we are going to do.

Main workflow of this section:
1. Join data, filter dates and clean missings
2. Compute lags and trends
3. Add country details

**Disclaimer**: this data enrichment is not mandatory and we could end up not using all of the new features in our models. However I consider it a didactical step that will surely add some value, for example in an in-depth exploratory analysis._____no_output_____## 3.1. Join data, filter dates and clean missings(Data Preprocessing)

First of all, we perform some pre-processing prepare the dataset, consisting on:

* **Join data**. Join train/test to facilitate data transformations
* **Filter dates**. According to the challenge conditions, remove ConfirmedCases and Fatalities post 2020-03-12. Create additional date columns
* **Missings**. Analyze and fix missing values_____no_output_____

# Merge train and test, exclude overlap
dates_overlap = ['2020-03-12','2020-03-13','2020-03-14','2020-03-15','2020-03-16','2020-03-17','2020-03-18',
                 '2020-03-19','2020-03-20','2020-03-21','2020-03-22','2020-03-23', '2020-03-24']
train2 = train.loc[~train['Date'].isin(dates_overlap)]
all_data = pd.concat([train2, test], axis = 0, sort=False)
# Double check that there are no informed ConfirmedCases and Fatalities after 2020-03-11
all_data.loc[all_data['Date'] >= '2020-03-12', 'ConfirmedCases'] = np.nan
all_data.loc[all_data['Date'] >= '2020-03-12', 'Fatalities'] = np.nan
all_data['Date'] = pd.to_datetime(all_data['Date'])

# Create date columns
le = preprocessing.LabelEncoder()
all_data['Day_num'] = le.fit_transform(all_data.Date)
all_data['Day'] = all_data['Date'].dt.day
all_data['Month'] = all_data['Date'].dt.month
all_data['Year'] = all_data['Date'].dt.year

# Fill null values given that we merged train-test datasets
all_data['Province/State'].fillna("None", inplace=True)
all_data['ConfirmedCases'].fillna(0, inplace=True)
all_data['Fatalities'].fillna(0, inplace=True)
all_data['Id'].fillna(-1, inplace=True)
all_data['ForecastId'].fillna(-1, inplace=True)

# Aruba has no Lat nor Long. Inform it manually
all_data.loc[all_data['Lat'].isna()==True, 'Lat'] = 12.510052
all_data.loc[all_data['Long'].isna()==True, 'Long'] = -70.009354
Time spent:  0.2120370864868164

**Observations**:
* "ConfirmedCases" and "Fatalities" are now only informed for dates previous to 2020-03-12
* The dataset includes all countries and dates, which is required for the lag/trend step
* Missing values for "ConfirmedCases" and "Fatalities" have been replaced by 0, which may be dangerous if we do not remember it at the end of the process. However, since we will train only on dates previous to 2020-03-12, this won't impact our prediction algorithm
* A new column "Day" has been created, as a day counter starting from the first date
_____no_output_____## 3.2. Compute lags and trends

Enriching a dataset is key to obtain good results. In this case we will apply 2 different transformations:

**Lag**. Lags are a way to compute the previous value of a column, so that the lag 1 for ConfirmedCases would inform the this column from the previous day. The lag 3 of a feature X is simply:
$$X_{lag3}(t) = X(t-3)$$

**Trend**. Transformig a column into its trend gives the natural tendency of this column, which is different from the raw value. The definition of trend I will apply is:
$$Trend_{X} = {X(t) - X(t-1) \over X(t-1)}$$

The backlog of lags I'll apply is 14 days, while for trends is 7 days.  For ConfirmedCases and Fatalities:_____no_output_____

def calculate_trend(df, lag_list, column):
    for lag in lag_list:
        trend_column_lag = "Trend_" + column + "_" + str(lag)
        df[trend_column_lag] = (df[column]-df[column].shift(lag, fill_value=-999))/df[column].shift(lag, fill_value=0)
    return df

def calculate_lag(df, lag_list, column):
    for lag in lag_list:
        column_lag = column + "_" + str(lag)
        df[column_lag] = df[column].shift(lag, fill_value=0)
    return df

ts = time.time()
all_data = calculate_lag(all_data, range(1,7), 'ConfirmedCases')
all_data = calculate_lag(all_data, range(1,7), 'Fatalities')
all_data = calculate_trend(all_data, range(1,7), 'ConfirmedCases')
all_data = calculate_trend(all_data, range(1,7), 'Fatalities')
all_data.replace([np.inf, -np.inf], 0, inplace=True)
all_data.fillna(0, inplace=True)
print("Time spent: ", time.time()-ts)
Time spent:  0.19962286949157715

As you see, the process is really fast. An example of some of the lag/trend columns for Spain:_____no_output_____## 3.3. Add country details

Variables like the total population of a country, the average age of citizens or the fraction of peoople living in cities may strongly impact on the COVID-19 transmission behavior. Hence, it's important to consider these factors. I'm using [Tanu's dataset](https://www.kaggle.com/tanuprabhu/population-by-country-2020) based on Web Scrapping for this purpose._____no_output_____

# Load countries data file
world_population = pd.read_csv("../input/population_by_country_2020.csv")

# Select desired columns and rename some of them
world_population = world_population[['Country (or dependency)', 'Population (2020)', 'Density (P/Km²)', 'Land Area (Km²)', 'Med. Age', 'Urban Pop %']]
world_population.columns = ['Country (or dependency)', 'Population (2020)', 'Density', 'Land Area', 'Med Age', 'Urban Pop']

# Replace United States by US
world_population.loc[world_population['Country (or dependency)']=='United States', 'Country (or dependency)'] = 'US'

# Remove the % character from Urban Pop values
world_population['Urban Pop'] = world_population['Urban Pop'].str.rstrip('%')

# Replace Urban Pop and Med Age "N.A" by their respective modes, then transform to int
world_population.loc[world_population['Urban Pop']=='N.A.', 'Urban Pop'] = int(world_population.loc[world_population['Urban Pop']!='N.A.', 'Urban Pop'].mode()[0])
world_population['Urban Pop'] = world_population['Urban Pop'].astype('int16')
world_population.loc[world_population['Med Age']=='N.A.', 'Med Age'] = int(world_population.loc[world_population['Med Age']!='N.A.', 'Med Age'].mode()[0])
world_population['Med Age'] = world_population['Med Age'].astype('int16')

# Now join the dataset to our previous DataFrame and clean missings (not match in left join)- label encode cities

all_data = all_data.merge(world_population, left_on='Country/Region', right_on='Country (or dependency)', how='left')
all_data[['Population (2020)', 'Density', 'Land Area', 'Med Age', 'Urban Pop']] = all_data[['Population (2020)', 'Density', 'Land Area', 'Med Age', 'Urban Pop']].fillna(0)

# Label encode countries and provinces. Save dictionary for exploration purposes
all_data.drop('Country (or dependency)', inplace=True, axis=1)
all_data['Country/Region'] = le.fit_transform(all_data['Country/Region'])
number_c = all_data['Country/Region']
countries = le.inverse_transform(all_data['Country/Region'])
country_dict = dict(zip(countries, number_c))
all_data['Province/State'] = le.fit_transform(all_data['Province/State'])
number_p = all_data['Province/State']
province = le.inverse_transform(all_data['Province/State'])
province_dict = dict(zip(province, number_p))

_____no_output_____

# 4. Predictions with machine learning

Our obective in this section consists on  predicting the evolution of the expansion from a data-centric perspective, like any other regression problem. To do so, remember that the challenge specifies that submissions on the public LB shouldn only contain data previous to 2020-03-12.

Models to apply:
1. Linear Regression for one country
2. Linear Regression for all countries (method 1)
3. Linear Regression for all countries (method 2)_____no_output_____## 4.1. Linear Regression for one country

Since we are interested into predicting the future time evolution of the pandemic, our first approach consists on a simple Linear Regression. However, remind that the evolution is not linear but exponential (only in the beginning of the infection), so that a preliminar log transformation is needed.

Visual comparison of both cases for Spain and with data from last 10 days informed, starting on March 1st:_____no_output_____As you see, the log transformation results in a fancy straight-like line, which is awesome for Linear Regression. However, let me clarify two important points:

* This "roughly exponential behavior" is only true for the initial infection stages of the pandemic (the initial increasing of infections on the SIR model), but that's exactly the point where most countries are at the moment.

* Why do I only extract the last 10 days of data? For three reasons:
    1. In order to capture exactly the very short term component of the evolution
    2. To prevent the effects of certain variables that have been impacting the transmition speed (quarantine vs free circulation)
    3. To prevent differences on criteria when confirming cases (remember that weird slope on the China plot?)_____no_output_____This first model is very simple, and only elemental features will be considered: Country/Region, date information, Long and Lat. Lags. Engineered columns like lags, trends and country details are not introduced as an input. Finally, the workflow for the Basic Linear Regression model is:
1. **Features**. Select features
2. **Dates**. Filter train data from 2020-03-01 to 2020-03-11
2. **Log transformation**. Apply log transformation to ConfirmedCases and Fatalities
3. **Infinites**. Replace infinites from the logarithm with 0. Given the asymptotic behavior of the logarithm for log(0),this implies that when applying the inverse transformation (exponential) a 1 will be returned instead of a 0. This problem does not impact many countries, but still needs to be tackled sooner or later in order to obtain a clean solution.
4. **Train/test split**. Split into train/valid/test
5. **Prediction**. Linear Regression, training country by country and joining data
6. **Submit**. Submit results in the correct format, and applying exponential to reverse log transformation _____no_output_____

# Filter selected features
data = all_data.copy()
features = ['Id', 'ForecastId', 'Country/Region', 'Province/State', 'ConfirmedCases', 'Fatalities',
       'Day_num', 'Day', 'Month', 'Year', 'Long', 'Lat']
data = data[features]

# Apply log transformation to all ConfirmedCases and Fatalities columns, except for trends
data[['ConfirmedCases', 'Fatalities']] = data[['ConfirmedCases', 'Fatalities']].astype('float64')
data[['ConfirmedCases', 'Fatalities']] = data[['ConfirmedCases', 'Fatalities']].apply(lambda x: np.log(x))

# Replace infinites
data.replace([np.inf, -np.inf], 0, inplace=True)

# Split data into train/test
def split_data(data):

    # Train set
    x_train = data[data.ForecastId == -1].drop(['ConfirmedCases', 'Fatalities'], axis=1)
    y_train_1 = data[data.ForecastId == -1]['ConfirmedCases']
    y_train_2 = data[data.ForecastId == -1]['Fatalities']

    # Test set
    x_test = data[data.ForecastId != -1].drop(['ConfirmedCases', 'Fatalities'], axis=1)

    # Clean Id columns and keep ForecastId as index
    x_train.drop('Id', inplace=True, errors='ignore', axis=1)
    x_train.drop('ForecastId', inplace=True, errors='ignore', axis=1)
    x_test.drop('Id', inplace=True, errors='ignore', axis=1)
    x_test.drop('ForecastId', inplace=True, errors='ignore', axis=1)

    return x_train, y_train_1, y_train_2, x_test

# Linear regression model
def lin_reg(X_train, Y_train, X_test):
    # Create linear regression object
    regr = linear_model.LinearRegression()

    # Train the model using the training sets
    regr.fit(X_train, Y_train)

    # Make predictions using the testing set
    y_pred = regr.predict(X_test)

    return regr, y_pred

# Submission function
def get_submission(df, target1, target2):

    prediction_1 = df[target1]
    prediction_2 = df[target2]

    # Submit predictions
    prediction_1 = [int(item) for item in list(map(round, prediction_1))]
    prediction_2 = [int(item) for item in list(map(round, prediction_2))]

    submission = pd.DataFrame({
        "ForecastId": df['ForecastId'].astype('int32'),
        "ConfirmedCases": prediction_1,
        "Fatalities": prediction_2
    })
    submission.to_csv('submission.csv', index=False)_____no_output_____

Let's try to see results when training with a single country:
_____no_output_____

# Select train (real) data from March 1 to March 22nd
dates_list = ['2020-03-01', '2020-03-02', '2020-03-03', '2020-03-04', '2020-03-05', '2020-03-06', '2020-03-07', '2020-03-08', '2020-03-09',
                 '2020-03-10', '2020-03-11','2020-03-12','2020-03-13','2020-03-14','2020-03-15','2020-03-16','2020-03-17','2020-03-18',
                 '2020-03-19','2020-03-20','2020-03-21','2020-03-22','2020-03-23', '2020-03-24']

# Filter Spain, run the Linear Regression workflow
country_name = "Spain"
day_start = 39
data_country = data[data['Country/Region']==country_dict[country_name]]
data_country = data_country.loc[data_country['Day_num']>=day_start]
X_train, Y_train_1, Y_train_2, X_test = split_data(data_country)
model, pred = lin_reg(X_train, Y_train_1, X_test)

# Create a df with both real cases and predictions (predictions starting on March 12th)
X_train_check = X_train.copy()
X_train_check['Target'] = Y_train_1

X_test_check = X_test.copy()
X_test_check['Target'] = pred

X_final_check = pd.concat([X_train_check, X_test_check])

# Select predictions from March 1st to March 24th
predicted_data = X_final_check.loc[(X_final_check['Day_num'].isin(list(range(day_start, day_start+len(dates_list)))))].Target
real_data = train.loc[(train['Country/Region']==country_name) & (train['Date'].isin(dates_list))]['ConfirmedCases']
dates_list_num = list(range(0,len(dates_list)))

# Plot results
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,6))

ax1.plot(dates_list_num, np.exp(predicted_data))
ax1.plot(dates_list_num, real_data)
ax1.axvline(10, linewidth=2, ls = ':', color='grey', alpha=0.5)
ax1.legend(['Predicted cases', 'Actual cases', 'Train-test split'], loc='upper left')
ax1.set_xlabel("Day count (from March 1st to March 22nd)")
ax1.set_ylabel("Confirmed Cases")

ax2.plot(dates_list_num, predicted_data)
ax2.plot(dates_list_num, np.log(real_data))
ax2.axvline(10, linewidth=2, ls = ':', color='grey', alpha=0.5)
ax2.legend(['Predicted cases', 'Actual cases', 'Train-test split'], loc='upper left')
ax2.set_xlabel("Day count (from March 1st to March 22nd)")
ax2.set_ylabel("Log Confirmed Cases")

plt.suptitle(("ConfirmedCases predictions based on Linear Regression for "+country_name))

# Filter Italy, run the Linear Regression workflow
country_name = "Italy"
day_start = 39
data_country = data[data['Country/Region']==country_dict[country_name]]
data_country = data_country.loc[data_country['Day_num']>=day_start]
X_train, Y_train_1, Y_train_2, X_test = split_data(data_country)
model, pred = lin_reg(X_train, Y_train_1, X_test)

# Create a df with both real cases and predictions (predictions starting on March 12th)
X_train_check = X_train.copy()
X_train_check['Target'] = Y_train_1

X_test_check = X_test.copy()
X_test_check['Target'] = pred

X_final_check = pd.concat([X_train_check, X_test_check])

# Select predictions from March 1st to March 24th
predicted_data = X_final_check.loc[(X_final_check['Day_num'].isin(list(range(day_start, day_start+len(dates_list)))))].Target
real_data = train.loc[(train['Country/Region']==country_name) & (train['Date'].isin(dates_list))]['ConfirmedCases']
dates_list_num = list(range(0,len(dates_list)))

# Plot results
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,6))

ax1.plot(dates_list_num, np.exp(predicted_data))
ax1.plot(dates_list_num, real_data)
ax1.axvline(10, linewidth=2, ls = ':', color='grey', alpha=0.5)
ax1.legend(['Predicted cases', 'Actual cases', 'Train-test split'], loc='upper left')
ax1.set_xlabel("Day count (from March 1st to March 22nd)")
ax1.set_ylabel("Confirmed Cases")

ax2.plot(dates_list_num, predicted_data)
ax2.plot(dates_list_num, np.log(real_data))
ax2.axvline(10, linewidth=2, ls = ':', color='grey', alpha=0.5)
ax2.legend(['Predicted cases', 'Actual cases', 'Train-test split'], loc='upper left')
ax2.set_xlabel("Day count (from March 1st to March 22nd)")
ax2.set_ylabel("Log Confirmed Cases")

plt.suptitle(("ConfirmedCases predictions based on Linear Regression for "+country_name))_____no_output_____

**Observations**:

* The general evolution is captured despite the simplicity of the model
* With training data starting on March 1st, predictions for the first 5 test days tend to be accurate
* Estimations are increasingly worse as time passes (harder to extrapolate)
* Countries that recently confirmed their first contagions are difficult to predict (less data points)
* Countries with 0 cases in the whole training dataset are predicted as non-infected (no datapoints)

** Questions to tackle in next subsections**:
* How to obtain the full submission set?
* What to do for countries with different Provinces/State informed?
* Is there any alternative to manually setting the size of the train data? _____no_output_____# 4.4. Logistic regression with lags

With all the previous results in mind, I quite believe that Linear Regression is a good approach for the early stages of the COVID-19 spread. Of course, this is only true for the initial outbreak we are analysing, and there's no way our model could predict when the number of new infections is going to decrease. But for short-term prediction purposes everything is fine, and we are in disposition to try to improve the results.

Remember those lagged variables we computed some sections before? Now it's time to use them, but first there's a problem to solve. If we use our dataset to predict the next following days of contagions, for the first day all the lags will be reported (from the previous days), but what about the next days? **Many of the lags will be unknown** (flagged as 0), since the number of ConfirmedCases is only known for the train subset. The most simple approach to overcome this is:

1. Begin with the train dataset, with all cases and lags reported
2. Forecast only the following day, through the Linear Regression
3. Set the new prediction as a confirmed case
4. Recompute lags
5. Repeat from step 2 to step 4 for all remaining days

As usual, I'll start training on single countries in order to analyze the behavior of the model with these new features._____no_output_____

# New split function, for one forecast day
def split_data_one_day(data, d):

    #Train
    x_train = data[data.Day_num<d]
    y_train_1 = x_train.ConfirmedCases
    y_train_2 = x_train.Fatalities
    x_train.drop(['ConfirmedCases', 'Fatalities'], axis=1, inplace=True)

    #Test
    x_test = data[data.Day_num==d]
    x_test.drop(['ConfirmedCases', 'Fatalities'], axis=1, inplace=True)

    # Clean Id columns and keep ForecastId as index
    x_train.drop('Id', inplace=True, errors='ignore', axis=1)
    x_train.drop('ForecastId', inplace=True, errors='ignore', axis=1)
    x_test.drop('Id', inplace=True, errors='ignore', axis=1)
    x_test.drop('ForecastId', inplace=True, errors='ignore', axis=1)

    return x_train, y_train_1, y_train_2, x_test

def plot_real_vs_prediction_country(data, train, country_name, day_start, dates_list):

    # Select predictions from March 1st to March 24th
    predicted_data = data.loc[(data['Day_num'].isin(list(range(day_start, day_start+len(dates_list)))))].ConfirmedCases
    real_data = train.loc[(train['Country/Region']==country_name) & (train['Date'].isin(dates_list))]['ConfirmedCases']
    dates_list_num = list(range(0,len(dates_list)))

    # Plot results
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,6))

    ax1.plot(dates_list_num, np.exp(predicted_data))
    ax1.plot(dates_list_num, real_data)
    ax1.axvline(10, linewidth=2, ls = ':', color='grey', alpha=0.5)
    ax1.legend(['Predicted cases', 'Actual cases', 'Train-test split'], loc='upper left')
    ax1.set_xlabel("Day count (starting on March 1st)")
    ax1.set_ylabel("Confirmed Cases")

    ax2.plot(dates_list_num, predicted_data)
    ax2.plot(dates_list_num, np.log(real_data))
    ax2.axvline(10, linewidth=2, ls = ':', color='grey', alpha=0.5)
    ax2.legend(['Predicted cases', 'Actual cases', 'Train-test split'], loc='upper left')
    ax2.set_xlabel("Day count (starting on March 1st)")
    ax2.set_ylabel("Log Confirmed Cases")

    plt.suptitle(("ConfirmedCases predictions based on Logistic Regression for "+country_name))

def plot_real_vs_prediction_country_fatalities(data, train, country_name, day_start, dates_list):

    # Select predictions from March 1st to March 24th
    predicted_data = data.loc[(data['Day_num'].isin(list(range(day_start, day_start+len(dates_list)))))].Fatalities
    real_data = train.loc[(train['Country/Region']==country_name) & (train['Date'].isin(dates_list))]['Fatalities']
    dates_list_num = list(range(0,len(dates_list)))

    # Plot results
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,6))

    ax1.plot(dates_list_num, np.exp(predicted_data))
    ax1.plot(dates_list_num, real_data)
    ax1.axvline(10, linewidth=2, ls = ':', color='grey', alpha=0.5)
    ax1.legend(['Predicted cases', 'Actual cases', 'Train-test split'], loc='upper left')
    ax1.set_xlabel("Day count (starting on March 1st)")
    ax1.set_ylabel("Fatalities Cases")

    ax2.plot(dates_list_num, predicted_data)
    ax2.plot(dates_list_num, np.log(real_data))
    ax2.axvline(10, linewidth=2, ls = ':', color='grey', alpha=0.5)
    ax2.legend(['Predicted cases', 'Actual cases', 'Train-test split'], loc='upper left')
    ax2.set_xlabel("Day count (starting on March 1st)")
    ax2.set_ylabel("Log Fatalities Cases")

    plt.suptitle(("Fatalities predictions based on Logistic Regression for "+country_name))

# Function to compute the Linear Regression predictions with lags, for a certain Country/Region
def lin_reg_with_lags_country(all_data, country_name, day_start, lag_size, country_dict):

    ts = time.time()

    # Filter country and features from all_data (dataset without data leaking)
    data = all_data.copy()
    features = ['Id', 'Province/State', 'Country/Region', 'Lat', 'Long',
           'ConfirmedCases', 'Fatalities', 'ForecastId', 'Day_num']
    data = data[features]

    # Select country an data start (all days)
    data = data[data['Country/Region']==country_dict[country_name]]
    data = data.loc[data['Day_num']>=day_start]

    # Lags
    data = calculate_lag(data, range(1,lag_size), 'ConfirmedCases')
    data = calculate_lag(data, range(1,8), 'Fatalities')

    filter_col_confirmed = [col for col in data if col.startswith('Confirmed')]
    filter_col_fatalities= [col for col in data if col.startswith('Fataliti')]
    filter_col = np.append(filter_col_confirmed, filter_col_fatalities)

    # Apply log transformation
    data[filter_col] = data[filter_col].apply(lambda x: np.log(x))
    data.replace([np.inf, -np.inf], 0, inplace=True)
    data.fillna(0, inplace=True)

    # Start/end of forecast
    start_fcst = all_data[all_data['Id']==-1].Day_num.min()
    end_fcst = all_data[all_data['Id']==-1].Day_num.max()

    for d in list(range(start_fcst, end_fcst+1)):
        X_train, Y_train_1, Y_train_2, X_test = split_data_one_day(data, d)
        model_1, pred_1 = lin_reg(X_train, Y_train_1, X_test)
        data.loc[(data['Country/Region']==country_dict[country_name])
                 & (data['Day_num']==d), 'ConfirmedCases'] = pred_1[0]
        model_2, pred_2 = lin_reg(X_train, Y_train_2, X_test)
        data.loc[(data['Country/Region']==country_dict[country_name])
                 & (data['Day_num']==d), 'Fatalities'] = pred_2[0]

        # Recompute lags
        data = calculate_lag(data, range(1,lag_size), 'ConfirmedCases')
        data = calculate_lag(data, range(1,8), 'Fatalities')
        data.replace([np.inf, -np.inf], 0, inplace=True)
        data.fillna(0, inplace=True)

    print("Process for ", country_name, "finished in ", round(time.time() - ts, 2), " seconds")

    return data

# Function to compute the Linear Regression predictions with lags, for a certain Country/Region and State/province
def lin_reg_with_lags_country_province(all_data, country_name, province_name, day_start, lag_size, country_dict):

    ts = time.time()

    # Filter country and features from all_data (dataset without data leaking)
    data = all_data.copy()
    features = ['Id', 'Province/State', 'Country/Region', 'Lat', 'Long',
           'ConfirmedCases', 'Fatalities', 'ForecastId', 'Day_num']
    data = data[features]

    # Select country an data start (all days)
    data = data[(data['Country/Region']==country_dict[country_name]) & (data['Province/State']==province_dict[province_name])]
    data = data.loc[data['Day_num']>=day_start]

    # Lags
    data = calculate_lag(data, range(1,lag_size), 'ConfirmedCases')
    data = calculate_lag(data, range(1,lag_size), 'Fatalities')

    # Apply log transformation
    filter_col_confirmed = [col for col in data if col.startswith('Confirmed')]
    filter_col_fatalities= [col for col in data if col.startswith('Fataliti')]
    filter_col = np.append(filter_col_confirmed, filter_col_fatalities)
    data[filter_col] = data[filter_col].apply(lambda x: np.log(x))
    data.replace([np.inf, -np.inf], 0, inplace=True)
    data.fillna(0, inplace=True)

    # Start/end of forecast
    start_fcst = all_data[all_data['Id']==-1].Day_num.min()
    end_fcst = all_data[all_data['Id']==-1].Day_num.max()

    for d in list(range(start_fcst, end_fcst+1)):
        X_train, Y_train_1, Y_train_2, X_test = split_data_one_day(data, d)
        model_1, pred_1 = lin_reg(X_train, Y_train_1, X_test)
        data.loc[(data['Country/Region']==country_dict[country_name]) & (data['Province/State']==province_dict[province_name])
                 & (data['Day_num']==d), 'ConfirmedCases'] = pred_1[0]
        model_2, pred_2 = lin_reg(X_train, Y_train_2, X_test)
        data.loc[(data['Country/Region']==country_dict[country_name]) & (data['Province/State']==province_dict[province_name])
                 & (data['Day_num']==d), 'Fatalities'] = pred_2[0]

        # Recompute lags
        data = calculate_lag(data, range(1,lag_size), 'ConfirmedCases')
        data = calculate_lag(data, range(1,lag_size), 'Fatalities')
        data.replace([np.inf, -np.inf], 0, inplace=True)
        data.fillna(0, inplace=True)

    print("Process for ", country_name, "/", province_name, "finished in ", round(time.time() - ts, 2), " seconds")

    return data

_____no_output_____# Run the model for Spain
country_name = 'Spain'
day_start = 35
lag_size = 30

data = lin_reg_with_lags_country(all_data, country_name, day_start, lag_size, country_dict)
plot_real_vs_prediction_country(data, train, country_name, 39, dates_list)
plot_real_vs_prediction_country_fatalities(data, train, country_name, 39, dates_list)

ts = time.time()

# Inputs
country_name = "Italy"
day_start = 35
lag_size = 30

data = lin_reg_with_lags_country(all_data, country_name, day_start, lag_size, country_dict)
plot_real_vs_prediction_country(data, train, country_name, 39, dates_list)
plot_real_vs_prediction_country_fatalities(data, train, country_name, 39, dates_list)Process for  Spain finished in  17.87  seconds
Process for  Italy finished in  17.79  seconds

**Observations**:
* **Parameters**. Two full weeks of training used (from February 26th to March 11th), with their previous 30 lags
* **Enough data**. (Spain, Italy, Germany). For countries with several ConfirmedCases!=0 in the train dataset (prior to March 11th), predictions are very precise and similar to actual confirmed data
* **Poor data**. (Algeria). Countries with a small number of datapoints in the train dataset show a potentially disastrous prediction. Given the small number of cases, the log transformation followed by a Linear Regression is not able to capture the future behavior
* **No data**. (Andorra). When there are no confirmed cases in the train dataset, the model predicts always 0 infections
* **Fatalities**. As we clarified in [section 4.1](#section41)., filling negative infinities by 0 impacts the inverse transformation of the logarithm. Values that should be 0 are now 1 ($e^0$), and that's the reason of the "weird" (see Andorra) plots for countries that have no fatalities in the training dataset_____no_output_____


# Notebook from yelabucsf/scrna-parameter-estimation
Path: analysis/simulation/legacy/method_of_moments/gmm.ipynb

# Speed optimization

Largest optimization introduced is pre-computing groupwise means_____no_output_____

import numpy as np
import scipy.stats as stats
import time
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import scipy as sp
import sklearn.linear_model as sk_lm
import statsmodels.api as sm
import imp
from IPython.core.display import display, HTML
display(HTML(""))_____no_output_____import numpy as np
from scipy.special import psi
from statsmodels.sandbox.regression.gmm import GMM_____no_output_____import sys
sys.path.append('/home/mkim7/Github/scrna-parameter-estimation/examples/simulation')
import simulate as sim_____no_output_____# def mean_objective_perm(params, perm_means, perm_counts):

#     x_mat = df_mean[['treatment', 'intercept']].values
#     y_val, y_counts = perm_means, perm_counts
#     beta_hat = perm_start.reshape(-1, num_perm, order='F')

#     momcond = np.einsum('nm,nk->nmk', x_mat, y_val) - np.einsum('nm,nk->nmk', x_mat,np.exp(x_mat@beta_hat) )
#     weighted_momcond = (momcond * y_counts).mean(axis=0)

#     return (weighted_momcond**2).sum()_____no_output_____def var_objective_perm(params):

    y_val, y_counts = df['y'].values.reshape(-1, 1), perm_proportions.reshape(-1, 1, num_perm)/N
    x_mat = df[['treatment', 'intercept']].values

    beta_hat = params.reshape(-1, num_perm, order='F')
    mean_beta = self.mean_beta.reshape(-1, 1)

    var_beta = params.reshape(-1, 1)

    a = (x_mat*y**2)/self.q_sq
    b = -(x_mat*y)*(1-self.q_sq/self.q)/self.q_sq
    c = -2*y*np.exp(x_mat@mean_beta)*x_mat/self.q
    d = np.exp(2*x_mat@mean_beta)*x_mat
    e = -x_mat*np.exp(x_mat@(self.gamma*mean_beta + var_beta))
    return a + b + c + d + e_____no_output_____

### Set up some data_____no_output_____

p = 0.1
noise_level = 0.2049755522580501
p_sq = (noise_level+1)*p**2
N = 10000
num_perm = 10000
X, z = sim.simulate_true_counts(
    N=N,
    mean_coef = np.array([1.0, 0.1]),
    var_coef = np.array([1.0, 0.1]),
    num_levels=4,
    p=1)

y = sim.simulate_dropout(z, p, p_sq)

hi

### Preprocessing step - must occur for each gene_____no_output_____

%%time
df = pd.DataFrame(
    data=np.hstack([X, y]),
    columns=['treatment', 'intercept', 'y'])
df_val = df.groupby(df.columns.tolist()).size().reset_index(name='count')
df_val['proportion'] = df_val['count'] / X.shape[0]

perm_proportions = stats.multinomial.rvs(
    N,
    df_val['proportion'],
    size=num_perm).T

df_mean = df.groupby(['treatment', 'intercept']).agg({'treatment':'size', 'y':'mean'}).rename(columns={'treatment':'group_count', 'y':'group_mean'}).reset_index()
df_mean['group_id'] = np.arange(df_mean.shape[0])
df_val = df_val.merge(df_mean, on=['treatment', 'intercept'], how='left')
df_mean['group_proportion'] = df_mean['group_count']/N

perm_vals = df_val['y'].values.reshape(-1, 1) * perm_proportions / df_val['group_count'].values.reshape(-1, 1)

perm_means = np.zeros((df_mean.shape[0], num_perm))
perm_counts = np.zeros((df_mean.shape[0], num_perm))
for group_id in range(df_mean.shape[0]):
    perm_means[group_id, :] = perm_vals[df_val['group_id']==group_id, :].sum(axis=0)
    perm_counts[group_id, :] = perm_proportions[df_val['group_id']==group_id, :].sum(axis=0)CPU times: user 71.1 ms, sys: 0 ns, total: 71.1 ms
Wall time: 70.3 ms
def mean_objective_perm(params, perm_means, perm_proportions, df_mean, sep=False):

    x_mat = df_mean[['treatment', 'intercept']].values
    y_val, y_counts = perm_means, perm_proportions/N
    beta_hat = params.reshape(-1, num_perm, order='F')

    momcond = np.einsum('nm,nk->nmk', x_mat, y_val)/p

    momcond -= np.einsum('nm,nk->nmk', x_mat,np.exp(x_mat@beta_hat) )

    weighted_momcond = np.einsum('ijk,ik->ijk', momcond, y_counts).mean(axis=0)

    #return tp1, tp2, tp3, tp4, tp5
    return (weighted_momcond**2).sum(axis=0) if sep else (weighted_momcond**2).sum()

def mean_objective_perm_old(params, sep=False):

    y_val, y_counts = df_val['y'].values.reshape(-1, 1), perm_proportions.reshape(-1, 1, num_perm)/N
    x_mat = df_val[['treatment', 'intercept']].values

    beta_hat = params.reshape(-1, num_perm, order='F')

    part_1 = ((x_mat*y_val)/p)

    part_1 = part_1.reshape(part_1.shape[0], part_1.shape[1], -1)

    part_2 = x_mat.reshape(x_mat.shape[0], x_mat.shape[1], -1)

    part_3 = np.exp(x_mat@beta_hat).reshape(-1, 1, num_perm)

    momcond = (part_1 - part_2*part_3)
    weighted_momcond = (momcond * y_counts).mean(axis=0)

    return (weighted_momcond**2).sum(axis=0) if sep else (weighted_momcond**2).sum()

def mean_objective(params, y_val, y_counts, x_mat):

#     y_val, y_counts = df['y'].values.reshape(-1, 1), df['proportion'].values.reshape(-1, 1)
#     x_mat = df[['treatment', 'intercept']].values

    beta_hat = params.reshape(-1, 1)

    a =  (x_mat*y_val)/p - (x_mat*np.exp(x_mat@beta_hat))

    momcond = (a * y_counts).mean(axis=0)

    return (momcond**2).sum()

_____no_output_____

### Starting values for large optimization_____no_output_____

perm_start = np.concatenate([np.array([1.0042849880247497, 0.11339659771439722]) for i in range(num_perm)])_____no_output_____

### Debugging objective functions_____no_output_____

%%time
mean_objective_perm_old(perm_start)#[:, :, 0]CPU times: user 41.6 ms, sys: 509 µs, total: 42.1 ms
Wall time: 11.9 ms
%%time
times = mean_objective_perm(perm_start, perm_means, perm_counts,df_mean)CPU times: user 12.4 ms, sys: 0 ns, total: 12.4 ms
Wall time: 2.99 ms
%%time
res_old = sp.optimize.minimize(
        lambda x: mean_objective_perm_old(x, sep=False),
        perm_start,
        method='l-bfgs-b',
        jac=lambda x: gradient(x, lambda y: mean_objective_perm_old(y, sep=True), num_features=2),
        options={'disp': True})
res_old.xCPU times: user 3.24 s, sys: 32.3 ms, total: 3.28 s
Wall time: 827 ms
%%time
res = sp.optimize.minimize(
        lambda x: mean_objective_perm(x, perm_means, perm_counts, df_mean),
        perm_start,
        method='l-bfgs-b',
        options={'gtol':1e-4,'ftol':1e-5},
        jac=lambda x: gradient(x, lambda y:  mean_objective_perm(y, perm_means, perm_counts, df_mean, sep=True), num_features=2))
res.xCPU times: user 670 ms, sys: 15.5 ms, total: 686 ms
Wall time: 171 ms
sns.distplot(res.x[::2])
sns.distplot(res_old.x[::2])_____no_output_____%%time
gradient(perm_start, lambda y: mean_objective_perm(y, perm_means, perm_counts, df_mean, sep=True))CPU times: user 2.31 ms, sys: 111 µs, total: 2.42 ms
Wall time: 2.32 ms
%%time
gradient(perm_start, lambda y: mean_objective_perm_old(y, sep=True))CPU times: user 2.55 ms, sys: 0 ns, total: 2.55 ms
Wall time: 2.43 ms
perm_start[0]/2_____no_output_____np.split(perm_start, perm_start.shape[0]/2)_____no_output_____sp.linalg.block_diag(*np.split(perm_start, perm_start.shape[0]/2))_____no_output_____EPS = np.MachAr().eps

def _get_epsilon(x, s, epsilon, n):
    if epsilon is None:
        h = EPS**(1. / s) * np.maximum(np.abs(x), 0.1)
    else:
        if np.isscalar(epsilon):
            h = np.empty(n)
            h.fill(epsilon)
        else:  # pragma : no cover
            h = np.asarray(epsilon)
            if h.shape != x.shape:
                raise ValueError("If h is not a scalar it must have the same"
                                 " shape as x.")
    return h

def approx_fprime(x, f, epsilon=None, args=(), kwargs={}, centered=False):

    n = len(x)
    # TODO:  add scaled stepsize
    f0 = f(*((x,)+args), **kwargs)
    print(f0)
    dim = np.atleast_1d(f0).shape  # it could be a scalar
    grad = np.zeros((n,) + dim, np.promote_types(float, x.dtype))
    #print(grad.shape)
    ei = np.zeros((n,), float)

    if not centered:
        epsilon = _get_epsilon(x, 2, epsilon, n)
        for k in range(n):
            ei[k] = epsilon[k]
            print(epsilon)
            print(f(*((x+ei,) + args), **kwargs))
            grad[k, :] = (f(*((x+ei,) + args), **kwargs) - f0)/epsilon[k]
            ei[k] = 0.0
    else:
        epsilon = _get_epsilon(x, 3, epsilon, n) / 2.
        for k in range(len(x)):
            ei[k] = epsilon[k]
            grad[k, :] = (f(*((x+ei,)+args), **kwargs) -
                          f(*((x-ei,)+args), **kwargs))/(2 * epsilon[k])
            ei[k] = 0.0
    return grad.squeeze().T

def gradient(x, f_sep, num_features=2, epsilon=None, mat=False, args=(), kwargs={}):

    n = len(x)
    f0 = f_sep(*((x,)+args), **kwargs)
    dim = (1,)
    grad = np.zeros((n,) + dim, np.promote_types(float, x.dtype))
    epsilon = _get_epsilon(x, 2, None, n)
    ei = np.zeros((n,),float)

    for start in range(num_features):
        ei[start::num_features] = epsilon[start::num_features]
        res = ((f_sep(*((x+ei,) + args), **kwargs) - f0)/epsilon[start::num_features]).reshape(-1, 1)
        grad[start::num_features, :] = res
        ei[start::num_features] = 0.0

    grad = grad.squeeze().T
    return sp.linalg.block_diag(*np.split(grad, grad.shape[0]/num_features)) if mat else grad

def mean_hessian_p(x, p, num_features=2, epsilon=None, args=(), kwargs={'num_features':2}):

    n = len(x)
    f = mean_gradient
    f0 = f(*((x,)+args), **kwargs)
    epsilon = _get_epsilon(x, 2, None, n)
    ei = np.zeros((n,),float)
    hess = [] # list of length num_features

    for start in range(num_features):
        ei[start::num_features] = epsilon[start::num_features]
        hess.append((f(*((x+ei,) + args), **kwargs) - f0)/epsilon[start::num_features][0])
        ei[start::num_features] = 0.0

    hess = np.vstack(hess)

    Hp_components = [hess[:, i:(i+num_features)]@p[i:(i+num_features)] for i in range(int(x.shape[0]/num_features))]

    return np.concatenate(Hp_components)
    _____no_output_____perm_start_____no_output_____perm_start[0::2]_____no_output_____


# Notebook from mwcraig/glowing-waffles
Path: glowing_waffles/notebooks/image-viewer-working-copy.ipynb

from __future__ import print_function, division

import numpy as np
import pandas as pd

import astropy.units as u
import astropy.coordinates as apycoord
from astropy.visualization import scale_image
from astropy.nddata import block_replicate
from astropy.table import Table, Column, MaskedColumn
from astropy.time import Time

import glowing_waffles.differential_photometry as df

from glowing_waffles.io import parse_aij_table

from ccdproc import CCDData

from scipy import optimize

%matplotlib inline
import matplotlib.pyplot as plt
_____no_output_____

# Read image_____no_output_____

ccd = CCDData.read('kelt-1-055R.fit', unit='adu')   #Read in a single fits image_____no_output_____# This function really should be somewhere else eventually.
def scale_and_downsample(data, downsample=4,
                         min_percent=20,
                         max_percent=99.5):

    scaled_data = scale_image(data,
                              min_percent=min_percent,
                              max_percent=max_percent)

    if downsample > 1:
        scaled_data = block_reduce(scaled_data,
                                   block_size=(downsample, downsample))
    return scaled_data
_____no_output_____#use the catalog_search function to find the apass stars in the frame of the image read above
apass, apass_x, apass_y = df.catalog_search(ccd.wcs, ccd.shape, 'II/336/apass9',
                                            ra_column='RAJ2000',
                                            dec_column='DEJ2000',
                                            radius=2.0,
                                            clip_by_frame=False)

#Get any known variable stars from a new catalog search of VSX
vsx, vsx_x, vsx_y = df.catalog_search(ccd.wcs, ccd.shape, 'B/vsx/vsx',
                                      ra_column='RAJ2000',
                                      dec_column='DEJ2000',
                                      radius=2.0,
                                      clip_by_frame=False)
vsx_names = vsx['Name']      #Get the names of the variables_____no_output_____#Creates a boolean array of the apass stars that have well defined magnitudes and color
keep_rows_that_have = {
    'e_r_mag': '<0.05',
    'u_e_r_mag': '=0',
    'e_B-V': '<0.1'
}

apass_in_bright2 = df.catalog_clean(apass, remove_rows_with_mask=True, **keep_rows_that_have)

apass_x, apass_y = ccd.wcs.all_world2pix(apass_in_bright2['RAJ2000'], apass_in_bright2['DEJ2000'], 0)
#create new lists of apass stars and x y pixel coordinates using boolean array
#apass_in_bright, in_apass_x, in_apass_y = apass[apass_bright], apass_x[apass_bright], apass_y[apass_bright]_____no_output_____disp = scale_and_downsample(ccd.data, downsample=1)_____no_output_____plt.figure(figsize=(12, 7))
plt.imshow(disp, cmap='gray', origin='lower')
plt.scatter(vsx_x, vsx_y, c='none', s=100, edgecolor='cyan')
plt.title('Blue: VSX, Yellow: APASS', fontsize=20)

for x, y, m in zip(vsx_x, vsx_y, vsx_names):
    plt.text(x, y, str(m), fontsize=18, color='cyan')

plt.scatter(apass_x, apass_y, c='none', s=50, edgecolor='yellow', alpha=0.5, marker='o')

for x, y, c in zip(apass_x, apass_y, apass_in_bright2):
    plt.text(x, y, c['r_mag'], fontsize=12, color='yellow')

plt.xlim(0, ccd.shape[1])
plt.ylim(0, ccd.shape[0])_____no_output_____#Read in the raw measurements file in case the parse function doesn't work as expected
aij_raw = Table.read('all_kelt_1_good_rows.csv')
#Get the sources
#sources = uniformize_source_names(aij_raw)

#Use glowing waffles to parse the measurements file
aij_stars = parse_aij_table('all_kelt_1_good_rows.csv')_____no_output_____#get the ra (which aij gives in hour angle) from the raw aij data
aij_ra = [np.mean(aij_stars[source].ra) for source in range(len(aij_stars))]
#get the dec from the raw aij data
aij_dec = [np.mean(aij_stars[source].dec) for source in range(len(aij_stars))]
#get the julian date from parsed measurements file
aij_jd = aij_stars[0].jd_utc_start

#Create a list of aij coordinates using apycoord.SkyCoord function
aij_coordinates = apycoord.SkyCoord(aij_ra, aij_dec, unit=(u.deg, u.deg))
#create a list of well defined apass stars' coordinates using apycoord.SkyCoord function
apass_coordinates = apycoord.SkyCoord(apass_in_bright2['RAJ2000'],
                                      apass_in_bright2['DEJ2000'],
                                      unit='deg')

vsx_coordinates = apycoord.SkyCoord(vsx['RAJ2000'], vsx['DEJ2000'], unit='deg')

_____no_output_____vsx_idx, apass_idx, d2d, d3d = apass_coordinates.search_around_sky(vsx_coordinates, 1 * u.arcsec)
if len(set(apass_idx)) != len(apass_idx):
    raise RuntimeError('Multiple matches found.')

print(len(apass_in_bright2))
# Remove any APASS stars that are in VSX
print(apass_idx)
print(vsx_coordinates[vsx_idx])
print(apass_coordinates[apass_idx])
if len(apass_idx):
    apass_in_bright2.remove_rows(apass_idx)

print(len(apass_in_bright2))
# Recreate APASS coordinate list after deleting some entries
apass_coordinates = apycoord.SkyCoord(apass_in_bright2['RAJ2000'],
                                      apass_in_bright2['DEJ2000'],
                                      unit='deg')_____no_output_____plt.plot(aij_stars[4].magnitude)     #Plot one of the stars just to see what it looks like
plt.ylabel('Instrumental Mag')
plt.xlabel('Image')

#compare different ways to get magnitudes.
#print(aij_mags[0][0], aij_stars[0].magnitude[0])
#print(-2.5*np.log10(174940.8)+2.5*np.log10(60))_____no_output_____

# APASS Filter Corrections
## Transform the APASS r magnitudes into R magnitudes using APASS r and i magnitudes

The equation used is R-feder - r-apass = A*c**3 + B*c**2 + C*c + D

Where..._____no_output_____

#Transform the apass magnitudes into the R filter we use
apass_R_mags = df.filter_transform(apass_in_bright2, 'R', r='r_mag', i='i_mag')_____no_output_____
apass_in_bright2.add_column(Column(data=apass_R_mags, name='R'))_____no_output_____# Yuck. Just....yuck

def single_time_table(stars, row):
    mags = []
    mag_err = []
    RA = []
    Dec = []

    for star in stars:
        mags.append(star.magnitude[row])
        RA.append(star.ra[row].value)
        Dec.append(star.dec[row])
        SNR = star.counts[row]/star.error[row]
        mag_err.append(1.0857/SNR)
    mags = Column(name='R', data=mags)
    RA = Column(name='RA', data=RA, unit='degree')
    Dec = Column(name='Dec', data=Dec, unit='degree')
    errs = Column(name='e_R', data=mag_err)

    return Table([mags, RA, Dec, errs])_____no_output_____aij_index, apass_index, d2d, d3d = apass_coordinates.search_around_sky(aij_coordinates, 1 * u.arcsec)

# Check for bad match (more than one object in APASS matches one object in AIJ)
#

# This checks for duplicates, because a python set consists of the unique elements
# of a list.
assert len(set(aij_index)) == len(aij_index)_____no_output_____apass_color = apass_in_bright2['Bmag'][apass_index] - apass_in_bright2['Vmag'][apass_index]
aij_table = single_time_table(aij_stars, 0)[aij_index]
transform_params = df.calculate_transform_coefficients(aij_table['R'],
                                                       apass_in_bright2['R'][apass_index],
                                                       apass_color,
                                                       input_mag_error=aij_table['e_R'])_____no_output_____transform_params_____no_output_____plt.plot(apass_color, apass_in_bright2['R'][apass_index] - aij_table['R'], 'o')
xlims = np.array(plt.xlim())
plt.plot(xlims, transform_params[1] + transform_params[0] * xlims)_____no_output_____#Create empty list for the corrections (slope and intercept)
corrections = []
#Create empy list for the error in the corrections
fit_error = []
all_Rminusr_error = []
#loop over all images
for idx in range(len(aij_stars[0].ra)):
    # Calculate APASS color
    #    NOTE: why not use the B-V column???
    apass_color = apass_in_bright2['Bmag'][apass_index] - apass_in_bright2['Vmag'][apass_index]

    aij_table = single_time_table(aij_stars, idx)[aij_index]
    transform_params = df.calculate_transform_coefficients(aij_table['R'],
                                                           apass_in_bright2['R'][apass_index],
                                                           apass_color,
                                                           input_mag_error=aij_table['e_R'])
    corrections.append(transform_params)
    # Plot every 25 fits...
    if not (idx % 25):
        #Plot the BminusV vs Rminusr graph for the first image
        Rminusr = apass_in_bright2['R'][apass_index]-aij_table['R']
        plt.plot(apass_color, Rminusr, 'o')
        x = np.array(apass_color)
        plt.plot(x, Rminusr, '+')
        plt.plot(x, transform_params[0] * x + transform_params[1])
        plt.title(str(idx))
        plt.show()

_____no_output_____plt.figure(figsize=(10, 5))
#This just plots the slope of the linear fit for all images
for image in range(len(aij_stars[0].ra)):
    time = Time(aij_stars[0].jd_utc_start[image], scale='utc', format='jd')
    ax = plt.scatter(image, corrections[image][0])
plt.ylabel('Color correction')
plt.xlabel('Image number')_____no_output_____for image in range(len(aij_stars[0].ra)):
    plt.scatter(aij_stars[0].airmass[image], corrections[image][0])
plt.ylabel('Color correction')
plt.xlabel('airmass')
plt.xlim(1, 2)_____no_output_____plt.figure(figsize=(10, 5))
#This just plots the slope of the linear fit for all images
for image in range(len(aij_stars[0].ra)):
    time = Time(aij_stars[0].jd_utc_start[image], scale='utc', format='jd')
    ax = plt.scatter(image, corrections[image][1])
plt.ylabel('Zero point')
plt.xlabel('Image number')_____no_output_____

• Source Radius: radius of the aperture used to calculate
net integrated counts. In fixed aperture mode, this value
is the aperture radius set by the user. In variable aperture
mode, this value represents the actual aperture radius
calculated as the product of the average FWHM in
the image and the multiplicative factor set in the MultiAperture
Measurements set up panel.

• FWHM mult: in variable aperture mode, this value is
the FWHM multiplier set in the Multi-Aperture Measurements
set up panel. In fixed aperture mode, this
column is not included in the table.

• Source Rad(base): in variable aperture mode, this
value represents the fixed aperture radius set by the user
and should be set to a number greater than 1.5 times the
maximum FWHM expected to ensure proper measurement
of FWHM. In fixed aperture mode, this column is
not included in the table.

• Sky Rad(min): radius of the inner edge of the annulus
used to calculate the sky background

• Sky Rad(max): radius of the outer edge of the annulus
used to calculate the sky background_____no_output_____

def source_error(source_number):             #Returns the column heading for a star's source-sky error in the measurements file
    col_name = 'Source-Sky_C'+str(source_number + 1)
    return col_name

#Calculate the error in the magnitudes to be used in the apass calibration
#%matplotlib inline
#define the gain of the ccd
gain = 1.5
#define the read noise of the ccd
read_noise = 30.0

npix = np.pi * 15**2  # pixel^2, estimated aperture size
n_sky = 50.0   # counts/pixel, estimated upper limit

#create a magnitude error list
mag_err = []
for source in range(len(aij_stars)):
    #use source_error function to get error in source-sky
    err = aij_stars[source].error
    #calculate the signal to noise ratio
    #     gain * aij_raw[source_column(source)]
    snr = aij_stars[source].counts / err
    #calculate magnitude error
    mag_e = 1/snr
    #add magnitude error to list mag_err
    mag_err.append(mag_e)_____no_output_____mag_err[0]_____no_output_____

Make a table with columns for:

+ time
+ instrumental magnitudes
+ transform color term
+ transform zero point
+ APASS color

Then the transform for each star is just a matter of multiplying/adding columns.

Well, or maybe not._____no_output_____

apass_coordinates = apycoord.SkyCoord(apass['RAJ2000'],
                                      apass['DEJ2000'],
                                      unit='deg')
all_apass_index, d2d, d3d = aij_coordinates.match_to_catalog_sky(apass_coordinates)

good = d2d < 3 * u.arcsec
if not good.all():
    print("Darn.")
    print(good.sum())
    print(d2d)

apass_colors = apass['B-V'][all_apass_index]
apass_color_errors = apass['e_B-V'][all_apass_index]_____no_output_____#create an array to store the corrected curves
corrected_curves = [] # np.zeros_like(aij_stars[0].magnitude)
#create an array to store the error in the corrected curves
corrected_curves_er = [] # np.zeros_like(corrected_curves)
#create an array to store the signal to noise ratios for all stars
all_SNR = [] # np.zeros_like(corrected_curves)
#loop over all of the stars

corrections = np.array(corrections)

for obj, star in enumerate(aij_stars):
    #print a little update just to know where the progress is at
    print('here in the object loop, we are now studying object number', str(obj))

    #get the error in the apass color for the star
    er_BV = apass_color_errors[obj]
    #get the apass color for the star
    BminusV = apass_colors[obj]
    #loop over all of the images

    transformed_mag = star.magnitude + apass_colors[obj] * corrections[:, 0] + corrections[:, 1]
    corrected_curves.append(np.array(transformed_mag))

#Plot the corrected light curve of star index 0
plt.plot(corrected_curves[5])    #looks kinda screwed up... shouldn't they meet into eachother smoothly with the
                                 #apass corrections. They are on different nights and all but still...?_____no_output_____#create an array of the dates each image was taken at (JD rounded to day)
night = np.floor(np.array(aij_stars[0].mjd_start + 0.5)) -1
#print(night)
#find the unique nights in the array of nights
unique_nights = np.unique(night)
#take out one of the nights if you want
unique_nights = set(unique_nights)# - set([57249.0])
#sort the list of nights obviously
unique_nights = sorted(unique_nights)
#find the number of nights to be used for plotting
number_of_nights = len(unique_nights)

night_stdev = [[] for a_night in range(number_of_nights)]
for index, star in enumerate(aij_stars):

    #loop over all of the nights and their index in unique nights
    for i, this_night in enumerate(unique_nights):
        #create a night mask that is a boolean in the shape of night
        night_mask = (night == this_night)
        stdev = np.std(corrected_curves[index][night_mask])
        night_stdev[i].append(stdev)

comp_stars = []

for index, stdevs in enumerate(night_stdev):
    good_comps = list(np.argpartition(np.array(stdevs),5)[2:7])
    comp_stars.append(good_comps)_____no_output_____

## Functions for making single magnitude plot
### This section is taken from the multi-night photometry notebook from a kelt-1 notebook._____no_output_____

#Define the function to be used to plot the differential magnitudes
def plot_magnitudes(mags=None, errors=None, times=None, source=None, night=None, ref_mag=0, color=None, name=''):
    #calcualte the mean of the magnitudes passed
    mean = mags.mean()
    #calcualte the standard deviation of the magntiudes passed
    std = mags.std()
    #plot the magnitudes vs time
    plt.errorbar(times, mags, yerr=errors, fmt='o',
                 label='{}, stdev: {:5.3f}\nnight: {}'.format(source, std, night))
    #change the xlims of the plot to reflect the times
    plt.xlim(times.min(), times.max())
    #Plots a line correspinding to the mean
    plt.plot(plt.xlim(), [mean, mean], 'k--', )
    #plots a line corresponding to the upper limit of the mean
    plt.plot(plt.xlim(), [mean + std, mean + std], 'k:')
    #plots a line corresponding to the lower limit of the mean
    plt.plot(plt.xlim(), [mean - std, mean - std], 'k:')
    #plt.plot(pd.rolling_mean(times, 20, center=True),
    #         pd.rolling_mean(mags, 20, center=True),
    #         color='gray', linewidth=3)
    # Make sure plot range is at least 0.1 mag...
    min_range = 0.1
    #find the ylim of the plot
    ylim = plt.ylim()
    #check if the difference in the limits is less then the min range
    if ylim[1] - ylim[0] < min_range:
        #if less then the mid range then change the y limits to be min_range different
        plt.ylim(mean - min_range/2, mean + min_range/2)

    #find the new ylim of the plot
    ylim = plt.ylim()
    # Reverse vertical axis so brighter is higher
    plt.ylim((ylim[1], ylim[0]))
    plt.title(str(color) + name)
    plt.legend()
    #send back the mean and the standard deviation of the plot
    return mean, std_____no_output_____#define array for the comparison magnitudes
corrected_counts = 10**(-(corrected_curves-2.5*np.log10(aij_raw['EXPOSURE']))/2.5)
comp_counts = []

for i, this_night in enumerate(unique_nights):
        #create a night mask that is a boolean in the shape of night
        night_mask = (night == this_night)
        comp = np.zeros(sum(night_mask))
        for star in comp_stars[i]:
            if star not in (4, 12, 17):
                comp += corrected_counts[star][night_mask]
        comp_counts += list(comp)
comp_counts = np.array(comp_counts)

#calculate the differential magnitudes by simply taking the difference between the corrected curves and the comparison magnitudes"""
diff_corrected = -2.5*np.log10(corrected_counts/comp_counts)_____no_output_____vsx_index, d2d, d3d = aij_coordinates.match_to_catalog_sky(vsx_coordinates)

vsx_match = d2d < 1 * u.arcsec_____no_output_____#loop over all of the stars and their index
for index, star in enumerate(aij_stars):
    #start a figure...
    plt.figure(figsize=(5*number_of_nights, 5))

    #define a list for night means
    night_means = []
    #define a list for night standard deviations
    night_stds = []
    #define a list for night bins? I suppose
    night_bins = []
    #get the color of the star
    BminusV = apass_colors[index]
    if vsx_match[index]:
        name = " VAR: " + vsx_names[vsx_index[index]]
    else:
        name = ''
    #loop over all of the nights and their index in unique nights
    for i, this_night in enumerate(unique_nights):
        plt.subplot(1, number_of_nights + 1, i + 1)
        #create a night mask that is a boolean in the shape of night
        night_mask = (night == this_night)
        #get the night mean and std from the plot magnitudes function and plot the magnitudes
        night_mean, night_std = plot_magnitudes(mags=corrected_curves[index][night_mask],
                                                times=aij_raw['BJD_TDB'][night_mask], source = index+1,
                                               night = this_night, color = BminusV, name=name)
""", errors = corrected_curves_er[index][night_mask]"""_____no_output_____


# Notebook from mavrix93/LightCurvesClassifier
Path: example_notebooks/compl_procedure.ipynb

import sys, os
import pandas as pd
from sklearn.model_selection import train_test_split

from lcc.db_tier.connectors import FileManager, OgleII, OgleII
from lcc.data_manager.package_reader import PackageReader
from lcc.stars_processing.systematic_search.stars_searcher import StarsSearcher, StarsSearcherRedis
from lcc.stars_processing.tools.visualization import plotProbabSpace
from lcc.stars_processing.tools.params_estim import ParamsEstimator
from lcc.utils.output_process_modules import saveIntoFile, loadFromFile
from lcc.utils.stars import saveStars, plotStarsPicture
from lcc.utils.helpers import get_combinations
from lcc.api.input_parse import parse_tun_query

%matplotlib inline

from matplotlib import pylab
pylab.rcParams['figure.figsize'] = (24.0, 8.0)_____no_output_____

# Define the task
In this example our task is to make a model which would distinguish quasars from Be stars which light curves are very simmilar. _____no_output_____

# We will be using Abbe value and variogram slope to describe each light curve
# and the "brain" of the filter will be Gradient Boosting
descr_name1 = "AbbeValueDescr"
descr_name2 = "VariogramSlopeDescr"
decid_name = "GradBoostDec"

# Parameters to tune and their ranges
tun_param1 = "bins"
bin_from = 10
bin_to = 150
bin_step = 50

tun_param2 = "days_per_bin"
dpb_from = 30
dpb_to = 110
dpb_step = 10

# Load example stars which are included in the package
obt_method = "FileManager"
quasars_path = PackageReader.getSamplePath("qso")
be_stars_path = PackageReader.getSamplePath("be_stars")_____no_output_____

# Show available descriptors and classifiers
Note: You can arbitrary number of descriptors and classifiers_____no_output_____

all_descriptors = PackageReader().getClassesDict("descriptors")
all_deciders = PackageReader().getClassesDict("deciders")

print("Descriptors: {}\n".format(", ".join(all_descriptors.keys())))
print("Deciders: {}".format(", ".join(all_deciders.keys())))Using TensorFlow backend.

Note: `all_descriptors` and `all_deciders` are dictionaries of descriptors and deciders classes which can be directly used. For example:
```
my_descriptor = all_descriptors["AbbeValueDescr"](bins=100)
```

is same as:
```
from lcc.stars_processing.descriptors import AbbeValueDescr

my_descriptor = AbbeValueDescr(bins=100)
```_____no_output_____# Get descriptors and a classifier_____no_output_____

abbe_descr = all_descriptors.get(descr_name1)
vario_slope_descr = all_descriptors.get(descr_name2)
decider = all_deciders.get(decid_name)_____no_output_____

# Make list of all parameters combination_____no_output_____

combinations = get_combinations([":".join([descr_name1, tun_param1]), ":".join([descr_name2, tun_param2])],
                                 range(bin_from, bin_to, bin_step), range(dpb_from, dpb_to, dpb_step))
tun_params = parse_tun_query(combinations)_____no_output_____tun_params[:3]_____no_output_____

# Load stars_____no_output_____

quasars = FileManager({"path": quasars_path}).getStars()
be_stars = FileManager({"path": be_stars_path}).getStars()Loading FITS files:: 100%|██████████| 137/137 [00:15<00:00,  8.90it/s]
Loading FITS files:: 100%|██████████| 221/221 [00:18<00:00, 12.26it/s]

### Clean sample
Keep just stars with light curves which have certain size_____no_output_____

quasars = [st for st in quasars if st.lightCurve and len(st.lightCurve.mag) >= bin_to]
be_stars = [st for st in be_stars if st.lightCurve and len(st.lightCurve.mag) >= bin_to]_____no_output_____

### Train - test split_____no_output_____

quasars_train, quasars_test = train_test_split(quasars, train_size=0.8)
be_stars_train, be_stars_test = train_test_split(be_stars, train_size=0.8)_____no_output_____

### Show random quasars_____no_output_____

import random

random.shuffle(quasars)
for star in quasars[:3]:
    print("*"*20)
    print(star)
    print()********************
MACHO identifier:	name: 64.8092.454

********************
xray identifier:	name:
mqs identifier:	name: MQS J053212.24-693130.9
radio identifier:	name:
OgleII identifier:	name: LMC_SC2_407170	db_ident: {'field': 'LMC_SC2', 'starid': 407170}
	Coordinate: 05h32m12.3684s -69d31m30.792s

********************
xray identifier:	name: 3XMM J050713.6-685952
mqs identifier:	name: MQS J050713.52-685952.5
radio identifier:	name:
OgleII identifier:	name: LMC_SC13_220502	db_ident: {'field': 'LMC_SC13', 'starid': 220502}
	Coordinate: 05h07m13.6308s -68d59m52.512s

### Plot distribution of magnitudes, variogram and light curve of random quasars_____no_output_____

plotStarsPicture(quasars[:3])_____no_output_____# Light curves description
quasars[0].lightCurve.meta_____no_output_____

# Find optimal parameters_____no_output_____

# Estimate all combinations and get the best one
es = ParamsEstimator(searched=quasars_train,
                     others=be_stars_train,
                     descriptors=[abbe_descr, vario_slope_descr],
                     deciders=[decider],
                     tuned_params=tun_params)

star_filter, best_stats, best_params = es.fit()All 24 combinations have been evaluated/Users/martinvo/workspace/private2/LightCurvesClassifier/lcc/stars_processing/tools/params_estim.py:200: UserWarning:
Error during saving outputs...:
	module 'types' has no attribute 'InstanceType'
  warnings.warn("\nError during saving outputs...:\n\t%s" % e)
print("Optimal parameters: {}".format(es.tuned_params[0]))
pd.DataFrame(es.stats)Optimal parameters: {'AbbeValueDescr': {'bins': 10}, 'VariogramSlopeDescr': {'days_per_bin': 30}}

### Show features for some quasars_____no_output_____

star_filter.getSpaceCoordinates(quasars[:5])_____no_output_____

# Evaluate model
Now we can use tuned filter to predict whether given stars are quasars or not. Note that we're evaluating on the test
sample._____no_output_____

prediction_quasars = star_filter.evaluateStars(quasars_test)
prediction_be_stars = star_filter.evaluateStars(be_stars_test)_____no_output_____prediction_quasars.head()_____no_output_____

### Probability plot
Purple stars are quasars, black are Be stars. Color on the background represents probability of an object
to be quasar (scaled by colorbar on the right)_____no_output_____

plotProbabSpace(star_filter)_____no_output_____

# Find a quasar candidates in OGLEII
NOTE: Sometimes OgleII doesn't work because their database, so if the query takes too long, maybe it's that case _____no_output_____

# This should return a star from OGLEII
ogle_star = OgleII({"starid": 1, "field_num": 2, "target": "lmc"}).getStars()_____no_output_____# Query for OgleII
db_name = "OgleII"
starid_from = 1
starid_to = 10
field_num_from = 1
field_num_to = 2
target = "lmc"_____no_output_____# Prepare queries and run sequential systematic search by using filter
queries = get_combinations(["starid", "field_num", "target"],
                           range(starid_from, starid_to),
                           range(field_num_from, field_num_to),
                           [target])

searcher = StarsSearcher([star_filter],
                         db_connector=db_name,
                         stat_file_path="/tmp/lcc_status.csv",
                        save_coords=True)
searcher.queryStars(queries)

passed_stars = searcher.getPassedStars()/Users/martinvo/workspace/private2/LightCurvesClassifier/lcc/stars_processing/systematic_search/stars_searcher.py:181: UserWarning: Removing existing status file /tmp/lcc_status.csv
  warnings.warn("Removing existing status file {}".format(stat_file_path))
Loading FITS files:: 100%|██████████| 9/9 [00:00<00:00, 13.70it/s]
searcher.getStatus()_____no_output_____saveStars(quasars[:5], "/tmp")_____no_output_____

## Redis searcher_____no_output_____

redis_searcher = StarsSearcherRedis([star_filter], db_connector=db_name, save_coords=True)
redis_searcher.queryStars(queries)

passed_stars = redis_searcher.getPassedStars()Remaining jobs: 5
Remaining jobs: 5
Remaining jobs: 5
Remaining jobs: 1
Remaining jobs: 1
Remaining jobs: 1
# redis_searcher.getStatus()_____no_output_____


# Notebook from Boris69bg/biosignalsnotebooks
Path: header_footer/biosignalsnotebooks_environment/categories/MainFiles/by_tag.ipynb

_____no_output_____<link rel="stylesheet" href="../../styles/theme_style.css">

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">

         Notebooks Grouped by Tag Values

_____no_output_____Each Notebook content is summarized in his header through a quantitative scale ('"Difficulty" between 1 and 5 stars) and some keywords (Group of "tags").

Grouping Notebooks by difficulty level, by signal type to which it applies or by tags is an extremelly important task, in order to ensure that the **biosignalsnotebooks** user could navigate efficiently in this learning environment.
_____no_output_____

            acquire

		 Signal Acquisition [OpenSignals]

_____no_output_____

            acquisition

		 Stone, Paper or Scissor Game - Train and Classify [Volume 1]

_____no_output_____

            anaconda

		 Download, Install and Execute Anaconda

_____no_output_____

            biosignalsplux

		 Pairing a Device at Windows 10 [biosignalsplux]

_____no_output_____

            bluetooth

		 Pairing a Device at Windows 10 [biosignalsplux]

_____no_output_____

            connect

		 Pairing a Device at Windows 10 [biosignalsplux]

_____no_output_____

            conversion

		 Generation of a time axis (conversion of samples into seconds)

		ECG Sensor - Unit Conversion

		EEG Sensor - Unit Conversion

		EMG Sensor - Unit Conversion

		fNIRS Sensor - Unit Conversion

_____no_output_____

            cross-validation

		 Stone, Paper or Scissor Game - Train and Classify [Volume 4]

_____no_output_____

            detect

		 Event Detection - Muscular Activations (EMG)

		 Event Detection - R Peaks (ECG)

_____no_output_____

            download

		 Download, Install and Execute Anaconda

		 Download, Install and Execute Jypyter Notebook Environment

_____no_output_____

            eeg

		EEG Sensor - Unit Conversion

_____no_output_____

            evaluate

		 Stone, Paper or Scissor Game - Train and Classify [Volume 4]

_____no_output_____

            extract

		 EMG Analysis - Time and Frequency Parameters

_____no_output_____

            extraction

		 Stone, Paper or Scissor Game - Train and Classify [Volume 2]

_____no_output_____

            fNIRS

		fNIRS Sensor - Unit Conversion

_____no_output_____

            features

		 Stone, Paper or Scissor Game - Train and Classify [Orange]

		 Stone, Paper or Scissor Game - Train and Classify [Volume 2]

		 Stone, Paper or Scissor Game - Train and Classify [Volume 3]

		 Stone, Paper or Scissor Game - Train and Classify [Volume 4]

_____no_output_____

            filter

		 Digital Filtering - A Fundamental Pre-Processing Step

_____no_output_____

            h5

		Load acquired data from .h5 file

_____no_output_____

            header

		Signal Loading - Working with File Header

_____no_output_____

            install

		 Download, Install and Execute Anaconda

		 Download, Install and Execute Jypyter Notebook Environment

_____no_output_____

            jupyter

		 Download, Install and Execute Anaconda

		 Download, Install and Execute Jypyter Notebook Environment

_____no_output_____

            load

		Load acquired data from .h5 file

		 Load Signals after Acquisition [OpenSignals]

		Load acquired data from .txt file

		Signal Loading - Working with File Header

_____no_output_____

            machine-learning

		 Stone, Paper or Scissor Game - Train and Classify [Orange]

		 Stone, Paper or Scissor Game - Train and Classify [Volume 1]

		 Stone, Paper or Scissor Game - Train and Classify [Volume 2]

		 Stone, Paper or Scissor Game - Train and Classify [Volume 3]

		 Stone, Paper or Scissor Game - Train and Classify [Volume 4]

_____no_output_____

            metadata

		Signal Loading - Working with File Header

_____no_output_____

            muscular-activations

		 EMG Analysis - Time and Frequency Parameters

_____no_output_____

            nearest-neighbour

		 Stone, Paper or Scissor Game - Train and Classify [Orange]

		 Stone, Paper or Scissor Game - Train and Classify [Volume 3]

_____no_output_____

            notebook

		 Download, Install and Execute Anaconda

		 Download, Install and Execute Jypyter Notebook Environment

_____no_output_____

            open

		Load acquired data from .h5 file

		 Load Signals after Acquisition [OpenSignals]

		Load acquired data from .txt file

_____no_output_____

            opensignals

		 Signal Acquisition [OpenSignals]

		 Store Files after Acquisition [OpenSignals]

		 Load Signals after Acquisition [OpenSignals]

_____no_output_____

            orange

		 Stone, Paper or Scissor Game - Train and Classify [Orange]

_____no_output_____

            pairing

		 Pairing a Device at Windows 10 [biosignalsplux]

_____no_output_____

            post-acquisition

		 Store Files after Acquisition [OpenSignals]

		 Load Signals after Acquisition [OpenSignals]

_____no_output_____

            pre-process

		 Digital Filtering - A Fundamental Pre-Processing Step

		Fatigue Evaluation - Evolution of Median Power Frequency

		 Generation of a time axis (conversion of samples into seconds)

		ECG Sensor - Unit Conversion

		EEG Sensor - Unit Conversion

		EMG Sensor - Unit Conversion

		fNIRS Sensor - Unit Conversion

_____no_output_____

            problems

		Problems of low sampling rate (aliasing)

_____no_output_____

            quality

		 Stone, Paper or Scissor Game - Train and Classify [Volume 4]

_____no_output_____

            r-peaks

		 Event Detection - R Peaks (ECG)

_____no_output_____

            real-time

		 Signal Acquisition [OpenSignals]

_____no_output_____

            record

		 Signal Acquisition [OpenSignals]

		 Resolution - The difference between smooth and abrupt variations

		Problems of low sampling rate (aliasing)

		 Store Files after Acquisition [OpenSignals]

_____no_output_____

            resolution

		 Resolution - The difference between smooth and abrupt variations

_____no_output_____

            sampling rate

		Problems of low sampling rate (aliasing)

_____no_output_____

            store

		 Store Files after Acquisition [OpenSignals]

_____no_output_____

            time

		 Generation of a time axis (conversion of samples into seconds)

_____no_output_____

            tkeo

		 Event Detection - Muscular Activations (EMG)

_____no_output_____

            train

		 Stone, Paper or Scissor Game - Train and Classify [Orange]

		 Stone, Paper or Scissor Game - Train and Classify [Volume 3]

_____no_output_____

            train_and_classify

		 Stone, Paper or Scissor Game - Train and Classify [Orange]

		 Stone, Paper or Scissor Game - Train and Classify [Volume 1]

		 Stone, Paper or Scissor Game - Train and Classify [Volume 2]

		 Stone, Paper or Scissor Game - Train and Classify [Volume 3]

_____no_output_____

            txt

		Load acquired data from .txt file

_____no_output_____

            windows10

		 Pairing a Device at Windows 10 [biosignalsplux]

_____no_output_____

            ☌ Project Presentation

            ☌ GitHub Repository

            ☌ How to install biosignalsnotebooks Python package ?

            ☌ Signal Library

            ☌ Notebook Categories

            ☌ Notebooks by Difficulty

            ☌ Notebooks by Signal Type

            ☌ Notebooks by Tag

_____no_output_____**Auxiliary Code Segment (should not be replicated by
the user)**_____no_output_____

from biosignalsnotebooks.__notebook_support__ import css_style_apply
css_style_apply()_____no_output_____%%html
_____no_output_____