# Notebook from janeite/course-content
Path: tutorials/W1D3_ModelFitting/student/W1D3_Tutorial4.ipynb

_____no_output_____# Tutorial 4: Multiple linear regression and polynomial regression
**Week 1, Day 3: Model Fitting**

**By Neuromatch Academy**

**Content creators**: Pierre-Étienne Fiquet, Anqi Wu, Alex Hyafil with help from Byron Galbraith, Ella Batty

**Content reviewers**: Lina Teichmann, Saeed Salehi, Patrick Mineault, Michael Waskom_____no_output_____**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

_____no_output_____---
# Tutorial Objectives

*Estimated timing of tutorial: 35 minutes*

This is Tutorial 4 of a series on fitting models to data. We start with simple linear regression, using least squares optimization (Tutorial 1) and Maximum Likelihood Estimation (Tutorial 2). We will use bootstrapping to build confidence intervals around the inferred linear model parameters (Tutorial 3). We'll finish our exploration of regression models by generalizing to multiple linear regression and polynomial regression (Tutorial 4). We end by learning how to choose between these various models. We discuss the bias-variance trade-off (Tutorial 5) and Cross Validation for model selection (Tutorial 6).

In this tutorial, we will generalize the regression model to incorporate multiple features.
- Learn how to structure inputs for regression using the 'Design Matrix'
- Generalize the MSE for multiple features using the ordinary least squares estimator
- Visualize data and model fit in multiple dimensions
- Fit polynomial regression models of different complexity
- Plot and evaluate the polynomial regression fits
_____no_output_____

# @title Tutorial slides

# @markdown These are the slides for the videos in all tutorials today
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/2mkq4/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)_____no_output_____# @title Video 1: Multiple Linear Regression and Polynomial Regression
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV11Z4y1u7cf", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="d4nfTki6Ejc", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)_____no_output_____

---
# Setup_____no_output_____

# Imports

import numpy as np
import matplotlib.pyplot as plt_____no_output_____#@title Figure Settings
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")_____no_output_____# @title Plotting Functions

def evaluate_fits(order_list, mse_list):
  """ Compare the quality of multiple polynomial fits
  by plotting their MSE values.

  Args:
    order_list (list): list of the order of polynomials to be compared
    mse_list (list): list of the MSE values for the corresponding polynomial fit
  """
  fig, ax = plt.subplots()
  ax.bar(order_list, mse_list)
  ax.set(title='Comparing Polynomial Fits', xlabel='Polynomial order', ylabel='MSE')

def plot_fitted_polynomials(x, y, theta_hat):
  """ Plot polynomials of different orders

  Args:
    x (ndarray): input vector of shape (n_samples)
    y (ndarray): vector of measurements of shape (n_samples)
    theta_hat (dict): polynomial regression weights for different orders
  """

  x_grid = np.linspace(x.min() - .5, x.max() + .5)

  plt.figure()

  for order in range(0, max_order + 1):
    X_design = make_design_matrix(x_grid, order)
    plt.plot(x_grid, X_design @ theta_hat[order]);

  plt.ylabel('y')
  plt.xlabel('x')
  plt.plot(x, y, 'C0.');
  plt.legend([f'order {o}' for o in range(max_order + 1)], loc=1)
  plt.title('polynomial fits')
  plt.show()_____no_output_____

---
# Section 1: Multiple Linear Regression

*Estimated timing to here from start of tutorial: 8 min*

This video covers linear regression with multiple inputs (more than 1D) and polynomial regression.

 Click here for text recap of video

Now that we have considered the univariate case and how to produce confidence intervals for our estimator, we turn to the general linear regression case, where we can have more than one regressor, or feature, in our input.

Recall that our original univariate linear model was given as

\begin{align}
y = \theta x + \epsilon
\end{align}

where $\theta$ is the slope and $\epsilon$ some noise. We can easily extend this to the multivariate scenario by adding another parameter for each additional feature

\begin{align}
y = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + ... +\theta_d x_d + \epsilon
\end{align}

where $\theta_0$ is the intercept and $d$ is the number of features (it is also the dimensionality of our input).

We can condense this succinctly using vector notation for a single data point

\begin{align}
y_i = \boldsymbol{\theta}^{\top}\mathbf{x}_i + \epsilon
\end{align}

and fully in matrix form

\begin{align}
\mathbf{y} = \mathbf{X}\boldsymbol{\theta} + \mathbf{\epsilon}
\end{align}

where $\mathbf{y}$ is a vector of measurements, $\mathbf{X}$ is a matrix containing the feature values (columns) for each input sample (rows), and $\boldsymbol{\theta}$ is our parameter vector.

This matrix $\mathbf{X}$ is often referred to as the "[design matrix](https://en.wikipedia.org/wiki/Design_matrix)".

We want to find an optimal vector of paramters $\boldsymbol{\hat\theta}$. Recall our analytic solution to minimizing MSE for a single regressor:

\begin{align}
\hat\theta = \frac{\sum_{i=1}^N x_i y_i}{\sum_{i=1}^N x_i^2}.
\end{align}

The same holds true for the multiple regressor case, only now expressed in matrix form

\begin{align}
\boldsymbol{\hat\theta} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}.
\end{align}

This is called the [ordinary least squares](https://en.wikipedia.org/wiki/Ordinary_least_squares) (OLS) estimator._____no_output_____For this tutorial we will focus on the two-dimensional case ($d=2$), which allows us to easily visualize our results. As an example, think of a situation where a scientist records the spiking response of a retinal ganglion cell to patterns of light signals that vary in contrast and in orientation. Then contrast and orientation values can be used as features / regressors to predict the cells response.

In this case our model can be writen for a single data point as:

\begin{align}
y = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \epsilon
\end{align}

or for multiple data points in matrix form where

\begin{align}
\mathbf{X} =
\begin{bmatrix}
1 & x_{1,1} & x_{1,2} \\
1 & x_{2,1} & x_{2,2} \\
\vdots & \vdots & \vdots \\
1 & x_{n,1} & x_{n,2}
\end{bmatrix},
\boldsymbol{\theta} =
\begin{bmatrix}
\theta_0 \\
\theta_1 \\
\theta_2 \\
\end{bmatrix}
\end{align}

When we refer to $x_{i, j}$, we mean that it is the i-th data point and the j-th feature of that data point.

For our actual exploration dataset we shall set $\boldsymbol{\theta}=[0, -2, -3]$ and draw $N=40$ noisy samples from $x \in [-2,2)$. Note that setting the value of $\theta_0 = 0$ effectively ignores the offset term._____no_output_____

# @markdown Execute this cell to simulate some data

# Set random seed for reproducibility
np.random.seed(1234)

# Set parameters
theta = [0, -2, -3]
n_samples = 40

# Draw x and calculate y
n_regressors = len(theta)
x0 = np.ones((n_samples, 1))
x1 = np.random.uniform(-2, 2, (n_samples, 1))
x2 = np.random.uniform(-2, 2, (n_samples, 1))
X = np.hstack((x0, x1, x2))
noise = np.random.randn(n_samples)
y = X @ theta + noise

ax = plt.subplot(projection='3d')
ax.plot(X[:,1], X[:,2], y, '.')

ax.set(
    xlabel='$x_1$',
    ylabel='$x_2$',
    zlabel='y'
)
plt.tight_layout()_____no_output_____

## Coding Exercise 1: Ordinary Least Squares Estimator

In this exercise you will implement the OLS approach to estimating $\boldsymbol{\hat\theta}$ from the design matrix $\mathbf{X}$ and measurement vector $\mathbf{y}$. You can use the `@` symbol for matrix multiplication, `.T` for transpose, and `np.linalg.inv` for matrix inversion.
_____no_output_____

def ordinary_least_squares(X, y):
  """Ordinary least squares estimator for linear regression.

  Args:
    x (ndarray): design matrix of shape (n_samples, n_regressors)
    y (ndarray): vector of measurements of shape (n_samples)

  Returns:
    ndarray: estimated parameter values of shape (n_regressors)
  """
  ######################################################################
  ## TODO for students: solve for the optimal parameter vector using OLS
  # Fill out function and remove
  raise NotImplementedError("Student exercise: solve for theta_hat vector using OLS")
  ######################################################################

  # Compute theta_hat using OLS
  theta_hat = ...

  return theta_hat

theta_hat = ordinary_least_squares(X, y)
print(theta_hat)_____no_output_____

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_25849be9.py)

_____no_output_____After filling in this function, you should see that $\boldsymbol{\hat\theta}$ = [ 0.13861386, -2.09395731, -3.16370742]_____no_output_____Now that we have our $\boldsymbol{\hat\theta}$, we can obtain $\hat{\mathbf{y}}$ and thus our mean squared error._____no_output_____

# Compute predicted data
theta_hat = ordinary_least_squares(X, y)
y_hat = X @ theta_hat

# Compute MSE
print(f"MSE = {np.mean((y - y_hat)**2):.2f}")_____no_output_____

Finally, the following code will plot a geometric visualization of the data points (blue) and fitted plane. _____no_output_____

# @markdown Execute this cell to visualize data and predicted plane

theta_hat = ordinary_least_squares(X, y)
xx, yy = np.mgrid[-2:2:50j, -2:2:50j]
y_hat_grid = np.array([xx.flatten(), yy.flatten()]).T @ theta_hat[1:]
y_hat_grid = y_hat_grid.reshape((50, 50))

ax = plt.subplot(projection='3d')
ax.plot(X[:, 1], X[:, 2], y, '.')
ax.plot_surface(xx, yy, y_hat_grid, linewidth=0, alpha=0.5, color='C1',
                cmap=plt.get_cmap('coolwarm'))

for i in range(len(X)):
  ax.plot((X[i, 1], X[i, 1]),
          (X[i, 2], X[i, 2]),
          (y[i], y_hat[i]),
          'g-', alpha=.5)

ax.set(
    xlabel='$x_1$',
    ylabel='$x_2$',
    zlabel='y'
)
plt.tight_layout()_____no_output_____

---
# Section 2:  Polynomial Regression

So far today, you learned how to predict outputs from inputs by fitting a linear regression model. We can now model all sort of relationships, including in neuroscience!

One potential problem with this approach is the simplicity of the model. Linear regression, as the name implies, can only capture a linear relationship between the inputs and outputs. Put another way, the predicted outputs are only a weighted sum of the inputs. What if there are more complicated computations happening? Luckily, many more complex models exist (and you will encounter many more over the next 3 weeks). One model that is still very simple to fit and understand, but captures more complex relationships, is **polynomial regression**, an extension of linear regression.

 Click here for text recap of relevant part of video

Since polynomial regression is an extension of linear regression, everything you learned so far will come in handy now! The goal is the same: we want to predict the dependent variable $y$ given the input values $x$. The key change is the type of relationship between inputs and outputs that the model can capture.

Linear regression models predict the outputs as a weighted sum of the inputs:

\begin{align}
y = \theta_0 + \theta x + \epsilon
\end{align}

With polynomial regression, we model the outputs as a polynomial equation based on the inputs. For example, we can model the outputs as:

\begin{align}
y & = \theta_0 + \theta_1 x + \theta_2 x^2 + \theta_3 x^3 + \epsilon
\end{align}

We can change how complex a polynomial is fit by changing the order of the polynomial. The order of a polynomial refers to the highest power in the polynomial. The equation above is a third order polynomial because the highest value x is raised to is 3. We could add another term ($+ \theta_4 x^4$) to model an order 4 polynomial and so on.

_____no_output_____First, we will simulate some data to practice fitting polynomial regression models. We will generate random inputs $x$ and then compute y according to $y = x^2 - x - 2 $, with some extra noise both in the input and the output to make the model fitting exercise closer to a real life situation._____no_output_____

# @markdown Execute this cell to simulate some data

# setting a fixed seed to our random number generator ensures we will always
# get the same psuedorandom number sequence
np.random.seed(121)
n_samples = 30
x = np.random.uniform(-2, 2.5, n_samples)  # inputs uniformly sampled from [-2, 2.5)
y =  x**2 - x - 2   # computing the outputs

output_noise = 1/8 * np.random.randn(n_samples)
y += output_noise  # adding some output noise

input_noise = 1/2 * np.random.randn(n_samples)
x += input_noise  # adding some input noise

fig, ax = plt.subplots()
ax.scatter(x, y)  # produces a scatter plot
ax.set(xlabel='x', ylabel='y');_____no_output_____

## Section 2.1: Design matrix for polynomial regression

*Estimated timing to here from start of tutorial: 16 min*

Now we have the basic idea of polynomial regression and some noisy data, let's begin! The key difference between fitting a linear regression model and a polynomial regression model lies in how we structure the input variables.

Let's go back to one feature for each data point. For linear regression, we used $\mathbf{X} = \mathbf{x}$ as the input data, where $\mathbf{x}$ is a vector where each element is the input for a single data point. To add a constant bias (a y-intercept in a 2-D plot), we use $\mathbf{X} = \big[ \boldsymbol 1, \mathbf{x} \big]$, where $\boldsymbol 1$ is a column of ones.  When fitting, we learn a weight for each column of this matrix. So we learn a weight that multiples with column 1 - in this case that column is all ones so we gain the bias parameter ($+ \theta_0$).

This matrix $\mathbf{X}$ that we use for our inputs is known as a **design matrix**. We want to create our design matrix so we learn weights for $\mathbf{x}^2, \mathbf{x}^3,$ etc. Thus, we want to build our design matrix $X$ for polynomial regression of order $k$ as:

\begin{align}
\mathbf{X} = \big[ \boldsymbol 1 , \mathbf{x}^1, \mathbf{x}^2 , \ldots , \mathbf{x}^k \big],
\end{align}

where $\boldsymbol{1}$ is the vector the same length as $\mathbf{x}$ consisting of of all ones, and $\mathbf{x}^p$ is the vector $\mathbf{x}$ with all elements raised to the power $p$. Note that $\boldsymbol{1} = \mathbf{x}^0$ and $\mathbf{x}^1 = \mathbf{x}$.

If we have inputs with more than one feature, we can use a similar design matrix but include all features raised to each power. Imagine that we have two features per data point: $\mathbf{x}_m$ is a vector of one feature per data point and  $\mathbf{x}_n$ is another.  Our design matrix for a polynomial regression would be:
\begin{align}
\mathbf{X} = \big[ \boldsymbol 1 , \mathbf{x}_m^1, \mathbf{x}_n^1, \mathbf{x}_m^2 , \mathbf{x}_n^2\ldots , \mathbf{x}_m^k , \mathbf{x}_n^k \big],
\end{align}_____no_output_____### Coding Exercise 2.1: Structure design matrix

Create a function (`make_design_matrix`) that structures the design matrix given the input data and the order of the polynomial you wish to fit. We will print part of this design matrix for our data and order 5._____no_output_____

def make_design_matrix(x, order):
  """Create the design matrix of inputs for use in polynomial regression

  Args:
    x (ndarray): input vector of shape (n_samples)
    order (scalar): polynomial regression order

  Returns:
    ndarray: design matrix for polynomial regression of shape (samples, order+1)
  """
  ########################################################################
  ## TODO for students: create the design matrix ##
  # Fill out function and remove
  raise NotImplementedError("Student exercise: create the design matrix")
  ########################################################################

  # Broadcast to shape (n x 1) so dimensions work
  if x.ndim == 1:
    x = x[:, None]

  #if x has more than one feature, we don't want multiple columns of ones so we assign
  # x^0 here
  design_matrix = np.ones((x.shape[0], 1))

  # Loop through rest of degrees and stack columns (hint: np.hstack)
  for degree in range(1, order + 1):
      design_matrix = ...

  return design_matrix

order = 5
X_design = make_design_matrix(x, order)

print(X_design[0:2, 0:2])_____no_output_____

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_5e30078a.py)

_____no_output_____You should see that the printed section of this design matrix is `[[ 1.         -1.51194917]
 [ 1.         -0.35259945]]`_____no_output_____## Section 2.2: Fitting polynomial regression models

*Estimated timing to here from start of tutorial: 24 min*

Now that we have the inputs structured correctly in our design matrix, fitting a polynomial regression is the same as fitting a linear regression model! All of the polynomial structure we need to learn is contained in how the inputs are structured in the design matrix. We can use the same least squares solution we computed in previous exercises. _____no_output_____### Coding Exercise 2.2: Fitting polynomial regression models with different orders _____no_output_____Here, we will fit polynomial regression models to find the regression coefficients ($\theta_0, \theta_1, \theta_2,$ ...) by solving the least squares problem. Create a function `solve_poly_reg` that loops over different order polynomials (up to `max_order`), fits that model, and saves out the weights for each. You may invoke the `ordinary_least_squares` function.

We will then qualitatively inspect the quality of our fits for each order by plotting the fitted polynomials on top of the data. In order to see smooth curves, we evaluate the fitted polynomials on a grid of $x$ values (ranging between the largest and smallest of the inputs present in the dataset)._____no_output_____

def solve_poly_reg(x, y, max_order):
  """Fit a polynomial regression model for each order 0 through max_order.

  Args:
    x (ndarray): input vector of shape (n_samples)
    y (ndarray): vector of measurements of shape (n_samples)
    max_order (scalar): max order for polynomial fits

  Returns:
    dict: fitted weights for each polynomial model (dict key is order)
  """

  # Create a dictionary with polynomial order as keys,
  # and np array of theta_hat (weights) as the values
  theta_hats = {}

  # Loop over polynomial orders from 0 through max_order
  for order in range(max_order + 1):

    ##################################################################################
    ## TODO for students: Create design matrix and fit polynomial model for this order
    # Fill out function and remove
    raise NotImplementedError("Student exercise: fit a polynomial model")
    ##################################################################################

    # Create design matrix
    X_design = ...

    # Fit polynomial model
    this_theta = ...

    theta_hats[order] = this_theta

  return theta_hats

max_order = 5
theta_hats = solve_poly_reg(x, y, max_order)

# Visualize
plot_fitted_polynomials(x, y, theta_hats)_____no_output_____

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_f5217dbd.py)

*Example output:*

_____no_output_____## Section 2.3: Evaluating fit quality

*Estimated timing to here from start of tutorial: 29 min*

As with linear regression, we can compute mean squared error (MSE) to get a sense of how well the model fits the data.

We compute MSE as:

\begin{align}
\mathrm{MSE} = \frac 1 N ||\mathbf{y} - \hat{\mathbf{y}}||^2 = \frac 1 N \sum_{i=1}^N (y_i - \hat y_i)^2
\end{align}

where the predicted values for each model are given by $ \hat{\mathbf{y}} = \mathbf{X}\boldsymbol{\hat\theta}$.

*Which model (i.e. which polynomial order) do you think will have the best MSE?*_____no_output_____### Coding Exercise 2.3: Compute MSE and compare models

We will compare the MSE for different polynomial orders with a bar plot._____no_output_____

mse_list = []
order_list = list(range(max_order + 1))

for order in order_list:

  X_design = make_design_matrix(x, order)

  ########################################################################
  ## TODO for students
  # Fill out function and remove
  raise NotImplementedError("Student exercise: compute MSE")
  ########################################################################

  # Get prediction for the polynomial regression model of this order
  y_hat = ...

  # Compute the residuals
  residuals = ...

  # Compute the MSE
  mse = ...

  mse_list.append(mse)

# Visualize MSE of fits
evaluate_fits(order_list, mse_list)_____no_output_____

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_89324713.py)

*Example output:*

_____no_output_____---
# Summary

*Estimated timing of tutorial: 35 minutes*

* Linear regression generalizes naturally to multiple dimensions
* Linear algebra affords us the mathematical tools to reason and solve such problems beyond the two dimensional case

* To change from a linear regression model to a polynomial regression model, we only have to change how the input data is structured

* We can choose the complexity of the model by changing the order of the polynomial model fit

* Higher order polynomial models tend to have lower MSE on the data they're fit with

**Note**: In practice, multidimensional least squares problems can be solved very efficiently (thanks to numerical routines such as LAPACK).

_____no_output_____---
# Notation

\begin{align}
x &\quad \text{input, independent variable}\\
y &\quad \text{response measurement, dependent variable}\\
\epsilon &\quad \text{measurement error, noise contribution}\\
\theta &\quad \text{slope parameter}\\
\hat{\theta} &\quad \text{estimated slope parameter}\\
\mathbf{x} &\quad \text{vector of inputs where each element is a different data point}\\
\mathbf{X} &\quad \text{design matrix}\\
\mathbf{y} &\quad \text{vector of measurements}\\
\mathbf{\hat y} &\quad \text{vector of estimated measurements}\\
\boldsymbol{\theta} &\quad \text{vector of parameters}\\
\boldsymbol{\hat\theta} &\quad \text{vector of estimated parameters}\\
d &\quad \text{dimensionality of input}\\
N &\quad \text{number of samples}\\
\end{align}_____no_output_____**Suggested readings**

[Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares](http://vmls-book.stanford.edu/)
Stephen Boyd and Lieven Vandenberghe_____no_output_____


# Notebook from esahin90/covid-19-online-machine-learning
Path: work/covid19-online-machine-learning.ipynb

# Online model definition, data preparation and example_____no_output_____## Table of Content
1. [Model definition](#model-def)
    * [Linear Regression](#linear-regression)
    * [Passive-aggressive Regressor](#pa-regressor)
2. [Upload model to chantilly](#chantilly-rest)
3. [Download GoogleCloudPlatform/covid-19-open-data](#download-dataset)
4. [Simulate data stream and view live dashboard for model evaluation](#simulate)
    * [Open Chantilly dashboard](#dashboard)
5. [Data preparation](#data-prep)
    * [Filter by state Bremen](#bremen-data)
    * [Merge vaccination data](#vaccination-data)
6. [Example predict, learn and measure metric including vaccination data](#example-vaccination)
    * [SNARIMAX model definition](#snarimax)
    * [Pick one feature x and label y](#pick-one)
    * [Define metrics for evaluation of the mode](#metric-eval)
    * [Walkthrough prediction, metric update & training](#walkthrough)
    * [Full cycle of predict, metric update & train on vaccination data](#full-loop)
7. [Playground](#playground)_____no_output_____## 1. Model definition _____no_output_____Creme is used for defining our online machine learning models, which can be trained and predict by one sample at a time

More information about creme can be found here: https://github.com/MaxHalford/creme_____no_output_____##### Pipeline definition
1. To include the date inside the linear regression, we're first transforming it into a unix timestamp with compose.FuncTransformer.
2. Then compose.Select is used to specify, what our input features are to train the model on.
3. For standardizing and scaling down the input values, preprocessing.StandardScaler is used, which scales each input feature by subtracting the mean and dividing by the standard deviation to shift the distribution to have a mean of zero and a standard deviation of one.
4. As the last step the disered model is define._____no_output_____Choose one of the following models for uploading to chantilly by executing the cell or define your own model with creme._____no_output_____### Linear Regression
Linear Regression is a basic linear model, which assumes a linear relationship input variables and the output variables_____no_output_____

from creme import compose
from creme import linear_model
from creme import preprocessing

def parse(row):
    import datetime as dt
    row['date'] = dt.datetime.fromisoformat(row['date']).timestamp()
    return row

model = compose.FuncTransformer(parse) \
    | compose.Select('date', 'new_deceased', 'new_recovered',
                     'cumulative_confirmed', 'cumulative_deceased',
                     'cumulative_recovered') \
    | preprocessing.StandardScaler() \
    | linear_model.LinearRegression()
_____no_output_____

### Passive-aggressive Regressor
The passive-aggressive regressor is model, which aims to learn from large-scale data and is defined by:

* Passive: If the prediction is correct, keep the model and do not make any changes.
* Aggressive: If the prediction is incorrect, make changes to the model._____no_output_____

from creme import compose
from creme import linear_model
from creme import preprocessing

def parse(row):
    import datetime as dt
    row['date'] = dt.datetime.fromisoformat(row['date']).timestamp()
    return row

model = compose.FuncTransformer(parse) \
    | compose.Select('date', 'new_deceased', 'new_recovered',
                     'cumulative_confirmed', 'cumulative_deceased',
                     'cumulative_recovered') \
    | preprocessing.StandardScaler() \
    | linear_model.PARegressor()_____no_output_____

## 2. Upload model to chantilly _____no_output_____The model is uploaded via a REST-API to chantilly, which then can be used our trained through REST-Requests. To serialize the payload for the REST-Request, dill is used to pickle the model.

More information about chantilly can be found here: https://github.com/online-ml/chantilly_____no_output_____

import dill
import requests

requests.post('http://localhost:5000/api/model', data=dill.dumps(model))_____no_output_____

## 3. Download GoogleCloudPlatform/covid-19-open-data _____no_output_____In this tutorial the COVID-19 Open-Data dataset is used. It attemps to assemble the largest Covid-19 epidemiological database.
More information about the dataset can be found here: https://github.com/GoogleCloudPlatform/covid-19-open-data_____no_output_____First we need to download the base epidemiology.csv, which lists the numbers for new cases, tests, recovered & deceased.
The schema definition can be found here: [Epidemiology Schema](https://github.com/GoogleCloudPlatform/covid-19-open-data/blob/main/docs/table-epidemiology.md)_____no_output_____

!wget https://storage.googleapis.com/covid19-open-data/v3/epidemiology.csv_____no_output_____

For the example inside this notebook we're also using the vaccination dataset, which lists the numbers for new vaccinations, etc..The schema definition can be found here: [Vaccinations Schema](https://github.com/GoogleCloudPlatform/covid-19-open-data/blob/main/docs/table-vaccinations.md)_____no_output_____

!wget https://storage.googleapis.com/covid19-open-data/v3/vaccinations.csv_____no_output_____

## 4. Simulate data stream and view live dashboard for model evaluation _____no_output_____For simulating a data stream and evaluating different models the python script `simulate.py` was developed.

```
# python simulate.py --help
usage: simulate.py [-h] [--file [FILE]] [speed_up]

positional arguments:
  speed_up

optional arguments:
  -h, --help     show this help message and exit
  --file [FILE]
```

The script will use `epidemiology.csv` as a default filepath to simulate a data stream for our specific case. You can also specify a custom filepath by using --file like `python --file bremen_epidemiology.csv` for example, if you continue with the notebook and create the dataset for the state Bremen in Germany.
The sleep timer is currently set to 1 second between each, which can be speeded up with a positional argument, when starting the script._____no_output_____There are two ways to use this script. Either run it in the cell below, which will spam the notebook. Or start a new terminal in jupyterlab and execute the python script in the terminal_____no_output_____

!python simulate.py_____no_output_____

### Open Chantilly dashboard _____no_output_____To evaluate the performance of the model, which was uploaded to chantilly, open [Chantilly Dashboard](http://localhost:5000)._____no_output_____## 5. Data preparation _____no_output_____In this sections we're trying to parse and prepare the data to only contain the state Bremen in Germany and clean up columns and rows, which contain NaN-values. For this we're using pandas, which is widely used in the data science community and is also open source.

More information can be found here: https://pandas.pydata.org_____no_output_____

import pandas as pd_____no_output_____data = pd.read_csv('epidemiology.csv')_____no_output_____

#### Filter by state Bremen (DE_HB)

The location key of the specific region is built by using a combination of codes. See [GoogleCloudPlatform/covid-19-open-data](https://github.com/GoogleCloudPlatform/covid-19-open-data) for more information_____no_output_____

bremen_data = data[data.location_key.apply(lambda x: "DE_HB" == x if isinstance(x, str) else False)]
# Drop na columns with no data
bremen_data = bremen_data.drop(columns=["new_tested", "cumulative_tested"])_____no_output_____

#### Quick look inside our dataframe_____no_output_____To understand how our dataframe looks like, a quick and easy way is to use head(), which prints the first five rows inside our dataframe._____no_output_____

bremen_data.head()_____no_output_____

An easy way to have a quick overview of the value in pandas, which we're trying to predict, is to use describe() on the Series, which outputs basic statiscal values_____no_output_____

bremen_data.new_confirmed.describe()_____no_output_____

To export the dataframe in a csv-file, to_csv(path) can be used, which will create a new csv-file with the data from the dataframe._____no_output_____

bremen_data.to_csv("bremen_epidemiology.csv")_____no_output_____

### Merge vaccination data

Here we're trying to parse the vaccination data and merge it with the epidemiology dataframe._____no_output_____

vacc_data = pd.read_csv('vaccinations.csv')_____no_output_____

To merge the dataframes we need to select the unqiue identifiers on both dataframes, which are the date and the location_key. Pandas provides an merge-function, which only uses the both dataframes as input parameters and a list of unique-identifiers, which are used to merge the dataframes._____no_output_____

merged_data = pd.merge(data, vacc_data, on=['date', 'location_key'])_____no_output_____

Filter again for the state Bremen in Germany (DE_HB)_____no_output_____

bremen_vacc_data = merged_data[merged_data.location_key.apply(lambda x: "DE_HB" == x if isinstance(x, str) else False)]_____no_output_____

Select specific columns, which we want to include in our dataframe_____no_output_____

bremen_vacc_data = bremen_vacc_data[['date', 'location_key', 'new_confirmed', 'new_deceased',
       'new_recovered', 'cumulative_confirmed', 'cumulative_deceased',
       'cumulative_recovered', 'new_persons_vaccinated',
       'cumulative_persons_vaccinated', 'new_persons_fully_vaccinated',
       'cumulative_persons_fully_vaccinated', 'new_vaccine_doses_administered',
       'cumulative_vaccine_doses_administered']]
# Drop na rows
bremen_vacc_data = bremen_vacc_data.dropna()_____no_output_____

Quick look inside the dataframe_____no_output_____

bremen_vacc_data.head()_____no_output_____

Describe can also be used on a dataframe, which will generate all the basic statistical values for each column_____no_output_____

bremen_vacc_data.describe()_____no_output_____

## 6. Example predict, learn and measure metric including vaccination data _____no_output_____Since the dataset including vaccinations for Bremen seems to be pretty small currently, we wanted to demonstrate, how the basic workflow with creme looks like. One of the models called SNARIMAX, which we wanted to test on chantilly, also had some problems with the compatibility. Therefore we decided to only use it also in this example._____no_output_____### SNARIMAX model definition _____no_output_____SNARIMAX stands for (S)easonal (N)on-linear (A)uto(R)egressive (I)ntegrated (M)oving-(A)verage with e(X)ogenous inputs model.

This model generalizes many established time series models in a single interface that can be trained online. It assumes that the provided training data is ordered in time and is uniformly spaced.

Documentation to the model can be found here: https://riverml.xyz/dev/api/time-series/SNARIMAX/_____no_output_____

from creme import compose
from creme import linear_model
from creme import preprocessing
from creme import time_series

def parse(row):
    import datetime as dt
    row['date'] = dt.datetime.fromisoformat(row['date']).toordinal()
    return row

model = compose.FuncTransformer(parse) \
    | compose.Select('date', 'new_deceased', 'new_recovered', 'cumulative_confirmed',
                     'cumulative_deceased', 'cumulative_recovered', 'new_persons_vaccinated',
                     'cumulative_persons_vaccinated', 'new_persons_fully_vaccinated',
                     'cumulative_persons_fully_vaccinated', 'new_vaccine_doses_administered',
                     'cumulative_vaccine_doses_administered') \
    | time_series.SNARIMAX(p=1, d=1, q=14)_____no_output_____

### Pick one feature x and label y _____no_output_____

test_dict = bremen_vacc_data.to_dict(orient="records")[-1]
y = test_dict.pop("new_confirmed")_____no_output_____test_dict_____no_output_____y_____no_output_____

### Define metrics for evaluation of the model _____no_output_____SMAPE stands (S)ymmetric (M)ean (A)bsolute (P)ercentage (E)rror and is used to evaluate the absolute percentage error in respect to the actual ground truth. This metric gives a good representation, how the model is performing and learning over time._____no_output_____

from creme import metrics
smape = metrics.SMAPE()_____no_output_____

### Walkthrough prediction, metric update & training _____no_output_____Predict value_____no_output_____

y_pred = model.forecast(horizon=1, xs=[test_dict.copy()])[0]_____no_output_____

Update metric_____no_output_____

smape.update(y_true=y, y_pred=y_pred)_____no_output_____

Train model on actual value_____no_output_____

model.fit_one(x=test_dict.copy(), y=y)_____no_output_____

Repeat prediction_____no_output_____

model.forecast(horizon=1, xs=[test_dict.copy()])[0]_____no_output_____

### Full cycle of predict, metric update & train on vaccination data _____no_output_____This is how a continuous loop of prediction, metric update and training on the merged vaccination dataframe could look like. A sleep timer of one second is used in the for loop to see the training process and metric improvement per cycle._____no_output_____

import time
for data_dict in bremen_vacc_data.to_dict(orient="records"):
    y = data_dict.pop("new_confirmed")
    y_pred = model.forecast(horizon=1, xs=[test_dict.copy()])[0]
    smape.update(y_true=y, y_pred=y_pred)
    model.fit_one(x=test_dict.copy(), y=y)
    print(f"Prediction: {y_pred}, Truth: {y}, SMAPE: {smape.get()}")
    time.sleep(1.0)_____no_output_____

## Playground

Try out stuff yourself!_____no_output_____


# Notebook from shigeodayo/ex_design_analysis
Path: src/statistical_test/examples/mean/parametric/smart_face.ipynb

# Parametric analysis with repeated data (ANOVA & Tukey-HSD)
Naoto Nakazato, Shigeo Yoshida, Sho Sakurai, Takuji Narumi, Tomohiro Tanikawa, and Michitaka Hirose. 2014. Smart Face: enhancing creativity during video conferences using real-time facial deformation. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing (CSCW '14). Association for Computing Machinery, New York, NY, USA, 75–83. DOI:https://doi.org/10.1145/2531602.2531637

Within-participant design (3 groups: smiley face, neutral face, sad face)

To use ANOVA, you need to make sure that the data fulfill these requirements:
- Normality
- Homogeneity of variances / Sphericity_____no_output_____

import numpy as np
import pandas as pd
import scipy.stats as stats
from statsmodels.stats.multicomp import pairwise_tukeyhsd
import matplotlib.pyplot as plt
import seaborn as sns
import os

# %matplotlib inline
plt.style.use('seaborn-darkgrid')_____no_output_____CSV_PATH = '../../../../../data/statistical_test/examples/smart_face/n_idea.csv'
NUM_OF_PARTICIPANTS = 10
ALPHA = 0.05
# OUTPUT_PATH = 'output/sf'
# if not os.path.isdir(OUTPUT_PATH):
    # os.makedirs(OUTPUT_PATH)_____no_output_____data = pd.read_csv(CSV_PATH)
print(data)_____no_output_____

## Normality check_____no_output_____

_, p = stats.shapiro(data['smiley face'])
print('p={:.5f}'.format(p))
if p > ALPHA:
    print('Normality check: passed')
else:
    print('Normality check: rejected')

_, p = stats.shapiro(data['neutral'])
print('p={:.5f}'.format(p))
if p > ALPHA:
    print('Normality check: passed')
else:
    print('Normality check: rejected')

_, p = stats.shapiro(data['sad face'])
print('p={:.5f}'.format(p))
if p > ALPHA:
    print('Normality check: passed')
else:
    print('Normality check: rejected')_____no_output_____

## Homogeneity of variances / Sphericity check_____no_output_____

_, p = stats.bartlett(data['smiley face'], data['neutral'], data['sad face'])
if p > ALPHA:
    print('Sphericity check: passed')
else:
    print('Sphericity check: rejected')_____no_output_____

## ANOVA_____no_output_____

_, p = stats.f_oneway(data['smiley face'], data['neutral'], data['sad face'])
print('ANOVA: p={:.5f}'.format(p))

if p > ALPHA:
    print('Same distributions')
    exit()
else:
    print('Different distributions. You can do a post-hoc test.')_____no_output_____

## Multiple comparisons
There are several ways to do a multiple comparision (as a post-hoc test).

I use 'Tukey-HSD' and 't-test (with Bonferroni correction)' here.

You can use whichever you like.
_____no_output_____### Tukey-HSD_____no_output_____

# https://qiita.com/TaigaU121/items/12c480f51a026ca9f333
def tukey_hsd(ind, *args):
    data_arr = np.hstack( args )

    ind_arr = np.array([])
    for x in range(len(args)):
        ind_arr = np.append(ind_arr, np.repeat(ind[x], len(args[x])))
    print(pairwise_tukeyhsd(data_arr,ind_arr))_____no_output_____tukey_hsd(['smiley face', 'neutral face', 'sad face'], data['smiley face'], data['neutral'], data['sad face'])_____no_output_____

### t-test (with Bonferroni correction)_____no_output_____

# Smiley face vs Neutral face
_, p = stats.ttest_rel(data['smiley face'], data['neutral'])
print('smile vs neutral: p={:.5f}'.format(p * 3))  # Bonferroni correction

# Neutral face vs Sad face
_, p = stats.ttest_rel(data['neutral'], data['sad face'])
print('neutral vs sad: p={:.5f}'.format(p * 3))  # Bonferroni correction

# Sad faec vs Smiley face
_, p = stats.ttest_rel(data['sad face'], data['smiley face'])
print('sad vs smile: p={:.5f}'.format(p * 3))  # Bonferroni correction_____no_output_____

## Visualization
3 different data visualizations (Bar plot, Box plot, Violin plot)_____no_output_____### Bar plot (average & standard error)
pros: easy to compare multiple data (average)

cons: less informative (average and standard error)_____no_output_____

# Mean
smile_mu = data['smiley face'].mean()
neutral_mu = data['neutral'].mean()
sad_mu = data['sad face'].mean()

# Standard deviation
smile_sd = data['smiley face'].std()
neutral_sd = data['neutral'].std()
sad_sd = data['sad face'].std()

# Standard error
smile_se = smile_sd / np.sqrt(NUM_OF_PARTICIPANTS)
neutral_se = neutral_sd / np.sqrt(NUM_OF_PARTICIPANTS)
sad_se = sad_sd / np.sqrt(NUM_OF_PARTICIPANTS)

y = np.array([smile_mu, neutral_mu, sad_mu])
e = np.array([smile_se, neutral_se, sad_se])

x = np.array(["Smiley face", 'Neutral face', 'Sad face'])
x_position = np.arange(len(x))
error_bar_set = dict(lw=1, capthik=1, capsize=10)

fig = plt.figure(figsize=(10, 6))
ax = fig.add_subplot(1, 1, 1)
ax.bar(x_position, y, yerr=e, tick_label=x, error_kw=error_bar_set, color=['salmon', 'palegreen', 'aqua'])
ax.set_xlabel('Facial expression', fontsize=14)
ax.set_ylabel('Number of ideas', fontsize=14)
ax.set_ylim(0, 18)

# plt.savefig(os.path.join(OUTPUT_PATH, 'sf_bar.pdf'))
plt.show()_____no_output_____

### Boxplot
pros:
more informative than bar plot

cons:
unable to understand the data distribution (box plot only show summary statistics)_____no_output_____

# error bar: min/max
# box: 25/50(median)/75 percentile
# circle: outlier (1.5 times bigger/smaller than box)
fig = plt.figure(figsize=(10, 6))
ax = fig.add_subplot(1, 1, 1)
ax.boxplot([data['smiley face'], data['neutral'], data['sad face']], labels=['Smiley face', 'Neutral face', 'Sad face'])
ax.set_xlabel('Facial expression', fontsize=14)
ax.set_ylabel('Number of ideas', fontsize=14)
ax.set_ylim(0, 18)

# plt.savefig(os.path.join(OUTPUT_PATH, 'sf_box.pdf'))
plt.show()_____no_output_____

### Violin plot
pros: more informative than box plot (beacuse violin plot represents data distribution)

cons:less popular (their meaning can be harder to grasp for many readers not familiar with the violin plot representation)### Violin plot_____no_output_____

# Similar to box plot, but also represents kernel density estimation (estimated distribution of data)
fig = plt.figure(figsize=(10, 6))
ax = fig.add_subplot(1, 1, 1)
sns.violinplot(data=[data['smiley face'], data['neutral'], data['sad face']], palette=['salmon', 'palegreen', 'aqua'])
ax.set_xticklabels(['Smiley face', 'Neutral face', 'Sad face'])
ax.set_xlabel('Facial expression', fontsize=14)
ax.set_ylabel('Number of ideas', fontsize=14)
ax.set_ylim(0, 18)

# plt.savefig(os.path.join(OUTPUT_PATH, 'sf_violin.pdf'))
plt.show()_____no_output_____


# Notebook from weiqi-luo/Predicting-Opening-Weekend-Box-Office-Sales
Path: code/Web_Crawler/WebCrawler4Boxoffice.ipynb

from bs4 import BeautifulSoup
from urllib.request import urlopen
import urllib.parse as urlparse
import re
import random
import webbrowser
import requests
from re import sub
from decimal import Decimal
import numpy as np
import pandas as pd_____no_output_____

### Extract name list from kaggle data_____no_output_____

df_kaggle = pd.read_csv("./kaggle_data/train.csv",sep=",")

name_of_movie = df_kaggle.original_title

lan_list =  df_kaggle.original_language
en_list = [x for x in lan_list if x!="en"]
en_indx = df_kaggle[df_kaggle.original_language=="en"]["id"]_____no_output_____

### Search the corresponding first-week box office_____no_output_____

def find_box_office(movie_id=None,box=None):
    if movie_id == None:
        return None
    else:
        base_url_movie = "https://www.boxofficemojo.com/movies/"
        param_movie = {"page":"weekend","id":movie_id}
        r_ = requests.get(base_url_movie, params=param_movie)
        url_ = r_.url
        html_ = urlopen(url_).read()
        soup_ = BeautifulSoup(html_,'lxml')
        sub_ = soup_.find_all('font',{'size':re.compile('2'), 'face':None})
        # title_ = soup_.find_all("font",{"face":re.compile("Verdana"),"size":re.compile("[4-6]")})
        # title_ = title_[0].get_text()
        for k in range(len(sub_)):
            if ('$' in sub_[k].get_text()):
                box = sub_[k].get_text()
                return Decimal(sub(r'[^\d.]', '', box))
        return box_____no_output_____def find_webID(movie_name,url_list,webID=None):
    for i in range(len(url_list)):
        if(movie_name in url_list[i].get_text() or url_list[i].get_text() in movie_name):
            webID = urlparse.parse_qs(urlparse.urlparse(url_list[i]["href"]).query)
            return webID["id"]
    return None_____no_output_____url_search = "https://www.boxofficemojo.com/search/"
# name_of_movie = ["Insight","Control Room","A Mighty Wind"]

counter = 0 # comment the counter out when establishing the data set

for i in en_indx:
    param = {'q': name_of_movie[i-1]}
    r_search = requests.get(url_search,params=param)
    url_search = r_search.url
    html_search = urlopen(url_search).read()
    soup_search = BeautifulSoup(html_search,"lxml")

    sub_url_search = soup_search.find_all("a",{"href":re.compile("/movies/\?id")})

    webID = find_webID(name_of_movie[i-1],sub_url_search)

    # par_search_id = urlparse.parse_qs(urlparse.urlparse(sub_url_search[1]["href"]).query)

    box_office = find_box_office(webID)

    print(i," <",name_of_movie[i-1],">"," the first week box office is: ",box_office)

    ####### comment this out when establishing the data set #######
    counter+=1
    if counter==100:
        break
    ###############################################################1  < Hot Tub Time Machine 2 >  the first week box office is:  5963324
2  < The Princess Diaries 2: Royal Engagement >  the first week box office is:  22956453
3  < Whiplash >  the first week box office is:  135388
6  < Pinocchio and the Emperor of the Night >  the first week box office is:  None
7  < The Possession >  the first week box office is:  None
8  < Control Room >  the first week box office is:  27125
9  < Muppet Treasure Island >  the first week box office is:  7906689
10    the first week box office is:  2112140
11  < Rocky >  the first week box office is:  None
12  < Revenge of the Nerds II: Nerds in Paradise >  the first week box office is:  7900879
13  < American Beauty >  the first week box office is:  861531
14  < Be Cool >  the first week box office is:  23450212
15  < Minority Report >  the first week box office is:  35677125
17  < Red Eye >  the first week box office is:  16167662
18  < The Invisible Woman >  the first week box office is:  31948
19  < Chalet Girl >  the first week box office is:  192
20  < Transporter 2 >  the first week box office is:  16540720
21  < Lost in Space >  the first week box office is:  20154919
22  < Friday the 13th Part III >  the first week box office is:  36690067
23  < V/H/S >  the first week box office is:  1221
24  < InSight >  the first week box office is:  9571
25  < Black Sheep >  the first week box office is:  16102
26  < Dr. Giggles >  the first week box office is:  2714815
27  < The Spanish Prisoner >  the first week box office is:  9586993
28  < What If >  the first week box office is:  133898
29  < The Transformers: The Movie >  the first week box office is:  6469244
30  < Changing Lanes >  the first week box office is:  17128062
33  < The Last Witch Hunter >  the first week box office is:  10812861
34  < ABCs of Death 2 >  the first week box office is:  4022
35  < Rush Hour 3 >  the first week box office is:  49100158
36  < Safe >  the first week box office is:  None
37  < About Adam >  the first week box office is:  33300
38  < Outbreak >  the first week box office is:  13420387
39  < Final: The Rapture >  the first week box office is:  31900
40  < Major Dundee >  the first week box office is:  3520
41  < Cocaine Cowboys >  the first week box office is:  36295
42  < Cool Hand Luke >  the first week box office is:  None
43  < The Next Best Thing >  the first week box office is:  5870387
44  < 5 Flights Up >  the first week box office is:  238491
45  < The Wedding Date >  the first week box office is:  16628170
47  < Ice Age: The Meltdown >  the first week box office is:  68033544
48  < The Hateful Eight >  the first week box office is:  4610676
49  < Wilson >  the first week box office is:  336227
50  < Star Trek: Generations >  the first week box office is:  23116394
52  < The Great Race >  the first week box office is:  None
53  < The Last Flight of Noah's Ark >  the first week box office is:  None
54  < For Keeps >  the first week box office is:  201
55  < Risen >  the first week box office is:  11801271
56  < Son in Law >  the first week box office is:  7033310
58  < Batman & Robin >  the first week box office is:  None
59  < Were The World Mine >  the first week box office is:  864
60  < All About the Benjamins >  the first week box office is:  10007291
61  < Casino Royale >  the first week box office is:  40833156
62  < Trapeze >  the first week box office is:  None
63  < Fallen >  the first week box office is:  None
64  < Friday the 13th Part VIII: Jason Takes Manhattan >  the first week box office is:  None
65  < Nebraska >  the first week box office is:  140401
66  < Bloodsport >  the first week box office is:  None
67  < Henry's Crime >  the first week box office is:  8726
69  < To Kill a Mockingbird >  the first week box office is:  5457
70  < The Fast and the Furious: Tokyo Drift >  the first week box office is:  23973840
71  < The Jacket >  the first week box office is:  2723682
73  < Cry-Baby >  the first week box office is:  3004905
75  < Jackson County Jail >  the first week box office is:  None
76  < Fright Night >  the first week box office is:  7714388
78  < Caddyshack >  the first week box office is:  4436330
79  < Sharky's Machine >  the first week box office is:  3022041
80  < 31 >  the first week box office is:  11842
81  < Dark Blue >  the first week box office is:  3880688
82  < Shopgirl >  the first week box office is:  229685
83  < The Skeleton Key >  the first week box office is:  16057945
84  < 2010 >  the first week box office is:  38594
85  < Silent House >  the first week box office is:  6661234
86  < Congo >  the first week box office is:  3690
87  < Kramer vs. Kramer >  the first week box office is:  5531000
89  < The Zookeeper's Wife >  the first week box office is:  3288835
90  < Sommersby >  the first week box office is:  8104624
91  < Kill Bill: Vol. 1 >  the first week box office is:  22089322
93  < Le Hussard sur le toit >  the first week box office is:  None
94  < Hunger >  the first week box office is:  3709
95  < Mulholland Drive >  the first week box office is:  587591
100  < Colors >  the first week box office is:  12045395
101  < Dark Angel >  the first week box office is:  None
102    the first week box office is:  367151
104  < Won't Back Down >  the first week box office is:  2603370
106  < Belle >  the first week box office is:  38017
107  < Extraordinary Measures >  the first week box office is:  6012594
108  < Braveheart >  the first week box office is:  9938276
109  < An Ideal Husband >  the first week box office is:  192802
110  < Hawaii >  the first week box office is:  None
111  < Bandits >  the first week box office is:  9511
112  < Little Black Book >  the first week box office is:  7075217
113  < In the Heart of the Sea >  the first week box office is:  11053366
114  < Biloxi Blues >  the first week box office is:  7093325
115  < The Lord of the Rings: The Fellowship of the Ring >  the first week box office is:  315544750
116  < Undercover Brother >  the first week box office is:  12037685
118  < Wild Hogs >  the first week box office is:  39699023
119  < Boxing Helena >  the first week box office is:  787724
121  < Rush Hour 2 >  the first week box office is:  67408222
122  < Tarzan >  the first week box office is:  38527856


# Notebook from xup5/Computational-Neuroscience-Class
Path: Lab 1 Poisson/Spike_trains_Full.ipynb

_____no_output_____This is a tutorial for generating Poisson spike trains.
Original tutorial: David Heeger (for Cold Spring Harbor Lab summer school).
Modified by: Odelia Schwartz and Xu Pan (2022)._____no_output_____# Generating Samples of a Poisson Process.

Spike trains in neurons are often modeled as a Poisson random
process.

One of the easiest ways to generate a Poisson spike
train is to rely on the approximation that in any small
time interval the probability of a spike is proportional
to the mean spike count (firing rate) times the time step:

   Prob{spike during (t,t+deltaT)} = r(t) * deltaT

where r(t) is the firing rate and deltaT is
the time step.  This approximation is only valid when deltaT is
very short so that there is essentially no chance of there
being more than one spike per time interval._____no_output_____## Part 1
_____no_output_____

# Let's begin by choosing a time step and by choosing an average
# firing rate.
# After running this, try and also choose different firing rates.

deltaT = 1e-3 			# Time step: 1e-3 secs = 1 msec
rate = 10			    	# Firing rate: spikes/sec
duration = 1			  # 1 sec simulation_____no_output_____# Choose the time intervals

import numpy as np

times = np.arange(0, duration, deltaT)
print(times.shape)
print(times)_____no_output_____# We can generate one random number between 0 and 1 (uniform distribution)
# Try this many times

xr_one = np.random.rand()
print(xr_one)_____no_output_____# Let's check if the mean firing rate times the time interval
# is greater than the random number
# Again, we can repeat choosing a random number and checking
# many times

intervalRate = rate * deltaT
print(intervalRate)
print(xr_one)
intervalRate > xr_one _____no_output_____# We want to do this process not just for one interval,
# but for all the time intervals over the duration
# We'll choose a bunch of random numbers, one for each time step,
# uniformly distributed between 0 and 1.

print(times.shape)
xr = np.random.rand(*times.shape)   # generate random numbers in an array the same size as times
print(xr.shape)
print(xr)

# *times.shape unpacks the tuple
# see what happens without the *
# see also https://stackoverflow.com/questions/43587526/pass-shape-tuple-to-numpy-random-rand_____no_output_____# Check that values are reasonably random by plotting a histogram

import matplotlib.pyplot as plt

plt.hist(xr, bins=20);
plt.xlabel('random draws', fontsize=16)
plt.ylabel('Probability', fontsize=16)_____no_output_____# Now, insert a spike whenever the probability of firing
# (rate*deltaT) is greater than the corresponding random number:
# Python (numpy) allows you to do this operation in one line, without a
# for loop.
# Remember, we defined: intervalRate = rate * deltaT

neuralResponse = intervalRate > xr

# We'll plot the neural responses (spikes) using the plt.stem()
# which is a lollipop plot...
plt.stem(times, neuralResponse, basefmt=" ");
plt.xlabel('Time', fontsize=16)
plt.ylabel('Neural Response (Spikes)', fontsize=16)

# You can repeat this several times, defining new random numbers xr_____no_output_____# To get a better understanding of what this does, we
# can look at the values for a vector of length 5
# Try changing the vector to get more spikes

tmpxr = np.array([.001, .1, .2, .5, .06])
print(intervalRate)
tmpResponse = intervalRate > tmpxr
print(tmpResponse)_____no_output_____# Calculate the spike count, which are the number of spikes during the time duration

spikeCount = np.sum(neuralResponse)
print(spikeCount)_____no_output_____

## Part 2_____no_output_____

# TODO: Repeat 5000 times: generating random numbers xr, computing neuralResponse and calculating
# spikeCount. Save the resulting number of spikes into spikeCounts each of the 5000 times.

len = 5000
spikeCounts = np.zeros(len)
print(spikeCounts.shape)

for repeat in range(5000):
  # Your code here:
  xr = np.random.rand(*times.shape)
  neuralResponse = intervalRate > xr
  spikeCounts[repeat] = np.sum(neuralResponse)_____no_output_____# What is the mean of the spikeCounts? How does it
# compare to the firing rate?

print(np.mean(spikeCounts))
print(rate)_____no_output_____# Plot a spike count histogram

print(np.min(spikeCounts))
print(np.max(spikeCounts))
x=np.arange(0, 30);
plt.hist(spikeCounts, bins=x, align='left')
plt.xlabel('Spike count')
plt.ylabel('Probability')_____no_output_____# Plot a normalized histogram (that sums to 1)
# for comparison with the Poisson

from scipy.stats import poisson

plt.hist(spikeCounts, bins=x, density=True, align='left')
plt.xlabel('Spike count')
plt.ylabel('Probability')
y = poisson.pmf(x, mu=np.mean(spikeCounts))
plt.plot(x,y)_____no_output_____

# To Do_____no_output_____

#   Here are some things for you to do in class for the code above:

# - Try repeating the above code by choosing a lower firing rate
#   (3) or a higher rate (15) by changing the variable rate at the top.
#   Plot the resulting distributions. How does the mean change?
#   Also plot examples of the spike trains in the repeated runs.

# - Try changing the number of repeats from 1000, to 50, 10, and 10000.
#   Plot the resulting distributions. How good is the fit to Poisson?_____no_output_____


# Notebook from trevorfoote/HST-Data-Reduction
Path: Example_WASP_79_Analysis.ipynb

import numpy as np
import matplotlib.pyplot as plt
import statistics
import pickle
import glob
from astropy.io import fits
import warnings
warnings.filterwarnings("ignore")
from astropy.visualization import simple_norm
import matplotlib.patches as patches

### personally developed pipeline scripts
import wfc3_extract
import mjd_bjd
import wfc3_systematics
import bat_model
import wfc3_mcmc
import wfc3_spec_trace
import binning
import wfc3_extract_spec

### set-up graphs for publication
import matplotlib as mpl
mpl.rcParams['xtick.major.size'] = 10
mpl.rcParams['xtick.major.width'] = 3
mpl.rcParams['ytick.major.size'] = 10
mpl.rcParams['ytick.major.width'] = 3
mpl.rcParams['mathtext.fontset'] = 'custom'
mpl.rcParams['axes.labelweight'] = 'bold'
mpl.rcParams['font.weight'] = 'bold'
mpl.rcParams['font.size'] = 18
mpl.rcParams['axes.linewidth'] = 6
plt.rcParams['font.size'] = '18'_____no_output_____

# 1) WLC Analysis_____no_output_____## 1.1)Data Reduction_____no_output_____

#fpath='/Users/Trevor/Pythoncode/HST/WASP79/'
fpath = '/home/trevor/OneDrive/Coding/HST/WASP79/'_____no_output_____### 1) to run extraction script for first time
'''
MJD_UTC, flux, flux_err, raDeg, decDeg, scidata_0, scidata, xybox = wfc3_extract.wfc3_extract()
BJD_TDB_f = mjd_bjd.mjd_bjd(MJD_UTC, flux, flux_err, raDeg, decDeg)
'''
### 2) if extraction has been completed and saved as file already and need MJD
fileObject = open(fpath+'W79_data/wlc_extract_out', 'rb')
MJD_UTC, flux, flux_err, raDeg, decDeg, scidata_0, scidata, xybox = pickle.load(fileObject)
fileObject.close()

fileObject = open(fpath+'W79_data/BJD_TDB', 'rb')
BJD_TDB_f, flux, flux_err = pickle.load(fileObject)
fileObject.close()
#'''_____no_output_____###View NDR of first image of first visit

# Create an ImageNormalize object
norm = simple_norm(scidata_0, 'sqrt', min_cut=0)

# Display the image
fig = plt.figure(figsize=(12,8))
ax = fig.add_subplot(1, 1, 1)
im = ax.imshow(scidata_0, origin='lower', norm=norm)

#First order spectra search
rect1 = patches.Rectangle((xybox[0],xybox[2]),(xybox[1]-xybox[0]),(xybox[3]-xybox[2]),
                          fill=False, ec='r', linewidth=2)
rect2 = patches.Rectangle((100,160),(170),(210), fill=False, ec='r', linewidth=2)
#Background region
rect3 = patches.Rectangle((15,xybox[3]+50),(485),(50), fill=False, ec='y', linewidth=2)

ax.add_patch(rect1)
ax.add_patch(rect2)
ax.add_patch(rect3)

fig.colorbar(im, label='Electrons')
ax.set_xlabel('Dispersion direction', fontsize=18)
ax.set_ylabel('Scanning direction', fontsize=18)
fig.show()_____no_output_____obs_start_BJD = np.amin(BJD_TDB_f)
print('Start of observation in BJD(TDB):', obs_start_BJD)

tt = BJD_TDB_f-obs_start_BJD

plt.figure()
plt.errorbar(tt, flux, flux_err, fmt='o', color='k')
plt.title('Raw light curve')
plt.xlabel('Time since obs start(min)')
plt.ylabel('Flux e-')
plt.show()Start of observation in BJD(TDB): 2457707.704777927
print('Total exposures:', len(flux), len(BJD_TDB_f))
flux, ttt, flux_err, BJD_TDB_ff = wfc3_systematics.remove_first_orbit(
    flux, tt, flux_err, BJD_TDB_f)
print('Total exposures after removing first orbit:', len(flux),
      len(BJD_TDB_ff))
flux, t, flux_err, BJD_TDB = wfc3_systematics.remove_first_exposures(
    flux, ttt, flux_err, BJD_TDB_ff)
print('Total exposures after removing first exposures:', len(flux), len(BJD_TDB))Total exposures: 64 64
Total exposures after removing first orbit: 52 52
First exposures = [ 0 13 26 39]
Total exposures after removing first exposures: 48 48
#### Remove second exposure from second orbit
flux = np.delete(flux,0)
t = np.delete(t,0)
flux_err = np.delete(flux_err,0)
BJD_TDB = np.delete(BJD_TDB,0)
print('Total usable exposures:', len(flux), len(BJD_TDB))Total usable exposures: 47 47
P_HST = 96.0/(24.0*60.0)
HSTphase = (BJD_TDB)/P_HST
HSTphase = HSTphase-np.floor(HSTphase)
HSTphase = HSTphase-np.amin(HSTphase)

plt.figure()
plt.errorbar(HSTphase, flux, flux_err, fmt='o', color='k')
plt.title('HST Phase')
plt.xlabel('HST Phase')
plt.ylabel('Flux e-')
plt.show()_____no_output_____m, b = wfc3_systematics.guess_lin_coeffs(t, flux)
S, phi, C = wfc3_systematics.guess_hook_coeffs(HSTphase, flux)_____no_output_____

## 1.2) Introduce BATMAN model_____no_output_____

#specify orbital parameters and limb darkening
t0 = 2457706.0503976-obs_start_BJD 			#time of inferior conjunction in BJD(TDB)
per = 3.66239264			#orbital period
rp = 0.10675				#planet radius (in units of stellar radii)
a = 7.292					#semi-major axis (in units of stellar radii)
inc = 85.929				#orbital inclination (in degrees)
ecc = 0.					#eccentricity
w = 90.						#longitude of periastron (in degrees)
u = [0.5012, 0.2630, -0.1034, -0.0301] 		 #limb darkening coefficients
limb_dark = "nonlinear"     #limb darkening model
transit_type = "secondary"  #transit type: primary (transit), seconday(eclipse)

#parameters specific for eclipse modeling include:
fp = 0.0002                 #planet-to-star flux ratio
t_secondary = 2457707.88159392-obs_start_BJD   #central eclipse time in BJD(TDB)_____no_output_____params, init_bat_model = bat_model.initialize_model(t, t0, per, rp, a,
                            inc, ecc, w, u, limb_dark, transit_type,
                            fp, t_secondary)

init_bat_lc = init_bat_model.light_curve(params)
plt.plot(t, init_bat_lc)
plt.show()_____no_output_____

### 1.2.1) free t_secondary_____no_output_____

###initial theta guesses
fixed_t_sec = False
if fixed_t_sec == True:
    theta_guess = np.zeros(6)
else:
    theta_guess = np.zeros(7)

theta_guess[0] = m    #linear coeff m
theta_guess[1] = b    #linear coeff b
theta_guess[2] = S    #hook coeff S
theta_guess[3] = phi  #hook coeff phi
theta_guess[4] = C    #hook coeff C
theta_guess[5] = fp
if fixed_t_sec == False:
    theta_guess[6] = t_secondary

fit_theta, fit_corr_bat_lc, fit_bat_lc, linearfit, hookfit = bat_model.fit_model(
    theta_guess, init_bat_model, params, t, HSTphase, flux, flux_err,
    t_secondary, fixed_t_sec)_____no_output_____plt.figure()
plt.scatter(t, flux/linearfit/hookfit)
plt.xlabel('time since start of obs (days)')
plt.ylabel('Flux')
plt.ylim(0.9995,1.0005)
plt.title('Flux after removing Linear & Hook (free t)')
plt.show()

res = flux/linearfit/hookfit
#print(res)
print('Sum of the residuals =', sum(res))
SDNR = statistics.stdev(res)
print('SDNR =', SDNR)

for i in np.arange(len(flux_err)):
    flux_err[i] = flux[i]*SDNR_____no_output_____res = bat_model.residuals(fit_theta, init_bat_model, params, t, HSTphase, flux,
                flux_err, t_secondary, fixed_t_sec)

plt.figure()
plt.scatter(t, res)
plt.xlabel('time since start of obs (days)')
plt.ylabel('Residual')
plt.title('Updated residuals with SDNR correction')
plt.show()

chi2 = sum(res*res)
print('Chi-square=', chi2)

dof = len(flux)-len(fit_theta)
print('Deg of freedom=', dof)

rchi2 = chi2/dof
print('Reduced Chi-square=', rchi2)

fit_corrs = linearfit*hookfit

plt.figure()
plt.errorbar(t, flux, flux_err, fmt='o', color='b', label='obs')
plt.plot(t, fit_corr_bat_lc, 'ro',
         label='fitted function')
#plt.plot(BJD_TDB, mean_flux*fit_corrs, 'go', label = 'just corrections')
plt.xlabel('time since start of obs (days)')
plt.ylabel('Flux')
plt.legend(bbox_to_anchor=(1.05, 1.0), loc='upper left')
plt.show()

#print(fit_theta)_____no_output_____

### Fig 1 for paper_____no_output_____

norm_flux = flux/np.mean(flux)
n_m, n_b = wfc3_systematics.guess_lin_coeffs(t, norm_flux)
n_S, n_phi, n_C = wfc3_systematics.guess_hook_coeffs(HSTphase, norm_flux)
norm_lin_fit = wfc3_systematics.lin_trend(t, n_m, n_b)
norm_hook_fit = wfc3_systematics.hook_trend(HSTphase, n_S, n_phi, n_C)
norm_corr = norm_lin_fit*norm_hook_fit_____no_output_____fig, axs = plt.subplots(4,1, sharex='col', gridspec_kw={'hspace':0.05},
                        figsize=(20,15))

for ax in axs:
    plt.xticks(weight='bold')
    plt.yticks(weight='bold')

### Raw light curve
r_wl = norm_flux
axs[0].plot(BJD_TDB, r_wl, 'ko')
axs[0].plot(BJD_TDB, fit_bat_lc, linewidth=5)
axs[0].set_ylabel('Relative Flux')
axs[0].set_title('Raw light curve', fontsize=20, fontweight='bold')
axs[0].yaxis.set_major_formatter(mpl.ticker.FormatStrFormatter('%.4f'))

#raw residual
r_res = r_wl - fit_bat_lc

axs[1].plot(BJD_TDB, r_res*1e6, 'ko')
axs[1].set_xlabel('$\mathbf{BJD_{TDB}}$')
axs[1].set_ylabel('Raw Residuals (ppm)')
axs[1].set_gridspec_kw={'hspace':5}

### Corrected light curve
c_wl = norm_flux/norm_corr
axs[2].plot(BJD_TDB, c_wl, 'ko')
axs[2].plot(BJD_TDB, fit_bat_lc, linewidth=5)
axs[2].set_ylabel('Relative Flux', fontsize=14, fontweight='bold')
axs[2].set_title('Corrected light curve', fontsize=20, fontweight='bold')
axs[2].yaxis.set_major_formatter(mpl.ticker.FormatStrFormatter('%.4f'))

### Residuals after corrections
c_res = c_wl-fit_bat_lc
axs[3].plot(BJD_TDB, c_res*1e6, 'ko', )
axs[3].set_xlabel('$\mathbf{BJD_{TDB}}$', fontsize=14)
axs[3].set_ylabel('Residuals (ppm)', fontsize=14, fontweight='bold')_____no_output_____

## 1.3) MCMC_____no_output_____

# -*- coding: utf-8 -*-
"""
Created on Fri Sep 18 14:36:27 2020

@author: Trevor
"""
import numpy as np
import matplotlib.pyplot as plt
import emcee
import corner
import bat_model

def mcmc(theta, BJD_TDB, flux, flux_err, model, params, HSTphase, t_secondary,
         fixed_t_sec, G_t, G_err, SDNR, P):
    ndim, nwalkers, nsteps = len(theta), 50, 10000 #6*len(theta) #3*len(theta)
    sampler=emcee.EnsembleSampler(nwalkers, ndim, lnprob, args=(params, model,
                                        BJD_TDB, flux, flux_err, HSTphase,
                                        t_secondary, fixed_t_sec, G_t, G_err, P))
    #pos=[theta*(1.+1.e-2*np.random.randn(ndim)) for i in range(nwalkers)]
    pos=[theta*(1.+SDNR*np.random.randn(ndim)) for i in range(nwalkers)]
    if fixed_t_sec == False:
        for i in np.arange(len(pos)):
            pos[i][6] = theta[6] + (0.005*np.random.randn())
    sampler.run_mcmc(pos, nsteps, progress=True);

    #Burn in removal
    samples=sampler.chain[:,int(0.2*nsteps):,:].reshape((-1,ndim))
    #samples=sampler.chain[:,:,:].reshape((-1,ndim))

    if fixed_t_sec == False:
        m_mcmc, b_mcmc, S_mcmc, phi_mcmc, C_mcmc, fp_mcmc, t0_mcmc=map(
                lambda v: (v[1], v[2]-v[1], v[1]-v[0]),
                zip(*np.percentile(samples, [16, 50, 84], axis=0)))
        fig = corner.corner(samples, labels = ['m', 'b', 'S', 'phi', 'C',
                                               'fp', 't_secondary'])
        mcmc_params = [m_mcmc, b_mcmc, S_mcmc, phi_mcmc, C_mcmc, fp_mcmc, t0_mcmc]

    else:
        m_mcmc, b_mcmc, S_mcmc, phi_mcmc, C_mcmc, fp_mcmc =map(
                lambda v: (v[1], v[2]-v[1], v[1]-v[0]),
                zip(*np.percentile(samples, [16, 50, 84], axis=0)))

        fig = corner.corner(samples, labels = ['m', 'b', 'S', 'phi', 'C',
                                               'fp'])
        mcmc_params = [m_mcmc, b_mcmc, S_mcmc, phi_mcmc, C_mcmc, fp_mcmc]
    plt.show()

    print('parameter', )
    print('m', m_mcmc)
    print('b', b_mcmc)
    print('S', S_mcmc)
    print('phi', phi_mcmc)
    print('C', C_mcmc)
    print('fp', fp_mcmc)
    if fixed_t_sec == False:
        print('t_sec', t0_mcmc)

    return sampler, samples, pos, mcmc_params

#posterior probability
def lnprob(theta, params, model, BJD_TDB, flux, flux_err, HSTphase,
           t_secondary, fixed_t_sec, G_t, G_err, P):
    lp = lnprior(theta, G_t, G_err, fixed_t_sec, P)
    if not np.isfinite(lp):
        return -np.inf
    return lp + lnlike(theta, params, model, BJD_TDB, flux, flux_err, HSTphase,
                       t_secondary, fixed_t_sec)

#prior
def lnprior(theta, G_t, G_err, fixed_t_sec, P):
    if fixed_t_sec == False:
        m, b, S, phi, C, fp, t_secondary = theta
        if fp>1 or fp<0:
        #if fp>1 or fp<0 or t_secondary>G_t+P/3 or t_secondary<G_t-P/3:
            return -np.inf
        else:
            #return 0.0
            mu = G_t
            sigma = G_err
            return np.log((1.0/(np.sqrt(2*np.pi)*sigma**2))*np.exp(-0.5*(t_secondary-mu)**2/sigma**2))

    else:
        m, b, S, phi, C, fp = theta
        if fp>1 or fp<0:
            return -np.inf
        else:
            return 0.0

#likelihood function
def lnlike(theta, params, model, BJD_TDB, flux, flux_err, HSTphase,
           t_secondary, fixed_t_sec):
    res = bat_model.residuals(theta, model, params, BJD_TDB, HSTphase,
                              flux, flux_err, t_secondary, fixed_t_sec)

    ln_likelihood = -0.5*(np.sum((res)**2 + np.log(2.0*np.pi*(flux_err)**2)))

    return ln_likelihood
_____no_output_____G_t0 = 57004.70593+2400000 - obs_start_BJD
G_err = 0.00227
P = 3.66239264
P_err = 5.6e-7

### Calculating eclipse times backwards
##t_max = (fit_theta[6]-G_t0)/P + 10 #number of periods from T0 in Sotzen back to 0 in BJD(TDB)
t_max = (t_secondary-G_t0)/P + 10
t_range = np.arange(1, t_max, 1)
Calc_T = np.zeros(len(t_range))
Calc_err = np.zeros(len(t_range))

for i in np.arange(len(t_range)):
    Calc_T[i] = (G_t0+t_range[i]*P)
    Calc_err[i] = np.sqrt((G_err)*(G_err)+(t_range[i]*P_err)*(t_range[i]*P_err))
#print('T', Calc_T)
#print('error', Calc_err)

##idx_closest = (np.abs(fit_theta[6]-Calc_T)).argmin()
idx_closest = (np.abs(t_secondary-Calc_T)).argmin()

G_t = Calc_T[idx_closest]
G_err = Calc_err[idx_closest]

print('Closest Calculated Eclipse', G_t, '+/-', G_err, 'days since start of obs')
print('Start of Observation', obs_start_BJD)_____no_output_____sampler, samples, pos, mcmc_params = mcmc(fit_theta, t, flux, flux_err,
                                  init_bat_model, params, HSTphase,
                                  t_secondary, fixed_t_sec, G_t, G_err, SDNR, P)_____no_output_____mu = G_t
sigma = 10*G_err
t_test = np.linspace(0.0, 1., 1000)
y = np.log((1.0/(np.sqrt(2*np.pi)*sigma**2))*np.exp(-0.5*(t_test-mu)**2/sigma**2))
print(G_t)
plt.plot(t_test,y, 'r')

mu2 = 0.18227913120319336
sigma2 = 0.013308739868601427
#t_test2 = np.linspace(0.170, 0.191, 1000)
y2 = np.log((1.0/(np.sqrt(2*np.pi)*sigma2**2))*np.exp(-0.5*(t_test-mu2)**2/sigma2**2))

plt.plot(t_test,y2, 'b')_____no_output_____mu = G_t
sigma = G_err
t_test = np.linspace(0.0, 1., 1000)
y = np.log((1.0/(np.sqrt(2*np.pi)*sigma**2))*np.exp(-0.5*(t_test-mu)**2/sigma**2))
print(G_t)
plt.plot(t_test,y, 'r')

mu2 = 0.18052917968431473
sigma2 = 0.002728958277691901
#t_test2 = np.linspace(0.170, 0.191, 1000)
y2 = np.log((1.0/(np.sqrt(2*np.pi)*sigma2**2))*np.exp(-0.5*(t_test-mu2)**2/sigma2**2))

plt.plot(t_test,y2, 'b')
_____no_output_____mcmc_bat_params, mcmc_bat_model = bat_model.initialize_model(t, t0, per, rp, a,
                                                    inc, ecc, w, u, limb_dark,
                                                    transit_type, mcmc_params[5][0],
                                                    mcmc_params[6][0])

wlc_bat = mcmc_bat_model.light_curve(mcmc_bat_params)
wlc_line = wfc3_systematics.lin_trend(t, mcmc_params[0][0], mcmc_params[1][0])
wlc_hook = wfc3_systematics.hook_trend(HSTphase, mcmc_params[2][0], mcmc_params[3][0],
                                       mcmc_params[4][0])

mcmc_full = wlc_bat*wlc_line*wlc_hook

plt.plot(t, mcmc_full, 'r*', linewidth=4, zorder=10)
plt.errorbar(t, flux, flux_err, fmt='bo', zorder=0)
plt.show()

wlc_res = (flux-mcmc_full)/flux_err
#print(wlc_res)

wlc_rchi2 = (wlc_res**2).sum()/(len(flux)-len(mcmc_params))

print('Reduced Chi Square =', wlc_rchi2)
print(t_secondary - mcmc_params[6][0])
init_bat_lc = init_bat_model.light_curve(params)
plt.plot(t, init_bat_lc)
plt.plot(t, wlc_bat)
plt.show()_____no_output_____wlc_m = (-495332.81617154245)
wlc_b = (474326845.5100336)
wlc_S = (3.956467238119049e+35)
wlc_phi = (2.3502374230657126e+35)
wlc_C = (0.002293176169255235)
wlc_fp = (0.00015379521351322595)
wlc_t_sec = (0.18055797174061183)
wlc_theta = (wlc_m, wlc_b, wlc_S, wlc_phi, wlc_C, wlc_fp, wlc_t_sec)
wlc_line = wfc3_systematics.lin_trend(BJD_TDB, wlc_m, wlc_b)
wlc_hook = wfc3_systematics.hook_trend(HSTphase, wlc_S, wlc_phi, wlc_C)
full_wlc_model, wlc_model, wlclin, wlchook = bat_model.modify_model(wlc_theta, init_bat_model, params,
                                                                    t, HSTphase, t_secondary,
                                                                    fixed_t_sec)
plt.figure(figsize=[15,10])
plt.plot(BJD_TDB, full_wlc_model, 'ro', label='MCMC best fit')
plt.errorbar(BJD_TDB, flux, flux_err, fmt='o', color='k', label='Observations')
plt.plot(BJD_TDB, fit_bat_lc*np.mean(flux), linewidth=5, label='Model Eclipse')
plt.xlabel('Time ($\mathbf{BJD_{TDB}}$)')
plt.ylabel('Flux')
plt.legend(bbox_to_anchor=(1.05, 1.0), loc='upper left')
plt.show()
_____no_output_____plt.rcParams['font.size'] = '14'
fig, axs = plt.subplots(2,2, sharex='col', gridspec_kw={'hspace':0.05,
                        'wspace':0.3, 'height_ratios':[2,1]}, figsize=(15,8))

for ax in axs:
    plt.xticks(weight='bold')
    plt.yticks(weight='bold')

### Raw light curve
r_wl = norm_flux
axs[0,0].plot(BJD_TDB, r_wl, 'ko')
axs[0,0].plot(BJD_TDB, fit_bat_lc, linewidth=5)
axs[0,0].set_ylabel('Relative Flux', fontsize=18, fontweight='bold')
axs[0,0].set_title('Raw light curve', fontsize=20, fontweight='bold')
axs[0,0].yaxis.set_major_formatter(mpl.ticker.FormatStrFormatter('%.4f'))

#raw residual
r_res = r_wl - fit_bat_lc

axs[1,0].plot(BJD_TDB, r_res*1e6, 'ko')
axs[1,0].set_xlabel('$\mathbf{BJD_{TDB}}$', fontsize=18)
axs[1,0].set_ylabel('Residuals (ppm)', fontsize=18, fontweight='bold')
axs[1,0].set_ylim(-1000, 600)

### Corrected light curve
c_wl = norm_flux/norm_corr

axs[0,1].plot(BJD_TDB, c_wl, 'ko')
axs[0,1].plot(BJD_TDB, fit_bat_lc, linewidth=5)
axs[0,1].set_ylabel('Relative Flux', fontsize=18, fontweight='bold')
axs[0,1].set_title('Corrected light curve', fontsize=20, fontweight='bold')
axs[0,1].yaxis.set_major_formatter(mpl.ticker.FormatStrFormatter('%.4f'))

### Residuals after corrections
c_res = c_wl-fit_bat_lc

axs[1,1].plot(BJD_TDB, c_res*1e6, 'ko', )
axs[1,1].set_xlabel('$\mathbf{BJD_{TDB}}$', fontsize=18)
axs[1,1].set_ylabel('Residuals (ppm)', fontsize=18, fontweight='bold')
axs[1,1].set_ylim(-1000, 600)_____no_output_____

# 2) Spectral Analysis_____no_output_____

xsum = []
for i in range(xybox[0], xybox[1]+1):
    xsum.append(np.sum(scidata_0[xybox[2]:xybox[3],i]))
xrng = np.arange(xybox[0], xybox[1]+1)

plt.plot(xrng, xsum)
plt.xlabel('xpix')
plt.ylabel('Flux (e$^-$)')
print("# of Columns =", len(xrng))_____no_output_____

## 2.1) Convert from pix to wl via _____no_output_____

#yref = xybox[3]-xybox[2]
yref = (xybox[3]+xybox[2])/2
xref = xybox[1]+50
x = xrng

DLDP_A_0 = [8.95431E+03,    9.35925E-02,            0.0,             0.0,           0.0,            0.0]
DLDP_A_1 = [4.51423E+01,    3.17239E-04,    2.17055E-03,    -7.42504E-07,   3.48639E-07,    3.09213E-07]

DLDP_0 = DLDP_A_0[0] + DLDP_A_0[1]*xref + DLDP_A_0[2]*yref
DLDP_1 = DLDP_A_1[0] + DLDP_A_1[1]*xref + DLDP_A_1[2]*yref + \
         DLDP_A_1[3]*xref**2 + DLDP_A_1[4]*xref*yref + DLDP_A_1[5]*yref**2

wl_y = DLDP_0 + DLDP_1*(x-xref) + yref

plt.plot(1+wl_y/10000, xsum)
plt.xlabel('wavelength')
plt.ylabel('Flux (e$^-$)')
plt.show()_____no_output_____

## 2.2) Introduce Phoenix Model for comparison_____no_output_____

up_wl = 1.77
low_wl = 0.995

w79 = np.loadtxt(fpath+'WASP-79_lte066-4.0-0.0a+0.0.BT-Settl.txt')
w79_wl = []
w79_sflux = []
for i in np.arange(len(w79[:,0])):
    w79_wl.append(w79[i,0])
    w79_sflux.append(w79[i,1])
w79_wl = np.array(w79_wl)
w79_sflux = np.array(w79_sflux)
w79_wl = w79_wl/10000
w79_sflux = 10**(w79_sflux-8.)*10

model_wl = []
model_sflux = []
for i in np.arange(len(w79_wl)):
    if low_wl <= w79_wl[i] <= up_wl:
        model_wl.append(w79_wl[i])
        model_sflux.append(w79_sflux[i])
'''
plt.plot(model_wl, model_sflux, 'b-')
plt.title('Stellar Model')
plt.xlabel('Wavelength ($\mu$m)')
plt.ylabel('Flux (e$^-$)')
plt.show()
'''
print('# of data pts in Stellar model=', len(model_wl))_____no_output_____flatimage=glob.glob(fpath + '*G141.1st*')
hdulist = fits.open(flatimage[0])
sens_data =hdulist[1].data

sens_wl = np.zeros(len(sens_data))
sens = np.zeros(len(sens_data))
sens_err = np.zeros(len(sens_data))
for i in np.arange(len(sens_data)):
    sens_wl[i]=sens_data[i][0]
    sens[i]=sens_data[i][1]
    sens_err[i] = sens_data[i][2]

sens_wl = sens_wl/10000
norm_sens = sens/max(sens)
print('# of data pts in sensitivity model =', len(sens_wl))_____no_output_____

### 2.2.1) Bin Stellar model down to sensitivity model _____no_output_____

### 1) to run extraction script for first time
#final = binning.binning(model_wl, model_sflux, newx = sens_wl)
#bin_wl, bin_sflux = final['bin_x'], final['bin_y']

### 2) if extraction has been completed and saved as file already and need MJD
fileObject = open(fpath+'W79_data/bin_extract_out', 'rb')
bin_sflux, bin_wl, bin_edge, bin_dy, bin_n = pickle.load(fileObject)
fileObject.close()

bin_sflux_mod = sens*bin_sflux
norm_bin_sflux = bin_sflux_mod/max(bin_sflux_mod)

plt.plot(bin_wl, norm_bin_sflux)
plt.xlabel('Wavelength ($\mu$m)')
plt.ylabel('Relative Signal')
plt.title('Norm binned stellar model w/ sensitivity')
plt.show()_____no_output_____

## 2.3) Overlay Phoenix model with observation_____no_output_____

fig, (ax1, ax2) = plt.subplots(2,1, sharex='col', gridspec_kw={'hspace':0.0},
                               figsize=(7,7))

### Wavelength calibration (uncorrected)
norm_xsum = xsum/max(xsum)
ax1.plot(bin_wl, norm_bin_sflux, 'b-', label='Phoenix model w/sensitivity curve')
ax1.plot(1.+wl_y/10000, norm_xsum, 'r-', linewidth=3.0, label='Calibrated observation')
#ax1.set_xlabel('Wavelength ($\mu m$)')
ax1.set_ylabel('Relative Signal')
ax1.legend(fontsize=11)

### Wavelength calibration (corrected)
ax2.plot(bin_wl, norm_bin_sflux, 'b-', label='Phoenix model w/sensitivity curve')
ax2.plot((1.-0.02)+wl_y/10000, norm_xsum, 'r-',linewidth=3.0, label='Calibrated observation w/offset')
ax2.set_xlabel('Wavelength ($\mu m$)')
ax2.set_ylabel('Relative Signal')
#ax2.legend()

#label_axes(fig, labels = ('a)', 'b)'), loc=(-.15, 1), ha='left')_____no_output_____

## 2.4) Data Reduction_____no_output_____

### 1) to run extraction script for first time
#bin_wls, bin_flux, bin_flux_err = wfc3_extract_spec.wfc3_extract(num_bins=3)

### 2) if extraction has been completed and saved as file already and need MJD
fileObject = open(fpath+'W79_data/spec_extract_out', 'rb')
bin_wls, bin_flux, bin_flux_err = pickle.load(fileObject)
fileObject.close()_____no_output_____plt.errorbar(tt, bin_flux[:,0], bin_flux_err[:,0], fmt='o', color='b')
plt.title('Raw 1.18$\mu$m light curve')
plt.xlabel('Time since obs start(min)')
plt.ylabel('Flux e-')
plt.show()

plt.errorbar(tt, bin_flux[:,1], bin_flux_err[:,1], fmt='o', color='r')
plt.title('Raw 1.38$\mu$m light curve')
plt.xlabel('Time since obs start(min)')
plt.ylabel('Flux e-')
plt.show()

plt.errorbar(tt, bin_flux[:,2], bin_flux_err[:,2], fmt='o', color='g')
plt.title('Raw 1.58$\mu$m light curve')
plt.xlabel('Time since obs start(min)')
plt.ylabel('Flux e-')
plt.show()_____no_output_____print('Total exposures:', len(bin_flux[:,0]), len(bin_flux[:,1]), len(bin_flux[:,2]))
bflux1, ttt, bflux_err1, BJD_TDB_ff = wfc3_systematics.remove_first_orbit(bin_flux[:,0], tt, bin_flux_err[:,0], BJD_TDB_f)
bflux2, ttt, bflux_err2, BJD_TDB_ff = wfc3_systematics.remove_first_orbit(bin_flux[:,1], tt, bin_flux_err[:,1], BJD_TDB_f)
bflux3, ttt, bflux_err3, BJD_TDB_ff = wfc3_systematics.remove_first_orbit(bin_flux[:,2], tt, bin_flux_err[:,2], BJD_TDB_f)

print('Total exposures after removing first orbit:', len(bflux1), len(bflux2), len(bflux3))
bflux1, t, bflux_err1, BJD_TDB = wfc3_systematics.remove_first_exposures(bflux1, ttt, bflux_err1, BJD_TDB_ff)
bflux2, t, bflux_err2, BJD_TDB = wfc3_systematics.remove_first_exposures(bflux2, ttt, bflux_err2, BJD_TDB_ff)
bflux3, t, bflux_err3, BJD_TDB = wfc3_systematics.remove_first_exposures(bflux3, ttt, bflux_err3, BJD_TDB_ff)

print('Total exposures after removing first exposures:', len(bflux1), len(bflux2), len(bflux3))_____no_output_____#### Remove second exposure from second orbit
bflux1 = np.delete(bflux1,0)
bflux2 = np.delete(bflux2,0)
bflux3 = np.delete(bflux3,0)
t = np.delete(t,0)
bflux_err1 = np.delete(bflux_err1,0)
bflux_err2 = np.delete(bflux_err2,0)
bflux_err3 = np.delete(bflux_err3,0)
BJD_TDB = np.delete(BJD_TDB,0)
print('Total usable exposures:', len(bflux1), len(BJD_TDB))
_____no_output_____### Normalize flux
norm_bflux1 = bflux1/np.mean(bflux1)
norm_bflux2 = bflux2/np.mean(bflux2)
norm_bflux3 = bflux3/np.mean(bflux3)

### Help graph
nc_flux1 = norm_bflux1/norm_corr
mid_b1 = np.mean(nc_flux1)
ran_b1 = np.max(nc_flux1)-np.min(nc_flux1)

nc_flux2 = norm_bflux2/norm_corr
mid_b2 = np.mean(nc_flux2)
ran_b2 = np.max(nc_flux2)-np.min(nc_flux2)

nc_flux3 = norm_bflux3/norm_corr
mid_b3 = np.mean(nc_flux3)
ran_b3 = np.max(nc_flux3)-np.min(nc_flux3)_____no_output_____

### 2.4.1) Correct linear and hook effects_____no_output_____

plt.figure()
plt.scatter(t, nc_flux1)
plt.xlabel('time since start of obs (days)')
plt.ylabel('Normalized Flux')
plt.ylim(mid_b1-ran_b1, mid_b1+ran_b1)
plt.title('Flux after removing Linear & Hook (free t)')
plt.show()

nres1 = nc_flux1
print('Sum of the residuals =', sum(nres1))
nSDNR1 = statistics.stdev(nres1)
print('SDNR =', nSDNR1)

for i in np.arange(len(bflux_err1)):
    bflux_err1[i] = bflux1[i]*nSDNR1

#######################################
plt.figure()
plt.scatter(t, nc_flux2)
plt.xlabel('time since start of obs (days)')
plt.ylabel('Normalized Flux')
plt.ylim(mid_b2-ran_b2, mid_b2+ran_b2)
plt.title('Flux after removing Linear & Hook (free t)')
plt.show()

nres2 = nc_flux2
print('Sum of the residuals =', sum(nres2))
nSDNR2 = statistics.stdev(nres2)
print('SDNR =', nSDNR2)

for i in np.arange(len(bflux_err2)):
    bflux_err2[i] = bflux2[i]*nSDNR2

##############################################
plt.figure()
plt.scatter(t, nc_flux3)
plt.xlabel('time since start of obs (days)')
plt.ylabel('Normalized Flux')
plt.ylim(mid_b3-ran_b3, mid_b3+ran_b3)
plt.title('Flux after removing Linear & Hook (free t)')
plt.show()

nres3 = nc_flux3
print('Sum of the residuals =', sum(nres3))
nSDNR3 = statistics.stdev(nres3)
print('SDNR =', nSDNR3)

for i in np.arange(len(bflux_err3)):
    bflux_err3[i] = bflux3[i]*nSDNR3_____no_output_____

### 2.4.2 Correct spectroscopic linear systematic_____no_output_____

m1, b1 = wfc3_systematics.guess_lin_coeffs(t, nc_flux1)
m2, b2 = wfc3_systematics.guess_lin_coeffs(t, nc_flux2)
m3, b3 = wfc3_systematics.guess_lin_coeffs(t, nc_flux3)_____no_output_____linearfit1 = m1*t + b1
nc_flux1 = nc_flux1/linearfit1
mid_b1 = np.mean(nc_flux1)
ran_b1 = np.max(nc_flux1)-np.min(nc_flux1)

linearfit2 = m2*t + b2
nc_flux2 = nc_flux2/linearfit2
mid_b2 = np.mean(nc_flux2)
ran_b2 = np.max(nc_flux2)-np.min(nc_flux2)

linearfit3 = m3*t + b3
nc_flux3 = nc_flux3/linearfit3
mid_b3 = np.mean(nc_flux3)
ran_b3 = np.max(nc_flux3)-np.min(nc_flux3)_____no_output_____plt.figure()
plt.scatter(t, nc_flux1)
plt.xlabel('time since start of obs (days)')
plt.ylabel('Normalized Flux')
plt.ylim(mid_b1-ran_b1, mid_b1+ran_b1)
plt.title('Flux after removing Linear & Hook (free t)')
plt.show()

nres1 = nc_flux1
print('Sum of the residuals =', sum(nres1))
nSDNR1 = statistics.stdev(nres1)
print('SDNR =', nSDNR1)

for i in np.arange(len(bflux_err1)):
    bflux_err1[i] = bflux1[i]*nSDNR1

#######################################
plt.figure()
plt.scatter(t, nc_flux2)
plt.xlabel('time since start of obs (days)')
plt.ylabel('Normalized Flux')
plt.ylim(mid_b2-ran_b2, mid_b2+ran_b2)
plt.title('Flux after removing Linear & Hook (free t)')
plt.show()

nres2 = nc_flux2
print('Sum of the residuals =', sum(nres2))
nSDNR2 = statistics.stdev(nres2)
print('SDNR =', nSDNR2)

for i in np.arange(len(bflux_err2)):
    bflux_err2[i] = bflux2[i]*nSDNR2

##############################################
plt.figure()
plt.scatter(t, nc_flux3)
plt.xlabel('time since start of obs (days)')
plt.ylabel('Normalized Flux')
plt.ylim(mid_b3-ran_b3, mid_b3+ran_b3)
plt.title('Flux after removing Linear & Hook (free t)')
plt.show()

nres3 = nc_flux3
print('Sum of the residuals =', sum(nres3))
nSDNR3 = statistics.stdev(nres3)
print('SDNR =', nSDNR3)

for i in np.arange(len(bflux_err3)):
    bflux_err3[i] = bflux3[i]*nSDNR3_____no_output_____plt.figure()
plt.scatter(t, nres1)
plt.xlabel('time since start of obs (days)')
plt.ylabel('Residuals')
plt.ylim(min(nres1)-0.0001, max(nres1)+0.0001)
plt.title('Residual of bin 1')
plt.show()

plt.figure()
plt.scatter(t, nres2)
plt.xlabel('time since start of obs (days)')
plt.ylabel('Residuals')
plt.ylim(min(nres2)-0.0001, max(nres2)+0.0001)
plt.title('Residual of bin 2')
plt.show()

plt.figure()
plt.scatter(t, nres3)
plt.xlabel('time since start of obs (days)')
plt.ylabel('Residuals')
plt.ylim(min(nres3)-0.0001, max(nres3)+0.0001)
plt.title('Residual of bin 3')
plt.show()_____no_output_____print('For binned data centered at wavelength 1.18$\mu$m')
sampler, samples, pos, fp_mcmc = wfc3_mcmc.mcmc(fit_theta, t, bflux1, bflux_err1,
                                  init_bat_model, params, HSTphase,
                                  t_secondary, fixed_t_sec, G_t, G_err, nSDNR1, P)_____no_output_____print('For binned data centered at wavelength 1.38$\mu$m')
sampler, samples, pos, fp_mcmc = wfc3_mcmc.mcmc(fit_theta, t, bflux2, bflux_err2,
                                  init_bat_model, params, HSTphase,
                                  t_secondary, fixed_t_sec, G_t, G_err, nSDNR2, P)_____no_output_____print('For binned data centered at wavelength 1.58$\mu$m')
sampler, samples, pos, fp_mcmc = wfc3_mcmc.mcmc(fit_theta, t, bflux3, bflux_err3,
                                  init_bat_model, params, HSTphase,
                                  t_secondary, fixed_t_sec, G_t, G_err, nSDNR3, P)_____no_output_____

### ----------------------- Testing Workspace below -------------------_____no_output_____

### 1) to run extraction script for first time
#import wfc3_extract
#MJD_UTC, flux, flux_err, raDeg, decDeg, scidata_0, scidata, xybox = wfc3_extract.wfc3_extract()

### 2) if extraction has been completed and saved as file already and need MJD
fileObject = open(fpath+'W79_data/spec_extract_out', 'rb')
bin_wls, bin_flux, bin_flux_err = pickle.load(fileObject)
fileObject.close()

fileObject = open(fpath+'W79_data/BJD_TDB', 'rb')
BJD_TDB_f, flux, flux_err = pickle.load(fileObject)
fileObject.close()

fileObject = open(fpath+'W79_data/wlc1_extract_out', 'rb')
MJD_UTC, flux, flux_err, raDeg, decDeg, scidata_0, scidata, xybox = pickle.load(fileObject)
fileObject.close()_____no_output_____print('spec wlc flux', sum(bin_flux[0,:]))
print('orig wlc flux',flux[0])

bin_wlc = np.zeros(len(bin_flux))
for i in np.arange(len(bin_flux)):
    bin_wlc[i] = sum(bin_flux[i,:])
diff_test = flux-bin_wlc
print(diff_test)
width = len(diff_test)/3
print(len(diff_test[0:int(width)]))
print(len(diff_test[int(width):2*int(width)]))
print(len(diff_test[2*int(width):3*int(width)+1]))
#print(diff_test[int(width):2*int(width)])
#print(diff_test[2*int(width):3*int(width)+1])_____no_output_____plt.figure()
plt.plot(tt, flux, 'ko')
plt.plot(tt, bin_wlc, 'ro')

plt.xlabel('Time since obs start(min)')
plt.ylabel('Flux e-')
plt.show()_____no_output_____

### Spectral binning into three bins_____no_output_____

obs_wl = (1.-0.01)+wl_y/10000
wl_len = len(obs_wl)
num_bins = 3
wl_width = wl_len/num_bins

bin_wl_1 = obs_wl[0:int(wl_width)]
bin_nf_1 = norm_xsum[0:int(wl_width)]

bin_wl_2 = obs_wl[int(wl_width):2*int(wl_width)]
bin_nf_2 = norm_xsum[int(wl_width):2*int(wl_width)]

bin_wl_3 = obs_wl[2*int(wl_width):3*int(wl_width)]
bin_nf_3 = norm_xsum[2*int(wl_width):3*int(wl_width)]

bin_flux1 = xsum[0:int(wl_width)]
bin_flux2 = xsum[int(wl_width):2*int(wl_width)]
bin_flux3 = xsum[2*int(wl_width):3*int(wl_width)]
_____no_output_____wl_1 = np.mean(bin_wl_1)
lc_1 = np.mean(bin_nf_1)
print(wl_1, lc_1)

wl_2 = np.mean(bin_wl_2)
lc_2 = np.mean(bin_nf_2)
print(wl_2, lc_2)

wl_3 = np.mean(bin_wl_3)
lc_3 = np.mean(bin_nf_3)
print(wl_3, lc_3)

flux1 = np.mean(bin_flux1)
flux2 = np.mean(bin_flux2)
flux3 = np.mean(bin_flux3)

print(flux1)_____no_output_____plt.plot(bin_wl_1, bin_nf_1, 'b-')
plt.plot(wl_1, lc_1, 'b*')
plt.plot(bin_wl_2, bin_nf_2, 'r-')
plt.plot(wl_2, lc_2, 'r*')
plt.plot(bin_wl_3, bin_nf_3, 'g-')
plt.plot(wl_3, lc_3, 'g*')
plt.xlabel('wavelength')
plt.ylabel('Flux (e$^-$)')
plt.show()_____no_output_____