# Notebook from MuchenSun/another_gp_tutorial
Path: worksheet.ipynb

# A(nother) hands-on intro to GP regression

**Original Author: Dan Foreman-Mackey** (foreman.mackey@gmail.com)

**A(nother) Author: Muchen Sun** (sunmch15@gmail.com)

~~~
Note:
This tutorial is heavily based on Dan Foreman-Mackey's original amazing hands-on GP tutorial, which literally helps me enter the world of GP. After finishing the tutorial, as a reader/user myself, I think this tutorial might be even more helpful if I could include more details, so here we are.

Compared with original version, here I added more explanation about multi-variate Gaussian distribution, the role of covariance matrix, and exposed more details of regression with GP.
~~~

In this tutorial, we'll work through an example problem in Gaussian process (GP) regression using Python.
The goal of the tutorial is to give you a qualitative sense of how GP regression works and a rough idea of how you might implement it in a real world problem.

This is designed to be open-ended and the last sections should be instructive even if you're already familiar with the implementation of GPs.
I've chosen to use Python because it is my programming language of choice and because it is commonly used in astronomy, but if you don't have experience coding in Python I've tried to include a lot of skeleton code so you should still be able to make progress without getting too caught up with Python's syntax.
Also, if you ever find yourself writing a `for` loop then something is wrong; Python is an interpreted language and, as a result, looping is *very* slow.
This entire worksheet can be done without looping.

**Remember: The best reference for all things GPs is [Rasmussen & Williams](http://www.gaussianprocess.org/gpml/).**

*And also: Richard Turner's [lecture](https://www.youtube.com/watch?v=92-98SYOdlY) is also a very good start point (especially when you got stuck at the book like me).*

## Getting started

First let's set up this notebook by importing everything we'll need and choose some settings so that the plots look better:_____no_output_____

%matplotlib inline
%config IPython.matplotlib.backend = "retina"

# Start by importing the specific code for this project.
import gp

# And then the standard numerical Python modules.
import emcee
import numpy as np
import matplotlib.pyplot as plt

# Finally set up the plotting to look a little nicer.
from matplotlib import rcParams
rcParams["savefig.dpi"] = 100
rcParams["figure.dpi"] = 100

# Automatically reload library
%load_ext autoreload
%autoreload 2_____no_output_____

## 0 - Hello, Gaussian!

Before we "officially" enter the world of Gaussian process, let's first review some concepts related to Gaussian distribution in this section.

### 0.1 - One-Dimensional Gaussian Distribution

Gaussian distribution can be represented by a probability density function(PDF), which tells us, how **likely** we may observe a sample under this distribution. For a one-dimensional distribution, the PDF is
$$
\begin{aligned}
p(x | \mu, \sigma) & = f(x, \mu, \sigma) \\
                   & = \frac{1}{\sigma\sqrt{2\pi}} \exp\left[\frac{(x-\mu)^2}{2\sigma^2}\right]
\end{aligned}
$$
Here the **mean** $\mu$ indicates where the likelihood is maximum, and the **standard deviation** represents how "flat" the curve it.

Let's implement the PDF and see how things go: finish the `pdf_1D` function below, and set it as input to `setup_gaussian_distr_1D` function to visualize it._____no_output_____

def pdf_1D(x, mu, sig):
    """
    The univariate Gaussian likelihood of vector ``x`` given a mean
    ``mean`` and a standard deviation ``stddev``.

    :param x: ``(N,)``   Points to bw evaluated at.
    :param mu: ``float`` Mean of the univariate Gaussian likelihood function.
    :param sig: ``float`` Standard Deviation of the univariate Gaussian
                             likelihood function.

    :returns ll: ``float`` The Gaussian likelihood.

    """
    # Erase the following line and implement the one-dimensional
    # Gaussian likelihood function.
    pass_____no_output_____widget = gp.interactive.setup_gaussian_distr_1D(pdf_1D)
widget_____no_output_____

### 0.2 - Two-Dimensional Gaussian Distribution

Now let's think about a two-dimensional Gaussian distribution, the mean becomes a 2-vector, and the variable that describe the "flatness" is now a 2-by-2 matrix $\Sigma$, which we call **covariance matrix**. The PDF can be represented as:
$$
\begin{aligned}
p(x | \mu, \Sigma) & = f(x, \mu, \Sigma) \\
                   & = \sqrt{det(2\pi\Sigma)} \exp\left(-\frac{1}{2} (x-\mu)^\top \Sigma^{-1} (x-\mu)\right)
\end{aligned}
$$

The diagonal elements in covariance matrix describes the uncertainty of each element in the vector, but that's only part of the story -- the off-diagonal elements describes how different elements in the vector are correlated to each other.

Let's implement this PDF below in `pdf_2D`, and set it as input to `setup_gaussian_distr_2D` to visualize the likelihood distribution. The mean is fixed at the origin, and diagonal elements in covariance matrix are both fixed at 1.0, slide the `off_diagonal` value (note that covariance matrix is always symmetric), and see how the PDF changes._____no_output_____

def pdf_2D(x, mu, sig):
    """
    The Gaussian likelihood of vector ``x`` given a mean
    ``mean`` and a standard deviation ``stddev``.

    :param x: ``(2,)``   Point to be evaluated at.
    :param mu: ``(2,)`` Mean of the Gaussian likelihood function.
    :param sig: ``(2,2)`` Covariance matrix of the Gaussian likelihood function.

    :returns ll: ``float`` The Gaussian likelihood.

    """
    # Erase the following line and implement the two-dimensional
    # Gaussian likelihood function.
    pass_____no_output_____widget = gp.interactive.setup_gaussian_distr_2D(pdf_2D)
widget_____no_output_____

### 0.3 - Marginal, Conditional and Joint Distribution

To understand what it means by the "correlation between two elements", we first introduce the concepts of **marginal** and **conditional distribution**.

We begin by re-write the two-dimensional Gaussian distribution in a more detailed manner (and with a new shorthand):
$$
\mu = \begin{bmatrix}\mu_x \\ \mu_y\end{bmatrix}, \Sigma = \begin{bmatrix}A & C \\ C^\top & B\end{bmatrix}
$$

Then, for a 2-vector that's under this distribution, we can represent it using the shorthand:
$$
\begin{bmatrix}x \\ y\end{bmatrix} \sim \mathcal{N}\left(\begin{bmatrix}\mu_x \\ \mu_y\end{bmatrix}, \begin{bmatrix}A & C \\ C^\top & B\end{bmatrix}\right)
$$

**Marginal distribution** indicates a subset of the original distribution, and a nice property of Gaussian distribution is, *the marginal distribution of Gaussian distribution is still a Gaussian distribution*. For example, if I only care about the distribution of $x$, then the marginal distribution of $x$ can be easily represented by extracting the corresponding elements from original mean and covariance:
$$
x \sim \mathcal{N}\left(\mu_x, A\right)
$$

But knowing $y$ might help us to better understand $x$, so we're also interested in the distribution of $x$ given the value of $y$, we call it **conditional distribution**, which (not that surprisingly) is also a Gaussian distribution:
$$
x|y \sim \mathcal{N}\left( \mu_x+CB^{-1}(y-\mu_y), A - CB^{-1}C^\top \right)
$$
Note that here we use the matrix form, this is for the general formulation of higher dimensional varialbles, in our two-dimenional example, $A, B, C$ are scalars. The above equation for conditional distribution will be very important later for understanding Gaussian process.

Now, implement the equation for conditional distribution in the function `conditional_pdf` below, then visualize the result using `setup_gaussian_distr_2D_conditional`. Try different values for $y$ and off-diagonal elements, you will find that, when off-diagonal elements are positive, the mean of $x$'s conditional distribution with increase with $y$. This means $x$ is positively correlated with $y$ -- when $y$ increases, $x$ will *likely* increase, too! When off-diagonal elements are zero, moving $y$ doesn't influence $x$'s conditional distribution, this means knowing $y$ tells us nothing about $x$.

*(hint: the conditional distribution here is just a one-dimensional Gaussian distribution, so feel free to use the function you previosly implemented.)*_____no_output_____

def conditional_pdf(x, y, mu, sig):
    """
    The conditional likelihood of ``x`` given a condition of ``y``,
    mean ``mean`` and a standard deviation ``stddev``.

    :param x: ``(N,)``   Point to be evaluated at.
    :param y: ``(1,)``   Point to be conditioned at.
    :param mu: ``(2,)`` Mean of the Gaussian likelihood function.
    :param sig: ``(2,2)`` Covariance matrix of the Gaussian likelihood function.

    :returns ll: ``float`` The Gaussian likelihood.

    """
    # Erase the following line and implement the conditional distribution
    pass_____no_output_____widget = gp.interactive.setup_gaussian_distr_2D_conditional(pdf_2D, conditional_pdf)
widget_____no_output_____

At last, sometimes we have two independent Gaussian distributions, and we want to know the likelihood of obtaining two samples at the same time, this is the **joint distribution** of two *independent* Gaussian distributions, which (again!) remains a Gaussian (the assumption of independence is important here, otherwise things will be much more complicated). In this case, the joint distribution is simply the product of two Gaussians:
$$
\mathcal{N}(x|a,A) \mathcal{N}(x|b,B) = Z^{-1} \mathcal{N}\left( (A^{-1}+B^{-1})^{-1} (A^{-1}a+B^{-1}b), (A^{-1}+B^{-1})^{-1} \right)
$$
Here $Z$ is the normalizer.

Finally, with everything ready, let's start to think about the problem of regression._____no_output_____## 1 – Fitting a "transit" with "correlated noise"

In this section, we'll work through how to fit a simplified transit model to data in the presence of "correlated noise".
I think that it's important to clarify what I mean by "noise" here because it isn't necessarily what you immediately expect.
When I say noise, I'm referring to *everything that contributes to generating the data that isn't an exoplanet transit*.
In the context of transit fitting, this means that everything that the spacecraft is doing (pointing and temperature variations, etc.) and everything that the star is doing (rotation, asteroseismic oscillations, granulation, etc.) is "noise".
As you can imagine, in many cases this noise might actually be the thing that you're trying to measure and that's fine – I spend most of my time measuring noise!
Therefore, more formally, "noise model" refers to anything that we're modeling as **stochastic** and the "mean model" refers to anything that we're modeling as **deterministic**.
For the purposes of this tutorial, we're not going to interpret the stochastic component of the signal and, instead, focus on studying the mean model, but that is not a limitation of GP modeling in general.

### 1.1 – The "mean model"

In this tutorial, we'll be fitting a very simple "transit" model with three parameters (depth, duration, and transit time) to a simulated dataset.
This model is implemented in the `gp` module that was included with this notebook and here's an example of how it works:_____no_output_____

t = np.linspace(-5, 5, 5000)
model = gp.SillyTransitModel(log_depth=np.log(200.0), log_duration=np.log(0.5), time=0.0)
plt.plot(t, model.get_value(t))
plt.xlabel("time [days]")
plt.ylabel("flux [ppm]");_____no_output_____

This obviously isn't meant to be realistic, but it's a good enough toy problem for today.
This model has a few nice features that might come in handy later:_____no_output_____

print("parameter_dict:")
print(model.get_parameter_dict())

print("\nparameter_vector:")
print(model.get_parameter_vector())

print("\nyou can freeze and thaw parameters...")
print("the depth, for example, is now fixed to the current value:")
model.freeze_parameter("log_depth")
print(model.get_parameter_dict())

print("\nand 'thawed':")
model.thaw_parameter("log_depth")
print(model.get_parameter_dict())_____no_output_____

### 1.2 – Fitting the usual way

I've included some code to simulate a dataset with a "transit" given by our silly model.
Here's how you get it and plot it:_____no_output_____

t, y, yerr = gp.data.generate_dataset()
plt.errorbar(t, y, yerr=yerr, fmt=".k")
plt.xlabel("time [days]")
plt.ylabel("flux [ppm]");_____no_output_____

It should be clear that there's some white noise (the error bars show that part) but that something else is going on.
There should be some sort of "trend" on top of which the transit is superimposed.
This might have been caused by spots rotating into and out of view, motion of the spacecraft, or something else.
As I mentioned at the top, we won't worry too much about interpreting these trends.

Now, let's start by fitting this data the usual way: ignoring the correlated noise.
(I recognize that I'm being a bit unfair here and point out that most transit fitting results also fit out some parametric form for the trend, but this demo gets today's point across so bear with me...)

The usual likelihood function that we can derive assuming independent Gaussian noise is (*recall our one-dimensional Gaussian likelihood in Section 0*):

$$
\ln p(\mathrm{data}\,|\,\mathrm{model}) = -\frac{1}{2}\sum_{n=1}^N\left[\frac{[y_n - m(t_n)]^2}{{\sigma_n}^2} + \ln(2\,\pi\,{\sigma_n}^2)\right ] = -\frac{1}{2}\,\chi^2 + \mathrm{constant}
$$

where $m(t)$ is the transit model and we might maximize this (or, equivalently, minimize $\chi^2$) to find the "best-fit" parameters.
Instead, we'll apply improper uniform priors (*Disclaimer: don't think that I condone this behavior*) and sample the posterior probability density for the parameters of the model using [emcee](http://dan.iel.fm/emcee)._____no_output_____

def simple_log_prob(params, model, t, y, yerr):
    model.set_parameter_vector(params)
    resid = y - model.get_value(t)
    return -0.5 * np.sum((resid/yerr)**2)

ndim = 3
nwalkers = 16
pos = 1e-4*np.random.randn(nwalkers, ndim)
pos[:, 0] += 5
sampler = emcee.EnsembleSampler(nwalkers, ndim, simple_log_prob,
                                args=(model, t, y, yerr))
pos, _, _ = sampler.run_mcmc(pos, 200)
sampler.reset()
sampler.run_mcmc(pos, 2000);

print('true parameters:\n', gp.data.true_parameters)

gp.corner(sampler.flatchain, truths=gp.data.true_parameters,
          labels=["ln(depth)", "ln(duration)", "transit time"]);_____no_output_____

In this figure, you should see the posterior constraints on the parameters compared to the true values that were used to simulate the data shown as blue lines.
If everything went as planned, this should be your feeling about the result:

😕

This didn't work because we ignored the correlated noise! In the above likelihood function, we only consider the marginal likelihood of each observation, but the missing conditional likelihood, or the correlation between the observations plays an important role here.

How to fix this? Let's try to consider the whole covariance matrix instead of only diagonal elements. And this brings us to...

### 1.3 – Building a Gaussian process model

Now let’s implement the code for a Gaussian process.
The main thing that you need to do in this section is write the ln-likelihood function.
What you need to do is convert the following mathematical equation (recall the two-dimensional Gaussian likelihood in Section 0) to Python:

$$
\ln p(\mathrm{data}\,|\,\mathrm{model}) = \ln L = -\frac{1}{2}\,\boldsymbol{r}^\mathrm{T}\,K^{-1}\,\boldsymbol{r} - \frac{1}{2}\,\ln \det \boldsymbol{K} - C
$$

where $C = N\,\ln(2\,\pi)/2$ is an irrelevant (for our purposes) constant, $\boldsymbol{r}$ is the residual vector (the difference between the measured $y$ and the model predictions for $y$), and $K$ is the covariance matrix describing the measurement uncertainties and the correlations between observations.

In the following cell, you'll implement a Python function that takes as input a residual vector `r` and a covariance matrix `K` and returns the right-hand side of the above equation.
In pseudocode, the function will look something like:

**Input:** Residual vector $\boldsymbol{r}$; covariance matrix $K$

**Output:** The ln-likelihood (up to a constant)
1. $\boldsymbol{\alpha} = \mathtt{np.linalg.solve}({K},\,\boldsymbol{r})$
2. $(s,\,d) = \mathtt{np.linalg.slogdet}(\boldsymbol{K})$
3. **return** $-0.5\,\left(\boldsymbol{r}\cdot\boldsymbol{\alpha} + d\right)$

When implementing this function, the two numpy (remember that we imported `numpy` as `np` above) methods used in the pseudocode ([`np.linalg.solve`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.solve.html) and [`np.linalg.slogdet`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.slogdet.html)) will come in handy.
The `solve` method is useful because you should *never* invert a matrix (it is numerically unstable).
Instead, you should always use `solve` if you want to find $\boldsymbol{x}$ in the equation:

$$
\boldsymbol{x} = {K}^{-1}\,\boldsymbol{r} \to \boldsymbol{r} = {K}\,\boldsymbol{x} \quad.
$$

The `slogdet` function is useful because it computes the ln-determinant of a matrix using a numerically stable method.
Determinants of physical matrices tend to be *very* tiny and computing them naively can result in numerical underflow.
Sticking to the ln-determinant will make you less suceptible to this problem.

Finally, if you want to take the dot product of two vectors or matrices, you should use the [`np.dot`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html) function.

(*If you're already an expert, you can actually speed this code up substantially using the [cho_factor](https://docs.scipy.org/doc/scipy-0.19.0/reference/generated/scipy.linalg.cho_factor.html) and [cho_solve](https://docs.scipy.org/doc/scipy-0.19.0/reference/generated/scipy.linalg.cho_solve.html) functions from scipy and figuring out how to compute the determinant of a matrix using its [Cholesky factorization](https://en.wikipedia.org/wiki/Cholesky_decomposition). Follow the links to documentation for `cho_factor` to see what, exactly, it returns. You can extract the diagonal elements from a matrix using the [`np.diag`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.diag.html) function.
Other functions you may want to use include `np.sum` and `np.log`.*)

OK. Go for it! Bonus points if your implementation fits in a Tweet (don't forget to include the #KeplerSciCon hashtag)._____no_output_____

def log_like(r, K):
    """
    The multivariate Gaussian ln-likelihood (up to a constant) for the
    vector ``r`` given a covariance matrix ``K``.

    :param r: ``(N,)``   The residual vector with ``N`` points.
    :param K: ``(N, N)`` The square (``N x N``) covariance matrix.

    :returns lnlike: ``float`` The Gaussian ln-likelihood.

    """
    # Erase the following line and implement the Gaussian process
    # ln-likelihood here.
    pass_____no_output_____

When you're happy with your implementation, execute the following cell.
If your implementation is correct, you'll see a smiley face (☺) otherwise, it will throw an exception._____no_output_____

gp.utils.test_log_like(log_like)_____no_output_____

Did you see a happy face when you executed the above line?
Edit your `log_like` function until you do and then let's move on to implementing a specific model.

### 1.4 – The covariance matrix/kernel function

Now that we have the likelihood function implemented, we need to choose a specific model of the covariance matrix $K$.
Each element of the matrix $K_{ij}$, will be given by

$$
K_{ij} = k_{\boldsymbol{\alpha}}(x_i,\,x_j)
$$

where we get to choose the "kernel function" $k_{\boldsymbol{\alpha}}$ (parameterized by ${\boldsymbol{\alpha}}$).
For now, let's implement the *exponential-squared kernel*:

$$
k_{\boldsymbol{\alpha}}(r_{ij}) = a^2 \, \exp \left ( -\frac{{\Delta x_{ij}}^2}{2\,l^2} \right )
$$

where $\boldsymbol{\alpha} = (a,\,l)$ and $\Delta x_{ij} = |x_i - x_j|$.

You need to implement this function below and it needs to work with a `numpy` array `dx` as input (use the `np.exp` function for exponentiation).
Note that `alpha` will be a list with two elements: `alpha[0] = amp` and `alpha[1] = ell`; Python arrays are zero-indexed._____no_output_____

def expsq_kernel(alpha, dx):
    """
    The exponential-squared kernel function. The difference matrix
    can be an arbitrarily shaped numpy array so make sure that you
    use functions like ``numpy.exp`` for exponentiation.

    :param alpha: ``(2,)`` The parameter vector ``(amp, ell)``.
    :param dx: ``numpy.array`` The difference matrix. This can be
        a numpy array with arbitrary shape.

    :returns K: The kernel matrix (should be the same shape as the
        input ``dx``).

    """
    # Erase the following line and implement your kernel function
    # there.
    pass_____no_output_____

Once you're happy with your implementation of `expsq_kernel`, try executing the following cell to test your code.
Again, if you've implemented it correctly, you'll see a smiley face._____no_output_____

gp.utils.test_kernel(expsq_kernel)_____no_output_____

### 1.5 – Qualitative effects of kernel (hyper)parameters

Now that we've implemented our kernel function, let's try to get a qualitative sense of what the hyperparameters do.

**Note:** *another good resource for trying this out is [gp.js](http://dan.iel.fm/gp.js/)*.

To start, we'll generate samples from the likelihood function that we implemented.
Remember that a likelihood is a probability distribution over *data* so, conditioned on specific hyperparameters, we can sample synthetic datasets from the likelihood.
For a Gaussian process, this probability function is a multivariate Gaussian where the covariance matrix is given by our kernel function.

In the following cell, we're using some IPython magic to create an interactive figure that displays 6 samples $\boldsymbol{y}_k$ from the likelihood for variable values of the hyperparameters.
Try executing the cell and then moving the sliders to get a feel for how the samples change with $a$ (`amp`) and $l$ (`ell`)._____no_output_____

gp.interactive.setup_likelihood_sampler(expsq_kernel)_____no_output_____

The conditional distribution is even more interesting.
That is, given the data that you have observed and a specific set of parameters, what is your prediction at other points $\boldsymbol{x}_\star$?
The math can be found in Chapter 2 of [R&W](http://www.gaussianprocess.org/gpml/), you can just execute the following cell and play around with the parameters to see how the predictions change with all the parameter choices.

In the figure, the "noiseless" transit function is shown as a dotted red line and the 6 samples from the conditional distribution are shown as black lines.
The ln-likelihood is shown in the title of the figure.
We'll use the results of your experiments to initialize the MCMC sampler in a bit so try to get a good fit._____no_output_____The conditional distribution is even more interesting. That is, given the data that you have observed, what is your prediction at other points? Let's denote the observations as vector $f\in\mathbb{R}^n$, and the points to be predicted as $f_*\in\mathbb{R}^{n_*}$. Combining the two vectors into a single one, and assume they're under the same Gaussian distribution (here we simply assume mean to be zero):
$$
\begin{bmatrix}f\\f_*\end{bmatrix} \sim \mathcal{N}\left( 0, \begin{bmatrix} K(X,X) & K(X,X_*) \\ K(X_*,X) & K(X_*,X_*) \end{bmatrix} \right)
$$

Here $K(X,X)\in\mathbb{R}^{n\times n}, K(X,X_*)\in\mathbb{R}^{n\times n_*}, K(X,X_*)=K(X,X_*)^\top, K(X_*,X_*)\in\mathbb{R}^{n_*\times n_*}$ are similar to $A, B, C$ in the two-dimensional Gaussian likelihood in Section 0, the difference is here we just wrote a kernel to describe this larger covariance matrix.

Now, to perform regression/prediction, we're interested in the distribution of $f_*$, conditioned on the observations $f$, following the equation in Section 0, we have:
$$
\begin{aligned}
f_* | f \sim \mathcal{N}\left(K(X_*,X) [K(X,X)+\sigma_n^2 I]^{-1} f, \\
\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad K(X_*,X_*)-K(X,X_*)^\top[K(X,X)+\sigma_n^2 I]^{-1}K(X,X_*)\right)
\end{aligned}
$$

The only difference here is $K(X,X)^{-1}$ is replaced by $[K(X,X)+\sigma_n^2 I]^{-1}$, this is because here we assume our observations are noisy, and $\sigma_n$ is the standard deviation for the independent observation noise (which is captured by the errorbar in the plot). If we take observation noise as zero, then the equation above is exactly the same as Section 0.

A reference pseudocode is shown below, and again, for the purpose of numerical stability, here we will use Cholesky factorization. The full algorithm for Gaussian process regression can be found in Chapter 2 of [R&W](http://www.gaussianprocess.org/gpml/), the only missing part here is the construction of kernel, but given the kernel function this should be straightforward. More details about the implementation can found at `gp/interactive.py`.

**Input:** observation $y$, covariance $K(X,X)$, covariance $K(X,X_*)$, covariance $K(X_*,X_*)$

**Output:** mean $\mu_*$ and covariance $\Sigma_*$ for the distribution of prediction, log marginal likelihood of observations $f$ under the predicted distribution
1. $L$ = scipy.linalg.cho_factor(K(X,X))
2. $\alpha$ = scipy.linalg.cho_solve(L, y)
3. $\mu_*$ = $K(x,x_*) \cdot \alpha$
4. $\Sigma_*$ = $K(x_*,x_*) - K(x,x_*) \cdot$ np.linalg.solve(K(x,x)$\cdot K(x,x_*)^\top$)
5. $ll$ = -0.5 $f^\top \alpha - \sum_i
\log(L_{ii})$ - $\frac{n}{2} \log(2\pi)$
6. **return: $\mu, \Sigma, ll$**

Now, implement the function `kernel_training_zero_mean` and feed it to `setup_conditional_sampler_zero_mean` to generate six sample predictions from the resulting Gaussian distribution. Tune the kernel parameters and see how they change the sample, and try to find a set of parameters that optimizes the log marginal likelihood._____no_output_____

def kernel_training_zero_mean(y, Kxx, Kxs, Kss):
    """
    Train GP kernel given a set of observations (assuming zero mean)

    :param y, : ``(n,)`` vector of observations.
    :param Kxx: ``(n,n)`` covariance matrix K(x,x)
    :param Kxs: ``(n,n*)`` covariance matrix K(x,x*)
    :param Kss: ``(n*,n*)`` covariance matrix K(x*,x*)

    :returns mu: mean of the prediction/conditional distributions
    :returns cov: covariance of the prediction/conditional distribution
    :returns ll: log likelihood of the given observations y under the
                    prediction/conditional distribution
    """
    from scipy.linalg import cho_factor, cho_solve
    # Erase the following line and implement your function for computing
    # the prediction distribution based on given observations and kernel
    pass_____no_output_____widget = gp.interactive.setup_conditional_sampler_zero_mean(t, y, yerr, expsq_kernel, kernel_training_zero_mean)
widget_____no_output_____

Sometimes, we may have some prior knowledge about the mean function. In this case, it might be better to indicate a certain mean instead of using the zero-mean function. Modify the kernel training function you just finished with a given non-zero mean function. Again, try to find a set of parameters that optimizes marginal likelihood.

*(hint: only one tiny modification is needed!)*_____no_output_____

def kernel_training_nonzero_mean(y, model_y, model_ys, Kxx, Kxs, Kss):
    """
    Train GP kernel given a set of observations (with non-zero mean)

    :param y, : ``(n,)`` vector of observations.
    :param model_y, : ``(n,)`` evaluation of non-zero mean function at
                                observation points
    :param model_ys, : ``(n*,)`` evaluation of non-zero mean function at
                                predictioin points
    :param Kxx: ``(n,n)`` covariance matrix K(x,x)
    :param Kxs: ``(n,n*)`` covariance matrix K(x,x*)
    :param Kss: ``(n*,n*)`` covariance matrix K(x*,x*)

    :returns mu: mean of the prediction/conditional distributions
    :returns cov: covariance of the prediction/conditional distribution
    :returns ll: log likelihood of the given observations y under the
                    prediction/conditional distribution
    """
    # Erase the following line and implement your function for computing
    # the prediction distribution based on given observations and kernel
    pass_____no_output_____widget = gp.interactive.setup_conditional_sampler_nonzero_mean(t, y, yerr, expsq_kernel, kernel_training_nonzero_mean)
widget_____no_output_____

Once you've found an initial guess that you're happy with, let's run some MCMC and see what our results look like. In this fit, we're going to sample for the parameters of the transit and the parameters of the GP (these ones are often called "hyperparameters") simultaneously. This means that any constraints that we get on the transit parameters will have the uncertainty in the noise model taken into account. Be warned, this MCMC will take a little longer to run than the last one because the GP model is a little bit more expensive to compute (we'll come back to this later)._____no_output_____

def gp_log_prob(params, log_like_fn, kernel_fn, mean_model, t, y, yerr):
    if np.any(params < -10.) or np.any(params > 10.):
        return -np.inf

    k = mean_model.vector_size

    # Compute the covariance matrix
    K = kernel_fn(np.exp(params[k:]), t[:, None] - t[None, :])
    K[np.diag_indices_from(K)] += yerr**2

    # Compute the residual vector
    mean_model.set_parameter_vector(params[:k])
    resid = y - model.get_value(t)

    # Compute the log likelihood
    return log_like_fn(resid, K)

ndim = 5
nwalkers = 16
pos = [np.log(np.abs(widget.kwargs[k])) for k in ["depth", "duration", "amp", "ell"]]
pos = np.array(pos[:2] + [widget.kwargs["time"]] + pos[2:])
pos = pos + 1e-4*np.random.randn(nwalkers, ndim)
sampler = emcee.EnsembleSampler(nwalkers, ndim, gp_log_prob,
                                args=(log_like, expsq_kernel, model, t, y, yerr))
pos, _, _ = sampler.run_mcmc(pos, 200)
sampler.reset()
sampler.run_mcmc(pos, 2000);

gp.corner(sampler.flatchain, truths=np.append(gp.data.true_parameters, [None, None]),
          labels=["ln(depth)", "ln(duration)", "transit time", "ln(amp)", "ln(ell)"])_____no_output_____

If everything went as planned, you should have found a much more acceptable constraint on the parameters.
You'll notice that the GP parameter panels don't have blue lines on them.
This is because we don't know the "true" noise model; we're only using the GP as an *effective* model.

## 2 - Optimize Hyperparameters For Maximal Likelihood

Until now, the way we found a "good" set of (hyper)parameters is through manually trying, an alternative is to optimize over the paramters. In order to do so, there must be an objective function, and one choice for the objective the marginal likelihood of observations under the predicted GP posterior distribution, and in this section, we will implement this.

Since, _____no_output_____

def gp_log_prob(params, log_like_fn, kernel_fn, mean_model, t, y, yerr):
    if np.any(params < -10.) or np.any(params > 10.):
        return -np.inf

    k = mean_model.vector_size

    # Compute the covariance matrix
    K = kernel_fn(np.exp(params[k:]), t[:, None] - t[None, :])
    K[np.diag_indices_from(K)] += yerr**2

    # Compute the residual vector
    mean_model.set_parameter_vector(params[:k])
    resid = y - model.get_value(t)

    # Compute the log likelihood
    return log_like_fn(resid, K)

from scipy.optimize import minimize

# this is the intial guess from the sliders above
pos = [np.log(np.abs(widget.kwargs[k])) for k in ["depth", "duration", "amp", "ell"]]
pos = np.array(pos[:2] + [widget.kwargs["time"]] + pos[2:])

# since we use `minimize`, we need to use the negative likelihood as objective
objective = lambda pos: -gp_log_prob(pos, log_like, expsq_kernel, model, t, y, yerr)

# minimize the objective / maximize likelihood
result = minimize(objective, pos)
print('maximal likelihood: ', gp_log_prob(result.x, log_like, expsq_kernel, model, t, y, yerr))
print('depth: ', np.exp(result.x[0]))
print('duration: ', np.exp(result.x[1]))
print('time: ', result.x[2])
print('amp: ', np.exp(result.x[3]))
print('ell: ', np.exp(result.x[4]))

# visualize
gp.interactive.setup_conditional_sampler_nonzero_mean_given_param(t, y, yerr, expsq_kernel, kernel_training_nonzero_mean,
                                                                    result.x[0], result.x[1], result.x[2], result.x[3], result.x[4])_____no_output_____

## 3 – Using GPs in Python

In the previous section, we developed a fully-functional GP code that you could now use for research, but it might be better to use an existing software package that makes it easier to try out different models and some other niceties.
There are packages in every programming language but, if you use Python as your programming language of choice, there are a few different software packages that you can use.
Here are a few options:

1. [george](http://dan.iel.fm/george) – Written (by this tutorial's original author!) with astronomy in mind. Good for general purpose GP modeling.
2. [scikit-learn](http://scikit-learn.org/stable/modules/gaussian_process.html) – Well supported package with a big community. Can be used for astronomy projects with some work.
3. [GPy](https://github.com/SheffieldML/GPy) – Very flexible with state-of-the-art inference algorithms designed by machine learning researchers.
4. [PyMC](https://pymc-devs.github.io/pymc3/notebooks/GP-introduction.html) – Implementation in the context of Hamiltonian MCMC sampling.
5. [Stan](http://mc-stan.org/) – Like PyMC, a Hamiltonian MCMC sampler with GP support.
6. [celerite](http://celerite.readthedocs.io/) – Scalable GP likelihood calculations for 1D datasets (e.g. time series).
7. etc. – I'm sure that there are others that I've missed!

## 4 – Choosing a kernel

Earlier in this tutorial, we chose to use the exponential-squared kernel function for the experiements.
This was a good idea because it is *exactly* the kernel that I used to generate the data.
That being said, in The Real World™, we never know what kernel to use and there are some other commonly used functions.
One good simple example is the Matern-3/2 function

$$
k_{\boldsymbol{\alpha}}(r_{ij}) = a^2 \, \left[1+\frac{\sqrt{3}\,|\Delta x_{ij}|}{l}\right]\, \exp \left ( -\frac{\sqrt{3}\,|\Delta x_{ij}|}{l} \right )
$$

Try implementing this function (or another one of your choice; see [Chapter 4](http://www.gaussianprocess.org/gpml/chapters/RW4.pdf) of [R&W](http://www.gaussianprocess.org/gpml/) for examples) and re-run the analysis with this kernel.
There is no included unit test for this kernel (since this is an optional extension) bu‘t you might consider making some plots or writing your own tests to make sure that the function returns sensible values._____no_output_____

def matern32_kernel(alpha, dx):
    """
    The Mater-3/2 kernel function. The difference matrix
    can be an arbitrarily shaped numpy array so make sure that you
    use functions like ``numpy.exp`` for exponentiation.

    :param alpha: ``(2,)`` The parameter vector ``(amp, ell)``.
    :param dx: ``numpy.array`` The difference matrix. This can be
        a numpy array with arbitrary shape.

    :returns K: The kernel matrix (should be the same shape as the
        input ``dx``).

    """
    # Erase the following line and implement your kernel function
    # there.
    pass_____no_output_____

Take a look at the conditional distribution given by this new choice of kernel:_____no_output_____

widget = gp.interactive.setup_conditional_sampler_nonzero_mean(t, y, yerr, matern32_kernel, kernel_training_nonzero_mean)
widget_____no_output_____

Try running the MCMC sampling with this kernel:_____no_output_____

pos = [np.log(np.abs(widget.kwargs[k])) for k in ["depth", "duration", "amp", "ell"]]
pos = np.array(pos[:2] + [widget.kwargs["time"]] + pos[2:])
pos = pos + 1e-4*np.random.randn(nwalkers, ndim)
sampler = emcee.EnsembleSampler(nwalkers, ndim, gp_log_prob,
                                args=(log_like, matern32_kernel, model, t, y, yerr))
pos, _, _ = sampler.run_mcmc(pos, 200)
sampler.reset()
sampler.run_mcmc(pos, 2000);

gp.corner(sampler.flatchain, truths=np.append(gp.data.true_parameters, [None, None]),
          labels=["ln(depth)", "ln(duration)", "transit time", "ln(amp)", "ln(ell)"])_____no_output_____

Depending on the specific choices that you made, this may or may not look awesome.
If not, try to debug the code (and make sure that you choose a good initial guess) until it does or, otherwise, come up with an argument for what is happening.

As a general guideline, the choice of kernel often won't significantly affect your inferences about the parameters of the mean model.
That being said, if you don't have a physical motivation for your kernel choice, it's useful to try some sort of formal model comparison.
There is nothing special about kernel selection compared to other model selection problems so the usual techniques ([Bayes factors](https://en.wikipedia.org/wiki/Bayes_factor), [cross-validation](https://en.wikipedia.org/wiki/Cross-validation_%28statistics%29), [information criteria](https://en.wikipedia.org/wiki/Bayesian_information_criterion), etc.) are all applicable._____no_output_____## 5 – Advanced: Outliers

If you've ever tried to fit a model to real data, I'm sure that you've run into issues with outliers.
In the context of white noise, there are [rigorous things that you can do](http://dan.iel.fm/posts/mixture-models/), but it gets a bit trickier with GPs.
A detailed discussion of why mixture models are tricky to implement with GPs is beyond the scope of this tutorial, but the qualitative reason is that every data point "cares" about the identity – outlier or otherwise – of every other data point.
There has been some work on alternatives to GPs that have heavier tails (e.g. [Student-*t* processes](https://arxiv.org/abs/1402.4306)), but the easiest (and most common) place to start is just our old friend sigma clipping.

To check this out, let's simulate a dataset where 10% of the points are outliers:_____no_output_____

t_out, y_out, yerr_out = gp.data.generate_dataset(outliers=75.0)
plt.errorbar(t_out, y_out, yerr=yerr_out, fmt=".k")
plt.xlabel("time [days]")
plt.ylabel("flux [ppm]");_____no_output_____

As before, let's play around with this to see how the predictions change with these outliers.
In particular, look at how the likelihood changes as you go to very short scale length `ell`._____no_output_____

kernel = expsq_kernel
widget = gp.interactive.setup_conditional_sampler_nonzero_mean(t_out, y_out, yerr_out, kernel, kernel_training_nonzero_mean)
widget_____no_output_____

Starting from here, let's run MCMC without dealing with the outliers._____no_output_____

pos = [np.log(np.abs(widget.kwargs[k])) for k in ["depth", "duration", "amp", "ell"]]
pos = np.array(pos[:2] + [widget.kwargs["time"]] + pos[2:])
pos = pos + 1e-4*np.random.randn(nwalkers, ndim)
sampler = emcee.EnsembleSampler(nwalkers, ndim, gp_log_prob,
                                args=(log_like, kernel, model, t_out, y_out, yerr_out))
pos, _, _ = sampler.run_mcmc(pos, 200)
sampler.reset()
sampler.run_mcmc(pos, 2000);

gp.corner(sampler.flatchain, truths=np.append(gp.data.true_parameters, [None, None]),
          labels=["ln(depth)", "ln(duration)", "transit time", "ln(amp)", "ln(ell)"])_____no_output_____

How does this look? Like garbage? This time, it's probably not a bug!

To deal with this, you will want to implement a sigma clipping procedure.
I'm not going to provide code for this, but the basic procedure is:

1. Fit for the maximum likelihood (or posteriori) parameters using the un-clipped data.
2. Implement Equations 2.22-2.24 from [R&W](http://www.gaussianprocess.org/gpml/chapters/) (or use an existing GP package) to compute the model prediction for the maximum likelihood parameters.
3. Clip data points that are more than $K$-sigma away from the prediction (if you can work it through, it's best if you can let datapoints re-enter the fit).
4. Return to Step 1 and repeat to convergence.

This procedure should converge quickly and reliably to a dataset with fewer outliers.
Some skeleton code is provided here, but it might be easier to use one of the Python packages mentioned above to implement this procedure._____no_output_____

from scipy.optimize import minimize

# Initial guess
pos = [np.log(np.abs(widget.kwargs[k])) for k in ["depth", "duration", "amp", "ell"]]
pos = np.array(pos[:2] + [widget.kwargs["time"]] + pos[2:])

# Remember that you want to *minimuze* the function
neg_log_prob = lambda *args: -gp_log_prob(*args)

# Implement your sigma clipping procedure here...

# This should be provided to the "args" argument in the minimize function
# In each loop of clipping, update "not_clipped"
not_clipped = np.ones(len(t_out), dtype=bool)
args = (log_like, kernel, model, t_out[not_clipped], y_out[not_clipped], yerr_out[not_clipped])_____no_output_____

After you are satisfied with your procedure, try running the MCMC again and see if you get more reasonable results._____no_output_____## 6 – Advanced: Computational cost and scaling of GPs

One of the biggest problems that you'll run into when you use GPs in practice is the computational efficiency and scaling.
To compute the likelihood of a GP model, you must invert (or, really, factorize) an $N \times N$ matrix where $N$ is the number of datapoints.
In general, the cost of this calcuation scales as the cube(!!!) of the size of the matrix.
In other words, the runtime is proportional to $N^3$.
This means that applying GP modeling to large datasets will be computationally intractable.

To see this in practice, let’s take a moment to benchmark our code and see it scales with the size of the dataset.
To test your implementation, run the following (this will take a minute or two to finish):_____no_output_____

params = sampler.flatchain[-1]
t_bench, y_bench, yerr_bench = gp.data.generate_dataset(N=2**10)
Ns = 2 ** np.arange(5, 11)
times = np.empty(len(Ns))
for i, N in enumerate(Ns):
    result = %timeit -qo gp_log_prob(params, log_like, expsq_kernel, model, t_bench[:N], y_bench[:N], yerr_bench[:N])
    times[i] = result.best_____no_output_____

Now let's plot this and compare it to the theoretical scaling:_____no_output_____

plt.plot(Ns, times, ".-k", ms=9, label="GP")
plt.plot(Ns, 0.5 * times[-1] * (Ns / Ns[-1])**3, label="N cubed")
plt.xscale("log")
plt.yscale("log")
plt.xlim(Ns.min(), Ns.max())
plt.ylim(0.8 * times.min(), 1.2 * times.max())
plt.legend();_____no_output_____

The theoretical cost of the computation of the ln-likelihood should be $\mathcal{O}(N^3)$ where $N$ is the number of datapoints.

**Questions:**
Is this what you get?
Why or why not?
How much time would it take to compute a model with 60,000-70,000 data points (not to mention the memory requirements!)?

There is a discussion of methods for scaling GPs to larger datasets in [Chapter 8 of R&W](http://www.gaussianprocess.org/gpml/chapters/).
The easiest place to start is by just sub-sampling your data.
There are also a few existing software packages that can be used (e.g. [this](http://dan.iel.fm/george/current/user/solvers/#hodlr-solver) and [this](http://celerite.readthedocs.io)).

_____no_output_____


# Notebook from sanchobarriga/course-content
Path: tutorials/W1D3_ModelFitting/student/W1D3_Tutorial5.ipynb

_____no_output_____
# Neuromatch Academy: Week 1, Day 3, Tutorial 5
# Model Fitting: Polynomial regression
_____no_output_____#Tutorial Objectives

This is Tutorial 5 of a series on fitting models to data. We start with simple linear regression, using least squares optimization (Tutorial 1) and Maximum Likelihood Estimation (Tutorial 2). We will use bootstrapping to build confidence intervals around the inferred linear model parameters (Tutorial 3). We'll finish our exploration of linear models by  generalizing to multiple linear regression (Tutorial 4). We then move on to polynomial regression (Tutorial 5). We end by learning how to choose between these various models. We discuss the bias-variance trade-off (Tutorial 6) and two common methods for model selection, AIC and Cross Validation (Tutorial 7).

In this tutorial, we will learn about a new model, polynomial regression, which is an extension of linear regression.

*   Learn how to structure our inputs for polynomial regression
*   Fit polynomial regression models of different complexity
* Plot and evaluate the polynomial regression fits

_____no_output_____

#@title Video Polynomial Regression
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="fGPSAjF2plo", width=854, height=480, fs=1)
print("Video available at https://youtu.be/fGPSAjF2plo" + video.id)
video
Video available at https://youtu.be/fGPSAjF2plofGPSAjF2plo

# Setup_____no_output_____

#@title  Imports
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import KFold
%matplotlib inline_____no_output_____#@title Figure Settings
%matplotlib inline
fig_w, fig_h = (8, 6)
plt.rcParams.update({'figure.figsize': (fig_w, fig_h)})
%config InlineBackend.figure_format = 'retina'_____no_output_____#@title Helper functions
def ordinary_least_squares(x, y):
  """Ordinary least squares estimator for linear regression.

  Args:
    x (ndarray): design matrix of shape (n_samples, n_regressors)
    y (ndarray): vecto"r of measurements of shape (n_samples)

  Returns:
    ndarray: estimated parameter values of shape (n_regressors)
  """
  return np.linalg.inv(x.T @ x) @ x.T @ y
_____no_output_____

# Polynomial Regression_____no_output_____So far today, you learned how to predict outputs from inputs by fitting a linear regression model. We can now model all sort of relationships, including in neuroscience! For example, you can fit a linear regression model to predict neural activity from the visual stimulus the mouse is viewing (neural encoding).

One potential problem with this approach is the simplicity of the model. Linear regression, as the name implies, can only capture a linear relationship between the inputs and outputs. Put another way, the predicted outputs are only a weighted sum of the inputs. What if there are more complicated computations happening? Luckily, many more complex models exist (and you will encounter many more over the next 3 weeks). One model that is still very simple to fit and understand, but captures more complex relationships, is **polynomial regression**, an extension of linear regression.

Since polynomial regression is an extension of linear regression, everything you learned so far will come in handy now! The goal is the same: we want to predict the dependent variable $y_{n}$ given the input values $x_{n}$. The key change is the type of relationship between inputs and outputs that the model can capture.

Linear regression models predict the outputs as a weighted sum of the inputs:

$$y_{n}= \theta_0 + \theta x_{n} + \epsilon_{n}$$

With polynomial regression, we model the outputs as a polynomial equation based on the inputs. In particular, we model the outputs as:

$$y_{n}= \theta_0 + \theta_1 x_{n} + \theta_2 x_{n}^2 + \theta_3 x_{n}^3 + \epsilon_{n}$$

We can change how complex a polynomial is fit by changing the order of the polynomial. The order of a polynomial refers to the highest power in the polynomial. The equation above is a third order polynomial because the highest value x is raised to is 3. We could add another term ($+ \theta_4 x_{n}^4$) to model an order 4 polynomial and so on.

_____no_output_____## Data Generation_____no_output_____First, we will simulate some data to practice fitting polynomial regression models. We will generate noisy samples $x$ and then compute y according to $y = x^2 - x - 2 $, with some extra noise to make the model fitting harder._____no_output_____

# setting a fixed seed to our random number generator ensures we will always
# get the same psuedorandom number sequence
np.random.seed(0)
n_samples = 50
x = np.random.uniform(-2, 2.5, n_samples) # sample from a uniform distribution over [-2, 2.5)
noise = np.random.randn(n_samples) # sample from a standard normal distribution
y =  x**2 - x - 2 + noise

fig, ax = plt.subplots()
ax.scatter(x, y)  # produces a scatter plot
ax.set(xlabel='x', ylabel='y'); _____no_output_____

## Design matrix for polynomial regression

Now we have the basic idea of polynomial regression and some noisy data, let's begin! The key difference between fitting a linear regression model and a polynomial regression model lies in how we structure the input variables.

For linear regression, we used $X = x$ as the input data. To add a constant bias (a y-intercept in a 2-D plot), we use $X = \big[ \boldsymbol 1, x \big]$, where $\boldsymbol 1$ is a column of ones.  When fitting, we learn a weight for each column of this matrix. So we learn a weight that multiples with column 1 - in this case that column is all ones so we gain the bias parameter ($+ \theta_0$). We also learn a weight for every column, or every feature of x, as we learned in the **Multiple Linear Regression** section in Part 1.

This matrix $X$ that we use for our inputs is known as a **design matrix**. We want to create our design matrix so we learn weights for $x^2, x^3,$ etc. Thus, we want to build our design matrix $X$ for polynomial regression of order $k$ as:

$$X = \big[ \boldsymbol 1 , x^1, x^2 , \ldots , x^k \big],$$

where $\boldsymbol{1}$ is the vector the same length as $x$ consisting of of all ones, and $x^p$ is the vector or matrix $x$ with all elements raised to the power $p$. Note that $\boldsymbol{1} = x^0$ and $x^1 = x$ _____no_output_____### Exercise: Structure design matrix

Create a function (`make_design_matrix`) that structures the design matrix given the input data and the order of the polynomial you wish to fit_____no_output_____

def make_design_matrix(x, order):
  """Create the design matrix of inputs for use in polynomial regression

  Args:
    x (ndarray): An array of shape (samples,) that contains the input values.
    order (scalar): The order of the polynomial we want to fit

  Returns:
    numpy array: The design matrix containing x raised to different powers
  """
  #################################################
  ## TODO for students: create the design matrix ##
  #################################################
    # Broadcast to shape (n x 1) if shape (n, ) so this function generalizes to multiple inputs
  if x.ndim == 1:
    x = x[:,None]

  #if x has more than one feature, we don't want multiple columns of ones so we assign
  # x^0 here
  design_matrix = np.ones((x.shape[0],1))

  # Loop through rest of degrees and stack columns (hint: np.hstack)
  #for degree in range(1, order+1):
      #design_matrix = YOUR CODE HERE

  #comment this out when you've filled the function
  raise NotImplementedError("Student excercise: create the design matrix")
  return design_matrix

# Uncomment once the make_design_matrix function is complete
max_order = 5
# X_design = make_design_matrix(x, max_order)_____no_output_____

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial5_Solution_74a1698e.py)

_____no_output_____## Fitting polynomial regression models_____no_output_____Now that we have the inputs structured correctly in our design matrix, fitting a polynomial regression is the same as fitting a linear regression model! All of the polynomial structure we need to learn is contained in how the inputs are structured in the design matrix. We can use the same least squares solution we computed in Tutorials 1 and 4. _____no_output_____### Exercise: Fitting polynomial regression models with different orders _____no_output_____Here, we will fit polynomial regression models to find the regression coefficients ($\theta_0, \theta_1, \theta_2,$ ...) by solving the least squares problem. Create a function `solve_poly_reg` that loops over different order polynomials (up to `max_order`), fits that model, and saves out the weights for each. You may invoke the `ordinary_least_squares` function from Tutorial 4. _____no_output_____

def solve_poly_reg(x, y, max_order):
  """Fit a polynomial regression model for each order 0 through max_order.

  Args:
    x (ndarray): An array of shape (samples, ) that contains the input values
    y (ndarray): An array of shape (samples, ) that contains the output values
    max_order (scalar): The order of the polynomial we want to fit

  Returns:
    dict: Each entry has the order as the key and the values are that models fitted weights
  """

  # Create a dictionary with polynomial order as keys, and np array of theta
  # (weights) as the values
  theta_hat = {}

  # Loop over polynomial orders from 0 through max_order
  for order in range(max_order+1):

    ##################################################################################
    ## TODO for students: Create design matrix and fit polynomial model for this order ##
    ##################################################################################
    #X = YOUR CODE HERE
    #this_theta = YOUR CODE HERE
    theta_hat[order] = this_theta

    #uncomment when you've filled in the function
    raise NotImplementedError("Student exercise: fit a polynomial model")

  return theta_hat

# Uncomment once the solve_poly_reg function is complete
max_order = 5
# theta_hat = solve_poly_reg(x, y, max_order)
_____no_output_____

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial5_Solution_a93ba152.py)

_____no_output_____## Visualizing polynomial fits

We've fit 5 different polynomials (orders 0-5) to our data. Let's qualitatively inspect the quality of our fits by plotting the fitted polynomials on top of the data. _____no_output_____### Exercise: Plot polynomial fits

In order to see smooth curves, you will have to evaluate the fitted polynomials on a grid of $x$ values (you can have them range between the largest and smallest of the inputs present in the dataset)._____no_output_____

################################################################################
## TODO for students:
# Get a range of x values between the smallest and largest values of the input
################################################################################

max_order = 5
plt.figure()

for order in range(0, max_order+1):
  ################################################################################
  ## TODO for students: For each order polynomial, create the design matrix and
  ## compute the predicted y values given the previous fits. Plot each result
  ################################################################################
    #X_design =
    #plt.plot(x_vec, np.dot(..., ....));

  pass

plt.ylabel('y')
plt.xlabel('x')
plt.plot(x, y, 'C0.');
plt.legend([f'order {o}' for o in range(max_order)], loc=1)
plt.title('polynomial fits');_____no_output_____

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial5_Solution_e8db9e7e.py)

*Example output:*

_____no_output_____## Evaluating fit quality

As with linear regression, we can compute mean squared error (MSE) to get a sense of how well the model fits the data.

We compute MSE as:

$$ MSE = \frac 1 N ||y - \hat y||^2 = \sum_{i=1}^N (y_i - \hat y_i)^2 $$

where the predicted values for each model are given by $ \hat y = X \hat \theta$.

*Which model (i.e. which polynomial order) do you think will have the best MSE?*_____no_output_____### Exercise: Compute MSE and compare models

We will compare the MSE for different polynomial orders with a bar plot._____no_output_____

################################################################################
## TODO for students: Loop over polynomial orders and:
## 0) Get the design matrix
## 1) Get predictions as you did in previous exercise
## 2) Compute MSE
## Should output mse, a numpy array of size(max_order+1,)) where each entry is the
## MSE for that order polynomial model
################################################################################""
##uncomment below and fill in with your code

mse = np.zeros((max_order+1))
#for order in range(0, max_order+1):
  #X_design =
  #y_hat =
  #residuals =
  #mse[order] =

# Uncomment once above exercise is complete
# with plt.xkcd():
#   plt.bar(range(max_order+1), mse);
#   plt.ylabel('MSE')
#   plt.xlabel('polynomial order')
#   plt.title('comparing polynomial fits');
_____no_output_____

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial5_Solution_341595ca.py)

*Example output:*

_____no_output_____# Summary

Polynomial regression is an extension of linear regression that can capture more complex patterns in the data.

* To change from a linear regression model to a polynomial regression model, we only have to change how the input data is structured

* We can choose the complexity of the model by changing the order of the polynomial model fit

* Higher order polynomial models tend to have lower MSE on the data they're fit with_____no_output_____