# Notebook from syj19950301/cs207_yijun_shen
Path: lectures/L13/L13.ipynb

# Lecture 13

### Monday, October 23rd 2017_____no_output_____## Last time:
* Data structures motivation
* Abstract data types
* Sequences
* Linked lists

## This time:
* Iterators and Iterables
* Trees, B-trees, and BSTs_____no_output_____# From pointers to iterators

One can simply follow the `next` pointers to the next **position** in a linked list.

This suggests an abstraction of the **position** to an **iterator**.

Such an abstraction allows us to treat arrays and linked lists with an identical interface.

The salient points of this abstraction are:
- The notion of a `next`
- The notion of a `first` to a `last`

We already implemented the sequence protocol.

Now we suggest an additional abstraction that is more fundamental than the notion of a sequence: the **iterable**._____no_output_____# Iterators and Iterables in `Python`

Just as a sequence is something implementing `__getitem__` and `__len__`, an **iterable** is something implementing `__iter__`.

`__len__` is not needed and indeed may not make sense.
```python
len(open('fname.txt')) # File iterator has no length
```

Example `14-1` in `Fluent Python`: The `Sentence` sequence and shows how it can be iterated upon._____no_output_____

import reprlib
class Sentence:
    def __init__(self, text):
        self.text = text
        self.words = text.split()

    def __getitem__(self, index):
        return self.words[index]

    def __len__(self):
        #completes sequence protocol, but not needed for iterable
        return len(self.words)

    def __repr__(self):
        return 'Sentence(%s)' % reprlib.repr(self.text)_____no_output_____# Sequence'
s = Sentence("Dogs will save the world.")
print(len(s), "   ", s[3], "   ",  s)5     the     Sentence('Dogs will save the world.')
min(s), max(s)_____no_output_____list(s)_____no_output_____

To iterate over an object `x`, `Python` automatically calls `iter(x)` (i.e. `x.__iter__`).

An **iterable** is something which, when `iter` is called on it, returns an **iterator**.

(1) If `__iter__` is defined, it is called to implement an iterator.

(2) If not, `__getitem__` is called starting from index `0`.

(3) If no `__iter__` and no `__getitem__` then raise a `TypeError`.

Any `Python` sequence is iterable because sequences implement `__getitem__`. The standard sequences also implement `__iter__`; for future proofing you should too because  (2) might be deprecated in a future version of `Python`._____no_output_____We know that `for` operates on iterables:_____no_output_____

for i in s:
    print(i)Dogs
will
save
the
world.

What's actually going on here?_____no_output_____

it = iter(s) # Build an iterator from an iterable
while True:
    try:
        nextval = next(it) # Get the next item in the iterator
        print(nextval)
    except StopIteration:
        del it # Iterator is exhausted.  Release reference and discard.
        breakDogs
will
save
the
world.

We can completely abstract away a sequence in favor of an iterable (i.e. we don't need to support indexing anymore)

Example `14-4` in `Fluent Python`:_____no_output_____

class SentenceIterator: # has __next__ and __iter__
    def __init__(self, words):
        self.words = words
        self.index = 0

    def __next__(self):
        try:
            word = self.words[self.index]
        except IndexError:
            raise StopIteration()
        self.index += 1
        return word

    def __iter__(self):
        return self

class Sentence: # An iterable b/c it has __iter__
    def __init__(self, text):
        self.text = text
        self.words = text.split()

    def __iter__(self):
        return SentenceIterator(self.words) # Returns an instance of the iterator

    def __repr__(self):
        return 'Sentence(%s)' % reprlib.repr(self.text)_____no_output_____s2 = Sentence("What is data science?")_____no_output_____for i in s2:
    print(i)What
is
data
science?
s2it=iter(s2) # Make the iterable an iterator
print(next(s2it), "\n\n") # Get the next entry
s2it2=iter(s2) # Reset the iterator
print(next(s2it), "   ", next(s2it2)) # Get the next entry of s2it and s2it2What

is     What

While we could have implemented `__next__` in Sentence itself, making it an iterator, we will run into the problem of "exhausting an iterator".

The iterator above keeps state in `self.index` and we must be able to start anew by creating a new instance if we want to re-iterate. Thus the `__iter__` in the iterable simply returns the `SentenceIterator`.

From `Fluent Python` ("Sentence Take #2:  A Classic Iterator"):
> A common cause of errors in building iterables and iterators is to confuse the two. To be clear: iterables have an `__iter__` method that instantiates a new iterator every time. Iterators implement a `__next__` method that returns individual items, and an `__iter__` method that returns self._____no_output_____`min()` and `max()` also work even though we no longer satisfy the sequence protocol.

`min` and `max` are pairwise comparisons and can be handled via iteration.

The take home message is that in programming with these iterators we don't need either the length or indexing to work to implement many algorithms: we have abstracted these away._____no_output_____

min(s2), max(s2)_____no_output_____

# Trees

- A tree is a hierarchical data structure that has a bunch of items.
- Each item may have a value.
- Items may point to other items or not (leaf nodes).
- Each item is pointed to by exactly one other item with the sole exception of the root item.  Nothing points to the root item.

There are many types of trees used to express data structures.  For example,
* Binary Tree
* B-Tree
* Heap
* Multi-way tree

Each of these has many subtypes._____no_output_____#### Trees arise everywhere:

- In parsing of code
- Evolutionary trees in biology
- Language origin trees
- Unix file system
- HTML tags

Just like with lists, one can consider looking at a tree in two ways: a collection of nodes or a tree with a root and multiple sub-trees.

Today, and in your homework, the focus is on Binary trees._____no_output_____Once again, one can represent trees using the recursive data structures we used to represent linked lists (from cs61a):

![](http://wla.berkeley.edu/~cs61a/fa11/lectures/img/tree.png)

_____no_output_____You could also use a tree in which the nodes all themselves have data. This is often used to represent a binary tree._____no_output_____## Some Terminology (Binary Tree)
* The root node is the only node that has nothing pointing to it.
* A node can point to other nodes called children.
* The node doing the pointing is called the parent node.
* In a binary tree, a given node only points to two other nodes (maximum of two children).
* If a child node does not contain a value, then it is said to be empty (or null).
* A node with pointers to two empty values is called a leaf._____no_output_____## Binary Search Trees (BSTs)
These are binary trees with the requirement that all children to the left of a particular node have lower values and all children to the right have higher values.

There are ways of dealing with repeated values (e.g. storing a counter), but we will just use BSTs that do not have repeated values.

A major benefit of BSTs:
* Efficient searching:  Can be $O(\log(n))$ for a balanced tree where $n$ is the height of the tree.

An algorithm for the deletion of a node is as follows:
1. If the node to be deleted has no children then just remove it.
2. If the node to be deleted has only one child, remove the node and replace it with its child.
3. If the node to be deleted has two children, replace the node to be deleted with the maximum value in the left subtree. Finally, delete the node with the maximum value in the left-subtree._____no_output_____## Practice

In your homework, you will implement a BST.  The implementation should be able to insert new values and optionally be able to delete values from the tree.

Consider the array $$\left[13, 7, 19, 17, 3, 29, 5, 31, 2, 11\right].$$  With your group, at your whiteboard, do the following:
1. Draw a BST for the array
2. Delete the node with value $13$_____no_output_____


# Notebook from ccbrain/nipype_tutorial
Path: notebooks/introduction_quickstart.ipynb

# Nipype Quickstart_____no_output_____![Nipype architecture](https://raw.github.com/satra/intro2nipype/master/images/arch.png)_____no_output_____- [Existing documentation](http://nipype.readthedocs.io/en/latest/)

- [Visualizing the evolution of Nipype](https://www.youtube.com/watch?v=cofpD1lhmKU)

- This notebook taken from [reproducible-imaging repository](https://github.com/ReproNim/reproducible-imaging)_____no_output_____#### Import a few things from nipype and external libraries_____no_output_____

import os
from os.path import abspath

from nipype import Workflow, Node, MapNode, Function
from nipype.interfaces.fsl import BET, IsotropicSmooth, ApplyMask

from nilearn.plotting import plot_anat
%matplotlib inline
import matplotlib.pyplot as plt_____no_output_____

## Interfaces
Interfaces are the core pieces of Nipype. The interfaces are python modules that allow you to use various external packages (e.g. FSL, SPM or FreeSurfer), even if they themselves are written in another programming language than python.

**Let's try to use `bet` from FSL:**_____no_output_____

# will use a T1w from ds000114 dataset
input_file =  abspath("/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_T1w.nii.gz")_____no_output_____# we will be typing here_____no_output_____

If you're lost the code is here:_____no_output_____

bet = BET()
bet.inputs.in_file = input_file
bet.inputs.out_file = "/output/T1w_nipype_bet.nii.gz"
res = bet.run()_____no_output_____

let's check the output:_____no_output_____

res.outputs_____no_output_____

and we can plot the output file_____no_output_____

plot_anat('/output/T1w_nipype_bet.nii.gz',
          display_mode='ortho', dim=-1, draw_cross=False, annotate=False);_____no_output_____

you can always check the list of arguments using `help` method_____no_output_____

BET.help()_____no_output_____

#### Exercise 1a
Import `IsotropicSmooth` from `nipype.interfaces.fsl` and find out the `FSL` command that is being run. What are the mandatory inputs for this interface?_____no_output_____

# type your code here_____no_output_____from nipype.interfaces.fsl import IsotropicSmooth
# all this information can be found when we run `help` method.
# note that you can either provide `in_file` and `fwhm` or `in_file` and `sigma`
IsotropicSmooth.help()_____no_output_____

#### Exercise 1b
Run the `IsotropicSmooth` for `/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_T1w.nii.gz` file with a smoothing kernel 4mm:_____no_output_____

# type your solution here_____no_output_____smoothing = IsotropicSmooth()
smoothing.inputs.in_file = "/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_T1w.nii.gz"
smoothing.inputs.fwhm = 4
smoothing.inputs.out_file = "/output/T1w_nipype_smooth.nii.gz"
smoothing.run()_____no_output_____# plotting the output
plot_anat('/output/T1w_nipype_smooth.nii.gz',
          display_mode='ortho', dim=-1, draw_cross=False, annotate=False);_____no_output_____

## Nodes and Workflows_____no_output_____Interfaces are the core pieces of Nipype that run the code of your desire. But to streamline your analysis and to execute multiple interfaces in a sensible order, you have to put them in something that we call a Node and create a Workflow.

In Nipype, a node is an object that executes a certain function. This function can be anything from a Nipype interface to a user specified function or an external script. Each node consists of a name, an interface and at least one input field and at least one output field.

Once you have multiple nodes you can use `Workflow` to connect each other and create a directed graph. Nipype workflow will take care of input and output of each interface and arrange the execution of each interface in the most efficient way._____no_output_____**Let's create the first node using `BET` interface:**_____no_output_____

# we will be typing here_____no_output_____

If you're lost the code is here:_____no_output_____

# Create Node
bet_node = Node(BET(), name='bet')
# Specify node inputs
bet_node.inputs.in_file = input_file
bet_node.inputs.mask = True

# bet node can be also defined this way:
#bet_node = Node(BET(in_file=input_file, mask=True), name='bet_node')_____no_output_____

#### Exercise 2
Create a `Node` for IsotropicSmooth interface._____no_output_____

# Type your solution here:

# smooth_node = _____no_output_____smooth_node = Node(IsotropicSmooth(in_file=input_file, fwhm=4), name="smooth")_____no_output_____

**We will now create one more Node for our workflow**_____no_output_____

mask_node = Node(ApplyMask(), name="mask")_____no_output_____

Let's check the the interface:_____no_output_____

ApplyMask.help()_____no_output_____

As you can see the interface takes two mandatory inputs: `in_file` and `mask_file`. We want to use the output of `smooth_node` as `in_file` and on of the output of `bet_file` (the `mask_file`) as `mask_file` input._____no_output_____** Let's initialize a `Workflow`:**_____no_output_____

# will be writing the code here:
_____no_output_____

if you're lost, full code is here:_____no_output_____

# Initiation of a workflow
wf = Workflow(name="smoothflow", base_dir="/output/working_dir")_____no_output_____

It's very important to specify `base_dir` (as absolute path), because otherwise all the outputs would be saved somewhere in the temporary files._____no_output_____**lets connect the `bet_node` output to `mask_node` input`**_____no_output_____

# we will be typing here:
_____no_output_____

if you're lost, the code is here:_____no_output_____

wf.connect(bet_node, "mask_file", mask_node, "mask_file")_____no_output_____

#### Exercise 3
Connect `out_file` of `smooth_node` to `in_file` of `mask_node`._____no_output_____

# type your code here_____no_output_____wf.connect(smooth_node, "out_file", mask_node, "in_file")_____no_output_____

**Let's see a graph describing our workflow:**_____no_output_____

wf.write_graph("workflow_graph.dot")
from IPython.display import Image
Image(filename="/output/working_dir/smoothflow/workflow_graph.png")_____no_output_____

you can also plot a more detailed graph:_____no_output_____

wf.write_graph(graph2use='flat')
from IPython.display import Image
Image(filename="/output/working_dir/smoothflow/graph_detailed.png")_____no_output_____

**and now let's run the workflow**_____no_output_____

# we will type our code here:_____no_output_____

if you're lost, full code is here:_____no_output_____

# Execute the workflow
res = wf.run()_____no_output_____

**and let's look at the results**_____no_output_____

# we can check the output of specific nodes from workflow
list(res.nodes)[0].result.outputs_____no_output_____

**we can see the fie structure that has been created:**_____no_output_____

! tree -L 3 /output/working_dir/smoothflow/_____no_output_____

**and we can plot the results:**_____no_output_____

import numpy as np
import nibabel as nb
#import matplotlib.pyplot as plt

# Let's create a short helper function to plot 3D NIfTI images
def plot_slice(fname):

    # Load the image
    img = nb.load(fname)
    data = img.get_data()

    # Cut in the middle of the brain
    cut = int(data.shape[-1]/2) + 10

    # Plot the data
    plt.imshow(np.rot90(data[..., cut]), cmap="gray")
    plt.gca().set_axis_off()

f = plt.figure(figsize=(12, 4))
for i, img in enumerate(["/data/ds000114/sub-01/ses-test/anat/sub-01_ses-test_T1w.nii.gz",
                         "/output/working_dir/smoothflow/smooth/sub-01_ses-test_T1w_smooth.nii.gz",
                         "/output/working_dir/smoothflow/bet/sub-01_ses-test_T1w_brain_mask.nii.gz",
                         "/output/working_dir/smoothflow/mask/sub-01_ses-test_T1w_smooth_masked.nii.gz"]):
    f.add_subplot(1, 4, i + 1)
    plot_slice(img)_____no_output_____

## Iterables

Some steps in a neuroimaging analysis are repetitive. Running the same preprocessing on multiple subjects or doing statistical inference on multiple files. To prevent the creation of multiple individual scripts, Nipype has as execution plugin for ``Workflow``, called **``iterables``**.

_____no_output_____Let's assume we have a workflow with two nodes, node (A) does simple skull stripping, and is followed by a node (B) that does isometric smoothing. Now, let's say, that we are curious about the effect of different smoothing kernels. Therefore, we want to run the smoothing node with FWHM set to 2mm, 8mm and 16mm._____no_output_____**let's just modify `smooth_node`:** _____no_output_____

# we will type the code here
_____no_output_____

if you're lost the code is here:_____no_output_____

smooth_node_it = Node(IsotropicSmooth(in_file=input_file), name="smooth")
smooth_node_it.iterables = ("fwhm", [4, 8, 16])_____no_output_____

we will define again bet and smooth nodes:_____no_output_____

bet_node_it = Node(BET(in_file=input_file, mask=True), name='bet_node')
mask_node_it = Node(ApplyMask(), name="mask")_____no_output_____

** will create a new workflow with a new `base_dir`:**_____no_output_____

# Initiation of a workflow
wf_it = Workflow(name="smoothflow_it", base_dir="/output/working_dir")
wf_it.connect(bet_node_it, "mask_file", mask_node_it, "mask_file")
wf_it.connect(smooth_node_it, "out_file", mask_node_it, "in_file")_____no_output_____

**let's run the workflow and check the output**_____no_output_____

res_it = wf_it.run()_____no_output_____

**let's see the graph**_____no_output_____

list(res_it.nodes)_____no_output_____

We can see the file structure that was created:_____no_output_____

! tree -L 3 /output/working_dir/smoothflow_it/_____no_output_____

you have now 7 nodes instead of 3!_____no_output_____### MapNode

If you want to iterate over a list of inputs, but need to feed all iterated outputs afterwards as one input (an array) to the next node, you need to use a **``MapNode``**. A ``MapNode`` is quite similar to a normal ``Node``, but it can take a list of inputs and operate over each input separately, ultimately returning a list of outputs.

Imagine that you have a list of items (lets say files) and you want to execute the same node on them (for example some smoothing or masking). Some nodes accept multiple files and do exactly the same thing on them, but some don't (they expect only one file). `MapNode` can solve this problem. Imagine you have the following workflow:

Node `A` outputs a list of files, but node `B` accepts only one file. Additionally `C` expects a list of files. What you would like is to run `B` for every file in the output of `A` and collect the results as a list and feed it to `C`. _____no_output_____** Let's run a simple numerical example using nipype `Function` interface **_____no_output_____

def square_func(x):
    return x ** 2

square = Function(input_names=["x"], output_names=["f_x"], function=square_func)_____no_output_____

If I want to know the results only for one `x` we can use `Node`:_____no_output_____

square_node = Node(square, name="square")
square_node.inputs.x = 2
res = square_node.run()
res.outputs_____no_output_____

let's try to ask for more values of `x`_____no_output_____

# NBVAL_SKIP
square_node = Node(square, name="square")
square_node.inputs.x = [2, 4]
res = square_node.run()
res.outputs_____no_output_____

**It will give an error since `square_func` do not accept list. But we can try `MapNode`:**_____no_output_____

square_mapnode = MapNode(square, name="square", iterfield=["x"])
square_mapnode.inputs.x = [2, 4]
res = square_mapnode.run()
res.outputs_____no_output_____

**Notice that `f_x` is a list again!**_____no_output_____


# Notebook from ofou/course-content
Path: tutorials/W1D4_GeneralizedLinearModels/W1D4_Tutorial2.ipynb

[![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/tutorials/W1D4_GeneralizedLinearModels/W1D4_Tutorial2.ipynb)_____no_output_____# Tutorial 2: Classifiers and regularizers
**Week 1, Day 4: Generalized Linear Models**

**By Neuromatch Academy**

__Content creators:__ Pierre-Etienne H. Fiquet, Ari Benjamin, Jakob Macke

__Content reviewers:__ Davide Valeriani, Alish Dipani, Michael Waskom, Ella Batty
_____no_output_____**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

_____no_output_____# Tutorial Objectives

*Estimated timing of tutorial: 1 hour, 35 minutes*

This is part 2 of a 2-part series about Generalized Linear Models (GLMs), which are a fundamental framework for supervised learning. In part 1, we learned about and implemented GLMs. In this tutorial, we’ll implement logistic regression, a special case of GLMs used to model binary outcomes.
Oftentimes the variable you would like to predict takes only one of two possible values. Left or right? Awake or asleep? Car or bus? In this tutorial, we will decode a mouse's left/right decisions from spike train data. Our objectives are to:
1.	Learn about logistic regression, how it is derived within the GLM theory, and how it is implemented in scikit-learn
2.	Apply logistic regression to decode choies from neural responses
3.	Learn about regularization, including the different approaches and the influence of hyperparameters

---
We would like to acknowledge [Steinmetz _et al._ (2019)](https://www.nature.com/articles/s41586-019-1787-x) for sharing their data, a subset of which is used here.
_____no_output_____

# @title Tutorial slides

# @markdown These are the slides for the videos in all tutorials today
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/upyjz/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)_____no_output_____

# Setup
_____no_output_____

# Imports

import numpy as np
import matplotlib.pyplot as plt

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score_____no_output_____#@title Figure settings
import ipywidgets as widgets

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")_____no_output_____# @title Plotting Functions

def plot_weights(models, sharey=True):
  """Draw a stem plot of weights for each model in models dict."""
  n = len(models)
  f = plt.figure(figsize=(10, 2.5 * n))
  axs = f.subplots(n, sharex=True, sharey=sharey)
  axs = np.atleast_1d(axs)

  for ax, (title, model) in zip(axs, models.items()):

    ax.margins(x=.02)
    stem = ax.stem(model.coef_.squeeze(), use_line_collection=True)
    stem[0].set_marker(".")
    stem[0].set_color(".2")
    stem[1].set_linewidths(.5)
    stem[1].set_color(".2")
    stem[2].set_visible(False)
    ax.axhline(0, color="C3", lw=3)
    ax.set(ylabel="Weight", title=title)
  ax.set(xlabel="Neuron (a.k.a. feature)")
  f.tight_layout()

def plot_function(f, name, var, points=(-10, 10)):
    """Evaluate f() on linear space between points and plot.

    Args:
      f (callable): function that maps scalar -> scalar
      name (string): Function name for axis labels
      var (string): Variable name for axis labels.
      points (tuple): Args for np.linspace to create eval grid.
    """
    x = np.linspace(*points)
    ax = plt.figure().subplots()
    ax.plot(x, f(x))
    ax.set(
      xlabel=f'${var}$',
      ylabel=f'${name}({var})$'
    )

def plot_model_selection(C_values, accuracies):
  """Plot the accuracy curve over log-spaced C values."""
  ax = plt.figure().subplots()
  ax.set_xscale("log")
  ax.plot(C_values, accuracies, marker="o")
  best_C = C_values[np.argmax(accuracies)]
  ax.set(
      xticks=C_values,
      xlabel="$C$",
      ylabel="Cross-validated accuracy",
      title=f"Best C: {best_C:1g} ({np.max(accuracies):.2%})",
  )

def plot_non_zero_coefs(C_values, non_zero_l1, n_voxels):
  """Plot the accuracy curve over log-spaced C values."""
  ax = plt.figure().subplots()
  ax.set_xscale("log")
  ax.plot(C_values, non_zero_l1, marker="o")
  ax.set(
    xticks=C_values,
    xlabel="$C$",
    ylabel="Number of non-zero coefficients",
  )
  ax.axhline(n_voxels, color=".1", linestyle=":")
  ax.annotate("Total\n# Neurons", (C_values[0], n_voxels * .98), va="top")_____no_output_____#@title Data retrieval and loading
import os
import requests
import hashlib

url = "https://osf.io/r9gh8/download"
fname = "W1D4_steinmetz_data.npz"
expected_md5 = "d19716354fed0981267456b80db07ea8"

if not os.path.isfile(fname):
  try:
    r = requests.get(url)
  except requests.ConnectionError:
    print("!!! Failed to download data !!!")
  else:
    if r.status_code != requests.codes.ok:
      print("!!! Failed to download data !!!")
    elif hashlib.md5(r.content).hexdigest() != expected_md5:
      print("!!! Data download appears corrupted !!!")
    else:
      with open(fname, "wb") as fid:
        fid.write(r.content)

def load_steinmetz_data(data_fname=fname):

  with np.load(data_fname) as dobj:
    data = dict(**dobj)

  return data_____no_output_____

---

#Section 1: Logistic regression_____no_output_____

# @title Video 1: Logistic regression
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1P54y1q7Qn", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="qfXFrUnLU0o", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)_____no_output_____

Logistic Regression is a binary classification model. It is a GLM with a *logistic* link function and a *Bernoulli* (i.e. coinflip) noise model.

Like in the last notebook, logistic regression invokes a standard procedure:

1.   Define a *model* of how inputs relate to outputs.
2.   Adjust the parameters to maximize (log) probability of your data given your model

## Section 1.1: The logistic regression model

*Estimated timing to here from start of tutorial: 8 min*

 Click here for text recap of relevant part of video

The fundamental input/output equation of logistic regression is:

\begin{align}
\hat{y} \equiv p(y=1|x,\theta) = \sigma(\theta^Tx)
\end{align}

Note that we interpret the output of logistic regression, $\hat{y}$, as the **probability that y = 1** given inputs $x$ and parameters $\theta$.

Here $\sigma()$ is a "squashing" function called the **sigmoid function** or **logistic function**. Its output is in the range $0 \leq y \leq 1$. It looks like this:

\begin{align}
\sigma(z) = \frac{1}{1 + \textrm{exp}(-z)}
\end{align}

Recall that $z = \theta^T x$. The parameters decide whether $\theta^T x$ will be very negative, in which case $\sigma(\theta^T x)\approx 0$, or very positive, meaning  $\sigma(\theta^T x)\approx 1$.
_____no_output_____### Coding Exercise 1.1: Implement the sigmoid function
_____no_output_____

def sigmoid(z):
  """Return the logistic transform of z."""
  ##############################################################################
  # TODO for students: Fill in the missing code (...) and remove the error
  raise NotImplementedError("Student exercise: implement the sigmoid function")
  ##############################################################################

  sigmoid = ...

  return sigmoid

# Visualize
plot_function(sigmoid, "\sigma", "z", (-10, 10))_____no_output_____# to_remove solution
def sigmoid(z):
  """Return the logistic transform of z."""

  sigmoid = 1 / (1 + np.exp(-z))

  return sigmoid

# Visualize
with plt.xkcd():
  plot_function(sigmoid, "\sigma", "z", (-10, 10))_____no_output_____

## Section 1.2: Using scikit-learn

*Estimated timing to here from start of tutorial: 13 min*

Unlike the previous notebook, we're not going to write the code that implements all of the Logistic Regression model itself. Instead, we're going to use the implementation in [scikit-learn](https://scikit-learn.org/stable/), a very popular library for Machine Learning.

The goal of this next section is to introduce `scikit-learn` classifiers and understand how to apply it to real neural data._____no_output_____---
# Section 2: Decoding neural data with logistic regression_____no_output_____## Section 2.1: Setting up the data

*Estimated timing to here from start of tutorial: 15 min*

In this notebook we'll use the Steinmetz dataset that you have seen previously. Recall that this dataset includes recordings of neurons as mice perform a decision task.

Mice had the task of turning a wheel to indicate whether they perceived a Gabor stimulus to the left, to the right, or not at all. Neuropixel probes measured spikes across the cortex. Check out the following task schematic below from the BiorXiv preprint.

_____no_output_____

# @markdown Execute to see schematic
import IPython
IPython.display.Image("http://kordinglab.com/images/others/steinmetz-task.png")_____no_output_____

Today we're going to **decode the decision from neural data** using Logistic Regression. We will only consider trials where the mouse chose "Left" or "Right" and ignore NoGo trials.

### Data format

In the hidden `Data retrieval and loading` cell, there is a function that loads the data:

- `spikes`: an array of normalized spike rates with shape `(n_trials, n_neurons)`
- `choices`: a vector of 0s and 1s, indicating the animal's behavioral response, with length `n_trials`._____no_output_____

data = load_steinmetz_data()
for key, val in data.items():
  print(key, val.shape)_____no_output_____

As with the GLMs you've seen in the previous tutorial (Linear and Poisson Regression), we will need two data structures:

- an `X` matrix with shape `(n_samples, n_features)`
- a `y` vector with length `n_samples`.

In the previous notebook, `y` corresponded to the neural data, and `X` corresponded to something about the experiment. Here, we are going to invert those relationships. That's what makes this a *decoding* model: we are going to predict behavior (`y`) from the neural responses (`X`):_____no_output_____

y = data["choices"]
X = data["spikes"]_____no_output_____

## Section 2.2: Fitting the model

*Estimated timing to here from start of tutorial: 25 min*

Using a Logistic Regression model within `scikit-learn` is very simple. _____no_output_____

# Define the model
log_reg = LogisticRegression(penalty="none")

# Fit it to data
log_reg.fit(X, y)_____no_output_____

There's two steps here:

- We *initialized* the model with a hyperparameter, telling it what penalty to use (we'll focus on this in the second part of the notebook)
- We *fit* the model by passing it the `X` and `y` objects.

_____no_output_____## Section 2.3: Classifying the training data

*Estimated timing to here from start of tutorial: 27 min*

Fitting the model performs maximum likelihood optimization, learning a set of *feature weights*. We can use those learned weights to *classify* new data, or predict the labels for each sample:_____no_output_____

y_pred = log_reg.predict(X)_____no_output_____

## Section 2.4: Evaluating the model

*Estimated timing to here from start of tutorial: 30 min*

Now we need to evaluate the model's predictions. We'll do that with an *accuracy* score. The accuracy of the classifier is the proportion of trials where the predicted label matches the true label.
_____no_output_____### Coding Exercise 2.4: Classifier accuracy

For the first exercise, implement a function to evaluate a classifier using the accuracy score. Use it to get the accuracy of the classifier on the *training* data._____no_output_____

def compute_accuracy(X, y, model):
  """Compute accuracy of classifier predictions.

  Args:
    X (2D array): Data matrix
    y (1D array): Label vector
    model (sklearn estimator): Classifier with trained weights.

  Returns:
    accuracy (float): Proportion of correct predictions.
  """
  #############################################################################
  # TODO Complete the function, then remove the next line to test it
  raise NotImplementedError("Implement the compute_accuracy function")
  #############################################################################

  y_pred = model.predict(X)

  accuracy = ...

  return accuracy

# Compute train accurcy
train_accuracy = compute_accuracy(X, y, log_reg)
print(f"Accuracy on the training data: {train_accuracy:.2%}")_____no_output_____# to_remove solution

def compute_accuracy(X, y, model):
  """Compute accuracy of classifier predictions.

  Args:
    X (2D array): Data matrix
    y (1D array): Label vector
    model (sklearn estimator): Classifier with trained weights.

  Returns:
    accuracy (float): Proportion of correct predictions.
  """

  y_pred = model.predict(X)

  accuracy = (y == y_pred).mean()

  return accuracy

# Compute train accurcy
train_accuracy = compute_accuracy(X, y, log_reg)
print(f"Accuracy on the training data: {train_accuracy:.2%}")_____no_output_____

## Section 2.5: Cross-validating the classifer

*Estimated timing to here from start of tutorial: 40 min*

Classification accuracy on the training data is 100%! That might sound impressive, but you should recall from yesterday the concept of *overfitting*: the classifier may have learned something idiosyncratic about the training data. If that's the case, it won't have really learned the underlying data->decision function, and thus won't generalize well to new data.

To check this, we can evaluate the *cross-validated* accuracy.
_____no_output_____

# @markdown Execute to see schematic
import IPython
IPython.display.Image("http://kordinglab.com/images/others/justCV-01.png")_____no_output_____

### Cross-validating using `scikit-learn` helper functions

Yesterday, we asked you to write your own functions for implementing cross-validation. In practice, this won't be necessary, because `scikit-learn` offers a number of [helpful functions](https://scikit-learn.org/stable/model_selection.html) that will do this for you. For example, you can cross-validate a classifer using `cross_val_score`.

`cross_val_score` takes a `sklearn` model like `LogisticRegression`, as well as your `X` and `y` data. It then retrains your model on test/train splits of `X` and `y`, and returns the test accuracy on each of the test sets._____no_output_____

accuracies = cross_val_score(LogisticRegression(penalty='none'), X, y, cv=8) # k=8 crossvalidation_____no_output_____#@title
#@markdown Run to plot out these `k=8` accuracy scores.
f, ax = plt.subplots(figsize=(8, 3))
ax.boxplot(accuracies, vert=False, widths=.7)
ax.scatter(accuracies, np.ones(8))
ax.set(
  xlabel="Accuracy",
  yticks=[],
  title=f"Average test accuracy: {accuracies.mean():.2%}"
)
ax.spines["left"].set_visible(False)_____no_output_____

The lower cross-validated accuracy compared to the training accuracy (100%) suggests that the model is being *overfit*. Is this surprising? Think about the shape of the $X$ matrix:_____no_output_____

X.shape_____no_output_____

The model has almost three times as many features as samples. This is a situation where overfitting is very likely (almost guaranteed).

**Link to neuroscience**: Neuro data commonly has more features than samples. Having more neurons than independent trials is one example. In fMRI data, there are commonly more measured voxels than independent trials.
_____no_output_____### Why more features than samples leads to overfitting

In brief, the variance of model estimation increases when there are more features than samples. That is, you would get a very different model every time you get new data and run `.fit()`. This is very related to the *bias/variance tradeoff* you learned about on day 1.

Why does this happen? Here's a tiny example to get your intuition going. Imagine trying to find a best-fit line in 2D when you only have 1 datapoint. There are simply a infinite number of lines that pass through that point. This is the situation we find ourselves in with more features than samples.

### What we can do about it
As you learned on day 1, you can decrease model variance if you don't mind increasing its bias. Here, we will increase bias by assuming that the correct parameters are all small. In our 2D example, this is like prefering the horizontal line to all others. This is one example of *regularization*. _____no_output_____-----

#Section 3: Regularization

*Estimated timing to here from start of tutorial: 50 min*
_____no_output_____

# @title Video 2: Regularization
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1Tg4y1i773", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="b2IaUCZ91bo", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)_____no_output_____

 Click here for text recap of video

Regularization forces a model to learn a set solutions you *a priori* believe to be more correct, which reduces overfitting because it doesn't have as much flexibility to fit idiosyncracies in the training data. This adds model bias, but it's a good bias because you know (maybe) that parameters should be small or mostly 0.

In a GLM, a common form of regularization is to *shrink* the classifier weights. In a linear model, you can see its effect by plotting the weights. We've defined a helper function, `plot_weights`, that we'll use extensively in this section.

Here is what the weights look like for a Logistic Regression model with no regularization:_____no_output_____

log_reg = LogisticRegression(penalty="none").fit(X, y)
plot_weights({"No regularization": log_reg})_____no_output_____

It's important to understand this plot. Each dot visualizes a value in our parameter vector $\theta$. (It's the same style of plot as the one showing $\theta$ in the video). Since each feature is the time-averaged response of a neuron, each dot shows how the model uses each neuron to estimate a decision.

Note the scale of the y-axis. Some neurons have values of about $20$, whereas others scale to $-20$._____no_output_____## Section 3.1: $L_2$ regularization

*Estimated timing to here from start of tutorial: 53 min*

Regularization comes in different flavors. A very common one uses an $L_2$ or "ridge" penalty. This changes the objective function to

\begin{align}
-\log\mathcal{L}'(\theta | X, y)=
-\log\mathcal{L}(\theta | X, y) +\frac\beta2\sum_i\theta_i^2,
\end{align}

where $\beta$ is a *hyperparameter* that sets the *strength* of the regularization.

You can use regularization in `scikit-learn` by changing the `penalty`, and you can set the strength of the regularization with the `C` hyperparameter ($C = \frac{1}{\beta}$, so this sets the *inverse* regularization).

Let's compare the unregularized classifier weights with the classifier weights when we use the default `C = 1`:_____no_output_____

log_reg_l2 = LogisticRegression(penalty="l2", C=1).fit(X, y)

# now show the two models
models = {
  "No regularization": log_reg,
  "$L_2$ (C = 1)": log_reg_l2,
}
plot_weights(models)_____no_output_____

Using the same scale for the two y axes, it's almost impossible to see the $L_2$ weights. Let's allow the y axis scales to adjust to each set of weights:_____no_output_____

plot_weights(models, sharey=False)_____no_output_____

Now you can see that the weights have the same basic pattern, but the regularized weights are an order-of-magnitude smaller.

### Interactive Demo 3.1: The effect of varying C on parameter size

We can use this same approach to see how the weights depend on the *strength* of the regularization:_____no_output_____

# @markdown Execute this cell to enable the widget!

# Precompute the models so the widget is responsive
log_C_steps = 1, 11, 1
penalized_models = {}
for log_C in np.arange(*log_C_steps, dtype=int):
  m = LogisticRegression("l2", C=10 ** log_C, max_iter=5000)
  penalized_models[log_C] = m.fit(X, y)

@widgets.interact
def plot_observed(log_C = widgets.FloatSlider(value=1, min=1, max=10, step=1)):
  models = {
    "No regularization": log_reg,
    f"$L_2$ (C = $10^{log_C}$)": penalized_models[log_C]
  }
  plot_weights(models)_____no_output_____

Recall from above that $C=\frac1\beta$ so larger `C` is less regularization. The top panel corresponds to $C=\infty$._____no_output_____## Section 3.2: $L_1$ regularization

*Estimated timing to here from start of tutorial: 1 hr, 3 min*

$L_2$ is not the only option for regularization. There is also the $L_1$, or "Lasso" penalty. This changes the objective function to

\begin{align}
-\log\mathcal{L}'(\theta | X, y)=
-\log\mathcal{L}(\theta | X, y) +\frac\beta2\sum_i|\theta_i|
\end{align}

In practice, using the summed absolute values of the weights causes *sparsity*: instead of just getting smaller, some of the weights will get forced to $0$:_____no_output_____

log_reg_l1 = LogisticRegression(penalty="l1", C=1, solver="saga", max_iter=5000)
log_reg_l1.fit(X, y)
models = {
  "$L_2$ (C = 1)": log_reg_l2,
  "$L_1$ (C = 1)": log_reg_l1,
}
plot_weights(models)_____no_output_____

Note: You'll notice that we added two additional parameters: `solver="saga"` and `max_iter=5000`. The `LogisticRegression` class can use several different optimization algorithms ("solvers"), and not all of them support the $L_1$ penalty. At a certain point, the solver will give up if it hasn't found a minimum value. The `max_iter` parameter tells it to make more attempts; otherwise, we'd see an ugly warning about "convergence"._____no_output_____## Section 3.3: The key difference between $L_1$ and $L_2$ regularization: sparsity

*Estimated timing to here from start of tutorial: 1 hr, 10 min*

When should you use $L_1$ vs. $L_2$ regularization? Both penalties shrink parameters, and both will help reduce overfitting. However, the models they lead to are different.

In particular, the $L_1$ penalty encourages *sparse* solutions in which most parameters are 0. Let's unpack the notion of sparsity.

A "dense" vector has mostly nonzero elements:
$\begin{bmatrix}
  0.1 \\ -0.6\\-9.1\\0.07
\end{bmatrix}$.
A "sparse" vector has mostly zero elements:
$\begin{bmatrix}
  0 \\ -0.7\\ 0\\0
\end{bmatrix}$.

The same is true of matrices:_____no_output_____

# @markdown Execute to plot a dense and a sparse matrix
np.random.seed(50)
n = 5
M = np.random.random((n, n))
M_sparse = np.random.choice([0,1], size=(n, n), p=[0.8, 0.2])

fig, axs = plt.subplots(1, 2, sharey=True, figsize=(10,5))

axs[0].imshow(M)
axs[1].imshow(M_sparse)
axs[0].axis('off')
axs[1].axis('off')
axs[0].set_title("A dense matrix", fontsize=15)
axs[1].set_title("A sparse matrix", fontsize=15)
text_kws = dict(ha="center", va="center")
for i in range(n):
  for j in range(n):
    iter_parts = axs, [M, M_sparse], ["{:.1f}", "{:d}"]
    for ax, mat, fmt in zip(*iter_parts):
      val = mat[i, j]
      color = ".1" if val > .7 else "w"
      ax.text(j, i, fmt.format(val), c=color, **text_kws)_____no_output_____

### Coding Exercise 3.3: The effect of $L_1$ regularization on parameter sparsity

Please complete the following function to fit a regularized `LogisticRegression` model and return **the number of coefficients in the parameter vector that are equal to 0**.

Don't forget to check out the [scikit-learn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html).
_____no_output_____

def count_non_zero_coefs(X, y, C_values):
  """Fit models with different L1 penalty values and count non-zero coefficients.

  Args:
    X (2D array): Data matrix
    y (1D array): Label vector
    C_values (1D array): List of hyperparameter values

  Returns:
    non_zero_coefs (list): number of coefficients in each model that are nonzero

  """
  #############################################################################
  # TODO Complete the function and remove the error
  raise NotImplementedError("Implement the count_non_zero_coefs function")
  #############################################################################

  non_zero_coefs = []
  for C in C_values:

    # Initialize and fit the model
    # (Hint, you may need to set max_iter)
    model = ...
    ...

    # Get the coefs of the fit model (in sklearn, we can do this using model.coef_)
    coefs = ...

    # Count the number of non-zero elements in coefs
    non_zero = ...
    non_zero_coefs.append(non_zero)

  return non_zero_coefs

# Use log-spaced values for C
C_values = np.logspace(-4, 4, 5)

# Count non zero coefficients
non_zero_l1 = count_non_zero_coefs(X, y, C_values)

# Visualize
plot_non_zero_coefs(C_values, non_zero_l1, n_voxels=X.shape[1])_____no_output_____# to_remove solution

def count_non_zero_coefs(X, y, C_values):
  """Fit models with different L1 penalty values and count non-zero coefficients.

  Args:
    X (2D array): Data matrix
    y (1D array): Label vector
    C_values (1D array): List of hyperparameter values

  Returns:
    non_zero_coefs (list): number of coefficients in each model that are nonzero

  """
  non_zero_coefs = []
  for C in C_values:

    # Initialize and fit the model
    # (Hint, you may need to set max_iter)
    model = LogisticRegression(penalty="l1", C=C, solver="saga", max_iter=5000)
    model.fit(X,y)

    # Get the coefs of the fit model (in sklearn, we can do this using model.coef_)
    coefs = model.coef_

    # Count the number of non-zero elements in coefs
    non_zero = np.sum(coefs != 0)
    non_zero_coefs.append(non_zero)

  return non_zero_coefs

# Use log-spaced values for C
C_values = np.logspace(-4, 4, 5)

# Count non zero coefficients
non_zero_l1 = count_non_zero_coefs(X, y, C_values)

# Visualize
with plt.xkcd():
  plot_non_zero_coefs(C_values, non_zero_l1, n_voxels=X.shape[1])_____no_output_____

Smaller `C` (bigger $\beta$) leads to sparser solutions.

**Link to neuroscience**: When is it OK to assume that the parameter vector is sparse? Whenever it is true that most features don't affect the outcome. One use-case might be decoding low-level visual features from whole-brain fMRI: we may expect only voxels in V1 and thalamus should be used in the prediction.

**WARNING**: be careful when interpreting $\theta$. Never interpret the nonzero coefficients as *evidence* that only those voxels/neurons/features carry information about the outcome. This is a product of our regularization scheme, and thus *our prior assumption that the solution is sparse*. Other regularization types or models may find very distributed relationships across the brain. Never use a model as evidence for a phenomena when that phenomena is encoded in the assumptions of the model. _____no_output_____## Section 3.4: Choosing the regularization penalty

*Estimated timing to here from start of tutorial: 1 hr, 25 min*

In the examples above, we just picked arbitrary numbers for the strength of regularization. How do you know what value of the hyperparameter to use?

The answer is the same as when you want to know whether you have learned good parameter values: use cross-validation. The best hyperparameter will be the one that allows the model to generalize best to unseen data._____no_output_____### Coding Exercise 3.4: Model selection

In the final exercise, we will use cross-validation to evaluate a set of models, each with a different $L_2$ penalty. Your `model_selection` function should have a for-loop that gets the mean cross-validated accuracy for each penalty value (use the `cross_val_score` function that we introduced above)._____no_output_____

def model_selection(X, y, C_values):
  """Compute CV accuracy for each C value.

  Args:
    X (2D array): Data matrix
    y (1D array): Label vector
    C_values (1D array): Array of hyperparameter values

  Returns:
    accuracies (1D array): CV accuracy with each value of C

  """
  #############################################################################
  # TODO Complete the function and remove the error
  raise NotImplementedError("Implement the model_selection function")
  #############################################################################

  accuracies = []
  for C in C_values:

    # Initialize and fit the model
    # (Hint, you may need to set max_iter)
    model = ...

    # Get the accuracy for each test split using cross-validation
    accs = ...

    # Store the average test accuracy for this value of C
    accuracies.append(...)

  return accuracies

# Use log-spaced values for C
C_values = np.logspace(-4, 4, 9)

# Compute accuracies
accuracies = model_selection(X, y, C_values)

# Visualize
plot_model_selection(C_values, accuracies)_____no_output_____# to_remove solution

def model_selection(X, y, C_values):
  """Compute CV accuracy for each C value.

  Args:
    X (2D array): Data matrix
    y (1D array): Label vector
    C_values (1D array): Array of hyperparameter values.

  Returns:
    accuracies (1D array): CV accuracy with each value of C.

  """
  accuracies = []
  for C in C_values:

    # Initialize and fit the model
    # (Hint, you may need to set max_iter)
    model = LogisticRegression(penalty="l2", C=C, max_iter=5000)

    # Get the accuracy for each test split using cross-validation
    accs = cross_val_score(model, X, y, cv=8)

    # Store the average test accuracy for this value of C
    accuracies.append(accs.mean())

  return accuracies

# Use log-spaced values for C
C_values = np.logspace(-4, 4, 9)

# Compute accuracies
accuracies = model_selection(X, y, C_values)

# Visualize
with plt.xkcd():
  plot_model_selection(C_values, accuracies)_____no_output_____

This plot suggests that the right value of $C$ does matter — up to a point. Remember that C is the *inverse* regularization. The plot shows that models where the regularization was too strong (small C values) performed very poorly. For $C > 10^{-2}$, the differences are marginal, but the best performance was obtained with an intermediate value ($C \approx 10^1$)._____no_output_____---
# Summary

*Estimated timing of tutorial: 1 hour, 35 minutes*

In this notebook, we learned about Logistic Regression, a fundamental algorithm for *classification*. We applied the algorithm to a *neural decoding* problem: we tried to predict an animal's behavioral choice from its neural activity. We saw again how important it is to use *cross-validation* to evaluate complex models that are at risk for *overfitting*, and we learned how *regularization* can be used to fit models that generalize better. Finally, we learned about some of the different options for regularization, and we saw how cross-validation can be useful for *model selection*._____no_output_____---
# Notation

\begin{align}
x &\quad \text{input}\\
y &\quad \text{measurement, response}\\
\theta &\quad \text{parameter}\\
\sigma(z) &\quad \text{logistic function}\\
C &\quad \text{inverse regularization strength parameter}\\
\beta &\quad \text{regularization strength parameter}\\
\hat{y} &\quad \text{estimated output}\\
\mathcal{L}(\theta| y_i, x_i) &\quad \text{likelihood of that parameter } \theta \text{ producing response } y_i \text{ from input } x_i\\
L_1 &\quad \text{Lasso regularization}\\
L_2 &\quad \text{ridge regularization}\\
\end{align}_____no_output_____---
# Bonus
_____no_output_____---
## Bonus Section 1: The Logistic Regression model in full

The fundamental input/output equation of logistic regression is:

\begin{align}
p(y_i = 1 |x_i, \theta) = \sigma(\theta^Tx_i)
\end{align}

**The logistic link function**

You've seen $\theta^T x_i$ before, but the $\sigma$ is new. It's the *sigmoidal* or *logistic* link function that "squashes" $\theta^T x_i$ to keep it between $0$ and $1$:

\begin{align}
\sigma(z) = \frac{1}{1 + \textrm{exp}(-z)}
\end{align}

**The Bernoulli likelihood**

You might have noticed that the output of the sigmoid, $\hat{y}$ is not a binary value (0 or 1), even though the true data $y$ is! Instead, we interpret the value of $\hat{y}$ as the *probability that y = 1*:

\begin{align}
\hat{y_i} \equiv p(y_i=1|x_i,\theta) = \frac{1}{{1 + \textrm{exp}(-\theta^Tx_i)}}
\end{align}

To get the likelihood of the parameters, we need to define *the probability of seeing $y$ given $\hat{y}$*. In logistic regression, we do this using the Bernoulli distribution:

\begin{align}
P(y_i\ |\ \hat{y}_i) = \hat{y}_i^{y_i}(1 - \hat{y}_i)^{(1 - y_i)}
\end{align}

So plugging in the regression model:

\begin{align}
P(y_i\ |\ \theta, x_i) = \sigma(\theta^Tx_i)^{y_i}\left(1 - \sigma(\theta^Tx_i)\right)^{(1 - y_i)}.
\end{align}

This expression effectively measures how good our parameters $\theta$ are. We can also write it as the likelihood of the parameters given the data:

\begin{align}
\mathcal{L}(\theta\ |\ y_i, x_i) = P(y_i\ |\ \theta, x_i),
\end{align}

and then use this as a target of optimization, considering all of the trials independently:

\begin{align}
\log\mathcal{L}(\theta | X, y) = \sum_{i=1}^Ny_i\log\left(\sigma(\theta^Tx_i)\right)\ +\ (1-y_i)\log\left(1 - \sigma(\theta^Tx_i)\right).
\end{align}_____no_output_____---
## Bonus Section 2: More detail about model selection

In the final exercise, we used all of the data to choose the hyperparameters. That means we don't have any fresh data left over to evaluate the performance of the selected model. In practice, you would want to have two *nested* layers of cross-validation, where the final evaluation is performed on data that played no role in selecting or training the model.

Indeed, the proper method for splitting your data to choose hyperparameters can get confusing. Here's a guide that the authors of this notebook developed while writing a tutorial on using machine learning for neural decoding (https://arxiv.org/abs/1708.00909).

_____no_output_____

# @markdown Execute to see schematic
import IPython
IPython.display.Image("http://kordinglab.com/images/others/CV-01.png")_____no_output_____


# Notebook from loganylchen/create_reference
Path: 05_recipes.ipynb

<code>
# default_exp recipes_____no_output_____%load_ext autoreload
%autoreload 2_____no_output_____# export

from create_reference.tasks import *_____no_output_____# export

def recipe(workflow,species_paras,args):
    get_fasta = [
        workflow.add_task(
            func=task_get_file,
            params=dict(
                link=para['link_genome_fasta'],
                filename=para['local_files']['local_genome_fasta_gz'],
            ),
            uid='get_fasta_%s_%s' % (para['species'],para['version']) ,
            stage_name='get_fasta'
        ) for para in species_paras]

    gunzip_fasta = [
        workflow.add_task(
            func=task_gunzip,
            params=dict(
                gzipfile=para['local_files']['local_genome_fasta_gz']
            ),
            uid='gunzip_fasta_%s_%s' % (para['species'],para['version']) ,
            parents=get_fasta[i],
            stage_name='gunzip_fasta'
        ) for i,para in enumerate(species_paras)]

    get_gtf = [
        workflow.add_task(
            func=task_get_file,
            params=dict(
                link=para['link_transcriptome_gtf'],
                filename=para['local_files']['local_transcriptome_gtf_gz'],
            ),
            uid='get_gtf_%s_%s' % (para['species'],para['version']) ,
            stage_name='get_gtf'
        ) for para in species_paras]

    gunzip_gtf = [
        workflow.add_task(
            func=task_gunzip,
            params=dict(
                gzipfile=para['local_files']['local_transcriptome_gtf_gz']
            ),
            uid='gunzip_fasta_%s_%s' % (para['species'],para['version']) ,
            parents=get_gtf[i],
            stage_name='gunzip_gtf'
        ) for i,para in enumerate(species_paras)]

    get_rRNA_bed = [
        workflow.add_task(
            func=task_extract_rRNA_bed,
            params=dict(
                gtf=para['local_files']['local_transcriptome_gtf'],
                bed=para['local_files']['rrna_bed']
            ),
            uid='get_rRNA_bed_%s_%s' % (para['species'],para['version']) ,
            parents=gunzip_gtf[i],
            stage_name='get_rRNA_bed'
        ) for i,para in enumerate(species_paras)]

    if 'picard' in args.indexs:
        get_picard_idx = [
            workflow.add_task(
                func=task_picard_build_index,
                params=dict(
                    software=args.picard,
                    reference=para['local_files']['local_genome_fasta'],
                    reference_dict=para['local_files']['picard_idx'],
                    tmp=para['local_files']['tmp']
                ),
                uid='get_picard_idx_%s_%s' % (para['species'],para['version']) ,
                parents=gunzip_fasta[i],
                stage_name='get_picard_idx'
            ) for i,para in enumerate(species_paras) ]

        get_rRNA_intervals = [
            workflow.add_task(
                func=task_build_rRNA_intervals,
                params=dict(
                    software=args.picard,
                    bed=para['local_files']['rrna_bed'],
                    intervals=para['local_files']['rrna_intervals'],
                    reference_dict=para['local_files']['picard_idx'],
                    tmp=para['local_files']['tmp']
                ),
                uid='get_rRNA_intervals_%s_%s' % (para['species'],para['version']) ,
                parents=[get_picard_idx[i],get_rRNA_bed[i]],
                stage_name='get_rRNA_intervals'
            ) for i,para in enumerate(species_paras) ]

    if 'bwa' in args.indexs:
        get_bwa_idx = [
            workflow.add_task(
                func=task_bwa_build_index,
                params=dict(
                    software=args.bwa,
                    reference=para['local_files']['local_genome_fasta'],
                ),
                uid='get_bwa_index_%s_%s' % (para['species'],para['version']) ,
                parents=gunzip_fasta[i],
                stage_name='get_bwa_index'
            ) for i,para in enumerate(species_paras)]

    if 'samtools' in args.indexs:
        get_bwa_idx = [
            workflow.add_task(
                func=task_samtools_build_index,
                params=dict(
                    software=args.samtools,
                    reference=para['local_files']['local_genome_fasta'],
                ),
                uid='get_samtools_index_%s_%s' % (para['species'],para['version']) ,
                parents=gunzip_fasta[i],
                stage_name='get_samtools_index'
            ) for i,para in enumerate(species_paras)]

    if 'bowtie' in args.indexs:
        get_bowtie_idx = [
            workflow.add_task(
                func=task_bowtie_build_index,
                params=dict(
                    software=args.bowtie,
                    reference=para['local_files']['local_genome_fasta'],
                    prefix=para['local_files']['bowtie_idx']
                ),
                uid='get_bowtie_index_%s_%s' % (para['species'],para['version']) ,
                parents=gunzip_fasta[i],
                stage_name='get_bowtie_index'
            ) for i,para in enumerate(species_paras)]

    if 'bowtie2' in args.indexs:
        get_bowtie2_idx = [
            workflow.add_task(
                func=task_bowtie2_build_index,
                params=dict(
                    software=args.bwa,
                    reference=para['local_files']['local_genome_fasta'],
                    prefix=para['local_files']['bowtie2_idx']
                ),
                uid='get_bowtie2_index_%s_%s' % (para['species'],para['version']) ,
                parents=gunzip_fasta[i],
                stage_name='get_bowtie2_index'
            ) for i,para in enumerate(species_paras)]

    if 'hisat2' in args.indexs:
        get_hisat2_idx = [
            workflow.add_task(
                func=task_hisat2_build_index,
                params=dict(
                    software=args.hisat2,
                    reference=para['local_files']['local_genome_fasta'],
                    prefix=para['local_files']['hisat2_idx'],
                    snp='',
                    gtf=para['local_files']['local_transcriptome_gtf']
                ),
                uid='get_hisat2_index_%s_%s' % (para['species'],para['version']) ,
                parents=[gunzip_fasta[i],gunzip_gtf[i]],
                stage_name='get_hisat2_index'
            ) for i,para in enumerate(species_paras)]

    if 'star' in args.indexs:
        get_star_idx = [
            workflow.add_task(
                func=task_star_build_index,
                params=dict(
                    software=args.star,
                    reference=para['local_files']['local_genome_fasta'],
                    prefix=para['local_files']['star_idx_prefix']+'_'+str(read_length),
                    gtf=para['local_files']['local_transcriptome_gtf'],
                    read_length=read_length
                ),
                uid='get_star_index_%s_%s_%s' % (para['species'],para['version'],str(read_length)) ,
                parents=[gunzip_fasta[i],gunzip_gtf[i]],
                stage_name='get_star_index'
            ) for i,para in enumerate(species_paras) for read_length in para['read_length']]
    _____no_output_____#hide
from nbdev.export import notebook2script
notebook2script()Converted 00_utils.ipynb.
Converted 01_defaults.ipynb.
Converted 02_tasks.ipynb.
Converted 03_commands.ipynb.
Converted 04_pipelines.ipynb.
Converted 05_recipes.ipynb.
Converted index.ipynb.

</code>