# Notebook from berombau/deep-learning-biology
Path: exercises/solutions/dlb-2-cnn-segmentation.ipynb

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JorisRoels/deep-learning-biology/blob/main/exercises/solutions/dlb-2-cnn-segmentation.ipynb)

# Exercise 3: Convolutional neural networks for segmentation

In this notebook, we will be using convolutional neural networks for segmentation of neurons in electron microscopy data.

The structure of these exercises is as follows:

1. [Import libraries and download data](#scrollTo=ScagUEMTMjlK)
2. [Data visualization and pre-processing](#scrollTo=ohZHyOTnI35b)
3. [Segmentation: a pixel classification problem](#scrollTo=UyspYtez5J8a)
4. [Building a U-Net with PyTorch](#scrollTo=wXbjn29WOOJ3)
5. [Training & validating the network](#scrollTo=zh8Pf_3HF_hi)

This notebook is largely based on the research published in:

Arganda-Carreras, I., Turaga, S. C., Berger, D. R., Ciresan, D. C., Giusti, A., Gambardella, L. M., Schmidhuber, J., Laptev, D., Dwivedi, S., Buhmann, J. M., Liu, T., Seyedhosseini, M., Tasdizen, T., Kamentsky, L., Burget, R., Uher, V., Tan, X., Sun, C., Pham, T. D., … Seung, H. S. (2015). Crowdsourcing the creation of image segmentation algorithms for connectomics. Frontiers in Neuroanatomy, 9. https://doi.org/10.3389/fnana.2015.00142_____no_output_____## 1. Import libraries and download data
Let's start with importing the necessary libraries. _____no_output_____

!pip install neuralnetsCollecting neuralnets
  Downloading https://files.pythonhosted.org/packages/74/82/451bf60b79d630a13100327562b7af8e8cce4e7856deef2f66f805fd92e6/neuralnets-1.1.6.3-py3-none-any.whl (56kB)

     |█████▉                          | 10kB 21.6MB/s eta 0:00:01
     |███████████▋                    | 20kB 28.0MB/s eta 0:00:01
     |█████████████████▌              | 30kB 26.8MB/s eta 0:00:01
     |███████████████████████▎        | 40kB 23.2MB/s eta 0:00:01
     |█████████████████████████████▏  | 51kB 24.0MB/s eta 0:00:01
     |████████████████████████████████| 61kB 8.7MB/s
Requirement already satisfied: tensorboard in /usr/local/lib/python3.6/dist-packages (from neuralnets) (2.3.0)
Requirement already satisfied: scikit-image in /usr/local/lib/python3.6/dist-packages (from neuralnets) (0.16.2)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from neuralnets) (1.15.0)
Requirement already satisfied: tifffile in /usr/local/lib/python3.6/dist-packages (from neuralnets) (2020.9.3)
Requirement already satisfied: torchvision in /usr/local/lib/python3.6/dist-packages (from neuralnets) (0.8.1+cu101)
Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from neuralnets) (2.10.0)
Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from neuralnets) (0.16.0)
Collecting progress
  Downloading https://files.pythonhosted.org/packages/38/ef/2e887b3d2b248916fc2121889ce68af8a16aaddbe82f9ae6533c24ff0d2b/progress-1.5.tar.gz
Requirement already satisfied: sklearn in /usr/local/lib/python3.6/dist-packages (from neuralnets) (0.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from neuralnets) (1.18.5)
Requirement already satisfied: opencv-python in /usr/local/lib/python3.6/dist-packages (from neuralnets) (4.1.2.30)
Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (from neuralnets) (1.7.0+cu101)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard->neuralnets) (50.3.2)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard->neuralnets) (0.4.2)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard->neuralnets) (3.3.3)
Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard->neuralnets) (1.33.2)
Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard->neuralnets) (1.17.2)
Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard->neuralnets) (2.23.0)
Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard->neuralnets) (3.12.4)
Requirement already satisfied: wheel>=0.26; python_version >= "3" in /usr/local/lib/python3.6/dist-packages (from tensorboard->neuralnets) (0.35.1)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard->neuralnets) (1.0.1)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.6/dist-packages (from tensorboard->neuralnets) (0.10.0)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard->neuralnets) (1.7.0)
Requirement already satisfied: imageio>=2.3.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image->neuralnets) (2.4.1)
Requirement already satisfied: matplotlib!=3.0.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image->neuralnets) (3.2.2)
Requirement already satisfied: pillow>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image->neuralnets) (7.0.0)
Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image->neuralnets) (2.5)
Requirement already satisfied: PyWavelets>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image->neuralnets) (1.1.1)
Requirement already satisfied: scipy>=0.19.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image->neuralnets) (1.4.1)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from sklearn->neuralnets) (0.22.2.post1)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch->neuralnets) (3.7.4.3)
Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/dist-packages (from torch->neuralnets) (0.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard->neuralnets) (1.3.0)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard->neuralnets) (2.0.0)
Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3" in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard->neuralnets) (4.6)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard->neuralnets) (4.1.1)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard->neuralnets) (0.2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard->neuralnets) (1.24.3)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard->neuralnets) (2.10)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard->neuralnets) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard->neuralnets) (2020.11.8)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->neuralnets) (1.3.1)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->neuralnets) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->neuralnets) (2.4.7)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->neuralnets) (2.8.1)
Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx>=2.0->scikit-image->neuralnets) (4.4.2)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sklearn->neuralnets) (0.17.0)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard->neuralnets) (3.1.0)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard->neuralnets) (3.4.0)
Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from rsa<5,>=3.1.4; python_version >= "3"->google-auth<2,>=1.6.3->tensorboard->neuralnets) (0.4.8)
Building wheels for collected packages: progress
  Building wheel for progress (setup.py) ... done
  Created wheel for progress: filename=progress-1.5-cp36-none-any.whl size=8074 sha256=5a74fcb22156810461bcdf53708811bb1cb3f36e898ab3dcc93d5f0d66e54bde
  Stored in directory: /root/.cache/pip/wheels/6c/c8/80/32a294e3041f006c661838c05a411c7b7ffc60ff939d14e116
Successfully built progress
Installing collected packages: progress, neuralnets
Successfully installed neuralnets-1.1.6.3 progress-1.5
import pickle
import numpy as np
import random
import os
import matplotlib.pyplot as plt
plt.rcdefaults()
import pandas as pd
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.manifold import TSNE
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from progressbar import ProgressBar, Percentage, Bar, ETA, FileTransferSpeed
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch.utils.data as data
from torch.utils.data import DataLoader
from torchvision import datasets
import gdown
import zipfile
import os
import progressbar
import time

from neuralnets.util.io import read_tif
from neuralnets.util.visualization import overlay_____no_output_____

As you will notice, Colab environments come with quite a large library pre-installed. If you need to import a module that is not yet specified, you can add it in the previous cell (make sure to run it again). If the module is not installed, you can install it with `pip`.

To make your work reproducible, it is advised to initialize all modules with stochastic functionality with a fixed seed. Re-running this script should give the same results as long as the seed is fixed. _____no_output_____

# make sure the results are reproducible
seed = 0
np.random.seed(seed)
random.seed(seed)
torch.manual_seed(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

# run all computations on the GPU if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print('Running computations with %s' % torch.device(device))
if torch.cuda.is_available():
    print(torch.cuda.get_device_properties(device))Running computations with cuda
_CudaDeviceProperties(name='Tesla T4', major=7, minor=5, total_memory=15079MB, multi_processor_count=40)

We will now download the required data from a public Google Drive repository. The data is stored as a zip archive and automatically extracted to the `data` directory in the current directory. _____no_output_____

# fields
url = 'http://data.bits.vib.be/pub/trainingen/DeepLearning/data-3.zip'
cmp_data_path = 'data.zip'

# download the compressed data
gdown.download(url, cmp_data_path, quiet=False)

# extract the data
zip = zipfile.ZipFile(cmp_data_path)
zip.extractall('')

# remove the compressed data
os.remove(cmp_data_path)Downloading...
From: http://data.bits.vib.be/pub/trainingen/DeepLearning/data-3.zip
To: /content/data.zip
100%|██████████| 14.9M/14.9M [00:01<00:00, 12.7MB/s]

## 2. Data visualization and pre-processing

The data used for this exercise session originates from an ISBI segmentation challenge on neuron structures in electron microscopy image stacks. The organizers provide a training volume and their corresponding labels. The test set is also provided, however predictions can be validated by uploading them to the challenge central server. This is to avoid overfitting the model on the test set.

Each data volume is provided as a .tif file that contains a 3D array. For the input data, these are simply the intensity values measured by the microscope. For the labels, this is a binary value: 0 for membrane, 255 for inner neuron structure.

We briefly visualize the labeled data. _____no_output_____

# specify where the data is stored
data_dir = 'data-3'

# load the datadispensers
x = read_tif(os.path.join(data_dir, 'train-volume.tif'))
y = read_tif(os.path.join(data_dir, 'train-labels.tif'))

# print out size
print('Size of the labeled volume: %d x %d x %d' % x.shape)

# show example
x_overlay = overlay(x[0] / 255, 1-(y[0]>0), colors=[(1, 0, 0)], alpha=0.4)
plt.subplot(1, 3, 1)
plt.imshow(x[0], cmap='gray')
plt.title('Input data')
plt.axis('off')
plt.subplot(1, 3, 2)
plt.imshow(y[0], cmap='gray')
plt.title('Labels')
plt.axis('off')
plt.subplot(1, 3, 3)
plt.imshow(x_overlay)
plt.title('Membrane overlay')
plt.axis('off')
plt.show()Size of the labeled volume: 30 x 512 x 512

Annotation of datasets like this typically involve lots of expertise and manual labour and is therefore extremely costly. This is a general issue in biomedical image-based datasets. For this reason, there has been increasing attention in developing automated segmentation techniques for biomedical imaging datasets. In the following, you will see how deep learning models can achieve relatively high accuracy on complex task such as neuronal structure segmentation.

As in practically any image analysis application, it is good common practice to rescale the data to the [0,1] interval, map the labels to subsequent values (0 and 1 in this case) and split the data in a train and test set. _____no_output_____

# normalize the data
x = (x - np.min(x)) / (np.max(x) - np.min(x))

# map the 255 labels to 1
y[y == 255] = 1

# split the data in a train and test set
test_ratio = 0.33 # we will use 33% of the data for testing
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=test_ratio, random_state=seed)

# print out size
print('Training volume: %d x %d x %d' % x_train.shape)
print('Testing volume: %d x %d x %d' % x_test.shape)

#
class_distribution = [np.sum(y_train == 0) / y_train.size, np.sum(y_train == 1) / y_train.size]
print('Class balance: ' )
print('    0: %.3f' % class_distribution[0])
print('    1: %.3f' % class_distribution[1])Training volume: 20 x 512 x 512
Testing volume: 10 x 512 x 512
Class balance:
    0: 0.221
    1: 0.779

## 3. Segmentation: a pixel classification problem

From a machine learning point of view, image segmentation can be seen as a classification problem. For each pixel in the image, the goal is to predict the corresponding class (membrane or non-membrane). Up to 2012, most techniques were based on extracting a set of features from a local or global region (e.g. intensity, edges, etc.) around the pixel and training a shallow classifier (e.g. a random forest). The choice of features would typically be the crucial factor and is different for each application.

Convolutional neural networks however, are able to solve this issue, as the feature extractor is learned based on the training data. To do this, we have to do two things: implement a dataset that extracts a local window for each pixel and train a CNN that performs binary classification. Let's start with the dataset.

**Exercise**: Implement the `EMWindowDataset` class:
- The `__init__` function should save the inputs and labels
- The `__getitem__` function should return a local window around the i'th pixel (use slice-by-slice raster ordering for this) and the corresponding label of that pixel. Note that extracting local windows near the bounds of the image may result in out-of-bound errors. You can omit this by [padding](https://numpy.org/doc/stable/reference/generated/numpy.pad.html) the data.
- The `__len__` function should ideally return the amount of pixels in the data. However, a single epoch would then require lots of iterations. Cap this with an upper bound `max_iter_epoch`.
- Data augmentation is not yet required.

_____no_output_____

# helper function: transform a linear index from a 3D array to 3D coordinates
# assuming slice-by-slice raster scanning ordering
def delinearize_index(i, sz):
    z_, y_, x_ = sz
    x = np.mod(i, x_)
    j = (i - x) // x_
    y = np.mod(j, y_)
    z = (j - y) // y_
    return z, y, x

# dataset useful for sampling (and many other things)
class EMWindowDataset(data.Dataset):

    def __init__(self, x, y, wnd_sz, max_iter_epoch):

        # window size
        self.wnd_sz = wnd_sz

        # offset (for padding)
        self.offset = (wnd_sz[0] // 2, wnd_sz[1] // 2)

        # size of the data
        self.size = x.shape

        # maximum number of iterations per epoch
        self.max_iter_epoch = max_iter_epoch

        # pad the data to avoid boundary issues
        self.x_padded = np.pad(x, ((0, 0), (self.offset[0], self.offset[0]), (self.offset[1], self.offset[1])), 'symmetric')

        # save the labels, no padding required
        self.y = y

    def __getitem__(self, i):

        # delinearize the index
        z, y, x = delinearize_index(i, self.size)

        # extract the window, don't forget the offsets
        oy, ox = self.offset
        wnd = self.x_padded[z:z + 1, y:y + 2 * oy + 1, x:x + 2 * ox + 1]

        return wnd, self.y[z, y, x]

    def __len__(self):
        return self.max_iter_epoch

# parameters
window_size = (32, 32)
batch_size = 256
max_iter_epoch = 2**12

# make an instance of the dataset for training and testing
ds_train = EMWindowDataset(x_train, y_train, window_size, max_iter_epoch)
ds_test = EMWindowDataset(x_test, y_test, window_size, max_iter_epoch)

# test the class
n = np.random.randint(len(ds_train))
wnd, label = ds_train[n]
plt.imshow(wnd[0], cmap='gray')
plt.title(label)
plt.show()

# setup the data loader
train_loader = DataLoader(ds_train, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(ds_test, batch_size=batch_size, shuffle=True)_____no_output_____

The positive side to segmentation labels is that each pixel corresponds to a single annotation. In contrast to usual classification dataset, this may give the impression that there are lots of labels. However, keep in mind that the data is heavily correlated, especially locally. In other words, neighboring pixel labels of a reference pixel won't provide that much more information than the reference pixel label.

The classification network that we will use is exactly the same as the one we used in the previous session. _____no_output_____

class ConvNormRelu(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size=3, padding=1):
        super(ConvNormRelu, self).__init__()
        self.unit = nn.Sequential()
        self.unit.add_module('conv', nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, padding=padding))
        self.unit.add_module('norm', nn.BatchNorm2d(int(out_channels)))
        self.unit.add_module('activation', nn.ReLU())

    def forward(self, inputs):
        return self.unit(inputs)

class CNN(nn.Module):
    def __init__(self, feature_maps=16):
        super(CNN, self).__init__()

        self.feature_maps = feature_maps

        self.conv1 = ConvNormRelu(in_channels=1, out_channels=feature_maps)
        self.conv2 = ConvNormRelu(in_channels=feature_maps, out_channels=feature_maps)
        self.conv3 = ConvNormRelu(in_channels=feature_maps, out_channels=feature_maps)
        self.conv4 = ConvNormRelu(in_channels=feature_maps, out_channels=feature_maps)
        self.pool = nn.MaxPool2d(2, 2)
        self.fc = nn.Linear(feature_maps*2*2, 2)

    def forward(self, x):
        x = self.pool(self.conv1(x))
        x = self.pool(self.conv2(x))
        x = self.pool(self.conv3(x))
        x = self.pool(self.conv4(x))
        x = x.view(-1, self.feature_maps*2*2)
        x = self.fc(x)
        return x

net = CNN(feature_maps=16)
print(net)CNN(
  (conv1): ConvNormRelu(
    (unit): Sequential(
      (conv): Conv2d(1, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (norm): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (activation): ReLU()
    )
  )
  (conv2): ConvNormRelu(
    (unit): Sequential(
      (conv): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (norm): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (activation): ReLU()
    )
  )
  (conv3): ConvNormRelu(
    (unit): Sequential(
      (conv): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (norm): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (activation): ReLU()
    )
  )
  (conv4): ConvNormRelu(
    (unit): Sequential(
      (conv): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (norm): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (activation): ReLU()
    )
  )
  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (fc): Linear(in_features=64, out_features=2, bias=True)
)

We will use an alternative metric for validation. The accuracy is a metric motivated from a classification point of view. A more suitable segmentation metric is the Dice coefficient:

$$
D = \frac{2 \times \left| Y \cap \hat{Y} \right|}{\left| Y \right| + \left| \hat{Y} \right|}
$$

where $Y$ and $\hat{Y}$ are the ground truth and predicted segmentation, respectively. If the prediction perfectly overlaps, the Dice coefficient will be 1. The code below illustrates the resulting segmentation of the best model so far and the corresponding dice score.

**Exercise**: Train the classification network for pixel-wise label prediction:
- Implement the `train_net`, `train_epoch` and `test_epoch`. To make life easier, you can reuse parts of the code of the previous exercise.
- Implement the `dice` function that computes the metric describe above. Save the model with the highest Dice score (averaged over the two classes).
- Evaluate the (average) Dice metric at the end of each test epoch and save it, so that it can be plotted (similar to the loss).
- Train the network for 20 epochs at a learning rate of 0.001. _____no_output_____

# dice coefficient implementation
def dice(y, y_pred):
    intersection = np.sum(y * y_pred)
    return 2 * intersection / (np.sum(y) + np.sum(y_pred))

# implementation of a single training epoch
def train_epoch(net, loader, loss_fn, optimizer):

    # set the network in training mode
    net.train()

    # keep track of the loss
    loss_cum = 0
    cnt = 0

    for i, data in enumerate(loader):

        # sample data
        x, y = data

        # transfer data to GPU and correct format
        x = x.float().to(device)
        y = y.long().to(device)

        # set all gradients equal to zero
        net.zero_grad()

        # feed the batch to the network and compute the outputs
        y_pred = net(x)

        # compare the outputs to the labels with the loss function
        loss = loss_fn(y_pred, y)
        loss_cum += loss.data.cpu().numpy()
        cnt += 1

        # backpropagate the gradients w.r.t. computed loss
        loss.backward()

        # apply one step in the optimization
        optimizer.step()

    # compute the average loss
    loss_avg = loss_cum / cnt

    return loss_avg

# implementation of a single testing epoch
def test_epoch(net, loader, loss_fn):

    # set the network in training mode
    net.eval()

    # keep track of the loss and predictions
    preds = np.zeros((len(loader.dataset), 2))
    ys = np.zeros((len(loader.dataset)))
    loss_cum = 0
    cnt = 0

    for i, data in enumerate(loader):

        # sample data
        x, y = data

        # transfer data to GPU and correct format
        x = x.float().to(device)
        y = y.long().to(device)

        # feed the batch to the network and compute the outputs
        y_pred = net(x)

        # compare the outputs to the labels with the loss function
        loss = loss_fn(y_pred, y)
        loss_cum += loss.data.cpu().numpy()
        cnt += 1

        # get the class probability predictions and save them for validation
        y_ = torch.softmax(y_pred, dim=1)
        b = i * loader.batch_size
        preds[b: b + y_.size(0), :] = y_.detach().cpu().numpy()
        ys[b: b + y_.size(0)] = y.detach().cpu().numpy()

    # compute accuracy
    d = 0.5 * (dice(1 - ys, 1 - preds.argmax(axis=1)) + dice(ys, preds.argmax(axis=1)))

    # compute the average loss
    loss_avg = loss_cum / cnt

    return loss_avg, d

def train_net(net, train_loader, test_loader, loss_fn, optimizer, epochs, log_dir):

    # transfer the network to the GPU
    net.to(device)

    best_dice = 0
    train_loss = np.zeros((epochs))
    test_loss = np.zeros((epochs))
    test_dice = np.zeros((epochs))
    for epoch in range(epochs):

        # training
        train_loss[epoch] = train_epoch(net, train_loader, loss_fn, optimizer)

        # testing
        test_loss[epoch], test_dice[epoch] = test_epoch(net, test_loader, loss_fn)

        # check if accuracy has increased
        if test_dice[epoch] > best_dice:
            best_dice = test_dice[epoch]

            # save the model
            torch.save(net.state_dict(), 'cnn_best.cpt')

        print('Epoch %5d - Train loss: %.6f - Test loss: %.6f - Test dice avg: %.6f' % (epoch, train_loss[epoch], test_loss[epoch], test_dice[epoch]))

    return train_loss, test_loss, test_dice

# parameters
learning_rate = 0.001
n_epochs = 20
log_dir = '.'

# define the optimizer
class_weights = torch.from_numpy(np.divide(1, class_distribution)).float().to(device)
class_weights = class_weights / class_weights.sum()
loss_fn = nn.CrossEntropyLoss(weight=class_weights)
# loss_fn = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=learning_rate)

# start training
train_loss_cnn, test_loss_cnn, test_dice_cnn = train_net(net, train_loader, test_loader, loss_fn, optimizer, n_epochs, log_dir)

# show the training curve and accuracy
plt.subplot(1, 2, 1)
plt.plot(train_loss_cnn)
plt.plot(test_loss_cnn)
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend(('Train', 'Test'))
plt.subplot(1, 2, 2)
plt.plot(test_dice_cnn)
plt.xlabel('Epochs')
plt.ylabel('Dice avg')
plt.show()Epoch     0 - Train loss: 0.414006 - Test loss: 0.693262 - Test dice avg: 0.441277
Epoch     1 - Train loss: 0.230934 - Test loss: 0.721943 - Test dice avg: 0.497957
Epoch     2 - Train loss: 0.161893 - Test loss: 0.673282 - Test dice avg: 0.692215
Epoch     3 - Train loss: 0.119127 - Test loss: 0.639826 - Test dice avg: 0.738619
Epoch     4 - Train loss: 0.091824 - Test loss: 0.601032 - Test dice avg: 0.754433
Epoch     5 - Train loss: 0.075326 - Test loss: 0.657208 - Test dice avg: 0.755952
Epoch     6 - Train loss: 0.062996 - Test loss: 0.788308 - Test dice avg: 0.739927
Epoch     7 - Train loss: 0.054004 - Test loss: 0.871502 - Test dice avg: 0.744548
Epoch     8 - Train loss: 0.043278 - Test loss: 0.694500 - Test dice avg: 0.766866
Epoch     9 - Train loss: 0.039117 - Test loss: 0.965212 - Test dice avg: 0.726596
Epoch    10 - Train loss: 0.030984 - Test loss: 0.829635 - Test dice avg: 0.747853
Epoch    11 - Train loss: 0.029167 - Test loss: 0.869089 - Test dice avg: 0.745000
Epoch    12 - Train loss: 0.024234 - Test loss: 1.022034 - Test dice avg: 0.735058
Epoch    13 - Train loss: 0.021572 - Test loss: 1.028657 - Test dice avg: 0.741081
Epoch    14 - Train loss: 0.016402 - Test loss: 1.012114 - Test dice avg: 0.740054
Epoch    15 - Train loss: 0.013843 - Test loss: 0.995476 - Test dice avg: 0.740326
Epoch    16 - Train loss: 0.013747 - Test loss: 1.122460 - Test dice avg: 0.733761
Epoch    17 - Train loss: 0.011495 - Test loss: 1.075303 - Test dice avg: 0.738531
Epoch    18 - Train loss: 0.010605 - Test loss: 1.338515 - Test dice avg: 0.718386
Epoch    19 - Train loss: 0.008500 - Test loss: 1.205790 - Test dice avg: 0.737104

You should obtain an average Dice score between 0.70 and 0.75. However, note that according to the learning curves, the model seems to be overfitting relatively fast. This is mainly due to the locality of the feature extraction. We will now illustrate how a test sample can be segmented with the trained network.

**Exercise**: Implement the `segment_slice` function:
- The function takes a 2D slice, a pretrained network, a window size and batch size as input, and computes the segmentation (a binary 2D array).
- The easiest way to loop through all the pixels of an image is by using the `EMWindowDataset` without shuffling. However, you will have to adjust the maximum number of iterations.
- As a first step, you can assume `batch_size=1`. Keep in mind, this can be inefficient, because GPUs become beneficial as the amount of parallel operations increases. Higher batch sizes therefore benefit computing time, but this is of course bounded to the available GPU memory. _____no_output_____

def segment_slice(x, net, window_size, batch_size):

    # setup data loader
    x_orig = x
    ds = EMWindowDataset(x[np.newaxis, ...], np.zeros_like(x[np.newaxis, ...]), window_size, x.size)
    loader = DataLoader(ds, batch_size=batch_size)

    # set the network in training mode
    net.eval()

    # keep track of the loss and predictions
    preds = np.zeros((len(loader.dataset), 2))

    for i, data in enumerate(loader):

        # sample data
        x, _ = data

        # transfer data to GPU and correct format
        x = x.float().to(device)

        # feed the batch to the network and compute the outputs
        y_pred = net(x)

        # get the class probability predictions and save them for validation
        y_ = torch.softmax(y_pred, dim=1)
        b = i * loader.batch_size
        preds[b: b + y_.size(0), :] = y_.detach().cpu().numpy()

    # compute the segmentation
    preds = preds.argmax(axis=1)
    segmentation = np.reshape(preds, x_orig.shape)

    return segmentation.astype('uint8')

# load the best parameters
state_dict = torch.load('cnn_best.cpt')
net.load_state_dict(state_dict)

# perform segmentation
n = 0
t = time.time()
y_pred = segment_slice(x_test[n], net, window_size, batch_size)
print('Elapsed time: %.2f seconds' % (time.time() - t))

# show example
x_overlay = overlay(x_test[n], 1 - y_test[n], colors=[(1, 0, 0)], alpha=0.4)
x_pred_overlay = overlay(x_test[n], 1 - y_pred, colors=[(1, 0, 0)], alpha=0.4)
plt.subplot(1, 2, 1)
plt.imshow(x_pred_overlay)
plt.title('Predictions')
plt.axis('off')
plt.subplot(1, 2, 2)
plt.imshow(x_overlay)
plt.title('Labels')
plt.axis('off')
plt.show()Elapsed time: 5.41 seconds

Visually, the segmentation result does not look perfect. Clearly, the network is making lots of mistakes, especially along membranes, because these regions require more context.

Another disadvantage to this methodology is the computational inefficiency. Even with larger batches, segmentation of a relatively small image patch can take seconds, which is impractical for larger datasets. The reason is obviously that for each pixel, a forward call of the network is required. _____no_output_____## 4. Building a U-Net with PyTorch

The lack of global context and computational efficiency of the pixel classification approach results in poor practical performance. As an alternative, the [U-Net](https://link.springer.com/chapter/10.1007/978-3-319-24574-4_28) network was proposed. This is an end-to-end segmentation network that takes an image as input and computes a complete segmentation of the input. Let's start by defining the network architecture:

![U-Net](https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/u-net-architecture.png)

**Exercise**: Implement the U-Net architecture:
- The architecture consists of an encoder, a bottleneck, a decoder and skip connections between the encoder and decoder.
- The basic building blocks of the U-Net architecture are two consecutive convolutional layers with ReLU activation that take $n$ feature maps and output $m$ feature maps. The convolutional layers have $3 \times 3$ kernels. For simplicity we will also pad the inputs by 1 pixel to make sure the inputs and outputs have the same size. Implement this block in the `ConvBlock` class.
- Downsampling is relatively simply with $2 \times 2$ max-pooling. However, upsampling is performed using bilinear upsampling layers, followed by a convolution layer. This operation has been implemented in the [`ConvTranspose2d`](https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html) class with $2 \times 2$ kernels and a stride of $2$.
- Implement the skip connections by using the PyTorch [concatenation](https://pytorch.org/docs/stable/generated/torch.cat.html) function.
- The initial number of feature maps is a parameter that can be chosen. From then on, the amount of feature maps doubles with every `ConvBlock` in the encoder and halves with every `ConvBlock` in the decoder. _____no_output_____

class ConvBlock(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size=3, padding=1):
        super(ConvBlock, self).__init__()
        self.unit = nn.Sequential()
        self.unit.add_module('conv1', ConvNormRelu(in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size, padding=padding))
        self.unit.add_module('conv2', ConvNormRelu(in_channels=out_channels, out_channels=out_channels, kernel_size=kernel_size, padding=padding))

    def forward(self, inputs):
        return self.unit(inputs)

class UNet(nn.Module):
    def __init__(self, feature_maps=16, out_channels=2):
        super(UNet, self).__init__()

        self.feature_maps = feature_maps

        # encoder layers
        self.conv_enc_1 = ConvBlock(in_channels=1, out_channels=feature_maps)
        self.conv_enc_2 = ConvBlock(in_channels=feature_maps, out_channels=2*feature_maps)
        self.conv_enc_3 = ConvBlock(in_channels=2*feature_maps, out_channels=4*feature_maps)
        self.conv_enc_4 = ConvBlock(in_channels=4*feature_maps, out_channels=8*feature_maps)
        self.pool = nn.MaxPool2d(2, 2)

        # bottleneck layers
        self.conv_btl = ConvBlock(in_channels=8*feature_maps, out_channels=16*feature_maps)

        # decoder layers
        self.conv_dec_4_up = nn.ConvTranspose2d(in_channels=16*feature_maps, out_channels=8*feature_maps, kernel_size=2, stride=2)
        self.conv_dec_4 = ConvBlock(in_channels=16*feature_maps, out_channels=8*feature_maps)
        self.conv_dec_3_up = nn.ConvTranspose2d(in_channels=8*feature_maps, out_channels=4*feature_maps, kernel_size=2, stride=2)
        self.conv_dec_3 = ConvBlock(in_channels=8*feature_maps, out_channels=4*feature_maps)
        self.conv_dec_2_up = nn.ConvTranspose2d(in_channels=4*feature_maps, out_channels=2*feature_maps, kernel_size=2, stride=2)
        self.conv_dec_2 = ConvBlock(in_channels=4*feature_maps, out_channels=2*feature_maps)
        self.conv_dec_1_up = nn.ConvTranspose2d(in_channels=2*feature_maps, out_channels=feature_maps, kernel_size=2, stride=2)
        self.conv_dec_1 = ConvBlock(in_channels=2*feature_maps, out_channels=feature_maps)

        # final segmentation layer
        self.conv_final = nn.Conv2d(in_channels=feature_maps, out_channels=out_channels, kernel_size=1)

    def forward(self, x):

        # encoder path
        x1 = self.conv_enc_1(x)
        x = self.pool(x1)
        x2 = self.conv_enc_2(x)
        x = self.pool(x2)
        x3 = self.conv_enc_3(x)
        x = self.pool(x3)
        x4 = self.conv_enc_4(x)
        x = self.pool(x4)

        # bottleneck
        x = self.conv_btl(x)

        # decoder path
        x = torch.cat((self.conv_dec_4_up(x), x4), dim=1)
        x = self.conv_dec_4(x)
        x = torch.cat((self.conv_dec_3_up(x), x3), dim=1)
        x = self.conv_dec_3(x)
        x = torch.cat((self.conv_dec_2_up(x), x2), dim=1)
        x = self.conv_dec_2(x)
        x = torch.cat((self.conv_dec_1_up(x), x1), dim=1)
        x = self.conv_dec_1(x)

        # final segmentation
        x = self.conv_final(x)

        return x

net = UNet(feature_maps=64)
print(net)UNet(
  (conv_enc_1): ConvBlock(
    (unit): Sequential(
      (conv1): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(1, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
      (conv2): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
    )
  )
  (conv_enc_2): ConvBlock(
    (unit): Sequential(
      (conv1): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
      (conv2): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
    )
  )
  (conv_enc_3): ConvBlock(
    (unit): Sequential(
      (conv1): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
      (conv2): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
    )
  )
  (conv_enc_4): ConvBlock(
    (unit): Sequential(
      (conv1): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
      (conv2): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
    )
  )
  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (conv_btl): ConvBlock(
    (unit): Sequential(
      (conv1): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(512, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
      (conv2): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
    )
  )
  (conv_dec_4_up): ConvTranspose2d(1024, 512, kernel_size=(2, 2), stride=(2, 2))
  (conv_dec_4): ConvBlock(
    (unit): Sequential(
      (conv1): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
      (conv2): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
    )
  )
  (conv_dec_3_up): ConvTranspose2d(512, 256, kernel_size=(2, 2), stride=(2, 2))
  (conv_dec_3): ConvBlock(
    (unit): Sequential(
      (conv1): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(512, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
      (conv2): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
    )
  )
  (conv_dec_2_up): ConvTranspose2d(256, 128, kernel_size=(2, 2), stride=(2, 2))
  (conv_dec_2): ConvBlock(
    (unit): Sequential(
      (conv1): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(256, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
      (conv2): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
    )
  )
  (conv_dec_1_up): ConvTranspose2d(128, 64, kernel_size=(2, 2), stride=(2, 2))
  (conv_dec_1): ConvBlock(
    (unit): Sequential(
      (conv1): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(128, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
      (conv2): ConvNormRelu(
        (unit): Sequential(
          (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (norm): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activation): ReLU()
        )
      )
    )
  )
  (conv_final): Conv2d(64, 2, kernel_size=(1, 1), stride=(1, 1))
)

Of course, this network requires an image as input and the corresponding label image as output. Therefore, we have to modify our dataset.

**Exercise**: Modify the `EMWindowDataset` class:
- Implement the `__getitem__` method, the remaining functions have already been implemented. _____no_output_____

# dataset useful for sampling
class EMWindowDataset(data.Dataset):

    def __init__(self, x, y, wnd_sz, max_iter_epoch):

        # window size
        self.wnd_sz = wnd_sz

        # maximum number of iterations per epoch
        self.max_iter_epoch = max_iter_epoch

        # save the data
        self.x = x

        # save the labels
        self.y = y

    def __getitem__(self, i):

        # generate random location in the data
        z = np.random.randint(self.x.shape[0])
        y = np.random.randint(self.x.shape[1] - self.wnd_sz[0] + 1)
        x = np.random.randint(self.x.shape[2] - self.wnd_sz[1] + 1)

        # extract the input and label windows
        wnd_x = self.x[z:z + 1, y:y + self.wnd_sz[0], x:x + self.wnd_sz[1]]
        wnd_y = self.y[z:z + 1, y:y + self.wnd_sz[0], x:x + self.wnd_sz[1]]

        return wnd_x, wnd_y

    def __len__(self):
        return self.max_iter_epoch

# parameters
window_size = (128, 128)
batch_size = 2
max_iter_epoch = 128

# make an instance of the dataset for training and testing
ds_train = EMWindowDataset(x_train, y_train, window_size, max_iter_epoch)
ds_test = EMWindowDataset(x_test, y_test, window_size, max_iter_epoch)

# test the class
n = np.random.randint(len(ds_train))
x, y = ds_train[n]
x_overlay = overlay(x[0], 1 - y[0], colors=[(1, 0, 0)], alpha=0.4)
plt.imshow(x_overlay)
plt.show()

# setup the data loader
train_loader = DataLoader(ds_train, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(ds_test, batch_size=batch_size, shuffle=True)_____no_output_____

## 5. Training & validating the network

Now that we have a U-Net network and a data loader, it is time to train the network! For the sake of repetitiveness, you are not required to implement the complete training loop.

**Exercise**: Train the U-Net architecture:
- Have a look at the training loop code and identify the differences with classical CNN training.
- Train the network for 20 epochs with a learning rate of 0.001. _____no_output_____

# implementation of a single training epoch
def train_epoch(net, loader, loss_fn, optimizer):

    # set the network in training mode
    net.train()

    # keep track of the loss
    loss_cum = 0
    cnt = 0

    for i, data in enumerate(loader):

        # sample data
        x, y = data

        # transfer data to GPU and correct format
        x = x.float().to(device)
        y = y.long()[:, 0, :, :].to(device)

        # set all gradients equal to zero
        net.zero_grad()

        # feed the batch to the network and compute the outputs
        y_pred = net(x)

        # compare the outputs to the labels with the loss function
        loss = loss_fn(y_pred, y)
        loss_cum += loss.data.cpu().numpy()
        cnt += 1

        # backpropagate the gradients w.r.t. computed loss
        loss.backward()

        # apply one step in the optimization
        optimizer.step()

    # compute the average loss
    loss_avg = loss_cum / cnt

    return loss_avg

# implementation of a single testing epoch
def test_epoch(net, loader, loss_fn):

    # set the network in training mode
    net.eval()

    # keep track of the loss and predictions
    preds = np.zeros((len(loader.dataset), *loader.dataset.wnd_sz))
    ys = np.zeros((len(loader.dataset), *loader.dataset.wnd_sz))
    loss_cum = 0
    cnt = 0

    for i, data in enumerate(loader):

        # sample data
        x, y = data

        # transfer data to GPU and correct format
        x = x.float().to(device)
        y = y.long()[:, 0, :, :].to(device)

        # feed the batch to the network and compute the outputs
        y_pred = net(x)

        # compare the outputs to the labels with the loss function
        loss = loss_fn(y_pred, y)
        loss_cum += loss.data.cpu().numpy()
        cnt += 1

        # get the class probability predictions and save them for validation
        y_ = torch.softmax(y_pred, dim=1)
        b = i * loader.batch_size
        preds[b: b + y_.size(0), ...] = y_.argmax(dim=1).detach().cpu().numpy()
        ys[b: b + y_.size(0), ...] = y.detach().cpu().numpy()

    # compute accuracy
    d = dice(ys.flatten(), preds.flatten())

    # compute the average loss
    loss_avg = loss_cum / cnt

    return loss_avg, d

def train_net(net, train_loader, test_loader, loss_fn, optimizer, epochs, log_dir):

    # transfer the network to the GPU
    net.to(device)

    best_dice = 0
    train_loss = np.zeros((epochs))
    test_loss = np.zeros((epochs))
    test_dice = np.zeros((epochs))
    for epoch in range(epochs):

        # training
        train_loss[epoch] = train_epoch(net, train_loader, loss_fn, optimizer)

        # testing
        test_loss[epoch], test_dice[epoch] = test_epoch(net, test_loader, loss_fn)

        # check if accuracy has increased
        if test_dice[epoch] > best_dice:
            best_dice = test_dice[epoch]

            # save the model
            torch.save(net.state_dict(), 'unet_best.cpt')

        print('Epoch %5d - Train loss: %.6f - Test loss: %.6f - Test dice avg: %.6f' % (epoch, train_loss[epoch], test_loss[epoch], test_dice[epoch]))

    return train_loss, test_loss, test_dice

# parameters
learning_rate = 0.001
n_epochs = 20
log_dir = '.'

# define the optimizer
class_weights = torch.from_numpy(np.divide(1, class_distribution)).float().to(device)
class_weights = class_weights / class_weights.sum()
loss_fn = nn.CrossEntropyLoss(weight=class_weights)
# loss_fn = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=learning_rate)

# start training
train_loss_unet, test_loss_unet, test_dice_unet = train_net(net, train_loader, test_loader, loss_fn, optimizer, n_epochs, log_dir)

# show the training curve and accuracy
plt.subplot(1, 2, 1)
plt.plot(train_loss_cnn)
plt.plot(test_loss_cnn)
plt.plot(train_loss_unet)
plt.plot(test_loss_unet)
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend(('Train (CNN)', 'Test (CNN)', 'Train (U-Net)', 'Test (U-Net)'))
plt.subplot(1, 2, 2)
plt.plot(test_dice_cnn)
plt.plot(test_dice_unet)
plt.xlabel('Epochs')
plt.ylabel('Dice avg')
plt.legend(('CNN', 'U-Net'))
plt.show()Epoch     0 - Train loss: 0.383133 - Test loss: 0.354965 - Test dice avg: 0.856172
Epoch     1 - Train loss: 0.361292 - Test loss: 0.352362 - Test dice avg: 0.902300
Epoch     2 - Train loss: 0.324768 - Test loss: 0.306067 - Test dice avg: 0.887419
Epoch     3 - Train loss: 0.325949 - Test loss: 0.334848 - Test dice avg: 0.850665
Epoch     4 - Train loss: 0.328289 - Test loss: 0.310259 - Test dice avg: 0.913652
Epoch     5 - Train loss: 0.315503 - Test loss: 0.305358 - Test dice avg: 0.878493
Epoch     6 - Train loss: 0.316873 - Test loss: 0.324832 - Test dice avg: 0.908888
Epoch     7 - Train loss: 0.303972 - Test loss: 0.295798 - Test dice avg: 0.932093
Epoch     8 - Train loss: 0.305955 - Test loss: 0.300341 - Test dice avg: 0.886833
Epoch     9 - Train loss: 0.288136 - Test loss: 0.260361 - Test dice avg: 0.931314
Epoch    10 - Train loss: 0.273979 - Test loss: 0.266878 - Test dice avg: 0.914591
Epoch    11 - Train loss: 0.282206 - Test loss: 0.261618 - Test dice avg: 0.903781
Epoch    12 - Train loss: 0.275372 - Test loss: 0.256586 - Test dice avg: 0.916227
Epoch    13 - Train loss: 0.274017 - Test loss: 0.276006 - Test dice avg: 0.896439
Epoch    14 - Train loss: 0.266158 - Test loss: 0.303701 - Test dice avg: 0.896790
Epoch    15 - Train loss: 0.280715 - Test loss: 0.256113 - Test dice avg: 0.938594
Epoch    16 - Train loss: 0.280409 - Test loss: 0.281226 - Test dice avg: 0.897781
Epoch    17 - Train loss: 0.258447 - Test loss: 0.248680 - Test dice avg: 0.929210
Epoch    18 - Train loss: 0.269229 - Test loss: 0.516527 - Test dice avg: 0.653548
Epoch    19 - Train loss: 0.259088 - Test loss: 0.263776 - Test dice avg: 0.906309

That should look much better! The U-Net architecture is significantly less suffering from overfitting and the mean Dice coefficient is substantially higher (should be approximately 0.90).

Now let's have a look how the actual predictions look like.

**Exercise**: Modify the `segment_slice` so that it segments a 2D image with a U-Net network:
- Simplify the original code of `segment_slice` by propagating the image straightforward through the network.
- Can you figure out why alternative image sizes can also be fed into the network? Is that always possible? _____no_output_____

def segment_slice(x, net):

    # set the network in training mode
    net.eval()

    # convert to torch tensor
    x = torch.from_numpy(x[np.newaxis, np.newaxis, ...]).float().to(device)

    # forward propagation
    pred = net(x).detach().cpu().numpy()

    # compute the segmentation
    segmentation = pred.argmax(axis=1)[0]

    return segmentation

# load the best parameters
state_dict = torch.load('unet_best.cpt')
net.load_state_dict(state_dict)

# perform segmentation
n = 0
t = time.time()
y_pred = segment_slice(x_test[n], net)
print('Elapsed time: %.2f seconds' % (time.time() - t))

# show example
x_overlay = overlay(x_test[n], 1 - y_test[n], colors=[(1, 0, 0)], alpha=0.4)
x_pred_overlay = overlay(x_test[n], 1 - y_pred, colors=[(1, 0, 0)], alpha=0.4)
plt.subplot(1, 2, 1)
plt.imshow(x_pred_overlay)
plt.title('Predictions')
plt.axis('off')
plt.subplot(1, 2, 2)
plt.imshow(x_overlay)
plt.title('Labels')
plt.axis('off')
plt.show()Elapsed time: 0.10 seconds

That is a much more usable segmentation result! Also note how much faster the segmentation computation is. The U-Net architecture excells in biomedical datasets because it can cope relatively well with small amounts of training data. If you have time left, you can always try to improve the results by adding data augmentation! _____no_output_____


# Notebook from willfrank98/3100-Models_of_Computation
Path: notebooks/tutorial/A4JSoln.ipynb

** IMPORTANT: Update your Imports/ directory! **

Update your Imports directory with these files replacing what you have:

1. Def_RE2NFA.py  - fixed a few lines to generate state names in a new way

2. StateNameSanitizers.py - fixed a few lines to generate state names in a new way

3. Def_DFA.py - fixed comp_dfa which was slightly 'non-kosher'

** IMPORTANT: Now download A4JSoln.ipynb! **

Run all the cells of A4JSoln.ipynb when you do the exercises below. If any crashes, let us know asap. Else
you are good to go._____no_output_____# Asg4: Context-free Parsing and Derivative Scanning

This Jove file covers two topics. The first, context-free parsing, helps us design a parser for regular expressions. In your assignment, you are to experiment with context-free parsing as well as derivative scanning._____no_output_____## Context-free Parsing

We will present the parser for regular expressions. We closely follow the conventions set forth by David Beazley, the author of PLY. Please refer to some of the excellent tutorials by Dave at http://www.dabeaz.com/ply/ to further dig into how to design lexing and parsing rules.

We will first strive to support a basic context-free grammar that allows the following types of regular expressions, given by these context-free grammar rules:

R -> Epsilon | char | RR | R+R | (R) | R*

Of course we can't encode this very grammar directly, as it is ambiguous.

For instance, ab+cd can be parsed as

* (ab) + (cd)
* a (b+c) d
* a (b+(cd))
* ((ab)+c))d

To avoid ambiguity, we layer the grammar in the usual way. We will provide the grammar
finally encoded, below.

We then will try and extend the CFG to support two additional operators:

* The regular expression negation operator, !

* The regular expression conjunction operator, &

The addition of these operators will enrich the grammar to include !R and R&R also as regular expressions.

The final grammar, as generated by 'parser.out' (automatically generated by PLY) is below.

** Note: parser.out is worth studying in depth during a compiler course. ** We won't have the occasion to lead you through it.

'''

## This is the parser.out file created for our RE CFG:

Created by PLY version 3.4 (http://www.dabeaz.com/ply)

Grammar

Rule 0     S' -> expression

Rule 1     expression -> expression PLUS catexp

Rule 2     expression -> catexp

Rule 3     catexp -> catexp andexp

Rule 4     catexp -> andexp

Rule 5     andexp -> andexp AND ordyexp

Rule 6     andexp -> ordyexp

Rule 7     ordyexp -> ordyexp STAR

Rule 8     ordyexp -> NOT ordyexp

Rule 9     ordyexp -> LPAREN expression RPAREN

Rule 10    ordyexp -> EPS

Rule 11    ordyexp -> STR

Terminals, with rules where they appear

AND                  : 5

EPS                  : 10

LPAREN               : 9

NOT                  : 8

PLUS                 : 1

RPAREN               : 9

STAR                 : 7

STR                  : 11

error                :

Nonterminals, with rules where they appear

andexp               : 3 4 5

catexp               : 1 2 3

expression           : 1 9 0

ordyexp              : 5 6 7 8

Parsing method: LALR

'''_____no_output_____

from lex import lex
from yacc import yacc
from jove.StateNameSanitizers import ResetStNum, NxtStateStr
from jove.SystemImports       import *_____no_output_____

## Token definitions

This is the lexer for REs. We begin with token definitions_____no_output_____

tokens = ('EPS','STR','LPAREN','RPAREN','PLUS','STAR', 'NOT', 'AND') # A4J: NOT and AND were commented out before.

# Tokens
t_PLUS    = r'\+'
t_STAR    = r'\*'
t_LPAREN  = r'\('
t_RPAREN  = r'\)'
t_EPS     = r'\'\'|\"\"'
t_STR     = r'[a-zA-Z0-9]'
t_NOT     = r'\!' # A4J: ADD THE REGULAR EXPRESSION FOR NEGATION HERE
t_AND     = r'\&' # A4J: ADD THE REGULAR EXPRESSION FOR CONJUNCTION HERE

# Ignored characters
t_ignore = " \t"

def t_newline(t):
    r'\n+'
    t.lexer.lineno += t.value.count("\n")

def t_error(t):
    print("Illegal character '%s'" % t.value[0])
    t.lexer.skip(1)
    _____no_output_____

# Exercise: Extended CFG for REs

** Q1(a): **

Extend the CFG to include "&" and "!", by uncommenting various grammar definition steps, below.

Look for hints marked "A4J" below to uncomment. (This is to make you go through the process in a hand-held way.)

Your answer simply consists of writing below what all you uncommented. You may state what you uncommented, showing the uncommented result, below. Mention why you are uncommenting it (what this new definitions helps with).
_____no_output_____
** Q1(a) Answer: **

1. Uncommented to include 'NOT' and 'AND' here:

   > tokens = ('EPS','STR','LPAREN','RPAREN','PLUS','STAR', 'NOT', 'AND')

2. Uncommented to include these token rules (defined via regular expressions):

   > t_NOT     = r'\!' # A4J: ADD THE REGULAR EXPRESSION FOR NEGATION HERE

   > t_AND     = r'\&' # A4J: ADD THE REGULAR EXPRESSION FOR CONJUNCTION HERE

3. Uncommented AND and NOT within the precedence rules:

> precedence = (

>    ('left','PLUS'),

>    ('left', 'AND'),

>    ('left','STAR'),

>    ('right','NOT')

>    )
_____no_output_____## These parsing rules specify many things.

We begin with operator precedence rules that are essentially to help the "LALR parser" (also known as the bottom-up parser) resolve 'shift-reduce conflicts'._____no_output_____

# Parsing rules

# A4J: Suggested new precedence rules for AND and NOT are in comments below.
# Uncomment the AND and NOT lines in your extension
#
precedence = (
   ('left','PLUS'),
   ('left', 'AND'),  # A4J: AND was not included initially
   ('left','STAR'),
   ('right','NOT')   # A4J: NOT was not included initially
   )_____no_output_____

## CFG productions and semantic actions

These Python functions whose names begin with "p_" house (1) the CFG production rules within their documentation strings. (2) the semantic actions within their body. The semantic actions can refer to grammar symbol attributes within CFG productions. We will explain one of these rules now.

Take the rules

 expression -> expression PLUS catexp
 expression -> catexp

1) This function defines the first production rule

def p_expression_plus(t):

   a) This comment string expresses the production rule

    '''expression : expression PLUS catexp'''

   b) This line below tells us that the occurrence of 'expression' on
      the left-hand side is marked t[0], and its value is determined by
      applying function attrDyadicInfix onto its three arguments below.
      Here, t[1] is the attribute of 'expression' coming after the colon (:)
      and the attribute of catexp is t[3]

    t[0] = attrDyadicInfix("+", t[1], t[3])

2) This function expresses the second related production rule where the
   basis case

def p_expression_plus1(t):
    '''expression : catexp'''

    t[0] = t[1]  _____no_output_____

def p_expression_plus(t):
    'expression : expression PLUS catexp'
    #
    t[0] = attrDyadicInfix("+", t[1], t[3])

def p_expression_plus1(t):
    'expression : catexp'
    #
    t[0] = t[1]

def p_expression_cat(t):
    'catexp :  catexp andexp'
    #
    t[0] = attrDyadicInfix(".", t[1], t[2])

def p_expression_cat1(t):
    'catexp :  andexp'
    #
    t[0] = t[1]

# A4J: The 'andexp AND ordyexp' rule was commented out.
# We uncommented the same.

def p_expression_ordy(t):
    'andexp : andexp AND ordyexp'
    #
    t[0] = attrDyadicInfix("&", t[1], t[3])

def p_expression_ordy1(t):
    'andexp : ordyexp'
    #
    t[0] = t[1]

# We employ field 'ast' of the dict to record the abstract syntax tree.
# Field 'dig' holds a digraph. It too is a dict.
# Its fields are nl for the node list and el for the edge list

def p_expression_ordy_star(t):
    'ordyexp : ordyexp STAR'
    #
    ast = ('*', t[1]['ast'])

    nlin = t[1]['dig']['nl']
    elin = t[1]['dig']['el']

    rootin = nlin[0]

    root = NxtStateStr("R*_")
    right = NxtStateStr("*_")

    t[0] = {'ast' : ast,
            'dig' : {'nl' : [root] + nlin + [right], # this order important for proper layout!
                     'el' : elin + [ (root, rootin),
                                     (root, right) ]
                    }}

# A4J: The 'NOT ordyexp' rule was commented out.
# We uncommented the same.

def p_expression_ordy_not(t):
    'ordyexp : NOT ordyexp'
    #
    ast  = ('!', t[2]['ast'])

    nlin = t[2]['dig']['nl']
    elin = t[2]['dig']['el']

    rootin = nlin[0]

    root = NxtStateStr("!R_")
    left = NxtStateStr("!_")

    t[0] = {'ast' : ast,
            'dig' : {'nl' : [ root, left ] + nlin, # this order important for proper layout!
                     'el' : elin + [ (root, left),
                                     (root, rootin) ]
                    }}

def p_expression_ordy_paren(t):
    'ordyexp : LPAREN expression RPAREN'
    #
    ast  = t[2]['ast']

    nlin = t[2]['dig']['nl']
    elin = t[2]['dig']['el']

    rootin = nlin[0]

    root = NxtStateStr("(R)_")
    left = NxtStateStr("(_")
    right= NxtStateStr(")_")

    t[0] = {'ast' : ast,
            'dig' : {'nl' : [root, left] + nlin + [right], #order important f. proper layout!
                     'el' : elin + [ (root, left),
                                     (root, rootin),
                                     (root, right) ]
                    }}

def p_expression_ordy_eps(t):
    'ordyexp : EPS'
    #
    strn = '@'
    ast  = ('@', strn)
    t[0] = { 'ast' : ast,
             'dig' : {'nl' : [ strn + NxtStateStr("_") ],
                      'el' : []
                     }}

def p_expression_ordy_str(t):
    'ordyexp : STR'
    #
    strn = t[1]
    ast  = ('str', strn)
    t[0] = {'ast' : ast,
            'dig' : {'nl' : [ strn + NxtStateStr("_") ],
                     'el' : []
                    }}

def p_error(t):
    print("Syntax error at '%s'" % t.value)

#--

def attrDyadicInfix(op, attr1, attr3):
    ast  = (op, (attr1['ast'], attr3['ast']))

    nlin1 = attr1['dig']['nl']
    nlin3 = attr3['dig']['nl']
    nlin  = nlin1 + nlin3

    elin1 = attr1['dig']['el']
    elin3 = attr3['dig']['el']
    elin  = elin1 + elin3

    rootin1 = nlin1[0]
    rootin3 = nlin3[0]

    root   = NxtStateStr("R1"+op+"R2"+"_") # NxtStateStr("$_")
    left   = rootin1
    middle = NxtStateStr(op+"_")
    right  = rootin3

    return {'ast' : ast,
            'dig' : {'nl' : [ root, left, middle, right ] + nlin,
                     'el' : elin + [ (root, left),
                                     (root, middle),
                                     (root, right) ]
                     }}

#===
# This is the main function in this Jove file. Give
#===

def parseRE(s):
    """In: a string s containing a regular expression.
       Out: An attribute triple consisting of
            1) An abstract syntax tree suitable for processing in the derivative-based scanner
            2) A node-list for the parse-tree digraph generated. Good for drawing a parse tree
               using the drawPT function below
            3) An edge list for the parse-tree generated (again good for drawing using the
               drawPT function below)
    """
    mylexer  = lex()
    myparser = yacc()
    pt = myparser.parse(s, lexer = mylexer)
    return (pt['ast'], pt['dig']['nl'], pt['dig']['el'])

def drawPT(ast_nl_el, comment="PT"):
    """Given an (ast, nl, el) triple where nl is the node and el the edge-list,
       draw the Parse Tree by returning a dot object.
    """
    (ast, nl, el) = ast_nl_el
    print("Drawing AST for ", ast)
    dotObj_pt = Digraph(comment)
    dotObj_pt.graph_attr['rankdir'] = 'TB'
    for n in nl:
        prNam = n.split('_')[0]
        dotObj_pt.node(n, prNam, shape="oval", peripheries="1")
    for e in el:
        dotObj_pt.edge(e[0], e[1])
    return dotObj_pt_____no_output_____

# Exercise: Study of Parsing by Drawing Parse Trees

** Question Q1(b): **

Some simple parse-tree examples are now given.

Please produce __three__ more interesting-looking parser-trees as described, __after you include AND and NOT ( as asked of you in Q1(a))__.

* One parse-tree must include '!' and '*' in one expression. For this, you must try "!!ab**"  and explain the resulting parse tree in terms of whether it is doing the right thing.

 > Comment on the precedences between '!' and concatenation. Also comment on the precedences between '*' and concatentation. Also comment on the associativity of '!' and '*'.

* Another parse-tree must include + and & in one expression. For this, you must try "ab+cd&ef" and explain the resulting parse tree in terms of whether it is doing the right thing.

 > Comment on the precedences between '+' , concatenation, and '&'

* A third one can be a complicated RE. However, try to limit yourself to about eight leaf nodes and about the same number of operators (rough guideline only). Write two short sentences describing the parse-tree for your expression, and its choice of operators._____no_output_____

drawPT(parseRE("''"))_____no_output_____drawPT(parseRE("1"))_____no_output_____drawPT(parseRE("(0*1*)*"))_____no_output_____

** Your answer for Q1(b) below **_____no_output_____

# Demonstrating the parsing of & and + in the same expression

drawPT(parseRE("!!ab**"))_____no_output_____

Q1(b): The above parse tree demonstrates how "!" binds tighter than concatenation and is right-associative. It also demonstrates that "*" binds tightly and is left-associative._____no_output_____

# Demonstrating the parsing of + and & in the same expression. Note that concatenation is also involved

drawPT(parseRE("ab+cd&ef"))_____no_output_____

Q1(b): The above parse tree demonstrates how "&" binds the tightest, followed by concatenation, and finally "+"_____no_output_____

# Demonstrating the parsing of all operators in one RE

drawPT(parseRE("(ab)*+!(cd)&e!f*"))_____no_output_____

Q1(b): We simply don't recommend an unreadable RE such as we just now fed. However, some of these REs may be
machine-generated, and then we may not be able to control their length or ugliness. In any case, it appears to
parse exactly as intended!_____no_output_____# Exercise: Produce the Parse Tree for an RE, and Explain

** Q1(c): ** Draw the parse tree for "0+11*" and explain its shape. Does it implement precedences that you expect "naturally"?_____no_output_____

drawPT(parseRE("0+11*"))_____no_output_____

** Your answer for Q1(c) here: ** _____no_output_____We obtained the parse-tree as intended because "*" binds tightly, and after that, concatenation binds tighter than "+"_____no_output_____# Exercise : RE parsing : Precedences

** Q1(d): **

Create variants of the expression "0+11*" by putting parentheses, and describe the parse-trees obtained. For example, calling the above RE R1, we have

* R1 = "0+11*"

* R2 = "(0+1)1*" is one variant.

Similarly, produce two other variants, calling them R3 and R4 merely by placing parentheses differently.

* Argue in two clear sentences that R1 through R4 denote different languages_____no_output_____** Your answer for Q1(d) below **_____no_output_____Indeed the parses are different for R1 and R2. In one case, we override the precedence of concatenation by giving "+" higher precedence. We can produce two more parses

* R3 = "0+(11)*"

* R4 = "(0+11)*"

However, our grammar __will not__ produce these parses._____no_output_____# Produce parse-trees for extended REs

** Q1(e): **  Turn on the parsing rules for "!" and "&". Test the new parser on the following REs

* RE with * and ! : "a!b*"

* RE with & involved : "cd&ef"

* RE with & and + involved : "a+b&c+d"
_____no_output_____

drawPT(parseRE("a!b"))_____no_output_____drawPT(parseRE("cd&ef"))_____no_output_____drawPT(parseRE("a+b&c+d"))_____no_output_____

** Your answer for Q1(e) below **_____no_output_____The parses were found to be correct_____no_output_____# Derivative-based Pattern Matching

We now present the routines that help implement a derivative-based pattern matcher_____no_output_____

#=== Now comes derivMatch as illustration of RE Derivative scanning

# These four functions are simple extractors of the operator and arguments

def opr(E):
    """Retrieves the operator of an expression.
    """
    return E[0]

def arg1(E):
    """Retrieves the first argument of a binary operator-based expression.
    """
    return E[1][0]

def arg2(E):
    """Retrieves the second argument of a binary operator-based expression.
    """
    return E[1][1]

def arg(E):
    """Retrieves the only argument of a unary operator-based expression.
    """
    return E[1]

def nullable(E):
    """This is the nullability test defined in Chapter 10.
    """
    if (opr(E) == "str") :
        return False
    elif (opr(E) == '@') :
        return True
    elif (opr(E) == "mty") :
        return False
    elif (opr(E) == "*"):
        return True
    elif (opr(E) == "!"):
         return not nullable(arg(E))
    elif (opr(E) == '+') :
        return nullable(arg1(E)) or nullable(arg2(E))
    elif (opr(E) == '.') :
        return nullable(arg1(E)) and nullable(arg2(E))
    elif (opr(E) == '&') :
        return nullable(arg1(E)) and nullable(arg2(E))
    else:
        return "???"

def dv(E, c):
    """This function computes the derivative
       of a regular expression E with respect
       to character "c".
    """
    if (opr(E) == "str") :
        if (arg(E) == c):
            return ('@', '@')
        else:
            return ("mty", "mty")
    elif (opr(E) == '@') :
        return ("mty", "mty")
    elif (opr(E) == "mty") :
        return ("mty", "mty")
    elif (opr(E) == "*"):
        return (".", (dv(arg(E), c), E))
    elif (opr(E) == "!"):
        return ("!", dv(arg(E), c))
    elif (opr(E) == '+') :
        return ("+", (dv(arg1(E), c), dv(arg2(E), c)))
    elif (opr(E) == '&') :
        return ("&", (dv(arg1(E), c), dv(arg2(E), c)))
    elif (opr(E) == '.') :
        if nullable(arg1(E)):
            return ("+", ( ('.', (dv(arg1(E), c), arg2(E))), dv(arg2(E), c) ))
        else:
            return ('.', (dv(arg1(E), c), arg2(E)))
    else:
        return "???"

def matches(w, E):
    if w=="":
        return nullable(E)
    else:
        derivative = dv(E, w[0])
        return matches(w[1:], derivative)_____no_output_____

## Testing Derivative-based Parsing_____no_output_____###  Testing Nullability

A regular expression is nullable if its language contains epsilon.
Regular-expression based pattern-matching works as follows:

* Keep obtaining the derivatives of a given RE under the characters comprising a string

* When the string is empty, check whether the RE is nullable._____no_output_____

(E, nl, el) = parseRE("''")
nullable(E)_____no_output_____(E, nl, el) = parseRE("c")
nullable(E)_____no_output_____(E, nl, el) = parseRE("c*")
nullable(E)_____no_output_____(E, nl, el) = parseRE("!c*")
nullable(E)_____no_output_____(E, nl, el) = parseRE("(!c)*")
nullable(E)_____no_output_____(E, nl, el) = parseRE("a+b*")
nullable(E)_____no_output_____(E, nl, el) = parseRE("a+bb*")
nullable(E)_____no_output_____(E, nl, el) = parseRE("a*b*")
nullable(E)_____no_output_____(E, nl, el) = parseRE("a*+bb*")
nullable(E)_____no_output_____(E, nl, el) = parseRE("aa*+bb*")
nullable(E)_____no_output_____

### Exercise: Explaining nullability tests

**Q1(f):** Explain the following nullability tests._____no_output_____

(E, nl, el) = parseRE("!a*")
nullable(E)_____no_output_____

** This is true because !a is starred and the star of any language is nullable. **_____no_output_____

drawPT(parseRE("!a*"))_____no_output_____

** The fact that !a is starred is seen from the parse-tree above. **_____no_output_____

(E, nl, el) = parseRE("!(a*)")
nullable(E)_____no_output_____

** It is because "a*" is nullable, and so its complement must not be. **_____no_output_____

(E, nl, el) = parseRE("!aa*+b&b*")
nullable(E)_____no_output_____drawPT(parseRE("  !aa*+b&b* "  ))_____no_output_____

** Answer: It is because "!a" is nullable and so is "a*". These are concatenated to be the left subexpression
    of "+" which only cares for one of its 'arms' to be nullable before the whole is declared nullable. **_____no_output_____## Exercise: Explaining nullability tests

**Q1(g):** Explain the following nullability test. Why is it returning true?_____no_output_____

(E, nl, el) = parseRE("!a*&b*")
nullable(E)_____no_output_____drawPT(parseRE("!a*&b*"))_____no_output_____

** Your answer for Q1(g): **_____no_output_____Both the conjuncts are nullable, and so the whole is nullable!_____no_output_____# Pattern-matching tests_____no_output_____** Q1(h): ** Pattern matching

Explain the following outcomes in terms of the behavior of the pattern matcher_____no_output_____

re = 'ab'
(E, nl, el) = parseRE(re)
matches("a", E)_____no_output_____re = 'ab'
(E, nl, el) = parseRE(re)
matches("ab", E)_____no_output_____re = 'a*b*'
(E, nl, el) = parseRE(re)
matches("aabbb", E)_____no_output_____re = '(a*b)&(bc*)'
drawPT(parseRE(re))_____no_output_____re = '(a*b)&(bc*)'
(E, nl, el) = parseRE(re)
matches("b", E)_____no_output_____

##  Match test answers

re = 'ab'

(E, nl, el) = parseRE(re)

matches("a", E) -> false

** This is justified because the input is "a" while the RE requires "ab", and so the match fails **

re = 'ab'

(E, nl, el) = parseRE(re)

matches("ab", E) -> true

** This is justified because the input is "ab" and the RE requires "ab", and so the match succeeds **

re = 'a*b*'

(E, nl, el) = parseRE(re)

matches("aabbb", E) -> true

** This is justified because the input is "aabbb" and the RE requires "a*b*", and so the match succeeds **

re = '(a*b)&(bc*)'

(E, nl, el) = parseRE(re)

matches("b", E) -> true

** This is justified because the input is "b" and the RE requires "(a*b)(bc*)", and so the match succeeds, as 'b' is common to both languages. **
_____no_output_____

B1 = "1(0+1)*"

E1 = "(0+1)*1"

B1andE1 =  " ( 1(0+1)* ) & ( (0+1)*1  ) "

NB1orNE1 =  "  !(  ( 1(0+1)* ) & ( (0+1)*1  )    ) "
_____no_output_____drawPT(parseRE(B1))_____no_output_____drawPT(parseRE(E1))_____no_output_____drawPT(parseRE(B1andE1))_____no_output_____NB1orNE1_____no_output_____drawPT(parseRE(NB1orNE1))_____no_output_____(E, nl, el) = parseRE(NB1orNE1)_____no_output_____matches("01", E)_____no_output_____matches("10", E)_____no_output_____matches("00", E)_____no_output_____matches("11", E)_____no_output_____matches("1010110110", E)_____no_output_____matches("1010110111", E)_____no_output_____matches("0010110111", E)_____no_output_____matches("0010110110", E)_____no_output_____

** The above four tests reflect the semantics of RE with respect to the DeMorgan's Law. Only when the word does not begin with a 1 or does not end with a 1 must we get a match. In this case, it is evident from the tests. **_____no_output_____# QUESTIONS FOR YOU TO ANSWER_____no_output_____1. Summarize sections 1-8 of this Jove file by answering the following questions. All "sentences" mentioned below are to be around 16 words or so.

 a. (One sentence) how did you add the single-character token "!" into the grammar, ensuring its precedence with respect to "*" ?

 b. (One sentence) how did you add the single-character token "&" into the grammar, ensuring its precedence with respect to "+" ?

 c. (One sentence) def p_expression_ordy(t)  and   def p_expression_plus(t)  look the same except for passing "&" or "+" to function attrDyadicInfix. What is the real difference between these functions? Hint: See where in the layered grammar they are situated (and hence what precedence with which they get invoked).

 d. (One sentence) How did "!!ab**" get parsed? Indicate by putting parentheses. E.g. if a+b+c got parsed right associative, I would put a+(b+c) in my answer. Like that add parentheses.

 e. (One sentence) Without disambiguation, how many parses are there for "0+11" ? Show your answer by putting parentheses.

 f. Show that string "aa" is accepted by RE "!aaa". Show all your steps. Hint: Simulate the first 'a'. If you get an empty set, carry on with your work using the rules table.

2. Run the commands embedded in Section 8 of this Jove file.

 a. Produce an RE capturing 'begins with 1' -- call it B1. Display its parse tree. The commands are waiting above. Run the commands, observe the result. Move on.

 b. Produce an RE capturing 'ends with 1' -- call it E1. Display its parse tree. The commands are waiting above. Run the commands, observe the result. Move on.

 c. Produce an RE capturing B1 and E1. Call this RE B1andE1. Use the "&" operator. Display its parse tree. The commands are waiting above. Run the commands, observe the result. Move on.

 d. Produce an RE capturing the negation of B1andE1. Call this RE NB1orNE1.  Use the "!" operator. Display its parse tree. The commands are waiting above. Run the commands, observe the result. Move on.

 e. Test NB1orNE1 on the following four strings: 00, 01, 10, and 11. Explain the results you obtain. The commands are waiting above. Run the commands, observe the result. Move on.

 e. Repeat for 1010110110, 1010110111, 0010110111, and 0010110110. Explain your results. The commands are waiting above. Run the commands, observe the result. Move on.

 f. Clearly explain why matches("0010110110", E)  returns true? Use 3 sentences.
_____no_output_____


# Notebook from cs-me/logical_musings
Path: _notebooks/2020-04-21-Artifical-General-Intelligence-The-Prelude-to-Brain-Connectivity.ipynb

# Artificial General Intelligence
> The Prelude to Brain Connectivity
- toc: false
- branch: master
- badges: False
- comments: true
- categories: [Brain Connectivity]_____no_output_____

#hide
from IPython.display import Image
_____no_output_____

This is a series of blog posts which will explain functional Brain Connectivity i.e. the study of statiscally significant temporal connections in anatomically segregated brain regions which are activated during a task. Don't worry if you didn't get any of the above. I will be explaining everything in a series of blog posts starting with the motivation behind selecting the topic. The motivation (atleast for me) is what is generally known as "Artifical General Intelligence (AGI)".

Though explaining it deserves a series of blog posts in itself, for now, I am leaving it for the future. In this post, I will try to explain AGI as a pre-cursor to Brain Connectivity and how neuroscience inspired AI is the need of the hour to achieve AGI.

Without much ado, let's begin!

AGI is the holy grail of today's AI research. This is the pen-ultimate goal of achieving human-like intelligence, the ultimate goal being development of sentient beings i.e. Humanoids with Consciousness. Normally, research in AI is divided into two parts known as Strong AI and Weak AI. Strong AI refers to a system which has the capacity to learn like humans do while Weak AI refers to a machine which is really good at performing a single task. The most amazing attribute of human intelligence is it's robustness i.e. the ability to learn diverse things with  very limited number of training examples. For instance you can learn to identify cats and dogs after seeing one or two examples of each only. Even a baby learn so many things and so quickly, that it seems impossible for our present artificial machines to replicate that. Speaking of the present capabilities of AI, one begins to wonder that we are not far away from acheiving the goal. The famous **[Ray Kurtzweil](href=(https://en.wikipedia.org/wiki/Ray_Kurzweil)** argued that the **[Singularity](https://en.wikipedia.org/wiki/Technological_singularity)** will be achieved by 2045.

Having said that, many people opine that *Deep Learning* is the tool we were missing and now that it has matured to some incredible results, it will be the vehicle which takes us to the Singularity. Others argue that Deep Learning is not actually the hail mary of the technologies and lacks much of what is needed to simulate true intelligence.

**[Gary Marcus](https://en.wikipedia.org/wiki/Gary_Marcus)** is one of the leading critique of Deep Learning and has been long advocating what **[Yoshua Bengio](https://en.wikipedia.org/wiki/Yoshua_Bengio)** said recently which is and I quote "we need to be able to extend deep learning to do things like reasoning, learning causality, and exploring the world in order to learn and acquire information " . You can find more details [here](https://medium.com/@GaryMarcus/the-deepest-problem-with-deep-learning-91c5991f5695). Similarly **[Andrew Ng](https://hbr.org/2016/11/what-artificial-intelligence-can-and-cant-do-right-now)** argued that AI at present is nowhere near ready to take over the world and such claims are no more than science fiction. In the current scenario, it is difficult to say whether we will be able to extend Deep Learning to such an extent where it can realize AGI given that it is more or less a statistical exercise and does not make use of or generate a world model to develop an understading around it which we humans are able to do effortlessly.

Hence, the question is *which* is the best technology to realize AGI in machines. The answer is we don't know and the last resort for us petty humans is imitation. Remember we are trying to achieve human intelligence and the tool that humans use for doing that is ***the brain***. Yes, the below represented blob of around 3 pounds and 15cm length which somehow is able to achieve what arguably a collection of best *brains* in the world don't seem to understand._____no_output_____

#hide_input
Image(filename="images/Brain_Connectivity/Cerebrum.jpg", width=300, height=300)
_____no_output_____

But imitation doesn't always guarantee replication as was argued by **[Yann LeCunn](http://yann.lecun.com/)** in one of his talks. He said that the brain is an existential proof of intelligence just like birds are for heavier-than-air flight. The Avion III, was a primitive steam-powered aircraft built by Clément Ader and was heavily inspired by nature (bats in particular) with articulated wings, however the vehicle was unable to fly and research on it was stopped in 1898 by the French Government. The first flying machine was built by the Wright brothers instead using principles of aerodynamics. It didn't mean they didn't study bird's flight. They spent a great deal of time understanding the birds' flight.

In 2017, **[Demis Hassabis](https://en.wikipedia.org/wiki/Demis_Hassabis)** and team wrote "Neuroscience inspired AI" which establishes the connection between achieving AGI inspired by Neuroscience. They mentioned two-fold benefits of examining biological intelligence.
    - Neuroscience provides rich source of information for new types of algorithms and infrastructure independent of and complementary to current     mathematical approaches
    - Neuroscience can provide validation of already existing AI techniques.

Note that taking inspiration for developing algorithms and infrastructure is different than plainly mimicking the brain or reverse-engineering the circuitory which is also being tried in projects like [Blue Brain Project](https://www.epfl.ch/research/domains/bluebrain/) or [Neuromorphic Computing](https://en.wikipedia.org/wiki/Neuromorphic_engineering). Demis et al. illustrated that some of the recent breakthroughs in AI are in fact closely associated with neuroscientific considerations. Some of the brain functions that the AI models mimic are Attention, Episodic Memory, Continual Learning, Imagination & Planning and Inference. The following figure illustrates some of the parallels between latest AI systems and Neural Models._____no_output_____

#hide_input
Image(filename="images/Brain_Connectivity/screenshot.png",width=300,height=300)
_____no_output_____

So to sum up, we need to get inspiration from brain and build our own frameworks exploiting the existing mathematical models of optimality and the cognitive models of intelligence which enables us to generalize to unforeseen events. In order to do that we must first understand how the brain works. Given the amazing technological advancement seen in the fields of Physics, Maths, Biology and Engineering, it's surprising that we don't know much about how our own brain works. However, things are changing fast and brain imaging technologies like *[EEG](https://en.wikipedia.org/wiki/Electroencephalography),[fMRI](https://en.wikipedia.org/wiki/Functional_magnetic_resonance_imaging), [PET](https://en.wikipedia.org/wiki/Positron_emission_tomography)* have enabled us to take a peek inside the magnificient and incredible working of brain.

One of the techniques to get some insights on how brain works is to map the different brain areas activated by a particular behavior/task. For example Amygdala is responsible for controlling our emotions and Hippocampus is responsible for memories. And recently, there's been a great increased interest in augmenting these types of analysis with connectivity studies. This study seek to describe how brain regions interact with one another, and how these interactions depend on experimental conditions and behavioral measures. The functional organization of the brain is characterized by segregation and integration of information being processed. Brain connectivity is thus crucial to elucidating how neurons and neural networks process information.

Brain Connectivity studies can help us in understanding how different regions of the brain communicate with each other and more importantly, what this connection or the lack thereof represent. For example brain connectivity can be used to diagnose diseases like schizophrenia, autism, and clinical depression. Both healthy and diseased brains may be mapped to study memory, learning, aging, and drug effects, following which a classifier using Deep Learning/ Machine Learning techniques can be used to build a diagnostic tool. It may also be crucial to understanding traumatic brain injuries (as in the case of Phineas Gage) and improving brain injury treatment.

With this very primitive definition and a detailed motivation, let's move on to the second part of the series [Introduction to Brain Connectivity] which will cover what is brain connectivity, different types of brain connecitivity and more importantly how to code things to analyze the brain connectivity data._____no_output_____


# Notebook from asad200/DNA-Sequence-Classification
Path: DNA_Classification.ipynb

import sys
import sklearn
import numpy as np
import pandas as pd

# import different classifiers from scikitlearn
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.linear_model import LogisticRegression, SGDClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.neural_network import MLPClassifier
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.metrics import confusion_matrix, accuracy_score, classification_report
from sklearn import model_selection_____no_output_____

### Import the Molecular Biology (Promoter Gene Sequences) Data Set_____no_output_____

# web address of the data set
url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/molecular-biology/promoter-gene-sequences/promoters.data'
columns = ['Class', 'id', 'Sequence']
data = pd.read_csv(url, names = columns)
print(data.head())  Class         id                                           Sequence
0     +        S10  \t\ttactagcaatacgcttgcgttcggtggttaagtatgtataat...
1     +       AMPC  \t\ttgctatcctgacagttgtcacgctgattggtgtcgttacaat...
2     +       AROH  \t\tgtactagagaactagtgcattagcttatttttttgttatcat...
3     +      DEOP2  \taattgtgatgtgtatcgaagtgtgttgcggagtagatgttagaa...
4     +  LEU1_TRNA  \ttcgataattaactattgacgaaaagctgaaaaccactagaatgc...

### Pre process the data set_____no_output_____

# extract the classes from the data set
classes = data['Class']

# extract the DNA sequences from the data set
sequences = data['Sequence']

print(classes.head(4))
print(sequences.head(4))0    +
1    +
2    +
3    +
Name: Class, dtype: object
0    \t\ttactagcaatacgcttgcgttcggtggttaagtatgtataat...
1    \t\ttgctatcctgacagttgtcacgctgattggtgtcgttacaat...
2    \t\tgtactagagaactagtgcattagcttatttttttgttatcat...
3    \taattgtgatgtgtatcgaagtgtgttgcggagtagatgttagaa...
Name: Sequence, dtype: object
# remove the leading \tabs from sequences and append the class of the sequence
sequences = list(sequences)
dataset = {}
for i, seq in enumerate(sequences):
    bases = list(seq)
    bases = [x for x in bases if x != '\t']
    bases.append(classes[i])
    dataset[i] = bases
# convert the pandas dataframe
df = pd.DataFrame(dataset)
df = df.transpose()
print(df.head())  0  1  2  3  4  5  6  7  8  9   ... 48 49 50 51 52 53 54 55 56 57
0  t  a  c  t  a  g  c  a  a  t  ...  g  c  t  t  g  t  c  g  t  +
1  t  g  c  t  a  t  c  c  t  g  ...  c  a  t  c  g  c  c  a  a  +
2  g  t  a  c  t  a  g  a  g  a  ...  c  a  c  c  c  g  g  c  g  +
3  a  a  t  t  g  t  g  a  t  g  ...  a  a  c  a  a  a  c  t  c  +
4  t  c  g  a  t  a  a  t  t  a  ...  c  c  g  t  g  g  t  a  g  +

[5 rows x 58 columns]
df.rename(columns = {57: 'Class'}, inplace=True)_____no_output_____# calculate the ratio of each base in a sequence
seq = []
for column in df.columns:
    seq.append(df[column].value_counts())

results = pd.DataFrame(seq).transpose()
print(results)
      0     1     2     3     4     5     6     7     8     9  ...    48  \
t  38.0  26.0  27.0  26.0  22.0  24.0  30.0  32.0  32.0  28.0  ...  21.0
c  27.0  22.0  21.0  30.0  19.0  18.0  21.0  20.0  22.0  22.0  ...  36.0
a  26.0  34.0  30.0  22.0  36.0  42.0  38.0  34.0  33.0  36.0  ...  23.0
g  15.0  24.0  28.0  28.0  29.0  22.0  17.0  20.0  19.0  20.0  ...  26.0
-   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...   NaN
+   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...   NaN

     49    50    51    52    53    54    55    56  Class
t  22.0  23.0  33.0  35.0  30.0  23.0  29.0  34.0    NaN
c  42.0  31.0  32.0  21.0  32.0  29.0  29.0  17.0    NaN
a  24.0  28.0  27.0  25.0  22.0  26.0  24.0  27.0    NaN
g  18.0  24.0  14.0  25.0  22.0  28.0  24.0  28.0    NaN
-   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   53.0
+   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   53.0

[6 rows x 58 columns]
# replace the alphabets with numeric values
df = pd.get_dummies(df)
df = df.drop(columns=['Class_-'])
df.rename(columns = {'Class_+': 'Class'}, inplace=True)
print(df.head())   0_a  0_c  0_g  0_t  1_a  1_c  1_g  1_t  2_a  2_c  ...  54_t  55_a  55_c  \
0    0    0    0    1    1    0    0    0    0    1  ...     0     0     0
1    0    0    0    1    0    0    1    0    0    1  ...     0     1     0
2    0    0    1    0    0    0    0    1    1    0  ...     0     0     1
3    1    0    0    0    1    0    0    0    0    0  ...     0     0     0
4    0    0    0    1    0    1    0    0    0    0  ...     1     1     0

   55_g  55_t  56_a  56_c  56_g  56_t  Class
0     1     0     0     0     0     1      1
1     0     0     1     0     0     0      1
2     0     0     0     0     1     0      1
3     0     1     0     1     0     0      1
4     0     0     0     0     1     0      1

[5 rows x 229 columns]
# Get all the relevent columns from the DataFrame
columns = df.columns.tolist()

# Filter the columns to remove the data we don't need for training
columns = [c for c in columns if c not in ["Class"]]

# Store the labels separetely
target = "Class"

x = df[columns]
y = df[target]

print(x.shape)
print(y.shape)(106, 228)
(106,)
seed = 1
x_train, x_test, y_train, y_test = model_selection.train_test_split(x, y, test_size=0.25, random_state=seed)_____no_output_____print(x_train.shape)
print(y_train.shape)
print(x_test.shape)
print(y_test.shape)(79, 228)
(79,)
(27, 228)
(27,)
# get the models
modelNames = ['SVC Linear', 'SVC RBF', 'SVC Simoid', 'K-Nearest Neighbours', 'Decision Tree',
              'Gaussian Process', 'Random Forest', 'Neural Net', 'AdaBoost', 'Naive Bayes']

classifiers = [SVC(kernel='linear'),
               SVC(kernel='rbf'),
               SVC(kernel='sigmoid'),
               KNeighborsClassifier(n_neighbors=3),
               DecisionTreeClassifier(max_depth=5),
               GaussianProcessClassifier(1.0 * RBF(1.0)),
               RandomForestClassifier(max_depth=5, n_estimators=10, max_features=1),
               MLPClassifier(alpha=1, max_iter=2000),
               AdaBoostClassifier(),
               GaussianNB()

]

models = list(zip(modelNames, classifiers))
#print(models)_____no_output_____# fit the models

scoring = 'accuracy'
results = []
modelNames = []

for name, model in models:
    k_fold = model_selection.KFold(n_splits=10, shuffle=True, random_state=seed)
    val_results = model_selection.cross_val_score(model, x_train, y_train, cv=k_fold, scoring=scoring)
    results.append(val_results)
    modelNames.append(name)
    print('{}: {} ({})'.format(name, val_results.mean(), val_results.std()))SVC Linear: 0.9125 (0.09762812094883318)
SVC RBF: 0.875 (0.11180339887498948)
SVC Simoid: 0.925 (0.1)
K-Nearest Neighbours: 0.8107142857142857 (0.099808490089158)
Decision Tree: 0.7071428571428571 (0.1524042020953705)
Gaussian Process: 0.8553571428571429 (0.1606051216556957)
Random Forest: 0.6160714285714286 (0.18250139781404862)
Neural Net: 0.9125 (0.09762812094883318)
AdaBoost: 0.8625 (0.14197270864500683)
Naive Bayes: 0.8375 (0.1125)
# test on tesing data

for name, model in models:
    model.fit(x_train, y_train)
    predictions = model.predict(x_test)
    print(name)
    print(accuracy_score((y_test), predictions))
    print(classification_report(y_test, predictions))SVC Linear
0.9629629629629629
              precision    recall  f1-score   support

           0       1.00      0.94      0.97        17
           1       0.91      1.00      0.95        10

    accuracy                           0.96        27
   macro avg       0.95      0.97      0.96        27
weighted avg       0.97      0.96      0.96        27

SVC RBF
0.9259259259259259
              precision    recall  f1-score   support

           0       1.00      0.88      0.94        17
           1       0.83      1.00      0.91        10

    accuracy                           0.93        27
   macro avg       0.92      0.94      0.92        27
weighted avg       0.94      0.93      0.93        27

SVC Simoid
0.9259259259259259
              precision    recall  f1-score   support

           0       1.00      0.88      0.94        17
           1       0.83      1.00      0.91        10

    accuracy                           0.93        27
   macro avg       0.92      0.94      0.92        27
weighted avg       0.94      0.93      0.93        27

K-Nearest Neighbours
0.7777777777777778
              precision    recall  f1-score   support

           0       1.00      0.65      0.79        17
           1       0.62      1.00      0.77        10

    accuracy                           0.78        27
   macro avg       0.81      0.82      0.78        27
weighted avg       0.86      0.78      0.78        27

Decision Tree
0.8148148148148148
              precision    recall  f1-score   support

           0       1.00      0.71      0.83        17
           1       0.67      1.00      0.80        10

    accuracy                           0.81        27
   macro avg       0.83      0.85      0.81        27
weighted avg       0.88      0.81      0.82        27

Gaussian Process
0.8888888888888888
              precision    recall  f1-score   support

           0       1.00      0.82      0.90        17
           1       0.77      1.00      0.87        10

    accuracy                           0.89        27
   macro avg       0.88      0.91      0.89        27
weighted avg       0.91      0.89      0.89        27

Random Forest
0.7407407407407407
              precision    recall  f1-score   support

           0       0.92      0.65      0.76        17
           1       0.60      0.90      0.72        10

    accuracy                           0.74        27
   macro avg       0.76      0.77      0.74        27
weighted avg       0.80      0.74      0.74        27

Neural Net
0.8888888888888888
              precision    recall  f1-score   support

           0       1.00      0.82      0.90        17
           1       0.77      1.00      0.87        10

    accuracy                           0.89        27
   macro avg       0.88      0.91      0.89        27
weighted avg       0.91      0.89      0.89        27

AdaBoost
0.8518518518518519
              precision    recall  f1-score   support

           0       1.00      0.76      0.87        17
           1       0.71      1.00      0.83        10

    accuracy                           0.85        27
   macro avg       0.86      0.88      0.85        27
weighted avg       0.89      0.85      0.85        27

Naive Bayes
0.9259259259259259
              precision    recall  f1-score   support

           0       1.00      0.88      0.94        17
           1       0.83      1.00      0.91        10

    accuracy                           0.93        27
   macro avg       0.92      0.94      0.92        27
weighted avg       0.94      0.93      0.93        27