# Notebook from richardyan93/ZeroCostDL4Mic
Path: Colab_notebooks/Beta notebooks/SplineDist_2D_ZeroCostDL4Mic.ipynb

# **SplineDist (2D)**
---

**SplineDist 2D** is a deep-learning method that can be used to segment objects from bioimages and was first published by [Mandal *et al.* in 2020, on biorXiv](https://www.biorxiv.org/content/10.1101/2020.10.27.357640v1). SplineDist uses a flexible and general representation by modelling objects as planar parametric spline curves.

 **This particular notebook enables the segmentation of 2D dataset. If you are interested in 3D dataset, you should use the StarDist or U-net 3D notebooks instead.**

---
*Disclaimer*:

This notebook is part of the Zero-Cost Deep-Learning to Enhance Microscopy project (https://github.com/HenriquesLab/DeepLearning_Collab/wiki). Jointly developed by the Jacquemet (link to https://cellmig.org/) and Henriques (https://henriqueslab.github.io/) laboratories.

This notebook is largely based on the paper:

**SplineDist: Automated Cell Segmentation with Spline Curves** from Mandal *et al.*, bioRxiv. (https://www.biorxiv.org/content/10.1101/2020.10.27.357640v1)

**The Original code** is freely available in GitHub:
https://gitlab.ebi.ac.uk/smandal/splinedist

**Please also cite this original paper when using or developing this notebook.**
_____no_output_____# **How to use this notebook?**

---

Video describing how to use our notebooks are available on youtube:
  - [**Video 1**](https://www.youtube.com/watch?v=GzD2gamVNHI&feature=youtu.be): Full run through of the workflow to obtain the notebooks and the provided test datasets as well as a common use of the notebook
  - [**Video 2**](https://www.youtube.com/watch?v=PUuQfP5SsqM&feature=youtu.be): Detailed description of the different sections of the notebook

---
###**Structure of a notebook**

The notebook contains two types of cell:

**Text cells** provide information and can be modified by douple-clicking the cell. You are currently reading the text cell. You can create a new text by clicking `+ Text`.

**Code cells** contain code and the code can be modfied by selecting the cell. To execute the cell, move your cursor on the `[ ]`-mark on the left side of the cell (play button appears). Click to execute the cell. After execution is done the animation of play button stops. You can create a new coding cell by clicking `+ Code`.

---
###**Table of contents, Code snippets** and **Files**

On the top left side of the notebook you find three tabs which contain from top to bottom:

*Table of contents* = contains structure of the notebook. Click the content to move quickly between sections.

*Code snippets* = contain examples how to code certain tasks. You can ignore this when using this notebook.

*Files* = contain all available files. After mounting your google drive (see section 1.) you will find your files and folders here.

**Remember that all uploaded files are purged after changing the runtime.** All files saved in Google Drive will remain. You do not need to use the Mount Drive-button; your Google Drive is connected in section 1.2.

**Note:** The "sample data" in "Files" contains default files. Do not upload anything in here!

---
###**Making changes to the notebook**

**You can make a copy** of the notebook and save it to your Google Drive. To do this click file -> save a copy in drive.

To **edit a cell**, double click on the text. This will show you either the source code (in code cells) or the source text (in text cells).
You can use the `#`-mark in code cells to comment out parts of the code. This allows you to keep the original code piece in the cell as a comment._____no_output_____#**0. Before getting started**
---
 For SplineDist to train, **it needs to have access to a paired training dataset made of images of nuclei and their corresponding masks**. Information on how to generate a training dataset is available in our Wiki page: https://github.com/HenriquesLab/ZeroCostDL4Mic/wiki

**We strongly recommend that you generate extra paired images. These images can be used to assess the quality of your trained model**. The quality control assessment can be done directly in this notebook.

The data structure is important. It is necessary that all the input data are in the same folder and that all the output data is in a separate folder. The provided training dataset is already split in two folders called "Training - Images" (Training_source) and "Training - Masks" (Training_target).

Additionally, the corresponding Training_source and Training_target files need to have **the same name**.

Please note that you currently can **only use .tif files!**

You can also provide a folder that contains the data that you wish to analyse with the trained network once all training has been performed. This can include Test dataset for which you have the equivalent output and can compare to what the network provides.

Here's a common data structure that can work:
*   Experiment A
    - **Training dataset**
      - Images (Training_source)
        - img_1.tif, img_2.tif, ...
      - Masks (Training_target)
        - img_1.tif, img_2.tif, ...
    - **Quality control dataset**
     - Images
        - img_1.tif, img_2.tif
      - Masks
        - img_1.tif, img_2.tif
    - **Data to be predicted**
    - **Results**

---
**Important note**

- If you wish to **Train a network from scratch** using your own dataset (and we encourage everyone to do that), you will need to run **sections 1 - 4**, then use **section 5** to assess the quality of your model and **section 6** to run predictions using the model that you trained.

- If you wish to **Evaluate your model** using a model previously generated and saved on your Google Drive, you will only need to run **sections 1 and 2** to set up the notebook, then use **section 5** to assess the quality of your model.

- If you only wish to **run predictions** using a model previously generated and saved on your Google Drive, you will only need to run **sections 1 and 2** to set up the notebook, then use **section 6** to run the predictions on the desired model.
---_____no_output_____# **1. Initialise the Colab session**

---

_____no_output_____
## **1.1. Check for GPU access**
---

By default, the session should be using Python 3 and GPU acceleration, but it is possible to ensure that these are set properly by doing the following:

Go to **Runtime -> Change the Runtime type**

**Runtime type: Python 3** *(Python 3 is programming language in which this program is written)*

**Accelator: GPU** *(Graphics processing unit)*
_____no_output_____

#@markdown ##Run this cell to check if you have GPU access
# %tensorflow_version 1.x

import tensorflow as tf
if tf.test.gpu_device_name()=='':
  print('You do not have GPU access.')
  print('Did you change your runtime ?')
  print('If the runtime setting is correct then Google did not allocate a GPU for your session')
  print('Expect slow performance. To access GPU try reconnecting later')

else:
  print('You have GPU access')
  !nvidia-smi

print('TensorFlow version: '+tf.__version__)_____no_output_____

## **1.2. Mount your Google Drive**
---
 To use this notebook on the data present in your Google Drive, you need to mount your Google Drive to this notebook.

 Play the cell below to mount your Google Drive and follow the link. In the new browser window, select your drive and select 'Allow', copy the code, paste into the cell and press enter. This will give Colab access to the data on the drive.

 Once this is done, your data are available in the **Files** tab on the top left of notebook._____no_output_____

#@markdown ##Play the cell to connect your Google Drive to Colab

#@markdown * Click on the URL.

#@markdown * Sign in your Google Account.

#@markdown * Copy the authorization code.

#@markdown * Enter the authorization code.

#@markdown * Click on "Files" site on the right. Refresh the site. Your Google Drive folder should now be available here as "drive".

# mount user's Google Drive to Google Colab.
from google.colab import drive
drive.mount('/content/gdrive')_____no_output_____

# **2. Install SplineDist and dependencies**
---
_____no_output_____

Notebook_version = ['1.12']

#@markdown ##Install SplineDist and dependencies
#%tensorflow_version 1.x

import tensorflow
print(tensorflow.__version__)
print("Tensorflow enabled.")

# Install packages which are not included in Google Colab

!pip install tifffile # contains tools to operate tiff-files
!pip install csbdeep  # contains tools for restoration of fluorescence microcopy images (Content-aware Image Restoration, CARE). It uses Keras and Tensorflow.
!pip install stardist # contains tools to operate STARDIST.
!pip install gputools # improves STARDIST performances
!pip install edt # improves STARDIST performances
!pip install wget
!pip install fpdf
!pip install PTable # Nice tables

!git clone https://github.com/uhlmanngroup/splinedist
# !git clone https://gitlab.ebi.ac.uk/smandal/splinedist

import os

os.chdir("/content/splinedist")

!python setup.py install
!python splinegenerator.py install

import splinegenerator as sg
from splinedist.utils import phi_generator, grid_generator, get_contoursize_max, export_imagej_rois
from splinedist import fill_label_holes, random_label_cmap, calculate_extents, gputools_available
from splinedist.matching import matching, matching_dataset
from splinedist.models import Config2D, SplineDist2D, SplineDistData2D

os.chdir("/content")
# ------- Variable specific to Stardist -------
from glob import glob
from tqdm import tqdm
from csbdeep.utils import Path, normalize
import numpy as np

np.random.seed(42)
lbl_cmap = random_label_cmap()

# ------- Common variable to all ZeroCostDL4Mic notebooks -------
import numpy as np
from matplotlib import pyplot as plt
import urllib
import os, random
import shutil
import zipfile

from tifffile import imsave, imread

import time
import sys
import wget
from pathlib import Path
import pandas as pd
import csv
from glob import glob
from scipy import signal
from scipy import ndimage
from skimage import io
from sklearn.linear_model import LinearRegression
from skimage.util import img_as_uint
import matplotlib as mpl
from skimage.metrics import structural_similarity
from skimage.metrics import peak_signal_noise_ratio as psnr
from astropy.visualization import simple_norm
from skimage import img_as_float32, img_as_ubyte, img_as_float
from skimage.util import img_as_ubyte, img_as_uint
from tqdm import tqdm
import cv2
from fpdf import FPDF, HTMLMixin
from datetime import datetime
from pip._internal.operations.freeze import freeze
import subprocess

# For sliders and dropdown menu and progress bar
from ipywidgets import interact
import ipywidgets as widgets

# def get_contoursize_percentile(Y_trn, percentile):
#   # Percentile needs to be between 0 and 100
#     contoursize = []
#     for i in range(len(Y_trn)):
#         mask = Y_trn[i]
#         obj_list = np.unique(mask)
#         obj_list = obj_list[1:]

#         for j in range(len(obj_list)):
#             mask_temp = mask.copy()
#             mask_temp[mask_temp != obj_list[j]] = 0
#             mask_temp[mask_temp > 0] = 1

#             mask_temp = mask_temp.astype(np.uint8)
#             contours,_ = cv2.findContours(mask_temp, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
#             # areas = [cv2.contourArea(cnt) for cnt in contours]
#             # max_ind = np.argmax(areas)
#             # contour = np.squeeze(contours[max_ind])
#             contour = np.squeeze(contours[0])
#             contour = np.reshape(contour,(-1,2))
#             contour = np.append(contour,contour[0].reshape((-1,2)),axis=0)
#             contoursize = np.append(contoursize,contour.shape[0])

#     contoursize_percentile = np.percentile(contoursize, percentile)
#     return contoursize_percentile

def get_contoursize_percentile_from_path(target_path, percentile = 99, show_histogram = False):
  # Percentile needs to be between 0 and 100
  contoursize = []
  Y_list = glob(target_path+"/*.tif")
  for y in tqdm(Y_list):
    Y_im = imread(y)
    Y_im = fill_label_holes(Y_im)
    obj_list = np.unique(Y_im)
    obj_list = obj_list[1:]

    for j in range(len(obj_list)):
        mask_temp = Y_im.copy()
        mask_temp[mask_temp != obj_list[j]] = 0
        mask_temp[mask_temp > 0] = 1

        mask_temp = mask_temp.astype(np.uint8)
        contours,_ = cv2.findContours(mask_temp, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
        perimeter = cv2.arcLength(contours[0],True)
        contoursize = np.append(contoursize, perimeter)

  contoursize_max = np.amax(contoursize)
  contoursize_percentile = np.percentile(contoursize, percentile)

  if show_histogram:
    # Histogram display
    n, bins, patches = plt.hist(x=contoursize, bins='auto', color='#0504aa',
                                alpha=0.7, rwidth=0.85)
    plt.grid(axis='y', alpha=0.75)
    plt.xlabel('Contour size')
    plt.ylabel('Frequency')
    plt.title('Contour size distribution')
    plt.text(200, 300, r'$Max='+str(round(contoursize_max,2))+'$')
    plt.text(200, 280, r'$'+str(percentile)+'th-per.='+str(round(contoursize_percentile,2))+'$')
    maxfreq = n.max();
    # Set a clean upper y-axis limit.
    plt.ylim(ymax=np.ceil(maxfreq / 10) * 10 if maxfreq % 10 else maxfreq + 10);

  return contoursize_percentile

def pdf_export(trained = False, augmentation = False, pretrained_model = False):
  class MyFPDF(FPDF, HTMLMixin):
    pass

  pdf = MyFPDF()
  pdf.add_page()
  pdf.set_right_margin(-1)
  pdf.set_font("Arial", size = 11, style='B')

  Network = 'SplineDist 2D'
  day = datetime.now()
  datetime_str = str(day)[0:10]

  Header = 'Training report for '+Network+' model ('+model_name+')\nDate: '+datetime_str
  pdf.multi_cell(180, 5, txt = Header, align = 'L')

  # add another cell
  if trained:
    training_time = "Training time: "+str(hour)+ "hour(s) "+str(mins)+"min(s) "+str(round(sec))+"sec(s)"
    pdf.cell(190, 5, txt = training_time, ln = 1, align='L')
  pdf.ln(1)

  Header_2 = 'Information for your materials and method:'
  pdf.cell(190, 5, txt=Header_2, ln=1, align='L')

  all_packages = ''
  for requirement in freeze(local_only=True):
    all_packages = all_packages+requirement+', '
  #print(all_packages)

  #Main Packages
  main_packages = ''
  version_numbers = []
  for name in ['tensorflow','numpy','Keras','csbdeep']:
    find_name=all_packages.find(name)
    main_packages = main_packages+all_packages[find_name:all_packages.find(',',find_name)]+', '
    #Version numbers only here:
    version_numbers.append(all_packages[find_name+len(name)+2:all_packages.find(',',find_name)])

  cuda_version = subprocess.run('nvcc --version',stdout=subprocess.PIPE, shell=True)
  cuda_version = cuda_version.stdout.decode('utf-8')
  cuda_version = cuda_version[cuda_version.find(', V')+3:-1]
  gpu_name = subprocess.run('nvidia-smi',stdout=subprocess.PIPE, shell=True)
  gpu_name = gpu_name.stdout.decode('utf-8')
  gpu_name = gpu_name[gpu_name.find('Tesla'):gpu_name.find('Tesla')+10]

  shape = io.imread(Training_source+'/'+os.listdir(Training_source)[1]).shape
  dataset_size = len(os.listdir(Training_source))

  text = 'The '+Network+' model was trained from scratch for '+str(number_of_epochs)+' epochs on '+str(dataset_size)+' paired image patches (image dimensions: '+str(shape)+', patch size: ('+str(patch_size)+','+str(patch_size)+')) with a batch size of '+str(batch_size)+' and a '+conf.train_dist_loss+' loss function, using the '+Network+' ZeroCostDL4Mic notebook (v '+Notebook_version[0]+') (von Chamier & Laine et al., 2020). Key python packages used include tensorflow (v '+version_numbers[0]+'), Keras (v '+version_numbers[2]+'), csbdeep (v '+version_numbers[3]+'), numpy (v '+version_numbers[1]+'), cuda (v '+cuda_version+'). The training was accelerated using a '+gpu_name+'GPU.'

  if pretrained_model:
    text = 'The '+Network+' model was trained for '+str(number_of_epochs)+' epochs on '+str(dataset_size)+' paired image patches (image dimensions: '+str(shape)+', patch size: ('+str(patch_size)+','+str(patch_size)+')) with a batch size of '+str(batch_size)+' and a '+conf.train_dist_loss+' loss function, using the '+Network+' ZeroCostDL4Mic notebook (v '+Notebook_version[0]+') (von Chamier & Laine et al., 2020). The model was retrained from a pretrained model. Key python packages used include tensorflow (v '+version_numbers[0]+'), Keras (v '+version_numbers[2]+'), csbdeep (v '+version_numbers[3]+'), numpy (v '+version_numbers[1]+'), cuda (v '+cuda_version+'). The training was accelerated using a '+gpu_name+'GPU.'

  pdf.set_font('')
  pdf.set_font_size(10.)
  pdf.multi_cell(190, 5, txt = text, align='L')
  pdf.set_font('')
  pdf.set_font('Arial', size = 10, style = 'B')
  pdf.ln(1)
  pdf.cell(28, 5, txt='Augmentation: ', ln=0)
  pdf.set_font('')
  if augmentation:
    aug_text = 'The dataset was augmented by a factor of '+str(Multiply_dataset_by)

  else:
    aug_text = 'No augmentation was used for training.'
  pdf.multi_cell(190, 5, txt=aug_text, align='L')
  pdf.set_font('Arial', size = 11, style = 'B')
  pdf.ln(1)
  pdf.cell(180, 5, txt = 'Parameters', align='L', ln=1)
  pdf.set_font('')
  pdf.set_font_size(10.)
  if Use_Default_Advanced_Parameters:
    pdf.cell(200, 5, txt='Default Advanced Parameters were enabled')
  pdf.cell(200, 5, txt='The following parameters were used for training:')
  pdf.ln(1)
  html = """

      Parameter
      Value

      number_of_epochs
      {0}

      patch_size
      {1}

      batch_size
      {2}

      number_of_steps
      {3}

      percentage_validation
      {4}

      contoursize_max
      {5}

      grid_parameter
      {6}

      number_of_contour_points
      {7}

      initial_learning_rate
      {8}

  """.format(number_of_epochs,str(patch_size)+'x'+str(patch_size),batch_size,number_of_steps,percentage_validation,round(contoursize,2),grid_parameter,number_of_control_points,initial_learning_rate)
  pdf.write_html(html)

  #pdf.multi_cell(190, 5, txt = text_2, align='L')
  pdf.set_font("Arial", size = 11, style='B')
  pdf.ln(1)
  pdf.cell(190, 5, txt = 'Training Dataset', align='L', ln=1)
  pdf.set_font('')
  pdf.set_font('Arial', size = 10, style = 'B')
  pdf.cell(30, 5, txt= 'Training_source:', align = 'L', ln=0)
  pdf.set_font('')
  pdf.multi_cell(170, 5, txt = Training_source, align = 'L')
  pdf.set_font('')
  pdf.set_font('Arial', size = 10, style = 'B')
  pdf.cell(28, 5, txt= 'Training_target:', align = 'L', ln=0)
  pdf.set_font('')
  pdf.multi_cell(170, 5, txt = Training_target, align = 'L')
  #pdf.cell(190, 5, txt=aug_text, align='L', ln=1)
  pdf.ln(1)
  pdf.set_font('')
  pdf.set_font('Arial', size = 10, style = 'B')
  pdf.cell(21, 5, txt= 'Model Path:', align = 'L', ln=0)
  pdf.set_font('')
  pdf.multi_cell(170, 5, txt = model_path+'/'+model_name, align = 'L')
  pdf.ln(1)
  pdf.cell(60, 5, txt = 'Example Training pair', ln=1)
  pdf.ln(1)
  exp_size = io.imread('/content/TrainingDataExample_SplineDist2D.png').shape
  pdf.image('/content/TrainingDataExample_SplineDist2D.png', x = 11, y = None, w = round(exp_size[1]/8), h = round(exp_size[0]/8))
  pdf.ln(1)
  ref_1 = 'References:\n - ZeroCostDL4Mic: von Chamier, Lucas & Laine, Romain, et al. "ZeroCostDL4Mic: an open platform to simplify access and use of Deep-Learning in Microscopy." BioRxiv (2020).'
  pdf.multi_cell(190, 5, txt = ref_1, align='L')
  ref_2 = '- SplineDist 2D: Mandal et al. "SplineDist: Automated Cell Segmentation with Spline Curves. bioRxiv 2020.'
  pdf.multi_cell(190, 5, txt = ref_2, align='L')

  pdf.ln(3)
  reminder = 'Important:\nRemember to perform the quality control step on all newly trained models\nPlease consider depositing your training dataset on Zenodo'
  pdf.set_font('Arial', size = 11, style='B')
  pdf.multi_cell(190, 5, txt=reminder, align='C')

  pdf.output(model_path+'/'+model_name+'/'+model_name+"_training_report.pdf")

def qc_pdf_export():
  class MyFPDF(FPDF, HTMLMixin):
    pass

  pdf = MyFPDF()
  pdf.add_page()
  pdf.set_right_margin(-1)
  pdf.set_font("Arial", size = 11, style='B')

  Network = 'SplineDist 2D'

  day = datetime.now()
  date_time = str(day)[0:16]

  Header = 'Quality Control report for '+Network+' model ('+QC_model_name+')\nDate and Time: '+date_time
  pdf.multi_cell(180, 5, txt = Header, align = 'L')

  Header_2 = 'Information for your materials and methods:'
  pdf.cell(190, 5, txt=Header_2, ln=1, align='L')

  all_packages = ''
  for requirement in freeze(local_only=True):
    all_packages = all_packages+requirement+', '

  pdf.set_font('')
  pdf.set_font('Arial', size = 11, style = 'B')
  pdf.ln(2)
  pdf.cell(190, 5, txt = 'Development of Training Losses', ln=1, align='L')
  pdf.ln(1)
  exp_size = io.imread(full_QC_model_path+'/Quality Control/QC_example_data.png').shape
  if os.path.exists(full_QC_model_path+'/Quality Control/lossCurvePlots.png'):
    pdf.image(full_QC_model_path+'/Quality Control/lossCurvePlots.png', x = 11, y = None, w = round(exp_size[1]/16), h = round(exp_size[0]/4))
  else:
    pdf.set_font('')
    pdf.set_font('Arial', size=10)
    pdf.multi_cell(190, 5, txt='If you would like to see the evolution of the loss function during training please play the first cell of the QC section in the notebook.')
  pdf.ln(2)
  pdf.set_font('')
  pdf.set_font('Arial', size = 10, style = 'B')
  pdf.ln(3)
  pdf.cell(80, 5, txt = 'Example Quality Control Visualisation', ln=1)
  pdf.ln(1)
  exp_size = io.imread(full_QC_model_path+'/Quality Control/QC_example_data.png').shape
  pdf.image(full_QC_model_path+'/Quality Control/QC_example_data.png', x = 16, y = None, w = round(exp_size[1]/10), h = round(exp_size[0]/10))
  pdf.ln(1)
  pdf.set_font('')
  pdf.set_font('Arial', size = 11, style = 'B')
  pdf.ln(1)
  pdf.cell(180, 5, txt = 'Quality Control Metrics', align='L', ln=1)
  pdf.set_font('')
  pdf.set_font_size(10.)

  pdf.ln(1)
  html = """

"""
  with open(full_QC_model_path+'/Quality Control/Quality_Control for '+QC_model_name+'.csv', 'r') as csvfile:
    metrics = csv.reader(csvfile)
    header = next(metrics)
    #image = header[0]
    #PvGT_IoU = header[1]
    fp = header[2]
    tp = header[3]
    fn = header[4]
    precision = header[5]
    recall = header[6]
    acc = header[7]
    f1 = header[8]
    n_true = header[9]
    n_pred = header[10]
    mean_true = header[11]
    mean_matched = header[12]
    panoptic = header[13]
    header = """

    {0}
    {1}
    {2}
    {3}
    {4}
    {5}
    {6}
    {7}
    {8}
    {9}
    {10}
    {11}
    {12}
    {13}

""".format("image #","Prediction v. GT IoU",'false pos.','true pos.','false neg.',precision,recall,acc,f1,n_true,n_pred,mean_true,mean_matched,panoptic)
    html = html+header
    i=0
    for row in metrics:
      i+=1
      #image = row[0]
      PvGT_IoU = row[1]
      fp = row[2]
      tp = row[3]
      fn = row[4]
      precision = row[5]
      recall = row[6]
      acc = row[7]
      f1 = row[8]
      n_true = row[9]
      n_pred = row[10]
      mean_true = row[11]
      mean_matched = row[12]
      panoptic = row[13]
      cells = """

          {0}
          {1}
          {2}
          {3}
          {4}
          {5}
          {6}
          {7}
          {8}
          {9}
          {10}
          {11}
          {12}
          {13}

""".format(str(i),str(round(float(PvGT_IoU),3)),fp,tp,fn,str(round(float(precision),3)),str(round(float(recall),3)),str(round(float(acc),3)),str(round(float(f1),3)),n_true,n_pred,str(round(float(mean_true),3)),str(round(float(mean_matched),3)),str(round(float(panoptic),3)))
      html = html+cells
    html = html+"""
"""

  pdf.write_html(html)

  pdf.ln(1)
  pdf.set_font('')
  pdf.set_font_size(10.)
  ref_1 = 'References:\n - ZeroCostDL4Mic: von Chamier, Lucas & Laine, Romain, et al. "ZeroCostDL4Mic: an open platform to simplify access and use of Deep-Learning in Microscopy." BioRxiv (2020).'
  pdf.multi_cell(190, 5, txt = ref_1, align='L')
  ref_2 = '- StarDist 2D: Schmidt, Uwe, et al. "Cell detection with star-convex polygons." International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2018.'
  pdf.multi_cell(190, 5, txt = ref_2, align='L')

  pdf.ln(3)
  reminder = 'To find the parameters and other information about how this model was trained, go to the training_report.pdf of this model which should be in the folder of the same name.'

  pdf.set_font('Arial', size = 11, style='B')
  pdf.multi_cell(190, 5, txt=reminder, align='C')

  pdf.output(full_QC_model_path+'/Quality Control/'+QC_model_name+'_QC_report.pdf')

# Colors for the warning messages
class bcolors:
  WARNING = '\033[31m'
W  = '\033[0m'  # white (normal)
R  = '\033[31m' # red

#Disable some of the tensorflow warnings
import warnings
warnings.filterwarnings("ignore")

print("Libraries installed")

print('-----------------------------------------')

# Check if this is the latest version of the notebook
Latest_notebook_version = pd.read_csv("https://raw.githubusercontent.com/HenriquesLab/ZeroCostDL4Mic/master/Colab_notebooks/Latest_ZeroCostDL4Mic_Release.csv")
print('Notebook version: '+Notebook_version[0])
strlist = Notebook_version[0].split('.')
Notebook_version_main = strlist[0]+'.'+strlist[1]
if Notebook_version_main == Latest_notebook_version.columns:
  print("This notebook is up-to-date.")
else:
  print(bcolors.WARNING +"A new version of this notebook has been released. We recommend that you download it at https://github.com/HenriquesLab/ZeroCostDL4Mic/wiki")

# Exporting requirements.txt for local run
!pip freeze > requirements.txt
_____no_output_____

# **3. Select your parameters and paths**
---_____no_output_____## **3.1. Setting main training parameters**
---
 _____no_output_____ **Paths for training, predictions and results**

**`Training_source:`, `Training_target`:** These are the paths to your folders containing the Training_source (images of nuclei) and Training_target (masks) training data respecively. To find the paths of the folders containing the respective datasets, go to your Files on the left of the notebook, navigate to the folder containing your files and copy the path by right-clicking on the folder, **Copy path** and pasting it into the right box below.

**`model_name`:** Use only my_model -style, not my-model (Use "_" not "-"). Do not use spaces in the name. Avoid using the name of an existing model (saved in the same folder) as it will be overwritten.

**`model_path`**: Enter the path where your model will be saved once trained (for instance your result folder).

**Training parameters**

**`number_of_epochs`:** Input how many epochs (rounds) the network will be trained. Preliminary results can already be observed after a 50-100 epochs, but a full training should run for up to 400 epochs. Evaluate the performance after training (see 5.). **Default value: 100**

**`patch_size`:** Input the size of the patches use to train SplineDist 2D (length of a side). The value should be smaller or equal to the dimensions of the image. Make the patch size as large as possible and divisible by 8. **If your runtime crashes when you start the training, decrease your `patch_size`. Default value: 256**

**Advanced Parameters - experienced users only**

**`batch_size:`** This parameter defines the number of patches seen in each training step. Reducing or increasing the **batch size** may slow or speed up your training, respectively, and can influence network performance. **Default value: 4**

**`number_of_steps`:** Define the number of training steps by epoch. By default this parameter is calculated so that each image / patch is seen at least once per epoch. **Default value: Number of patch / batch_size**, **the default value is also used when set to 0**.

**`contoursize`:** Define the size of contour to use (object perimeter). **Default value: the 99th percentile of the perimeter distribution found in the training dataset**, **the default value is also used when set to 0**.

**`percentage_validation`:**  Input the percentage of your training dataset you want to use to validate the network during the training. **Default value: 10**

**`grid_parameter`:** increase this number if the object to segment are very large or decrease it if they are very small. **Default value: 2**

**`number_of_control_points`:** choose the number of control points. **Default value: 16**

**`initial_learning_rate`:** Input the initial value to be used as learning rate. **Default value: 0.0003**

**If you get an Out of memory (OOM) error during the training,  manually decrease the patch_size value until the OOM error disappear.**

_____no_output_____

#@markdown ###Path to training images:
Training_source = "" #@param {type:"string"}
Training_target = "" #@param {type:"string"}

#@markdown ###Name of the model and path to model folder:
model_name = "" #@param {type:"string"}

model_path = "" #@param {type:"string"}
#trained_model = model_path

#@markdown ### Other parameters for training:
number_of_epochs =  100#@param {type:"number"}
patch_size =  256#@param {type:"number"}

#@markdown ###Advanced Parameters
Use_Default_Advanced_Parameters = True #@param {type:"boolean"}

#@markdown ###If not, please input:

#GPU_limit = 90 #@param {type:"number"}
batch_size =  4#@param {type:"number"}
number_of_steps = 100#@param {type:"number"}
contoursize = 0#@param {type:"number"}
percentage_validation =  10 #@param {type:"number"}
grid_parameter = 2#@param [1, 2, 4, 8, 16, 32] {type:"raw"}
number_of_control_points =  16#@param {type:"number"}
initial_learning_rate = 0.0003 #@param {type:"number"}

if (Use_Default_Advanced_Parameters):
  print("Default advanced parameters enabled")
  batch_size = 4
  percentage_validation = 10
  grid_parameter = 2
  number_of_control_points = 16
  initial_learning_rate = 0.0003
  print('Estimating optimal contour size...')
  contoursize = get_contoursize_percentile_from_path(Training_target, 99, False)

else:
  if (contoursize == 0):
    print('Estimating optimal contour size...')
    contoursize = get_contoursize_percentile_from_path(Training_target, 99, True)

percentage = percentage_validation/100

#here we check that no model with the same name already exist, if so delete
if os.path.exists(model_path+'/'+model_name):
  print(bcolors.WARNING +"!! WARNING: Folder already exists and has been removed !!" + W)
  shutil.rmtree(model_path+'/'+model_name)

# Here we open will randomly chosen input and output image
random_choice = random.choice(os.listdir(Training_source))
x = imread(Training_source+"/"+random_choice)

# Here we check the image dimensions

Image_Y = x.shape[0]
Image_X = x.shape[1]

print('Loaded images (width, length) =', x.shape)

#Hyperparameters failsafes

# Here we check that patch_size is smaller than the smallest xy dimension of the image

if patch_size > min(Image_Y, Image_X):
  patch_size = min(Image_Y, Image_X)
  print(bcolors.WARNING + " Your chosen patch_size is bigger than the xy dimension of your image; therefore the patch_size chosen is now:",patch_size)

if patch_size > 2048:
  patch_size = 2048
  print(bcolors.WARNING + " Your image dimension is large; therefore the patch_size chosen is now:",patch_size)

# Here we check that the patch_size is divisible by 16
if not patch_size % 16 == 0:
    patch_size = ((int(patch_size / 16)-1) * 16)
    print(bcolors.WARNING + " Your chosen patch_size is not divisible by 8; therefore the patch_size chosen is:",patch_size)

# Here we disable pre-trained model by default (in case the next cell is not ran)
Use_pretrained_model = False

# Here we disable data augmentation by default (in case the cell is not ran)

Use_Data_augmentation = False

print("Patch size: ", patch_size)
print("Contour size: ", round(contoursize,2))
print("Parameters initiated.")

os.chdir(Training_target)
y = imread(Training_target+"/"+random_choice)

#Here we use a simple normalisation strategy to visualise the image
norm = simple_norm(x, percent = 99)

f=plt.figure(figsize=(16,8))
plt.subplot(1,2,1)
plt.imshow(x, interpolation='nearest', norm=norm, cmap='magma')
plt.title('Training source')
plt.axis('off');

plt.subplot(1,2,2)
plt.imshow(y, interpolation='nearest', cmap=lbl_cmap)
plt.title('Training target')
plt.axis('off');
plt.savefig('/content/TrainingDataExample_SplineDist2D.png',bbox_inches='tight',pad_inches=0)_____no_output_____

## **3.2. Data augmentation**
---
_____no_output_____Data augmentation can improve training progress by amplifying differences in the dataset. This can be useful if the available dataset is small since, in this case, it is possible that a network could quickly learn every example in the dataset (overfitting), without augmentation. Augmentation is not necessary for training and if your training dataset is large you should disable it.

Data augmentation is performed here via random rotations, flips, and intensity changes.

 **However, data augmentation is not a magic solution and may also introduce issues. Therefore, we recommend that you train your network with and without augmentation, and use the QC section to validate that it improves overall performances.**
_____no_output_____

#Data augmentation

Use_Data_augmentation = True #@param {type:"boolean"}

#@markdown ####Choose a factor by which you want to multiply your original dataset

Multiply_dataset_by = 1 #@param {type:"slider", min:1, max:10, step:1}

def random_fliprot(img, mask):
    assert img.ndim >= mask.ndim
    axes = tuple(range(mask.ndim))
    perm = tuple(np.random.permutation(axes))
    img = img.transpose(perm + tuple(range(mask.ndim, img.ndim)))
    mask = mask.transpose(perm)
    for ax in axes:
        if np.random.rand() > 0.5:
            img = np.flip(img, axis=ax)
            mask = np.flip(mask, axis=ax)
    return img, mask

def random_intensity_change(img):
    img = img*np.random.uniform(0.6,2) + np.random.uniform(-0.2,0.2)
    return img

def augmenter(x, y):
    """Augmentation of a single input/label image pair.
    x is an input image
    y is the corresponding ground-truth label image
    """
    x, y = random_fliprot(x, y)
    x = random_intensity_change(x)
    # add some gaussian noise
    sig = 0.02*np.random.uniform(0,1)
    x = x + sig*np.random.normal(0,1,x.shape)
    return x, y

if Use_Data_augmentation:
  augmenter = augmenter
  print("Data augmentation enabled")

if not Use_Data_augmentation:
  augmenter = None
  print(bcolors.WARNING+"Data augmentation disabled")

_____no_output_____

## **3.3. Using weights from a pre-trained model as initial weights**
---
  Here, you can set the the path to a pre-trained model from which the weights can be extracted and used as a starting point for this training session. **This pre-trained model needs to be a SplineDist model**.

 This option allows you to perform training over multiple Colab runtimes or to do transfer learning using models trained outside of ZeroCostDL4Mic. **You do not need to run this section if you want to train a network from scratch**.

 In order to continue training from the point where the pre-trained model left off, it is adviseable to also **load the learning rate** that was used when the training ended. This is automatically saved for models trained with ZeroCostDL4Mic and will be loaded here. If no learning rate can be found in the model folder provided, the default learning rate will be used. _____no_output_____

# @markdown ##Loading weights from a pre-trained network

Use_pretrained_model = False #@param {type:"boolean"}

pretrained_model_choice = "Model_from_file" #@param ["Model_from_file"]

Weights_choice = "best" #@param ["last", "best"]

#@markdown ###If you chose "Model_from_file", please provide the path to the model folder:
pretrained_model_path = "" #@param {type:"string"}

# --------------------- Check if we load a previously trained model ------------------------
if Use_pretrained_model:

# --------------------- Load the model from the choosen path ------------------------
  if pretrained_model_choice == "Model_from_file":
    h5_file_path = os.path.join(pretrained_model_path, "weights_"+Weights_choice+".h5")

# --------------------- Add additional pre-trained models here ------------------------

# --------------------- Check the model exist ------------------------
# If the model path chosen does not contain a pretrain model then use_pretrained_model is disabled,
  if not os.path.exists(h5_file_path):
    print(bcolors.WARNING+'WARNING: weights_last.h5 pretrained model does not exist' + W)
    Use_pretrained_model = False

# If the model path contains a pretrain model, we load the training rate,
  if os.path.exists(h5_file_path):
#Here we check if the learning rate can be loaded from the quality control folder
    if os.path.exists(os.path.join(pretrained_model_path, 'Quality Control', 'training_evaluation.csv')):

      with open(os.path.join(pretrained_model_path, 'Quality Control', 'training_evaluation.csv'),'r') as csvfile:
        csvRead = pd.read_csv(csvfile, sep=',')
        #print(csvRead)

        if "learning rate" in csvRead.columns: #Here we check that the learning rate column exist (compatibility with model trained un ZeroCostDL4Mic bellow 1.4)
          print("pretrained network learning rate found")
          #find the last learning rate
          lastLearningRate = csvRead["learning rate"].iloc[-1]
          #Find the learning rate corresponding to the lowest validation loss
          min_val_loss = csvRead[csvRead['val_loss'] == min(csvRead['val_loss'])]
          #print(min_val_loss)
          bestLearningRate = min_val_loss['learning rate'].iloc[-1]

          if Weights_choice == "last":
            print('Last learning rate: '+str(lastLearningRate))

          if Weights_choice == "best":
            print('Learning rate of best validation loss: '+str(bestLearningRate))

        if not "learning rate" in csvRead.columns: #if the column does not exist, then initial learning rate is used instead
          bestLearningRate = initial_learning_rate
          lastLearningRate = initial_learning_rate
          print(bcolors.WARNING+'WARNING: The learning rate cannot be identified from the pretrained network. Default learning rate of '+str(bestLearningRate)+' will be used instead' + W)

#Compatibility with models trained outside ZeroCostDL4Mic but default learning rate will be used
    if not os.path.exists(os.path.join(pretrained_model_path, 'Quality Control', 'training_evaluation.csv')):
      print(bcolors.WARNING+'WARNING: The learning rate cannot be identified from the pretrained network. Default learning rate of '+str(initial_learning_rate)+' will be used instead'+ W)
      bestLearningRate = initial_learning_rate
      lastLearningRate = initial_learning_rate

# Display info about the pretrained model to be loaded (or not)
if Use_pretrained_model:
  print('Weights found in:')
  print(h5_file_path)
  print('will be loaded prior to training.')

else:
  print(bcolors.WARNING+'No pretrained network will be used.')

_____no_output_____

#**4. Train the network**
---_____no_output_____## **4.1. Prepare the training data and model for training**
---
Here, we use the information from 3. to build the model and convert the training data into a suitable format for training._____no_output_____

#@markdown ##Create the model and dataset objects

Training_source_dir = Training_source
Training_target_dir = Training_target
# --------------------- ------------------------------------------------

training_images_tiff=Training_source_dir+"/*.tif"
mask_images_tiff=Training_target_dir+"/*.tif"

# this funtion imports training images and masks and sorts them suitable for the network
X = sorted(glob(training_images_tiff))
Y = sorted(glob(mask_images_tiff))

# assert -funtion check that X and Y really have images. If not this cell raises an error
assert all(Path(x).name==Path(y).name for x,y in zip(X,Y))

# Here we map the training dataset (images and masks).
X = list(map(imread,X))
Y = list(map(imread,Y))
n_channel = 1 if X[0].ndim == 2 else X[0].shape[-1]

#Normalize images and fill small label holes.
if n_channel == 1:
  axis_norm = (0,1)   # normalize channels independently
  print("Normalizing image channels independently")

if n_channel > 1:
  axis_norm = (0,1,2) # normalize channels jointly
  print("Normalizing image channels jointly")
  sys.stdout.flush()

X = [normalize(x,1,99.8,axis=axis_norm) for x in tqdm(X)]
Y = [fill_label_holes(y) for y in tqdm(Y)]

#Here we split the training dataset into training images (90 %) and validation images (10 %).
#It is advisable to use 10 % of your training dataset for validation. This ensures the truthfull validation error value. If only few validation images are used network may choose too easy or too challenging images for validation.
# split training data (images and masks) into training images and validation images.
assert len(X) > 1, "not enough training data"
rng = np.random.RandomState(42)
ind = rng.permutation(len(X))
n_val = max(1, int(round(percentage * len(ind))))
ind_train, ind_val = ind[:-n_val], ind[-n_val:]
X_val, Y_val = [X[i] for i in ind_val]  , [Y[i] for i in ind_val]
X_trn, Y_trn = [X[i] for i in ind_train], [Y[i] for i in ind_train]
print('number of images: %3d' % len(X))
print('- training:       %3d' % len(X_trn))
print('- validation:     %3d' % len(X_val))

# Use OpenCL-based computations for data generator during training (requires 'gputools')
# Currently always False for stability
use_gpu = False and gputools_available()

#Here we ensure that our network has a minimal number of steps

if (Use_Default_Advanced_Parameters) or (number_of_steps == 0):
  # number_of_steps = (int(len(X)/batch_size)+1)
  number_of_steps = Image_X*Image_Y/(patch_size*patch_size)*(int(len(X)/batch_size)+1)
  if (Use_Data_augmentation):
    augmentation_factor = Multiply_dataset_by
    number_of_steps = number_of_steps * augmentation_factor

print('Number of steps: '+str(number_of_steps))

# --------------------- Using pretrained model ------------------------
#Here we ensure that the learning rate set correctly when using pre-trained models
if Use_pretrained_model:
  if Weights_choice == "last":
    initial_learning_rate = lastLearningRate

  if Weights_choice == "best":
    initial_learning_rate = bestLearningRate
# --------------------- ---------------------- ------------------------

# # compute the size of the largest contour present in the image-set
# Now executed above
# if contoursize_max == 0:
#   # contoursize_max = get_contoursize_max(Y_trn)
#   contoursize_99p = get_contoursize_percentile(Y_trn, 99) # 99th percentile

# # print('Maximum contour size: '+str(contoursize_max))
# print('99th percentile contour size: '+str(contoursize_99p))

conf = Config2D (
    use_gpu      = use_gpu,
    train_batch_size = batch_size,
    n_channel_in = n_channel,
    train_patch_size = (patch_size, patch_size),
    grid = (grid_parameter, grid_parameter),
    train_learning_rate = initial_learning_rate,
    n_params        = 2*number_of_control_points,
    contoursize_max = contoursize,
)

# Here we create a model according to section 5.3.
model = SplineDist2D(conf, name=model_name, basedir=model_path)

os.chdir(model_path+'/'+model_name)
phi_generator(number_of_control_points, conf.contoursize_max)
grid_generator(number_of_control_points, conf.train_patch_size, conf.grid)

# --------------------- Using pretrained model ------------------------
# Load the pretrained weights
if Use_pretrained_model:
  model.load_weights(h5_file_path)

# --------------------- ---------------------- ------------------------

#Here we check the FOV of the network.
median_size = calculate_extents(list(Y), np.median)
fov = np.array(model._axes_tile_overlap('YX'))
if any(median_size > fov):
    print(bcolors.WARNING+"WARNING: median object size larger than field of view of the neural network.")
print(conf)

#Export pdf summary of training parameters
pdf_export(augmentation = Use_Data_augmentation, pretrained_model = Use_pretrained_model)

_____no_output_____

## **4.2. Start training**
---
When playing the cell below you should see updates after each epoch (round). Network training can take some time.

* **CRITICAL NOTE:** Google Colab has a time limit for processing (to prevent using GPU power for datamining). Training time must be less than 12 hours! If training takes longer than 12 hours, please decrease the number of epochs or number of patches. Another way circumvent this is to save the parameters of the model after training and start training again from this point.

Once training is complete, the trained model is automatically saved on your Google Drive, in the **model_path** folder that was selected in Section 3. It is however wise to download the folder from Google Drive as all data can be erased at the next training if using the same folder.
_____no_output_____

start = time.time()

#@markdown ##Start training

# Training the model.

history = model.train(X_trn, Y_trn, validation_data=(X_val,Y_val), augmenter=augmenter,
                      epochs=number_of_epochs, steps_per_epoch=number_of_steps)

print("Training done")

print("Network optimization in progress")
#Here we optimize the network.

model.optimize_thresholds(X_val, Y_val)

print("Done")

# convert the history.history dict to a pandas DataFrame:
lossData = pd.DataFrame(history.history)

if os.path.exists(model_path+"/"+model_name+"/Quality Control"):
  shutil.rmtree(model_path+"/"+model_name+"/Quality Control")

os.makedirs(model_path+"/"+model_name+"/Quality Control")

# The training evaluation.csv is saved (overwrites the Files if needed).
lossDataCSVpath = model_path+'/'+model_name+'/Quality Control/training_evaluation.csv'
with open(lossDataCSVpath, 'w') as f:
  writer = csv.writer(f)
  writer.writerow(['loss','val_loss', 'learning rate'])
  for i in range(len(history.history['loss'])):
    writer.writerow([history.history['loss'][i], history.history['val_loss'][i], history.history['lr'][i]])

# Displaying the time elapsed for training
dt = time.time() - start
mins, sec = divmod(dt, 60)
hour, mins = divmod(mins, 60)
print("Time elapsed:",hour, "hour(s)",mins,"min(s)",round(sec),"sec(s)")

#Create a pdf document with training summary
pdf_export(trained = True, augmentation = Use_Data_augmentation, pretrained_model = Use_pretrained_model)

# Displaying the validation data

Y_val_pred = [model.predict_instances(x, n_tiles=model._guess_n_tiles(x), show_tile_progress=False)[0]
              for x in tqdm(X_val)]

def plot_img_label(img, lbl, img_title="image", lbl_title="label", **kwargs):
  fig, (ai,al) = plt.subplots(1,2, figsize=(12,5), gridspec_kw=dict(width_ratios=(1.25,1)))
  im = ai.imshow(img, cmap='gray', clim=(0,1))
  ai.set_title(img_title)
  fig.colorbar(im, ax=ai)
  al.imshow(lbl, cmap=lbl_cmap)
  al.set_title(lbl_title)
  plt.tight_layout()

plot_img_label(X_val[0],Y_val[0], lbl_title="label GT")
plot_img_label(X_val[0],Y_val_pred[0], lbl_title="label Pred")

_____no_output_____

# **5. Evaluate your model**
---

This section allows the user to perform important quality checks on the validity and generalisability of the trained model.

**We highly recommend to perform quality control on all newly trained models.**

_____no_output_____

# model name and path
#@markdown ###Do you want to assess the model you just trained ?
Use_the_current_trained_model = True #@param {type:"boolean"}

#@markdown ###If not, please provide the path to the model folder:

QC_model_folder = "" #@param {type:"string"}

#Here we define the loaded model name and path
QC_model_name = os.path.basename(QC_model_folder)
QC_model_path = os.path.dirname(QC_model_folder)

if (Use_the_current_trained_model):
  QC_model_name = model_name
  QC_model_path = model_path

full_QC_model_path = QC_model_path+'/'+QC_model_name+'/'
if os.path.exists(full_QC_model_path):
  print("The "+QC_model_name+" network will be evaluated")
else:
  print(bcolors.WARNING+'!! WARNING: The chosen model does not exist !!')
  print('Please make sure you provide a valid model path and model name before proceeding further.')
_____no_output_____

## **5.1. Inspection of the loss function**
---

First, it is good practice to evaluate the training progress by comparing the training loss with the validation loss. The latter is a metric which shows how well the network performs on a subset of unseen data which is set aside from the training dataset. For more information on this, see for example [this review](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6381354/) by Nichols *et al.*

**Training loss** describes an error value after each epoch for the difference between the model's prediction and its ground-truth target.

**Validation loss** describes the same error value between the model's prediction on a validation image and compared to it's target.

During training both values should decrease before reaching a minimal value which does not decrease further even after more training. Comparing the development of the validation loss with the training loss can give insights into the model's performance.

Decreasing **Training loss** and **Validation loss** indicates that training is still necessary and increasing the `number_of_epochs` is recommended. Note that the curves can look flat towards the right side, just because of the y-axis scaling. The network has reached convergence once the curves flatten out. After this point no further training is required. If the **Validation loss** suddenly increases again an the **Training loss** simultaneously goes towards zero, it means that the network is overfitting to the training data. In other words the network is remembering the exact patterns from the training data and no longer generalizes well to unseen data. In this case the training dataset has to be increased._____no_output_____

#@markdown ##Play the cell to show a plot of training errors vs. epoch number

lossDataFromCSV = []
vallossDataFromCSV = []

with open(QC_model_path+'/'+QC_model_name+'/Quality Control/training_evaluation.csv','r') as csvfile:
    csvRead = csv.reader(csvfile, delimiter=',')
    next(csvRead)
    for row in csvRead:
        lossDataFromCSV.append(float(row[0]))
        vallossDataFromCSV.append(float(row[1]))

epochNumber = range(len(lossDataFromCSV))
plt.figure(figsize=(15,10))

plt.subplot(2,1,1)
plt.plot(epochNumber,lossDataFromCSV, label='Training loss')
plt.plot(epochNumber,vallossDataFromCSV, label='Validation loss')
plt.title('Training loss and validation loss vs. epoch number (linear scale)')
plt.ylabel('Loss')
plt.xlabel('Epoch number')
plt.legend()

plt.subplot(2,1,2)
plt.semilogy(epochNumber,lossDataFromCSV, label='Training loss')
plt.semilogy(epochNumber,vallossDataFromCSV, label='Validation loss')
plt.title('Training loss and validation loss vs. epoch number (log scale)')
plt.ylabel('Loss')
plt.xlabel('Epoch number')
plt.legend()
plt.savefig(QC_model_path+'/'+QC_model_name+'/Quality Control/lossCurvePlots.png')
plt.show()

_____no_output_____

## **5.2. Error mapping and quality metrics estimation**
---
This section will calculate the Intersection over Union score for all the images provided in the Source_QC_folder and Target_QC_folder ! The result for one of the image will also be displayed.

The **Intersection over Union** (IuO) metric is a method that can be used to quantify the percent overlap between the target mask and your prediction output. **Therefore, the closer to 1, the better the performance.** This metric can be used to assess the quality of your model to accurately predict nuclei.

Here, the IuO is both calculated over the whole image and on a per-object basis. The value displayed below is the IuO value calculated over the entire image. The IuO value calculated on a per-object basis is used to calculate the other metrics displayed.

“n_true” refers to the number of objects present in the ground truth image. “n_pred” refers to the number of objects present in the predicted image.

When a segmented object has an IuO value above 0.5 (compared to the corresponding ground truth), it is then considered a true positive. The number of “**true positives**” is available in the table below. The number of “false positive” is then defined as  “**false positive**” = “n_pred” - “true positive”. The number of “false negative” is defined as “false negative” = “n_true” - “true positive”.

The mean_matched_score is the mean IoUs of matched true positives. The mean_true_score is the mean IoUs of matched true positives but normalized by the total number of ground truth objects. The panoptic_quality is calculated as described by [Kirillov et al. 2019](https://arxiv.org/abs/1801.00868).

For more information about the other metric displayed, please consult the SI of the paper describing ZeroCostDL4Mic.

 The results can be found in the "*Quality Control*" folder which is located inside your "model_folder"._____no_output_____

#@markdown ##Choose the folders that contain your Quality Control dataset

Source_QC_folder = "" #@param{type:"string"}
Target_QC_folder = "" #@param{type:"string"}

#Create a quality control Folder and check if the folder already exist
if os.path.exists(QC_model_path+"/"+QC_model_name+"/Quality Control") == False:
  os.makedirs(QC_model_path+"/"+QC_model_name+"/Quality Control")

if os.path.exists(QC_model_path+"/"+QC_model_name+"/Quality Control/Prediction"):
  shutil.rmtree(QC_model_path+"/"+QC_model_name+"/Quality Control/Prediction")

os.makedirs(QC_model_path+"/"+QC_model_name+"/Quality Control/Prediction")

# Generate predictions from the Source_QC_folder and save them in the QC folder

Source_QC_folder_tif = Source_QC_folder+"/*.tif"

np.random.seed(16)
lbl_cmap = random_label_cmap()
Z = sorted(glob(Source_QC_folder_tif))
Z = list(map(imread,Z))
n_channel = 1 if Z[0].ndim == 2 else Z[0].shape[-1]

print('Number of test dataset found in the folder: '+str(len(Z)))

#Normalize images.

if n_channel == 1:
  axis_norm = (0,1)   # normalize channels independently
  print("Normalizing image channels independently")

if n_channel > 1:
  axis_norm = (0,1,2) # normalize channels jointly
  print("Normalizing image channels jointly")

model = SplineDist2D(None, name=QC_model_name, basedir=QC_model_path)
names = [os.path.basename(f) for f in sorted(glob(Source_QC_folder_tif))]

# modify the names to suitable form: path_images/image_numberX.tif

lenght_of_Z = len(Z)

print('Running predictions...')
for i in tqdm(range(lenght_of_Z)):
  img = normalize(Z[i], 1,99.8, axis=axis_norm)
  os.chdir(QC_model_path+"/"+QC_model_name)
  labels, details = model.predict_instances(img)
  os.chdir(QC_model_path+"/"+QC_model_name+"/Quality Control/Prediction")
  imsave(names[i], labels, details)

# Here we start testing the differences between GT and predicted masks

with open(QC_model_path+"/"+QC_model_name+"/Quality Control/Quality_Control for "+QC_model_name+".csv", "w", newline='') as file:
  writer = csv.writer(file)
  writer.writerow(["image","Prediction v. GT Intersection over Union", "false positive", "true positive", "false negative", "precision", "recall", "accuracy", "f1 score", "n_true", "n_pred", "mean_true_score", "mean_matched_score", "panoptic_quality"])

# define the images

  for n in os.listdir(Source_QC_folder):

    if not os.path.isdir(os.path.join(Source_QC_folder,n)):
      print('Running QC on: '+n)
      test_input = io.imread(os.path.join(Source_QC_folder,n))
      test_prediction = io.imread(os.path.join(QC_model_path+"/"+QC_model_name+"/Quality Control/Prediction",n))
      test_ground_truth_image = io.imread(os.path.join(Target_QC_folder, n))

      # Calculate the matching (with IoU threshold `thresh`) and all metrics

      stats = matching(test_prediction, test_ground_truth_image, thresh=0.5)

      #Convert pixel values to 0 or 255
      test_prediction_0_to_255 = test_prediction
      test_prediction_0_to_255[test_prediction_0_to_255>0] = 255

      #Convert pixel values to 0 or 255
      test_ground_truth_0_to_255 = test_ground_truth_image
      test_ground_truth_0_to_255[test_ground_truth_0_to_255>0] = 255

      # Intersection over Union metric

      intersection = np.logical_and(test_ground_truth_0_to_255, test_prediction_0_to_255)
      union = np.logical_or(test_ground_truth_0_to_255, test_prediction_0_to_255)
      iou_score =  np.sum(intersection) / np.sum(union)
      writer.writerow([n, str(iou_score), str(stats.fp), str(stats.tp), str(stats.fn), str(stats.precision), str(stats.recall), str(stats.accuracy), str(stats.f1), str(stats.n_true), str(stats.n_pred), str(stats.mean_true_score), str(stats.mean_matched_score), str(stats.panoptic_quality)])

from tabulate import tabulate

df = pd.read_csv (QC_model_path+"/"+QC_model_name+"/Quality Control/Quality_Control for "+QC_model_name+".csv")
print(tabulate(df, headers='keys', tablefmt='psql'))

from astropy.visualization import simple_norm

# ------------- For display ------------
print('--------------------------------------------------------------')
@interact
def show_QC_results(file = os.listdir(Source_QC_folder)):

  plt.figure(figsize=(25,5))
  source_image = io.imread(os.path.join(Source_QC_folder, file), as_gray = True)
  target_image = io.imread(os.path.join(Target_QC_folder, file), as_gray = True)
  prediction = io.imread(QC_model_path+"/"+QC_model_name+"/Quality Control/Prediction/"+file, as_gray = True)

  stats = matching(prediction, target_image, thresh=0.5)

  target_image_mask = np.empty_like(target_image)
  target_image_mask[target_image > 0] = 255
  target_image_mask[target_image == 0] = 0

  prediction_mask = np.empty_like(prediction)
  prediction_mask[prediction > 0] = 255
  prediction_mask[prediction == 0] = 0

  intersection = np.logical_and(target_image_mask, prediction_mask)
  union = np.logical_or(target_image_mask, prediction_mask)
  iou_score =  np.sum(intersection) / np.sum(union)

  norm = simple_norm(source_image, percent = 99)

  #Input
  plt.subplot(1,4,1)
  plt.axis('off')
  plt.imshow(source_image, aspect='equal', norm=norm, cmap='magma', interpolation='nearest')
  plt.title('Input')

  #Ground-truth
  plt.subplot(1,4,2)
  plt.axis('off')
  plt.imshow(target_image_mask, aspect='equal', cmap='Greens')
  plt.title('Ground Truth')

  #Prediction
  plt.subplot(1,4,3)
  plt.axis('off')
  plt.imshow(prediction_mask, aspect='equal', cmap='Purples')
  plt.title('Prediction')

  #Overlay
  plt.subplot(1,4,4)
  plt.axis('off')
  plt.imshow(target_image_mask, cmap='Greens')
  plt.imshow(prediction_mask, alpha=0.5, cmap='Purples')
  plt.title('Ground Truth and Prediction, Intersection over Union:'+str(round(iou_score,3 )));
  plt.savefig(full_QC_model_path+'/Quality Control/QC_example_data.png',bbox_inches='tight',pad_inches=0)

qc_pdf_export()_____no_output_____

# **6. Using the trained model**

---

In this section the unseen data is processed using the trained model (in section 4). First, your unseen images are uploaded and prepared for prediction. After that your trained model from section 4 is activated and finally saved into your Google Drive._____no_output_____

## **6.1 Generate prediction(s) from unseen dataset**
---

In this section the unseen data is processed using the trained model (in section 4). First, your unseen images are uploaded and prepared for prediction. After that your trained model from section 4 is activated and finally saved into your Google Drive.

---

The current trained model (from section 4.3) can now be used to process images. If an older model needs to be used, please untick the **Use_the_current_trained_model** box and enter the name and path of the model to use. Predicted output images are saved in your **Prediction_folder** folder as restored image stacks (ImageJ-compatible TIFF images).

**`Data_folder`:** This folder should contains the images that you want to predict using the network that you will train.

**`Result_folder`:** This folder will contain the predicted output ROI.

**`Data_type`:** Please indicate if the images you want to predict are single images or stacks

In SplineDist the following results can be exported:

- The predicted mask images
- A tracking file that can easily be imported into Trackmate to track the nuclei (Stacks only).
- A CSV file that contains the number of nuclei detected per image (single image only).
- A CSV file that contains the coordinate the centre of each detected nuclei (single image only).

_____no_output_____

Single_Images = 1
Stacks = 2

#@markdown ### Provide the path to your dataset and to the folder where the prediction will be saved (Result folder), then play the cell to predict output on your unseen images.

Data_folder = "" #@param {type:"string"}
Results_folder = "" #@param {type:"string"}

#@markdown ###Are your data single images or stacks?

Data_type = Single_Images #@param ["Single_Images", "Stacks"] {type:"raw"}

#@markdown ###What outputs would you like to generate?

Mask_images = True #@param {type:"boolean"}
Tracking_file = False #@param {type:"boolean"}

# model name and path
#@markdown ###Do you want to use the current trained model?
Use_the_current_trained_model = False #@param {type:"boolean"}

#@markdown ###If not, please provide the path to the model folder:

Prediction_model_folder = "" #@param {type:"string"}

#Here we find the loaded model name and parent path
Prediction_model_name = os.path.basename(Prediction_model_folder)
Prediction_model_path = os.path.dirname(Prediction_model_folder)

if (Use_the_current_trained_model):
  print("Using current trained network")
  Prediction_model_name = model_name
  Prediction_model_path = model_path

full_Prediction_model_path = Prediction_model_path+'/'+Prediction_model_name+'/'
if os.path.exists(full_Prediction_model_path):
  print("The "+Prediction_model_name+" network will be used.")
else:
  print(bcolors.WARNING+'!! WARNING: The chosen model does not exist !!'+W)
  print('Please make sure you provide a valid model path and model name before proceeding further.')

#single images

if Data_type == 1 :
  print("Single images are now being predicted")

  Nuclei_number = []

  for image in os.listdir(Data_folder):

    print("Performing prediction on: "+image)

    X = imread(Data_folder+"/"+image)
    n_channel = 1 if X.ndim == 2 else X.shape[-1]

    if n_channel == 1:
      axis_norm = (0,1)   # normalize channels independently
      print("Normalizing image channels independently")

    if n_channel > 1:
      axis_norm = (0,1,2) # normalize channels jointly
      print("Normalizing image channels jointly")
      sys.stdout.flush()

    model = SplineDist2D(None, name = Prediction_model_name, basedir = Prediction_model_path)
    names = [os.path.basename(f) for f in sorted(glob(Data_folder))]

    short_name = os.path.splitext(image)

  # Save all ROIs and masks into results folder

    img = normalize(X, 1,99.8, axis = axis_norm)
    os.chdir(full_Prediction_model_path)
    labels, details = model.predict_instances(img)
    os.chdir(Results_folder)

    if Mask_images:
      imsave(str(short_name[0])+".tif", labels)

    if Tracking_file:
      print(bcolors.WARNING+"Tracking files are only generated when stacks are predicted"+W)

    Nuclei_centre_coordinate = details['points']
    my_df2 = pd.DataFrame(Nuclei_centre_coordinate)
    my_df2.columns =['Y', 'X']

    my_df2.to_csv(Results_folder+'/'+str(short_name[0])+'_object_centre.csv', index=False, header=True)

    Nuclei_array = details['coord']
    Nuclei_array2 = [str(short_name[0]), Nuclei_array.shape[0]]
    Nuclei_number.append(Nuclei_array2)

  my_df = pd.DataFrame(Nuclei_number)
  my_df.to_csv(Results_folder+'/object_count.csv', index=False, header=False)

  # One example is displayed

  print("One example image is displayed bellow:")
  plt.figure(figsize=(10,10))
  plt.imshow(img if img.ndim==2 else img[...,:3], clim=(0,1), cmap='gray')
  plt.imshow(labels, cmap=lbl_cmap, alpha=0.5)
  plt.axis('off');
  plt.savefig(str(short_name[0])+"_overlay.tif")

if Data_type == 2 :
  print("Stacks are being predicted")
  np.random.seed(42)
  lbl_cmap = random_label_cmap()

  # normalize channels independently
  axis_norm = (0,1)
  print("Normalizing image channels independently")

  #Load a pretrained network
  model = SplineDist2D(None, name = Prediction_model_name, basedir = Prediction_model_path)

  for image in os.listdir(Data_folder):
    print("Performing prediction on: "+image)

    timelapse = imread(Data_folder+"/"+image)

    short_name = os.path.splitext(image)

    timelapse = normalize(timelapse, 1,99.8, axis=(0,)+tuple(1+np.array(axis_norm)))

    n_timepoint = timelapse.shape[0]
    prediction_stack = np.zeros((n_timepoint, timelapse.shape[1], timelapse.shape[2]))

    Tracking_stack = np.zeros((n_timepoint, timelapse.shape[2], timelapse.shape[1]))

# Save the masks in the result folder
    if Mask_images or Tracking_file:
      for t in tqdm(range(n_timepoint)):
        img_t = timelapse[t]
        os.chdir(full_Prediction_model_path)
        labels, details = model.predict_instances(img_t)
        prediction_stack[t] = labels

# Create a tracking file for trackmate

        for point in details['points']:
          cv2.circle(Tracking_stack[t],tuple(point),0,(1), -1)

      prediction_stack_32 = img_as_float32(prediction_stack, force_copy=False)

      Tracking_stack_32 = img_as_float32(Tracking_stack, force_copy=False)
      Tracking_stack_8 = img_as_ubyte(Tracking_stack_32, force_copy=True)

      Tracking_stack_8_rot = np.rot90(Tracking_stack_8, axes=(1,2))
      Tracking_stack_8_rot_flip = np.fliplr(Tracking_stack_8_rot)

      os.chdir(Results_folder)
      if Mask_images:
        imsave("Predicted_"+str(short_name[0]+".tif"), prediction_stack_32)
      if Tracking_file:
        imsave(str(short_name[0])+"_tracking_file.tif", Tracking_stack_8_rot_flip)

print('---------------------')
print("Predictions completed.")   _____no_output_____

## **6.2. Download your predictions**
---

**Store your data** and ALL its results elsewhere by downloading it from Google Drive and after that clean the original folder tree (datasets, results, trained model etc.) if you plan to train or use new networks. Please note that the notebook will otherwise **OVERWRITE** all files which have the same name._____no_output_____
#**Thank you for using SplineDist 2D!**_____no_output_____


# Notebook from driscolljt/Machine-Learning-for-Algorithmic-Trading-Second-Edition_Original
Path: 14_working_with_text_data/06_sentiment_analysis_yelp.ipynb

# Text classification and sentiment analysis: Yelp Reviews_____no_output_____Once text data has been converted into numerical features using the natural language processing techniques discussed in the previous sections, text classification works just like any other classification task.

In this notebook, we will apply these preprocessing technique to Yelp business reviews to classify them by review scores and sentiment polarity. More specifically, we will apply sentiment analysis to the significantly larger Yelp business review dataset with five outcome classes. _____no_output_____## Imports_____no_output_____

import warnings
warnings.filterwarnings('ignore')_____no_output_____%matplotlib inline

from pathlib import Path
import json
from time import time

import numpy as np
import pandas as pd

from scipy import sparse

# spacy, textblob and nltk for language processing
from textblob import TextBlob

# sklearn for feature extraction & modeling
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix
import joblib

import lightgbm as lgb

# Visualization
import matplotlib.pyplot as plt
import seaborn as sns
_____no_output_____sns.set_style('white')
np.random.seed(42)_____no_output_____

## Yelp Challenge: business reviews dataset_____no_output_____### Load Data_____no_output_____Follow the [instructions](../data/create_yelp_review_data.ipynb) to create the dataset. _____no_output_____

data_dir = Path('..', 'data', 'yelp')_____no_output_____yelp_reviews = pd.read_parquet(data_dir / 'user_reviews.parquet')_____no_output_____yelp_reviews.info(null_counts=True)<class 'pandas.core.frame.DataFrame'>
Int64Index: 8021122 entries, 0 to 8021121
Data columns (total 25 columns):
 #   Column              Non-Null Count    Dtype
---  ------              --------------    -----
 0   stars               8021122 non-null  float64
 1   useful              8021122 non-null  int64
 2   funny               8021122 non-null  int64
 3   cool                8021122 non-null  int64
 4   text                8021122 non-null  object
 5   year                8021122 non-null  int64
 6   month               8021122 non-null  int64
 7   review_count        8021122 non-null  int64
 8   useful_user         8021122 non-null  int64
 9   funny_user          8021122 non-null  int64
 10  cool_user           8021122 non-null  int64
 11  fans                8021122 non-null  int64
 12  average_stars       8021122 non-null  float64
 13  compliment_hot      8021122 non-null  int64
 14  compliment_more     8021122 non-null  int64
 15  compliment_profile  8021122 non-null  int64
 16  compliment_cute     8021122 non-null  int64
 17  compliment_list     8021122 non-null  int64
 18  compliment_note     8021122 non-null  int64
 19  compliment_plain    8021122 non-null  int64
 20  compliment_cool     8021122 non-null  int64
 21  compliment_funny    8021122 non-null  int64
 22  compliment_writer   8021122 non-null  int64
 23  compliment_photos   8021122 non-null  int64
 24  member_yrs          8021122 non-null  int64
dtypes: float64(2), int64(22), object(1)
memory usage: 1.6+ GB

### Explore data_____no_output_____

yelp_dir = Path('results', 'yelp')

text_features_dir = yelp_dir / 'data'
if not text_features_dir.exists():
    text_features_dir.mkdir(exist_ok=True, parents=True)_____no_output_____

The following figure shows the number of reviews and the average number of stars per year._____no_output_____#### Reviews & Stars by Year_____no_output_____

fig, axes = plt.subplots(ncols=3, figsize=(18, 4))
yelp_reviews.year.value_counts().sort_index().plot.bar(title='Reviews per Year', ax=axes[0], rot=0);
sns.lineplot(x='year', y='stars', data=yelp_reviews, ax=axes[1])
axes[1].set_title('Stars per year')

stars_dist = yelp_reviews.stars.value_counts(normalize=True).sort_index().mul(100)
stars_dist.index = stars_dist.index.astype(int)
stars_dist.plot.barh(title='# Stars Breakdown', ax=axes[2])
axes[2].set_xlabel('Share of all Ratings (%)')
axes[2].set_ylabel('Number of Stars');

sns.despine()
fig.tight_layout();_____no_output_____

#### Years of Membership Breakdown_____no_output_____

ax = yelp_reviews.member_yrs.value_counts().div(1000).sort_index().plot.bar(title='Years of Membership',
                                                                            rot=0)
ax.set_xlabel('Number of Years')
ax.set_ylabel("Number of Members  ('000)")
sns.despine()
plt.tight_layout()_____no_output_____

### Create train-test split_____no_output_____

train = yelp_reviews[yelp_reviews.year < 2019].sample(frac=.25)
test = yelp_reviews[yelp_reviews.year == 2019]_____no_output_____print(f'# Training Obs: {len(train):,.0f} | # Test Obs: {len(test):,.0f}')# Training Obs: 1,701,322 | # Test Obs: 1,215,836
train.to_parquet(text_features_dir / 'train.parquet')
test.to_parquet(text_features_dir / 'test.parquet')_____no_output_____del yelp_reviews_____no_output_____

#### Reload stored data_____no_output_____

train = pd.read_parquet(text_features_dir / 'train.parquet')
test = pd.read_parquet(text_features_dir / 'test.parquet')_____no_output_____

## Create Yelp review document-term matrix_____no_output_____

vectorizer = CountVectorizer(stop_words='english', ngram_range=(1, 2), max_features=10000)
train_dtm = vectorizer.fit_transform(train.text)
train_dtm_____no_output_____sparse.save_npz(text_features_dir / 'train_dtm', train_dtm)_____no_output_____test_dtm = vectorizer.transform(test.text)
sparse.save_npz(text_features_dir / 'test_dtm', test_dtm)_____no_output_____

### Reload stored data_____no_output_____

train_dtm = sparse.load_npz(text_features_dir / 'train_dtm.npz')
test_dtm = sparse.load_npz(text_features_dir / 'test_dtm.npz')_____no_output_____

## Combine non-text features with the document-term matrix_____no_output_____The dataset contains various numerical features. The vectorizers produce [scipy.sparse matrices](https://docs.scipy.org/doc/scipy/reference/sparse.html). To combine the vectorized text data with other features, we need to first convert these to sparse matrices as well; many sklearn objects and other libraries like lightgbm can handle these very memory-efficient data structures. Converting the sparse matrix to a dense numpy array risks memory overflow.

Most variables are categorical so we use one-hot encoding since we have a fairly large dataset to accommodate the increase in features.

We convert the encoded numerical features and combine them with the document-term matrix:_____no_output_____### One-hot-encoding _____no_output_____

df = pd.concat([train.drop(['text', 'stars'], axis=1).assign(source='train'),
                test.drop(['text', 'stars'], axis=1).assign(source='test')])_____no_output_____uniques = df.nunique()
binned = pd.concat([(df.loc[:, uniques[uniques > 20].index]
                     .apply(pd.qcut, q=10, labels=False, duplicates='drop')),
                    df.loc[:, uniques[uniques <= 20].index]], axis=1)
binned.info(null_counts=True)<class 'pandas.core.frame.DataFrame'>
Int64Index: 2917158 entries, 4906334 to 8021121
Data columns (total 24 columns):
 #   Column              Non-Null Count    Dtype
---  ------              --------------    -----
 0   useful              2917158 non-null  int64
 1   funny               2917158 non-null  int64
 2   cool                2917158 non-null  int64
 3   review_count        2917158 non-null  int64
 4   useful_user         2917158 non-null  int64
 5   funny_user          2917158 non-null  int64
 6   cool_user           2917158 non-null  int64
 7   fans                2917158 non-null  int64
 8   average_stars       2917158 non-null  int64
 9   compliment_hot      2917158 non-null  int64
 10  compliment_more     2917158 non-null  int64
 11  compliment_profile  2917158 non-null  int64
 12  compliment_cute     2917158 non-null  int64
 13  compliment_list     2917158 non-null  int64
 14  compliment_note     2917158 non-null  int64
 15  compliment_plain    2917158 non-null  int64
 16  compliment_cool     2917158 non-null  int64
 17  compliment_funny    2917158 non-null  int64
 18  compliment_writer   2917158 non-null  int64
 19  compliment_photos   2917158 non-null  int64
 20  year                2917158 non-null  int64
 21  month               2917158 non-null  int64
 22  member_yrs          2917158 non-null  int64
 23  source              2917158 non-null  object
dtypes: int64(23), object(1)
memory usage: 556.4+ MB
dummies = pd.get_dummies(binned,
                         columns=binned.columns.drop('source'),
                         drop_first=True)
dummies.info()<class 'pandas.core.frame.DataFrame'>
Int64Index: 2917158 entries, 4906334 to 8021121
Columns: 111 entries, source to member_yrs_15
dtypes: object(1), uint8(110)
memory usage: 350.5+ MB
train_dummies = dummies[dummies.source=='train'].drop('source', axis=1)
train_dummies.info()<class 'pandas.core.frame.DataFrame'>
Int64Index: 1701322 entries, 4906334 to 4178053
Columns: 110 entries, useful_1 to member_yrs_15
dtypes: uint8(110)
memory usage: 191.5 MB

### Train set_____no_output_____

# Cast other feature columns to float and convert to a sparse matrix.
train_numeric = sparse.csr_matrix(train_dummies.astype(np.uint8))
train_numeric.shape_____no_output_____# Combine sparse matrices.
train_dtm_numeric = sparse.hstack((train_dtm, train_numeric))
train_dtm_numeric.shape_____no_output_____sparse.save_npz(text_features_dir / 'train_dtm_numeric',
                train_dtm_numeric)_____no_output_____

### Repeat for test set_____no_output_____

test_dummies = dummies[dummies.source=='test'].drop('source', axis=1)
test_numeric = sparse.csr_matrix(test_dummies.astype(np.int8))
test_dtm_numeric = sparse.hstack((test_dtm, test_numeric))
test_dtm_numeric.shape_____no_output_____sparse.save_npz(text_features_dir / 'test_dtm_numeric', test_dtm_numeric)_____no_output_____

### Reload stored data_____no_output_____

train_dtm_numeric = sparse.load_npz(text_features_dir / 'train_dtm_numeric.npz')
test_dtm_numeric = sparse.load_npz(text_features_dir / 'test_dtm_numeric.npz')_____no_output_____

## Benchmark Accuracy_____no_output_____

accuracy, runtime = {}, {}
predictions = test[['stars']].copy()_____no_output_____

Using the most frequent number of stars (=5) to predict the test set achieve an accuracy close to 51%:_____no_output_____

naive_prediction = np.full_like(predictions.stars,
                                fill_value=train.stars.mode().iloc[0])_____no_output_____naive_benchmark = accuracy_score(predictions.stars, naive_prediction)_____no_output_____naive_benchmark_____no_output_____

## Model Evaluation Helper_____no_output_____

def evaluate_model(model, X_train, X_test, name, store=False):
    start = time()
    model.fit(X_train, train.stars)
    runtime[name] = time() - start
    predictions[name] = model.predict(X_test)
    accuracy[result] = accuracy_score(test.stars, predictions[result])
    if store:
        joblib.dump(model, f'results/{result}.joblib')_____no_output_____

## Multiclass Naive Bayes_____no_output_____

nb = MultinomialNB()_____no_output_____

### Text Features_____no_output_____Next, we train a Naive Bayes classifier using a document-term matrix produced by the CountVectorizer with default settings._____no_output_____

result = 'nb_text'_____no_output_____evaluate_model(nb, train_dtm, test_dtm, result, store=False)_____no_output_____

#### Accuracy_____no_output_____The prediction produces 64.4% accuracy on the test set, a 24.2% improvement over the benchmark:_____no_output_____

accuracy[result]_____no_output_____

#### Confusion Matrix_____no_output_____

stars = index = list(range(1, 6))
pd.DataFrame(confusion_matrix(test.stars,
                              predictions[result]),
             columns=stars,
             index=stars)_____no_output_____

### Text & Numeric Features_____no_output_____

result = 'nb_combined'_____no_output_____evaluate_model(nb, train_dtm_numeric, test_dtm_numeric, result, store=False)_____no_output_____

#### Accuracy_____no_output_____

accuracy[result]_____no_output_____

## Multinomial Logistic Regression_____no_output_____Logistic regression also provides a multinomial training option that is faster and more accurate than the one-vs-all implementation. We use the lbfgs solver (see sklearn [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) for details)._____no_output_____

Cs = np.logspace(-5, 5, 11)_____no_output_____

### Text Features_____no_output_____

log_reg_text_accuracy = {}
log_reg_text_runtime = []
for i, C in enumerate(Cs):
    start = time()
    model = LogisticRegression(C=C,
                               multi_class='multinomial',
                               solver='lbfgs')

    model.fit(train_dtm, train.stars)
    log_reg_text_runtime.append(time() - start)
    log_reg_text_accuracy[C] = accuracy_score(test.stars,
                                              model.predict(test_dtm))

    print(f'{C:12.5f}: {log_reg_text_runtime[i]:.2f}s | {log_reg_text_accuracy[C]:.2%}', flush=True)     0.00001: 34.93s | 62.02%
     0.00010: 74.01s | 70.89%
     0.00100: 126.02s | 73.95%
     0.01000: 122.22s | 74.85%
     0.10000: 125.65s | 74.80%
     1.00000: 130.17s | 74.83%
    10.00000: 126.39s | 74.80%
   100.00000: 125.82s | 74.81%
  1000.00000: 122.73s | 74.83%
 10000.00000: 123.29s | 74.80%
100000.00000: 126.80s | 74.80%
pd.Series(log_reg_text_accuracy).to_csv(yelp_dir / 'logreg_text.csv')_____no_output_____accuracy['lr_text'] = pd.Series(log_reg_text_accuracy).max()
runtime['lr_text'] = np.mean(log_reg_text_runtime)_____no_output_____

### Combined Features_____no_output_____

log_reg_comb_accuracy = {}
log_reg_comb_runtime = []
for i, C in enumerate(Cs):
    start = time()
    model = LogisticRegression(C=C,
                               multi_class='multinomial',
                               solver='lbfgs')

    model.fit(train_dtm_numeric, train.stars)
    log_reg_comb_runtime.append(time() - start)
    log_reg_comb_accuracy[C] = accuracy_score(test.stars,
                                              model.predict(test_dtm_numeric))

    print(f'{C:12.5f}: {log_reg_comb_runtime[i]:.2f}s | {log_reg_comb_accuracy[C]:.2%}', flush=True)     0.00001: 55.26s | 63.98%
     0.00010: 99.05s | 72.94%
     0.00100: 137.62s | 75.12%
     0.01000: 139.11s | 75.55%
     0.10000: 140.26s | 75.28%
     1.00000: 138.90s | 75.32%
    10.00000: 135.13s | 75.35%
   100.00000: 137.76s | 75.32%
  1000.00000: 139.09s | 75.36%
 10000.00000: 135.01s | 75.37%
100000.00000: 134.73s | 75.38%
pd.Series(log_reg_comb_accuracy).to_csv(yelp_dir / 'logreg_combined.csv')_____no_output_____accuracy['lr_comb'] = pd.Series(log_reg_comb_accuracy).max()
runtime['lr_comb'] = np.mean(log_reg_comb_runtime)_____no_output_____

## Gradient Boosting_____no_output_____For illustration, we also train a lightgbm Gradient Boosting tree ensemble with default settings and multiclass objective._____no_output_____

lgb_train = lgb.Dataset(data=train_dtm_numeric.tocsr().astype(np.float32),
                        label=train.stars.sub(1),
                        categorical_feature=list(range(train_dtm_numeric.shape[1])))_____no_output_____lgb_test = lgb.Dataset(data=test_dtm_numeric.tocsr().astype(np.float32),
                       label=test.stars.sub(1),
                       reference=lgb_train)_____no_output_____param = {'objective': 'multiclass',
         'metrics': ['multi_error'],
         'num_class': 5}_____no_output_____booster = lgb.train(params=param,
                    train_set=lgb_train,
                    num_boost_round=2000,
                    early_stopping_rounds=25,
                    valid_sets=[lgb_train, lgb_test],
                    verbose_eval=25)Training until validation scores don't improve for 25 rounds
[25]	training's multi_error: 0.405759	valid_1's multi_error: 0.320156
[50]	training's multi_error: 0.370905	valid_1's multi_error: 0.297562
[75]	training's multi_error: 0.353564	valid_1's multi_error: 0.285789
[100]	training's multi_error: 0.34278	valid_1's multi_error: 0.278771
[125]	training's multi_error: 0.335184	valid_1's multi_error: 0.273733
[150]	training's multi_error: 0.329293	valid_1's multi_error: 0.269682
[175]	training's multi_error: 0.324728	valid_1's multi_error: 0.266682
[200]	training's multi_error: 0.320829	valid_1's multi_error: 0.264156
[225]	training's multi_error: 0.317401	valid_1's multi_error: 0.262016
[250]	training's multi_error: 0.314696	valid_1's multi_error: 0.259945
[275]	training's multi_error: 0.312185	valid_1's multi_error: 0.258219
[300]	training's multi_error: 0.3099	valid_1's multi_error: 0.256977
[325]	training's multi_error: 0.307957	valid_1's multi_error: 0.25591
[350]	training's multi_error: 0.306137	valid_1's multi_error: 0.254868
[375]	training's multi_error: 0.304627	valid_1's multi_error: 0.254025
[400]	training's multi_error: 0.302983	valid_1's multi_error: 0.253086
[425]	training's multi_error: 0.30154	valid_1's multi_error: 0.252281
[450]	training's multi_error: 0.300309	valid_1's multi_error: 0.251648
[475]	training's multi_error: 0.299089	valid_1's multi_error: 0.250989
[500]	training's multi_error: 0.297708	valid_1's multi_error: 0.250408
[525]	training's multi_error: 0.296714	valid_1's multi_error: 0.249912
[550]	training's multi_error: 0.295725	valid_1's multi_error: 0.249531
[575]	training's multi_error: 0.294649	valid_1's multi_error: 0.24918
[600]	training's multi_error: 0.293748	valid_1's multi_error: 0.248822
[625]	training's multi_error: 0.292762	valid_1's multi_error: 0.248478
[650]	training's multi_error: 0.291932	valid_1's multi_error: 0.248306
[675]	training's multi_error: 0.291082	valid_1's multi_error: 0.248
[700]	training's multi_error: 0.290352	valid_1's multi_error: 0.247695
[725]	training's multi_error: 0.289579	valid_1's multi_error: 0.247381
[750]	training's multi_error: 0.288837	valid_1's multi_error: 0.247131
[775]	training's multi_error: 0.288094	valid_1's multi_error: 0.246972
[800]	training's multi_error: 0.287377	valid_1's multi_error: 0.247014
Early stopping, best iteration is:
[776]	training's multi_error: 0.288074	valid_1's multi_error: 0.246952
booster.save_model((yelp_dir / 'lgb_model.txt').as_posix());_____no_output_____y_pred_class = booster.predict(test_dtm_numeric.astype(float))_____no_output_____

The basic settings did not improve over the multinomial logistic regression, but further parameter tuning remains an unused option._____no_output_____

accuracy['lgb_comb'] = accuracy_score(test.stars, y_pred_class.argmax(1) + 1)_____no_output_____

## Comparison_____no_output_____

model_map = {'nb_combined': 'Naive Bayes',
             'lr_comb': 'Logistic Regression',
             'lgb_comb': 'LightGBM'}_____no_output_____accuracy_ = {model_map[k]: v for k, v in accuracy.items() if model_map.get(k)}_____no_output_____log_reg_text = pd.read_csv(yelp_dir / 'logreg_text.csv',
                       index_col=0,
                       squeeze=True)
log_reg_combined = pd.read_csv(yelp_dir / 'logreg_combined.csv',
                       index_col=0,
                       squeeze=True)_____no_output_____fig, axes = plt.subplots(ncols=2, figsize=(14, 4))
pd.Series(accuracy_).sort_values().plot.barh(
    ax=axes[0], xlim=(.45, .75), title='Accuracy by Model')
axes[0].axvline(naive_benchmark, ls='--', lw=1, c='k')

log_reg = (log_reg_text.to_frame('text')
           .join(log_reg_combined.to_frame('combined')))
log_reg.plot(logx=True,
             ax=axes[1],
             title='Logistic Regression - Model Tuning')
axes[1].set_xlabel('Regularization')
axes[1].set_ylabel('Accuracy')
axes[0].set_xlabel('Accuracy')
sns.despine()
fig.tight_layout()_____no_output_____

## Textblob for Sentiment Analysis_____no_output_____

sample_review = train.text.sample(1).iloc[0]
print(sample_review)It's good to see new restaurants in Henderson along Stephanie Street, and judging by the packed parking lot a lot of people seem to like Miller's Ale House. I can't say that I'm a fan. I'll agree with Ashley B's review that it's better than Chili's and far better than suffering through Applebees, but it's really just a ho-hum nothing special chain restaurant with TV's plastered everywhere, and a boring menu of the same tired classics that can be easily prepared by a bunch of distracted teenagers working the kitchen.

If you're hungry and the parking lot is not jammed and you can't decide on what to eat at least you can shut your stomach up if you stop here. But if you want something different or original remember this is a chain....you won't find what you're craving here.

2 Stars...as in I was so underwhelmed by the place I almost fell asleep writing this one. Yawn!
# Polarity ranges from -1 (most negative) to 1 (most positive).
TextBlob(sample_review).sentiment.polarity_____no_output_____# Define a function that accepts text and returns the polarity.
def detect_sentiment(text):
    return TextBlob(text).sentiment.polarity_____no_output_____train['sentiment'] = train.text.apply(detect_sentiment)_____no_output_____sample_reviews = train[['stars', 'text']].sample(100000)_____no_output_____# Create a new DataFrame column for sentiment (Warning: SLOW!).
sample_reviews['sentiment'] = sample_reviews.text.apply(detect_sentiment)_____no_output_____# Box plot of sentiment grouped by stars
sns.boxenplot(x='stars', y='sentiment', data=train);_____no_output_____# Widen the column display.
pd.set_option('max_colwidth', 500)_____no_output_____# Reviews with most negative sentiment
train[train.sentiment == -1].text.head()_____no_output_____# Negative sentiment in a 5-star review
train.loc[(train.stars == 5) & (train.sentiment < -0.3), 'text'].head(1)_____no_output_____# Positive sentiment in a 1-star review
train.loc[(train.stars == 1) & (train.sentiment > 0.5), 'text'].head(1)_____no_output_____# Reset the column display width.
pd.reset_option('max_colwidth')_____no_output_____


# Notebook from Z0m6ie/Zombie_Code
Path: Data_Science_Course/Michigan Data Analysis Course/1 Applied Plotting, Charting & Data Representation/Week 2/Week2.ipynb

# Basic Plotting with matplotlib_____no_output_____You can show matplotlib figures directly in the notebook by using the `%matplotlib notebook` and `%matplotlib inline` magic commands.

`%matplotlib notebook` provides an interactive environment._____no_output_____

%matplotlib notebook_____no_output_____import matplotlib as mpl
mpl.get_backend()_____no_output_____import matplotlib.pyplot as plt
plt.plot?_____no_output_____# because the default is the line style '-',
# nothing will be shown if we only pass in one point (3,2)
plt.plot(3, 2)_____no_output_____# we can pass in '.' to plt.plot to indicate that we want
# the point (3,2) to be indicated with a marker '.'
plt.plot(3, 2, '*')_____no_output_____

Let's see how to make a plot without using the scripting layer._____no_output_____

# First let's set the backend without using mpl.use() from the scripting layer
from matplotlib.backends.backend_agg import FigureCanvasAgg
from matplotlib.figure import Figure

# create a new figure
fig = Figure()

# associate fig with the backend
canvas = FigureCanvasAgg(fig)

# add a subplot to the fig
ax = fig.add_subplot(111)

# plot the point (3,2)
ax.plot(3, 2, '.')

# save the figure to test.png
# you can see this figure in your Jupyter workspace afterwards by going to
# https://hub.coursera-notebooks.org/
canvas.print_png('test.png')_____no_output_____

We can use html cell magic to display the image._____no_output_____

%%html
_____no_output_____# create a new figure
plt.figure()

# plot the point (3,2) using the circle marker
plt.plot(3, 2, 'o')

# get the current axes
ax = plt.gca()

# Set axis properties [xmin, xmax, ymin, ymax]
ax.axis([0,6,0,10])_____no_output_____# create a new figure
plt.figure()

# plot the point (1.5, 1.5) using the circle marker
plt.plot(1.5, 1.5, 'o')
# plot the point (2, 2) using the circle marker
plt.plot(2, 2, '*')
# plot the point (2.5, 2.5) using the circle marker
plt.plot(2.5, 2.5, 'o')_____no_output_____# get current axes
ax = plt.gca()
# get all the child objects the axes contains
ax.get_children()_____no_output_____

# Scatterplots_____no_output_____

import numpy as np

x = np.array([1,2,3,4,5,6,7,8])
y = x

plt.figure()
plt.scatter(x, y) # similar to plt.plot(x, y, '.'), but the underlying child objects in the axes are not Line2D_____no_output_____import numpy as np

x = np.array([1,2,3,4,5,6,7,8])
y = x

# create a list of colors for each point to have
# ['green', 'green', 'green', 'green', 'green', 'green', 'green', 'red']
colors = ['green']*(len(x)-1)
colors.append('red')

plt.figure()

# plot the point with size 100 and chosen colors
plt.scatter(x, y, s=100, c=colors)_____no_output_____# convert the two lists into a list of pairwise tuples
zip_generator = zip([1,2,3,4,5], [6,7,8,9,10])

print(list(zip_generator))
# the above prints:
# [(1, 6), (2, 7), (3, 8), (4, 9), (5, 10)]

zip_generator = zip([1,2,3,4,5], [6,7,8,9,10])
# The single star * unpacks a collection into positional arguments
print(*zip_generator)
# the above prints:
# (1, 6) (2, 7) (3, 8) (4, 9) (5, 10)[(1, 6), (2, 7), (3, 8), (4, 9), (5, 10)]
(1, 6) (2, 7) (3, 8) (4, 9) (5, 10)
# use zip to convert 5 tuples with 2 elements each to 2 tuples with 5 elements each
print(list(zip((1, 6), (2, 7), (3, 8), (4, 9), (5, 10))))
# the above prints:
# [(1, 2, 3, 4, 5), (6, 7, 8, 9, 10)]

zip_generator = zip([1,2,3,4,5], [6,7,8,9,10])
# let's turn the data back into 2 lists
x, y = zip(*zip_generator) # This is like calling zip((1, 6), (2, 7), (3, 8), (4, 9), (5, 10))
print(x)
print(y)
# the above prints:
# (1, 2, 3, 4, 5)
# (6, 7, 8, 9, 10)[(1, 2, 3, 4, 5), (6, 7, 8, 9, 10)]
(1, 2, 3, 4, 5)
(6, 7, 8, 9, 10)
plt.figure()
# plot a data series 'Tall students' in red using the first two elements of x and y
plt.scatter(x[:2], y[:2], s=100, c='red', label='Tall students')
# plot a second data series 'Short students' in blue using the last three elements of x and y
plt.scatter(x[2:], y[2:], s=100, c='blue', label='Short students')_____no_output_____# add a label to the x axis
plt.xlabel('The number of times the child kicked a ball')
# add a label to the y axis
plt.ylabel('The grade of the student')
# add a title
plt.title('Relationship between ball kicking and grades')_____no_output_____# add a legend (uses the labels from plt.scatter)
plt.legend()_____no_output_____# add the legend to loc=4 (the lower right hand corner), also gets rid of the frame and adds a title
plt.legend(loc=4, frameon=False, title='Legend')_____no_output_____# get children from current axes (the legend is the second to last item in this list)
plt.gca().get_children()_____no_output_____# get the legend from the current axes
legend = plt.gca().get_children()[-2]_____no_output_____# you can use get_children to navigate through the child artists
legend.get_children()[0].get_children()[1].get_children()[0].get_children()_____no_output_____# import the artist class from matplotlib
from matplotlib.artist import Artist

def rec_gc(art, depth=0):
    if isinstance(art, Artist):
        # increase the depth for pretty printing
        print("  " * depth + str(art))
        for child in art.get_children():
            rec_gc(child, depth+2)

# Call this function on the legend artist to see what the legend is made up of
rec_gc(plt.legend())Legend
    <matplotlib.offsetbox.VPacker object at 0x7fee7305bd30>
        <matplotlib.offsetbox.TextArea object at 0x7fee7305bac8>
            Text(0,0,'None')
        <matplotlib.offsetbox.HPacker object at 0x7fee708ed710>
            <matplotlib.offsetbox.VPacker object at 0x7fee708ed358>
                <matplotlib.offsetbox.HPacker object at 0x7fee73046eb8>
                    <matplotlib.offsetbox.DrawingArea object at 0x7fee708e8940>
                        <matplotlib.collections.PathCollection object at 0x7fee708e8208>
                    <matplotlib.offsetbox.TextArea object at 0x7fee708ed390>
                        Text(0,0,'Tall students')
                <matplotlib.offsetbox.HPacker object at 0x7fee7304f048>
                    <matplotlib.offsetbox.DrawingArea object at 0x7fee7305d7f0>
                        <matplotlib.collections.PathCollection object at 0x7fee7305d0f0>
                    <matplotlib.offsetbox.TextArea object at 0x7fee708e8320>
                        Text(0,0,'Short students')
    FancyBboxPatch(0,0;1x1)

# Line Plots_____no_output_____

import numpy as np

linear_data = np.array([1,2,3,4,5,6,7,8])
exponential_data = linear_data**2

plt.figure()
# plot the linear data and the exponential data
plt.plot(linear_data, '-o', exponential_data, '-o')_____no_output_____# plot another series with a dashed red line
plt.plot([22,44,55], '--r')_____no_output_____plt.xlabel('Some data')
plt.ylabel('Some other data')
plt.title('A title')
# add a legend with legend entries (because we didn't have labels when we plotted the data series)
plt.legend(['Baseline', 'Competition', 'Us'])_____no_output_____# fill the area between the linear data and exponential data
plt.gca().fill_between(range(len(linear_data)),
                       linear_data, exponential_data,
                       facecolor='blue',
                       alpha=0.25)_____no_output_____

Let's try working with dates!_____no_output_____

plt.figure()

observation_dates = np.arange('2017-01-01', '2017-01-09', dtype='datetime64[D]')

plt.plot(observation_dates, linear_data, '-o',  observation_dates, exponential_data, '-o')_____no_output_____

Let's try using pandas_____no_output_____

import pandas as pd

plt.figure()
observation_dates = np.arange('2017-01-01', '2017-01-09', dtype='datetime64[D]')
observation_dates = map(pd.to_datetime, observation_dates) # trying to plot a map will result in an error
plt.plot(observation_dates, linear_data, '-o',  observation_dates, exponential_data, '-o')_____no_output_____plt.figure()
observation_dates = np.arange('2017-01-01', '2017-01-09', dtype='datetime64[D]')
observation_dates = list(map(pd.to_datetime, observation_dates)) # convert the map to a list to get rid of the error
plt.plot(observation_dates, linear_data, '-o',  observation_dates, exponential_data, '-o')_____no_output_____x = plt.gca().xaxis

# rotate the tick labels for the x axis
for item in x.get_ticklabels():
    item.set_rotation(45)_____no_output_____# adjust the subplot so the text doesn't run off the image
plt.subplots_adjust(bottom=0.25)_____no_output_____ax = plt.gca()
ax.set_xlabel('Date')
ax.set_ylabel('Units')
ax.set_title('Exponential vs. Linear performance')_____no_output_____# you can add mathematical expressions in any text element
ax.set_title("Exponential ($x^2$) vs. Linear ($x$) performance")_____no_output_____

# Bar Charts_____no_output_____

plt.figure()
xvals = range(len(linear_data))
plt.bar(xvals, linear_data, width = 0.3)_____no_output_____new_xvals = []

# plot another set of bars, adjusting the new xvals to make up for the first set of bars plotted
for item in xvals:
    new_xvals.append(item+0.3)

plt.bar(new_xvals, exponential_data, width = 0.3 ,color='red')_____no_output_____from random import randint
linear_err = [randint(0,15) for x in range(len(linear_data))]

# This will plot a new set of bars with errorbars using the list of random error values
plt.bar(xvals, linear_data, width = 0.3, yerr=linear_err)_____no_output_____# stacked bar charts are also possible
plt.figure()
xvals = range(len(linear_data))
plt.bar(xvals, linear_data, width = 0.3, color='b')
plt.bar(xvals, exponential_data, width = 0.3, bottom=linear_data, color='r')_____no_output_____# or use barh for horizontal bar charts
plt.figure()
xvals = range(len(linear_data))
plt.barh(xvals, linear_data, height = 0.3, color='b')
plt.barh(xvals, exponential_data, height = 0.3, left=linear_data, color='r')_____no_output_____