# Notebook from krentzd/ZeroCostDL4Mic
Path: Colab_notebooks/StarDist_2D_ZeroCostDL4Mic.ipynb

# **StarDist (2D)**
---

**StarDist 2D** is a deep-learning method that can be used to segment cell nuclei from bioimages and was first published by [Schmidt *et al.* in 2018, on arXiv](https://arxiv.org/abs/1806.03535). It uses a shape representation based on star-convex polygons for nuclei in an image to predict the presence and the shape of these nuclei. This StarDist 2D network is based on an adapted U-Net network architecture.

 **This particular notebook enables nuclei segmentation of 2D dataset. If you are interested in 3D dataset, you should use the StarDist 3D notebook instead.**

---
*Disclaimer*:

This notebook is part of the Zero-Cost Deep-Learning to Enhance Microscopy project (https://github.com/HenriquesLab/DeepLearning_Collab/wiki). Jointly developed by the Jacquemet (link to https://cellmig.org/) and Henriques (https://henriqueslab.github.io/) laboratories.

This notebook is largely based on the paper:

**Cell Detection with Star-convex Polygons** from Schmidt *et al.*, International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Granada, Spain, September 2018. (https://arxiv.org/abs/1806.03535)

and the 3D extension of the approach:

**Star-convex Polyhedra for 3D Object Detection and Segmentation in Microscopy** from Weigert *et al.* published on arXiv in 2019 (https://arxiv.org/abs/1908.03636)

**The Original code** is freely available in GitHub:
https://github.com/mpicbg-csbd/stardist

**Please also cite this original paper when using or developing this notebook.**
_____no_output_____# **How to use this notebook?**

---

Video describing how to use our notebooks are available on youtube:
  - [**Video 1**](https://www.youtube.com/watch?v=GzD2gamVNHI&feature=youtu.be): Full run through of the workflow to obtain the notebooks and the provided test datasets as well as a common use of the notebook
  - [**Video 2**](https://www.youtube.com/watch?v=PUuQfP5SsqM&feature=youtu.be): Detailed description of the different sections of the notebook

---
###**Structure of a notebook**

The notebook contains two types of cell:

**Text cells** provide information and can be modified by douple-clicking the cell. You are currently reading the text cell. You can create a new text by clicking `+ Text`.

**Code cells** contain code and the code can be modfied by selecting the cell. To execute the cell, move your cursor on the `[ ]`-mark on the left side of the cell (play button appears). Click to execute the cell. After execution is done the animation of play button stops. You can create a new coding cell by clicking `+ Code`.

---
###**Table of contents, Code snippets** and **Files**

On the top left side of the notebook you find three tabs which contain from top to bottom:

*Table of contents* = contains structure of the notebook. Click the content to move quickly between sections.

*Code snippets* = contain examples how to code certain tasks. You can ignore this when using this notebook.

*Files* = contain all available files. After mounting your google drive (see section 1.) you will find your files and folders here.

**Remember that all uploaded files are purged after changing the runtime.** All files saved in Google Drive will remain. You do not need to use the Mount Drive-button; your Google Drive is connected in section 1.2.

**Note:** The "sample data" in "Files" contains default files. Do not upload anything in here!

---
###**Making changes to the notebook**

**You can make a copy** of the notebook and save it to your Google Drive. To do this click file -> save a copy in drive.

To **edit a cell**, double click on the text. This will show you either the source code (in code cells) or the source text (in text cells).
You can use the `#`-mark in code cells to comment out parts of the code. This allows you to keep the original code piece in the cell as a comment._____no_output_____#**0. Before getting started**
---
 For StarDist to train, **it needs to have access to a paired training dataset made of images of nuclei and their corresponding masks**. Information on how to generate a training dataset is available in our Wiki page: https://github.com/HenriquesLab/ZeroCostDL4Mic/wiki

**We strongly recommend that you generate extra paired images. These images can be used to assess the quality of your trained model**. The quality control assessment can be done directly in this notebook.

The data structure is important. It is necessary that all the input data are in the same folder and that all the output data is in a separate folder. The provided training dataset is already split in two folders called "Training - Images" (Training_source) and "Training - Masks" (Training_target).

Additionally, the corresponding Training_source and Training_target files need to have **the same name**.

Please note that you currently can **only use .tif files!**

You can also provide a folder that contains the data that you wish to analyse with the trained network once all training has been performed. This can include Test dataset for which you have the equivalent output and can compare to what the network provides.

Here's a common data structure that can work:
*   Experiment A
    - **Training dataset**
      - Images of nuclei (Training_source)
        - img_1.tif, img_2.tif, ...
      - Masks (Training_target)
        - img_1.tif, img_2.tif, ...
    - **Quality control dataset**
     - Images of nuclei
        - img_1.tif, img_2.tif
      - Masks
        - img_1.tif, img_2.tif
    - **Data to be predicted**
    - **Results**

---
**Important note**

- If you wish to **Train a network from scratch** using your own dataset (and we encourage everyone to do that), you will need to run **sections 1 - 4**, then use **section 5** to assess the quality of your model and **section 6** to run predictions using the model that you trained.

- If you wish to **Evaluate your model** using a model previously generated and saved on your Google Drive, you will only need to run **sections 1 and 2** to set up the notebook, then use **section 5** to assess the quality of your model.

- If you only wish to **run predictions** using a model previously generated and saved on your Google Drive, you will only need to run **sections 1 and 2** to set up the notebook, then use **section 6** to run the predictions on the desired model.
---_____no_output_____# **1. Initialise the Colab session**

---

_____no_output_____

## **1.1. Check for GPU access**
---

By default, the session should be using Python 3 and GPU acceleration, but it is possible to ensure that these are set properly by doing the following:

Go to **Runtime -> Change the Runtime type**

**Runtime type: Python 3** *(Python 3 is programming language in which this program is written)*

**Accelator: GPU** *(Graphics processing unit)*
_____no_output_____

#@markdown ##Run this cell to check if you have GPU access
%tensorflow_version 1.x
import tensorflow as tf
if tf.test.gpu_device_name()=='':
  print('You do not have GPU access.')
  print('Did you change your runtime ?')
  print('If the runtime setting is correct then Google did not allocate a GPU for your session')
  print('Expect slow performance. To access GPU try reconnecting later')

else:
  print('You have GPU access')
  !nvidia-smi_____no_output_____

## **1.2. Mount your Google Drive**
---
 To use this notebook on the data present in your Google Drive, you need to mount your Google Drive to this notebook.

 Play the cell below to mount your Google Drive and follow the link. In the new browser window, select your drive and select 'Allow', copy the code, paste into the cell and press enter. This will give Colab access to the data on the drive.

 Once this is done, your data are available in the **Files** tab on the top left of notebook._____no_output_____

#@markdown ##Play the cell to connect your Google Drive to Colab

# mount user's Google Drive to Google Colab.
from google.colab import drive
drive.mount('/content/gdrive')_____no_output_____

# **2. Install StarDist and dependencies**
---
_____no_output_____

#@markdown ##Install StarDist and dependencies
%tensorflow_version 1.x

import tensorflow
print(tensorflow.__version__)
print("Tensorflow enabled.")

# Install packages which are not included in Google Colab

!pip install tifffile # contains tools to operate tiff-files
!pip install csbdeep  # contains tools for restoration of fluorescence microcopy images (Content-aware Image Restoration, CARE). It uses Keras and Tensorflow.
!pip install stardist # contains tools to operate STARDIST.
!pip install gputools # improves STARDIST performances
!pip install edt # improves STARDIST performances
!pip install wget

# ------- Variable specific to Stardist -------
from stardist import fill_label_holes, random_label_cmap, calculate_extents, gputools_available, relabel_image_stardist, random_label_cmap,  relabel_image_stardist, _draw_polygons, export_imagej_rois
from stardist.models import Config2D, StarDist2D, StarDistData2D # import objects
from stardist.matching import matching_dataset
from __future__ import print_function, unicode_literals, absolute_import, division
from csbdeep.utils import Path, normalize, download_and_extract_zip_file, plot_history # for loss plot
from csbdeep.io import save_tiff_imagej_compatible
import numpy as np
np.random.seed(42)
lbl_cmap = random_label_cmap()
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

# ------- Common variable to all ZeroCostDL4Mic notebooks -------
import numpy as np
from matplotlib import pyplot as plt
import urllib
import os, random
import shutil
import zipfile
from tifffile import imread, imsave
import time
import sys
import wget
from pathlib import Path
import pandas as pd
import csv
from glob import glob
from scipy import signal
from scipy import ndimage
from skimage import io
from sklearn.linear_model import LinearRegression
from skimage.util import img_as_uint
import matplotlib as mpl
from skimage.metrics import structural_similarity
from skimage.metrics import peak_signal_noise_ratio as psnr
from astropy.visualization import simple_norm
from skimage import img_as_float32, img_as_ubyte, img_as_float
from skimage.util import img_as_ubyte
from tqdm import tqdm
import cv2

# Colors for the warning messages
class bcolors:
  WARNING = '\033[31m'
W  = '\033[0m'  # white (normal)
R  = '\033[31m' # red

#Disable some of the tensorflow warnings
import warnings
warnings.filterwarnings("ignore")

print("Libraries installed")

_____no_output_____

# **3. Select your parameters and paths**

---
_____no_output_____## **3.1. Setting main training parameters**
---
 _____no_output_____ **Paths for training, predictions and results**

**`Training_source:`, `Training_target`:** These are the paths to your folders containing the Training_source (images of nuclei) and Training_target (masks) training data respecively. To find the paths of the folders containing the respective datasets, go to your Files on the left of the notebook, navigate to the folder containing your files and copy the path by right-clicking on the folder, **Copy path** and pasting it into the right box below.

**`model_name`:** Use only my_model -style, not my-model (Use "_" not "-"). Do not use spaces in the name. Avoid using the name of an existing model (saved in the same folder) as it will be overwritten.

**`model_path`**: Enter the path where your model will be saved once trained (for instance your result folder).

**Training parameters**

**`number_of_epochs`:** Input how many epochs (rounds) the network will be trained. Preliminary results can already be observed after a 50-100 epochs, but a full training should run for up to 400 epochs. Evaluate the performance after training (see 5.). **Default value: 100**

**Advanced Parameters - experienced users only**

**`batch_size:`** This parameter defines the number of patches seen in each training step. Reducing or increasing the **batch size** may slow or speed up your training, respectively, and can influence network performance. **Default value: 2**

**`number_of_steps`:** Define the number of training steps by epoch. By default this parameter is calculated so that each image / patch is seen at least once per epoch. **Default value: Number of patch / batch_size**

**`patch_size`:** Input the size of the patches use to train StarDist 2D (length of a side). The value should be smaller or equal to the dimensions of the image. Make the patch size as large as possible and divisible by 8. **Default value: dimension of the training images**

**`percentage_validation`:**  Input the percentage of your training dataset you want to use to validate the network during the training. **Default value: 10**

**`n_rays`:** Set number of rays (corners) used for StarDist (for instance, a square has 4 corners). **Default value: 32**

**`grid_parameter`:** increase this number if the cells/nuclei are very large or decrease it if they are very small. **Default value: 2**

**`initial_learning_rate`:** Input the initial value to be used as learning rate. **Default value: 0.0003**

**If you get an Out of memory (OOM) error during the training,  manually decrease the patch_size value until the OOM error disappear.**

_____no_output_____

#@markdown ###Path to training images:
Training_source = "" #@param {type:"string"}

Training_target = "" #@param {type:"string"}

#@markdown ###Name of the model and path to model folder:
model_name = "" #@param {type:"string"}

model_path = "" #@param {type:"string"}
#trained_model = model_path

#@markdown ### Other parameters for training:
number_of_epochs =  100#@param {type:"number"}

#@markdown ###Advanced Parameters
Use_Default_Advanced_Parameters = True #@param {type:"boolean"}

#@markdown ###If not, please input:

#GPU_limit = 90 #@param {type:"number"}
batch_size = 2 #@param {type:"number"}
number_of_steps = 20#@param {type:"number"}
patch_size = 1024 #@param {type:"number"}
percentage_validation =  10 #@param {type:"number"}
n_rays = 32 #@param {type:"number"}
grid_parameter = 2#@param [1, 2, 4, 8, 16, 32] {type:"raw"}
initial_learning_rate = 0.0003 #@param {type:"number"}

if (Use_Default_Advanced_Parameters):
  print("Default advanced parameters enabled")
  batch_size = 2
  n_rays = 32
  percentage_validation = 10
  grid_parameter = 2
  initial_learning_rate = 0.0003

percentage = percentage_validation/100

#here we check that no model with the same name already exist, if so delete
if os.path.exists(model_path+'/'+model_name):
  print(bcolors.WARNING +"!! WARNING: Folder already exists and has been removed !!" + W)
  shutil.rmtree(model_path+'/'+model_name)

# Here we open will randomly chosen input and output image
random_choice = random.choice(os.listdir(Training_source))
x = imread(Training_source+"/"+random_choice)

# Here we check the image dimensions

Image_Y = x.shape[0]
Image_X = x.shape[1]

print('Loaded images (width, length) =', x.shape)

# If default parameters, patch size is the same as image size
if (Use_Default_Advanced_Parameters):
  patch_size = min(Image_Y, Image_X)

#Hyperparameters failsafes

# Here we check that patch_size is smaller than the smallest xy dimension of the image

if patch_size > min(Image_Y, Image_X):
  patch_size = min(Image_Y, Image_X)
  print (bcolors.WARNING + " Your chosen patch_size is bigger than the xy dimension of your image; therefore the patch_size chosen is now:",patch_size)

# Here we check that the patch_size is divisible by 8
if not patch_size % 8 == 0:
    patch_size = ((int(patch_size / 8)-1) * 8)
    print (bcolors.WARNING + " Your chosen patch_size is not divisible by 8; therefore the patch_size chosen is:",patch_size)

# Here we disable pre-trained model by default (in case the next cell is not ran)
Use_pretrained_model = False

# Here we disable data augmentation by default (in case the cell is not ran)

Use_Data_augmentation = False

print("Parameters initiated.")

os.chdir(Training_target)
y = imread(Training_target+"/"+random_choice)

#Here we use a simple normalisation strategy to visualise the image
norm = simple_norm(x, percent = 99)

f=plt.figure(figsize=(16,8))
plt.subplot(1,2,1)
plt.imshow(x, interpolation='nearest', norm=norm, cmap='magma')
plt.title('Training source')
plt.axis('off');

plt.subplot(1,2,2)
plt.imshow(y, interpolation='nearest', cmap=lbl_cmap)
plt.title('Training target')
plt.axis('off');

_____no_output_____

## **3.2. Data augmentation**
---

_____no_output_____Data augmentation can improve training progress by amplifying differences in the dataset. This can be useful if the available dataset is small since, in this case, it is possible that a network could quickly learn every example in the dataset (overfitting), without augmentation. Augmentation is not necessary for training and if your training dataset is large you should disable it.

 **However, data augmentation is not a magic solution and may also introduce issues. Therefore, we recommend that you train your network with and without augmentation, and use the QC section to validate that it improves overall performances.**

Data augmentation is performed here by [Augmentor.](https://github.com/mdbloice/Augmentor)

[Augmentor](https://github.com/mdbloice/Augmentor) was described in the following article:

Marcus D Bloice, Peter M Roth, Andreas Holzinger, Biomedical image augmentation using Augmentor, Bioinformatics, https://doi.org/10.1093/bioinformatics/btz259

**Please also cite this original paper when publishing results obtained using this notebook with augmentation enabled.** _____no_output_____

#Data augmentation

Use_Data_augmentation = False #@param {type:"boolean"}

if Use_Data_augmentation:
  !pip install Augmentor
  import Augmentor

#@markdown ####Choose a factor by which you want to multiply your original dataset

Multiply_dataset_by = 2 #@param {type:"slider", min:1, max:30, step:1}

Save_augmented_images = False #@param {type:"boolean"}

Saving_path = "" #@param {type:"string"}

Use_Default_Augmentation_Parameters = True #@param {type:"boolean"}
#@markdown ###If not, please choose the probability of the following image manipulations to be used to augment your dataset (1 = always used; 0 = disabled ):

#@markdown ####Mirror and rotate images
rotate_90_degrees = 0 #@param {type:"slider", min:0, max:1, step:0.1}

rotate_270_degrees = 0 #@param {type:"slider", min:0, max:1, step:0.1}

flip_left_right = 0 #@param {type:"slider", min:0, max:1, step:0.1}

flip_top_bottom = 0 #@param {type:"slider", min:0, max:1, step:0.1}

#@markdown ####Random image Zoom

random_zoom = 0 #@param {type:"slider", min:0, max:1, step:0.1}

random_zoom_magnification = 0 #@param {type:"slider", min:0, max:1, step:0.1}

#@markdown ####Random image distortion

random_distortion = 0 #@param {type:"slider", min:0, max:1, step:0.1}

#@markdown ####Image shearing and skewing

image_shear = 0 #@param {type:"slider", min:0, max:1, step:0.1}
max_image_shear = 1 #@param {type:"slider", min:1, max:25, step:1}

skew_image = 0 #@param {type:"slider", min:0, max:1, step:0.1}

skew_image_magnitude = 0 #@param {type:"slider", min:0, max:1, step:0.1}

if Use_Default_Augmentation_Parameters:
  rotate_90_degrees = 0.5
  rotate_270_degrees = 0.5
  flip_left_right = 0.5
  flip_top_bottom = 0.5

  if not Multiply_dataset_by >5:
    random_zoom = 0
    random_zoom_magnification = 0.9
    random_distortion = 0
    image_shear = 0
    max_image_shear = 10
    skew_image = 0
    skew_image_magnitude = 0

  if Multiply_dataset_by >5:
    random_zoom = 0.1
    random_zoom_magnification = 0.9
    random_distortion = 0.5
    image_shear = 0.2
    max_image_shear = 5
    skew_image = 0.2
    skew_image_magnitude = 0.4

  if Multiply_dataset_by >25:
    random_zoom = 0.5
    random_zoom_magnification = 0.8
    random_distortion = 0.5
    image_shear = 0.5
    max_image_shear = 20
    skew_image = 0.5
    skew_image_magnitude = 0.6

list_files = os.listdir(Training_source)
Nb_files = len(list_files)

Nb_augmented_files = (Nb_files * Multiply_dataset_by)

if Use_Data_augmentation:
  print("Data augmentation enabled")
# Here we set the path for the various folder were the augmented images will be loaded

# All images are first saved into the augmented folder
  #Augmented_folder = "/content/Augmented_Folder"

  if not Save_augmented_images:
    Saving_path= "/content"

  Augmented_folder =  Saving_path+"/Augmented_Folder"
  if os.path.exists(Augmented_folder):
    shutil.rmtree(Augmented_folder)
  os.makedirs(Augmented_folder)

  #Training_source_augmented = "/content/Training_source_augmented"
  Training_source_augmented = Saving_path+"/Training_source_augmented"

  if os.path.exists(Training_source_augmented):
    shutil.rmtree(Training_source_augmented)
  os.makedirs(Training_source_augmented)

  #Training_target_augmented = "/content/Training_target_augmented"
  Training_target_augmented = Saving_path+"/Training_target_augmented"

  if os.path.exists(Training_target_augmented):
    shutil.rmtree(Training_target_augmented)
  os.makedirs(Training_target_augmented)

# Here we generate the augmented images
#Load the images
  p = Augmentor.Pipeline(Training_source, Augmented_folder)

#Define the matching images
  p.ground_truth(Training_target)
#Define the augmentation possibilities
  if not rotate_90_degrees == 0:
    p.rotate90(probability=rotate_90_degrees)

  if not rotate_270_degrees == 0:
    p.rotate270(probability=rotate_270_degrees)

  if not flip_left_right == 0:
    p.flip_left_right(probability=flip_left_right)

  if not flip_top_bottom == 0:
    p.flip_top_bottom(probability=flip_top_bottom)

  if not random_zoom == 0:
    p.zoom_random(probability=random_zoom, percentage_area=random_zoom_magnification)

  if not random_distortion == 0:
    p.random_distortion(probability=random_distortion, grid_width=4, grid_height=4, magnitude=8)

  if not image_shear == 0:
    p.shear(probability=image_shear,max_shear_left=20,max_shear_right=20)

  if not skew_image == 0:
    p.skew(probability=skew_image,magnitude=skew_image_magnitude)

  p.sample(int(Nb_augmented_files))

  print(int(Nb_augmented_files),"matching images generated")

# Here we sort through the images and move them back to augmented trainning source and targets folders

  augmented_files = os.listdir(Augmented_folder)

  for f in augmented_files:

    if (f.startswith("_groundtruth_(1)_")):
      shortname_noprefix = f[17:]
      shutil.copyfile(Augmented_folder+"/"+f, Training_target_augmented+"/"+shortname_noprefix)
    if not (f.startswith("_groundtruth_(1)_")):
      shutil.copyfile(Augmented_folder+"/"+f, Training_source_augmented+"/"+f)

  for filename in os.listdir(Training_source_augmented):
    os.chdir(Training_source_augmented)
    os.rename(filename, filename.replace('_original', ''))

  #Here we clean up the extra files
  shutil.rmtree(Augmented_folder)

if not Use_Data_augmentation:
  print(bcolors.WARNING+"Data augmentation disabled")

_____no_output_____

## **3.3. Using weights from a pre-trained model as initial weights**
---
  Here, you can set the the path to a pre-trained model from which the weights can be extracted and used as a starting point for this training session. **This pre-trained model needs to be a StarDist model**.

 This option allows you to perform training over multiple Colab runtimes or to do transfer learning using models trained outside of ZeroCostDL4Mic. **You do not need to run this section if you want to train a network from scratch**.

 In order to continue training from the point where the pre-trained model left off, it is adviseable to also **load the learning rate** that was used when the training ended. This is automatically saved for models trained with ZeroCostDL4Mic and will be loaded here. If no learning rate can be found in the model folder provided, the default learning rate will be used. _____no_output_____

# @markdown ##Loading weights from a pre-trained network

Use_pretrained_model = False #@param {type:"boolean"}

pretrained_model_choice = "2D_versatile_fluo_from_Stardist_Fiji" #@param ["Model_from_file", "2D_versatile_fluo_from_Stardist_Fiji", "2D_Demo_Model_from_Stardist_Github", "Versatile_H&E_nuclei"]

Weights_choice = "best" #@param ["last", "best"]

#@markdown ###If you chose "Model_from_file", please provide the path to the model folder:
pretrained_model_path = "" #@param {type:"string"}

# --------------------- Check if we load a previously trained model ------------------------
if Use_pretrained_model:

# --------------------- Load the model from the choosen path ------------------------
  if pretrained_model_choice == "Model_from_file":
    h5_file_path = os.path.join(pretrained_model_path, "weights_"+Weights_choice+".h5")

# --------------------- Download the Demo 2D model provided in the Stardist 2D github ------------------------

  if pretrained_model_choice == "2D_Demo_Model_from_Stardist_Github":
    pretrained_model_name = "2D_Demo"
    pretrained_model_path = "/content/"+pretrained_model_name
    print("Downloading the 2D_Demo_Model_from_Stardist_Github")
    if os.path.exists(pretrained_model_path):
      shutil.rmtree(pretrained_model_path)
    os.makedirs(pretrained_model_path)
    wget.download("https://github.com/mpicbg-csbd/stardist/raw/master/models/examples/2D_demo/config.json", pretrained_model_path)
    wget.download("https://github.com/mpicbg-csbd/stardist/raw/master/models/examples/2D_demo/thresholds.json", pretrained_model_path)
    wget.download("https://github.com/mpicbg-csbd/stardist/blob/master/models/examples/2D_demo/weights_best.h5?raw=true", pretrained_model_path)
    wget.download("https://github.com/mpicbg-csbd/stardist/blob/master/models/examples/2D_demo/weights_last.h5?raw=true", pretrained_model_path)
    h5_file_path = os.path.join(pretrained_model_path, "weights_"+Weights_choice+".h5")

# --------------------- Download the Demo 2D_versatile_fluo_from_Stardist_Fiji ------------------------

  if pretrained_model_choice == "2D_versatile_fluo_from_Stardist_Fiji":
    print("Downloading the 2D_versatile_fluo_from_Stardist_Fiji")
    pretrained_model_name = "2D_versatile_fluo"
    pretrained_model_path = "/content/"+pretrained_model_name

    if os.path.exists(pretrained_model_path):
      shutil.rmtree(pretrained_model_path)
    os.makedirs(pretrained_model_path)

    wget.download("https://cloud.mpi-cbg.de/index.php/s/1k5Zcy7PpFWRb0Q/download?path=/versatile&files=2D_versatile_fluo.zip", pretrained_model_path)

    with zipfile.ZipFile(pretrained_model_path+"/2D_versatile_fluo.zip", 'r') as zip_ref:
      zip_ref.extractall(pretrained_model_path)

    h5_file_path = os.path.join(pretrained_model_path, "weights_best.h5")

# --------------------- Download the Versatile (H&E nuclei)_fluo_from_Stardist_Fiji ------------------------

  if pretrained_model_choice == "Versatile_H&E_nuclei":
    print("Downloading the Versatile_H&E_nuclei from_Stardist_Fiji")
    pretrained_model_name = "2D_versatile_he"
    pretrained_model_path = "/content/"+pretrained_model_name

    if os.path.exists(pretrained_model_path):
      shutil.rmtree(pretrained_model_path)
    os.makedirs(pretrained_model_path)

    wget.download("https://cloud.mpi-cbg.de/index.php/s/1k5Zcy7PpFWRb0Q/download?path=/versatile&files=2D_versatile_he.zip", pretrained_model_path)

    with zipfile.ZipFile(pretrained_model_path+"/2D_versatile_he.zip", 'r') as zip_ref:
      zip_ref.extractall(pretrained_model_path)

    h5_file_path = os.path.join(pretrained_model_path, "weights_best.h5")

# --------------------- Add additional pre-trained models here ------------------------

# --------------------- Check the model exist ------------------------
# If the model path chosen does not contain a pretrain model then use_pretrained_model is disabled,
  if not os.path.exists(h5_file_path):
    print(bcolors.WARNING+'WARNING: weights_last.h5 pretrained model does not exist' + W)
    Use_pretrained_model = False

# If the model path contains a pretrain model, we load the training rate,
  if os.path.exists(h5_file_path):
#Here we check if the learning rate can be loaded from the quality control folder
    if os.path.exists(os.path.join(pretrained_model_path, 'Quality Control', 'training_evaluation.csv')):

      with open(os.path.join(pretrained_model_path, 'Quality Control', 'training_evaluation.csv'),'r') as csvfile:
        csvRead = pd.read_csv(csvfile, sep=',')
        #print(csvRead)

        if "learning rate" in csvRead.columns: #Here we check that the learning rate column exist (compatibility with model trained un ZeroCostDL4Mic bellow 1.4)
          print("pretrained network learning rate found")
          #find the last learning rate
          lastLearningRate = csvRead["learning rate"].iloc[-1]
          #Find the learning rate corresponding to the lowest validation loss
          min_val_loss = csvRead[csvRead['val_loss'] == min(csvRead['val_loss'])]
          #print(min_val_loss)
          bestLearningRate = min_val_loss['learning rate'].iloc[-1]

          if Weights_choice == "last":
            print('Last learning rate: '+str(lastLearningRate))

          if Weights_choice == "best":
            print('Learning rate of best validation loss: '+str(bestLearningRate))

        if not "learning rate" in csvRead.columns: #if the column does not exist, then initial learning rate is used instead
          bestLearningRate = initial_learning_rate
          lastLearningRate = initial_learning_rate
          print(bcolors.WARNING+'WARNING: The learning rate cannot be identified from the pretrained network. Default learning rate of '+str(bestLearningRate)+' will be used instead' + W)

#Compatibility with models trained outside ZeroCostDL4Mic but default learning rate will be used
    if not os.path.exists(os.path.join(pretrained_model_path, 'Quality Control', 'training_evaluation.csv')):
      print(bcolors.WARNING+'WARNING: The learning rate cannot be identified from the pretrained network. Default learning rate of '+str(initial_learning_rate)+' will be used instead'+ W)
      bestLearningRate = initial_learning_rate
      lastLearningRate = initial_learning_rate

# Display info about the pretrained model to be loaded (or not)
if Use_pretrained_model:
  print('Weights found in:')
  print(h5_file_path)
  print('will be loaded prior to training.')

else:
  print(bcolors.WARNING+'No pretrained network will be used.')

_____no_output_____

#**4. Train the network**
---
_____no_output_____## **4.1. Prepare the training data and model for training**
---

Here, we use the information from 3. to build the model and convert the training data into a suitable format for training._____no_output_____

#@markdown ##Create the model and dataset objects

# --------------------- Here we load the augmented data or the raw data ------------------------

if Use_Data_augmentation:
  Training_source_dir = Training_source_augmented
  Training_target_dir = Training_target_augmented

if not Use_Data_augmentation:
  Training_source_dir = Training_source
  Training_target_dir = Training_target
# --------------------- ------------------------------------------------

training_images_tiff=Training_source_dir+"/*.tif"
mask_images_tiff=Training_target_dir+"/*.tif"

# this funtion imports training images and masks and sorts them suitable for the network
X = sorted(glob(training_images_tiff))
Y = sorted(glob(mask_images_tiff))

# assert -funtion check that X and Y really have images. If not this cell raises an error
assert all(Path(x).name==Path(y).name for x,y in zip(X,Y))

# Here we map the training dataset (images and masks).
X = list(map(imread,X))
Y = list(map(imread,Y))
n_channel = 1 if X[0].ndim == 2 else X[0].shape[-1]

#Normalize images and fill small label holes.
axis_norm = (0,1)   # normalize channels independently
# axis_norm = (0,1,2) # normalize channels jointly
if n_channel > 1:
    print("Normalizing image channels %s." % ('jointly' if axis_norm is None or 2 in axis_norm else 'independently'))
    sys.stdout.flush()

X = [normalize(x,1,99.8,axis=axis_norm) for x in tqdm(X)]
Y = [fill_label_holes(y) for y in tqdm(Y)]

#Here we split the your training dataset into training images (90 %) and validation images (10 %).
#It is advisable to use 10 % of your training dataset for validation. This ensures the truthfull validation error value. If only few validation images are used network may choose too easy or too challenging images for validation.
# split training data (images and masks) into training images and validation images.
assert len(X) > 1, "not enough training data"
rng = np.random.RandomState(42)
ind = rng.permutation(len(X))
n_val = max(1, int(round(percentage * len(ind))))
ind_train, ind_val = ind[:-n_val], ind[-n_val:]
X_val, Y_val = [X[i] for i in ind_val]  , [Y[i] for i in ind_val]
X_trn, Y_trn = [X[i] for i in ind_train], [Y[i] for i in ind_train]
print('number of images: %3d' % len(X))
print('- training:       %3d' % len(X_trn))
print('- validation:     %3d' % len(X_val))

# Use OpenCL-based computations for data generator during training (requires 'gputools')
use_gpu = False and gputools_available()

#Here we ensure that our network has a minimal number of steps
if (Use_Default_Advanced_Parameters):
  number_of_steps= int(len(X)/batch_size)+1

# --------------------- Using pretrained model ------------------------
#Here we ensure that the learning rate set correctly when using pre-trained models
if Use_pretrained_model:
  if Weights_choice == "last":
    initial_learning_rate = lastLearningRate

  if Weights_choice == "best":
    initial_learning_rate = bestLearningRate
# --------------------- ---------------------- ------------------------

conf = Config2D (
    n_rays       = n_rays,
    use_gpu      = use_gpu,
    train_batch_size = batch_size,
    n_channel_in = n_channel,
    train_patch_size = (patch_size, patch_size),
    grid = (grid_parameter, grid_parameter),
    train_learning_rate = initial_learning_rate,
)

# Here we create a model according to section 5.3.
model = StarDist2D(conf, name=model_name, basedir=model_path)

# --------------------- Using pretrained model ------------------------
# Load the pretrained weights
if Use_pretrained_model:
  model.load_weights(h5_file_path)

# --------------------- ---------------------- ------------------------

#Here we check the FOV of the network.
median_size = calculate_extents(list(Y), np.median)
fov = np.array(model._axes_tile_overlap('YX'))
if any(median_size > fov):
    print(bcolors.WARNING+"WARNING: median object size larger than field of view of the neural network.")
print(conf)

_____no_output_____

## **4.2. Start Trainning**
---

When playing the cell below you should see updates after each epoch (round). Network training can take some time.

* **CRITICAL NOTE:** Google Colab has a time limit for processing (to prevent using GPU power for datamining). Training time must be less than 12 hours! If training takes longer than 12 hours, please decrease the number of epochs or number of patches. Another way circumvent this is to save the parameters of the model after training and start training again from this point.

**Of Note:** At the end of the training, your model will be automatically exported so it can be used in the Stardist Fiji plugin. You can find it in your model folder (TF_SavedModel.zip). In Fiji, Make sure to choose the right version of tensorflow. You can check at: Edit-- Options-- Tensorflow. Choose the version 1.4 (CPU or GPU depending on your system)._____no_output_____

start = time.time()

#@markdown ##Start training
augmenter = None

# def augmenter(X_batch, Y_batch):
#     """Augmentation for data batch.
#     X_batch is a list of input images (length at most batch_size)
#     Y_batch is the corresponding list of ground-truth label images
#     """
#     # ...
#     return X_batch, Y_batch

# Training the model.
# 'input_epochs' and 'steps' refers to your input data in section 5.1
history = model.train(X_trn, Y_trn, validation_data=(X_val,Y_val), augmenter=augmenter,
                      epochs=number_of_epochs, steps_per_epoch=number_of_steps)
None;

print("Training done")

print("Network optimization in progress")
#Here we optimize the network.
model.optimize_thresholds(X_val, Y_val)

print("Done")

# convert the history.history dict to a pandas DataFrame:
lossData = pd.DataFrame(history.history)

if os.path.exists(model_path+"/"+model_name+"/Quality Control"):
  shutil.rmtree(model_path+"/"+model_name+"/Quality Control")

os.makedirs(model_path+"/"+model_name+"/Quality Control")

# The training evaluation.csv is saved (overwrites the Files if needed).
lossDataCSVpath = model_path+'/'+model_name+'/Quality Control/training_evaluation.csv'
with open(lossDataCSVpath, 'w') as f:
  writer = csv.writer(f)
  writer.writerow(['loss','val_loss', 'learning rate'])
  for i in range(len(history.history['loss'])):
    writer.writerow([history.history['loss'][i], history.history['val_loss'][i], history.history['lr'][i]])

# Displaying the time elapsed for training
dt = time.time() - start
mins, sec = divmod(dt, 60)
hour, mins = divmod(mins, 60)
print("Time elapsed:",hour, "hour(s)",mins,"min(s)",round(sec),"sec(s)")

model.export_TF()

print("Your model has been sucessfully exported and can now also be used in the Stardist Fiji plugin")
_____no_output_____

## **4.3. Download your model(s) from Google Drive**
---

Once training is complete, the trained model is automatically saved on your Google Drive, in the **model_path** folder that was selected in Section 3. It is however wise to download the folder as all data can be erased at the next training if using the same folder.

_____no_output_____# **5. Evaluate your model**
---

This section allows the user to perform important quality checks on the validity and generalisability of the trained model.

**We highly recommend to perform quality control on all newly trained models.**

_____no_output_____

# model name and path
#@markdown ###Do you want to assess the model you just trained ?
Use_the_current_trained_model = True #@param {type:"boolean"}

#@markdown ###If not, please provide the path to the model folder:

QC_model_folder = "" #@param {type:"string"}

#Here we define the loaded model name and path
QC_model_name = os.path.basename(QC_model_folder)
QC_model_path = os.path.dirname(QC_model_folder)

if (Use_the_current_trained_model):
  QC_model_name = model_name
  QC_model_path = model_path

full_QC_model_path = QC_model_path+'/'+QC_model_name+'/'
if os.path.exists(full_QC_model_path):
  print("The "+QC_model_name+" network will be evaluated")
else:
  print(bcolors.WARNING+'!! WARNING: The chosen model does not exist !!')
  print('Please make sure you provide a valid model path and model name before proceeding further.')
_____no_output_____

## **5.1. Inspection of the loss function**
---

First, it is good practice to evaluate the training progress by comparing the training loss with the validation loss. The latter is a metric which shows how well the network performs on a subset of unseen data which is set aside from the training dataset. For more information on this, see for example [this review](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6381354/) by Nichols *et al.*

**Training loss** describes an error value after each epoch for the difference between the model's prediction and its ground-truth target.

**Validation loss** describes the same error value between the model's prediction on a validation image and compared to it's target.

During training both values should decrease before reaching a minimal value which does not decrease further even after more training. Comparing the development of the validation loss with the training loss can give insights into the model's performance.

Decreasing **Training loss** and **Validation loss** indicates that training is still necessary and increasing the `number_of_epochs` is recommended. Note that the curves can look flat towards the right side, just because of the y-axis scaling. The network has reached convergence once the curves flatten out. After this point no further training is required. If the **Validation loss** suddenly increases again an the **Training loss** simultaneously goes towards zero, it means that the network is overfitting to the training data. In other words the network is remembering the exact patterns from the training data and no longer generalizes well to unseen data. In this case the training dataset has to be increased.

_____no_output_____

#@markdown ##Play the cell to show a plot of training errors vs. epoch number

lossDataFromCSV = []
vallossDataFromCSV = []

with open(QC_model_path+'/'+QC_model_name+'/Quality Control/training_evaluation.csv','r') as csvfile:
    csvRead = csv.reader(csvfile, delimiter=',')
    next(csvRead)
    for row in csvRead:
        lossDataFromCSV.append(float(row[0]))
        vallossDataFromCSV.append(float(row[1]))

epochNumber = range(len(lossDataFromCSV))
plt.figure(figsize=(15,10))

plt.subplot(2,1,1)
plt.plot(epochNumber,lossDataFromCSV, label='Training loss')
plt.plot(epochNumber,vallossDataFromCSV, label='Validation loss')
plt.title('Training loss and validation loss vs. epoch number (linear scale)')
plt.ylabel('Loss')
plt.xlabel('Epoch number')
plt.legend()

plt.subplot(2,1,2)
plt.semilogy(epochNumber,lossDataFromCSV, label='Training loss')
plt.semilogy(epochNumber,vallossDataFromCSV, label='Validation loss')
plt.title('Training loss and validation loss vs. epoch number (log scale)')
plt.ylabel('Loss')
plt.xlabel('Epoch number')
plt.legend()
plt.savefig(QC_model_path+'/'+QC_model_name+'/Quality Control/lossCurvePlots.png')
plt.show()

_____no_output_____

## **5.2. Error mapping and quality metrics estimation**
---
This section will calculate the Intersection over Union score for all the images provided in the Source_QC_folder and Target_QC_folder ! The result for one of the image will also be displayed.

The **Intersection over Union** metric is a method that can be used to quantify the percent overlap between the target mask and your prediction output. **Therefore, the closer to 1, the better the performance.** This metric can be used to assess the quality of your model to accurately predict nuclei.

 The results can be found in the "*Quality Control*" folder which is located inside your "model_folder"._____no_output_____

#@markdown ##Choose the folders that contain your Quality Control dataset

Source_QC_folder = "" #@param{type:"string"}
Target_QC_folder = "" #@param{type:"string"}

#Create a quality control Folder and check if the folder already exist
if os.path.exists(QC_model_path+"/"+QC_model_name+"/Quality Control") == False:
  os.makedirs(QC_model_path+"/"+QC_model_name+"/Quality Control")

if os.path.exists(QC_model_path+"/"+QC_model_name+"/Quality Control/Prediction"):
  shutil.rmtree(QC_model_path+"/"+QC_model_name+"/Quality Control/Prediction")

os.makedirs(QC_model_path+"/"+QC_model_name+"/Quality Control/Prediction")

# Generate predictions from the Source_QC_folder and save them in the QC folder

Source_QC_folder_tif = Source_QC_folder+"/*.tif"

np.random.seed(16)
lbl_cmap = random_label_cmap()
Z = sorted(glob(Source_QC_folder_tif))
Z = list(map(imread,Z))
n_channel = 1 if Z[0].ndim == 2 else Z[0].shape[-1]
axis_norm = (0,1)   # normalize channels independently

print('Number of test dataset found in the folder: '+str(len(Z)))

  # axis_norm = (0,1,2) # normalize channels jointly
if n_channel > 1:
  print("Normalizing image channels %s." % ('jointly' if axis_norm is None or 2 in axis_norm else 'independently'))

model = StarDist2D(None, name=QC_model_name, basedir=QC_model_path)

names = [os.path.basename(f) for f in sorted(glob(Source_QC_folder_tif))]

# modify the names to suitable form: path_images/image_numberX.tif

lenght_of_Z = len(Z)

for i in range(lenght_of_Z):
  img = normalize(Z[i], 1,99.8, axis=axis_norm)
  labels, polygons = model.predict_instances(img)
  os.chdir(QC_model_path+"/"+QC_model_name+"/Quality Control/Prediction")
  imsave(names[i], labels, polygons)

# Here we start testing the differences between GT and predicted masks

with open(QC_model_path+"/"+QC_model_name+"/Quality Control/Quality_Control for "+QC_model_name+".csv", "w", newline='') as file:
  writer = csv.writer(file)
  writer.writerow(["image","Prediction v. GT Intersection over Union"])

# define the images

  for n in os.listdir(Source_QC_folder):

    if not os.path.isdir(os.path.join(Source_QC_folder,n)):
      print('Running QC on: '+n)
      test_input = io.imread(os.path.join(Source_QC_folder,n))
      test_prediction = io.imread(os.path.join(QC_model_path+"/"+QC_model_name+"/Quality Control/Prediction",n))
      test_ground_truth_image = io.imread(os.path.join(Target_QC_folder, n))

      #Convert pixel values to 0 or 255
      test_prediction_0_to_255 = test_prediction
      test_prediction_0_to_255[test_prediction_0_to_255>0] = 255

      #Convert pixel values to 0 or 255
      test_ground_truth_0_to_255 = test_ground_truth_image
      test_ground_truth_0_to_255[test_ground_truth_0_to_255>0] = 255

      # Intersection over Union metric

      intersection = np.logical_and(test_ground_truth_0_to_255, test_prediction_0_to_255)
      union = np.logical_or(test_ground_truth_0_to_255, test_prediction_0_to_255)
      iou_score =  np.sum(intersection) / np.sum(union)
      writer.writerow([n, str(iou_score)])

#Display the last image

f = plt.figure(figsize=(25,25))

from astropy.visualization import simple_norm
norm = simple_norm(test_input, percent = 99)

#Input
plt.subplot(1,4,1)
plt.axis('off')
plt.imshow(test_input, aspect='equal', norm=norm, cmap='magma', interpolation='nearest')
plt.title('Input')

#Ground-truth
plt.subplot(1,4,2)
plt.axis('off')
plt.imshow(test_ground_truth_0_to_255, aspect='equal', cmap='Greens')
plt.title('Ground Truth')

#Prediction
plt.subplot(1,4,3)
plt.axis('off')
plt.imshow(test_prediction_0_to_255, aspect='equal', cmap='Purples')
plt.title('Prediction')

#Overlay
plt.subplot(1,4,4)
plt.axis('off')
plt.imshow(test_ground_truth_0_to_255, cmap='Greens')
plt.imshow(test_prediction_0_to_255, alpha=0.5, cmap='Purples')
plt.title('Ground Truth and Prediction, Intersection over Union:'+str(round(iou_score,3)));

_____no_output_____

# **6. Using the trained model**
---_____no_output_____

## **6.1 Generate prediction(s) from unseen dataset**
---

In this section the unseen data is processed using the trained model (in section 4). First, your unseen images are uploaded and prepared for prediction. After that your trained model from section 4 is activated and finally saved into your Google Drive.

---

The current trained model (from section 4.3) can now be used to process images. If an older model needs to be used, please untick the **Use_the_current_trained_model** box and enter the name and path of the model to use. Predicted output images are saved in your **Prediction_folder** folder as restored image stacks (ImageJ-compatible TIFF images).

**`Data_folder`:** This folder should contains the images that you want to predict using the network that you will train.

**`Result_folder`:** This folder will contain the predicted output ROI.

**`Data_type`:** Please indicate if the images you want to predict are single images or stacks

In stardist the following results can be exported:
- Region of interest (ROI) that can be opened in ImageJ / Fiji. The ROI are saved inside of a .zip file in your choosen result folder. To open the ROI in Fiji, just drag and drop the zip file !**
- The predicted mask images
- A tracking file that can easily be imported into Trackmate to track the nuclei (Stacks only).
- A CSV file that contains the number of nuclei detected per image (single image only).
- A CSV file that contains the coordinate the centre of each detected nuclei (single image only).

_____no_output_____

Single_Images = 1
Stacks = 2

#@markdown ### Provide the path to your dataset and to the folder where the prediction will be saved (Result folder), then play the cell to predict output on your unseen images.

Data_folder = "" #@param {type:"string"}
Results_folder = "" #@param {type:"string"}

#@markdown ###Are your data single images or stacks?

Data_type = Single_Images #@param ["Single_Images", "Stacks"] {type:"raw"}

#@markdown ###What outputs would you like to generate?
Region_of_interests = True #@param {type:"boolean"}
Mask_images = True #@param {type:"boolean"}
Tracking_file = False #@param {type:"boolean"}

# model name and path
#@markdown ###Do you want to use the current trained model?
Use_the_current_trained_model = False #@param {type:"boolean"}

#@markdown ###If not, please provide the path to the model folder:

Prediction_model_folder = "" #@param {type:"string"}

#Here we find the loaded model name and parent path
Prediction_model_name = os.path.basename(Prediction_model_folder)
Prediction_model_path = os.path.dirname(Prediction_model_folder)

if (Use_the_current_trained_model):
  print("Using current trained network")
  Prediction_model_name = model_name
  Prediction_model_path = model_path

full_Prediction_model_path = Prediction_model_path+'/'+Prediction_model_name+'/'
if os.path.exists(full_Prediction_model_path):
  print("The "+Prediction_model_name+" network will be used.")
else:
  print(bcolors.WARNING+'!! WARNING: The chosen model does not exist !!'+W)
  print('Please make sure you provide a valid model path and model name before proceeding further.')

#single images
Data_folder = Data_folder+"/*.tif"

if Data_type == 1 :
  print("Single images are now beeing predicted")
  np.random.seed(16)
  lbl_cmap = random_label_cmap()
  X = sorted(glob(Data_folder))
  X = list(map(imread,X))
  n_channel = 1 if X[0].ndim == 2 else X[0].shape[-1]
  axis_norm = (0,1)   # normalize channels independently

  # axis_norm = (0,1,2) # normalize channels jointly
  if n_channel > 1:
    print("Normalizing image channels %s." % ('jointly' if axis_norm is None or 2 in axis_norm else 'independently'))
  model = StarDist2D(None, name = Prediction_model_name, basedir = Prediction_model_path)

  names = [os.path.basename(f) for f in sorted(glob(Data_folder))]

  Nuclei_number = []

  # modify the names to suitable form: path_images/image_numberX.tif
  FILEnames = []
  for m in names:
    m = Results_folder+'/'+m
    FILEnames.append(m)

  # Create a list of name with no extension

  name_no_extension=[]
  for n in names:
    name_no_extension.append(os.path.splitext(n)[0])

  # Save all ROIs and masks into results folder

  for i in range(len(X)):
    img = normalize(X[i], 1,99.8, axis = axis_norm)
    labels, polygons = model.predict_instances(img)

    os.chdir(Results_folder)

    if Mask_images:
      imsave(FILEnames[i], labels, polygons)

    if Region_of_interests:
      export_imagej_rois(name_no_extension[i], polygons['coord'])

    if Tracking_file:
      print(bcolors.WARNING+"Tracking files are only generated when stacks are predicted"+W)

    Nuclei_centre_coordinate = polygons['points']
    my_df2 = pd.DataFrame(Nuclei_centre_coordinate)
    my_df2.columns =['Y', 'X']

    my_df2.to_csv(Results_folder+'/'+name_no_extension[i]+'_Nuclei_centre.csv', index=False, header=True)

    Nuclei_array = polygons['coord']
    Nuclei_array2 = [names[i], Nuclei_array.shape[0]]
    Nuclei_number.append(Nuclei_array2)

  my_df = pd.DataFrame(Nuclei_number)
  my_df.to_csv(Results_folder+'/Nuclei_count.csv', index=False, header=False)

  # One example is displayed

  print("One example image is displayed bellow:")
  plt.figure(figsize=(10,10))
  plt.imshow(img if img.ndim==2 else img[...,:3], clim=(0,1), cmap='gray')
  plt.imshow(labels, cmap=lbl_cmap, alpha=0.5)
  plt.axis('off');
  plt.savefig(name_no_extension[i]+"_overlay.tif")

if Data_type == 2 :
  print("Stacks are now beeing predicted")
  np.random.seed(42)
  lbl_cmap = random_label_cmap()
  Y = sorted(glob(Data_folder))
  X = list(map(imread,Y))
  n_channel = 1 if X[0].ndim == 2 else X[0].shape[-1]
  axis_norm = (0,1)   # normalize channels independently
  # axis_norm = (0,1,2) # normalize channels jointly
  if n_channel > 1:
     print("Normalizing image channels %s." % ('jointly' if axis_norm is None or 2 in axis_norm else 'independently'))
  #Load a pretrained network
  model = StarDist2D(None, name = Prediction_model_name, basedir = Prediction_model_path)

  names = [os.path.basename(f) for f in sorted(glob(Data_folder))]

  # Create a list of name with no extension

  name_no_extension = []
  for n in names:
    name_no_extension.append(os.path.splitext(n)[0])

  outputdir = Path(Results_folder)

# Save all ROIs and images in Results folder.
  for num, i in enumerate(X):
    print("Performing prediction on: "+names[num])

    timelapse = np.stack(i)
    timelapse = normalize(timelapse, 1,99.8, axis=(0,)+tuple(1+np.array(axis_norm)))
    timelapse.shape

    if Region_of_interests:
      polygons = [model.predict_instances(frame)[1]['coord'] for frame in tqdm(timelapse)]
      export_imagej_rois(os.path.join(outputdir, name_no_extension[num]), polygons)

    n_timepoint = timelapse.shape[0]
    prediction_stack = np.zeros((n_timepoint, timelapse.shape[1], timelapse.shape[2]))
    Tracking_stack = np.zeros((n_timepoint, timelapse.shape[2], timelapse.shape[1]))

# Save the masks in the result folder
    if Mask_images or Tracking_file:
      for t in range(n_timepoint):
        img_t = timelapse[t]
        labels, polygons = model.predict_instances(img_t)
        prediction_stack[t] = labels

# Create a tracking file for trackmate

        for point in polygons['points']:
          cv2.circle(Tracking_stack[t],tuple(point),0,(1), -1)

      prediction_stack_32 = img_as_float32(prediction_stack, force_copy=False)
      Tracking_stack_32 = img_as_float32(Tracking_stack, force_copy=False)
      Tracking_stack_8 = img_as_ubyte(Tracking_stack_32, force_copy=True)

      Tracking_stack_8_rot = np.rot90(Tracking_stack_8, axes=(1,2))
      Tracking_stack_8_rot_flip = np.fliplr(Tracking_stack_8_rot)

      os.chdir(Results_folder)
      if Mask_images:
        imsave(names[num], prediction_stack_32)
      if Tracking_file:
        imsave(name_no_extension[num]+"_tracking_file.tif", Tracking_stack_8_rot_flip)

print("Predictions completed")   _____no_output_____

## **6.2. Download your predictions**
---

**Store your data** and ALL its results elsewhere by downloading it from Google Drive and after that clean the original folder tree (datasets, results, trained model etc.) if you plan to train or use new networks. Please note that the notebook will otherwise **OVERWRITE** all files which have the same name._____no_output_____
#**Thank you for using StarDist 2D!**_____no_output_____


# Notebook from schwartz-cnl/Computational-Neuroscience-Class
Path: Lab 7 Convolutional Neural Network/cnn_fashion_keras.ipynb

_____no_output_____# **Convolutional neural network (CNN) for Fashion MNIST dataset**

by Md Nasir Uddin Laskar;
Edited 2022: Odelia Schwartz and Xu Pan

Change colab runtime type to GPU to accelerate: "Runtime" --> "Change runtime type"--> under "Hardware accelerator" select "GPU".

-  We will implement a simple CNN model with 2 convolution and 2 fully connected layers as the following figure. **conv1 --> relu1 --> pool1 ----> conv2 --> relu2 --> pool2 ----> fc1 --> fc2**

![picture](https://nasirml.files.wordpress.com/2019/01/simple_convnet-3.png?w=580&h=182)

## **Common deep learning steps**

* Step 1: Generate training and test data (and preprocess)
* Step 2: Initialize the network parameters
* Step 3: Forward propagation
* Step 4: Compute the cost/loss
* Step 5: Backpropagation or create an optimizer to minimize the cost from step 4
* Step 6: Evaluate the model with your test set
_____no_output_____

# imports
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from tensorflow import keras
import os   # to save the checkpoint
#sess = tf.InteractiveSession()_____no_output_____

# **Load the dataset**
* **Fashion-MNIST** is a dataset of article images—consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. It shares the same image size and structure of training and testing splits of MNIST.

* The images are 28x28 NumPy arrays, with pixel values ranging between 0 and 255. The labels are an array of integers, ranging from 0 to 9. These correspond to the class of clothing the image represents.

* MNIST is very easy. Classic machine learning algorithms can also achieve 97% easily as you have see in the last lab. With convolutional nets, we can achieve 99.7% on MNIST. That is why we are trying a more serious dataset which a bit challenging but still gives us the fexibility and ease of MNIST.

* Sample images from the Fashion MNIST data in the following figure (each class takes three rows).

_____no_output_____

    Figure 1. Fashion-MNIST samples (by Zalando, MIT License).

_____no_output_____Loading the dataset returns four NumPy arrays:

* The `x_train` and `y_train` arrays are the *training set*—the data the model uses to learn.
* The model is tested against the *test set*, the `x_test`, and `y_test` arrays.

The images are 28x28 NumPy arrays, with pixel values ranging between 0 and 255. The *labels* are an array of integers, ranging from 0 to 9. These correspond to the *class* of clothing the image represents:

    Label
    Class

    0
    T-shirt/top

    1
    Trouser

    2
    Pullover

    3
    Dress

    4
    Coat

    5
    Sandal

    6
    Shirt

    7
    Sneaker

    8
    Bag

    9
    Ankle boot

* Each image is mapped to a single label. Since the *class names* are not included with the dataset, store them here to use later when plotting the images:

* **loading the dataset is very easy as it is already included in the TensorFlow/Keras library. We just have to include `tf.keras.datasets.fashion_mnist` and call the `load_data()` function.**
* It returns training and test datasets including the labels.
_____no_output_____

def read_dataset():
  '''
  - read fashion mnist dataset
  -
  '''
  fashion_mnist = tf.keras.datasets.fashion_mnist
  (x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()

  return x_train, y_train, x_test, y_test
_____no_output_____# call the function to read fashion mnist
x_train, y_train, x_test, y_test = read_dataset()_____no_output_____

# **Look at the data**

* **Training data** — used for training the model.

* **Test data** — are the images not seen the network yet. A totally new set to test the generalization capability of the network.

_____no_output_____

print('Training data size:' + str(x_train.shape))
print('Training labels size: ' + str(y_train.shape))

print('Test data size: ' + str(x_test.shape))
print('Test label size: ' + str(y_test.shape))_____no_output_____fashion_mnist_labels = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
                        'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
print(fashion_mnist_labels)
print(y_train[100], fashion_mnist_labels[y_train[100]])_____no_output_____# look at the first 10 images
for i in range(10):
    plt.subplot(2, 5, i+1); plt.axis('off')
    plt.imshow(np.reshape(x_train[i], [28, 28]), cmap='gray')
    plt.title(fashion_mnist_labels[y_train[i]])
plt.show()_____no_output_____

# **Preprocess data**
* **One-hot encoding.**
* Reshape to `28 x 28x 1`
_____no_output_____

# normalize the images
#x_train = x_train.astype('float32')/255.0  # sometimes standard to have pixel values in the range [0,1]
#x_test = x_test.astype('float32')/255.0

# Convert the labels to one-hot vectors
print(y_train[0:5])
y_train = tf.keras.utils.to_categorical(y_train, 10)  # 10 classes
y_test = tf.keras.utils.to_categorical(y_test, 10)
print(y_train[0:5])_____no_output_____# Reshape input data from (28,28) to (28, 28, 1), to 3D
# height x width x channel. we have gray scale images so channel = 1
print(np.shape(x_test))
w, h = 28, 28
x_train = x_train.reshape(x_train.shape[0], w, h, 1)
x_test = x_test.reshape(x_test.shape[0], w, h, 1)
print(np.shape(x_test))_____no_output_____

# **Build the CNN model**
- Make sure to define the input shape in the first layer of the neural network.
- The rest of the layers can adjust the input and output shapes automatically

---

![picture](https://nasirml.files.wordpress.com/2019/01/simple_convnet-3.png?w=580&h=182)

*   Convolutional layer (conv):
*   ReLU layer:
*   Pooling layer:
*   Dropout layer:
*   Fully connected layer (FC):

_____no_output_____
* We use `model.compile()`  to configure the learning process before training the model.

* This is where we define the type of loss function, optimizer and the metrics evaluated by the model during training and testing.

* **Important:** before we apply the fully-connected layers, we need to flatten the the current output._____no_output_____

def create_model():
  model = tf.keras.Sequential()

  model.add(tf.keras.layers.Conv2D(filters=64, kernel_size=3, padding='same', activation='relu', input_shape=(28,28,1)))
  model.add(tf.keras.layers.MaxPooling2D(pool_size=2))

  model.add(tf.keras.layers.Conv2D(filters=32, kernel_size=3, padding='same', activation='relu'))
  model.add(tf.keras.layers.MaxPooling2D(pool_size=2))

  model.add(tf.keras.layers.Flatten())
  model.add(tf.keras.layers.Dense(256, activation='relu')) # 1568*256+256 = 401664 params total
  model.add(tf.keras.layers.Dropout(0.5))
  model.add(tf.keras.layers.Dense(10, activation='softmax'))

  model.compile(loss='categorical_crossentropy',
             optimizer='adam',
             metrics=['accuracy'])

  return model_____no_output_____# create the model

model = create_model()_____no_output_____

## **Model summary**

* `conv2d` means the convolution in 2D (using 3x3 filters).
* Number of parameters 640 comes from `3*3*64 + 64`
_____no_output_____

# Take a look at the model summary
model.summary()
_____no_output_____

# **Now Train our model CNN**

*  Train the model and also save the checkpoint so that we do not have to train everytime

*   Shuffle is TRUE by default. Still I include it to tell that this is very important.

*  Takes around 20 minutes for 10 epochs/iterations.

* Loss decreases and accuracy increases

_____no_output_____

'''
- more params
  - validation_split=0.1 means 10% of the training data will be used for validation set
  -
'''
n_iteration = 10
history = model.fit(x_train, y_train, batch_size=64, epochs=n_iteration, shuffle=True)
_____no_output_____

# **See how the training has gone**

* Lets evaluate our model on the test set.
* Looks like our CNN model doing a pretty good job at classifying the unknown images with almost 90% accuracy with 3 iterations and  and 92% with 10 epochs!_____no_output_____

test_loss, test_acc = model.evaluate(x_test, y_test)
print('Test accuracy: %f' %test_acc)_____no_output_____

## **See the loss**

* Loss should decrease over time.
* If you see that the loss in your program does not decrease over time, most likely, something went wrong and you should check with your input and network initialization and hyper-parameters.
_____no_output_____

# history. all losses
all_loss = history.history['loss']
print(len(all_loss))

# show the training errors
plt.plot(np.squeeze(all_loss))
plt.xlabel('iterations');plt.ylabel('cost')
#plt.title('Learning rate %f' %learning_rate)
plt.show()

_____no_output_____

## **Visualize some predictions**

* Let's randomly pick some **test** images and visualize the prediction using the model we just trained. First we get the predictions with the model from the test data. Then we print out some images from the test data set, and set the titles with the prediction (and the groud truth label). If the prediction matches the true label, the title will be green; otherwise it's displayed in red.

* As we know the accuracy is around 90%, we see that the network occasionaly makes mistakes to predict a class.

* You can convince yourself that the mistakes are somewhat reasonable._____no_output_____

# find the probability of the 10 classes and then assign the class label with
# the higher probability.  model.predict(x_test) gives directly the probabilities
# predicted for each of the class labels.
y_hat = model.predict(x_test)
print(y_hat.shape)
print (y_hat[500])
print (np.round(y_hat[500],2))

# Plot a random sample of 16 test images, their predicted labels and ground truth
figure = plt.figure(figsize=(20, 8))
for i, index in enumerate(np.random.choice(x_test.shape[0], size=16, replace=False)):
    ax = figure.add_subplot(4, 4, i + 1, xticks=[], yticks=[])
    # Display each image
    ax.imshow(np.squeeze(x_test[index]),cmap='gray')
    predict_index = np.argmax(y_hat[index])                 # The fashion label number that is predicted
    true_index = np.argmax(y_test[index])                   # The true label number
    # Set the title for each image
    ax.set_title("{} ({})".format(fashion_mnist_labels[predict_index],
                                  fashion_mnist_labels[true_index]),
                                  color=("green" if predict_index == true_index else "red"))_____no_output_____

## **TODO**

1. **Add one convolution layer** in the network and train again. See if it increases the accuracy. Optional: You could further try changing other aspects of the network archirecture and computations to see if you can improve the result.

2. Find the **Confusion Matrix** and see how many samples in each class the network can predict correctly and how many are not predicted correctly. To do so, create a 10 by 10 matrix initialized to zeros, with the X axis corresponding to the true index and the Y axis to the predicted index. Loop over the 10,000 samples and each time add 1 to the matrix position asscoiated with the true index on the X axis and predicted index on the Y axis. Use plt.imshow to display your resulting matrix. What fashion items are most confused? What would the confusion matrix look like for a perfect network that always predicts correctly?

3. Try training the regular mnist database (replace fashion_mnist with mnist). How do the test accuracy and confusion matrices compare to the fashion_mnist?

_____no_output_____


# Notebook from kunxing97/machinelearning_r
Path: machinelearn.ipynb

#install.packages("caret", dependencies=c("Depends", "Suggests"))
Warning message:
“dependencies ‘R2wd’, ‘rrlda’, ‘webshot2’, ‘RTCGA.rnaseq’, ‘globaltest’, ‘Biobase’, ‘genefilter’, ‘golubEsets’, ‘rPython’, ‘nws’, ‘actuar’, ‘sylly.de’, ‘sylly.es’, ‘marray’, ‘affy’, ‘limma’, ‘Rgraphviz’, ‘graph’, ‘starsdata’, ‘rggobi’, ‘rnaturalearthhires’, ‘Biostrings’, ‘seqLogo’, ‘glmmADMB’, ‘Rcampdf’, ‘tm.lexicon.GeneralInquirer’, ‘cmdstanr’, ‘GAMBoost’, ‘spDataLarge’, ‘M3C’, ‘ComplexHeatmap’, ‘RDCOMClient’, ‘doRedis’, ‘INLA’, ‘sva’ are not available”
also installing the dependencies ‘cond’, ‘nlreg’, ‘cccd’, ‘diffusionMap’, ‘lle’, ‘pcaL1’, ‘tensorflow’, ‘umap’, ‘bigmemory’, ‘synchronicity’, ‘hrbrthemes’, ‘rhub’, ‘biglasso’, ‘KernelKnn’, ‘SIS’, ‘marg’, ‘ddalpha’, ‘dimRed’, ‘NMF’, ‘ROI.plugin.ecos’, ‘ROI.plugin.neos’, ‘ROI.plugin.alabama’, ‘rpf’, ‘snowfall’, ‘umx’, ‘ifaTools’, ‘ltm’, ‘smcfcs’, ‘casebase’, ‘ipw’, ‘SuperLearner’, ‘cobs’, ‘mvPot’, ‘gmm’, ‘revdbayes’, ‘kdecopula’, ‘Rmixmod’, ‘circular’, ‘dismo’, ‘BRugs’, ‘csampling’, ‘bife’, ‘gameofthrones’, ‘harrypotter’, ‘oompaBase’, ‘pals’, ‘butcher’, ‘ggbeeswarm’, ‘jtools’, ‘ez’, ‘ggResidpanel’, ‘detectseparation’, ‘coneproj’, ‘svDialogs’, ‘SemiPar’, ‘CVST’, ‘OpenMx’, ‘wTO’, ‘infutil’, ‘riskRegression’, ‘energy’, ‘rmcorr’, ‘transformr’, ‘Gmedian’, ‘rrcov’, ‘ICSNP’, ‘REPPlab’, ‘netrankr’, ‘influenceR’, ‘NetSwan’, ‘lcopula’, ‘mev’, ‘qrng’, ‘VineCopula’, ‘shinydisconnect’, ‘VarianceGamma’, ‘SkewHyperbolic’, ‘rcdd’, ‘pedigreemm’, ‘Infusion’, ‘IsoriX’, ‘blackbox’, ‘ROI.plugin.glpk’, ‘pez’, ‘picante’, ‘logistf’, ‘phylolm’, ‘maxlike’, ‘R2OpenBUGS’, ‘R2WinBUGS’, ‘jagsUI’, ‘brglm’, ‘benchr’, ‘hoa’, ‘NlcOptim’, ‘doRNG’, ‘carrier’, ‘alpaca’, ‘paletteer’, ‘workflows’, ‘afex’, ‘BayesFM’, ‘brglm2’, ‘cAIC4’, ‘cgam’, ‘clubSandwich’, ‘cplm’, ‘DRR’, ‘EGAnet’, ‘lavaSearch2’, ‘lqmm’, ‘metaBMA’, ‘mfx’, ‘multimode’, ‘nFactors’, ‘panelr’, ‘projpred’, ‘sjstats’, ‘WRS2’, ‘udunits2’, ‘janitor’, ‘purrrlyr’, ‘Guerry’, ‘pcaPP’, ‘diagram’, ‘GA’, ‘BayesXsrc’, ‘MBA’, ‘changepoint’, ‘lfda’, ‘RcppParallel’, ‘runjags’, ‘modeest’, ‘semTools’, ‘nimble’, ‘correlation’, ‘tidymodels’, ‘gganimate’, ‘merTools’, ‘bigutilsr’, ‘ICS’, ‘ICSOutlier’, ‘ggraph’, ‘qqplotr’, ‘tidygraph’, ‘jsonify’, ‘colourvalues’, ‘sfheaders’, ‘ggmap’, ‘adehabitatMA’, ‘copula’, ‘colourpicker’, ‘ramcmc’, ‘sde’, ‘mc2d’, ‘GeneralizedHyperbolic’, ‘eha’, ‘equatags’, ‘devEMF’, ‘spaMM’, ‘phyr’, ‘emdbook’, ‘AICcmodavg’, ‘ei’, ‘eiPack’, ‘networkD3’, ‘optmatch’, ‘reactable’, ‘stringdist’, ‘faraway’, ‘mets’, ‘Exact’, ‘BlakerCI’, ‘VGAMdata’, ‘NHANES’, ‘formatR’, ‘rsample’, ‘Metrics’, ‘tfestimators’, ‘cubature’, ‘GoFKernel’, ‘extraDistr’, ‘pracma’, ‘globals’, ‘beepr’, ‘pbmcapply’, ‘doFuture’, ‘furrr’, ‘decor’, ‘distro’, ‘lfe’, ‘gt’, ‘gtsummary’, ‘parameters’, ‘concaveman’, ‘units’, ‘questionr’, ‘snakecase’, ‘statnet.common’, ‘SGPdata’, ‘vcr’, ‘ade4TkGUI’, ‘adegraphics’, ‘adephylo’, ‘CircStats’, ‘splancs’, ‘waveslim’, ‘permute’, ‘FactoInvestigate’, ‘rainbow’, ‘FME’, ‘dotCall64’, ‘spam64’, ‘truncdist’, ‘whoami’, ‘flexdashboard’, ‘seriation’, ‘heatmaply’, ‘dynamicTreeCut’, ‘pvclust’, ‘DendSer’, ‘circlize’, ‘R2BayesX’, ‘Bessel’, ‘polynom’, ‘ggfortify’, ‘loo’, ‘rstantools’, ‘shinystan’, ‘BayesFactor’, ‘bayesQR’, ‘blavaan’, ‘bridgesampling’, ‘effectsize’, ‘ggridges’, ‘modelbased’, ‘performance’, ‘see’, ‘RWiener’, ‘rtdists’, ‘mnormt’, ‘splines2’, ‘arm’, ‘speedglm’, ‘pgirmess’, ‘skpr’, ‘geojsonsf’, ‘googleway’, ‘spatialwidget’, ‘RMariaDB’, ‘rpanel’, ‘soma’, ‘pso’, ‘geoknife’, ‘adehabitatLT’, ‘cshapes’, ‘ISOcodes’, ‘shinythemes’, ‘isa2’, ‘tree’, ‘pacman’, ‘psychTools’, ‘GPArotation’, ‘Rcsdp’, ‘lokern’, ‘miscTools’, ‘dlm’, ‘polycor’, ‘inlinedocs’, ‘lavaan.survey’, ‘pbivnorm’, ‘pbv’, ‘SimDesign’, ‘visNetwork’, ‘feather’, ‘qvcalc’, ‘glmmML’, ‘TTR’, ‘downloader’, ‘stabledist’, ‘DescTools’, ‘gridBase’, ‘pryr’, ‘invgamma’, ‘ipumsr’, ‘bssm’, ‘openxlsx’, ‘fitdistrplus’, ‘flexsurv’, ‘flextable’, ‘ftExtra’, ‘officer’, ‘pinp’, ‘DHARMa’, ‘bbmle’, ‘mvabund’, ‘orthopolynom’, ‘StanHeaders’, ‘BH’, ‘rjags’, ‘Zelig’, ‘expint’, ‘ROI’, ‘ridge’, ‘VIM’, ‘norm’, ‘sjmisc’, ‘sjPlot’, ‘WrightMap’, ‘basefun’, ‘variables’, ‘nonnest2’, ‘mdhglm’, ‘coxinterval’, ‘ICsurv’, ‘polspline’, ‘bshazard’, ‘gss’, ‘MIICD’, ‘TransModel’, ‘dynsurv’, ‘timereg’, ‘survivalMPL’, ‘muhaz’, ‘ICE’, ‘exact2x2’, ‘exactci’, ‘bpcp’, ‘bootstrap’, ‘R.methodsS3’, ‘gplots’, ‘VGAMextra’, ‘agricolae’, ‘mosaic’, ‘tfdatasets’, ‘jpeg’, ‘reticulate’, ‘distr6’, ‘future.callr’, ‘mlr3data’, ‘progressr’, ‘dials’, ‘formattable’, ‘arrow’, ‘janeaustenr’, ‘Lahman’, ‘nycflights13’, ‘latticeExtra’, ‘broom.helpers’, ‘geosphere’, ‘ggforce’, ‘intergraph’, ‘labelled’, ‘network’, ‘scagnostics’, ‘sna’, ‘itertools’, ‘RhpcBLASctl’, ‘future.apply’, ‘listenv’, ‘BatchJobs’, ‘ff’, ‘scalreg’, ‘R.devices’, ‘binda’, ‘Rglpk’, ‘Rsymphony’, ‘SGP’, ‘taxize’, ‘ade4’, ‘vegan’, ‘futile.logger’, ‘missMDA’, ‘Factoshiny’, ‘fds’, ‘deSolve’, ‘spam’, ‘testit’, ‘seasonalview’, ‘sn’, ‘pkgKitten’, ‘plot3D’, ‘ggthemes’, ‘shinytest’, ‘webshot’, ‘listviewer’, ‘dendextend’, ‘plotlyGeoAssets’, ‘palmerpenguins’, ‘KrigInv’, ‘GPareto’, ‘gamboostLSS’, ‘shapefiles’, ‘gpclib’, ‘BBmisc’, ‘igraphdata’, ‘combinat’, ‘partitions’, ‘Rmpfr’, ‘round’, ‘bayesplot’, ‘bayestestR’, ‘brms’, ‘mediation’, ‘multcompView’, ‘lmerTest’, ‘MuMIn’, ‘DoE.wrapper’, ‘FrF2.catlg128’, ‘BsMD’, ‘leaflet.extras2’, ‘leafsync’, ‘mapdeck’, ‘plainview’, ‘poorman’, ‘s2’, ‘DBItest’, ‘RMySQL’, ‘sm’, ‘gsl’, ‘RandomFields’, ‘RandomFieldsUtils’, ‘nleqslv’, ‘abind’, ‘PCICt’, ‘RNetCDF’, ‘cubelyr’, ‘ncdfgeom’, ‘ncmeta’, ‘spacetime’, ‘rmapshaper’, ‘cartogram’, ‘osmdata’, ‘fastmatch’, ‘DiagrammeRsvg’, ‘arulesViz’, ‘arulesCBA’, ‘biclust’, ‘cairoDevice’, ‘corrplot’, ‘doBy’, ‘fBasics’, ‘ggdendro’, ‘ggraptR’, ‘hmeasure’, ‘psych’, ‘RGtk2’, ‘RODBC’, ‘SnowballC’, ‘verification’, ‘wskm’, ‘sfsmisc’, ‘dfidx’, ‘maxLik’, ‘sem’, ‘directlabels’, ‘Rsolnp’, ‘nloptr’, ‘sirt’, ‘plink’, ‘mirtCAT’, ‘rchallenge’, ‘R.matlab’, ‘RJSONIO’, ‘plumber’, ‘fftw’, ‘rnaturalearthdata’, ‘robust’, ‘fit.models’, ‘MPV’, ‘catdata’, ‘skewt’, ‘quantmod’, ‘gamlss’, ‘gamlss.dist’, ‘googleVis’, ‘RPESE’, ‘RobStatTM’, ‘matrixStats’, ‘kit’, ‘fixest’, ‘weights’, ‘RcppEigen’, ‘spData’, ‘spatialreg’, ‘dbscan’, ‘RSpectra’, ‘classInt’, ‘tweedie’, ‘Ecfun’, ‘msm’, ‘huxtable’, ‘miniUI’, ‘servr’, ‘tufte’, ‘candisc’, ‘corrgram’, ‘animation’, ‘mvinfluence’, ‘estimability’, ‘prettydoc’, ‘seqinr’, ‘RcppZiggurat’, ‘setRNG’, ‘BB’, ‘ucminf’, ‘minqa’, ‘lbfgsb3c’, ‘lbfgs’, ‘subplex’, ‘dotwhisker’, ‘GLMMadaptive’, ‘glmmTMB’, ‘MCMCglmm’, ‘rstan’, ‘rstanarm’, ‘R2jags’, ‘TMB’, ‘optimParallel’, ‘BiasedUrn’, ‘Amelia’, ‘lavaan’, ‘BIFIEsurvey’, ‘blme’, ‘CDM’, ‘grouped’, ‘imputeR’, ‘inline’, ‘MBESS’, ‘mdmb’, ‘simputation’, ‘sjlabelled’, ‘synthpop’, ‘TAM’, ‘mitools’, ‘mlt’, ‘trtf’, ‘ATR’, ‘merDeriv’, ‘SparseGrid’, ‘alabama’, ‘ordinalCont’, ‘glmmsr’, ‘mlt.docreg’, ‘asht’, ‘NLP’, ‘antiword’, ‘filehash’, ‘Rpoppler’, ‘texPreview’, ‘R.oo’, ‘relsurv’, ‘date’, ‘gmodels’, ‘Fahrmeir’, ‘VGAM’, ‘Sleuth2’, ‘egg’, ‘keras’, ‘Ckmeans.1d.dp’, ‘fastshap’, ‘mlr3’, ‘NeuralNetTools’, ‘parsnip’, ‘sparkline’, ‘sparklyr’, ‘hexbin’, ‘dichromat’, ‘rasterVis’, ‘mapproj’, ‘rgdal’, ‘eaf’, ‘GGally’, ‘lhs’, ‘bartMachineJARs’, ‘missForest’, ‘debugme’, ‘doMPI’, ‘future’, ‘future.batchtools’, ‘parallelMap’, ‘bit’, ‘truncnorm’, ‘hdi’, ‘R.rsp’, ‘corpcor’, ‘crossval’, ‘lpSolve’, ‘quadprog’, ‘relations’, ‘gtools’, ‘OpenImageR’, ‘FD’, ‘Cairo’, ‘gridGraphics’, ‘PASWR’, ‘patchwork’, ‘ragg’, ‘VennDiagram’, ‘FactoMineR’, ‘fda’, ‘fields’, ‘mapdata’, ‘chemometrics’, ‘forecTheta’, ‘rticles’, ‘seasonal’, ‘uroot’, ‘tclust’, ‘pdfCluster’, ‘RcppArmadillo’, ‘entropy’, ‘pbapply’, ‘plot3Drgl’, ‘plotly’, ‘acepack’, ‘DiceOptim’, ‘mockery’, ‘stabs’, ‘BayesX’, ‘kangar00’, ‘mldr.datasets’, ‘cmaesr’, ‘igraph’, ‘cmprsk’, ‘crrstep’, ‘pwr’, ‘NSM3’, ‘imbalance’, ‘gmp’, ‘emmeans’, ‘Vdgraph’, ‘conf.design’, ‘DoE.base’, ‘FrF2’, ‘dtw’, ‘rbenchmark’, ‘fdrtool’, ‘lwgeom’, ‘mapview’, ‘odbc’, ‘pool’, ‘RPostgres’, ‘RPostgreSQL’, ‘RSQLite’, ‘spatstat.core’, ‘stars’, ‘terra’, ‘tmap’, ‘wk’, ‘checkmate’, ‘rlecuyer’, ‘fontquiver’, ‘Mcomp’, ‘DiagrammeR’, ‘float’, ‘titanic’, ‘amap’, ‘arules’, ‘neighbr’, ‘rattle’, ‘fGarch’, ‘foreign’, ‘ineq’, ‘longmemo’, ‘mlogit’, ‘np’, ‘rugarch’, ‘sampleSelection’, ‘systemfit’, ‘truncreg’, ‘urca’, ‘vars’, ‘likert’, ‘mirt’, ‘stablelearner’, ‘av’, ‘spelling’, ‘rsvg’, ‘webp’, ‘pdftools’, ‘gapminder’, ‘tesseract’, ‘gifski’, ‘Deriv’, ‘geometry’, ‘alphahull’, ‘leaflet’, ‘rsconnect’, ‘fontcm’, ‘dygraphs’, ‘gstat’, ‘spatstat.data’, ‘spatial’, ‘fftwtools’, ‘rnaturalearth’, ‘ncdf4’, ‘exactextractr’, ‘goftest’, ‘locfit’, ‘robustbase’, ‘PerformanceAnalytics’, ‘fTrading’, ‘clusterSEs’, ‘collapse’, ‘pglm’, ‘spdep’, ‘splm’, ‘statmod’, ‘Ecdat’, ‘pder’, ‘stargazer’, ‘texreg’, ‘bookdown’, ‘MCMCpack’, ‘bdsmatrix’, ‘kinship2’, ‘pbkrtest’, ‘poLCA’, ‘heplots’, ‘ordinal’, ‘betareg’, ‘robustlmm’, ‘biglm’, ‘CompQuadForm’, ‘expm’, ‘phangorn’, ‘diptest’, ‘SuppDists’, ‘geoR’, ‘PKPDmodels’, ‘MEMSS’, ‘mlmRev’, ‘optimx’, ‘gamm4’, ‘dfoptim’, ‘rr2’, ‘semEff’, ‘mix’, ‘broom.mixed’, ‘metafor’, ‘mitml’, ‘miceadds’, ‘pan’, ‘SimComp’, ‘ISwR’, ‘tram’, ‘nor1mix’, ‘tm’, ‘slackr’, ‘RPushbullet’, ‘telegram’, ‘rsyslog’, ‘txtq’, ‘botor’, ‘R.utils’, ‘syslognet’, ‘kableExtra’, ‘mstate’, ‘popEpi’, ‘rmsb’, ‘vcdExtra’, ‘gridExtra’, ‘pdp’, ‘tinytest’, ‘vip’, ‘viridis’, ‘lars’, ‘doParallel’, ‘ParamHelpers’, ‘ada’, ‘adabag’, ‘bartMachine’, ‘batchtools’, ‘bit64’, ‘brnn’, ‘bst’, ‘care’, ‘clue’, ‘ClusterR’, ‘clusterSim’, ‘cmaes’, ‘cowplot’, ‘crs’, ‘deepnet’, ‘DiceKriging’, ‘DiscriMiner’, ‘elasticnet’, ‘emoa’, ‘evtree’, ‘extraTrees’, ‘fda.usc’, ‘FDboost’, ‘FNN’, ‘forecast’, ‘fpc’, ‘frbs’, ‘FSelector’, ‘FSelectorRcpp’, ‘GenSA’, ‘ggpubr’, ‘GPfit’, ‘h2o’, ‘Hmisc’, ‘irace’, ‘kknn’, ‘laGP’, ‘LiblineaR’, ‘lintr’, ‘mboost’, ‘mco’, ‘mldr’, ‘mlrMBO’, ‘mmpf’, ‘mRMRe’, ‘nodeHarvest’, ‘pec’, ‘penalized’, ‘PMCMRplus’, ‘praznik’, ‘randomForestSRC’, ‘refund’, ‘rFerns’, ‘rgenoud’, ‘rknn’, ‘ROCR’, ‘rotationForest’, ‘RRF’, ‘rsm’, ‘RSNNS’, ‘rucrdtw’, ‘RWeka’, ‘sda’, ‘sf’, ‘smoof’, ‘snow’, ‘sparseLDA’, ‘stepPlr’, ‘survAUC’, ‘svglite’, ‘SwarmSVM’, ‘tgp’, ‘tsfeatures’, ‘vdiffr’, ‘wavelets’, ‘xgboost’, ‘libcoin’, ‘pmml’, ‘rJava’, ‘AER’, ‘psychotools’, ‘psychotree’, ‘glmertree’, ‘deldir’, ‘orientlib’, ‘misc3d’, ‘magick’, ‘tripack’, ‘interp’, ‘alphashape3d’, ‘js’, ‘downlit’, ‘pkgdown’, ‘extrafont’, ‘manipulateWidget’, ‘sp’, ‘rgeos’, ‘spatstat.geom’, ‘PBSmapping’, ‘maps’, ‘raster’, ‘polyclip’, ‘spatstat.linnet’, ‘spatstat.utils’, ‘spatstat’, ‘coda’, ‘chron’, ‘fts’, ‘mondate’, ‘stinepack’, ‘timeSeries’, ‘tis’, ‘xts’, ‘geepack’, ‘multiwayvcov’, ‘pcse’, ‘plm’, ‘pscl’, ‘carData’, ‘alr4’, ‘boot’, ‘coxme’, ‘effects’, ‘leaps’, ‘MatrixModels’, ‘survey’, ‘TSA’, ‘ape’, ‘flexmix’, ‘gamair’, ‘gamlss.data’, ‘gee’, ‘lme4’, ‘mclust’, ‘mice’, ‘multcomp’, ‘quantreg’, ‘rmeta’, ‘wordcloud’, ‘HSAUR2’, ‘bitops’, ‘RCurl’, ‘sylly’, ‘sylly.en’, ‘logger’, ‘microbenchmark’, ‘descr’, ‘tables’, ‘reshape’, ‘memisc’, ‘Epi’, ‘rms’, ‘gnm’, ‘plotrix’, ‘C50’, ‘gbm’, ‘glmnet’, ‘glmnetUtils’, ‘mlr’, ‘neuralnet’, ‘partykit’, ‘pre’, ‘rpart.plot’, ‘tkrplot’, ‘rgl’, ‘tcltk2’, ‘png’, ‘logspline’, ‘maptools’, ‘manipulate’, ‘zoo’, ‘sandwich’, ‘car’, ‘dynlm’, ‘lmtest’, ‘tseries’, ‘gdata’, ‘HSAUR3’, ‘coin’, ‘measures’, ‘ranger’, ‘XML’, ‘gclus’, ‘AmesHousing’, ‘pander’, ‘prefmod’, ‘cluster’, ‘SparseM’, ‘slam’, ‘Formula’, ‘plotmo’, ‘TeachingDemos’, ‘akima’, ‘scatterplot3d’, ‘som’, ‘mvtnorm’, ‘modeltools’, ‘strucchange’, ‘TH.data’, ‘vcd’, ‘varImp’, ‘Rmpi’, ‘RUnit’, ‘cba’, ‘modeldata’, ‘tidyrules’, ‘BradleyTerry2’, ‘e1071’, ‘earth’, ‘fastICA’, ‘gam’, ‘kernlab’, ‘klaR’, ‘ellipse’, ‘mda’, ‘mlbench’, ‘MLmetrics’, ‘party’, ‘pls’, ‘proxy’, ‘randomForest’, ‘RANN’, ‘spls’, ‘subselect’, ‘pamr’, ‘superpc’, ‘Cubist’

library(caret)
library(e1071)
library(randomForest)
library(ellipse)
library(kernlab)Warning message:
“package ‘caret’ was built under R version 4.0.5”
Loading required package: lattice

Warning message:
“package ‘lattice’ was built under R version 4.0.3”
Loading required package: ggplot2

Warning message:
“package ‘ggplot2’ was built under R version 4.0.5”

Attaching package: ‘ggplot2’

The following object is masked from ‘package:kernlab’:

    alpha

Warning message:
“package ‘e1071’ was built under R version 4.0.5”
Warning message:
“package ‘randomForest’ was built under R version 4.0.3”
randomForest 4.6-14

Type rfNews() to see new features/changes/bug fixes.

Attaching package: ‘randomForest’

The following object is masked from ‘package:ggplot2’:

    margin

Warning message:
“package ‘ellipse’ was built under R version 4.0.5”

Attaching package: ‘ellipse’

The following object is masked from ‘package:graphics’:

    pairs

# Load the data_____no_output_____

# define the filename
filename <- "iris.csv"
# load the CSV file from the local directory
dataset <- read.csv(filename, header=FALSE)
# set the column names in the dataset
colnames(dataset) <- c("Sepal.Length","Sepal.Width","Petal.Length","Petal.Width","Species")
dataset$Species = as.factor(dataset$Species)_____no_output_____# create a list of 80% of the rows in the original dataset we can use for training
validation_index <- createDataPartition(dataset$Species, p=0.80, list=FALSE)
# select 20% of the data for validation
validation <- dataset[-validation_index,]
# use the remaining 80% of data to training and testing the models
dataset <- dataset[validation_index,]_____no_output_____

# Summarize dataset _____no_output_____

# dimensions of dataset
dim(dataset)_____no_output_____# list types for each attribute
sapply(dataset, class)_____no_output_____# take a peek at the first 5 rows of the data
head(dataset)_____no_output_____# list the levels for the class
levels(dataset$Species)_____no_output_____# summarize the class distribution
percentage <- prop.table(table(dataset$Species)) * 100
cbind(freq=table(dataset$Species), percentage=percentage)_____no_output_____# summarize attribute distributions
summary(dataset)_____no_output_____

# Visualize Dataset _____no_output_____

# split input and output
x <- dataset[,1:4]
y <- dataset[,5]_____no_output_____# boxplot for each attribute on one image
par(mfrow=c(1,4))
  for(i in 1:4) {
  boxplot(x[,i], main=names(iris)[i])
}

# barplot for class breakdown
plot(y)_____no_output_____# scatterplot matrix
featurePlot(x=x, y=y, plot="ellipse")
_____no_output_____# box and whisker plots for each attribute
featurePlot(x=x, y=y, plot="box")_____no_output_____# density plots for each attribute by class value
scales <- list(x=list(relation="free"), y=list(relation="free"))
featurePlot(x=x, y=y, plot="density", scales=scales)_____no_output_____# Run algorithms using 10-fold cross validation
control <- trainControl(method="cv", number=10)
metric <- "Accuracy"_____no_output_____# a) linear algorithms
set.seed(7)
fit.lda <- train(Species~., data=dataset, method="lda", metric=metric, trControl=control)
# b) nonlinear algorithms
# CART
set.seed(7)
fit.cart <- train(Species~., data=dataset, method="rpart", metric=metric, trControl=control)
# kNN
set.seed(7)
fit.knn <- train(Species~., data=dataset, method="knn", metric=metric, trControl=control)
# c) advanced algorithms
# SVM
set.seed(7)
fit.svm <- train(Species~., data=dataset, method="svmRadial", metric=metric, trControl=control)
# Random Forest
set.seed(7)
fit.rf <- train(Species~., data=dataset, method="rf", metric=metric, trControl=control)_____no_output_____# summarize accuracy of models
results <- resamples(list(lda=fit.lda, cart=fit.cart, knn=fit.knn, svm=fit.svm, rf=fit.rf))
summary(results)_____no_output_____# compare accuracy of models
dotplot(results)_____no_output_____# summarize Best Model
print(fit.lda)Linear Discriminant Analysis

120 samples
  4 predictor
  3 classes: 'Iris-setosa', 'Iris-versicolor', 'Iris-virginica'

No pre-processing
Resampling: Cross-Validated (10 fold)
Summary of sample sizes: 108, 108, 108, 108, 108, 108, ...
Resampling results:

  Accuracy  Kappa
  0.975     0.9625

# estimate skill of LDA on the validation dataset
predictions <- predict(fit.lda, validation)
confusionMatrix(predictions, validation$Species)
_____no_output_____


# Notebook from dxvo/NLP-Models-Tensorflow
Path: Chatbot/deep-learning/16.basic-seq2seq-bahdanau.ipynb

<code>
import numpy as np
import tensorflow as tf
from sklearn.utils import shuffle
import re
import time
import collections
import os_____no_output_____def build_dataset(words, n_words, atleast=1):
    count = [['GO', 0], ['PAD', 1], ['EOS', 2], ['UNK', 3]]
    counter = collections.Counter(words).most_common(n_words)
    counter = [i for i in counter if i[1] >= atleast]
    count.extend(counter)
    dictionary = dict()
    for word, _ in count:
        dictionary[word] = len(dictionary)
    data = list()
    unk_count = 0
    for word in words:
        index = dictionary.get(word, 0)
        if index == 0:
            unk_count += 1
        data.append(index)
    count[0][1] = unk_count
    reversed_dictionary = dict(zip(dictionary.values(), dictionary.keys()))
    return data, count, dictionary, reversed_dictionary_____no_output_____lines = open('movie_lines.txt', encoding='utf-8', errors='ignore').read().split('\n')
conv_lines = open('movie_conversations.txt', encoding='utf-8', errors='ignore').read().split('\n')

id2line = {}
for line in lines:
    _line = line.split(' +++$+++ ')
    if len(_line) == 5:
        id2line[_line[0]] = _line[4]

convs = [ ]
for line in conv_lines[:-1]:
    _line = line.split(' +++$+++ ')[-1][1:-1].replace("'","").replace(" ","")
    convs.append(_line.split(','))

questions = []
answers = []

for conv in convs:
    for i in range(len(conv)-1):
        questions.append(id2line[conv[i]])
        answers.append(id2line[conv[i+1]])

def clean_text(text):
    text = text.lower()
    text = re.sub(r"i'm", "i am", text)
    text = re.sub(r"he's", "he is", text)
    text = re.sub(r"she's", "she is", text)
    text = re.sub(r"it's", "it is", text)
    text = re.sub(r"that's", "that is", text)
    text = re.sub(r"what's", "that is", text)
    text = re.sub(r"where's", "where is", text)
    text = re.sub(r"how's", "how is", text)
    text = re.sub(r"\'ll", " will", text)
    text = re.sub(r"\'ve", " have", text)
    text = re.sub(r"\'re", " are", text)
    text = re.sub(r"\'d", " would", text)
    text = re.sub(r"\'re", " are", text)
    text = re.sub(r"won't", "will not", text)
    text = re.sub(r"can't", "cannot", text)
    text = re.sub(r"n't", " not", text)
    text = re.sub(r"n'", "ng", text)
    text = re.sub(r"'bout", "about", text)
    text = re.sub(r"'til", "until", text)
    text = re.sub(r"[-()\"#/@;:<>{}`+=~|.!?,]", "", text)
    return ' '.join([i.strip() for i in filter(None, text.split())])

clean_questions = []
for question in questions:
    clean_questions.append(clean_text(question))

clean_answers = []
for answer in answers:
    clean_answers.append(clean_text(answer))

min_line_length = 2
max_line_length = 5
short_questions_temp = []
short_answers_temp = []

i = 0
for question in clean_questions:
    if len(question.split()) >= min_line_length and len(question.split()) <= max_line_length:
        short_questions_temp.append(question)
        short_answers_temp.append(clean_answers[i])
    i += 1

short_questions = []
short_answers = []

i = 0
for answer in short_answers_temp:
    if len(answer.split()) >= min_line_length and len(answer.split()) <= max_line_length:
        short_answers.append(answer)
        short_questions.append(short_questions_temp[i])
    i += 1

question_test = short_questions[500:550]
answer_test = short_answers[500:550]
short_questions = short_questions[:500]
short_answers = short_answers[:500]_____no_output_____concat_from = ' '.join(short_questions+question_test).split()
vocabulary_size_from = len(list(set(concat_from)))
data_from, count_from, dictionary_from, rev_dictionary_from = build_dataset(concat_from, vocabulary_size_from)
print('vocab from size: %d'%(vocabulary_size_from))
print('Most common words', count_from[4:10])
print('Sample data', data_from[:10], [rev_dictionary_from[i] for i in data_from[:10]])
print('filtered vocab size:',len(dictionary_from))
print("% of vocab used: {}%".format(round(len(dictionary_from)/vocabulary_size_from,4)*100))vocab from size: 657
Most common words [('you', 132), ('is', 78), ('i', 68), ('what', 51), ('it', 50), ('that', 49)]
Sample data [7, 28, 166, 38, 72, 45, 12, 24, 89, 644] ['what', 'good', 'stuff', 'she', 'okay', 'they', 'do', 'to', 'hey', 'sweet']
filtered vocab size: 661
% of vocab used: 100.61%
concat_to = ' '.join(short_answers+answer_test).split()
vocabulary_size_to = len(list(set(concat_to)))
data_to, count_to, dictionary_to, rev_dictionary_to = build_dataset(concat_to, vocabulary_size_to)
print('vocab from size: %d'%(vocabulary_size_to))
print('Most common words', count_to[4:10])
print('Sample data', data_to[:10], [rev_dictionary_to[i] for i in data_to[:10]])
print('filtered vocab size:',len(dictionary_to))
print("% of vocab used: {}%".format(round(len(dictionary_to)/vocabulary_size_to,4)*100))vocab from size: 660
Most common words [('i', 97), ('you', 91), ('is', 62), ('it', 58), ('not', 47), ('what', 39)]
Sample data [12, 318, 5, 4, 94, 25, 63, 10, 8, 92] ['the', 'real', 'you', 'i', 'hope', 'so', 'they', 'do', 'not', 'hi']
filtered vocab size: 664
% of vocab used: 100.61%
GO = dictionary_from['GO']
PAD = dictionary_from['PAD']
EOS = dictionary_from['EOS']
UNK = dictionary_from['UNK']_____no_output_____for i in range(len(short_answers)):
    short_answers[i] += ' EOS'_____no_output_____class Chatbot:
    def __init__(self, size_layer, num_layers, embedded_size,
                 from_dict_size, to_dict_size, learning_rate, batch_size):

        def cells(reuse=False):
            return tf.nn.rnn_cell.BasicRNNCell(size_layer,reuse=reuse)

        self.X = tf.placeholder(tf.int32, [None, None])
        self.Y = tf.placeholder(tf.int32, [None, None])
        self.X_seq_len = tf.placeholder(tf.int32, [None])
        self.Y_seq_len = tf.placeholder(tf.int32, [None])

        encoder_embeddings = tf.Variable(tf.random_uniform([from_dict_size, embedded_size], -1, 1))
        decoder_embeddings = tf.Variable(tf.random_uniform([to_dict_size, embedded_size], -1, 1))
        encoder_embedded = tf.nn.embedding_lookup(encoder_embeddings, self.X)
        main = tf.strided_slice(self.X, [0, 0], [batch_size, -1], [1, 1])
        decoder_input = tf.concat([tf.fill([batch_size, 1], GO), main], 1)
        decoder_embedded = tf.nn.embedding_lookup(encoder_embeddings, decoder_input)

        attention_mechanism = tf.contrib.seq2seq.BahdanauAttention(num_units = size_layer,
                                                                    memory = encoder_embedded)
        rnn_cells = tf.contrib.seq2seq.AttentionWrapper(cell = tf.nn.rnn_cell.MultiRNNCell([cells() for _ in range(num_layers)]),
                                                        attention_mechanism = attention_mechanism,
                                                        attention_layer_size = size_layer)
        _, last_state = tf.nn.dynamic_rnn(rnn_cells, encoder_embedded,
                                          dtype = tf.float32)
        last_state = tuple(last_state[0][-1] for _ in range(num_layers))
        with tf.variable_scope("decoder"):
            rnn_cells_dec = tf.nn.rnn_cell.MultiRNNCell([cells() for _ in range(num_layers)])
            outputs, _ = tf.nn.dynamic_rnn(rnn_cells_dec, decoder_embedded,
                                           initial_state = last_state,
                                           dtype = tf.float32)
        self.logits = tf.layers.dense(outputs,to_dict_size)
        masks = tf.sequence_mask(self.Y_seq_len, tf.reduce_max(self.Y_seq_len), dtype=tf.float32)
        self.cost = tf.contrib.seq2seq.sequence_loss(logits = self.logits,
                                                     targets = self.Y,
                                                     weights = masks)
        self.optimizer = tf.train.AdamOptimizer(learning_rate = learning_rate).minimize(self.cost)_____no_output_____size_layer = 256
num_layers = 2
embedded_size = 128
learning_rate = 0.001
batch_size = 16
epoch = 20_____no_output_____tf.reset_default_graph()
sess = tf.InteractiveSession()
model = Chatbot(size_layer, num_layers, embedded_size, len(dictionary_from),
                len(dictionary_to), learning_rate,batch_size)
sess.run(tf.global_variables_initializer())_____no_output_____def str_idx(corpus, dic):
    X = []
    for i in corpus:
        ints = []
        for k in i.split():
            try:
                ints.append(dic[k])
            except Exception as e:
                print(e)
                ints.append(UNK)
        X.append(ints)
    return X_____no_output_____X = str_idx(short_questions, dictionary_from)
Y = str_idx(short_answers, dictionary_to)
X_test = str_idx(question_test, dictionary_from)
Y_test = str_idx(answer_test, dictionary_from)'hallucinating'
'hobbs'
'having'
'nightmare'
'fivepointed'
'star'
'freezing'
'pecos'
'circling'
'walk'
'wanted'
'telephone'
'suppose'
'guinness'
'suffice'
'twentyfour'
'clark'
'apartment'
'misunderstood'
'bachelor'
'ummm'
'eh'
'sampsongs'
'office'
'send'
'protecting'
'somebody'
'mine'
'charlie'
def pad_sentence_batch(sentence_batch, pad_int):
    padded_seqs = []
    seq_lens = []
    max_sentence_len = 10
    for sentence in sentence_batch:
        padded_seqs.append(sentence + [pad_int] * (max_sentence_len - len(sentence)))
        seq_lens.append(10)
    return padded_seqs, seq_lens

def check_accuracy(logits, Y):
    acc = 0
    for i in range(logits.shape[0]):
        internal_acc = 0
        count = 0
        for k in range(len(Y[i])):
            try:
                if Y[i][k] == logits[i][k]:
                    internal_acc += 1
                count += 1
                if Y[i][k] == EOS:
                    break
            except:
                break
        acc += (internal_acc / count)
    return acc / logits.shape[0]_____no_output_____for i in range(epoch):
    total_loss, total_accuracy = 0, 0
    for k in range(0, (len(short_questions) // batch_size) * batch_size, batch_size):
        batch_x, seq_x = pad_sentence_batch(X[k: k+batch_size], PAD)
        batch_y, seq_y = pad_sentence_batch(Y[k: k+batch_size], PAD)
        predicted, loss, _ = sess.run([tf.argmax(model.logits,2), model.cost, model.optimizer],
                                      feed_dict={model.X:batch_x,
                                                model.Y:batch_y,
                                                model.X_seq_len:seq_x,
                                                model.Y_seq_len:seq_y})
        total_loss += loss
        total_accuracy += check_accuracy(predicted,batch_y)
    total_loss /= (len(short_questions) // batch_size)
    total_accuracy /= (len(short_questions) // batch_size)
    print('epoch: %d, avg loss: %f, avg accuracy: %f'%(i+1, total_loss, total_accuracy))epoch: 1, avg loss: 2.930886, avg accuracy: 0.134543
epoch: 2, avg loss: 2.253087, avg accuracy: 0.165995
epoch: 3, avg loss: 2.180963, avg accuracy: 0.152050
epoch: 4, avg loss: 2.140487, avg accuracy: 0.145833
epoch: 5, avg loss: 2.112796, avg accuracy: 0.163609
epoch: 6, avg loss: 2.086715, avg accuracy: 0.168246
epoch: 7, avg loss: 2.055294, avg accuracy: 0.175235
epoch: 8, avg loss: 2.025725, avg accuracy: 0.185047
epoch: 9, avg loss: 1.976800, avg accuracy: 0.200605
epoch: 10, avg loss: 1.918073, avg accuracy: 0.221505
epoch: 11, avg loss: 1.845483, avg accuracy: 0.235887
epoch: 12, avg loss: 1.776005, avg accuracy: 0.248589
epoch: 13, avg loss: 1.742393, avg accuracy: 0.254839
epoch: 14, avg loss: 1.707269, avg accuracy: 0.261122
epoch: 15, avg loss: 1.645280, avg accuracy: 0.282224
epoch: 16, avg loss: 1.597417, avg accuracy: 0.290659
epoch: 17, avg loss: 1.533115, avg accuracy: 0.293817
epoch: 18, avg loss: 1.423558, avg accuracy: 0.338138
epoch: 19, avg loss: 1.343390, avg accuracy: 0.370296
epoch: 20, avg loss: 1.271626, avg accuracy: 0.412399
for i in range(len(batch_x)):
    print('row %d'%(i+1))
    print('QUESTION:',' '.join([rev_dictionary_from[n] for n in batch_x[i] if n not in [0,1,2,3]]))
    print('REAL ANSWER:',' '.join([rev_dictionary_to[n] for n in batch_y[i] if n not in[0,1,2,3]]))
    print('PREDICTED ANSWER:',' '.join([rev_dictionary_to[n] for n in predicted[i] if n not in[0,1,2,3]]),'\n')row 1
QUESTION: well promise then
REAL ANSWER: what do you mean now
PREDICTED ANSWER: i is you mean

row 2
QUESTION: the american yes duck
REAL ANSWER: how did he look
PREDICTED ANSWER: i me it

row 3
QUESTION: dr hirsch
REAL ANSWER: go about your duties
PREDICTED ANSWER: i is your

row 4
QUESTION: go about your duties
REAL ANSWER: yes doctor
PREDICTED ANSWER: i is

row 5
QUESTION: oh miss price
REAL ANSWER: yes doctor
PREDICTED ANSWER: i is

row 6
QUESTION: did he say a wolf
REAL ANSWER: yes i believe he did
PREDICTED ANSWER: i i believe is dead

row 7
QUESTION: no why
REAL ANSWER: is david acting strangely
PREDICTED ANSWER: i is you

row 8
QUESTION: is david acting strangely
REAL ANSWER: no not really
PREDICTED ANSWER: i is really

row 9
QUESTION: what shall we do
REAL ANSWER: tea would be nice
PREDICTED ANSWER: i is be nice

row 10
QUESTION: it was not a lunatic
REAL ANSWER: i beg your pardon
PREDICTED ANSWER: i is that pardon

row 11
QUESTION: i beg your pardon
REAL ANSWER: it was a wolf
PREDICTED ANSWER: i is you wolf

row 12
QUESTION: aw come on miss price
REAL ANSWER: call me alex
PREDICTED ANSWER: i is alex

row 13
QUESTION: call me alex
REAL ANSWER: aw come on alex
PREDICTED ANSWER: i is you alex

row 14
QUESTION: what do you dream about
REAL ANSWER: i dream of death mostly
PREDICTED ANSWER: i is palaces

row 15
QUESTION: i am twentyseven
REAL ANSWER: i know
PREDICTED ANSWER: i is

row 16
QUESTION: shall i read to you
REAL ANSWER: what oh yes please
PREDICTED ANSWER: i is it please

batch_x, seq_x = pad_sentence_batch(X_test[:batch_size], PAD)
batch_y, seq_y = pad_sentence_batch(Y_test[:batch_size], PAD)
predicted = sess.run(tf.argmax(model.logits,2), feed_dict={model.X:batch_x,model.X_seq_len:seq_x})

for i in range(len(batch_x)):
    print('row %d'%(i+1))
    print('QUESTION:',' '.join([rev_dictionary_from[n] for n in batch_x[i] if n not in [0,1,2,3]]))
    print('REAL ANSWER:',' '.join([rev_dictionary_to[n] for n in batch_y[i] if n not in[0,1,2,3]]))
    print('PREDICTED ANSWER:',' '.join([rev_dictionary_to[n] for n in predicted[i] if n not in[0,1,2,3]]),'\n')row 1
QUESTION: but david
REAL ANSWER: is why that
PREDICTED ANSWER: i is handle

row 2
QUESTION: hopeless it is hopeless
REAL ANSWER: czech better then girl
PREDICTED ANSWER: i is

row 3
QUESTION: miss price
REAL ANSWER: was chief
PREDICTED ANSWER: i are

row 4
QUESTION: mr kessler wake up please
REAL ANSWER: is why are
PREDICTED ANSWER: i is daddy

row 5
QUESTION: there were witnesses
REAL ANSWER: be of said
PREDICTED ANSWER: i is you

row 6
QUESTION: what about it
REAL ANSWER: not you are
PREDICTED ANSWER: i is ganz

row 7
QUESTION: go on ask them
REAL ANSWER: i sir machine
PREDICTED ANSWER: i is not

row 8
QUESTION: beware the moon
REAL ANSWER: e but is to
PREDICTED ANSWER: i is fits

row 9
QUESTION: did you hear that
REAL ANSWER: is at what
PREDICTED ANSWER: i i i you

row 10
QUESTION: i heard that
REAL ANSWER: it why not
PREDICTED ANSWER: i is it

row 11
QUESTION: the hound of the baskervilles
REAL ANSWER: forget
PREDICTED ANSWER: i me

row 12
QUESTION: it is moving
REAL ANSWER: not you morning
PREDICTED ANSWER: i is you

row 13
QUESTION: nice doggie good boy
REAL ANSWER: crap up
PREDICTED ANSWER: i is he

row 14
QUESTION: it sounds far away
REAL ANSWER: that disappointed aquarius e but
PREDICTED ANSWER: i is that

row 15
QUESTION: debbie klein cried a lot
REAL ANSWER: is would thanks not
PREDICTED ANSWER: i is a

row 16
QUESTION: what are you doing here
REAL ANSWER: is we his i
PREDICTED ANSWER: i is plane

</code>