# Notebook from junelsolis/ZeroCostDL4Mic
Path: Colab_notebooks/Beta notebooks/MaskRCNN_ZeroCostDL4Mic.ipynb

# **MaskRCNN**

---

 This notebook is an implementation of MaskRCNN. This neural network performs instance segmentation. This means it can be used to detect objects in images, segment these objects and classify them. This notebook is based on the work of [He et al.](https://arxiv.org/abs/1703.06870)

---

*Disclaimer*:

This notebook is part of the *Zero-Cost Deep-Learning to Enhance Microscopy* project (ZeroCostDL4Mic) (https://github.com/HenriquesLab/DeepLearning_Collab/wiki)

This notebook is based on the following paper:

**Mask R-CNN**, arxiv, 2018 by Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick [here](https://arxiv.org/abs/1703.06870)

And source code found in: *https://github.com/matterport/Mask_RCNN*

Provide information on dataset availability and link for download if applicable.

**Please also cite this original paper when using or developing this notebook.**_____no_output_____# **How to use this notebook?**

---

Video describing how to use ZeroCostDL4Mic notebooks are available on youtube:
  - [**Video 1**](https://www.youtube.com/watch?v=GzD2gamVNHI&feature=youtu.be): Full run through of the workflow to obtain the notebooks and the provided test datasets as well as a common use of the notebook
  - [**Video 2**](https://www.youtube.com/watch?v=PUuQfP5SsqM&feature=youtu.be): Detailed description of the different sections of the notebook

---
###**Structure of a notebook**

The notebook contains two types of cell:

**Text cells** provide information and can be modified by douple-clicking the cell. You are currently reading the text cell. You can create a new text by clicking `+ Text`.

**Code cells** contain code and the code can be modfied by selecting the cell. To execute the cell, move your cursor on the `[ ]`-mark on the left side of the cell (play button appears). Click to execute the cell. After execution is done the animation of play button stops. You can create a new coding cell by clicking `+ Code`.

---
###**Table of contents, Code snippets** and **Files**

On the top left side of the notebook you find three tabs which contain from top to bottom:

*Table of contents* = contains structure of the notebook. Click the content to move quickly between sections.

*Code snippets* = contain examples how to code certain tasks. You can ignore this when using this notebook.

*Files* = contain all available files. After mounting your google drive (see section 1.) you will find your files and folders here.

**Remember that all uploaded files are purged after changing the runtime.** All files saved in Google Drive will remain. You do not need to use the Mount Drive-button; your Google Drive is connected in section 1.2.

**Note:** The "sample data" in "Files" contains default files. Do not upload anything in here!

---
###**Making changes to the notebook**

**You can make a copy** of the notebook and save it to your Google Drive. To do this click file -> save a copy in drive.

To **edit a cell**, double click on the text. This will show you either the source code (in code cells) or the source text (in text cells).
You can use the `#`-mark in code cells to comment out parts of the code. This allows you to keep the original code piece in the cell as a comment._____no_output_____#**0. Before getting started**
---
**We strongly recommend that you generate extra paired images. These images can be used to assess the quality of your trained model (Quality control dataset)**. The quality control assessment can be done directly in this notebook.

 **Additionally, the corresponding input and output files need to have the same name**.

 Please note that while the file format is flexible (.tif, .png, .jpeg should all work) but these currently **must be of RGB** type.

Here's the data structure that you should use:
*   Experiment A
    - **Training dataset**
      - Training
        - img_1.png, img_1.png.csv, img_2.png, img_2.png.csv, ...
      - Validation
        - img_a.png, img_a.png.csv, img_b.png, img_b.png.csv,...
    - **Quality control dataset**
      - Validation
        - img_a.png, img_a.png.csv, img_b.png, img_b.png.csv
    - **Data to be predicted**
    - **Results**

---

 **Note: This notebook is still in the beta stage.
Currently, the notebook works only if the annotation files are in csv format with the following columns:**

***| filename | width | height | object_index | class_name | x | y |***

where each row in the csv will provide the coordinates **(x,y)** of an edge point in the segmentation mask of an individual object with a dedicated **object_index** (e.g. 1, 2, 3....) and its **class_name** (e.g. 'nucleus' or 'horse' etc.) on the image of dimensions **width** x **height** (pixels). If you already have a dataset with segmentation masks we can provide a fiji macro that can convert the dataset into the correct format.
*We are actively working on integrating more flexibility into the annotations this notebook can be used with.*

---

**Important note**

- If you wish to **Train a network from scratch** using your own dataset (and we encourage everyone to do that), you will need to run **sections 1 - 4**, then use **section 5** to assess the quality of your model and **section 6** to run predictions using the model that you trained.

- If you wish to **Evaluate your model** using a model previously generated and saved on your Google Drive, you will only need to run **sections 1 and 2** to set up the notebook, then use **section 5** to assess the quality of your model.

- If you only wish to **run predictions** using a model previously generated and saved on your Google Drive, you will only need to run **sections 1 and 2** to set up the notebook, then use **section 6** to run the predictions on the desired model.
---_____no_output_____# **1. Install MaskRCNN and dependencies**
---
_____no_output_____## **1.1. Install key dependencies**
---
 _____no_output_____

#@markdown ##Install MaskRCNN and dependencies
!pip install fpdf
!pip install imgaug
!pip install h5py==2.10
!git clone https://github.com/matterport/Mask_RCNN

#Force session restart
exit(0)_____no_output_____

## **1.2. Restart your runtime**
---

** Ignore the following message error message. Your Runtime has automatically restarted. This is normal.**

_____no_output_____## **1.3. Load key dependencies**
---
 _____no_output_____

Notebook_version = '1.13'
Network = 'MaskRCNN'

from builtins import any as b_any

def get_requirements_path():
    # Store requirements file in 'contents' directory
    current_dir = os.getcwd()
    dir_count = current_dir.count('/') - 1
    path = '../' * (dir_count) + 'requirements.txt'
    return path

def filter_files(file_list, filter_list):
    filtered_list = []
    for fname in file_list:
        if b_any(fname.split('==')[0] in s for s in filter_list):
            filtered_list.append(fname)
    return filtered_list

def build_requirements_file(before, after):
    path = get_requirements_path()

    # Exporting requirements.txt for local run
    !pip freeze > $path

    # Get minimum requirements file
    df = pd.read_csv(path, delimiter = "\n")
    mod_list = [m.split('.')[0] for m in after if not m in before]
    req_list_temp = df.values.tolist()
    req_list = [x[0] for x in req_list_temp]

    # Replace with package name and handle cases where import name is different to module name
    mod_name_list = [['sklearn', 'scikit-learn'], ['skimage', 'scikit-image']]
    mod_replace_list = [[x[1] for x in mod_name_list] if s in [x[0] for x in mod_name_list] else s for s in mod_list]
    filtered_list = filter_files(req_list, mod_replace_list)

    file=open(path,'w')
    for item in filtered_list:
        file.writelines(item + '\n')

    file.close()

import sys
before = [str(m) for m in sys.modules]

#@markdown ##Load Key Dependencies
%tensorflow_version 1.x

import os
import sys
import json
import datetime
import time
import numpy as np
import skimage.draw
from skimage import io
import imgaug
import pandas as pd
import csv
import random
import datetime
import shutil
from matplotlib import pyplot as plt
import matplotlib.lines as lines
from matplotlib.patches import Polygon
import IPython.display
from PIL import Image, ImageDraw, ImageFont
from fpdf import FPDF, HTMLMixin
from pip._internal.operations.freeze import freeze
import subprocess as sp

# Root directory of the project
ROOT_DIR = os.path.abspath("/content")
# !git clone https://github.com/matterport/Mask_RCNN
# Import Mask RCNN
sys.path.append(ROOT_DIR)  # To find local version of the library
os.chdir('/content/Mask_RCNN')

#Here we need to replace "self.keras_model.metrics_tensors.append(loss)" with "self.keras_model.add_metric(loss, name)"
# in model.py line 2199, otherwise we get version issues.
from tempfile import mkstemp
from shutil import move, copymode
from os import fdopen, remove
#This function replaces the old default files with new values
def replace(file_path, pattern, subst):
    #Create temp file
    fh, abs_path = mkstemp()
    with fdopen(fh,'w') as new_file:
        with open(file_path) as old_file:
            for line in old_file:
                new_file.write(line.replace(pattern, subst))
    #Copy the file permissions from the old file to the new file
    copymode(file_path, abs_path)
    #Remove original file
    remove(file_path)
    #Move new file
    move(abs_path, file_path)

replace("/content/Mask_RCNN/mrcnn/model.py",'self.keras_model.metrics_tensors.append(loss)','self.keras_model.add_metric(loss, name)')
#replace("/content/Mask_RCNN/mrcnn/model.py", "save_weights_only=True),", "save_weights_only=True),\n            keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor = 0.1, patience = 30, min_lr = 0, verbose = 1)")
#replace("/content/Mask_RCNN/mrcnn/model.py", "save_weights_only=True),", "save_weights_only=True),\n            keras.callbacks.CSVLogger('/content/results.csv'),")
replace("/content/Mask_RCNN/mrcnn/model.py",'workers = 0','workers = 1')
replace("/content/Mask_RCNN/mrcnn/model.py",'workers = multiprocessing.cpu_count()','workers = 1')
replace("/content/Mask_RCNN/mrcnn/model.py",'use_multiprocessing=True','use_multiprocessing=False')
replace("/content/Mask_RCNN/mrcnn/utils.py","shift = np.array([0, 0, 1, 1])","shift = np.array([0., 0., 1., 1.])")
replace("/content/Mask_RCNN/mrcnn/visualize.py", "i += 1","i += 1\n    plt.savefig('/content/TrainingDataExample_MaskRCNN.png',bbox_inches='tight',pad_inches=0)")
#replace("/content/Mask_RCNN/mrcnn/model.py","   class_ids","    if config.NUM_CLASSES == 2:\n     class_ids = tf.ones_like(probs[:, 0], dtype=tf.int32)\n   else:\n     class_ids")

#Using this command will allow display of detections below the 0.5 score threshold, if only 1 class beyond background is in the dataset
replace("/content/Mask_RCNN/mrcnn/model.py","class_ids = tf.argmax(probs","if config.NUM_CLASSES >= 2:\n     class_ids = tf.ones_like(probs[:, 0], dtype=tf.int32)\n    else:\n     class_ids = tf.argmax(probs")

from mrcnn.config import Config
from mrcnn import model as modellib, utils
from mrcnn import visualize
from mrcnn.model import log
from mrcnn import utils

def get_ax(rows=1, cols=1, size=8):
    """Return a Matplotlib Axes array to be used in
    all visualizations in the notebook. Provide a
    central point to control graph sizes.

    Change the default size attribute to control the size
    of rendered images
    """
    _, ax = plt.subplots(rows, cols, figsize=(size*cols, size*rows))
    return ax

############################################################
#  Dataset
############################################################

class ClassDataset(utils.Dataset):
    def load_coco(annotation_file):
        dataset = json.load(open(annotation_file, 'r'))
        assert type(dataset)==dict, 'annotation file format {} not supported'.format(type(dataset))
        self.dataset = dataset
        self.createIndex()

    def createIndex(self):
        # create index
        print('creating index...')
        anns, cats, imgs = {}, {}, {}
        imgToAnns,catToImgs = defaultdict(list),defaultdict(list)
        if 'annotations' in self.dataset:
            for ann in self.dataset['annotations']:
                imgToAnns[ann['image_id']].append(ann)
                anns[ann['id']] = ann

        if 'images' in self.dataset:
            for img in self.dataset['images']:
                imgs[img['id']] = img

        if 'categories' in self.dataset:
            for cat in self.dataset['categories']:
                cats[cat['id']] = cat

        if 'annotations' in self.dataset and 'categories' in self.dataset:
            for ann in self.dataset['annotations']:
                catToImgs[ann['category_id']].append(ann['image_id'])

        print('index created!')

        # create class members
        self.anns = anns
        self.imgToAnns = imgToAnns
        self.catToImgs = catToImgs
        self.imgs = imgs
        self.cats = cats

    def load_class(self, dataset_dir, subset):
        """Load a subset of the dataset.
        dataset_dir: Root directory of the dataset.
        subset: Subset to load: train or val
        """

        # Add classes. We have only one class to add.
        self.add_class("Training_Datasets", 1, "nucleus")

        # Train or validation dataset?
        assert subset in ["Training", "Validation"]
        dataset_dir = os.path.join(dataset_dir, subset)

        # Load annotations
        # VGG Image Annotator (up to version 1.6) saves each image in the form:
        # { 'filename': '28503151_5b5b7ec140_b.jpg',
        #   'regions': {
        #       '0': {
        #           'region_attributes': {},
        #           'shape_attributes': {
        #               'all_points_x': [...],
        #               'all_points_y': [...],
        #               'name': 'polygon'}},
        #       ... more regions ...
        #   },
        #   'size': 100202
        # }
        # We mostly care about the x and y coordinates of each region
        # Note: In VIA 2.0, regions was changed from a dict to a list.
        annotations = json.load(open(os.path.join(dataset_dir, "birds071220220_json.json")))
        annotations = list(annotations.values())  # don't need the dict keys

        # The VIA tool saves images in the JSON even if they don't have any
        # annotations. Skip unannotated images.
        annotations = [a for a in annotations if a['regions']]

        # Add images
        for a in annotations:
            # Get the x, y coordinaets of points of the polygons that make up
            # the outline of each object instance. These are stores in the
            # shape_attributes (see json format above)
            # The if condition is needed to support VIA versions 1.x and 2.x.
            if type(a['regions']) is dict:
                polygons = [r['shape_attributes'] for r in a['regions'].values()]
            else:
                polygons = [r['shape_attributes'] for r in a['regions']]

            #Get the class of the object
            obj_class = [c['region_attributes']['species'] for c in a['regions']]

            # load_mask() needs the image size to convert polygons to masks.
            # Unfortunately, VIA doesn't include it in JSON, so we must read
            # the image. This is only managable since the dataset is tiny.
            image_path = os.path.join(dataset_dir, a['filename'])
            image = skimage.io.imread(image_path)
            height, width = image.shape[:2]

            self.add_image(
                "Training_Datasets",
                image_id=a['filename'],  # use file name as a unique image id
                path=image_path,
                width=width, height=height,
                polygons=polygons,
                obj_class=obj_class)

    def load_image_csv(self, dataset_dir, subset):
        # Add classes. We have only one class to add.
        # self.add_class("Training_Datasets", 1, "nucleus")
        #self.add_class("Training_Datasets", 2, "Great tit")

        # Train or validation dataset?
        assert subset in ["Training", "Validation"]
        dataset_dir = os.path.join(dataset_dir, subset)
        #Data Format
        #csv file:
        #filename,width,height,object_index, class_name, x, y
        #file_1,256,256,1,nucleus, 1, 1
        #file_1,256,256,1,nucleus, 3, 10
        #file_1,256,256,1,nucleus, 1, 3
        #file_1,256,256,1,nucleus, 3, 7
        #file_1,256,256,2,nucleus, 17, 20
        #...
        class_index = 0
        obj_class_old = ""
        #class_names will hold all the classes we find in the dataset
        class_names = {obj_class_old:class_index}
        for csv_file_name in os.listdir(dataset_dir):
          if csv_file_name.endswith('.csv'):
            with open(os.path.join(dataset_dir,csv_file_name)) as csvfile_count:
              row_count = sum(1 for _ in csvfile_count)
            with open(os.path.join(dataset_dir,csv_file_name)) as csvfile:
              annotations = csv.reader(csvfile)
              next(annotations)
              polygons = []
              x_values = []
              y_values = []
              index_old = 1
              for line in annotations:
                img_file_name = line[0]
                index_new = int(line[4])
                obj_class = line[3]

                if not obj_class in class_names:
                  class_index+=1
                  class_names[obj_class] = class_index
                  self.add_class("Training_Datasets", class_index, obj_class)

                if index_new == index_old:
                  x_values.append(int(line[5]))
                  y_values.append(int(line[6]))

                  if row_count == annotations.line_num:
                    polygon = {"class_name":class_names[obj_class],"all_points_x":x_values,"all_points_y":y_values}
                    polygons.append(polygon)

                elif index_new != index_old:
                  polygon = {"class_name":class_names[obj_class_old],"all_points_x":x_values,"all_points_y":y_values}
                  polygons.append(polygon)
                  x_values = []
                  x_values.append(int(line[5]))
                  y_values = []
                  y_values.append(int(line[6]))

                index_old = int(line[4])
                obj_class_old = line[3]
                image_path = os.path.join(dataset_dir,img_file_name)

            self.add_image(
                "Training_Datasets",
                image_id=img_file_name,  # use file name as a unique image id
                path=image_path,
                width=int(line[1]), height=int(line[2]),
                polygons=polygons)
              #print(csv_file_name, class_index, polygons)
        return class_index

    def load_mask(self, image_id):
        info = self.image_info[image_id]
        #print(info)
        mask = np.zeros([info["height"], info["width"], len(info["polygons"])],
                        dtype=np.uint8)
        class_ids = []
        #class_index = 0
        for i, p in enumerate(info["polygons"]):

            class_name = p['class_name']
            # class_names = {class_name:class_index}
            # if class_name != class_name_old:
            #   class_index+=1
            #   class_names[class_name] = class_index

            # Get indexes of pixels inside the polygon and set them to 1
            # print(p['y_values'])
            rr, cc = skimage.draw.polygon(p['all_points_y'], p['all_points_x'])
            mask[rr, cc, i] = 1

            #class_name_old = p['class_name']
            class_ids.append(class_name)

        class_ids = np.array(class_ids)

        return mask.astype(np.bool), class_ids.astype(np.int32)

    # def load_mask(self, image_id):
    #     """Generate instance masks for an image.
    #    Returns:
    #     masks: A bool array of shape [height, width, instance count] with
    #         one mask per instance.
    #     class_ids: a 1D array of class IDs of the instance masks.
    #     """
    #     def clean_name(name):
    #       """Returns a shorter version of object names for cleaner display."""
    #       return ",".join(name.split(",")[:1])

    #     # If not a balloon dataset image, delegate to parent class.
    #     image_info = self.image_info[image_id]
    #     if image_info["source"] != "Training_Datasets":
    #         return super(self.__class__, self).load_mask(image_id)

    #     # Convert polygons to a bitmap mask of shape
    #     # [height, width, instance_count]
    #     info = self.image_info[image_id]

    #     mask = np.zeros([info["height"], info["width"], len(info["polygons"])],
    #                     dtype=np.uint8)
    #     for i, p in enumerate(info["polygons"]):
    #         # Get indexes of pixels inside the polygon and set them to 1
    #         rr, cc = skimage.draw.polygon(p['all_points_y'], p['all_points_x'])
    #         mask[rr, cc, i] = 1

    #     classes = info["obj_class"]
    #     class_list = [clean_name(c["name"]) for c in self.class_info]
    #     class_ids = np.array([class_list.index(s) for s in classes])

    #     # Return mask, and array of class IDs of each instance. Since we have
    #     # one class ID only, we return an array of 1s
    #     return mask.astype(np.bool), class_ids.astype(np.int32)#np.ones([mask.shape[-1]], dtype=np.int32)

    def image_reference(self, image_id):
        """Return the path of the image."""
        info = self.image_info[image_id]
        if info["source"] == "Training_Datasets":
            return info["path"]
        else:
            super(self.__class__, self).image_reference(image_id)

def train(model, augmentation=True):
    """Train the model."""
    # Training dataset.
    dataset_train = ClassDataset()
    dataset_train.load_class('/content/gdrive/MyDrive/MaskRCNN/Training_Datasets', "Training")
    dataset_train.prepare()

    # Validation dataset
    dataset_val = ClassDataset()
    dataset_val.load_class('/content/gdrive/MyDrive/MaskRCNN/Training_Datasets', "Validation")
    dataset_val.prepare()

    if augmentation == True:
      augment = imgaug.augmenters.Sometimes(0.5, imgaug.augmenters.OneOf([imgaug.augmenters.Fliplr(0.5),
                                                                         imgaug.augmenters.Flipud(0.5),
                                                                         imgaug.augmenters.Affine(rotate=45)]))
    else:
      augment = None
    # *** This training schedule is an example. Update to your needs ***
    # Since we're using a very small dataset, and starting from
    # COCO trained weights, we don't need to train too long. Also,
    # no need to train all layers, just the heads should do it.
    print("Training network heads")
    model.train(dataset_train, dataset_val,
                learning_rate=config.LEARNING_RATE,
                epochs=80,
                augmentation = augment,
                layers='heads')

def train_csv(model, training_folder, augmentation=True, epochs = 20, layers = 'heads'):
    """Train the model."""
    # Training dataset.
    dataset_train = ClassDataset()
    dataset_train.load_image_csv(training_folder, "Training")
    dataset_train.prepare()

    # Validation dataset
    dataset_val = ClassDataset()
    dataset_val.load_image_csv(training_folder, "Validation")
    dataset_val.prepare()

    if augmentation == True:
      augment = imgaug.augmenters.SomeOf((1,2),[imgaug.augmenters.OneOf([imgaug.augmenters.Affine(rotate=90),
                                                                        imgaug.augmenters.Affine(rotate=180),
                                                                        imgaug.augmenters.Affine(rotate=270)]),
                                                imgaug.augmenters.Fliplr(0.5),
                                                imgaug.augmenters.Flipud(0.5),
                                                imgaug.augmenters.Multiply((0.8, 1.5)),
                                                imgaug.augmenters.GaussianBlur(sigma=(0.0, 5.0))])
    else:
      augment = None
    # *** This training schedule is an example. Update to your needs ***
    # Since we're using a very small dataset, and starting from
    # COCO trained weights, we don't need to train too long. Also,
    # no need to train all layers, just the heads should do it.
    print("Training network heads")
    model.train(dataset_train, dataset_val,
                learning_rate=config.LEARNING_RATE,
                epochs=epochs,
                augmentation = augment,
                layers=layers)

def color_splash(image, mask):
    """Apply color splash effect.
    image: RGB image [height, width, 3]
    mask: instance segmentation mask [height, width, instance count]
    Returns result image.
    """
    # Make a grayscale copy of the image. The grayscale copy still
    # has 3 RGB channels, though.
    gray = skimage.color.gray2rgb(skimage.color.rgb2gray(image)) * 255
    # Copy color pixels from the original color image where mask is set
    if mask.shape[-1] > 0:
        # We're treating all instances as one, so collapse the mask into one layer
        mask = (np.sum(mask, -1, keepdims=True) >= 1)
        splash = np.where(mask, image, gray).astype(np.uint8)
    else:
        splash = gray.astype(np.uint8)
    return splash

def detect_and_color_splash(model, image_path=None, video_path=None):
    assert image_path or video_path

    # Image or video?
    if image_path:
        # Run model detection and generate the color splash effect
        print("Running on {}".format(args.image))
        # Read image
        image = skimage.io.imread(args.image)
        # Detect objects
        r = model.detect([image], verbose=1)[0]
        # Color splash
        splash = color_splash(image, r['masks'])
        # Save output
        file_name = "splash_{:%Y%m%dT%H%M%S}.png".format(datetime.datetime.now())
        skimage.io.imsave(file_name, splash)
    elif video_path:
        import cv2
        # Video capture
        vcapture = cv2.VideoCapture(video_path)
        width = int(vcapture.get(cv2.CAP_PROP_FRAME_WIDTH))
        height = int(vcapture.get(cv2.CAP_PROP_FRAME_HEIGHT))
        fps = vcapture.get(cv2.CAP_PROP_FPS)

        # Define codec and create video writer
        file_name = "splash_{:%Y%m%dT%H%M%S}.avi".format(datetime.datetime.now())
        vwriter = cv2.VideoWriter(file_name,
                                  cv2.VideoWriter_fourcc(*'MJPG'),
                                  fps, (width, height))

        count = 0
        success = True
        while success:
            print("frame: ", count)
            # Read next image
            success, image = vcapture.read()
            if success:
                # OpenCV returns images as BGR, convert to RGB
                image = image[..., ::-1]
                # Detect objects
                r = model.detect([image], verbose=0)[0]
                # Color splash
                splash = color_splash(image, r['masks'])
                # RGB -> BGR to save image to video
                splash = splash[..., ::-1]
                # Add image to video writer
                vwriter.write(splash)
                count += 1
        vwriter.release()
    print("Saved to ", file_name)

# Colors for the warning messages
class bcolors:
  WARNING = '\033[31m'
  NORMAL = '\033[0m'

class ClassConfig(Config):
    """Configuration for training on the toy  dataset.
    Derives from the base Config class and overrides some values.
    """
    # Give the configuration a recognizable name
    # We use a GPU with 12GB memory, which can fit two images.
    # Adjust down if you use a smaller GPU.
    IMAGES_PER_GPU = 1
    DETECTION_MIN_CONFIDENCE = 0
    NAME = "nucleus"
    # Backbone network architecture
    # Supported values are: resnet50, resnet101
    BACKBONE = "resnet50"
    # Input image resizing
    # Random crops of size 64x64
    IMAGE_RESIZE_MODE = "crop"
    IMAGE_MIN_DIM = 256
    IMAGE_MAX_DIM = 256
    IMAGE_MIN_SCALE = 2.0
    # Length of square anchor side in pixels
    RPN_ANCHOR_SCALES = (4, 8, 16, 32, 64)
    # ROIs kept after non-maximum supression (training and inference)
    POST_NMS_ROIS_TRAINING = 200
    POST_NMS_ROIS_INFERENCE = 400
    # Non-max suppression threshold to filter RPN proposals.
    # You can increase this during training to generate more propsals.
    RPN_NMS_THRESHOLD = 0.9
    # How many anchors per image to use for RPN training
    RPN_TRAIN_ANCHORS_PER_IMAGE = 64
    # Image mean (RGB)
    MEAN_PIXEL = np.array([43.53, 39.56, 48.22])
    # If enabled, resizes instance masks to a smaller size to reduce
    # memory load. Recommended when using high-resolution images.
    USE_MINI_MASK = True
    MINI_MASK_SHAPE = (56, 56)  # (height, width) of the mini-mask
    TRAIN_ROIS_PER_IMAGE = 128
    # Maximum number of ground truth instances to use in one image
    MAX_GT_INSTANCES = 100
    # Max number of final detections per image
    DETECTION_MAX_INSTANCES = 200

# Below we define a function which saves the predictions.
# It is from this branch:
# https://github.com/matterport/Mask_RCNN/commit/bc8f148b820ebd45246ed358a120c99b09798d71

def save_image(image, image_name, boxes, masks, class_ids, scores, class_names, filter_classs_names=None,
               scores_thresh=0.1, save_dir=None, mode=0):
    """
        image: image array
        image_name: image name
        boxes: [num_instance, (y1, x1, y2, x2, class_id)] in image coordinates.
        masks: [num_instances, height, width]
        class_ids: [num_instances]
        scores: confidence scores for each box
        class_names: list of class names of the dataset
        filter_classs_names: (optional) list of class names we want to draw
        scores_thresh: (optional) threshold of confidence scores
        save_dir: (optional) the path to store image
        mode: (optional) select the result which you want
                mode = 0 , save image with bbox,class_name,score and mask;
                mode = 1 , save image with bbox,class_name and score;
                mode = 2 , save image with class_name,score and mask;
                mode = 3 , save mask with black background;
    """
    mode_list = [0, 1, 2, 3]
    assert mode in mode_list, "mode's value should in mode_list %s" % str(mode_list)

    if save_dir is None:
        save_dir = os.path.join(os.getcwd(), "output")
        if not os.path.exists(save_dir):
            os.makedirs(save_dir)

    useful_mask_indices = []

    N = boxes.shape[0]
    if not N:
        print("\n*** No instances in image %s to draw *** \n" % (image_name))
        return
    else:
        assert boxes.shape[0] == masks.shape[-1] == class_ids.shape[0]

    for i in range(N):
        # filter
        class_id = class_ids[i]
        score = scores[i] if scores is not None else None
        if score is None or score < scores_thresh:
            continue

        label = class_names[class_id]
        if (filter_classs_names is not None) and (label not in filter_classs_names):
            continue

        if not np.any(boxes[i]):
            # Skip this instance. Has no bbox. Likely lost in image cropping.
            continue

        useful_mask_indices.append(i)

    if len(useful_mask_indices) == 0:
        print("\n*** No instances in image %s to draw *** \n" % (image_name))
        return

    colors = visualize.random_colors(len(useful_mask_indices))

    if mode != 3:
        masked_image = image.astype(np.uint8).copy()
    else:
        masked_image = np.zeros(image.shape).astype(np.uint8)

    if mode != 1:
        for index, value in enumerate(useful_mask_indices):
            masked_image = visualize.apply_mask(masked_image, masks[:, :, value], colors[index])

    masked_image = Image.fromarray(masked_image)

    if mode == 3:
        masked_image.save(os.path.join(save_dir, '%s' % (image_name)))
        return

    draw = ImageDraw.Draw(masked_image)
    colors = np.array(colors).astype(int) * 255

    for index, value in enumerate(useful_mask_indices):
        class_id = class_ids[value]
        score = scores[value]
        label = class_names[class_id]

        y1, x1, y2, x2 = boxes[value]
        if mode != 2:
            color = tuple(colors[index])
            draw.rectangle((x1, y1, x2, y2), outline=color)

        # Label
        font = ImageFont.load_default()
        draw.text((x1, y1), "%s %f" % (label, score), (255, 255, 255), font)

    masked_image.save(os.path.join(save_dir, '%s' % (image_name)))

def pdf_export(config, trained = False, augmentation = False, pretrained_model = False):
  class MyFPDF(FPDF, HTMLMixin):
    pass

  config_list = ""
  for a in dir(config):
    if not a.startswith("__") and not callable(getattr(config, a)):
      config_list += "{}:     {}\n".format(a, getattr(config, a))

  pdf = MyFPDF()
  pdf.add_page()
  pdf.set_right_margin(-1)
  pdf.set_font("Arial", size = 11, style='B')

  Network = 'MaskRCNN'
  day = datetime.datetime.now()
  datetime_str = str(day)[0:10]

  Header = 'Training report for '+Network+' model ('+model_name+'):\nDate: '+datetime_str
  pdf.multi_cell(180, 5, txt = Header, align = 'L')

  # add another cell
  if trained:
    training_time = "Training time: "+str(hour)+ "hour(s) "+str(mins)+"min(s) "+str(round(sec))+"sec(s)"
    pdf.cell(190, 5, txt = training_time, ln = 1, align='L')
  pdf.ln(1)

  Header_2 = 'Information for your materials and methods:'
  pdf.cell(190, 5, txt=Header_2, ln=1, align='L')

  all_packages = ''
  for requirement in freeze(local_only=True):
    all_packages = all_packages+requirement+', '
  #print(all_packages)

  #Main Packages
  main_packages = ''
  version_numbers = []
  for name in ['tensorflow','numpy','Keras']:
    find_name=all_packages.find(name)
    main_packages = main_packages+all_packages[find_name:all_packages.find(',',find_name)]+', '
    #Version numbers only here:
    version_numbers.append(all_packages[find_name+len(name)+2:all_packages.find(',',find_name)])

  cuda_version = sp.run('nvcc --version',stdout=sp.PIPE, shell=True)
  cuda_version = cuda_version.stdout.decode('utf-8')
  cuda_version = cuda_version[cuda_version.find(', V')+3:-1]
  gpu_name = sp.run('nvidia-smi',stdout=sp.PIPE, shell=True)
  gpu_name = gpu_name.stdout.decode('utf-8')
  gpu_name = gpu_name[gpu_name.find('Tesla'):gpu_name.find('Tesla')+10]
  #print(cuda_version[cuda_version.find(', V')+3:-1])
  #print(gpu_name)
  try:
    shape = io.imread(Training_source+'/Training/'+os.listdir(Training_source+'/Training')[0]).shape
  except:
    shape = io.imread(Training_source+'/Training/'+os.listdir(Training_source+'/Training')[0][:-4]).shape
  dataset_size = len(os.listdir(Training_source))/2

  text = 'The '+Network+' model was trained using weights initialised on the coco dataset for '+str(number_of_epochs)+' epochs on '+str(dataset_size)+' labelled images (image dimensions: '+str(shape)+') with a batch size of '+str(config.BATCH_SIZE)+' and custom loss functions for region proposal and classification, using the '+Network+' ZeroCostDL4Mic notebook (v '+Notebook_version[0]+') (von Chamier & Laine et al., 2020). Key python packages used include tensorflow (v '+version_numbers[0]+'), Keras (v '+version_numbers[2]+'), numpy (v '+version_numbers[1]+'), cuda (v '+cuda_version+'). The training was accelerated using a '+gpu_name+'GPU.'

  if pretrained_model:
    text = 'The '+Network+' model was trained for '+str(number_of_epochs)+' epochs on '+str(dataset_size)+' labelled images (image dimensions: '+str(shape)+') with a batch size of '+str(config.BATCH_SIZE)+' and custom loss functions for region proposal and classification, using the '+Network+' ZeroCostDL4Mic notebook (v '+Notebook_version[0]+') (von Chamier & Laine et al., 2020). The model was retrained from a previous model checkpoint (model: '+os.path.basename(pretrained_model_path)[:-8]+', checkpoint: '+str(int(pretrained_model_path[-7:-3]))+'). Key python packages used include tensorflow (v '+version_numbers[0]+'), Keras (v '+version_numbers[2]+'), numpy (v '+version_numbers[1]+'), cuda (v '+cuda_version+'). The training was accelerated using a '+gpu_name+'GPU.'

  pdf.set_font('')
  pdf.set_font_size(10.)
  pdf.multi_cell(190, 5, txt = text, align='L')
  pdf.set_font('')
  pdf.set_font('Arial', size = 10, style = 'B')
  pdf.ln(1)
  pdf.cell(28, 5, txt='Augmentation: ', ln=0)
  pdf.set_font('')
  if augmentation:
    aug_text = 'The dataset was augmented by vertical and horizontal flipping'
    # if multiply_dataset_by >= 2:
    #   aug_text = aug_text+'\n- flipping'
    # if multiply_dataset_by > 2:
    #   aug_text = aug_text+'\n- rotation'
  else:
    aug_text = 'No augmentation was used for training.'
  pdf.multi_cell(190, 5, txt=aug_text, align='L')
  pdf.set_font('Arial', size = 11, style = 'B')
  pdf.ln(1)
  pdf.cell(180, 5, txt = 'Parameters', align='L', ln=1)
  pdf.set_font('')
  pdf.set_font_size(10.)
  # if Use_Default_Advanced_Parameters:
  #   pdf.cell(200, 5, txt='Default Advanced Parameters were enabled')
  pdf.cell(200, 5, txt='The following parameters were used for training:')
  pdf.ln(4)
  pdf.multi_cell(200, 5, txt=config_list)

  pdf.set_font("Arial", size = 11, style='B')
  pdf.ln(1)
  pdf.cell(190, 5, txt = 'Training Dataset', align='L', ln=1)
  pdf.set_font('')
  pdf.set_font('Arial', size = 10, style = 'B')
  pdf.cell(30, 5, txt= 'Training_source:', align = 'L', ln=0)
  pdf.set_font('')
  pdf.multi_cell(170, 5, txt = Training_source+'/Training', align = 'L')
  pdf.set_font('')
  pdf.set_font('Arial', size = 10, style = 'B')
  pdf.cell(29, 5, txt= 'Validation:', align = 'L', ln=0)
  pdf.set_font('')
  pdf.multi_cell(170, 5, txt = Training_source+'/Validation', align = 'L')
  #pdf.cell(190, 5, txt=aug_text, align='L', ln=1)
  pdf.ln(1)
  pdf.set_font('')
  pdf.set_font('Arial', size = 10, style = 'B')
  pdf.cell(22, 5, txt= 'Model Path:', align = 'L', ln=0)
  pdf.set_font('')
  pdf.multi_cell(170, 5, txt = model_path+'/'+model_name, align = 'L')
  pdf.ln(1)
  pdf.cell(60, 5, txt = 'Example ground-truth annotation', ln=1)
  pdf.ln(1)
  exp_size = io.imread('/content/TrainingDataExample_MaskRCNN.png').shape
  pdf.image('/content/TrainingDataExample_MaskRCNN.png', x = 11, y = None, w = round(exp_size[1]/8), h = round(exp_size[0]/8))
  pdf.ln(1)
  ref_1 = 'References:\n - ZeroCostDL4Mic: von Chamier, Lucas & Laine, Romain, et al. "Democratising deep learning for microscopy with ZeroCostDL4Mic." Nature Communications (2021).'
  pdf.multi_cell(190, 5, txt = ref_1, align='L')
  ref_2 = '- MaskRCNN: Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick. "Mask R - CNN" arxiv. 2018.'
  pdf.multi_cell(190, 5, txt = ref_2, align='L')
  if augmentation:
    ref_3 = '- imgaug: Jung, Alexander et al., https://github.com/aleju/imgaug, (2020)'
    pdf.multi_cell(190, 5, txt = ref_3, align='L')
  pdf.ln(3)
  reminder = 'Important:\nRemember to perform the quality control step on all newly trained models\nPlease consider depositing your training dataset on Zenodo'
  pdf.set_font('Arial', size = 11, style='B')
  pdf.multi_cell(190, 5, txt=reminder, align='C')

  pdf.output(os.path.dirname(model.log_dir)+'/'+model_name+'_training_report.pdf')

  print('------------------------------')
  print('PDF report exported in '+model_path+'/'+model_name+'/')

def qc_pdf_export():
  class MyFPDF(FPDF, HTMLMixin):
    pass

  pdf = MyFPDF()
  pdf.add_page()
  pdf.set_right_margin(-1)
  pdf.set_font("Arial", size = 11, style='B')

  Network = 'MaskRCNN'

  day = datetime.datetime.now()
  datetime_str = str(day)[0:16]

  Header = 'Quality Control report for '+Network+' model ('+QC_model_name+', checkpoint:'+str(Checkpoint)+')\nDate and Time: '+datetime_str
  pdf.multi_cell(180, 5, txt = Header, align = 'L')

  all_packages = ''
  for requirement in freeze(local_only=True):
    all_packages = all_packages+requirement+', '

  pdf.set_font('')
  pdf.set_font('Arial', size = 11, style = 'B')
  pdf.ln(2)
  pdf.cell(190, 5, txt = 'Development of Training Losses', ln=1, align='L')
  pdf.ln(1)
  if os.path.exists(QC_model_folder+'/Quality Control/lossCurveAndmAPPlots.png'):
    exp_size = io.imread(QC_model_folder+'/Quality Control/lossCurveAndmAPPlots.png').shape
    pdf.image(QC_model_folder+'/Quality Control/lossCurveAndmAPPlots.png', x = 11, y = None, w = round(exp_size[1]/8), h = round(exp_size[0]/8))
  else:
    pdf.set_font('')
    pdf.set_font('Arial', size=10)
    pdf.multi_cell(190, 5, txt='If you would like to see the evolution of the loss function during training please play the first cell of the QC section in the notebook.')
  pdf.set_font('')
  pdf.set_font('Arial', size = 10, style = 'B')
  pdf.cell(80, 5, txt = 'P-R curves for test dataset', ln=1, align='L')
  pdf.ln(2)
  #for i in range(len(AP)):
  # os.path.exists(QC_model_folder+'/Quality Control/P-R_curve_'+config['model']['labels'][i]+'.png'):
  exp_size = io.imread(QC_model_folder+'/Quality Control/P-R_curve_'+QC_model_name+'.png').shape
  pdf.ln(1)
  pdf.image(QC_model_folder+'/Quality Control/P-R_curve_'+QC_model_name+'.png', x=16, y=None, w=round(exp_size[1]/4), h=round(exp_size[0]/4))
  #  else:
  #    pdf.cell(100, 5, txt='For the class '+config['model']['labels'][i]+' the model did not predict any objects.', ln=1, align='L')
  pdf.ln(3)
  pdf.set_font('')
  pdf.set_font('Arial', size = 11, style = 'B')
  pdf.ln(1)
  pdf.cell(180, 5, txt = 'Quality Control Metrics', align='L', ln=1)
  pdf.set_font('')
  pdf.set_font_size(10.)

  pdf.ln(1)
  html = """

"""
  with open(QC_model_folder+'/Quality Control/QC_results.csv', 'r') as csvfile:
    metrics = csv.reader(csvfile)
    header = next(metrics)
    class_name = header[0]
    gt = header[1]
    tp = header[2]
    fn = header[3]
    iou = header[4]
    mAP = header[5]
    header = """

    {0}
    {1}
    {2}
    {3}
    {4}
    {5}

""".format(class_name,gt,tp,fn,iou,mAP)
    html = html+header
    i=0
    for row in metrics:
      i+=1
      class_name = row[0]
      gt = row[1]
      tp = row[2]
      fn = row[3]
      iou = row[4]
      mAP = row[5]
      cells = """

          {0}
          {1}
          {2}
          {3}
          {4}
          {5}

""".format(class_name,str(gt),str(tp),str(fn),str(iou),str(mAP))
      html = html+cells
    html = html+"""
"""

  pdf.write_html(html)
  pdf.set_font('')
  pdf.set_font('Arial', size = 11, style = 'B')
  pdf.ln(3)
  pdf.cell(80, 5, txt = 'Example Quality Control Visualisation', ln=1)
  pdf.ln(3)
  exp_size = io.imread(QC_model_folder+'/Quality Control/QC_example_data.png').shape
  pdf.image(QC_model_folder+'/Quality Control/QC_example_data.png', x = 16, y = None, w = round(exp_size[1]/10), h = round(exp_size[0]/10))

  pdf.set_font('')
  pdf.set_font_size(10.)
  pdf.ln(3)
  ref_1 = 'References:\n - ZeroCostDL4Mic: von Chamier, Lucas & Laine, Romain, et al. "Democratising deep learning for microscopy with ZeroCostDL4Mic." Nature Communications (2021).'
  pdf.multi_cell(190, 5, txt = ref_1, align='L')
  ref_2 = '- MaskRCNN: Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick. "Mask R - CNN" arxiv. 2018.'
  pdf.multi_cell(190, 5, txt = ref_2, align='L')

  pdf.ln(3)
  reminder = 'To find the parameters and other information about how this model was trained, go to the training_report.pdf of this model which should be in the folder of the same name.'

  pdf.set_font('Arial', size = 11, style='B')
  pdf.multi_cell(190, 5, txt=reminder, align='C')

  pdf.output(QC_model_folder+'/Quality Control/'+QC_model_name+'_QC_report.pdf')

  print('------------------------------')
  print('PDF report exported in '+QC_model_folder+'/Quality Control/')

# Check if this is the latest version of the notebook
All_notebook_versions = pd.read_csv("https://raw.githubusercontent.com/HenriquesLab/ZeroCostDL4Mic/master/Colab_notebooks/Latest_Notebook_versions.csv", dtype=str)
print('Notebook version: '+Notebook_version)
Latest_Notebook_version = All_notebook_versions[All_notebook_versions["Notebook"] == Network]['Version'].iloc[0]
print('Latest notebook version: '+Latest_Notebook_version)
if Notebook_version == Latest_Notebook_version:
  print("This notebook is up-to-date.")
else:
  print(bcolors.WARNING +"A new version of this notebook has been released. We recommend that you download it at https://github.com/HenriquesLab/ZeroCostDL4Mic/wiki")

# Build requirements file for local run
after = [str(m) for m in sys.modules]
build_requirements_file(before, after)_____no_output_____

# **2. Initialise the Colab session**

---

_____no_output_____
## **2.1. Check for GPU access**
---

By default, the session should be using Python 3 and GPU acceleration, but it is possible to ensure that these are set properly by doing the following:

Go to **Runtime -> Change the Runtime type**

**Runtime type: Python 3** *(Python 3 is programming language in which this program is written)*

**Accelator: GPU** *(Graphics processing unit)*
_____no_output_____

#@markdown ##Run this cell to check if you have GPU access
%tensorflow_version 1.x
import tensorflow as tf
if tf.test.gpu_device_name()=='':
  print('You do not have GPU access.')
  print('Did you change your runtime ?')
  print('If the runtime setting is correct then Google did not allocate a GPU for your session')
  print('Expect slow performance. To access GPU try reconnecting later')

else:
  print('You have GPU access')
  !nvidia-smi_____no_output_____

## **2.2. Mount your Google Drive**
---
 To use this notebook on the data present in your Google Drive, you need to mount your Google Drive to this notebook.

 Play the cell below to mount your Google Drive and follow the link. In the new browser window, select your drive and select 'Allow', copy the code, paste into the cell and press enter. This will give Colab access to the data on the drive.

 Once this is done, your data are available in the **Files** tab on the top left of notebook._____no_output_____

#@markdown ##Run this cell to connect your Google Drive to Colab

#@markdown * Click on the URL.

#@markdown * Sign in your Google Account.

#@markdown * Copy the authorization code.

#@markdown * Enter the authorization code.

#@markdown * Click on "Files" site on the right. Refresh the site. Your Google Drive folder should now be available here as "drive".

#mounts user's Google Drive to Google Colab.

from google.colab import drive
drive.mount('/content/gdrive')_____no_output_____

** If you cannot see your files, reactivate your session by connecting to your hosted runtime.**

 Connect to a hosted runtime.
_____no_output_____# **3. Select your paths and parameters**

---

The code below allows the user to enter the paths to where the training data is and to define the training parameters.

If your dataset is large, this step can take a while.

**Note:** The BG class reported by MaskRCNN stands for 'background'. By default BG is the default class in MaskRCNN, so even if your dataset contains only one class, MaskRCNN will treat the dataset as a two-class set.
_____no_output_____## **3.1. Setting the main training parameters**
---
_____no_output_____ **Paths for training, predictions and results**

**`Training_source:`:** This is the path to your folder containing the subfolders *Training* and *Validation*, each containing images with their respective annotations. **If your files are not organised in this way, the notebook will NOT work. So make sure everything looks right!** To find the paths of the folders containing the respective datasets, go to your Files on the left of the notebook, navigate to the folder containing your files and copy the path by right-clicking on the folder, **Copy path** and pasting it into the right box below.

**`model_name`:** Use only my_model -style, not my-model (Use "_" not "-"). Do not use spaces in the name. Avoid using the name of an existing model (saved in the same folder) as it will be overwritten. **Note that MaskRCNN will add a timestamp to your model_name in the form: model_name*YearMonthDayTHourMinute***

**`model_path`**: Enter the path where your model will be saved once trained (for instance your result folder).

**Training parameters**

**`Training Depth`:** Here, you can choose how much you want to train the network. MaskRCNN is already pretrained on a large dataset which means its weights are already initialised. This means it may not be necessary to train the full model to reach satisfactory results on your dataset. To get the most out of the model, we recommend training the headlayers first for ca. 30 epochs, and then retraining the same model with an increasing depth for further 10s of epochs. To do this, use the same model_name in this section, with any other needed parameters and then load the desired weights file in section 3.3. **Default value: Head layers only**

**`number_of_epochs`:**Enter the number of epochs the networks will be trained for. Note that if you want to continue training a previously trained model, enter the final number of epochs you want to use, i.e. if your previous model was trained for 50 epochs and you want to train it to 80, enter 80 epochs here, not 30.
**Default value: 50**

**`detection_confidence`:** The network will assign scores of confidence to any predictions of ROIs it makes on the dataset during training. The detection confidence here indicates what threshold score you want to apply for the network to use accept any predicted ROIs. We recommend starting low here. If you notice your network is giving you too many ROIs, then increase this value gradually. **Default value: 0**

**`learning_rate:`** Input the initial value to be used as learning rate. The learning rate will decrease after 7 epochs if the validation loss does not improve. **Default value: 0.003**

_____no_output_____

#@markdown ###Path to training images:

Training_source = "" #@param {type:"string"}

# Ground truth images
#Training_validation = "" #@param {type:"string"}

# model name and path
##@markdown ###Name of the model and path to model folder:
model_name = "" #@param {type:"string"}
model_path = "" #@param {type:"string"}

full_model_path = os.path.join(model_path,model_name)
# if os.path.exists(full_model_path):
#   print(bcolors.WARNING+'Model folder already exists and will be overwritten.'+bcolors.NORMAL)

# other parameters for training.
#@markdown ###Training Parameters

Training_depth = "3+ resnet layers" #@param ["Head_layers_only", "3+ resnet layers", "4+ resnet layers", "5+ resnet layers", "all layers"]
##@markdown ###Advanced Parameters

#Use_Default_Advanced_Parameters = True #@param {type:"boolean"}
##@markdown ###If not, please input:

number_of_epochs =  10#@param {type:"integer"}

batch_size =  4#@param{type:"integer"}

image_resize_mode = "none"

detection_confidence = 0 #@param {type:"number"}

region_proposal_nms_threshold = 0.9 #@param{type:"number"}

learning_rate = 0.003 #@param {type:"number"}

#@markdown ###Loss weights

region_proposal_class_loss =  1#@param {type:"number"}
region_proposal_class_loss = float(region_proposal_class_loss)

region_proposal_bbox_loss =  1#@param {type:"number"}
region_proposal_bbox_loss = float(region_proposal_bbox_loss)

mrcnn_class_loss =  1#@param {type:"number"}
mrcnn_class_loss = float(mrcnn_class_loss)

mrcnn_bbox_loss =  1#@param {type:"number"}
mrcnn_bbox_loss = float(mrcnn_bbox_loss)

mrcnn_mask_loss =  1#@param {type:"number"}
mrcnn_mask_loss = float(mrcnn_mask_loss)

# Path to trained weights file
COCO_WEIGHTS_PATH = os.path.join(ROOT_DIR, "mask_rcnn_coco.h5")

# Directory to save logs and model checkpoints, if not provided
# through the command line argument --logs
DEFAULT_LOGS_DIR = model_path

dataset_train = ClassDataset()
dataset_train.load_image_csv(Training_source, "Training")
dataset_train.prepare()

print("Class Count: {}".format(dataset_train.num_classes))
for i, info in enumerate(dataset_train.class_info):
    print("{:3}. {:50}".format(i, info['name']))

############################################################
#  Configurations
############################################################

class ClassConfig(Config):
    """Configuration for training on the toy  dataset.
    Derives from the base Config class and overrides some values.
    """
    # Give the configuration a recognizable name
    NAME = model_name

    # We use a GPU with 12GB memory, which can fit two images.
    # Adjust down if you use a smaller GPU.
    IMAGES_PER_GPU = batch_size

    # Number of classes (including background)
    NUM_CLASSES = len(dataset_train.class_names) # Background + nucleus

    # Number of training steps per epoch
    STEPS_PER_EPOCH = (len(os.listdir(Training_source+"/Training"))/2)  // IMAGES_PER_GPU
    VALIDATION_STEPS = (len(os.listdir(Training_source+"/Validation"))/2) // IMAGES_PER_GPU

    # Skip detections with < 90% confidence
    # DETECTION_MIN_CONFIDENCE = detection_confidence

    LEARNING_RATE = learning_rate

    DETECTION_MIN_CONFIDENCE = 0

    # Backbone network architecture
    # Supported values are: resnet50, resnet101
    BACKBONE = "resnet101"

    # Input image resizing
    # Random crops of size 64x64
    IMAGE_RESIZE_MODE = image_resize_mode #"crop"
    IMAGE_MIN_DIM = 128
    IMAGE_MAX_DIM = 128
    IMAGE_MIN_SCALE = 2.0

    # Length of square anchor side in pixels
    RPN_ANCHOR_SCALES = (8, 16, 32, 64, 128)

    # ROIs kept after non-maximum supression (training and inference)
    POST_NMS_ROIS_TRAINING = 2000
    POST_NMS_ROIS_INFERENCE = 4000

    # Non-max suppression threshold to filter RPN proposals.
    # You can increase this during training to generate more propsals.
    RPN_NMS_THRESHOLD = region_proposal_nms_threshold

    # How many anchors per image to use for RPN training
    RPN_TRAIN_ANCHORS_PER_IMAGE = 128

    # Image mean (RGB)
    MEAN_PIXEL = np.array([43.53, 39.56, 48.22])

    # If enabled, resizes instance masks to a smaller size to reduce
    # memory load. Recommended when using high-resolution images.
    USE_MINI_MASK = False
    MINI_MASK_SHAPE = (56, 56)  # (height, width) of the mini-mask

    # Number of ROIs per image to feed to classifier/mask heads
    # The Mask RCNN paper uses 512 but often the RPN doesn't generate
    # enough positive proposals to fill this and keep a positive:negative
    # ratio of 1:3. You can increase the number of proposals by adjusting
    # the RPN NMS threshold.
    TRAIN_ROIS_PER_IMAGE = 128

    # Maximum number of ground truth instances to use in one image
    MAX_GT_INSTANCES = 100

    # Max number of final detections per image
    DETECTION_MAX_INSTANCES = 200

    LOSS_WEIGHTS = {
        "rpn_class_loss": region_proposal_class_loss,
        "rpn_bbox_loss": region_proposal_bbox_loss,
        "mrcnn_class_loss": mrcnn_class_loss,
        "mrcnn_bbox_loss": mrcnn_bbox_loss,
        "mrcnn_mask_loss": mrcnn_mask_loss
    }

if Training_depth == "Head_layers_only":
  layers = "heads"
elif Training_depth == "3+ resnet layers":
  layers = "3+"
elif Training_depth == "4+ resnet layers":
  layers = "4+"
elif Training_depth == "5+ resnet layers":
  layers = "5+"
else:
  layers = "all"

config = ClassConfig()
# Training dataset
# dataset_train = ClassDataset()
# num_classes = dataset_train.load_image_csv(Training_source, "Training")
# dataset_train.prepare()
# print("Class Count: {}".format(dataset_train.num_classes))
# for i, info in enumerate(dataset_train.class_info):
#     print("{:3}. {:50}".format(i, info['name']))

# Load and display random samples
image_ids = np.random.choice(dataset_train.image_ids, 1)
for image_id in image_ids:
    image = dataset_train.load_image(image_id)
    mask, class_ids = dataset_train.load_mask(image_id)
    visualize.display_top_masks(image, mask, class_ids, dataset_train.class_names, limit=dataset_train.num_classes-1)

# plt.savefig('/content/TrainingDataExample_MaskRCNN.png',bbox_inches='tight',pad_inches=0)

# image, image_meta, class_ids, bbox, mask = modellib.load_image_gt(
#         dataset_train, config, image_id, use_mini_mask=False)

# visualize.display_instances(image, bbox, mask, class_ids, dataset_train.class_names,
#                             show_bbox=False)
model = modellib.MaskRCNN(mode="training", config=config, model_dir=DEFAULT_LOGS_DIR)
config.display()
Use_pretrained_model = False
Use_Data_augmentation = False_____no_output_____

##**3.2. Data augmentation**

---

 Data augmentation can improve training progress by amplifying differences in the dataset. This can be useful if the available dataset is small since, in this case, it is possible that a network could quickly learn every example in the dataset (overfitting), without augmentation. Augmentation is not necessary for training and if the dataset the `Use_Data_Augmentation` box can be unticked.

 If the box is ticked a simple augmentation of horizontal and vertical flipping will be applied to the dataset._____no_output_____

#@markdown ##**Augmentation Options**

Use_Data_augmentation = True #@param {type:"boolean"}

if Use_Data_augmentation == True:
  # Number of training steps per epoch
  class AugClassConfig(ClassConfig):
    STEPS_PER_EPOCH = 10*((len(os.listdir(Training_source+"/Training"))/2)  // batch_size)
    VALIDATION_STEPS = 10*((len(os.listdir(Training_source+"/Validation"))/2) // batch_size)

if Use_Data_augmentation:
  config = AugClassConfig()_____no_output_____

## **3.3. Using weights from a pre-trained model as initial weights**
---
  Here, you can set the the path to a pre-trained model from which the weights can be extracted and used as a starting point for this training session. **This pre-trained model needs to be a MaskRCNN model**.

 This option allows you to perform training over multiple Colab runtimes or to do transfer learning using models trained outside of ZeroCostDL4Mic. **You do not need to run this section if you want to train a network from scratch**._____no_output_____

# @markdown ##Loading weights from a pre-trained network

Use_pretrained_model = True #@param {type:"boolean"}

#@markdown ###If yes, please provide the path to the model (this path should end with the file extension .h5):
pretrained_model_path = "" #@param {type:"string"}

if Use_Data_augmentation == True:
  config = AugClassConfig()
else:
  config = ClassConfig()

model = modellib.MaskRCNN(mode="training", config=config, model_dir=DEFAULT_LOGS_DIR)
model.load_weights(pretrained_model_path, by_name=True)_____no_output_____

#**4. Train the network**
---_____no_output_____## **4.1. Train the network**
---
When playing the cell below you should see updates after each epoch (round). Network training can take some time.

* **CRITICAL NOTE:** Google Colab has a time limit for processing (to prevent using GPU power for datamining). Training time must be less than 12 hours! If training takes longer than 12 hours, please decrease the number of epochs or number of patches._____no_output_____

#@markdown ##Start training

pdf_export(config, augmentation = Use_Data_augmentation, pretrained_model=Use_pretrained_model)

if os.path.exists(model.log_dir+"/Quality Control"):
  shutil.rmtree(model.log_dir+"/Quality Control")
os.makedirs(model.log_dir+"/Quality Control")

start = time.time()
#Here, we start the model training
train_csv(model, Training_source, augmentation=Use_Data_augmentation, epochs = number_of_epochs, layers = layers)
dt = time.time() - start
mins, sec = divmod(dt, 60)
hour, mins = divmod(mins, 60)
print("Time elapsed:",hour, "hour(s)",mins,"min(s)",round(sec),"sec(s)")

new_model_name = os.path.basename(model.log_dir)
#Here, we just save some interesting parameters from training as a csv file
if not os.path.exists(model_path+'/'+new_model_name+'/Quality Control/class_names.csv'):
  with open(model_path+'/'+new_model_name+'/Quality Control/class_names.csv','w') as class_count_csv:
    class_writer = csv.writer(class_count_csv)
    for class_name in dataset_train.class_names:
      class_writer.writerow([class_name])

if os.path.exists(model_path+'/'+new_model_name+'/Quality Control/training_evaluation.csv'):
  with open(model_path+'/'+new_model_name+'/Quality Control/training_evaluation.csv','a') as csvfile:
    writer = csv.writer(csvfile)
    #print('hello')
    #writer.writerow(['epoch','loss','val_loss','learning rate'])
    model_starting_checkpoint = int(pretrained_model_path[-7:-3])
    for i in range(len(model.keras_model.history.history['loss'])):
      writer.writerow([str(model_starting_checkpoint+i),model.keras_model.history.history['loss'][i], str(learning_rate)])
else:
  with open(model_path+'/'+new_model_name+'/Quality Control/training_evaluation.csv','w') as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(['epoch','loss','val_loss','learning rate'])
    for i in range(len(model.keras_model.history.history['loss'])):
      writer.writerow([str(i+1),model.keras_model.history.history['loss'][i], model.keras_model.history.history['val_loss'][i], str(learning_rate)])

pdf_export(config, trained=True, augmentation = Use_Data_augmentation, pretrained_model = Use_pretrained_model)_____no_output_____

# **5. Evaluate your model**
---

This section allows the user to perform important quality checks on the validity and generalisability of the trained model.

**We highly recommend to perform quality control on all newly trained models.**

_____no_output_____

#@markdown ###Do you want to assess the model you just trained ?
Use_the_current_trained_model = True #@param {type:"boolean"}

#@markdown ###If not, please provide the name of the model folder:

QC_model_folder = "" #@param {type:"string"}

if (Use_the_current_trained_model):
  QC_model_folder = model_path+'/'+new_model_name

QC_model_name = os.path.basename(QC_model_folder)

if os.path.exists(QC_model_folder):
  print("The "+QC_model_name+" model will be evaluated")
else:
  W  = '\033[0m'  # white (normal)
  R  = '\033[31m' # red
  print(R+'!! WARNING: The chosen model does not exist !!'+W)
  print('Please make sure you provide a valid model path before proceeding further.')_____no_output_____

## **5.1. Inspection of the loss function**
---

First, it is good practice to evaluate the training progress by comparing the training loss with the validation loss. The latter is a metric which shows how well the network performs on a subset of unseen data which is set aside from the training dataset. For more information on this, see for example [this review](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6381354/) by Nichols *et al.*

**Training loss** describes an error value after each epoch for the difference between the model's prediction and its ground-truth target.

**Validation loss** describes the same error value between the model's prediction on a validation image and compared to it's target.

During training both values should decrease before reaching a minimal value which does not decrease further even after more training. Comparing the development of the validation loss with the training loss can give insights into the model's performance.

Decreasing **Training loss** and **Validation loss** indicates that training is still necessary and increasing the `number_of_epochs` is recommended. Note that the curves can look flat towards the right side, just because of the y-axis scaling. The network has reached convergence once the curves flatten out. After this point no further training is required. If the **Validation loss** suddenly increases again an the **Training loss** simultaneously goes towards zero, it means that the network is overfitting to the training data. In other words the network is remembering the exact patterns from the training data and no longer generalizes well to unseen data. In this case the training dataset has to be increased.

In this notebook, the training loss curves are plotted using **tensorboard**. However, all the training results are also logged in a csv file in your model folder._____no_output_____

#@markdown ##Play the cell to show a plot of training errors vs. epoch number

if os.path.exists(QC_model_folder):
  os.chdir(QC_model_folder)
  %load_ext tensorboard
  %tensorboard --logdir "$QC_model_folder"
else:
  print("The chosen model or path does not exist. Check if your model_name was saved with a timestamp.")_____no_output_____

## **5.2. Error mapping and quality metrics estimation**
---

This section will display an overlay of the input images ground-truth (solid lines) and predicted boxes (dashed lines). Additionally, the below cell will show the mAP value of the model on the QC data together with plots of the Precision-Recall curves for all the classes in the dataset. If you want to read in more detail about these scores, we recommend [this brief explanation](https://medium.com/@jonathan_hui/map-mean-average-precision-for-object-detection-45c121a31173).

 In a nutshell:

**Precision:** This is the proportion of the correct classifications (true positives) in all the predictions made by the model.

**Recall:** This is the proportion of the detected true positives in all the detectable data.

 The files provided in the "QC_data_folder" should be under a subfolder called validation which contains the images (e.g. as .jpg) and annotations (.csv files)!_____no_output_____

#@markdown ### Provide the path to your quality control dataset.
DEFAULT_LOGS_DIR = "/content/gdrive/MyDrive"
QC_data_folder = "" #@param {type:"string"}
#Result_folder = "" #@param {type:"string"}

# model name and path

#Use_the_current_trained_model = False #@param {type:"boolean"}

#@markdown #####During training, the model files are automatically saved inside a folder named after model_name in section 3. Provide the path to this folder below.
#QC_model_folder = "/content/gdrive/MyDrive/maskrcnn_nucleus20210202T1206" #@param {type:"string"}

#@markdown ###Choose the checkpoint you want to evauluate:
Checkpoint =  8#@param {type:"integer"}

#Load the dataset
dataset_val = ClassDataset()
dataset_val.load_image_csv(QC_data_folder, "Validation")
dataset_val.prepare()

# Activate the (pre-)trained model

detection_min_confidence = 0.35 #@param{type:"number"}
region_proposal_nms_threshold = 0.99 #@param{type:"number"}
resize_mode = "none" #@param["none","square","crop","pad64"]

class InferenceConfig(ClassConfig):
    IMAGE_RESIZE_MODE = resize_mode
    RPN_NMS_THRESHOLD = region_proposal_nms_threshold
    NAME = "nucleus"
    IMAGES_PER_GPU = 1
    # Number of classes (including background)
    DETECTION_MIN_CONFIDENCE = detection_min_confidence
    NUM_CLASSES = len(dataset_val.class_names)  # Background + nucleus
    RPN_ANCHOR_SCALES = (8, 16, 32, 64, 128)
    POST_NMS_ROIS_INFERENCE = 15000
inference_config = InferenceConfig()

# Recreate the model in inference mode
#if Use_the_current_trained_model:
model = modellib.MaskRCNN(mode="inference",
                          config=inference_config,
                          model_dir=QC_model_folder)
# else:
#   model = modellib.MaskRCNN(mode="inference",
#                             config=inference_config,
#                             model_dir=QC_model_folder)

# Get path to saved weights
if Checkpoint < 10:
  qc_model_path = QC_model_folder+"/mask_rcnn_"+QC_model_name[:-13]+"_000"+str(Checkpoint)+".h5"
elif Checkpoint < 100:
  qc_model_path = QC_model_folder+"/mask_rcnn_"+QC_model_name[:-13]+"_00"+str(Checkpoint)+".h5"
elif Checkpoint < 1000:
  qc_model_path = QC_model_folder+"/mask_rcnn_"+QC_model_name[:-13]+"_0"+str(Checkpoint)+".h5"

# Load trained weights
print("Loading weights from ", qc_model_path)
model.load_weights(qc_model_path, by_name=True)

# dataset_val = ClassDataset()
# num_classes = dataset_val.load_image_csv(QC_data_folder, "Validation")
# dataset_val.prepare()

image_id = random.choice(dataset_val.image_ids)
original_image, image_meta, gt_class_id, gt_bbox, gt_mask =\
    modellib.load_image_gt(dataset_val, inference_config,
                           image_id, use_mini_mask=False)

results = model.detect([original_image], verbose=1)
r = results[0]
visualize.display_differences(original_image, gt_bbox, gt_class_id, gt_mask, r['rois'], r['class_ids'], r['scores'], r['masks'], dataset_val.class_names, iou_threshold = 0.8, score_threshold= 0.8)
# visualize.display_instances(original_image, gt_bbox, gt_mask, gt_class_id,
#                             dataset_val.class_names, figsize=(8, 8))

save_image(original_image, "QC_example_data.png", r['rois'], r['masks'],
        r['class_ids'],r['scores'],dataset_val.class_names,
      scores_thresh=0,mode=0,save_dir=QC_model_folder+'/Quality Control')_____no_output_____

##**5.3. Precision-Recall Curve**

 The p-r curve can give a quantification how well the model
Since the training saves model checkpoints for each epoch, you should choose which one you want to use for quality control in the `Checkpoint` box._____no_output_____

#@markdown ###Show the precision-recall curve of the QC data
#@markdown Choose an IoU threshold for the p-r plot (between 0 and 1), ignore that the plot title says AP@50:

iou_threshold = 0.3 #@param{type:"number"}
mAP, precisions, recalls, overlaps = utils.compute_ap(gt_bbox, gt_class_id, gt_mask,
               r['rois'], r['class_ids'], r['scores'], r['masks'],
               iou_threshold=iou_threshold)
visualize.plot_precision_recall(mAP, precisions, recalls)
plt.savefig(QC_model_folder+'/Quality Control/P-R_curve_'+QC_model_name+'.png',bbox_inches='tight',pad_inches=0)

gt_match, pred_match, overlaps = utils.compute_matches(gt_bbox, gt_class_id, gt_mask, r['rois'], r['class_ids'], r['scores'], r['masks'])

#TO DO: Implement for multiclasses
if len(dataset_val.class_names) == 2:
  with open (QC_model_folder+'/Quality Control/QC_results.csv','w') as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(['class','gt instances','True positives','False Negatives', 'IoU threshold', 'mAP'])
    for index in dataset_val.class_names:
      if index != 'BG':
        writer.writerow([index, str(len(gt_match)), str(len(pred_match)), str(len(gt_match)-len(pred_match)), str(iou_threshold), str(mAP)])
        qc_pdf_export()
else:
  print('Your dataset has more than one class. This means certain features may not be enabled. We are working on implementing this section fully for multiple classes.')_____no_output_____

# **6. Using the trained model**

---

In this section the unseen data is processed using the trained model (in section 4). First, your unseen images are uploaded and prepared for prediction. After that your trained model from section 4 is activated and finally saved into your Google Drive._____no_output_____## **6.1. Generate prediction(s) from unseen dataset**
---

The current trained model (from section 4.2) can now be used to process images. If you want to use an older model, untick the **Use_the_current_trained_model** box and enter the name and path of the model to use. Predicted output images are saved in your **Result_folder** folder as restored image stacks (ImageJ-compatible TIFF images).

**`Data_folder`:** This folder should contain the images that you want to use your trained network on for processing.

**`Result_folder`:** This folder will contain the predicted output images._____no_output_____

#@markdown ### Provide the path to your dataset and to the folder where the predictions are saved, then play the cell to predict outputs from your unseen images.
DEFAULT_LOGS_DIR = "/content/gdrive/MyDrive"
Data_folder = "" #@param {type:"string"}
Result_folder = "" #@param {type:"string"}

# model name and path
#@markdown ###Do you want to use the current trained model?
Use_the_current_trained_model = True #@param {type:"boolean"}

#@markdown ###If not, provide the name of the model and path to model folder:
Prediction_model_folder = "" #@param {type:"string"}

if Use_the_current_trained_model:
  Prediction_model_folder = model_path+'/'+new_model_name

#@markdown ###Choose the checkpoint you want to evaluate:
Checkpoint = 8#@param {type:"integer"}

if os.path.exists(Prediction_model_folder+'/Quality Control/class_names.csv'):
  print('Prediction classes detected! The model will predict the following classes:')
  class_names = []
  with open(Prediction_model_folder+'/Quality Control/class_names.csv', 'r') as class_names_csv:
    csvreader = csv.reader(class_names_csv)
    for row in csvreader:
      print(row[0])
      class_names.append(row[0])

detection_min_confidence = 0.1 #@param{type:"number"}
region_proposal_nms_threshold = 0.99 #@param{type:"number"}
resize_mode = "none" #@param["none","square","crop","pad64"]
post_nms_rois = 10000 #@param{type:"integer"}

#Load the dataset
dataset_val = ClassDataset()
dataset_val.load_image_csv(Data_folder, "Validation")
dataset_val.prepare()

  # Activate the (pre-)trained model
class InferenceConfig(ClassConfig):
    IMAGE_RESIZE_MODE = resize_mode
    IMAGE_MIN_DIM = 128
    IMAGE_MAX_DIM = 128
    IMAGE_MIN_SCALE = 2.0
    RPN_NMS_THRESHOLD = region_proposal_nms_threshold
    #DETECTION_NMS_THRESHOLD = 0.0
    NAME = "nucleus"
    IMAGES_PER_GPU = 1
    # Number of classes (including background)
    DETECTION_MIN_CONFIDENCE = detection_min_confidence
    NUM_CLASSES = len(dataset_val.class_names)  # Background + nucleus
    RPN_ANCHOR_SCALES = (8, 16, 32, 64, 128)
    POST_NMS_ROIS_INFERENCE = post_nms_rois

inference_config = InferenceConfig()

# Recreate the model in inference mode
model = modellib.MaskRCNN(mode="inference",
                          config=inference_config,
                          model_dir=Prediction_model_folder)

# Get path to saved weights
if Checkpoint < 10:
  pred_model_path = Prediction_model_folder+"/mask_rcnn_"+os.path.basename(Prediction_model_folder[:-13])+"_000"+str(Checkpoint)+".h5"
elif Checkpoint < 100:
  pred_model_path = Prediction_model_folder+"/mask_rcnn_"+os.path.basename(Prediction_model_folder[:-13])+"_00"+str(Checkpoint)+".h5"
elif Checkpoint < 1000:
  pred_model_path = Prediction_model_folder+"/mask_rcnn_"+os.path.basename(Prediction_model_folder[:-13])+"_0"+str(Checkpoint)+".h5"

# Load trained weights
print("Loading weights from ", pred_model_path)
model.load_weights(pred_model_path, by_name=True)

#@markdown ###Choose how you would like to export the predictions:
Export_mode = "image with class_name,score and mask" #@param["image with bbox, class_name, scores, masks","image with bbox,class_name and score","image with class_name,score and mask","mask with black background"]
if Export_mode == "image with bbox, class_name, scores, masks":
  export_mode = 0
elif Export_mode == "image with bbox,class_name and score":
  export_mode = 1
elif Export_mode == "image with class_name,score and mask":
  export_mode = 2
elif Export_mode == "mask with black background":
  export_mode = 3

file_path = os.path.join(Data_folder, 'Validation')
for input in os.listdir(file_path):
  if input.endswith('.png'):
    image = io.imread(os.path.join(file_path,input))
    results = model.detect([image], verbose=0)
    r = results[0]
    save_image(image, "predicted_"+input, r['rois'], r['masks'],
      r['class_ids'],r['scores'],class_names,
    scores_thresh=0,mode=export_mode,save_dir=Result_folder)
_____no_output_____

## **6.2. Inspect the predicted output**
---
_____no_output_____

#@markdown ##Run this cell to display a randomly chosen input with predicted mask.

detection_min_confidence = 0.1 #@param{type:"number"}
region_proposal_nms_threshold = 0.99 #@param{type:"number"}
resize_mode = "none" #@param["none","square","crop","pad64"]
post_nms_rois = 10000 #@param{type:"integer"}

  # Activate the (pre-)trained model
class InferenceConfig(ClassConfig):
    IMAGE_RESIZE_MODE = resize_mode
    IMAGE_MIN_DIM = 128
    IMAGE_MAX_DIM = 128
    IMAGE_MIN_SCALE = 2.0
    RPN_NMS_THRESHOLD = region_proposal_nms_threshold
    #DETECTION_NMS_THRESHOLD = 0.0
    NAME = "nucleus"
    IMAGES_PER_GPU = 1
    # Number of classes (including background)
    DETECTION_MIN_CONFIDENCE = detection_min_confidence
    NUM_CLASSES = len(dataset_val.class_names)  # Background + nucleus
    RPN_ANCHOR_SCALES = (8, 16, 32, 64, 128)
    POST_NMS_ROIS_INFERENCE = post_nms_rois

inference_config = InferenceConfig()

model = modellib.MaskRCNN(mode="inference",
                          config=inference_config,
                          model_dir=Prediction_model_folder)

model.load_weights(pred_model_path, by_name=True)
example_image = random.choice(os.listdir(os.path.join(Data_folder,'Validation')))

if example_image.endswith('.csv'):
  example_image = example_image[:-4]

display_image = io.imread(file_path+'/'+example_image)
results = model.detect([display_image], verbose=0)

r = results[0]

visualize.display_instances(display_image, r['rois'], r['masks'], r['class_ids'],
                            class_names, r['scores'], ax=get_ax())_____no_output_____

## **6.3. Download your predictions**
---

**Store your data** and ALL its results elsewhere by downloading it from Google Drive and after that clean the original folder tree (datasets, results, trained model etc.) if you plan to train or use new networks. Please note that the notebook will otherwise **OVERWRITE** all files which have the same name._____no_output_____# **7. Version log**
---
**v1.13**:

*   This notebook is new as ZeroCostDL4Mic version 1.13. and is currently a beta version.
*  Further edits to this notebook in future versions will be updated in this cell._____no_output_____#**Thank you for using MaskRCNN**!_____no_output_____


# Notebook from gao-lab/GLUE
Path: experiments/Atlas/s05_verify.ipynb

import os

import anndata
import networkx as nx
import scanpy as sc
from matplotlib import rcParams
from networkx.algorithms.bipartite import biadjacency_matrix

import scglue_____no_output_____scglue.plot.set_publication_params()
rcParams["figure.figsize"] = (3, 3)

PATH = "s05_verify"
os.makedirs(PATH, exist_ok=True)_____no_output_____

# Read data_____no_output_____

rna = anndata.read_h5ad("../../data/dataset/Cao-2020.h5ad")
atac = anndata.read_h5ad("../../data/dataset/Domcke-2020.h5ad")_____no_output_____rna_aligned = anndata.read_h5ad("s04_glue_final/full/rna.h5ad", backed="r")
atac_aligned = anndata.read_h5ad("s04_glue_final/full/atac.h5ad", backed="r")
rna.obsm["X_glue_umap"] = rna_aligned.obsm["X_glue_umap"]
atac.obsm["X_glue_umap"] = atac_aligned.obsm["X_glue_umap"]
del rna_aligned, atac_aligned_____no_output_____prior = nx.read_graphml("s01_preprocessing/full.graphml.gz")
biadj = biadjacency_matrix(prior, atac.var_names, rna.var_names)_____no_output_____atac = anndata.AnnData(
    X=atac.X @ biadj, obs=atac.obs, var=rna.var,
    obsm={"X_glue_umap": atac.obsm["X_glue_umap"]}
)_____no_output_____

# Normalization_____no_output_____

sc.pp.normalize_total(rna)
sc.pp.normalize_total(atac)_____no_output_____sc.pp.log1p(rna)
sc.pp.log1p(atac)_____no_output_____fig = sc.pl.embedding(rna, "X_glue_umap", color="Organ", return_fig=True)
fig.savefig(f"{PATH}/rna_organ.pdf")_____no_output_____fig = sc.pl.embedding(atac, "X_glue_umap", color="tissue", return_fig=True)
fig.savefig(f"{PATH}/atac_tissue.pdf")_____no_output_____

# Cerebrum_____no_output_____

rna_cerebrum = rna[rna.obs["Organ"] == "Cerebrum", :]
atac_cerebrum = atac[atac.obs["tissue"] == "cerebrum", :]_____no_output_____os.makedirs(f"{PATH}/cerebrum", exist_ok=True)_____no_output_____

## Neural progenitor markers_____no_output_____Likely radial glial cells_____no_output_____

MARKERS = [
    "PAX6", "SOX2", "HES1", "VIM",
    "HOPX", "LIFR", "PDGFD", "GLI3"
]_____no_output_____fig = sc.pl.embedding(
    rna_cerebrum, "X_glue_umap", gene_symbols="gene_name", color=MARKERS,
    vmin=0, vmax="p99.9", return_fig=True
)
for i, ax in enumerate(fig.axes):
    if i % 2 == 0:
        ax.set_xlabel("UMAP1")
        ax.set_ylabel("UMAP2")
fig.savefig(f"{PATH}/cerebrum/rna_neural_progenitor.pdf")_____no_output_____fig = sc.pl.embedding(
    atac_cerebrum, "X_glue_umap", gene_symbols="gene_name", color=MARKERS,
    vmin=0, vmax="p99.9", return_fig=True
)
for i, ax in enumerate(fig.axes):
    if i % 2 == 0:
        ax.set_xlabel("UMAP1")
        ax.set_ylabel("UMAP2")
fig.savefig(f"{PATH}/cerebrum/atac_neural_progenitor.pdf")_____no_output_____

## Excitatory neurons markers_____no_output_____

MARKERS = [
    "EOMES", "SOX5", "RUNX1T1", "EPHA3",
    "FAM19A1", "LMO3", "SATB2", "UNC5D"
]_____no_output_____fig = sc.pl.embedding(
    rna_cerebrum, "X_glue_umap", gene_symbols="gene_name", color=MARKERS,
    vmin=0, vmax="p99.9", return_fig=True
)
for i, ax in enumerate(fig.axes):
    if i % 2 == 0:
        ax.set_xlabel("UMAP1")
        ax.set_ylabel("UMAP2")
fig.savefig(f"{PATH}/cerebrum/rna_excitatory_neuron.pdf")_____no_output_____fig = sc.pl.embedding(
    atac_cerebrum, "X_glue_umap", gene_symbols="gene_name", color=MARKERS,
    vmin=0, vmax="p99.9", return_fig=True
)
for i, ax in enumerate(fig.axes):
    if i % 2 == 0:
        ax.set_xlabel("UMAP1")
        ax.set_ylabel("UMAP2")
fig.savefig(f"{PATH}/cerebrum/atac_excitatory_neuron.pdf")_____no_output_____

## Astrocyte markers_____no_output_____

MARKERS = [
    "SLC1A3", "AQP4", "PAX3", "GFAP",
    "RFX4", "PTN", "MMD2", "ATP1A2"
]_____no_output_____fig = sc.pl.embedding(
    rna_cerebrum, "X_glue_umap", gene_symbols="gene_name", color=MARKERS,
    vmin=0, vmax="p99.9", return_fig=True
)
for i, ax in enumerate(fig.axes):
    if i % 2 == 0:
        ax.set_xlabel("UMAP1")
        ax.set_ylabel("UMAP2")
fig.savefig(f"{PATH}/cerebrum/rna_astrocyte.pdf")_____no_output_____fig = sc.pl.embedding(
    atac_cerebrum, "X_glue_umap", gene_symbols="gene_name", color=MARKERS,
    vmin=0, vmax="p99.9", return_fig=True
)
for i, ax in enumerate(fig.axes):
    if i % 2 == 0:
        ax.set_xlabel("UMAP1")
        ax.set_ylabel("UMAP2")
fig.savefig(f"{PATH}/cerebrum/atac_astrocyte.pdf")_____no_output_____

## Oligodendrocyte markers_____no_output_____

MARKERS = [
    "SOX10", "LHFPL3", "PCDH15", "SCN1A",
    "ZNF365", "PDGFRA", "BRINP3", "PDE4B"
]_____no_output_____fig = sc.pl.embedding(
    rna_cerebrum, "X_glue_umap", gene_symbols="gene_name", color=MARKERS,
    vmin=0, vmax="p99.9", return_fig=True
)
for i, ax in enumerate(fig.axes):
    if i % 2 == 0:
        ax.set_xlabel("UMAP1")
        ax.set_ylabel("UMAP2")
fig.savefig(f"{PATH}/cerebrum/rna_oligodendrocyte.pdf")_____no_output_____fig = sc.pl.embedding(
    atac_cerebrum, "X_glue_umap", gene_symbols="gene_name", color=MARKERS,
    vmin=0, vmax="p99.9", return_fig=True
)
for i, ax in enumerate(fig.axes):
    if i % 2 == 0:
        ax.set_xlabel("UMAP1")
        ax.set_ylabel("UMAP2")
fig.savefig(f"{PATH}/cerebrum/atac_oligodendrocyte.pdf")_____no_output_____


# Notebook from biosignalsnotebooks/biosignalsnotebooks
Path: notebookToHtml/oldFiles/opensignalstools_html/Categories/Load/open_txt.ipynb

_____no_output_____<link rel="stylesheet" href="../../styles/theme_style.css">

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">

         Load acquired data from .txt file

_____no_output_____

        Difficulty Level:

                    Tags
                    open/\load/\txt

_____no_output_____
A text file is one of the simplest means to store information, being a format outputted by **OpenSignals**.

In this **Jupyter Notebook** it will be explained how to load/transpose the data inside .txt file to a **Python** list, which consists in a step that precedes all processing operations.
_____no_output_____
_____no_output_____
1 - Importation of the needed packages
_____no_output_____

# Packages used for accessing the URL of the files that contain electrophysiological signals
from bs4 import BeautifulSoup
from urllib.request import urlopen
import re

# Package for reading a file when his URL is known
from wget import download

# Package used for loading data from the input text file
from numpy import loadtxt

# opensignalstools python package
from opensignalstools import plot, generate_time_____no_output_____

2 - Access to electrophysiological signals list
_____no_output_____
2.1 - Enter biosignalsplux url
_____no_output__________no_output_____
2.2 - Navigate through biosignalsplux main page menu and enter in "Signal Samples" page
_____no_output__________no_output_____
2.3 - Interactive buttons for accessing each signal sample file
_____no_output__________no_output_____
2.4 - File url copy (right-click of the mouse in the desired signal file icon)
_____no_output__________no_output_____

copy_link = 'http://biosignalsplux.com/downloads/samples/sensor_samples/biosignalsplux_Blood_Volume_Pulse_(BVP)_Sample.txt'_____no_output_____

3 - Get file from URL (http://biosignalsplux.com/downloads/samples/sensor_samples/biosignalsplux_Blood_Volume_Pulse_(BVP)_Sample.txt)
_____no_output_____

# File download.
txt_file = download(copy_link, out="download_file_name.txt")_____no_output_____

4 - Transposition of data to a Python list
_____no_output_____

data = loadtxt(txt_file)_____no_output_____

5 - Identification of acquisition sampling rate in the file header ("sampling rate" key)
_____no_output_____

# Embedding of .pdf file
from IPython.display import HTML
HTML('')_____no_output_____

_____no_output_____

sampling_rate = 1000_____no_output_____

6 - Generation of time axis for signal plotting
_____no_output_____

time = generate_time(data, sampling_rate)_____no_output_____

7 - Final Output of the loaded data
_____no_output_____

print (data)[[  0.00000000e+00   0.00000000e+00   3.90720000e+04]
 [  1.00000000e+00   0.00000000e+00   3.90880000e+04]
 [  2.00000000e+00   0.00000000e+00   3.90880000e+04]
 ...,
 [  2.72970000e+04   0.00000000e+00   2.21440000e+04]
 [  2.72980000e+04   0.00000000e+00   2.21000000e+04]
 [  2.72990000e+04   0.00000000e+00   2.20440000e+04]]

Each line of the list defines a sample acquired at a specific time instant and each column can be the sample number (*nSeq* ), digital input (*DI* ) or a sample value (*CH1* ), like described in the file header bellow._____no_output_____

# Embedding of .pdf file
from IPython.display import HTML
HTML('')_____no_output_____

_____no_output_____The samples of the signal under analysis are stored at the third entry of each list element (index 2)._____no_output_____

channel_column = 2_____no_output_____

8 - Graphical representation of the signal
_____no_output_____

plot([list(time)], [list(data[:, channel_column])], x_axis_label="Time (s)", y_axis_label="Raw Data")_____no_output_____

*This procedure can be automatically done by **load** function of **opensignalstools** package*_____no_output_____

import opensignalstools as ost

ost.load("http://biosignalsplux.com/downloads/samples/sensor_samples/biosignalsplux_Blood_Volume_Pulse_(BVP)_Sample.txt", channels=[[1]], devices=['00:07:80:3B:46:61'], remote=True)_____no_output_____

            ☌ GitHub Repository

            ☌ Notebook Categories

            ☌ How to install opensignalstools Python package ?

            ☌ Signal Library

            ☌ Notebooks by Difficulty

            ☌ Notebooks by Signal Type

            ☌ Notebooks by Tag

_____no_output_____

from opensignalstools.__notebook_support__ import css_style_apply
css_style_apply().................... CSS Style Applied to Jupyter Notebook .........................


# Notebook from gbarvinok/federatedlearning
Path: examples/tutorials/translations/português/Parte 03 - Ferramentas Avançadas para Execução Remota.ipynb

# Parte 3: Ferramentas Avançadas para Execução Remota

Na última seção nós utilizamos o conceito de Federated Learning para treinarmos um modelo bem simples. Nós fizemos isto apenas chamando *.send()* e *.get()* em nosso modelo. Ou seja, enviando-o para o local onde se encontram os dados de treinamento, atualizando-o, e então trazendo-o de volta. No entando, ao final do exemplo nós percebemos que precisaríamos ir mais além para proteger a privacidade das pessoas. Isto é, queremos calcular a média dos gradientes **antes** de chamar *.get()*. Dessa forma, nunca veremos o gradiente exato de ninguém (protegendo melhor sua privacidade!!!)

Mas, para fazer isso, precisamos de mais algumas coisas:

- usar um ponteiro para enviar um Tensor diretamente para outro Worker

Além disso, enquanto estivermos aqui, vamos aprender sobre algumas operações mais avançadas com tensores, o que nos ajudará tanto com este exemplo quanto com alguns outros no futuro!

Autores:
- Andrew Trask - Twitter: [@iamtrask](https://twitter.com/iamtrask)

Tradução:
- Jeferson Silva - Twitter: [@jefersonnpn](https://twitter.com/jefersonnpn)_____no_output_____

import torch
import syft as sy
hook = sy.TorchHook(torch)_____no_output_____

# Seção 3.1 - De ponteiros para ponteiros

Como você sabe, os objetos do tipo *PointerTensor* se comportam da mesma forma que tensores normais. De fato, eles são *tão parecidos com tensores* que podemos até ter ponteiros que apontam **para** outros ponteiros. Confira!_____no_output_____

bob = sy.VirtualWorker(hook, id='bob')
alice = sy.VirtualWorker(hook, id='alice')_____no_output_____# este é um tensor local.
x = torch.tensor([1,2,3,4])
x_____no_output_____# enviando o tensor local para Bob
x_ptr = x.send(bob)

# agora temos um apontador/ponteiro
x_ptr_____no_output_____# agora podemos ENVIAR O APONTADOR para alice!!!
pointer_to_x_ptr = x_ptr.send(alice)

pointer_to_x_ptr_____no_output_____

### Mas o que acabamos de fazer?

Bom, no exemplo anterior, nós criamos um tensor chamado `x` e o enviamos para Bob, criando um ponteiro em nossa máquina local (`x_ptr`).

Em seguida, nós chamamos `x_ptr.send(alice)` que **envia o ponteiro** para Alice.

Note, isso NÃO move os dados! Em vez disso, move o ponteiro, isto é, o apontador para os dados!! _____no_output_____

# Como você pode ver acima, Bob ainda possui os dados reais (os dados sempre são armazenados com tipo LocalTensor).
bob._objects_____no_output_____# Alice, por outro lado, tem x_ptr!! (observe o ponteiro para Bob)
alice._objects_____no_output_____# podemos usar .get() para recuperar x_ptr de Alice

x_ptr = pointer_to_x_ptr.get()
x_ptr_____no_output_____# e então podemos usar x_ptr para recuperar x de Bob!

x = x_ptr.get()
x_____no_output_____

### Aritmética em Pointero -> Pointero -> Objeto com Dados

Assim como nos ponteiros normais, podemos executar quaisquer operações do PyTorch nesses tensores._____no_output_____

bob._objects_____no_output_____alice._objects_____no_output_____p2p2x = torch.tensor([1,2,3,4,5]).send(bob).send(alice)

y = p2p2x + p2p2x_____no_output_____bob._objects_____no_output_____alice._objects_____no_output_____y.get().get()_____no_output_____bob._objects_____no_output_____alice._objects_____no_output_____p2p2x.get().get()_____no_output_____bob._objects_____no_output_____alice._objects_____no_output_____

# Seção 3.2 - Operações com tensores em cadeia

Na última seção, sempre que usávamos uma operação *.send() ou *.get()*, chamávamos essas operações diretamente do tensor em nossa máquina local. No entanto, se você tiver uma cadeia de ponteiros, às vezes você deseja chamar essas operações como *.get()* ou *.send()* no **último** ponteiro da cadeia (como enviar dados diretamente de um *worker* para outro). Para conseguirmos isso, você deseja usar funções especialmente projetadas para lidar com operações que devem preservar sua privacidade.

Essas operações são:

- `meuPonteiro_para_outroPonteiro.mover(outro_worker)`_____no_output_____

# x agora é um ponteiro para um outro ponteiro para os dados que vivem na máquina de Bob
x = torch.tensor([1,2,3,4,5]).send(bob)_____no_output_____print('  bob:', bob._objects)
print('alice:', alice._objects)_____no_output_____x = x.move(alice)_____no_output_____print('  bob:', bob._objects)
print('alice:', alice._objects)_____no_output_____x_____no_output_____

Excelente! Agora estamos equipados com as ferramentas para executar a **média de gradientes** remota usando um agregador confiável!_____no_output_____# Parabéns!!! - Hora de se juntar a comunidade!

Parabéns por concluir esta etapa do tutorial! Se você gostou e gostaria de se juntar ao movimento em direção à proteção de privacidade, propriedade descentralizada e geração, demanda em cadeia, de dados em IA, você pode fazê-lo das seguintes maneiras!

### Dê-nos uma estrela em nosso repo do PySyft no GitHub

A maneira mais fácil de ajudar nossa comunidade é adicionando uma estrela nos nossos repositórios! Isso ajuda a aumentar a conscientização sobre essas ferramentas legais que estamos construindo.

- [Star PySyft](https://github.com/OpenMined/PySyft)

### Junte-se ao Slack!

A melhor maneira de manter-se atualizado sobre os últimos avanços é se juntar à nossa comunidade! Você pode fazer isso preenchendo o formulário em [http://slack.openmined.org](http://slack.openmined.org)

### Contribua com o projeto!

A melhor maneira de contribuir para a nossa comunidade é se tornando um contribuidor do código! A qualquer momento, você pode acessar a página de *Issues* (problemas) do PySyft no GitHub e filtrar por "Projetos". Isso mostrará todas as etiquetas (tags) na parte superior, com uma visão geral de quais projetos você pode participar! Se você não deseja ingressar em um projeto, mas gostaria de codificar um pouco, também pode procurar mais mini-projetos "independentes" pesquisando problemas no GitHub marcados como "good first issue".

- [Projetos do PySyft](https://github.com/OpenMined/PySyft/issues?q=is%3Aopen+is%3Aissue+label%3AProject)
- [Etiquetados como Good First Issue](https://github.com/OpenMined/PySyft/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)

### Doar

Se você não tem tempo para contribuir com nossa base de códigos, mas ainda deseja nos apoiar, também pode se tornar um Apoiador em nosso Open Collective. Todas as doações vão para hospedagem na web e outras despesas da comunidade, como hackathons e meetups!

[Página do Open Collective do OpenMined](https://opencollective.com/openmined)_____no_output_____