# Notebook from Ezibenroc/calibration_analysis
Path: dahu/blas/extraction_archives.ipynb

# Extraction and plot of monitoring information from DGEMM archives_____no_output_____

<code>
archive_names = [
    'nancy_2021-04-12_2965244.zip',
    'nancy_2021-04-19_2975590.zip',
    'nancy_2021-04-20_2977732.zip',
]_____no_output_____import io
import zipfile
import requests
import pandas
import re
import os
import numpy
from plotnine import *
import plotnine
plotnine.options.figure_size = (12, 8)
import cashew
print(cashew.__version__)
print(cashew.__git_version__)
from cashew import archive_extraction as ae

def download_file(archive_name, base_url='https://gricad-gitlab.univ-grenoble-alpes.fr/Ezibenroc/g5k_test/-/raw/master/data/', directory='/tmp'):
    data = requests.get(base_url + archive_name)
    if data.status_code != 200:
        raise ValueError(f'Could not download the file, got an error {data.status_code}')
    path = os.path.join(directory, archive_name)
    with open(path, 'wb') as f:
        f.write(data.content)
    return path0.2.10
697ee1f882418b003dd238ab6f7a1997176ac874
archives = [download_file(name) for name in archive_names]
archives_____no_output_____import re
import numpy

dataframes = []
for f in archives:
    tmp = ae. read_monitoring(f)
    tmp['expfile'] = ae.read_yaml(f, 'info.yaml')['expfile'][0]
    dataframes.append(tmp)

df = pandas.concat(dataframes)
df.head()_____no_output_____def plot_evolution(df, y_val):
    df = df.copy()
    for jobid in df['jobid'].unique():
        df.loc[df['jobid'] == jobid, 'start_job'] = df[df['jobid'] == jobid]['timestamp'].min()
    df['timestamp'] -= df['start_job']
    start = df[['jobid', 'start_exp', 'start_job']].drop_duplicates()
    stop = df[['jobid', 'stop_exp', 'start_job']].drop_duplicates()
    start['x'] = start['start_exp'] - start['start_job']
    stop['x'] = stop['stop_exp'] - stop['start_job']
    lines = pandas.concat([start, stop])
    measure = start.copy()
    measure['start'] = measure['x'] + 120
    measure['stop'] = measure['x'] + 120 + 240
    return (ggplot(df)
           + geom_rect(aes(xmin='start', xmax='stop'), ymin=float('-inf'), ymax=float('+inf'), data=measure, alpha=0.2)
           + geom_line(aes(x='timestamp', y=y_val, color='factor(cpu)'))
           + theme_bw()
           + geom_vline(aes(xintercept='x'), data=lines, linetype='dashed')
           + geom_vline(aes(xintercept='x'), data=lines, linetype='dashed')
           + facet_wrap(['jobid'], labeller='label_both')
           + labs(color='CPU', x='Timestamp (s)')
           + guides(color = guide_legend(override_aes = {'alpha': 1, 'size': 2}))

    )

def filter_dates(df, *dates):
    dates = {pandas.to_datetime(d).date() for d in dates}
    return df[df['date'].isin(dates)]

def filter_nodes(df, *nodes):
    nodes = set(nodes)
    return df[df['node'].isin(nodes)]_____no_output_____cluster = df['cluster'].unique()
assert len(cluster) == 1
cluster = cluster[0]

for node in [33,34,35,36]:
    tmp = filter_nodes(df[df['kind'] == 'power_cpu'], node)
    print(plot_evolution(tmp, 'value')
        + ggtitle(f'Power consumption evolution on node {cluster}-{node} (package)')
        + ylab('Power (W)')
        + expand_limits(y=0)
    )_____no_output_____for node in [33,34,35,36]:
    tmp = filter_nodes(df[df['kind'] == 'frequency'], node)
    print(plot_evolution(tmp, 'value')
        + ggtitle(f'CPU frequency on node {cluster}-{node}')
        + ylab('Frequency (GHz)')
        + expand_limits(y=0)
    )_____no_output_____
</code>


# Notebook from hsun3163/xqtl-pipeline
Path: code/data_preprocessing/reference_data.ipynb

# Reference data standardization

This module provides reference data download, indexing and preprocessing (if necessary), in preparation for use throughout the pipeline.

We have included the PDF document compiled by data standardization subgroup in the [on Google Drive](https://drive.google.com/file/d/1R5sw5o8vqk_mbQQb4CGmtH3ldu1T3Vu0/view?usp=sharing) as well as on [ADSP Dashboard](https://www.niagads.org/adsp/content/adspgcadgenomeresources-v2pdf). It contains the reference data to use for the project._____no_output_____## Overview

This module is based on the [TOPMed workflow from Broad](https://github.com/broadinstitute/gtex-pipeline/blob/master/TOPMed_RNAseq_pipeline.md). The reference data after we process it (details see Methods section and the rest of the analysis) can be found [in this folder on Google Drive](https://drive.google.com/drive/folders/19fmoII8yS7XE7HFcMU4OfvC2bL1zMD_P). Specifically, the list of reference files to be used are:

1. `GRCh38_full_analysis_set_plus_decoy_hla.noALT_noHLA_noDecoy_ERCC.{dict,fasta,fasta.fai}`
2. `Homo_sapiens.GRCh38.103.chr.reformatted.collapse_only.gene.ERCC.gtf` for stranded protocol, and `Homo_sapiens.GRCh38.103.chr.reformatted.gene.ERCC.gtf` for unstranded protocol.
3. Everything under `STAR_Index` folder
4. Everything under `RSEM_Index` folder

## Methods

Workflows implemented include:

### Convert transcript feature file gff3 to gtf

- Input: an uncompressed gff3 file.(i.e. can be view via cat)
- Output: a gtf file.

### Collapse transcript features into genes

- Input: a gtf file.
- Output: a gtf file with collapesed gene model.

### Generate STAR index based on gtf and reference fasta

- Input: a gtf file and an acompanying fasta file.
- Output: A folder of STAR index.

### Generate RSEM index based on gtf and reference fasta

- Input: a gtf file and an acompanying fasta file.
- Output: A folder of RSEM index._____no_output_____## Example commands

To download reference data:_____no_output_____

sos run reference_data.ipynb download_hg_reference --cwd reference_data
sos run reference_data.ipynb download_gene_annotation --cwd reference_data
sos run reference_data.ipynb download_ercc_reference --cwd reference_data_____no_output_____

To format reference data:_____no_output_____

sos run reference_data.ipynb hg_reference \
    --cwd reference_data \
    --ercc-reference reference_data/ERCC92.fa \
    --hg-reference reference_data/GRCh38_full_analysis_set_plus_decoy_hla.fa \
    --container container/rna_quantification.sif_____no_output_____
sos run pipeline/reference_data.ipynb hg_gtf \
    --cwd reference_data \
    --hg-gtf /mnt/mfs/statgen/snuc_pseudo_bulk/data/reference_data/genes.gtf \
    --hg-reference data/reference_data/GRCh38_full_analysis_set_plus_decoy_hla.noALT_noHLA_noDecoy_ERCC.fasta \
    --containers containers/rna_quantification.sif -J 1 -q csg -c csg.yml &_____no_output_____

To format gene feature data:_____no_output_____

sos run reference_data.ipynb gene_annotation \
    --cwd reference_data \
    --ercc-gtf reference_data/ERCC92.gtf \
    --hg-gtf reference_data/Homo_sapiens.GRCh38.103.chr.gtf \
    --hg-reference reference_data/GRCh38_full_analysis_set_plus_decoy_hla.noALT_noHLA_noDecoy_ERCC.fasta \
    --container container/rna_quantification.sif --stranded_____no_output_____

**Notice that for un-stranded RNA-seq protocol please use switch `--no-stranded` to the command above instead of `--stranded`. More details can be found later in the document.**_____no_output_____To generate STAR index using the GTF annotation file before gene model collapse:_____no_output_____

sos run reference_data.ipynb STAR_index \
    --cwd reference_data \
    --hg-reference reference_data/GRCh38_full_analysis_set_plus_decoy_hla.noALT_noHLA_noDecoy_ERCC.fasta \
    --hg-gtf reference_data/Homo_sapiens.GRCh38.103.chr.reformatted.ERCC.gtf \
    --container container/rna_quantification.sif \
    --mem 40G_____no_output_____

**Notice that command above requires at least 40G of memory, and takes quite a while to complete**._____no_output_____To generate RSEM index:_____no_output_____

sos run reference_data.ipynb RSEM_index \
    --cwd reference_data \
    --hg-reference reference_data/GRCh38_full_analysis_set_plus_decoy_hla.noALT_noHLA_noDecoy_ERCC.fasta \
    --hg-gtf reference_data/Homo_sapiens.GRCh38.103.chr.reformatted.ERCC.gtf \
    --container container/rna_quantification.sif \
    --mem 40G_____no_output_____

## Command interface_____no_output_____

sos run reference_data.ipynb -husage: sos run reference_data.ipynb [workflow_name | -t targets] [options] [workflow_options]
  workflow_name:        Single or combined workflows defined in this script
  targets:              One or more targets to generate
  options:              Single-hyphen sos parameters (see "sos run -h" for details)
  workflow_options:     Double-hyphen workflow-specific parameters

Workflows:
  download_hg_reference
  download_gene_annotation
  download_ercc_reference
  gff3_to_gtf
  hg_reference
  hg_gtf
  ercc_gtf
  gene_annotation
  STAR_index
  RSEM_indexing

Global Workflow Options:
  --cwd VAL (as path, required)
                        The output directory for generated files.
  --job-size 1 (as int)
                        For cluster jobs, number commands to run per job
  --walltime 5h
                        Wall clock time expected
  --mem 16G
                        Memory expected
  --numThreads 8 (as int)
                        Number of threads
  --container ''
                        Software container option

Sections
  download_hg_reference:
  download_gene_annotation:
  download_ercc_reference:
  gff3_to_gtf:
    Workflow Options:
      --gff3-file VAL (as path, required)
  hg_reference_1:
    Workflow Options:
      --hg-reference VAL (as path, required)
                        Path to HG reference file
  hg_reference_2:
    Workflow Options:
      --ercc-reference VAL (as path, required)
  hg_reference_3:
  hg_gtf_1:
    Workflow Options:
      --hg-reference VAL (as path, required)
      --hg-gtf VAL (as path, required)
  hg_gtf_2:
    Workflow Options:
      --[no-]collapse-only (default to False)
                        Use this for stranded protocol (optional)
  ercc_gtf:
    Workflow Options:
      --ercc-gtf VAL (as path, required)
  gene_annotation:
  STAR_index:
    Workflow Options:
      --hg-gtf VAL (as path, required)
      --hg-reference VAL (as path, required)
      --sjdbOverhang 100 (as int)
                        Specifies the length of the genomic sequence around the
                        annotated junction to be used in constructing the splice
                        junctions database. Ideally, this length should be equal
                        to the ReadLength-1, where ReadLength is the length of
                        the reads. Default choice follows from TOPMed pipeline
                        recommendation.
  RSEM_indexing:
    Workflow Options:
      --hg-gtf VAL (as path, required)
      --hg-reference VAL (as path, required)
[global]
# The output directory for generated files.
parameter: cwd = path
# For cluster jobs, number commands to run per job
parameter: job_size = 1
# Wall clock time expected
parameter: walltime = "5h"
# Memory expected
parameter: mem = "16G"
# Number of threads
parameter: numThreads = 8
# Software container option
parameter: container = ""
cwd = path(f'{cwd:a}')
from sos.utils import expand_size_____no_output_____

## Data download_____no_output_____

[download_hg_reference]
output: f"{cwd:a}/GRCh38_full_analysis_set_plus_decoy_hla.fa"
download: dest_dir = cwd
    ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/GRCh38_reference_genome/GRCh38_full_analysis_set_plus_decoy_hla.fa_____no_output_____[download_gene_annotation]
output: f"{cwd:a}/Homo_sapiens.GRCh38.103.chr.gtf"
download: dest_dir = cwd, decompress=True
    http://ftp.ensembl.org/pub/release-103/gtf/homo_sapiens/Homo_sapiens.GRCh38.103.chr.gtf.gz_____no_output_____[download_ercc_reference]
output: f"{cwd:a}/ERCC92.gtf", f"{cwd:a}/ERCC92.fa"
download: dest_dir = cwd, decompress=True
    https://tools.thermofisher.com/content/sfs/manuals/ERCC92.zip_____no_output_____

## GFF3 to GTF formatting_____no_output_____

[gff3_to_gtf]
parameter: gff3_file = path
input: gff3_file
output: f'{cwd}/{_input:bn}.gtf'
task: trunk_workers = 1, trunk_size = job_size, walltime = walltime,  mem = mem, tags = f'{step_name}_{_output:bn}'
bash: container=container, expand= "${ }", stderr = f'{_output:n}.stderr', stdout = f'{_output:n}.stdout'
        gffread ${_input} -T -o ${_output}_____no_output_____

## HG reference file preprocessing
1. Remove the HLA/ALT/Decoy record from the fasta -- because none of the downstreams RNA-seq calling pipeline component can handle them properly.
2. Adding in ERCC information to the fasta file -- even if ERCC is not included in the RNA-seq library it does not harm to add them.
3. Generating index for the fasta file_____no_output_____

[hg_reference_1 (HLA ALT Decoy removal)]
# Path to HG reference file
parameter: hg_reference = path
input: hg_reference
output:  f'{cwd}/{_input:bn}.noALT_noHLA_noDecoy.fasta'
task: trunk_workers = 1, trunk_size = job_size, walltime = walltime,  mem = mem, tags = f'{step_name}_{_output:bn}'
python: expand = "${ }", stderr = f'{_output}.stderr', stdout = f'{_output}.stdout', container = container
    with open('${_input}', 'r') as fasta:
        contigs = fasta.read()
        contigs = contigs.split('>')
        contig_ids = [i.split(' ', 1)[0] for i in contigs]

        # exclude ALT, HLA and decoy contigs
        filtered_fasta = '>'.join([c for i,c in zip(contig_ids, contigs)
        if not (i[-4:]=='_alt' or i[:3]=='HLA' or i[-6:]=='_decoy')])

    with open('${_output}', 'w') as fasta:
        fasta.write(filtered_fasta)_____no_output_____[hg_reference_2 (merge with ERCC reference)]
parameter: ercc_reference = path
output: f'{cwd}/{_input:bn}_ERCC.fasta'
task: trunk_workers = 1, trunk_size = job_size, walltime = walltime,  mem = mem, tags = f'{step_name}_{_output:bn}'
bash: expand = "${ }", stderr = f'{_output[0]}.stderr', stdout = f'{_output}.stdout', container = container
    sed 's/ERCC-/ERCC_/g' ${ercc_reference} >  ${ercc_reference:n}.patched.fa
    cat ${_input} ${ercc_reference:n}.patched.fa > ${_output}_____no_output_____[hg_reference_3 (index the fasta file)]
output: f'{cwd}/{_input:bn}.dict'
task: trunk_workers = 1, trunk_size = job_size, walltime = walltime,  mem = mem, tags = f'{step_name}_{_output:bn}'
bash: expand = "${ }", stderr = f'{_output}.stderr', stdout = f'{_output}.stdout', container = container
    samtools faidx ${_input}
    java -jar /opt/picard-tools/picard.jar \
    CreateSequenceDictionary \
    R=${_input} \
    O=${_output}_____no_output_____

## Transcript and gene model reference processing_____no_output_____This step modify the `gtf` file for following reasons:

1. RSEM require GTF input to have the same chromosome name format (with `chr` prefix) as the fasta file. **although for STAR, this problem can be solved by the now commented --sjdbGTFchrPrefix "chr" option, we have to add `chr` to it for use with RSEM**.
2. Gene model collapsing script `collapse_annotation.py` from GTEx require the gtf have `transcript_type` instead `transcript_biotype` in its annotation. We rename it here, although **this problem can also be solved by modifying the collapse_annotation.py while building the docker, since we are doing 1 above we think it is better to add in another customization here.**
3. Adding in ERCC information to the `gtf` reference.

We may reimplement 1 and 2 if the problem with RSEM is solved, or when RSEM is no longer needed._____no_output_____

[hg_gtf_1 (add chr prefix to gtf file)]
parameter: hg_reference = path
parameter: hg_gtf = path
input: hg_reference, hg_gtf
output: f'{cwd}/{_input[1]:bn}.reformatted.gtf'
task: trunk_workers = 1, trunk_size = job_size, walltime = walltime,  mem = mem, tags = f'{step_name}_{_output:bn}'
R: expand = "${ }", stderr = f'{_output}.stderr', stdout = f'{_output}.stdout', container = container
    library("readr")
    library("stringr")
    library("dplyr")
    options(scipen = 999)
    con <- file("${_input[0]}","r")
    fasta <- readLines(con,n=1)
    close(con)
    gtf = read_delim("${_input[1]}", "\t",  col_names  = F, comment = "#", col_types="ccccccccc")
    if(!str_detect(fasta,">chr")) {
        gtf_mod = gtf%>%mutate(X1 = str_remove_all(X1,"chr"))
    } else if (!any(str_detect(gtf$X1[1],"chr"))) {
        gtf_mod = gtf%>%mutate(X1 = paste0("chr",X1))
    } else (gtf_mod = gtf)
    if(any(str_detect(gtf_mod$X9, "transcript_biotype"))) {
      gtf_mod = gtf_mod%>%mutate(X9 = str_replace_all(X9,"transcript_biotype","transcript_type"))
    }
    gtf_mod%>%write.table("${_output}",sep = "\t",quote = FALSE,col.names = F,row.names = F)_____no_output_____

**Text below is taken from https://github.com/broadinstitute/gtex-pipeline/tree/master/gene_model**

Gene-level expression and eQTLs from the GTEx project are calculated based on a collapsed gene model (i.e., combining all isoforms of a gene into a single transcript), according to the following rules:

1. Transcripts annotated as “retained_intron” or “read_through” are excluded. Additionally, transcripts that overlap with annotated read-through transcripts may be blacklisted (blacklists for GENCODE v19, 24 & 25 are provided in this repository; no transcripts were blacklisted for v26).
2. The union of all exon intervals of each gene is calculated.
3. Overlapping intervals between genes are excluded from all genes.

The purpose of step 3 is primarily to exclude overlapping regions from genes annotated on both strands, which can't be unambiguously quantified from unstranded RNA-seq (GTEx samples were sequenced using an unstranded protocol). For stranded protocols, this step can be skipped by adding the `--collapse_only` flag.

Further documentation is available on the [GTEx Portal](https://gtexportal.org/home/documentationPage#staticTextAnalysisMethods)._____no_output_____

[hg_gtf_2 (collapsed gene model)]
parameter: stranded = bool
output: f'{_input:n}{".collapse_only" if stranded else ""}.gene.gtf'
task: trunk_workers = 1, trunk_size = job_size, walltime = walltime, mem = mem, tags = f'{step_name}_{_output:bn}'
bash: expand = "${ }", stderr = f'{_output}.stderr', stdout = f'{_output}.stdout', container = container
    collapse_annotation.py ${"--collapse_only" if stranded else ""} ${_input} ${_output}_____no_output_____[ercc_gtf (Preprocess ERCC gtf file)]
parameter: ercc_gtf = path
input: ercc_gtf
output: f'{cwd}/{_input:bn}.genes.patched.gtf'
task: trunk_workers = 1, trunk_size = job_size, walltime = walltime,  mem = mem, tags = f'{step_name}_{_output:bn}'
python: expand = "${ }", stderr = f'{_output}.stderr', stdout = f'{_output}.stdout', container = container
    with open('${_input}') as exon_gtf, open('${_output}', 'w') as gene_gtf:
        for line in exon_gtf:
            f = line.strip().split('\t')
            f[0] = f[0].replace('-','_')  # required for RNA-SeQC/GATK (no '-' in contig name)

            attr = f[8]
            if attr[-1]==';':
                attr = attr[:-1]
            attr = dict([i.split(' ') for i in attr.replace('"','').split('; ')])
            # add gene_name, gene_type
            attr['gene_name'] = attr['gene_id']
            attr['gene_type'] = 'ercc_control'
            attr['gene_status'] = 'KNOWN'
            attr['level'] = 2
            for k in ['id', 'type', 'name', 'status']:
                attr['transcript_'+k] = attr['gene_'+k]

            attr_str = []
            for k in ['gene_id', 'transcript_id', 'gene_type', 'gene_status', 'gene_name',
                'transcript_type', 'transcript_status', 'transcript_name']:
                attr_str.append('{0:s} "{1:s}";'.format(k, attr[k]))
            attr_str.append('{0:s} {1:d};'.format('level', attr['level']))
            f[8] = ' '.join(attr_str)

            # write gene, transcript, exon
            gene_gtf.write('\t'.join(f[:2]+['gene']+f[3:])+'\n')
            gene_gtf.write('\t'.join(f[:2]+['transcript']+f[3:])+'\n')
            f[8] = ' '.join(attr_str[:2])
            gene_gtf.write('\t'.join(f[:2]+['exon']+f[3:])+'\n')_____no_output_____[gene_annotation]
input: output_from("hg_gtf_1"), output_from("hg_gtf_2"), output_from("ercc_gtf")
output: f'{cwd}/{_input[0]:bn}.ERCC.gtf', f'{cwd}/{_input[1]:bn}.ERCC.gtf'
task: trunk_workers = 1, trunk_size = job_size, walltime = walltime,  mem = mem, tags = f'{step_name}_{_output:bn}'
bash: expand = "${ }", stderr = f'{_output[0]}.stderr', stdout = f'{_output[0]}.stdout', container = container
    cat ${_input[0]} ${_input[2]} > ${_output[0]}
    cat ${_input[1]} ${_input[2]} > ${_output[1]}_____no_output_____

## Generating index file for `STAR`

This step generate the index file for STAR alignment. This file just need to generate once and can be re-used.

**At least 40GB of memory is needed**.

### Step Inputs

* `gtf` and `fasta`: path to reference sequence. Both of them needs to be unzipped. `gtf` should be the one prior to collapse by gene.
* `sjdbOverhang`: specifies the length of the genomic sequence around the annotated junction to be used in constructing the splice junctions database. Ideally, this length should be equal to the ReadLength-1, where ReadLength is the length of the reads. We use 100 here as recommended by the TOPMed pipeline. See here for [some additional discussions](https://groups.google.com/g/rna-star/c/h9oh10UlvhI/m/BfSPGivUHmsJ).

### Step Output

* Indexing file stored in `{cwd}/STAR_index`, which will be used by `STAR`_____no_output_____

[STAR_index]
parameter: hg_gtf = path
parameter: hg_reference = path
# Specifies the length of the genomic sequence around the annotated junction to be used in constructing the splice junctions database. Ideally, this length should be equal to the ReadLength-1, where ReadLength is the length of the reads.
# Default choice follows from TOPMed pipeline recommendation.
parameter: sjdbOverhang = 100
fail_if(expand_size(mem) < expand_size('40G'), msg = "At least 40GB of memory is required for this step")
input: hg_reference, hg_gtf
output: f"{cwd}/STAR_Index/chrName.txt", f"{cwd}/STAR_Index/Log.out",
        f"{cwd}/STAR_Index/transcriptInfo.tab", f"{cwd}/STAR_Index/exonGeTrInfo.tab",
        f"{cwd}/STAR_Index/SAindex", f"{cwd}/STAR_Index/SA", f"{cwd}/STAR_Index/genomeParameters.txt",
        f"{cwd}/STAR_Index/chrStart.txt", f"{cwd}/STAR_Index/sjdbList.out.tab",
        f"{cwd}/STAR_Index/exonInfo.tab", f"{cwd}/STAR_Index/sjdbList.fromGTF.out.tab",
        f"{cwd}/STAR_Index/chrLength.txt", f"{cwd}/STAR_Index/sjdbInfo.txt",
        f"{cwd}/STAR_Index/Genome", f"{cwd}/STAR_Index/chrNameLength.txt",
        f"{cwd}/STAR_Index/geneInfo.tab"
task: trunk_workers = 1, trunk_size = job_size, walltime = walltime, mem = mem, tags = f'{step_name}_{_output[0]:bd}'
bash: container=container, expand= "${ }", stderr = f'{_output[0]}.stderr', stdout = f'{_output[0]}.stdout'
    STAR --runMode genomeGenerate \
         --genomeDir ${_output:d} \
         --genomeFastaFiles ${_input[0]} \
         --sjdbGTFfile ${_input[1]} \
         --sjdbOverhang ${sjdbOverhang} \
         --runThreadN ${numThreads} #--sjdbGTFchrPrefix "chr" _____no_output_____

## Generating index file for `RSEM`

This step generate the indexing file for `RSEM`. This file just need to generate once.

### Step Inputs

* `gtf` and `fasta`: path to reference sequence. `gtf` should be the one prior to collapse by gene.
* `sjdbOverhang`: specifies the length of the genomic sequence around the annotated junction to be used in constructing the splice junctions database. Ideally, this length should be equal to the ReadLength-1, where ReadLength is the length of the reads.

### Step Outputs
* Indexing file stored in `RSEM_index_dir`, which will be used by `RSEM`_____no_output_____

[RSEM_index]
parameter: hg_gtf = path
parameter: hg_reference = path
input: hg_reference, hg_gtf
output: f"{cwd}/RSEM_Index/rsem_reference.n2g.idx.fa", f"{cwd}/RSEM_Index/rsem_reference.grp",
        f"{cwd}/RSEM_Index/rsem_reference.idx.fa", f"{cwd}/RSEM_Index/rsem_reference.ti",
        f"{cwd}/RSEM_Index/rsem_reference.chrlist", f"{cwd}/RSEM_Index/rsem_reference.seq",
        f"{cwd}/RSEM_Index/rsem_reference.transcripts.fa"
task: trunk_workers = 1, trunk_size = job_size, walltime = walltime, mem = mem, tags = f'{step_name}_{_output[0]:bd}'
bash: container=container, expand= "${ }", stderr = f'{_output[0]}.stderr', stdout = f'{_output[0]}.stdout'
    rsem-prepare-reference \
            ${_input[0]} \
            ${_output[1]:n} \
            --gtf ${_input[1]} \
            --num-threads ${numThreads}_____no_output_____

## Generation of RefFlat file
This file is needed for picard CollectRnaSeqMetrics module, which in turn
>produces metrics describing the distribution of the bases within the transcripts. It calculates the total numbers and the fractions of nucleotides within specific genomic regions including untranslated regions (UTRs), introns, intergenic sequences (between discrete genes), and peptide-coding sequences (exons). This tool also determines the numbers of bases that pass quality filters that are specific to Illumina data (PF_BASES)._____no_output_____

[RefFlat_generation]
parameter: hg_gtf = path
input: hg_gtf
output: f'{_input:n}.ref.flat'
bash: container=container, expand= "${ }", stderr = f'{_output}.stderr', stdout = f'{_output}.stdout'
    gtfToGenePred ${_input}  ${_output}.tmp -genePredExt -geneNameAsName2
    awk -F'\t' -v OFS="\t" '{$1=$12 OFS $1;}7' ${_output}.tmp | cut -f 1-11 > ${_output}
    rm ${_output}.tmp_____no_output_____


# Notebook from Ninon-Dupeuble/Genius
Path: Math/MOOC_Verhulst_partie_3-2.ipynb

$\underline{\text{Modèle de Verhulst, partie 3 : introduction au chaos et fractales de Liapounov}}$_____no_output_____Bienvenue dans cette troisième et dernière partie du modèle de Verhulst ! Je te conseille de d'abord commencer par la vidéo ou le fichier PDF associés à cette partie, avant de te lancer dans les joies de la simulation !

Cet outil va te permettre d'exécuter du code Python facilement, à travers ce que j'ai déjà codé, ou de petits exercices où tu dois toi-même coder !

L'objectif est que tu découvres la puissance de l'informatique pour ce genre de calculs et pour l'intuition qu'elle peut t'apporter, mais aussi ses limites. Bonne découverte !

PS : Au début, appuie sur Cell->Run All pour tout lancer. Pour activer une cellule de code particulière (comme après une modification), clique dessus et appuie sur Cells->Run Cells, ou Shift+Enter ;)_____no_output_____

import os
import sys
import random as rd
import math
import numpy as np
import skimage.io
import matplotlib
import matplotlib.pyplot as plt
import time

from bokeh.io import show, output_notebook
from bokeh.plotting import figure
from bokeh.layouts import column, row
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
from bokeh.plotting import ColumnDataSource, output_file, show

output_notebook(hide_banner=True)

def f_mu(mu):
    return lambda x:mu*x*(1-x)_____no_output_____

$\underline{\text{Un chaos compréhensible : détermination aléatoire d'un antécédent de tout }x\in[0,1]}$

Nous avons pu voir que la dynamique de $f_4$ est chaotique. Néanmoins, le chaos qui y est associé répond à un certain comportement "prévisible". Nous allons donc l'illustrer ici grâce à un algorithme.

Celui-ci va nous permettre, pour tous $x\in[0,1]$, $\epsilon > 0$ et $N\in \mathbb{N}^{*}$ de trouver un $u_0$ tel que :
$$
f^N_4(u_0)\in[x-\epsilon, x+\epsilon]
$$

Ainsi, malgré la sensibilité aux conditions initiales, nous pouvons trouver une valeur initiale dont nous connaissons, à une précision choisie, la valeur de la suite associée à une itération choisie._____no_output_____

#Un chaos prédictible
def convergence_vers_x(x,preci,nb_etape):
    print("On note :")
    print("- x la valeur que l'on souhaite atteindre")
    print("- preci l'entier tel que l'on souhaite atteindre x à 10^{-preci} près")
    print("- nb_etape le nombre d'étapes de la convergence")
    p=preci
    preci=int(p*np.log(10)/np.log(2))+2
    a=np.arcsin(np.sqrt(x))/np.pi
    b=int((2**preci)*a)
    c=rd.randint(0,2**nb_etape)/(2**nb_etape)+b/(2**(nb_etape+preci))
    liste_valeurs=[np.sin(c*np.pi)**2]
    for i in range(nb_etape):
        liste_valeurs.append(f_4(liste_valeurs[-1]))
    print("A partir de u0={}, la valeur finale obtenue est de {} au lieu de {} après {} étapes.".format(liste_valeurs[0], liste_valeurs[-1], x, nb_etape))

    fig = figure(width=470, height=300, title="Convergence vers la valeur choisie")
    fig.line(list(range(nb_etape+1)), liste_valeurs, legend_label="Evolution de la suite", color='blue')
    fig.x([nb_etape],[x],legend_label="Valeur attendue", color="red")
    fig.legend.location = "top_right"

    fig1 = figure(width=470, height=300, y_axis_type="log", title="Ecart de la convergence vers la valeur choisie")
    fig1.line([0,nb_etape],[10**(-p), 10**(-p)], legend_label="Précision attendue", color="black")
    fig1.x(list(range(nb_etape+1)), abs(x-np.array(liste_valeurs)), legend_label="Evolution de l'écart à x", color='blue')
    fig1.legend.location = "bottom_left"
    show(row(fig, fig1))_____no_output_____interact(convergence_vers_x, x=widgets.FloatSlider(min=0, max=1, step=0.05, value=0.5, continuous_update=False), preci=widgets.IntSlider(min=1, max=15, step=1, value=4, continuous_update=False), nb_etape=widgets.IntSlider(min=1, max=50, step=1, value=10, continuous_update=False))_____no_output_____

On peut observer que les résultats obtenus correspondent bien en général aux conditions décidées lorsque $nb_{etape}\leq 20$. En effet, pour $nb_{etape}\geq 20$, les erreurs de calcul successives, alliées à la sensibilité aux conditions initiales, mènent à l'obtention de résultats pas aussi précis que ceux qui étaient atendus. Néanmoins, dans un cadre théorique, cet algorithme fonctionne à tous les coups : c'est la limite entre la théorie et la réalité matérielle !_____no_output_____$\underline{\text{Le diagramme de bifurcation complet}}$_____no_output_____Nous sommes enfin prêts pour admirer le diagramme de bifurcation complet : les points fixes attractifs, puis le cascade de doublement de périodes, et enfin toutes les choses étranges qui se passent après !_____no_output_____

def derniers_elements(mu, x0, epsi):
    liste=[x0]
    x=x0
    for k in range(200):
        x=f_mu(mu)(x)
        liste.append(x)
    if mu<=3:
        return np.array(liste[-1:])
    for k in range(1,200):
        if abs(liste[-1-k]-liste[-1])<epsi:
            return np.array(liste[-k:])

def affichage_global(nb_points):
    print("Attention, le calcul peut être un peu long avec un grand nombre de points !")
    l=np.linspace(0,4,nb_points)
    liste_x=[]
    liste_y=[]
    plt.close()
    for mu in l:
        try:
            cycle=derniers_elements(mu,0.1,0.001)
            for k in range(cycle.shape[0]):
                liste_x.append(mu)
                liste_y.append(cycle[k])
        except:
            pass
    fig = figure(width=900, height=500, title="Diagramme de bifurcation")
    fig.x(liste_x,liste_y, size=1)
    show(fig)_____no_output_____interact(affichage_global, nb_points=widgets.IntSlider(min=2000, max=100000, step=1000, value=2000, continuous_update=False))_____no_output_____

N'hésite pas à utiliser la fonction Zoom pour regarder plus précisément ce qu'il se passe entre 3 et 4 !_____no_output_____$\underline{\text{Les fractales de Liapounov}}$_____no_output_____

def expo_lyapunov(a,b,lim):
    x=0.5
    l=0
    x=f_mu(a)(x)
    l+=np.log(abs(b*(1-2*x)))
    x=f_mu(b)(x)
    for i in range(1,lim):
        l+=np.log(abs(a*(1-2*x)))
        x=f_mu(a)(x)
        l+=np.log(abs(b*(1-2*x)))
        x=f_mu(b)(x)
    return l/(2*lim-1)

def fractale(nb):
    print("Attention, le temps de calcul peut être très important !\nEn cas de problème, appuie sur Kernel-> Restart & Run All pour tout relancer")
    lx=np.linspace(2,4,nb)
    ly=np.linspace(2,4,nb)
    tab=np.zeros((nb,nb),dtype=float)
    t=time.time()
    for i in range(nb):
        for j in range(nb):
            tab[i,j]=expo_lyapunov(lx[i],ly[j],40)
        if i%(nb//100)==0: print("■", end="")
        if i==2*nb//100:
            print("Temps restant estimé : ",int(49*(time.time()-t))," s")
    plt.close()
    fig = plt.figure(frameon=False)
    fig.set_size_inches(50,50)
    #fig.savefig("chemin_d_acces\\fractale.png",dpi=300)
    plt.imshow(tab,"inferno")
    fig.show()_____no_output_____interact(fractale, nb=widgets.IntSlider(min=100, max=5000, step=100, value=200, continuous_update=False))_____no_output_____


# Notebook from ykat0/capital
Path: docs/tutorials/capital_tutorial.ipynb

# Tutorial_____no_output_____Setting matplotlib formats just for tutorials.
Please skip the next cell._____no_output_____

import matplotlib_inline.backend_inline
matplotlib_inline.backend_inline.set_matplotlib_formats('png')_____no_output_____

Import CAPITAL and Scanpy_____no_output_____

%matplotlib inline
import capital as cp
import scanpy as sc_____no_output_____

## Preprocessing scRNA-seq data
Read a count matrix (cells x genes) of your own data as an AnnData object using sc.read()._____no_output_____

#adata1 = sc.read("./your_data1.h5ad")_____no_output_____

Run a simple preproccesing recipe._____no_output_____

#cp.tl.preprocessing(adata1, n_Top_genes=2000, K=40)_____no_output_____

Specifically, cp.tl.preprocessing() runs a simple preprocessing recipe as follows:

```
sc.pp.filter_cells(adata, min_genes=Min_Genes)
sc.pp.filter_genes(adata, min_cells=Min_Cells)
sc.pp.normalize_total(adata, exclude_highly_expressed=True)
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, min_mean=Min_Mean, max_mean=Max_Mean, min_disp=Min_Disp, n_top_genes=n_Top_genes)
adata.raw = adata
adata = adata[:,adata.var['highly_variable']]
sc.tl.pca(adata, n_comps=N_pcs)
sc.pp.neighbors(adata, n_neighbors=K, n_pcs=N_pcs)
sc.tl.diffmap(adata)
sc.tl.umap(adata)
sc.tl.leiden(adata)
sc.tl.paga(adata, groups='leiden')
```_____no_output_____In this tutorial, we will use two datasets that have already been preprocessed.
Download might take a minute._____no_output_____

adata1 = cp.dataset.setty19()Downloading the dataset.
Download completed. The dataset is saved in ../data/capital_dataset/setty19_capital.h5ad
adata1_____no_output_____sc.pl.umap(adata1, color="leiden")_____no_output_____

Note that dataset 2 will be loaded later._____no_output_____## Computing a trajectory tree with CAPITAL
Pass the AnnData object and the root name in the trajectory to cp.tl.trajectory_tree().
You can pass your metadata in adata.obs by choosing a key in adata.obs.
You can also draw your own defined tree in networkx.DiGraph and pass it to the argument._____no_output_____

cp.tl.trajectory_tree(adata1, root_node="4", groupby="leiden", tree=None)_____no_output_____

Draw the calculated tree._____no_output_____

cp.pl.trajectory_tree(adata1)_____no_output_____

Apply the same process to one or more datasets that you would like to align._____no_output_____

adata2 = cp.dataset.velten17()Downloading the dataset.
Download completed. The dataset is saved in ../data/capital_dataset/velten17_capital.h5ad
adata2_____no_output_____sc.pl.umap(adata2, color="leiden")_____no_output_____cp.tl.trajectory_tree(adata2, root_node="0", groupby="leiden", tree=None)_____no_output_____cp.pl.trajectory_tree(adata2)_____no_output_____

## Aligning trajectory trees with CAPITAL
The alignment of trajectory trees is calculated by minimizing an alignment distance.

You can specify the number of highly variable genes for both of the data so that the intersection of those genes is used to calculate the tree alignment._____no_output_____

cdata = cp.tl.tree_alignment(adata1, adata2, num_genes1=2000, num_genes2=2000)Calculating tree alignment
411 genes are used to calculate cost of tree alignment.

Calculation finished.

Draw the tree alignment result._____no_output_____

cp.pl.tree_alignment(cdata)_____no_output_____

Each linear alignment from the root to each leaf (path) is labeled as alignmentXYZ where X, Y and Z indicate digit.
See the list of the alignments name by alignmentlist._____no_output_____

cdata.alignmentlist_____no_output_____

## Aligning cells along each trajectory path
Run dpt (diffusion pseudotime) to calculate pseudotime for each alignment.
Specify the alignment name (list of alignment names) to choose which path should be compared to calculate cell-cell matching.
Note: calculating pseudotime for all the alignments might take for a minute._____no_output_____

cp.tl.dpt(cdata, alignment=["alignment001", "alignment002"])_____no_output_____

Choose genes to do dynamic time warping._____no_output_____

genes = ["GATA1", "CA1", "LMNA"]_____no_output_____cp.tl.dtw(cdata, genes, alignment="alignment001")_____no_output_____

Draw the result of dynamic time warping.
The colors in the labels correspond to the colors of sc.pl.umap()._____no_output_____

cp.pl.dtw(cdata, genes, alignment="alignment001",
          data1_name="Setty+2019", data2_name="Velten+2017")_____no_output_____

Draw gene expression dynamics for both data.
Pseudotime in this figure is matched to adata1's pseudotime._____no_output_____

cp.pl.gene_expression_trend(cdata, genes, alignment="alignment001",
                            data1_name="Setty+2019", data2_name="Velten+2017")_____no_output_____

## Writing CAPITAL results
Specify a directory so that two AnnData objects, adata1 and adata2, and capital_data.npz are saved in the directory.
You can read CAPITAL data by passing the directory path to cp.tl.read_capital_data()._____no_output_____

cdata.write("../results/cdata")_____no_output_____cdata = cp.tl.read_capital_data("../results/cdata")_____no_output_____

## Calculating similarity scores of genes in cell-cell alignment
Computing dynamic time warping for each gene will generate a similarity score (see [here](https://tslearn.readthedocs.io/en/stable/gen_modules/metrics/tslearn.metrics.dtw_path.html#tslearn.metrics.dtw_path) for details).
You can pass a value to min_disp to choose how many genes are used to calculate the score.
The genes that have larger normalized dispersions per gene than min_disp in either data are chosen.
Results are in cdata.similarity_score.
The higher in the resulting list, the more similar the gene expression trend is._____no_output_____

cp.tl.genes_similarity_score(cdata, alignment="alignment002", min_disp=0.5)Calculating similarity score of 5148 genes in alignment002
Calculating finished
cdata.similarity_score["alignment002"]_____no_output_____

See gene expression trends that have similar trends in the alignment._____no_output_____

high_similarlity_genes = cdata.similarity_score["alignment002"][:4]_____no_output_____cp.tl.dtw(cdata, high_similarlity_genes, alignment="alignment002")_____no_output_____cp.pl.gene_expression_trend(cdata, high_similarlity_genes, alignment="alignment002",
                            data1_name="Setty+2019", data2_name="Velten+2017")_____no_output_____

See also gene expression trends that have lower similarlity score in the alignment._____no_output_____

low_similarlity_genes = cdata.similarity_score["alignment002"][::-1][:4]_____no_output_____cp.tl.dtw(cdata, low_similarlity_genes, alignment="alignment002")_____no_output_____cp.pl.gene_expression_trend(cdata, low_similarlity_genes, alignment="alignment002",
                            data1_name="Setty+2019", data2_name="Velten+2017")_____no_output_____


# Notebook from ZhouGeorge/TensorFlow-Tutorials
Path: 19_Hyper-Parameters.ipynb

# TensorFlow Tutorial #19
# Hyper-Parameter Optimization

by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)
/ [GitHub](https://github.com/Hvass-Labs/TensorFlow-Tutorials) / [Videos on YouTube](https://www.youtube.com/playlist?list=PL9Hr9sNUjfsmEu1ZniY0XpHSzl5uihcXZ)_____no_output_____## Introduction

There are many parameters you can select when building and training a Neural Network in TensorFlow. These are often called Hyper-Parameters. For example, there is a hyper-parameter for how many layers the network should have, and another hyper-parameter for how many nodes per layer, and another hyper-parameter for the activation function to use, etc. The optimization method also has one or more hyper-parameters you can select, such as the learning-rate.

One way of searching for good hyper-parameters is by hand-tuning, where you try one set of parameters and see how they perform, and then you try another set of parameters and see if they improve the performance. You try and build an intuition for what works well and guide your parameter-search accordingly. Not only is this extremely time-consuming for a human researcher, but the optimal parameters are often counter-intuitive to humans so you will not find them!

Another way of searching for good hyper-parameters is to divide each parameter's valid range into evenly spaced values, and then simply have the computer try all combinations of parameter-values. This is called Grid Search. Although it is run entirely by the computer, it quickly becomes extremely time-consuming because the number of parameter-combinations increases exponentially as you add more hyper-parameters. This problem is known as the Curse of Dimensionality. For example, if you have just 4 hyper-parameters to tune and each of them is allowed 10 possible values, then there is a total of 10^4 parameter-combinations. If you add just one more hyper-parameter then there are 10^5 parameter-combinations, and so on.

Yet another way of searching for good hyper-parameters is by random search. Instead of systematically trying every single parameter-combination as in Grid Search, we now try a number of parameter-combinations completely at random. This is like searching for "a needle in a haystack" and as the number of parameters increases, the probability of finding the optimal parameter-combinations by random sampling decreases to zero.

This tutorial uses a clever method for finding good hyper-parameters known as Bayesian Optimization. You should be familiar with TensorFlow, Keras and Convolutional Neural Networks, see Tutorials #01, #02 and #03-C._____no_output_____## Flowchart

The problem with hyper-parameter optimization is that it is extremely costly to assess the performance of a set of parameters. This is because we first have to build the corresponding neural network, then we have to train it, and finally we have to measure its performance on a test-set. In this tutorial we will use the small MNIST problem so this training can be done very quickly, but on more realistic problems the training may take hours, days or even weeks on a very fast computer. So we need an optimization method that can search for hyper-parameters as efficiently as possible, by only evaluating the actual performance when absolutely necessary.

The idea with Bayesian optimization is to construct another model of the search-space for hyper-parameters. One kind of model is known as a Gaussian Process. This gives us an estimate of how the performance varies with changes to the hyper-parameters. Whenever we evaluate the actual performance for a set of hyper-parameters, we know for a fact what the performance is - except perhaps for some noise. We can then ask the Bayesian optimizer to give us a new suggestion for hyper-parameters in a region of the search-space that we haven't explored yet, or hyper-parameters that the Bayesian optimizer thinks will bring us most improvement. We then repeat this process a number of times until the Bayesian optimizer has built a good model of how the performance varies with different hyper-parameters, so we can choose the best parameters.

The flowchart of the algorithm is roughly:

![Flowchart](images/19_flowchart_bayesian_optimization.png)_____no_output_____## Imports_____no_output_____

%matplotlib inline
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
import math/home/magnus/anaconda3/envs/tf-test/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
  return f(*args, **kwds)

We need to import several things from Keras._____no_output_____

# from tf.keras.models import Sequential  # This does not work!
from tensorflow.python.keras import backend as K
from tensorflow.python.keras.models import Sequential
from tensorflow.python.keras.layers import InputLayer, Input
from tensorflow.python.keras.layers import Reshape, MaxPooling2D
from tensorflow.python.keras.layers import Conv2D, Dense, Flatten
from tensorflow.python.keras.callbacks import TensorBoard
from tensorflow.python.keras.optimizers import Adam
from tensorflow.python.keras.models import load_model_____no_output_____

**NOTE:** We will save and load models using Keras so you need to have [h5py](http://docs.h5py.org/en/latest/build.html#install) installed. You also need to have [scikit-optimize](https://scikit-optimize.github.io/) installed for doing the hyper-parameter optimization.

You should be able to run the following command in a terminal to install them both:

`pip install h5py scikit-optimize`_____no_output_____**NOTE:** This Notebook requires plotting functions in `scikit-optimize` that have not been merged into the official release at the time of this writing. If this Notebook cannot run with the version of `scikit-optimize` installed by the command above, you may have to install `scikit-optimize` from a development branch by running the following command instead:

`pip install git+git://github.com/Hvass-Labs/scikit-optimize.git@dd7433da068b5a2509ef4ea4e5195458393e6555`_____no_output_____

import skopt
from skopt import gp_minimize, forest_minimize
from skopt.space import Real, Categorical, Integer
from skopt.plots import plot_convergence
from skopt.plots import plot_objective, plot_evaluations
from skopt.plots import plot_histogram, plot_objective_2D
from skopt.utils import use_named_args_____no_output_____

This was developed using Python 3.6 (Anaconda) and package versions:_____no_output_____

tf.__version_______no_output_____tf.keras.__version_______no_output_____skopt.__version_______no_output_____

## Hyper-Parameters

In this tutorial we want to find the hyper-parametes that makes a simple Convolutional Neural Network perform best at classifying the MNIST dataset for hand-written digits.

For this demonstration we want to find the following hyper-parameters:

* The learning-rate of the optimizer.
* The number of fully-connected / dense layers.
* The number of nodes for each of the dense layers.
* Whether to use 'sigmoid' or 'relu' activation in all the layers.

We will use the Python package `scikit-optimize` (or `skopt`) for finding the best choices of these hyper-parameters. Before we begin with the actual search for hyper-parameters, we first need to define the valid search-ranges or search-dimensions for each of these parameters.

This is the search-dimension for the learning-rate. It is a real number (floating-point) with a lower bound of `1e-6` and an upper bound of `1e-2`. But instead of searching between these bounds directly, we use a logarithmic transformation, so we will search for the number `k` in `1ek` which is only bounded between -6 and -2. This is better than searching the entire exponential range._____no_output_____

dim_learning_rate = Real(low=1e-6, high=1e-2, prior='log-uniform',
                         name='learning_rate')_____no_output_____

This is the search-dimension for the number of dense layers in the neural network. This is an integer and we want at least 1 dense layer and at most 5 dense layers in the neural network._____no_output_____

dim_num_dense_layers = Integer(low=1, high=5, name='num_dense_layers')_____no_output_____

This is the search-dimension for the number of nodes for each dense layer. This is also an integer and we want at least 5 and at most 512 nodes in each layer of the neural network._____no_output_____

dim_num_dense_nodes = Integer(low=5, high=512, name='num_dense_nodes')_____no_output_____

This is the search-dimension for the activation-function. This is a combinatorial or categorical parameter which can be either 'relu' or 'sigmoid'._____no_output_____

dim_activation = Categorical(categories=['relu', 'sigmoid'],
                             name='activation')_____no_output_____

We then combine all these search-dimensions into a list._____no_output_____

dimensions = [dim_learning_rate,
              dim_num_dense_layers,
              dim_num_dense_nodes,
              dim_activation]_____no_output_____

It is helpful to start the search for hyper-parameters with a decent choice that we have found by hand-tuning. But we will use the following parameters that do not perform so well, so as to better demonstrate the usefulness of hyper-parameter optimization: A learning-rate of 1e-5, a single dense layer with 16 nodes, and relu activation-functions.

Note that these hyper-parameters are packed in a single list. This is how `skopt` works internally on hyper-parameters. You therefore need to ensure that the order of the dimensions are consistent with the order given in `dimensions` above._____no_output_____

default_parameters = [1e-5, 1, 16, 'relu']_____no_output_____

### Helper-function for log-dir-name

We will log the training-progress for all parameter-combinations so they can be viewed and compared using TensorBoard. This is done by setting a common parent-dir and then have a sub-dir for each parameter-combination with an appropriate name._____no_output_____

def log_dir_name(learning_rate, num_dense_layers,
                 num_dense_nodes, activation):

    # The dir-name for the TensorBoard log-dir.
    s = "./19_logs/lr_{0:.0e}_layers_{1}_nodes_{2}_{3}/"

    # Insert all the hyper-parameters in the dir-name.
    log_dir = s.format(learning_rate,
                       num_dense_layers,
                       num_dense_nodes,
                       activation)

    return log_dir_____no_output_____

## Load Data_____no_output_____The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path._____no_output_____

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets('data/MNIST/', one_hot=True)Extracting data/MNIST/train-images-idx3-ubyte.gz
Extracting data/MNIST/train-labels-idx1-ubyte.gz
Extracting data/MNIST/t10k-images-idx3-ubyte.gz
Extracting data/MNIST/t10k-labels-idx1-ubyte.gz

The MNIST data-set has now been loaded and consists of 70,000 images and associated labels (i.e. classifications of the images). The data-set is split into 3 mutually exclusive sub-sets._____no_output_____

print("Size of:")
print("- Training-set:\t\t{}".format(len(data.train.labels)))
print("- Test-set:\t\t{}".format(len(data.test.labels)))
print("- Validation-set:\t{}".format(len(data.validation.labels)))Size of:
- Training-set:		55000
- Test-set:		10000
- Validation-set:	5000

The class-labels are One-Hot encoded, which means that each label is a vector with 10 elements, all of which are zero except for one element. The index of this one element is the class-number, that is, the digit shown in the associated image. We also need the class-numbers as integers for the test-set, so we calculate it now._____no_output_____

data.test.cls = np.argmax(data.test.labels, axis=1)_____no_output_____

We use the performance on the validation-set as an indication of which choice of hyper-parameters performs the best on previously unseen data. The Keras API needs the validation-set as a tuple._____no_output_____

validation_data = (data.validation.images, data.validation.labels)_____no_output_____

## Data Dimensions_____no_output_____The data dimensions are used in several places in the source-code below. They are defined once so we can use these variables instead of numbers throughout the source-code below._____no_output_____

# We know that MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
# This is used for plotting the images.
img_shape = (img_size, img_size)

# Tuple with height, width and depth used to reshape arrays.
# This is used for reshaping in Keras.
img_shape_full = (img_size, img_size, 1)

# Number of colour channels for the images: 1 channel for gray-scale.
num_channels = 1

# Number of classes, one class for each of 10 digits.
num_classes = 10_____no_output_____

### Helper-function for plotting images_____no_output_____Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image._____no_output_____

def plot_images(images, cls_true, cls_pred=None):
    assert len(images) == len(cls_true) == 9

    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(3, 3)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])

        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()_____no_output_____

### Plot a few images to see if data is correct_____no_output_____

# Get the first images from the test-set.
images = data.test.images[0:9]

# Get the true classes for those images.
cls_true = data.test.cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)_____no_output_____

### Helper-function to plot example errors

Function for plotting examples of images from the test-set that have been mis-classified._____no_output_____

def plot_example_errors(cls_pred):
    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # Boolean array whether the predicted class is incorrect.
    incorrect = (cls_pred != data.test.cls)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = data.test.images[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = data.test.cls[incorrect]

    # Plot the first 9 images.
    plot_images(images=images[0:9],
                cls_true=cls_true[0:9],
                cls_pred=cls_pred[0:9])_____no_output_____

## Hyper-Parameter Optimization

There are several steps required to do hyper-parameter optimization.

### Create the Model

We first need a function that takes a set of hyper-parameters and creates the Convolutional Neural Network corresponding to those parameters. We use Keras to build the neural network in TensorFlow, see Tutorial #03-C for more details._____no_output_____

def create_model(learning_rate, num_dense_layers,
                 num_dense_nodes, activation):
    """
    Hyper-parameters:
    learning_rate:     Learning-rate for the optimizer.
    num_dense_layers:  Number of dense layers.
    num_dense_nodes:   Number of nodes in each dense layer.
    activation:        Activation function for all layers.
    """

    # Start construction of a Keras Sequential model.
    model = Sequential()

    # Add an input layer which is similar to a feed_dict in TensorFlow.
    # Note that the input-shape must be a tuple containing the image-size.
    model.add(InputLayer(input_shape=(img_size_flat,)))

    # The input from MNIST is a flattened array with 784 elements,
    # but the convolutional layers expect images with shape (28, 28, 1)
    model.add(Reshape(img_shape_full))

    # First convolutional layer.
    # There are many hyper-parameters in this layer, but we only
    # want to optimize the activation-function in this example.
    model.add(Conv2D(kernel_size=5, strides=1, filters=16, padding='same',
                     activation=activation, name='layer_conv1'))
    model.add(MaxPooling2D(pool_size=2, strides=2))

    # Second convolutional layer.
    # Again, we only want to optimize the activation-function here.
    model.add(Conv2D(kernel_size=5, strides=1, filters=36, padding='same',
                     activation=activation, name='layer_conv2'))
    model.add(MaxPooling2D(pool_size=2, strides=2))

    # Flatten the 4-rank output of the convolutional layers
    # to 2-rank that can be input to a fully-connected / dense layer.
    model.add(Flatten())

    # Add fully-connected / dense layers.
    # The number of layers is a hyper-parameter we want to optimize.
    for i in range(num_dense_layers):
        # Name of the layer. This is not really necessary
        # because Keras should give them unique names.
        name = 'layer_dense_{0}'.format(i+1)

        # Add the dense / fully-connected layer to the model.
        # This has two hyper-parameters we want to optimize:
        # The number of nodes and the activation function.
        model.add(Dense(num_dense_nodes,
                        activation=activation,
                        name=name))

    # Last fully-connected / dense layer with softmax-activation
    # for use in classification.
    model.add(Dense(num_classes, activation='softmax'))

    # Use the Adam method for training the network.
    # We want to find the best learning-rate for the Adam method.
    optimizer = Adam(lr=learning_rate)

    # In Keras we need to compile the model so it can be trained.
    model.compile(optimizer=optimizer,
                  loss='categorical_crossentropy',
                  metrics=['accuracy'])

    return model_____no_output_____

### Train and Evaluate the Model

The neural network with the best hyper-parameters is saved to disk so it can be reloaded later. This is the filename for the model._____no_output_____

path_best_model = '19_best_model.keras'_____no_output_____

This is the classification accuracy for the model saved to disk. It is a global variable which will be updated during optimization of the hyper-parameters._____no_output_____

best_accuracy = 0.0_____no_output_____

This is the function that creates and trains a neural network with the given hyper-parameters, and then evaluates its performance on the validation-set. The function then returns the so-called fitness value (aka. objective value), which is the negative classification accuracy on the validation-set. It is negative because `skopt` performs minimization instead of maximization.

Note the function decorator `@use_named_args` which wraps the fitness function so that it can be called with all the parameters as a single list, for example: `fitness(x=[1e-4, 3, 256, 'relu'])`. This is the calling-style `skopt` uses internally._____no_output_____

@use_named_args(dimensions=dimensions)
def fitness(learning_rate, num_dense_layers,
            num_dense_nodes, activation):
    """
    Hyper-parameters:
    learning_rate:     Learning-rate for the optimizer.
    num_dense_layers:  Number of dense layers.
    num_dense_nodes:   Number of nodes in each dense layer.
    activation:        Activation function for all layers.
    """

    # Print the hyper-parameters.
    print('learning rate: {0:.1e}'.format(learning_rate))
    print('num_dense_layers:', num_dense_layers)
    print('num_dense_nodes:', num_dense_nodes)
    print('activation:', activation)
    print()

    # Create the neural network with these hyper-parameters.
    model = create_model(learning_rate=learning_rate,
                         num_dense_layers=num_dense_layers,
                         num_dense_nodes=num_dense_nodes,
                         activation=activation)

    # Dir-name for the TensorBoard log-files.
    log_dir = log_dir_name(learning_rate, num_dense_layers,
                           num_dense_nodes, activation)

    # Create a callback-function for Keras which will be
    # run after each epoch has ended during training.
    # This saves the log-files for TensorBoard.
    # Note that there are complications when histogram_freq=1.
    # It might give strange errors and it also does not properly
    # support Keras data-generators for the validation-set.
    callback_log = TensorBoard(
        log_dir=log_dir,
        histogram_freq=0,
        batch_size=32,
        write_graph=True,
        write_grads=False,
        write_images=False)

    # Use Keras to train the model.
    history = model.fit(x=data.train.images,
                        y=data.train.labels,
                        epochs=3,
                        batch_size=128,
                        validation_data=validation_data,
                        callbacks=[callback_log])

    # Get the classification accuracy on the validation-set
    # after the last training-epoch.
    accuracy = history.history['val_acc'][-1]

    # Print the classification accuracy.
    print()
    print("Accuracy: {0:.2%}".format(accuracy))
    print()

    # Save the model if it improves on the best-found performance.
    # We use the global keyword so we update the variable outside
    # of this function.
    global best_accuracy

    # If the classification accuracy of the saved model is improved ...
    if accuracy > best_accuracy:
        # Save the new model to harddisk.
        model.save(path_best_model)

        # Update the classification accuracy.
        best_accuracy = accuracy

    # Delete the Keras model with these hyper-parameters from memory.
    del model

    # Clear the Keras session, otherwise it will keep adding new
    # models to the same TensorFlow graph each time we create
    # a model with a different set of hyper-parameters.
    K.clear_session()

    # NOTE: Scikit-optimize does minimization so it tries to
    # find a set of hyper-parameters with the LOWEST fitness-value.
    # Because we are interested in the HIGHEST classification
    # accuracy, we need to negate this number so it can be minimized.
    return -accuracy_____no_output_____

### Test Run

Before we run the hyper-parameter optimization, let us first check that the various functions above actually work, when we pass the default hyper-parameters._____no_output_____

fitness(x=default_parameters)learning rate: 1.0e-05
num_dense_layers: 1
num_dense_nodes: 16
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 2.2525 - acc: 0.1995 - val_loss: 2.1754 - val_acc: 0.3578
Epoch 2/3
55000/55000 [==============================] - 2s - loss: 2.0279 - acc: 0.4612 - val_loss: 1.8432 - val_acc: 0.5558
Epoch 3/3
55000/55000 [==============================] - 4s - loss: 1.6227 - acc: 0.5998 - val_loss: 1.3877 - val_acc: 0.6654

Accuracy: 66.54%

### Run the Hyper-Parameter Optimization

Now we are ready to run the actual hyper-parameter optimization using Bayesian optimization from the scikit-optimize package. Note that it first calls `fitness()` with `default_parameters` as the starting point we have found by hand-tuning, which should help the optimizer locate better hyper-parameters faster.

There are many more parameters you can experiment with here, including the number of calls to the `fitness()` function which we have set to 40. But `fitness()` is very expensive to evaluate so it should not be run too many times, especially for larger neural networks and datasets.

You can also experiment with the so-called acquisition function which determines how to find a new set of hyper-parameters from the internal model of the Bayesian optimizer. You can also try using another Bayesian optimizer such as Random Forests._____no_output_____

search_result = gp_minimize(func=fitness,
                            dimensions=dimensions,
                            acq_func='EI', # Expected Improvement.
                            n_calls=40,
                            x0=default_parameters)learning rate: 1.0e-05
num_dense_layers: 1
num_dense_nodes: 16
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 2s - loss: 2.2287 - acc: 0.1868 - val_loss: 2.1264 - val_acc: 0.3182
Epoch 2/3
55000/55000 [==============================] - 2s - loss: 1.9607 - acc: 0.4438 - val_loss: 1.7713 - val_acc: 0.5082
Epoch 3/3
55000/55000 [==============================] - 5s - loss: 1.5763 - acc: 0.5579 - val_loss: 1.3832 - val_acc: 0.6166

Accuracy: 61.66%

learning rate: 6.1e-04
num_dense_layers: 2
num_dense_nodes: 474
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 1.3354 - acc: 0.5258 - val_loss: 0.3002 - val_acc: 0.9112
Epoch 2/3
55000/55000 [==============================] - 4s - loss: 0.2336 - acc: 0.9269 - val_loss: 0.1626 - val_acc: 0.9538
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 0.1403 - acc: 0.9563 - val_loss: 0.1113 - val_acc: 0.9692

Accuracy: 96.92%

learning rate: 6.1e-06
num_dense_layers: 2
num_dense_nodes: 333
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 2.1702 - acc: 0.5067 - val_loss: 1.9186 - val_acc: 0.6892
Epoch 2/3
55000/55000 [==============================] - 3s - loss: 1.4878 - acc: 0.7480 - val_loss: 1.0546 - val_acc: 0.7940
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 0.8226 - acc: 0.8264 - val_loss: 0.6324 - val_acc: 0.8514

Accuracy: 85.14%

learning rate: 1.7e-04
num_dense_layers: 4
num_dense_nodes: 252
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 2.3075 - acc: 0.1058 - val_loss: 2.2968 - val_acc: 0.1126
Epoch 2/3
55000/55000 [==============================] - 4s - loss: 1.5272 - acc: 0.4944 - val_loss: 0.8210 - val_acc: 0.7386
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 0.6595 - acc: 0.7967 - val_loss: 0.4940 - val_acc: 0.8544

Accuracy: 85.44%

learning rate: 7.3e-03
num_dense_layers: 3
num_dense_nodes: 166
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 0.1821 - acc: 0.9431 - val_loss: 0.0705 - val_acc: 0.9808
Epoch 2/3
55000/55000 [==============================] - 3s - loss: 0.0605 - acc: 0.9829 - val_loss: 0.0678 - val_acc: 0.9848
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 0.0549 - acc: 0.9855 - val_loss: 0.0736 - val_acc: 0.9846

Accuracy: 98.46%

learning rate: 6.1e-05
num_dense_layers: 2
num_dense_nodes: 209
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 2s - loss: 2.3187 - acc: 0.1073 - val_loss: 2.3030 - val_acc: 0.0924
Epoch 2/3
55000/55000 [==============================] - 2s - loss: 2.3016 - acc: 0.1121 - val_loss: 2.2993 - val_acc: 0.1126
Epoch 3/3
55000/55000 [==============================] - 5s - loss: 2.2858 - acc: 0.1573 - val_loss: 2.2243 - val_acc: 0.2898

Accuracy: 28.98%

learning rate: 1.8e-04
num_dense_layers: 4
num_dense_nodes: 453
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 0.3601 - acc: 0.8920 - val_loss: 0.1234 - val_acc: 0.9640
Epoch 2/3
55000/55000 [==============================] - 3s - loss: 0.0850 - acc: 0.9741 - val_loss: 0.0576 - val_acc: 0.9830
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 0.0566 - acc: 0.9824 - val_loss: 0.0535 - val_acc: 0.9856

Accuracy: 98.56%

learning rate: 5.5e-06
num_dense_layers: 4
num_dense_nodes: 186
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 2.3129 - acc: 0.1039 - val_loss: 2.3025 - val_acc: 0.1100
Epoch 2/3
55000/55000 [==============================] - 4s - loss: 2.3016 - acc: 0.1106 - val_loss: 2.3010 - val_acc: 0.1126
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 2.3013 - acc: 0.1123 - val_loss: 2.3011 - val_acc: 0.1126

Accuracy: 11.26%

learning rate: 3.1e-05
num_dense_layers: 3
num_dense_nodes: 427
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 2.3132 - acc: 0.1070 - val_loss: 2.3007 - val_acc: 0.1126
Epoch 2/3
55000/55000 [==============================] - 4s - loss: 2.3029 - acc: 0.1080 - val_loss: 2.3020 - val_acc: 0.1126
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 2.3021 - acc: 0.1093 - val_loss: 2.3016 - val_acc: 0.1126

Accuracy: 11.26%

learning rate: 1.4e-04
num_dense_layers: 2
num_dense_nodes: 29
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 2s - loss: 0.8474 - acc: 0.7524 - val_loss: 0.2954 - val_acc: 0.9190
Epoch 2/3
55000/55000 [==============================] - 2s - loss: 0.2392 - acc: 0.9315 - val_loss: 0.1741 - val_acc: 0.9512
Epoch 3/3
55000/55000 [==============================] - 5s - loss: 0.1643 - acc: 0.9517 - val_loss: 0.1346 - val_acc: 0.9612

Accuracy: 96.12%

learning rate: 3.7e-04
num_dense_layers: 4
num_dense_nodes: 338
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 2.1610 - acc: 0.1844 - val_loss: 1.0813 - val_acc: 0.6678
Epoch 2/3
55000/55000 [==============================] - 4s - loss: 0.5982 - acc: 0.8131 - val_loss: 0.3252 - val_acc: 0.9100
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 0.2712 - acc: 0.9201 - val_loss: 0.1858 - val_acc: 0.9468

Accuracy: 94.68%

learning rate: 1.7e-06
num_dense_layers: 4
num_dense_nodes: 512
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 2.2568 - acc: 0.3895 - val_loss: 2.1984 - val_acc: 0.6048
Epoch 2/3
55000/55000 [==============================] - 4s - loss: 2.0854 - acc: 0.6719 - val_loss: 1.9276 - val_acc: 0.7052
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 1.7106 - acc: 0.7158 - val_loss: 1.4589 - val_acc: 0.7290

Accuracy: 72.90%

learning rate: 1.4e-03
num_dense_layers: 2
num_dense_nodes: 62
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 2s - loss: 0.2396 - acc: 0.9249 - val_loss: 0.0643 - val_acc: 0.9822
Epoch 2/3
55000/55000 [==============================] - 2s - loss: 0.0587 - acc: 0.9819 - val_loss: 0.0536 - val_acc: 0.9838
Epoch 3/3
55000/55000 [==============================] - 6s - loss: 0.0427 - acc: 0.9867 - val_loss: 0.0480 - val_acc: 0.9842

Accuracy: 98.42%

learning rate: 2.7e-03
num_dense_layers: 2
num_dense_nodes: 364
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 1.3014 - acc: 0.5223 - val_loss: 0.2531 - val_acc: 0.9232
Epoch 2/3
55000/55000 [==============================] - 3s - loss: 0.1956 - acc: 0.9386 - val_loss: 0.1221 - val_acc: 0.9650
Epoch 3/3
55000/55000 [==============================] - 5s - loss: 0.1138 - acc: 0.9646 - val_loss: 0.0846 - val_acc: 0.9758

Accuracy: 97.58%

learning rate: 5.6e-04
num_dense_layers: 5
num_dense_nodes: 13
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 0.9357 - acc: 0.6775 - val_loss: 0.3024 - val_acc: 0.9184
Epoch 2/3
55000/55000 [==============================] - 3s - loss: 0.2416 - acc: 0.9338 - val_loss: 0.1749 - val_acc: 0.9520
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 0.1685 - acc: 0.9525 - val_loss: 0.1541 - val_acc: 0.9570

Accuracy: 95.70%

learning rate: 1.0e-02
num_dense_layers: 5
num_dense_nodes: 352
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 2.3316 - acc: 0.1049 - val_loss: 2.3019 - val_acc: 0.1070
Epoch 2/3
55000/55000 [==============================] - 3s - loss: 2.3024 - acc: 0.1090 - val_loss: 2.3017 - val_acc: 0.1126
Epoch 3/3
55000/55000 [==============================] - 8s - loss: 2.3020 - acc: 0.1104 - val_loss: 2.3014 - val_acc: 0.1126

Accuracy: 11.26%

learning rate: 1.5e-03
num_dense_layers: 1
num_dense_nodes: 5
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 2s - loss: 1.7072 - acc: 0.4784 - val_loss: 1.2153 - val_acc: 0.6980
Epoch 2/3
55000/55000 [==============================] - 2s - loss: 0.9949 - acc: 0.7914 - val_loss: 0.7749 - val_acc: 0.8564
Epoch 3/3
55000/55000 [==============================] - 6s - loss: 0.6663 - acc: 0.8663 - val_loss: 0.5469 - val_acc: 0.9014

Accuracy: 90.14%

learning rate: 1.0e-03
num_dense_layers: 3
num_dense_nodes: 496
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 0.1843 - acc: 0.9426 - val_loss: 0.0483 - val_acc: 0.9852
Epoch 2/3
55000/55000 [==============================] - 5s - loss: 0.0506 - acc: 0.9840 - val_loss: 0.0471 - val_acc: 0.9856
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 0.0347 - acc: 0.9889 - val_loss: 0.0451 - val_acc: 0.9856

Accuracy: 98.56%

learning rate: 3.7e-03
num_dense_layers: 5
num_dense_nodes: 512
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 4s - loss: 0.2060 - acc: 0.9377 - val_loss: 0.0739 - val_acc: 0.9832
Epoch 2/3
55000/55000 [==============================] - 5s - loss: 0.0781 - acc: 0.9814 - val_loss: 0.0765 - val_acc: 0.9842
Epoch 3/3
55000/55000 [==============================] - 8s - loss: 0.0908 - acc: 0.9818 - val_loss: 0.1368 - val_acc: 0.9766

Accuracy: 97.66%

learning rate: 1.0e-02
num_dense_layers: 5
num_dense_nodes: 512
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 2.3199 - acc: 0.1105 - val_loss: 2.3015 - val_acc: 0.1126
Epoch 2/3
55000/55000 [==============================] - 4s - loss: 2.3020 - acc: 0.1104 - val_loss: 2.3011 - val_acc: 0.1126
Epoch 3/3
55000/55000 [==============================] - 8s - loss: 2.3018 - acc: 0.1110 - val_loss: 2.3013 - val_acc: 0.1126

Accuracy: 11.26%

learning rate: 1.9e-04
num_dense_layers: 4
num_dense_nodes: 418
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 0.3595 - acc: 0.8999 - val_loss: 0.0888 - val_acc: 0.9732
Epoch 2/3
55000/55000 [==============================] - 3s - loss: 0.0868 - acc: 0.9738 - val_loss: 0.0686 - val_acc: 0.9782
Epoch 3/3
55000/55000 [==============================] - 8s - loss: 0.0584 - acc: 0.9821 - val_loss: 0.0478 - val_acc: 0.9850

Accuracy: 98.50%

learning rate: 2.4e-03
num_dense_layers: 4
num_dense_nodes: 144
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 0.1906 - acc: 0.9390 - val_loss: 0.0576 - val_acc: 0.9834
Epoch 2/3
55000/55000 [==============================] - 3s - loss: 0.0550 - acc: 0.9840 - val_loss: 0.0402 - val_acc: 0.9890
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 0.0380 - acc: 0.9885 - val_loss: 0.0459 - val_acc: 0.9880

Accuracy: 98.80%

learning rate: 6.8e-03
num_dense_layers: 2
num_dense_nodes: 105
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 2s - loss: 0.1552 - acc: 0.9507 - val_loss: 0.0498 - val_acc: 0.9860
Epoch 2/3
55000/55000 [==============================] - 2s - loss: 0.0485 - acc: 0.9853 - val_loss: 0.0534 - val_acc: 0.9836
Epoch 3/3
55000/55000 [==============================] - 6s - loss: 0.0417 - acc: 0.9875 - val_loss: 0.0496 - val_acc: 0.9852

Accuracy: 98.52%

learning rate: 2.5e-04
num_dense_layers: 2
num_dense_nodes: 435
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 0.3258 - acc: 0.9131 - val_loss: 0.1024 - val_acc: 0.9676
Epoch 2/3
55000/55000 [==============================] - 4s - loss: 0.0856 - acc: 0.9742 - val_loss: 0.0603 - val_acc: 0.9812
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 0.0601 - acc: 0.9819 - val_loss: 0.0477 - val_acc: 0.9868

Accuracy: 98.68%

learning rate: 2.5e-06
num_dense_layers: 1
num_dense_nodes: 409
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 2.2504 - acc: 0.3689 - val_loss: 2.1796 - val_acc: 0.5498
Epoch 2/3
55000/55000 [==============================] - 3s - loss: 2.0835 - acc: 0.6384 - val_loss: 1.9688 - val_acc: 0.6812
Epoch 3/3
55000/55000 [==============================] - 6s - loss: 1.8409 - acc: 0.7098 - val_loss: 1.6977 - val_acc: 0.7404

Accuracy: 74.04%

learning rate: 4.4e-03
num_dense_layers: 3
num_dense_nodes: 311
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 0.1504 - acc: 0.9523 - val_loss: 0.0746 - val_acc: 0.9800
Epoch 2/3
55000/55000 [==============================] - 4s - loss: 0.0559 - acc: 0.9842 - val_loss: 0.0751 - val_acc: 0.9812
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 0.0431 - acc: 0.9884 - val_loss: 0.0500 - val_acc: 0.9870

Accuracy: 98.70%

learning rate: 2.1e-03
num_dense_layers: 5
num_dense_nodes: 436
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 0.1884 - acc: 0.9418 - val_loss: 0.0664 - val_acc: 0.9840
Epoch 2/3
55000/55000 [==============================] - 4s - loss: 0.0598 - acc: 0.9837 - val_loss: 0.0454 - val_acc: 0.9880
Epoch 3/3
55000/55000 [==============================] - 8s - loss: 0.0435 - acc: 0.9887 - val_loss: 0.0553 - val_acc: 0.9864

Accuracy: 98.64%

learning rate: 1.9e-04
num_dense_layers: 3
num_dense_nodes: 441
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 0.3664 - acc: 0.8989 - val_loss: 0.1076 - val_acc: 0.9698
Epoch 2/3
55000/55000 [==============================] - 4s - loss: 0.0872 - acc: 0.9736 - val_loss: 0.0626 - val_acc: 0.9816
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 0.0583 - acc: 0.9824 - val_loss: 0.0504 - val_acc: 0.9856

Accuracy: 98.56%

learning rate: 1.7e-03
num_dense_layers: 1
num_dense_nodes: 512
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 1.2528 - acc: 0.5598 - val_loss: 0.2764 - val_acc: 0.9186
Epoch 2/3
55000/55000 [==============================] - 3s - loss: 0.2010 - acc: 0.9369 - val_loss: 0.1251 - val_acc: 0.9592
Epoch 3/3
55000/55000 [==============================] - 6s - loss: 0.1203 - acc: 0.9629 - val_loss: 0.0916 - val_acc: 0.9734

Accuracy: 97.34%

learning rate: 1.5e-03
num_dense_layers: 5
num_dense_nodes: 285
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 2.1378 - acc: 0.1588 - val_loss: 1.2723 - val_acc: 0.4116
Epoch 2/3
55000/55000 [==============================] - 4s - loss: 0.5670 - acc: 0.7991 - val_loss: 0.2616 - val_acc: 0.9266
Epoch 3/3
55000/55000 [==============================] - 8s - loss: 0.1877 - acc: 0.9460 - val_loss: 0.1365 - val_acc: 0.9618

Accuracy: 96.18%

learning rate: 3.3e-04
num_dense_layers: 5
num_dense_nodes: 5
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 2.3570 - acc: 0.0907 - val_loss: 2.3175 - val_acc: 0.0868
Epoch 2/3
55000/55000 [==============================] - 3s - loss: 2.3074 - acc: 0.0952 - val_loss: 2.3029 - val_acc: 0.1126
Epoch 3/3
55000/55000 [==============================] - 8s - loss: 2.3019 - acc: 0.1123 - val_loss: 2.3013 - val_acc: 0.1126

Accuracy: 11.26%

learning rate: 2.3e-04
num_dense_layers: 4
num_dense_nodes: 512
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 2.1591 - acc: 0.1861 - val_loss: 1.0381 - val_acc: 0.6422
Epoch 2/3
55000/55000 [==============================] - 5s - loss: 0.6686 - acc: 0.7868 - val_loss: 0.4403 - val_acc: 0.8662
Epoch 3/3
55000/55000 [==============================] - 8s - loss: 0.3814 - acc: 0.8831 - val_loss: 0.2920 - val_acc: 0.9090

Accuracy: 90.90%

learning rate: 2.6e-03
num_dense_layers: 1
num_dense_nodes: 126
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 2s - loss: 1.1633 - acc: 0.5922 - val_loss: 0.1928 - val_acc: 0.9422
Epoch 2/3
55000/55000 [==============================] - 2s - loss: 0.1490 - acc: 0.9550 - val_loss: 0.0859 - val_acc: 0.9778
Epoch 3/3
55000/55000 [==============================] - 5s - loss: 0.0885 - acc: 0.9729 - val_loss: 0.0735 - val_acc: 0.9786

Accuracy: 97.86%

learning rate: 5.7e-04
num_dense_layers: 1
num_dense_nodes: 246
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 6s - loss: 0.2579 - acc: 0.9261 - val_loss: 0.0748 - val_acc: 0.9782
Epoch 2/3
55000/55000 [==============================] - 6s - loss: 0.0691 - acc: 0.9787 - val_loss: 0.0502 - val_acc: 0.9858
Epoch 3/3
55000/55000 [==============================] - 6s - loss: 0.0465 - acc: 0.9854 - val_loss: 0.0423 - val_acc: 0.9880

Accuracy: 98.80%

learning rate: 2.4e-04
num_dense_layers: 1
num_dense_nodes: 164
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 2s - loss: 0.4321 - acc: 0.8849 - val_loss: 0.1429 - val_acc: 0.9608
Epoch 2/3
55000/55000 [==============================] - 2s - loss: 0.1163 - acc: 0.9654 - val_loss: 0.0821 - val_acc: 0.9766
Epoch 3/3
55000/55000 [==============================] - 4s - loss: 0.0794 - acc: 0.9762 - val_loss: 0.0679 - val_acc: 0.9796

Accuracy: 97.96%

learning rate: 1.0e-06
num_dense_layers: 2
num_dense_nodes: 5
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 2s - loss: 2.3000 - acc: 0.1046 - val_loss: 2.2987 - val_acc: 0.1122
Epoch 2/3
55000/55000 [==============================] - 2s - loss: 2.2981 - acc: 0.1124 - val_loss: 2.2965 - val_acc: 0.1224
Epoch 3/3
55000/55000 [==============================] - 4s - loss: 2.2959 - acc: 0.1221 - val_loss: 2.2941 - val_acc: 0.1290

Accuracy: 12.90%

learning rate: 1.3e-05
num_dense_layers: 2
num_dense_nodes: 512
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 1.6243 - acc: 0.6472 - val_loss: 0.7587 - val_acc: 0.8260
Epoch 2/3
55000/55000 [==============================] - 4s - loss: 0.5184 - acc: 0.8724 - val_loss: 0.3656 - val_acc: 0.9038
Epoch 3/3
55000/55000 [==============================] - 7s - loss: 0.3292 - acc: 0.9091 - val_loss: 0.2724 - val_acc: 0.9268

Accuracy: 92.68%

learning rate: 7.6e-05
num_dense_layers: 1
num_dense_nodes: 241
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 2s - loss: 0.7636 - acc: 0.8233 - val_loss: 0.2393 - val_acc: 0.9368
Epoch 2/3
55000/55000 [==============================] - 2s - loss: 0.1961 - acc: 0.9448 - val_loss: 0.1449 - val_acc: 0.9612
Epoch 3/3
55000/55000 [==============================] - 6s - loss: 0.1309 - acc: 0.9617 - val_loss: 0.1068 - val_acc: 0.9688

Accuracy: 96.88%

learning rate: 2.0e-03
num_dense_layers: 4
num_dense_nodes: 512
activation: relu

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 0.1668 - acc: 0.9474 - val_loss: 0.0605 - val_acc: 0.9832
Epoch 2/3
55000/55000 [==============================] - 4s - loss: 0.0548 - acc: 0.9845 - val_loss: 0.0419 - val_acc: 0.9902
Epoch 3/3
55000/55000 [==============================] - 8s - loss: 0.0408 - acc: 0.9890 - val_loss: 0.0596 - val_acc: 0.9844

Accuracy: 98.44%

learning rate: 2.2e-03
num_dense_layers: 2
num_dense_nodes: 326
activation: sigmoid

Train on 55000 samples, validate on 5000 samples
Epoch 1/3
55000/55000 [==============================] - 3s - loss: 1.1358 - acc: 0.5865 - val_loss: 0.2104 - val_acc: 0.9356
Epoch 2/3
55000/55000 [==============================] - 3s - loss: 0.1576 - acc: 0.9505 - val_loss: 0.0999 - val_acc: 0.9712
Epoch 3/3
55000/55000 [==============================] - 6s - loss: 0.1011 - acc: 0.9679 - val_loss: 0.0856 - val_acc: 0.9726

Accuracy: 97.26%

### Optimization Progress

The progress of the hyper-parameter optimization can be easily plotted. The best fitness value found is plotted on the y-axis, remember that this is the negated classification accuracy on the validation-set.

Note how few hyper-parameters had to be tried before substantial improvements were found._____no_output_____

plot_convergence(search_result)_____no_output_____

### Best Hyper-Parameters

The best hyper-parameters found by the Bayesian optimizer are packed as a list because that is what it uses internally._____no_output_____

search_result.x_____no_output_____

We can convert these parameters to a dict with proper names for the search-space dimensions.

First we need a reference to the search-space object._____no_output_____

space = search_result.space_____no_output_____

Then we can use it to create a dict where the hyper-parameters have the proper names of the search-space dimensions. This is a bit awkward._____no_output_____

space.point_to_dict(search_result.x)_____no_output_____

This is the fitness value associated with these hyper-parameters. This is a negative number because the Bayesian optimizer performs minimization, so we had to negate the classification accuracy which is posed as a maximization problem._____no_output_____

search_result.fun_____no_output_____

We can also see all the hyper-parameters tried by the Bayesian optimizer and their associated fitness values (the negated classification accuracies). These are sorted so the highest classification accuracies are shown first.

It appears that 'relu' activation was generally better than 'sigmoid'. Otherwise it can be difficult to see a pattern of which parameter choices are good. We really need to plot these results._____no_output_____

sorted(zip(search_result.func_vals, search_result.x_iters))_____no_output_____

### Plots

There are several plotting functions available in the `skopt` library. For example, we can plot a histogram for the `activation` parameter, which shows the distribution of samples during the hyper-parameter optimization._____no_output_____

fig, ax = plot_histogram(result=search_result,
                         dimension_name='activation')_____no_output_____

We can also make a landscape-plot of the estimated fitness values for two dimensions of the search-space, here taken to be `learning_rate` and `num_dense_layers`.

The Bayesian optimizer works by building a surrogate model of the search-space and then searching this model instead of the real search-space, because it is much faster. The plot shows the last surrogate model built by the Bayesian optimizer where yellow regions are better and blue regions are worse. The black dots show where the optimizer has sampled the search-space and the red star shows the best parameters found.

Several things should be noted here. Firstly, this surrogate model of the search-space may not be accurate. It is built from only 40 samples of calls to the `fitness()` function for training a neural network with a given choice of hyper-parameters. The modelled fitness landscape may differ significantly from its true values especially in regions of the search-space with few samples. Secondly, the plot may change each time the hyper-parameter optimization is run because of random noise in the training process of the neural network. Thirdly, this plot shows the effect of changing these two parameters `num_dense_layers` and `learning_rate` when averaged over all other dimensions in the search-space, this is also called a Partial Dependence plot and is a way of visualizing high-dimensional spaces in only 2-dimensions._____no_output_____

fig = plot_objective_2D(result=search_result,
                        dimension_name1='learning_rate',
                        dimension_name2='num_dense_layers',
                        levels=50)_____no_output_____

We cannot make a landscape plot for the `activation` hyper-parameter because it is a categorical variable that can be one of two strings `relu` or `sigmoid`. How this is encoded depends on the Bayesian optimizer, for example, whether it is using Gaussian Processes or Random Forests. But it cannot currently be plotted using the built-in functions of `skopt`.

Instead we only want to use the real- and integer-valued dimensions of the search-space which we identify by their names._____no_output_____

dim_names = ['learning_rate', 'num_dense_nodes', 'num_dense_layers']_____no_output_____

We can then make a matrix-plot of all combinations of these dimensions.

The diagonal shows the influence of a single dimension on the fitness. This is a so-called Partial Dependence plot for that dimension. It shows how the approximated fitness value changes with different values in that dimension.

The plots below the diagonal show the Partial Dependence for two dimensions. This shows how the approximated fitness value changes when we are varying two dimensions simultaneously.

These Partial Dependence plots are only approximations of the modelled fitness function - which in turn is only an approximation of the true fitness function in `fitness()`. This may be a bit difficult to understand. For example, the Partial Dependence is calculated by fixing one value for the `learning_rate` and then taking a large number of random samples for the remaining dimensions in the search-space. The estimated fitness for all these points is then averaged. This process is then repeated for other values of the `learning_rate` to show how it affects the fitness on average. A similar procedure is done for the plots that show the Partial Dependence plots for two dimensions._____no_output_____

fig, ax = plot_objective(result=search_result, dimension_names=dim_names)_____no_output_____

We can also show another type of matrix-plot. Here the diagonal shows histograms of the sample distributions for each of the hyper-parameters during the Bayesian optimization. The plots below the diagonal show the location of samples in the search-space and the colour-coding shows the order in which the samples were taken. For larger numbers of samples you will likely see that the samples eventually become concentrated in a certain region of the search-space._____no_output_____

fig, ax = plot_evaluations(result=search_result, dimension_names=dim_names)_____no_output_____

### Evaluate Best Model on Test-Set

We can now use the best model on the test-set. It is very easy to reload the model using Keras._____no_output_____

model = load_model(path_best_model)_____no_output_____

We then evaluate its performance on the test-set._____no_output_____

result = model.evaluate(x=data.test.images,
                        y=data.test.labels) 8960/10000 [=========================>....] - ETA: 0s

We can print all the performance metrics for the test-set._____no_output_____

for name, value in zip(model.metrics_names, result):
    print(name, value)loss 0.0363312054525
acc 0.9888

Or we can just print the classification accuracy._____no_output_____

print("{0}: {1:.2%}".format(model.metrics_names[1], result[1]))acc: 98.88%

### Predict on New Data

We can also predict the classification for new images. We will just use some images from the test-set but you could load your own images into numpy arrays and use those instead._____no_output_____

images = data.test.images[0:9]_____no_output_____

These are the true class-number for those images. This is only used when plotting the images._____no_output_____

cls_true = data.test.cls[0:9]_____no_output_____

Get the predicted classes as One-Hot encoded arrays._____no_output_____

y_pred = model.predict(x=images)_____no_output_____

Get the predicted classes as integers._____no_output_____

cls_pred = np.argmax(y_pred,axis=1)_____no_output_____plot_images(images=images,
            cls_true=cls_true,
            cls_pred=cls_pred)_____no_output_____

### Examples of Mis-Classified Images

We can plot some examples of mis-classified images from the test-set.

First we get the predicted classes for all the images in the test-set:_____no_output_____

y_pred = model.predict(x=data.test.images)_____no_output_____

Then we convert the predicted class-numbers from One-Hot encoded arrays to integers._____no_output_____

cls_pred = np.argmax(y_pred,axis=1)_____no_output_____

Plot some of the mis-classified images._____no_output_____

plot_example_errors(cls_pred)_____no_output_____

## Conclusion

This tutorial showed how to optimize the hyper-parameters of a neural network using Bayesian optimization. We used the scikit-optimize (`skopt`) library which is still under development, but it is already an extremely powerful tool. It was able to substantially improve on hand-tuned hyper-parameters in a small number of iterations. This is vastly superior to Grid Search and Random Search of the hyper-parameters, which would require far more computational time, and would most likely find inferior hyper-parameters, especially for more difficult problems._____no_output_____## Exercises

These are a few suggestions for exercises that may help improve your skills with TensorFlow. It is important to get hands-on experience with TensorFlow in order to learn how to use it properly.

You may want to backup this Notebook before making any changes.

* Try and run 100 or 200 iterations of the optimization instead of just 40 iterations. What happens to the plotted landscapes?
* Try some of the other optimization methods from scikit-optimize such as `forest_minimize` instead of `gp_minimize`. How do they perform?
* Try using another acquisition function for the optimizer e.g. Probability of Improvement.
* Try optimizing more hyper-parameters with the Bayesian optimization. For example, the kernel-size and number of filters in the convolutional-layers, or the batch-size used in training.
* Add a hyper-parameter for the number of convolutional layers and implement it in `create_model()`. Note that if you have pooling-layers after the convolution then the images are downsampled, so there is a limit to the number of layers you can have before the images become too small.
* Look at the plots. Do you think that some of the hyper-parameters may be irrelevant? Try and remove these parameters and redo the optimization of the remaining hyper-parameters.
* Use another and more difficult dataset with image-files.
* Train for more epochs. Does it improve the classification accuracy on the validiation- and test-sets? How does it affect the time-usage?
* Explain to a friend how the program works._____no_output_____## License (MIT)

Copyright (c) 2016-2018 by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE._____no_output_____