# Notebook from arisp99/MIPTools
Path: base_resources/analysis-template-with-qual.ipynb

# How to use code cells in this notebook
If a code cell starts with
```python
# RUN
```
Run the cell by CTRL+Enter, or the Run button above.

If a code cell starts with
```python
# USER INPUT
```
User input is needed before running the cell. Usually there will be a cell preceding this which gives an example for the values to be provided.

If a code cell starts with
```python
# OPTIONAL USER INPUT
```
User input is needed before running the cell. However, some defaults are provided, so make sure that either the settings will work for your run, or change them appropriately.

If a cell starts with
#### Example cell
These cells are not code cells but examples of user inputs from the test data analysis for the actual code cell that follows it, informing the user about the formatting etc.

**Important note on entering input:** When entering user input, please make sure you follow the formatting provided in the example cells. For example, when the parameter is text, make sure you have quotation marks around the parameters but when it is a number, do not enclose in quotes. If it is a list, then provide a list in brackets._____no_output_____

# RUN
import sys
sys.path.append("/opt/src")
import mip_functions as mip
import probe_summary_generator
import pickle
import json
import copy
import os
import numpy as np
import subprocess
%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('ggplot')
from matplotlib.lines import Line2D
plt.rcParams['svg.fonttype'] = 'none'
import pandas as pd
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
import allel
wdir = "/opt/analysis/"
data_dir = "/opt/data/"Classes reloading.
functions reloading

#### Example cell
```python

# provide the MIPWrangler output files
# which must be located in the /opt/data directory within the container.
# if more than one run is to be merged, provide all files
info_files = ["test_run_JJJ_DR1_VAR4_20191126.txt.gz"]

# sample sheets associated with each wrangler file,
# in the same order as the wrangler files.
sample_sheets = ["sample_list.tsv"]

# No input below
info_files = [data_dir + i for i in info_files]
sample_sheets = [data_dir + s for s in sample_sheets]
pd.concat([pd.read_table(s) for s in sample_sheets],
         ignore_index=True).groupby(["sample_set", "probe_set"]).first()
```_____no_output_____

# USER INPUT

# provide the MIPWrangler output files
# which must be located in the /opt/data directory within the container.
# if more than one run is to be merged, provide all files
info_files = []

# sample sheets associated with each wrangler file,
# in the same order as the wrangler files.
sample_sheets = []

# No input below
info_files = [data_dir + i for i in info_files]
sample_sheets = [data_dir + s for s in sample_sheets]
pd.concat([pd.read_table(s) for s in sample_sheets],
         ignore_index=True).groupby(["sample_set", "probe_set"]).first()_____no_output_____

The above table shows the unique sample_set, probe_set combinations in the sample sheets provided. Select which combinations should be used for this analysis._____no_output_____#### Example cell
```python
sample_groups = [["JJJ", "DR1,VAR4"]]
```

If more than one combination is to be used, the input will be a list of lists, for example:
```python
sample_groups = [["sample_set_1", "probe_set_1"], ["sample_set_2", "probe_set_2"]]
```_____no_output_____

# USER INPUT
sample_groups = [[]]_____no_output_____

### Specify the species and the probe set used
These two are important parameters to determine which files will be used for analysis.

For the species, the options are: "pf" for *Plasmodium falciparum*, "pv" for *Plasmodium vivax*, "hg19" for *Homo sapiens* genome assembly hg19/GRCh37 and "hg38" for *Homo sapiens* genome assembly hg38/GRCh38
___
Probe sets also must be specified. Check the output of the sample sheet summary above under **probe_set** field for a reminder of what the probe set of interest is. This is usually a three letter code or codes separated by a comma. _____no_output_____#### Example cell
```python
species = "pf"
probe_sets_used = "DR1,VAR4"
```

It is also possible to analyse just a subset of probe sets that has been used. For example, if the data has both DR1 and VAR4 probe sets but I want to analyse only the DR1 set:
```python
species = "pf"
probe_sets_used = "DR1"
```
Note that I'd still need to specifiy "DR1,VAR4" in the sample_groups above._____no_output_____

# USER INPUT
species =
probe_sets_used = _____no_output_____

#### Example cell
```python
# available cpu count
processorNumber = 20

## extra bwa options for haplotype alignment
# use "-a" for getting all alignments
# use "-L 500" to penalize soft clipping
# use "-t" to set number of available processors
bwaExtra = ["-t", str(processorNumber)]
```_____no_output_____

# OPTIONAL USER INPUT
# available cpu count
processorNumber = 20

## extra bwa options for haplotype alignment
# use "-a" for getting all alignments
# use "-L 500" to penalize soft clipping
# use "-t" to set number of available processors
bwaExtra = ["-t", str(processorNumber)]_____no_output_____

### Get/Set the analysis settings
Use the settings template for the species specified to get the  analysis settings and change the vaules specified in the above cell. This will create a template_settings.txt file in your analysis directory and a settings.txt file to be used for the analysis. These files also will serve as a reference of analysis settings for the sake of reproducibility.

The last step of the below cell attempts to save a file to the /opt/project_resources directory. If you do not have write permission to the location, you cannot save that file. However, if a file has been previously saved in the directory, it will be fine._____no_output_____

# RUN

# copy the template settings file
temp_settings_file = "/opt/resources/templates/analysis_settings_templates/settings.txt"
subprocess.call(["scp", temp_settings_file, "/opt/analysis/template_settings.txt"])

# extract the settings template
temp_settings = mip.get_analysis_settings("/opt/analysis/template_settings.txt")

# update bwa settings with the options set above
bwaOptions = temp_settings["bwaOptions"]
try:
    bwaOptions.extend(bwaExtra)
except AttributeError:
    bwaOptions = [bwaOptions]
    bwaOptions.extend(bwaExtra)

# Create a list from the probe_sets string
mipSetKey = probe_sets_used.split(",") + [""]

# create a dictionary for which settings should be updated
# using the user specified parameters.
update_keys = {"processorNumber": processorNumber,
               "bwaOptions": bwaOptions,
               "species": species,
               "mipSetKey" : mipSetKey}
# update the settings
for k, v in update_keys.items():
    temp_settings[k] = v
# create a settings file in the analysis directory.
settings_file = "settings.txt"
settings_path = os.path.join(wdir, settings_file)
mip.write_analysis_settings(temp_settings, settings_path)
settings = mip.get_analysis_settings(wdir + settings_file)
# create probe sets dictionary
try:
    mip.update_probe_sets("/opt/project_resources/mip_ids/mipsets.csv",
                         "/opt/project_resources/mip_ids/probe_sets.json")
except IOError:
    pass_____no_output_____

# Process run data
First section of the data analysis involves processing the MIPWrangler output files, combining data from multiple runs (if necessary), mapping haplotypes and creating summary files and plots showing how the sequencing runs went._____no_output_____## MIPWrangler output file processing
Below operation combines output files from multiple runs, summing up count data belonging to the same libraries.

Libraries are labeled by combining three fields in the sample sheet: sample_name-sample_set-replicate, which makes the Sample ID. If two different libraries has the same Sample ID (same three fields, but a different LibraryPrep identifier), the overlapping libraries will be assigned new replicate numbers such that there are no shared IDs any more. A warning will be printed in that case, and the original sample ID and the new one will be written to the samples.tsv file generated in the analysis directory.

If only a single output file is used, then the below operation just filters and renames some columns from the original file._____no_output_____

# RUN
if len(info_files) > 1:
    mip.combine_info_files(wdir,
                           settings_file,
                           info_files,
                           sample_sheets,
                           settings["mipsterFile"],
                           sample_sets=sample_groups)
else:
    mip.process_info_file(wdir,
                          settings_file,
                          info_files,
                          sample_sheets,
                          settings["mipsterFile"],
                          sample_sets=sample_groups)_____no_output_____

## Filter and map haplotype sequences
Align each haplotype sequence to the reference genome. Remove off target haplotypes. All haplotype mappings will be saved to the disk so off targets can be inspected if needed.

Some filters can be applied to remove noise:
*  minHaplotypeBarcodes: minimum total UMI cut off across all samples.
*  minHaplotypeSamples: minimum number of samples a haplotype is observed in.
*  minHaplotypeSampleFraction: minimum fraction of samples a haplotype is observed in.

It is probably safe to apply minimal count filters like at least 10 UMIs across samples and at least two samples. However, most data sets will be easily handled without these filters. So it may be better to not filter at this step unless the downstream operations are taking too much resources. However, filters can and should be applied after variant calls are made._____no_output_____#### Example cell
```python
# filter haplotype sequences based on the number of total supporting UMIs
settings["minHaplotypeBarcodes"] = 1
# filter haplotype sequences based on the number of samples they were observed in
settings["minHaplotypeSamples"] = 1
# filter haplotype sequences based on the fraction of samples they were observed in
settings["minHaplotypeSampleFraction"] = 0.0001
```_____no_output_____

# OPTIONAL USER INPUT
# filter haplotype sequences based on the number of total supporting UMIs
settings["minHaplotypeBarcodes"] = 1
# filter haplotype sequences based on the number of samples they were observed in
settings["minHaplotypeSamples"] = 1
# filter haplotype sequences based on the fraction of samples they were observed in
settings["minHaplotypeSampleFraction"] = 0.0001 _____no_output_____#RUN
mip.map_haplotypes(settings)
mip.get_haplotype_counts(settings)_____no_output_____

### Preview the mapping results
Plotting the probe coverage by samples is a good  way to see overall experiment perfomance. It shows if a probe has at least 1 barcode (or however many is specified below) for a given sample.

Dark columns point to poor performing probes whereas dark rows indicate poor samples. Note that this excludes samples with no reads at all. Use "all_barcode_counts.csv" file if those are of interest as well.

Some parameters can be supplied to the plotting function as noted in the comments._____no_output_____

# OPTIONAL USER INPUT

# coverage filter: anything below this number will be considered absent
barcode_threshold = 10
# font size for tick labels for x and y axis
tick_label_size=5
# font size for heat map color bar
cbar_label_size=5
# figure resolution
dpi=300
# present/absent colors
absent_color='black'
present_color='green'
# Save the plot in the analysis directory?
# If false, plots the graph here.
save=False
# How frequent the x and y-axis ticks should be
# every nth column will have  a tick
ytick_freq=None
xtick_freq=None
# rotation of xtick labels
xtick_rotation=90_____no_output_____# OPTIONAL USER INPUT
barcode_counts = pd.read_csv(wdir + "barcode_counts.csv",
             header = [0,1], index_col = 0)
mip.plot_performance(barcode_counts,
                     barcode_threshold=barcode_threshold,
                     tick_label_size=tick_label_size,
                     cbar_label_size=cbar_label_size,
                     dpi=dpi,
                     absent_color=absent_color,
                     present_color=present_color,
                     save=save,
                     ytick_freq=ytick_freq,
                     xtick_freq=xtick_freq,
                     xtick_rotation=xtick_rotation)_____no_output_____

### Look at summary stats
There are summary statistics and meta data (if provided) we can use to determine if coverage is enough, whether further sequencing is necessary, and how to proceed if further sequencing will be needed._____no_output_____

# RUN
sample_summary = pd.read_csv(wdir + "sample_summary.csv")
sample_summary.head()_____no_output_____

### Plot total barcode count vs probe coverage
A scatter plot of total barcode count vs number of probes covered at a certain barcode count is a good way to see how the relationship between total coverage and probe coverage, which is useful in determining how to proceed to the next experiments or analyses._____no_output_____

# RUN
f = sns.pairplot(data = sample_summary,
                x_vars = "Barcode Count",
                y_vars = "targets_with_10_barcodes",
                plot_kws={"s": 10})
f.fig.set_size_inches(5,5)
f.fig.set_dpi(150)
_ = plt.xticks(rotation=45)_____no_output_____

## Repooling capture reactions for further sequencing.
### Factors to consider:
1. What do you we want to accomplish? In most cases, we would like to get enough coverage for a number of probes for each sample. For example, the test data contains **50 probes** in total. Let's say it is sufficient if we had a coverage of **10** or more for each probe for a sample. Then, we would not want to sequence any more of that sample.
```python
target_coverage_count = 50
target_coverage_key='targets_with_10_barcodes'
```
Alternatively, we can set a goal of a fraction of total probes to reach a certain coverage rather than an absolute number of probes. For 95% of the maximum number of probes observed (47 in this case):
```python
target_coverage_fraction = 0.95
target_coverage_key='targets_with_10_barcodes'
```
Although we set our goal to 47 probes, it is likely that some sample will never reach that number regardless of how much we sequence, if there is a deletion in the region, for example. So it makes sense to set a total coverage threshold after which we don't expect more data. Looking at the plot above, it seems like after 1000 barcode counts, we would reach our goal for most samples.
```python
high_barcode_threshold = 10000
```
Another metric to use for determining if we want to sequence a sample more is the average read count per barcode count. This value indicates we have sequenced each unique molecular index in our sample so many times, so when the value is high, it is unlikely that we'd get more UMIs by sequencing the same library more. It makes more sense for a fresh MIP capture from these samples if more data is needed.
```python
barcode_coverage_threshold=10
```
Some samples perform very poorly for one reason or another. There are two options for these samples for repooling consideration: 1) Repool as much as we can for the next run, 2) Assuming there is a problem in the capture reaction, set up a new MIP capture reaction for these samples. It makes more sense to use option 1 if this is the first sequencing data using this library. Use option 2 if this library have been repooled at a higher volume already, but still producing poor data.
```python
barcode_count_threshold=100 # samples below total barcode count of this value is considered low coverage
low_coverage_action='Repool' # what to do for low coverage samples (Repool or Recapture)
```
Sometimes a handful of samples show uneven coverage of loci, i.e. they have very good coverage of a handful of loci but poor coverage in others, which may point to a problem with the sample or the experiment in general. These samples are determined by comparing the subset of samples that reached the goal we set (completed samples) and those that have not. We look at the number of barcodes per probe for _completed_ samples and get 25th percentile (or other percentile as set) and assume that if a sample on average has this many barcodes per target, it should have reached the set goal. For example, if on average _completed_ samples, i.e. samples that cover 47 probes at 10 barcodes or more, have 10000 total barcodes, they would have ~200 (10000/47) barcodes per target covered. And if an _incomplete_ sample has 5000 total barcodes and only 10 targets covered, this value would be 500 for that sample and it would be flagged as **uneven coverage** in repooling document.
```python
assesment_key='targets_with_1_barcodes' # coverage key to compare "complete" and "incomplete" samples
good_coverage_quantile=0.25 # percentile to set the threshold
```_____no_output_____#### Example cell
```python
high_barcode_threshold = 10000
target_coverage_count = None
target_coverage_fraction = 0.95
target_coverage_key = 'targets_with_10_barcodes'
barcode_coverage_threshold = 10
barcode_count_threshold = 100
low_coverage_action = 'Recapture'
assesment_key = 'targets_with_1_barcodes'
good_coverage_quantile = 0.25
```_____no_output_____

# USER INPUT
high_barcode_threshold =
low_coverage_action = _____no_output_____# OPTIONAL USER INPUT
target_coverage_count = None
target_coverage_fraction = 0.95
target_coverage_key = 'targets_with_10_barcodes'
barcode_coverage_threshold = 10
barcode_count_threshold = 100
assesment_key = 'targets_with_1_barcodes'
good_coverage_quantile = 0.25_____no_output_____# RUN
meta = pd.read_csv(wdir + "run_meta.csv")
data_summary = pd.merge(sample_summary, meta)
mip.repool(wdir,
           data_summary,
           high_barcode_threshold,
           target_coverage_count=target_coverage_count,
           target_coverage_fraction=target_coverage_fraction,
           target_coverage_key=target_coverage_key,
           barcode_coverage_threshold=barcode_coverage_threshold,
           barcode_count_threshold=barcode_count_threshold,
           low_coverage_action=low_coverage_action,
           assesment_key=assesment_key,
           good_coverage_quantile=good_coverage_quantile,
           output_file='repool.csv')_____no_output_____

### Inspect the repool document
Library to completion field in the repool document has the value (volume) of how much from a sample should be pooled for re-sequencing. These values are only rough estimates and care should be taken to make sure there will be enough material to sequence._____no_output_____

# RUN
pd.read_csv(wdir + "repool.csv").head()_____no_output_____

# Variant Calling
Second part of the analysis involves variant calling and variant analysis. _____no_output_____### Options for freebayes wrapper
```Python
align = True # Default is set to true, fastq files and bam files per sample
# will be created in 'fastq_dir' and 'bam_dir'.
# it should be set to false if bam files are available.

settings = settings # analysis settings dictionary created above.

bam_files = None # default is to use all bam files within the bam_dir.
# if specific files should be used, then they can be specified in a list.

verbose = True # prints errors and warnings as well as saving to disk.
# if set to false, it will print that there is an error which will
# be saved to disk which should be inspected for details.

targets_file = None # force calls on specific loci even if there is
# no observations satisfying filter criteria. Useful in cases of targeted
# mutations such as drug resistance mutations.
# Usually a file at "/opt/project_resources/targets.tsv" would be present
# if the project requires it. Then targets_file should be set to this path.

# paths for input-output files with default values that can be left unchanged
fastq_dir, bam_dir, vcf_file, settings_file, errors_file, warnings_file

# additional options to pass to freebayes directly:
options = [] # see below for suggestions and possibilities.
```
#### Additional options for freebayes caller.
Most of the freebayes options are shown below in the **freebayes help** section at the bottom of this document. Some options are integrated into the python wrapper freebayes_call, but others should be added depending on the data type, species etc.

integrated options:
```bash
    -r region
            limit calls to a specific region.
            This is done internally, splitting the results into contigs and processing each contig
            separately (in parallel if multiple cpus are available).
            Per-contig vcf files are concatenated at the end into a single file.
    -@ targets.vcf
            force calls on positions provided in the vcf file
            a vcf file is generated if a tab separated file containing targets are provided.
    -L --bam-list
            a list of bam files to be used. By default, all bams in bams directory will be used.
            A list of specific bams can be specified to freebayes_call as bam_files option.
```
options to consider adding for parasite sequencing:
```bash
    --pooled-continuous
             This option does not make assumptions about the ploidy when making genotype calls.
             It makes sense for a mixed ploidy sample such as parasite infected blood DNA.
             variants are still called as diploid.
    --min-alternate-fraction 0.01
             since we assume a pooled continuous sample, it would be better to set a within
             samlpe allele frequency threshold to remove noise.
             this is likely not needed when dealing with a diploid sample because a frequency
             of 0.01 will likely be considered noise for a diploid sample.
    --min-alternate-count 2
             number of reads supporting a variant to consider for genotype calls.
             having this at at least 2 is good. It will be possible to process
             variants with 1 reads in postprocessing steps if a specific variant
             is observed at least in one sample at > 1 reads. So this removes the
             variant from consideration if no sample has > 1 reads supporting it.
    --min-alternate-total 10
             total read support for a variant across samples.
```
options to consider for human sequences:
```bash
    --min-mapping-quality 0
             default for this setting is 1. I do not think this is helping much in
             addressing mapping issues. However, reads in copy number variant regions
             may have 0 mapping quality. These would be worth to keep, but they
             should be handled appropriately at postprocessing steps.
    --min-alternate-count 2
    --min-alternate-fraction 0.05 (default)
    --min-alternate-total 10
```_____no_output_____####  Example cell
```python
# provide freebayes options.
# These will be directy passed to freebayes

# example for plasmodium falciparum calls
options = ["--pooled-continuous",
           "--min-alternate-fraction", "0.01",
           "--min-alternate-count", "2",
           "--haplotype-length", "3",
           "--min-alternate-total", "10",
           "--use-best-n-alleles", "70",
           "--genotype-qualities", "--gvcf",
           "--gvcf-dont-use-chunk", "true"]

# example for human genome calls with gvcf output
options = ["--haplotype-length", "-1",
           "--use-best-n-alleles", "50",
           "--genotype-qualities", "--gvcf",
           "--gvcf-dont-use-chunk", "true"]

# example for human genome calls without gvcf output
options = ["--haplotype-length", "-1",
           "--use-best-n-alleles", "50",
           "--genotype-qualities"]
```_____no_output_____

# USER INPUT

# provide freebayes options.
# These will be directy passed to freebayes
options = _____no_output_____# OPTIONAL USER INPUT

align=True
verbose=True
# where to save generated fastq files
fastq_dir="/opt/analysis/padded_fastqs"
# where to save generated bam files
bam_dir="/opt/analysis/padded_bams"
# where to save the output vcf file
vcf_file="/opt/analysis/variants.vcf.gz"
# where is the targeted variants file
targets_file="/opt/project_resources/targets.tsv"
# where to save errors and warnings generated by freebayes
errors_file="/opt/analysis/freebayes_errors.txt"
warnings_file="/opt/analysis/freebayes_warnings.txt"_____no_output_____# OPTIONAL USER INPUT

# freebayes caller creates fastq files from the haplotype sequences
# by default 20 bp flanking sequence from the reference genome is added
# to ensure correct deletion calls when they are towards the ends.
# This assumes the 20 bp flank is wild type, however the sequence
# is given a quality of 1, which should help avoiding some issues.
# If this is not desired, set the below parameter to 0
fastq_padding = 20_____no_output_____# RUN
r = mip.freebayes_call(
        settings=settings,
        options=options,
        align=align,
        verbose=verbose,
        fastq_dir=fastq_dir,
        bam_dir=bam_dir,
        vcf_file=vcf_file,
        targets_file=targets_file,
        bam_files=None,
        errors_file=errors_file,
        warnings_file=warnings_file,
        fastq_padding=fastq_padding)_____no_output_____

# Potential Exit Point
The above cell should create the vcf file **variants.vcf.gz** in the analysis directory (assuming the vcf_file parameter was not changed). You can use this file in any downstream pipeline that utilizes vcf files. The variants are called rather generously, i.e. even when there is a good chance that a called variant is not there, with the assumption that the vcf will be further processed using whatever metric is deemed suitable for the data set.

In addition, you should now have a **padded_fastqs** subdirectory in your analysis directory containing fastq files for each sample. These fastq files contain 1 read per UMI and they are stitched together and cleaned up using MIPWrangler. You should be able to use these files in any pipeline that accepts fastq inputs (virtually all bioinformatics pipelines).

Finally, there is a **padded_bams** folder containing bam files for each sample obtained by mapping the *padded fastqs* to the reference genome.  _____no_output_____---
The next steps in this notebook are dealing with postprocessing the vcf file in the ways that we found useful so far._____no_output_____# Processing Variant Calls
Freebayes produces high quality vcf files with haplotype based variant calls. This is important for getting more accurate calls, especially for complex regions where SNVs may overlap with indels and there may be many possible alleles as opposed to a simple biallelic SNV call.

haplotype based variant example:

chr1  1000 AAA,AGC,TGC

However, it may be desired to "decompose" these complex variants for some applications. For example, if we are interested in knowing the prevalence of a specific drug resistance mutation, it would make sense to combine all variants containing this mutation even though they may be part of different haplotypes, and hence are represented in the vcf in different variants.

Decomposed variants:

chr1  1000 A T
chr1  1001 A G
chr1  1002 A C

vcf_to_tables function takes the vcf file generated by freebayes and generates allele count and coverage data in table form. It is possible to decompose and aggregate amino acid and/or nucleotide level variants. 3 files containing count data are generated: alternate_table.csv, reference_table.csv, coverage_table.csv, for alt allele, ref allele and coverage count values for each variant, respectively.

It first separates the multiallelic calls to bi-allelic calls.

#### annotate, default=True
It then annotates variants using snpEff.

#### geneid_to_genename, default=None
Variant annotation provides a gene ID (e.g. PF3D7_0709000) but it does not provide common gene names (e.g. crt). If common names are used in target files, or they are desired in general, a tab separated gene ID to gene name file can be used. **gene_name and gene_id** columns are required. If no file is provided, gene name will be the same as the gene ID.

#### aggregate_aminoacids, default=False
If aminoacid level aggregation is requested, it decomposes multi amino acid missense variants into single components and aggregates the alternate allele and coverage counts per amino acid change. For example, Asn75Glu change for crt gene is a known drug resistance mutation in Plasmodium falciparum. There may be 3 separate variants in the vcf file that contain this mutation: Asn75Glu, MetAsn75IleGlu, Asn75Glu_del76-80*. All three has the missense variant Asn75Glu. While the first two  are simple changes, the third is a complex change including a 5 amino acid deletion and a stop codon following Asn75Glu. In this case, it makes sense to combine the counts of the first two variants towards Asn75Glu counts but the third one is debatable because of the complexity; i.e. the drug resistance mutation Asn75Glu probably is not that improtant in that context because of the stop codon following it. So we decompose the simple changes and aggregate but leave complex changes as they are. If aminoacid aggregation is carried out, file names will contain AA tag.

#### target_aa_annotation, default=None
It is also possible to annotate the targeted variants (such as Asn75Glu above) in the generated tables as 'Targeted' in case some analysis should be carried out on targeted variants only. A tab separated file containing the annotation details is required for this operation. **gene_name, aminoacid_change and mutation_name** are required fields. If a variants gene_name and aminoacid_change are matching to a row in the target file, that variant will be marked as targeted and will have the correspondign mutation name. Note that if common gene name conversion (see above) is not used, the gene_name column in this file must match the actual gene ID and not the common name. It may be more convenient to keep the gene IDs in the target file as well and use that file for ID to name mapping. **aggregate_aminoacids must be set to True** for this option to be used.

#### aggregate_nucleotides, default=False
A similar aggregation can be done at nucleotide level. If specified, biallelic variants will be decomposed using the tool **vt decompose_blocksub**. By default it decomposes block substitutions that do not include indels. However, it is also possible to decompose complex variants including indels by providing -a option. For possible decompose options see vt help:
```bash
vt decompose_blocksub options :
  -p  Output phased genotypes and PS tags for decomposed variants [false]
  -m  keep MNVs (multi-nucleotide variants) [false]
  -a  enable aggressive/alignment mode [false]
  -d  MNVs max distance (when -m option is used) [2]
  -o  output VCF file [-]
  -I  file containing list of intervals []
  -i  intervals []
  -?  displays help
```
If nucleotide level aggregation is done, the file names will include AN tag.

#### target_nt_annotation, default=None
Annotation of targeted nucleotides requires a file similar to the targeted amino acid annotation. However, the required fields for this annotation are: CHROM, POS, REF, ALT and mutation_name. **aggregate_nucleotides must be set to True** for this option to be used.

#### aggregate_none, default=False
It is also possible to generate count tables without doing any aggregation. This will generate the 3 count files, and all of the variant information included in the vcf file will be a separate column in the table's index. For annotated initial vcf files, or if annotate option is selected, each subfield in the INFO/ANN field will have its own column.

#### min_site_qual, default=-1
Filter variant sites for a minimum QUAL value assigned by the variant caller. This value is described in freebayes manual as:
```bash
Of primary interest to most users is the QUAL field, which estimates the probability that there is a polymorphism at the loci described by the record. In freebayes, this value can be understood as 1 - P(locus is homozygous given the data). It is recommended that users use this value to filter their results, rather than accepting anything output by freebayes as ground truth.

By default, records are output even if they have very low probability of variation, in expectation that the VCF will be filtered using tools such as vcffilter in vcflib, which is also included in the repository under vcflib/. For instance,

freebayes -f ref.fa aln.bam | vcffilter -f "QUAL > 20" >results.vcf

removes any sites with estimated probability of not being polymorphic less than phred 20 (aka 0.01), or probability of polymorphism > 0.99.

In simulation, the receiver-operator characteristic (ROC) tends to have a very sharp inflection between Q1 and Q30, depending on input data characteristics, and a filter setting in this range should provide decent performance. Users are encouraged to examine their output and both variants which are retained and those they filter out. Most problems tend to occur in low-depth areas, and so users may wish to remove these as well, which can also be done by filtering on the DP flag.
```
Therefore, a **minimum of 1** should be used as a min_site_qual to remove low quality sites. If a site is annotated as **targeted**, the site will be kept regardless of its qual value, however, the alternate observation counts for the site may be reset to zero depending on the min_target_site_qual value described below.

#### min_target_site_qual, default=-1
If a variant site is targeted but the site qual is lower than this,
reset the alternate observation counts to 0. It may be best to leave
this at the default value since there is usually additional evidence
that a targeted variant exists in a samples compared to a de novo
variant, i.e. those variants that are targeted had been observed in other samples/studies._____no_output_____#### Example cell
```python
# provide a file that maps gene names to gene IDs
# this is necessary when targeted variant annotations use
# gene names instead of gene IDs
geneid_to_genename = "/opt/project_resources/geneid_to_genename.tsv"
# annotate targted amino acid changes in the tables.
target_aa_annotation = "/opt/project_resources/targets.tsv"
# decompose multi amino acid changes and combine counts of
# resulting single amino acid changes
aggregate_aminoacids = True
# decompose MNVs and combine counts for resulting SNVs
aggregate_nucleotides = True
# annotate targeted nucleotide changes in the tables.
target_nt_annotation = None
```_____no_output_____

# USER INPUT

# provide a file that maps gene names to gene IDs
# this is necessary when targeted variant annotations use
# gene names instead of gene IDs. Otherwise provide None
geneid_to_genename =
# annotate targeted amino acid changes in the tables
# using the file, or otherwise provide None
target_aa_annotation =
# decompose multi amino acid changes and combine counts of
# resulting single amino acid changes
aggregate_aminoacids =
# decompose MNVs and combine counts for resulting SNVs
aggregate_nucleotides =
# annotate targeted nucleotide changes in the tables.
target_nt_annotation = _____no_output_____# OPTIONAL USER INPUT

# analysis settings dictionary
settings = settings
# provide the path to the settings file
# if settings dictionary has not been loaded
settings_file = None
# use snpEff to annotate the variants
annotate = True
# additional vt options for decomposing nucleotides.
# Supply ["-a"] to include indels and complex variants
# in decomposition, or other options shown above if desired.
decompose_options = []
# was the initial vcf file was annotated by snpEff?
annotated_vcf = False
# create tables for variants as they are in the vcf file
# without decomposing compex variants or indels.
# Multiallelic variants will be split into biallelic.
aggregate_none = True
# filter variant sites for quality
min_site_qual = 1
# reset targeted variant counts to zero
# when the site quality is below this value
min_target_site_qual = 0
# reset genotypes in the vcf file to NA
# and depth to 0 if FORMAT/GQ value for a variant/sample
# is below this value:
min_genotype_qual = 1
# reset alt allele count in the vcf file to 0
# if FORMAT/QA value divided by FORMAT/AO for a variant/sample
# is below this value:
min_mean_alt_qual = 15 # average quality cut off for variants
# There are also available, similar filters for:
# min_mean_ref_qual : resetting low qual reference allele counts
# min_alt_qual : similar to min_mean_alt_qual, but for total qual score
# min_ref_qual : similar to min_alt_qual but for reference alleles

# prefix for output files, if desired.
# this is useful when different quality thresholds etc will be used
# to avoid overwriting the files. For example, if min_genotype_qual = 1
# and min_mean_alt_qual = 15 is used, a suitable prefix could be
# "gq1.mqa15."
output_prefix = ""_____no_output_____# RUN

# input vcf file
vcf_file = vcf_file.split("/")[-1]
mip.vcf_to_tables_fb(
     vcf_file,
     settings=settings,
     settings_file=settings_file,
     annotate=annotate,
     geneid_to_genename=geneid_to_genename,
     target_aa_annotation=target_aa_annotation,
     aggregate_aminoacids=aggregate_aminoacids,
     target_nt_annotation=target_nt_annotation,
     aggregate_nucleotides=aggregate_nucleotides,
     decompose_options=decompose_options,
     annotated_vcf=annotated_vcf,
     aggregate_none=aggregate_none,
     min_site_qual=min_site_qual,
     min_target_site_qual=min_target_site_qual,
     min_genotype_qual=min_genotype_qual,
     min_mean_alt_qual=min_mean_alt_qual,
     output_prefix=output_prefix)_____no_output_____

## Tables created
alternate_XX_table.csv files will contain the ALT allele count for that table type while coverage_XX_table.csv will contain the depth of coverage at each locus.
### Nucleotide changes (aggregated)
For some projects we may be interested in specific single nucleotide changes. For these, it would make sense to decompose multi nucleotide changes and combine counts of the same single nucleotide changes. Two tables will be generated for count and coverage data for aggregated nucleotide changes:

**alternate_AN_table.csv** file in the analysis directory is created if aggregate_nucleotides option was selected when creating data tables. This table has the UMI counts for each alternate nucleotide.

**coverage_AN_table.csv** file is the corresponding coverage depth for each variant's position.

**genotypes_AN_table.csv** file contains the aggregated value of the genotypes called by freebayes: 0/0->0, 0/1->1, 1/1->2, N/A (.) ->-1. When calls from multiple variants are aggregated; if all 0/0 then -> 0, if any 0/0 and non-0/0 then -> 1, if all 1/1 then -> 2_____no_output_____### Amino acid changes (aggregated)
For some projects we may be interested in the amino acid changes, particularly specific, targeted amino acid changes, such as drug resistance mutations in *Plasmodium falciparum*, which is the data set provided for pipeline test. For these type of projects, we may want to analyze the variants from the amino acid perspective, rather than nucleotide changes which is standard output for variant callers.

**alternate_AA_table.csv** file in the analysis directory is created if aggregate_aminoacids option was selected when creating data tables. This table has the UMI counts for each alternate amino acid.

**coverage_AA_table.csv** file is the corresponding coverage depth for each variant's position.

**genotypes_AA_table.csv** file contains the aggregated value of the genotypes called by freebayes: 0/0->0, 0/1->1, 1/1->2, N/A (.) ->-1. When calls from multiple variants are aggregated; if all 0/0 then -> 0, if any 0/0 and non-0/0 then -> 1, if all 1/1 then -> 2_____no_output_____### Nucleotide changes (not aggregated)
For some projects we may be interested in keeping composite variants as they are called by the pipeline. These will include MNVs, comlplex variants including indels, etc. Two tables will be generated for count and coverage data for original nucleotide changes:

**alternate_table.csv** file in the analysis directory is created if aggregate_none option was selected when creating data tables. This table has the UMI counts for each alternate nucleotide.

**coverage_table.csv** file is the corresponding coverage depth for each variant's position.

**genotypes_table.csv** file contains the aggregated value of the genotypes called by freebayes: 0/0->0, 0/1->1, 1/1->2, N/A (.) ->-1._____no_output_____# Calling genotypes, prevalences and filtering data
The original vcf file created by freebayes contain the genotypes determined by the program itself. In addition, genotype values for aggregated and non-aggregated nucleotides and aminoacids are also available as *_genotypes_table.csv files as described above.

However, the default parameters generating the vcf file are not very strict. In this part of the analysis we will apply various filters to the count tables and generate genotype calls based on those filters._____no_output_____### Chose which tables to analyse
Select the type of data to analyse. Make sure the count file is matching the coverage file. e.g. alternate_XX_table and coverage_XX_table, XX must be the same value (AA, AN or nothing)._____no_output_____#### Example cell
```python
mutation_count_file = "/opt/analysis/q1.mqa15.alternate_AA_table.csv"
mutation_coverage_file = "/opt/analysis/q1.mqa15.coverage_AA_table.csv"
```_____no_output_____

# USER INPUT

mutation_count_file =
mutation_coverage_file = _____no_output_____# RUN
mutation_counts = pd.read_csv(mutation_count_file,
                              header=list(range(6)),
                              index_col=0)
mutation_counts.head()_____no_output_____# RUN
mutation_coverage = pd.read_csv(mutation_coverage_file,
                                index_col=0,
                                header=list(range(6)))
mutation_coverage.head()_____no_output_____

### Set your filters
1.  **min_coverage**: how many UMIs are needed to for a genomic position for a sample to reliably call genotypes. If we set min_coverage = 10, any locus within a sample that is covered below this threshold will have an NA genotype.
2.  **min_count**: if a genomic position have enough coverage, how many UMIs supporting an ALT allele call is needed for a reliable call. If we set min_count = 2, any mutation with an  call that has less than 2 barcodes supporting the ALT call will revert to REF.
3.  **min_freq**: a minimum within sample allele frequency threshold to consider a variant valid. If set to 0.01, for example, a variant locus in a sample that is at 0.005 frequency for the ALT allele within the sample, the locus would be called REF, if the within sample AF is between 0.01 and 0.99, it would be considered HET, and if > 0.99, it would be homozygous ALT._____no_output_____#### Example cell
```python
# filter mutation counts for minimum count parameter
# by setting counts to zero if it is below threshold
min_count = 2
# filter loci without enough coverage by setting
# coverage to zero if it is below threshold
min_coverage = 10
# call genotypes using the minimum within sample
# allele frequency
min_freq = 0
```_____no_output_____

# USER INPUT

# filter mutation counts for minimum count parameter
# by setting counts to zero if it is below threshold
min_count =
# filter loci without enough coverage by setting
# coverage to zero if it is below threshold
min_coverage =
# call genotypes using the minimum within sample
# allele frequency
min_freq = _____no_output_____# RUN

# import the PCA module which has genotype calling and
# filtering functions
import PCA

gt_calls = PCA.call_genotypes(mutation_counts, mutation_coverage,
                              min_count, min_coverage, min_freq)
gt_calls.keys()_____no_output_____

### What are the dataframes generated by call_genotypes function and how  are they generated?_____no_output_____**filtered_mutation_counts**: take the mutation_counts table, if a cell's value is below *min_count*, reset that cell's value to zero, otherwise leave as is.  _____no_output_____

# RUN
filtered_mutation_counts = gt_calls["filtered_mutation_counts"]
filtered_mutation_counts.head()_____no_output_____

**filtered_mutation_coverage**: take the mutation_coverage table, if a cell's value is below *min_coverage*, reset that cell's value to zero, otherwise leave as is._____no_output_____

# RUN
filtered_mutation_coverage = gt_calls["filtered_mutation_coverage"]
filtered_mutation_coverage.head()_____no_output_____

**wsaf**: divide *filtered_mutation_counts* table by *filtered_mutation_coverage* table, yielding within sample allele frequencies.  _____no_output_____

# RUN
freq = gt_calls["wsaf"]
freq.head()_____no_output_____

**genotypes**: take the *wsaf* table, if a cell's value is less than *min_freq* set the genotype value to 0 (homozygous wild type); if the cell's value is more than (*1 - min_freq*) set the genotype value to 2 (homozygous mutant), if the cell's value is between *min_freq* and (*1 - min_freq*) set the genotype value to 1 (heterozygous/mixed).  _____no_output_____

# RUN
genotypes = gt_calls["genotypes"]
genotypes.head()_____no_output_____

**prevalences**: take the *genotypes* table, if a cell's value is 2, reset its value to 1; otherwise leave as is._____no_output_____

# RUN
prevalences = gt_calls["prevalences"]
prevalences.head()_____no_output_____

## Filter genotypes / prevalences
It is generally a good idea to do some basic noise removal once the genotypes are created. Some suggestions are provided here._____no_output_____### Filter variants that are always at low WSAF
If a variant is only seen at a low frequency within samples, it is a good indication that it could be just noise. Here we will set a number of samples and minimum WSAF threshold to remove such noise.

```python
num_samples_wsaf = 2
min_wsaf = 0.5
wsaf_filter = ((freq > min_wsaf).sum()) >= num_samples_wsaf
```

The above options will keep the variants that are in at > 0.5 WSAF in at least 2 samples._____no_output_____

# USER INPUT
num_samples_wsaf =
min_wsaf = _____no_output_____wsaf_filter = ((freq > min_wsaf).sum()) >= num_samples_wsaf
print(("{} of {} variants will remain after the wsaf filter").format(
    wsaf_filter.sum(), freq.shape[1]))_____no_output_____

### Filter variants that are observed with low UMI counts
If a variant is only supported by a low number of UMIs across the entire sample set, it is another indication of noise.

```python
num_samples_umi = 2
min_umi = 3
umi_filter = ((filtered_mutation_counts >= min_umi).sum()) > num_samples_umi
```

The above options will keep the variants that are supported by at least 3 UMIs in at least 2 samples._____no_output_____

# USER INPUT
num_samples_umi =
min_umi = _____no_output_____# RUN
umi_filter = ((filtered_mutation_counts >= min_umi).sum()) > num_samples_umi
print(("{} of {} variants will remain after the UMI filter").format(
    umi_filter.sum(), freq.shape[1]))_____no_output_____

### Keep variants that were targeted
In most projects there are a number of variants that we would like to report, even if they are not seen in the sample set. We would like to stop those variants from being removed by the above filters._____no_output_____

# RUN
targ = freq.columns.get_level_values("Targeted") == "Yes"_____no_output_____

### Combine filters
Keep the variants that are either targeted or passing filters_____no_output_____

variant_mask = targ | (wsaf_filter & umi_filter)
print(("{} variants will remain in the final call set.\n"
       "{} variants were targeted and will be kept; and {} will be removed by "
       "the combined UMI and WSAF filters.").format(
    variant_mask.sum(), targ.sum(), (wsaf_filter & umi_filter).sum()))_____no_output_____

## Filter data tables with the combined filters_____no_output_____

filtered_genotypes = genotypes.loc[:, variant_mask]
filtered_genotypes.head()_____no_output_____filtered_prevalences = prevalences.loc[:, variant_mask]
filtered_prevalences.head()_____no_output_____

## freebayes help documentation
Below are the various sections of freebayes --help output showing examples and options.
```bash
citation: Erik Garrison, Gabor Marth
          "Haplotype-based variant detection from short-read sequencing"
          arXiv:1207.3907 (http://arxiv.org/abs/1207.3907)

author:   Erik Garrison <erik.garrison@bc.edu>, Marth Lab, Boston College, 2010-2014
version:  v1.3.1-dirty
```

### overview:
```bash
    To call variants from aligned short-read sequencing data, supply BAM files and
    a reference.  FreeBayes will provide VCF output on standard out describing SNPs,
    indels, and complex variants in samples in the input alignments.

    By default, FreeBayes will consider variants supported by at least 2
    observations in a single sample (-C) and also by at least 20% of the reads from
    a single sample (-F).  These settings are suitable to low to high depth
    sequencing in haploid and diploid samples, but users working with polyploid or
    pooled samples may wish to adjust them depending on the characteristics of
    their sequencing data.

    FreeBayes is capable of calling variant haplotypes shorter than a read length
    where multiple polymorphisms segregate on the same read.  The maximum distance
    between polymorphisms phased in this way is determined by the
    --max-complex-gap, which defaults to 3bp.  In practice, this can comfortably be
    set to half the read length.

    Ploidy may be set to any level (-p), but by default all samples are assumed to
    be diploid.  FreeBayes can model per-sample and per-region variation in
    copy-number (-A) using a copy-number variation map.

    FreeBayes can act as a frequency-based pooled caller and describe variants
    and haplotypes in terms of observation frequency rather than called genotypes.
    To do so, use --pooled-continuous and set input filters to a suitable level.
    Allele observation counts will be described by AO and RO fields in the VCF output.

```

### examples:
```bash
    # call variants assuming a diploid sample
    freebayes -f ref.fa aln.bam >var.vcf

    # call variants assuming a diploid sample, providing gVCF output
    freebayes -f ref.fa --gvcf aln.bam >var.gvcf

    # require at least 5 supporting observations to consider a variant
    freebayes -f ref.fa -C 5 aln.bam >var.vcf

    # discard alignments overlapping positions where total read depth is greater than 200
    freebayes -f ref.fa -g 200 aln.bam >var.vcf

    # use a different ploidy
    freebayes -f ref.fa -p 4 aln.bam >var.vcf

    # assume a pooled sample with a known number of genome copies
    freebayes -f ref.fa -p 20 --pooled-discrete aln.bam >var.vcf

    # generate frequency-based calls for all variants passing input thresholds
    freebayes -f ref.fa -F 0.01 -C 1 --pooled-continuous aln.bam >var.vcf

    # use an input VCF (bgzipped + tabix indexed) to force calls at particular alleles
    freebayes -f ref.fa -@ in.vcf.gz aln.bam >var.vcf

    # generate long haplotype calls over known variants
    freebayes -f ref.fa --haplotype-basis-alleles in.vcf.gz \
                        --haplotype-length 50 aln.bam

    # naive variant calling: simply annotate observation counts of SNPs and indels
    freebayes -f ref.fa --haplotype-length 0 --min-alternate-count 1 \
        --min-alternate-fraction 0 --pooled-continuous --report-monomorphic >var.vcf
```

### input:
```bash
   -b --bam FILE   Add FILE to the set of BAM files to be analyzed.
   -L --bam-list FILE
                   A file containing a list of BAM files to be analyzed.
   -c --stdin      Read BAM input on stdin.
   -f --fasta-reference FILE
                   Use FILE as the reference sequence for analysis.
                   An index file (FILE.fai) will be created if none exists.
                   If neither --targets nor --region are specified, FreeBayes
                   will analyze every position in this reference.
   -t --targets FILE
                   Limit analysis to targets listed in the BED-format FILE.
   -r --region <chrom>:<start_position>-<end_position>
                   Limit analysis to the specified region, 0-base coordinates,
                   end_position not included (same as BED format).
                   Either '-' or '..' maybe used as a separator.
   -s --samples FILE
                   Limit analysis to samples listed (one per line) in the FILE.
                   By default FreeBayes will analyze all samples in its input
                   BAM files.
   --populations FILE
                   Each line of FILE should list a sample and a population which
                   it is part of.  The population-based bayesian inference model
                   will then be partitioned on the basis of the populations.
   -A --cnv-map FILE
                   Read a copy number map from the BED file FILE, which has
                   either a sample-level ploidy:
                      sample_name copy_number
                   or a region-specific format:
                      seq_name start end sample_name copy_number
                   ... for each region in each sample which does not have the
                   default copy number as set by --ploidy. These fields can be delimited
                   by space or tab.

```

### output:
```bash
   -v --vcf FILE   Output VCF-format results to FILE. (default: stdout)
   --gvcf
                   Write gVCF output, which indicates coverage in uncalled regions.
   --gvcf-chunk NUM
                   When writing gVCF output emit a record for every NUM bases.
   -& --gvcf-dont-use-chunk BOOL
                   When writing the gVCF output emit a record for all bases if
                   set to "true" , will also route an int to --gvcf-chunk
                   similar to --output-mode EMIT_ALL_SITES from GATK
   -@ --variant-input VCF
                   Use variants reported in VCF file as input to the algorithm.
                   Variants in this file will included in the output even if
                   there is not enough support in the data to pass input filters.
   -l --only-use-input-alleles
                   Only provide variant calls and genotype likelihoods for sites
                   and alleles which are provided in the VCF input, and provide
                   output in the VCF for all input alleles, not just those which
                   have support in the data.
   --haplotype-basis-alleles VCF
                   When specified, only variant alleles provided in this input
                   VCF will be used for the construction of complex or haplotype
                   alleles.
   --report-all-haplotype-alleles
                   At sites where genotypes are made over haplotype alleles,
                   provide information about all alleles in output, not only
                   those which are called.
   --report-monomorphic
                   Report even loci which appear to be monomorphic, and report all
                   considered alleles, even those which are not in called genotypes.
                   Loci which do not have any potential alternates have '.' for ALT.
   -P --pvar N     Report sites if the probability that there is a polymorphism
                   at the site is greater than N.  default: 0.0.  Note that post-
                   filtering is generally recommended over the use of this parameter.
   --strict-vcf
                   Generate strict VCF format (FORMAT/GQ will be an int)

```

### population model:
```bash
-T --theta N    The expected mutation rate or pairwise nucleotide diversity
                   among the population under analysis.  This serves as the
                   single parameter to the Ewens Sampling Formula prior model
                   default: 0.001
   -p --ploidy N   Sets the default ploidy for the analysis to N.  default: 2
   -J --pooled-discrete
                   Assume that samples result from pooled sequencing.
                   Model pooled samples using discrete genotypes across pools.
                   When using this flag, set --ploidy to the number of
                   alleles in each sample or use the --cnv-map to define
                   per-sample ploidy.
   -K --pooled-continuous
                   Output all alleles which pass input filters, regardles of
                   genotyping outcome or model.
```
### reference allele:
```bash
   -Z --use-reference-allele
                   This flag includes the reference allele in the analysis as
                   if it is another sample from the same population.
   --reference-quality MQ,BQ
                   Assign mapping quality of MQ to the reference allele at each
                   site and base quality of BQ.  default: 100,60
```
### allele scope:
```bash
   -n --use-best-n-alleles N
                   Evaluate only the best N SNP alleles, ranked by sum of
                   supporting quality scores.  (Set to 0 to use all; default: all)
   -E --max-complex-gap N
      --haplotype-length N
                   Allow haplotype calls with contiguous embedded matches of up
                   to this length. Set N=-1 to disable clumping. (default: 3)
   --min-repeat-size N
                   When assembling observations across repeats, require the total repeat
                   length at least this many bp.  (default: 5)
   --min-repeat-entropy N
                   To detect interrupted repeats, build across sequence until it has
                   entropy > N bits per bp. Set to 0 to turn off. (default: 1)
   --no-partial-observations
                   Exclude observations which do not fully span the dynamically-determined
                   detection window.  (default, use all observations, dividing partial
                   support across matching haplotypes when generating haplotypes.)

  These flags are meant for testing.
  They are not meant for filtering the output.
  They actually filter the input to the algorithm by throwing away alignments.
  This hurts performance by hiding information from the Bayesian model.
  Do not use them unless you can validate that they improve results!

   -I --throw-away-snp-obs     Remove SNP observations from input.
   -i --throw-away-indels-obs  Remove indel observations from input.
   -X --throw-away-mnp-obs     Remove MNP observations from input.
   -u --throw-away-complex-obs Remove complex allele observations from input.

  If you need to break apart haplotype calls to obtain one class of alleles,
  run the call with default parameters, then normalize and subset the VCF:
    freebayes ... | vcfallelicprimitives -kg >calls.vcf
  For example, this would retain only biallelic SNPs.
    <calls.vcf vcfsnps | vcfbiallelic >biallelic_snp_calls.vcf
```
### indel realignment:
```bash
   -O --dont-left-align-indels
                   Turn off left-alignment of indels, which is enabled by default.

```

### input filters:
```bash
   -4 --use-duplicate-reads
                   Include duplicate-marked alignments in the analysis.
                   default: exclude duplicates marked as such in alignments
   -m --min-mapping-quality Q
                   Exclude alignments from analysis if they have a mapping
                   quality less than Q.  default: 1
   -q --min-base-quality Q
                   Exclude alleles from analysis if their supporting base
                   quality is less than Q.  default: 0
   -R --min-supporting-allele-qsum Q
                   Consider any allele in which the sum of qualities of supporting
                   observations is at least Q.  default: 0
   -Y --min-supporting-mapping-qsum Q
                   Consider any allele in which and the sum of mapping qualities of
                   supporting reads is at least Q.  default: 0
   -Q --mismatch-base-quality-threshold Q
                   Count mismatches toward --read-mismatch-limit if the base
                   quality of the mismatch is >= Q.  default: 10
   -U --read-mismatch-limit N
                   Exclude reads with more than N mismatches where each mismatch
                   has base quality >= mismatch-base-quality-threshold.
                   default: ~unbounded
   -z --read-max-mismatch-fraction N
                   Exclude reads with more than N [0,1] fraction of mismatches where
                   each mismatch has base quality >= mismatch-base-quality-threshold
                   default: 1.0
   -$ --read-snp-limit N
                   Exclude reads with more than N base mismatches, ignoring gaps
                   with quality >= mismatch-base-quality-threshold.
                   default: ~unbounded
   -e --read-indel-limit N
                   Exclude reads with more than N separate gaps.
                   default: ~unbounded
   -0 --standard-filters  Use stringent input base and mapping quality filters
                   Equivalent to -m 30 -q 20 -R 0 -S 0
   -F --min-alternate-fraction N
                   Require at least this fraction of observations supporting
                   an alternate allele within a single individual in the
                   in order to evaluate the position.  default: 0.05
   -C --min-alternate-count N
                   Require at least this count of observations supporting
                   an alternate allele within a single individual in order
                   to evaluate the position.  default: 2
   -3 --min-alternate-qsum N
                   Require at least this sum of quality of observations supporting
                   an alternate allele within a single individual in order
                   to evaluate the position.  default: 0
   -G --min-alternate-total N
                   Require at least this count of observations supporting
                   an alternate allele within the total population in order
                   to use the allele in analysis.  default: 1
   --min-coverage N
                   Require at least this coverage to process a site. default: 0
   --limit-coverage N
                   Downsample per-sample coverage to this level if greater than this coverage.
                   default: no limit
   -g --skip-coverage N
                   Skip processing of alignments overlapping positions with coverage >N.
                   This filters sites above this coverage, but will also reduce data nearby.
                   default: no limit

```

### population priors:
```bash
   -k --no-population-priors
                   Equivalent to --pooled-discrete --hwe-priors-off and removal of
                   Ewens Sampling Formula component of priors.
```
### mappability priors:
```bash
   -w --hwe-priors-off
                   Disable estimation of the probability of the combination
                   arising under HWE given the allele frequency as estimated
                   by observation frequency.
   -V --binomial-obs-priors-off
                   Disable incorporation of prior expectations about observations.
                   Uses read placement probability, strand balance probability,
                   and read position (5'-3') probability.
   -a --allele-balance-priors-off
                   Disable use of aggregate probability of observation balance between alleles
                   as a component of the priors.
```
### genotype likelihoods:
```bash
   --observation-bias FILE
                   Read length-dependent allele observation biases from FILE.
                   The format is [length] [alignment efficiency relative to reference]
                   where the efficiency is 1 if there is no relative observation bias.
   --base-quality-cap Q
                   Limit estimated observation quality by capping base quality at Q.
   --prob-contamination F
                   An estimate of contamination to use for all samples.  default: 10e-9
   --legacy-gls    Use legacy (polybayes equivalent) genotype likelihood calculations
   --contamination-estimates FILE
                   A file containing per-sample estimates of contamination, such as
                   those generated by VerifyBamID.  The format should be:
                       sample p(read=R|genotype=AR) p(read=A|genotype=AA)
                   Sample '*' can be used to set default contamination estimates.
```
### algorithmic features:
```bash
   --report-genotype-likelihood-max
                   Report genotypes using the maximum-likelihood estimate provided
                   from genotype likelihoods.
   -B --genotyping-max-iterations N
                   Iterate no more than N times during genotyping step. default: 1000.
   --genotyping-max-banddepth N
                   Integrate no deeper than the Nth best genotype by likelihood when
                   genotyping. default: 6.
   -W --posterior-integration-limits N,M
                   Integrate all genotype combinations in our posterior space
                   which include no more than N samples with their Mth best
                   data likelihood. default: 1,3.
   -N --exclude-unobserved-genotypes
                   Skip sample genotypings for which the sample has no supporting reads.
   -S --genotype-variant-threshold N
                   Limit posterior integration to samples where the second-best
                   genotype likelihood is no more than log(N) from the highest
                   genotype likelihood for the sample.  default: ~unbounded
   -j --use-mapping-quality
                   Use mapping quality of alleles when calculating data likelihoods.
   -H --harmonic-indel-quality
                   Use a weighted sum of base qualities around an indel, scaled by the
                   distance from the indel.  By default use a minimum BQ in flanking sequence.
   -D --read-dependence-factor N
                   Incorporate non-independence of reads by scaling successive
                   observations by this factor during data likelihood

```_____no_output_____


# Notebook from MagriLab/API-ESN
Path: source/.ipynb_checkpoints/ESN_bias-checkpoint.ipynb

<code>
# In this file the Echo State Network reservoir is defined as a cell for a tensorflow Recurrent Neural Netowrk

# to pass to py_func
def np_eigenvals(x):
    return np.linalg.eigvals(x).astype('complex128')

# Definition of the EchoState NN as a child of RNNCell of tensorflow
# It generates the states of the reservoir; there is no output matrix
class EchoStateRNNCell(tf.keras.layers.AbstractRNNCell):

    def __init__(self, num_units, num_inputs=1, decay=0.1, rho=0.6,
                 sparseness=0.0,
                 sigma_in=1.0,
                 b_in=1.0,
                 rng=None,
                 activation=None,
                 reuse = False,
                 win=None,
                 wecho=None):
        """
        Args:
            num_units: int, Number of units in the ESN cell.
            num_inputs: int, The number of inputs to the RNN cell.
            decay: float, Decay/leaking of the ODE of each unit.
            rho: float, Target spectral radius 1.
            sparseness: float [0,1], sparseness of the inner weight matrix.
            rng: np.random.RandomState, random number generator. (to be able to have always the same random matrix...)
            activation: Nonlinearity to use.
            reuse: if reusing existing matrix
            win: input weights matrix
            wecho: echo state matrix
        """

        # Basic RNNCell initialization (see tensorflow documentation)
        super(EchoStateRNNCell, self).__init__() # use the initializer of RNNCell (i.e. defines a bunch of standard variables)

        # hyperparameters
        self._num_units  = num_units
        self._activation = activation
        self._num_inputs = num_inputs
        self.rho = rho
        self.decay = decay
        self.sparseness = sparseness
        self.sigma_in = sigma_in
        self.b_in  = np.reshape(b_in, [1,1])

        # Random number generator initialization
        self.rng = rng
        if rng is None:
            self.rng = np.random.RandomState()

        # build initializers for tensorflow variables
        if (reuse == False):
            self.win = self.buildInputMatrix()
            self.wecho = self.buildEchoMatrix()
        else: #in case they are passed as an argument
            self.win = win.astype('float64')
            self.wecho = wecho.astype('float64')

        # convert the weight to tf variable
        self.Win   = tf.Variable(self.win, name='Win', trainable=False)
        self.Wecho = tf.Variable(self.wecho, name='Wecho', trainable=False)

        self.setEchoStateProperty()

    @property
    def state_size(self):
        return self._num_units

    @property
    def output_size(self):
        return self._num_units

    # function that has to be implemented in the tensorflow framework
    # return the output and new state of the RNN for given inputs and current state of the RNN
    # state and output coincide in the reservoir
    def call(self, inputs, state):
        """ Reservoir state evolution:
            x = f(W*inp + U*g(x)).
        """

        new_state = self._activation(
                    tf.matmul(tf.concat([inputs,self.b_in],axis=1) , self.Win) +
                    tf.matmul(state[0], self.Wecho))

        return new_state, new_state

    def setEchoStateProperty(self):
        """ optimize U to obtain alpha-improved echo-state property """
        self.Wecho = self.normalizeEchoStateWeights(self.Wecho)

    # construct the Win matrix (dimension num_inputs x num_units)
    def buildInputMatrix(self):
        """
            Returns:

            Matrix representing the
            input weights to an ESN
        """

        # Each unit is connected randomly to a given input with a weight from a
        # uniform distribution between +- sigma_in (input scaling)
        # +1 for added input bias in the input matrix
        W = np.zeros((self._num_inputs+1,self._num_units))
        for i in range(self._num_units):
            W[self.rng.randint(0,self._num_inputs+1),i] = \
            self.rng.uniform(-self.sigma_in,self.sigma_in)

        return W.astype('float64')

    def getInputMatrix(self):
        return self.win

    def buildEchoMatrix(self):
        """
            Returns:

            Matrix representing the
            inner weights to an ESN
        """

        # Build random matrix from uniform distribution
        W = self.rng.uniform(-1.0,1.0, [self._num_units, self._num_units]).astype("float64") * \
                (self.rng.rand(self._num_units, self._num_units) < (1. - self.sparseness) ) # trick to add zeros to have the sparseness required
        return W

    def normalizeEchoStateWeights(self, W):
        # Sets the spectral radius rho

        eigvals = tf.py_function(np_eigenvals, [W], tf.complex128)
        W = W / tf.reduce_max(tf.abs(eigvals))*self.rho #imposes spectral radius

        return W

    def getEchoMatrix(self):
        return self.wecho_____no_output_____
</code>


# Notebook from alexisbloor/big-data-challenge
Path: pt1_big_data.ipynb

import os
# Find the latest version of spark 3.0  from http://www.apache.org/dist/spark/ and enter as the spark version
# For example:
# spark_version = 'spark-3.0.3'
spark_version = 'spark-3.2.0'
os.environ['SPARK_VERSION']=spark_version

# Install Spark and Java
!apt-get update
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q http://www.apache.org/dist/spark/$SPARK_VERSION/$SPARK_VERSION-bin-hadoop2.7.tgz
!tar xf $SPARK_VERSION-bin-hadoop2.7.tgz
!pip install -q findspark

# Set Environment Variables
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = f"/content/{spark_version}-bin-hadoop2.7"

# Start a SparkSession
import findspark
findspark.init()_____no_output_____!wget https://jdbc.postgresql.org/download/postgresql-42.2.9.jar--2022-01-25 22:40:20--  https://jdbc.postgresql.org/download/postgresql-42.2.9.jar
Resolving jdbc.postgresql.org (jdbc.postgresql.org)... 72.32.157.228, 2001:4800:3e1:1::228
Connecting to jdbc.postgresql.org (jdbc.postgresql.org)|72.32.157.228|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 914037 (893K) [application/java-archive]
Saving to: ‘postgresql-42.2.9.jar’

postgresql-42.2.9.j 100%[===================>] 892.61K  1.54MB/s    in 0.6s

2022-01-25 22:40:21 (1.54 MB/s) - ‘postgresql-42.2.9.jar’ saved [914037/914037]

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("BigData").config("spark.driver.extraClassPath","/content/postgresql-42.2.9.jar").getOrCreate()_____no_output_____from pyspark import SparkFiles
# Load in employee.csv from S3 into a DataFrame
url = "https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_us_Gift_Card_v1_00.tsv.gz"
spark.sparkContext.addFile(url)

df = spark.read.option('encoding', 'UTF-8').csv(SparkFiles.get(""), sep="\t", header=True, inferSchema=True)
df.show(10)+-----------+-----------+--------------+----------+--------------+--------------------+----------------+-----------+-------------+-----------+----+-----------------+--------------------+--------------------+-----------+
|marketplace|customer_id|     review_id|product_id|product_parent|       product_title|product_category|star_rating|helpful_votes|total_votes|vine|verified_purchase|     review_headline|         review_body|review_date|
+-----------+-----------+--------------+----------+--------------+--------------------+----------------+-----------+-------------+-----------+----+-----------------+--------------------+--------------------+-----------+
|         US|   24371595|R27ZP1F1CD0C3Y|B004LLIL5A|     346014806|Amazon eGift Card...|       Gift Card|          5|            0|          0|   N|                Y|          Five Stars|Great birthday gi...| 2015-08-31|
|         US|   42489718| RJ7RSBCHUDNNE|B004LLIKVU|     473048287|Amazon.com eGift ...|       Gift Card|          5|            0|          0|   N|                Y|Gift card for the...|It's an Amazon gi...| 2015-08-31|
|         US|     861463|R1HVYBSKLQJI5S|B00IX1I3G6|     926539283|Amazon.com Gift C...|       Gift Card|          5|            0|          0|   N|                Y|          Five Stars|                Good| 2015-08-31|
|         US|   25283295|R2HAXF0IIYQBIR|B00IX1I3G6|     926539283|Amazon.com Gift C...|       Gift Card|          1|            0|          0|   N|                Y|            One Star|                Fair| 2015-08-31|
|         US|     397970| RNYLPX611NB7Q|B005ESMGV4|     379368939|Amazon.com Gift C...|       Gift Card|          5|            0|          0|   N|                Y|          Five Stars|I can't believe h...| 2015-08-31|
|         US|   18513645|R3ALA9XXMBEDZR|B004KNWWU4|     326384774|Amazon Gift Card ...|       Gift Card|          5|            0|          0|   N|                Y|Perfect for most ...|Perfect!  Nice & ...| 2015-08-31|
|         US|   22484620|R3R8PHAVJFTPDF|B004LLIKVU|     473048287|Amazon.com eGift ...|       Gift Card|          5|            0|          0|   N|                Y|          Five Stars|            excelent| 2015-08-31|
|         US|   14765851|R18WWEK8OIXE30|BT00CTP2EE|     775486538|Amazon.com Gift C...|       Gift Card|          5|            0|          0|   N|                Y|          Five Stars|Great and Safe Gi...| 2015-08-31|
|         US|   18751931|R1EGUNQON2J277|B004LLIKVU|     473048287|Amazon.com eGift ...|       Gift Card|          1|            0|          0|   N|                Y|            One Star|       What?????????| 2015-08-31|
|         US|   15100528|R21Z4M4L98CPU2|B004W8D102|     595099956|Amazon Gift Card ...|       Gift Card|          5|            0|          0|   N|                Y|          Five Stars|This was just too...| 2015-08-31|
+-----------+-----------+--------------+----------+--------------+--------------------+----------------+-----------+-------------+-----------+----+-----------------+--------------------+--------------------+-----------+
only showing top 10 rows

total_rows = df.count()
print(f"The total number of records is {total_rows}.")The total number of records is 149086.

## Examine the schema_____no_output_____

df.printSchema()root
 |-- marketplace: string (nullable = true)
 |-- customer_id: integer (nullable = true)
 |-- review_id: string (nullable = true)
 |-- product_id: string (nullable = true)
 |-- product_parent: integer (nullable = true)
 |-- product_title: string (nullable = true)
 |-- product_category: string (nullable = true)
 |-- star_rating: integer (nullable = true)
 |-- helpful_votes: integer (nullable = true)
 |-- total_votes: integer (nullable = true)
 |-- vine: string (nullable = true)
 |-- verified_purchase: string (nullable = true)
 |-- review_headline: string (nullable = true)
 |-- review_body: string (nullable = true)
 |-- review_date: string (nullable = true)

## Rename columns_____no_output_____

df_review_id = df.select(['review_id', 'customer_id', 'product_id', 'product_parent', 'review_date'])
df_review_id.show(5)+--------------+-----------+----------+--------------+-----------+
|     review_id|customer_id|product_id|product_parent|review_date|
+--------------+-----------+----------+--------------+-----------+
|R27ZP1F1CD0C3Y|   24371595|B004LLIL5A|     346014806| 2015-08-31|
| RJ7RSBCHUDNNE|   42489718|B004LLIKVU|     473048287| 2015-08-31|
|R1HVYBSKLQJI5S|     861463|B00IX1I3G6|     926539283| 2015-08-31|
|R2HAXF0IIYQBIR|   25283295|B00IX1I3G6|     926539283| 2015-08-31|
| RNYLPX611NB7Q|     397970|B005ESMGV4|     379368939| 2015-08-31|
+--------------+-----------+----------+--------------+-----------+
only showing top 5 rows

df_review_id.printSchema()root
 |-- review_id: string (nullable = true)
 |-- customer_id: integer (nullable = true)
 |-- product_id: string (nullable = true)
 |-- product_parent: integer (nullable = true)
 |-- review_date: string (nullable = true)

from pyspark.sql.functions import to_date_____no_output_____review_df = df_review_id.withColumnRenamed('REVIEW_DATED', 'review_dated') _____no_output_____products_df = df.select(['product_id', 'product_title'])_____no_output_____products_df = products_df.distinct()_____no_output_____total_rows = products_df.count()
print(f"The total nember of records after removing duplicates {total_rows}")The total nember of records after removing duplicates 1780
from pyspark.sql.functions import desc
customers_df = df.select(['customer_id'])\
  .groupby("customer_id")\
  .agg({"customer_id": "count"})\
  .orderBy(desc("count(customer_id)"))

customers_df.show()_____no_output_____total_rows = customers_df.count()
print(total_rows)_____no_output_____vine_df = df.select(['review_id', 'star_rating', 'helpful_votes', 'total_votes', 'vine'])
vine_df.show()_____no_output_____# Configuration for RDS instance
mode="append"
jdbc_url = "jdbc:postgresql://<insert endpoint>:5432/my_data_class_db"
config = {"user":"",
          "password": "",
          "driver":"org.postgresql.Driver"}_____no_output_____# Write DataFrame to table

review_df.write.jdbc(url=jdbc_url, table='review_id_table', mode=mode, properties=config)_____no_output_____products_df.write.jdbc(url=jdbc_url, table='products_table', mode=mode, properties=config)_____no_output_____customers_df.write.jdbc(url=jdbc_url, table='customers_table', mode=mode, properties=config)_____no_output_____vine_df.write.jdbc(url=jdbc_url, table='vine_table', mode=mode, properties=config)_____no_output_____


# Notebook from theed-ml/notebooks
Path: 11-knn.ipynb

# K-Nearest Neighbors

Implement the [KNN classification algorithm](https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm) for classification.

### Objective

This notebook aims to introduce the nearest neighbors model in the classification problem. In this case, we will first implement the model and test it using the [Iris flower data set](https://archive.ics.uci.edu/ml/datasets/iris) where the goal is to the classe of a flower based on its features. Next, we will use the Endometrium vs. Uterus cancer data set, where goal is to separate **endometrium** tumors from the **uterine** ones.

### Learning objective

After finished this notebook, you should be able to explain **nearest neighbors model**, including how to use the scikit-learn implementation.

### Iris flower data set

![flower](https://archive.ics.uci.edu/ml/assets/MLimages/Large53.jpg)

The [Iris flower data set](https://archive.ics.uci.edu/ml/datasets/iris) consists of 150 data points, where each data point is a feature vector $\boldsymbol x \in \mathbb{R}^4$ describing the attribute of a flower in the data set.

The four dimensions represent:

1. sepal length in cm
2. sepal width in cm
3. petal length in cm
4. petal width in cm

and the corresponding target $y \in \mathbb{Z}$ describes the class of the flower. There are three classes of flowers in this data set. They are:

0. Iris Setosa
1. Iris Versicolour
2. Iris Virginica
_____no_output_____

import sys
assert sys.version_info >= (3, 6)

import numpy
assert numpy.__version__ >="1.17.3"
import numpy as np

import matplotlib.pyplot as plt

import pandas
assert pandas.__version__ >= "0.25.1"
import pandas as pd

import sklearn
assert sklearn.__version__ >= "0.21.3"

from sklearn import datasets

%matplotlib inline_____no_output_____

## 1. Implementing the nearest neighbors method_____no_output_____### 1.1. Loading the iris data set_____no_output_____

iris = datasets.load_iris()

print('data shape is {}'.format(iris.data.shape))
print('class shape is {}'.format(iris.target.shape))_____no_output_____

For the simplicity, we will only use the first two features (i.e., sepal length and sepal width) of the data set to classify the flowers.
_____no_output_____

X = iris.data[:, :2]
y = iris.target_____no_output_____

### 1.2. Visualizing the classes of the flowers_____no_output_____We create a scatter plot of the dataset below. The x and y axis represent the sepal length and sepal width of the dataset, and the color of the points represent the different classes of flowers._____no_output_____

from matplotlib.colors import ListedColormap

cmap_bold = ListedColormap(['#FF0000',  '#00FF00', '#0000FF'])

K = 3
x = X[-1]

fig, ax = plt.subplots(figsize=(4,4))
for i, iris_class in enumerate(['Iris Setosa', 'Iris Versicolour', 'Iris Virginica']):
    idx = y==i
    ax.scatter(X[idx,0], X[idx,1],
               c=cmap_bold.colors[i], edgecolor='k',
               s=20, label=iris_class);

ax.set(xlabel='sepal length (cm)', ylabel='sepal width (cm)')
ax.legend();_____no_output_____

The idea behind a k-nearest neighbor classifier is very simple:

1. Given a training set $\boldsymbol X \in \mathbb{R}^{N \times D}$ and $\boldsymbol y \in \mathbb{Z}^N$; where, $N$ is the number of data points in the data set, and $D$ is the dimensionality of the data.
2. We predict the label of a novel point $\boldsymbol x \in \mathbb{R}^{D}$ __as the label of the majority of its "K-nearest neighbor"__ by using distance measure (e.g., the Euclidean distance).

First, we need to compute the pairwise distance matrix with the distances between the rows of two matrices._____no_output_____

def pairwise_distance_matrix(X, Y):

    """Compute the pairwise distance between the rows of X and the rows of Y

    Arguments
    ----------
    X: ndarray of size (N, D)
    Y: ndarray of size (M, D)

    Returns
    --------
    distance_matrix: matrix of shape (N, M), each entry distance_matrix[i,j] is the distance between
    ith row of X and the jth row of Y.
    """

    N, D = X.shape
    M, _ = Y.shape

    distance_matrix = None

    return distance_matrix

def KNN(k, X, y, x):

    """K nearest neighbors

    Arguments
    ----------

    k: number of nearest neighbors
    X: training set
    y: training labels
    x: test set

    Returns
    --------

    number of k-nearest neighbors for each of the classes
    """

    x = x.reshape(1, -1) if len(x.shape) == 1 else x

    N, D = X.shape
    num_classes = None

    distances = None

    # make the predictions
    ypred = np.zeros(num_classes)

    # find the labels of the k-nearest neighbors
    classes = None

    for c in np.unique(classes):
        ypred[c] = len(classes[classes == c])

    return np.argmax(ypred)_____no_output_____

### 1.3. Visualizing the decision boundary of the kNN_____no_output_____

x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1

step = 0.1
xx, yy = np.meshgrid(np.arange(x_min, x_max, step),
                     np.arange(y_min, y_max, step))

ypred = []
for data in np.array([xx.ravel(), yy.ravel()]).T:
    ypred.append(KNN(K, X, y, data.reshape(1,2)))

fig, ax = plt.subplots(figsize=(5,5))

ax.pcolormesh(xx, yy, np.array(ypred).reshape(xx.shape), cmap=cmap_light)
ax.scatter(X[:,0], X[:,1], c=y, cmap=cmap_bold, edgecolor='k', s=20);_____no_output_____

## 2. Endometrium vs. Uterus cancer classification using scikit-learn

In this data set, each observation is a tumor, and it is described by the expression of 3,000 genes. The expression of a gene is a measure of how much of that gene is present in the biological sample. Because this affects how much of the protein this gene codes for is produced, and because proteins dictacte what cells can do, gene expression gives us valuable information about the tumor. In particular, the expression of the same gene in the same individual is different in different tissues.

The goal is to separate the **endometrium** tumors from the **uterine** ones._____no_output_____### 2.1. Loading the endometrium cancer data set_____no_output_____

# read the data tumor data set: `data/endometrium_uterus_tumor.csv`
endometrium_data = None_____no_output_____print(endometrium_data.shape)

endometrium_data.head()_____no_output_____np.unique(endometrium_data['Tissue'])_____no_output_____

1. Drop the **target** feature (i.e., `Tissue`) and the unnecessary ones for the classification.
2. Encode the target variable `Tissue` to be 0 for Endometrium tumeur and 1 for Uterus.

_Hint_: check the method `get_dummies` of pandas._____no_output_____

# drop from the dataframe the columns Tissue and ID_REF
X = None

# encode the target variable and get its values
y = None_____no_output_____

Plot a scatter of two variables of the data set. _____no_output_____

plt.scatter(X[:,0], X[:,1], c=y)_____no_output_____from sklearn import model_selection

def stratifiedKFold(y, num_folds):

    kf = model_selection.StratifiedKFold(n_splits=num_folds)
    folds_regr = [(tr, te) for (tr, te) in kf.split(np.zeros(y.size), y)]
    return folds_regr_____no_output_____

Create a 10-cross validation folds on the data_____no_output_____

cv_folds = stratifiedKFold(y, 10)_____no_output_____

Cross validate 20 *k*-NN classifiers on the loaded datset using the `cross_validate` function of the module `cross_validation`. _____no_output_____

from cross_validation import cross_validate_____no_output_____

### 2.2. *k*-nearest neighbours classifier

In scikit-learn, a k-neighbours classifier can be initialised as `knn_clf = sklearn.neighbors.KNeighborsClassifier(n_neighbors=k)`_____no_output_____

from sklearn import neighbors
from sklearn import metrics

aurocs_clf = []

# Create a range of values of k.
k_range    = range(1,40,2)

for k in k_range:
    clf_knn    = None # Create a k-neighbors classifier
    y_pred = cross_validate(X, y, clf_knn, cv_folds)

    fpr, tpr, thresholdss = metrics.roc_curve(y, y_pred[:,1])
    aurocs_clf.append(metrics.auc(fpr,tpr))_____no_output_____plt.plot(k_range, aurocs_clf, color='blue')
plt.xlabel('Number of nearest neighbours')
plt.ylabel('Cross-validated AUC')
plt.title('Nearest neighbours classification - cross validated AUC.')_____no_output_____

**Question.** Find the best value for the parameter `n_neighbors` by finding the one that gives the maximum value of AUC._____no_output_____

best_k = None
print(k_range[best_k])_____no_output_____

Use `sklearn.model_selection.GridSearchCV` to find the best number of neighbors (i.e., `n_neighbors` parameter)._____no_output_____

from sklearn import model_selection

classifier = None

param_grid = {'n_neighbors': k_range}
clf_knn_opt = model_selection.GridSearchCV(classifier,
                                           param_grid=param_grid,
                                           cv=cv_folds,
                                           scoring='roc_auc',
                                           iid=True)
clf_knn_opt.fit(X, y)_____no_output_____

Finding the best number of neighbors._____no_output_____

print (clf_knn_opt.best_params_)_____no_output_____

__Question__: Is the optimum equal to the one computed earlier (i.e. `best_k`)?_____no_output_____ _____no_output_____


# Notebook from IntelSoftware/scikit-learn_essentials
Path: 04_scikit-learn-intelex-KNN-using_kdtree/04_AnalyzeGalaxyBatch.ipynb

# Classify Stars in Colliding Galaxies

Using ~5000 labeled stars out of ~80,000 in each galaxy (170,000 stars total), make a model to predict and/or classify all the stars in the collection of the galaxy collision.

![KNN Classifier results for colliding galaxies](Assets/KNNPredictionResult.png "Classifying Stars in Colliding Galaxies")

As an optional assignment:
- Find all neighboring pairs of points in two approaching galaxies.

Follow the creation of two synthetically created star locations in Euclidean coordinates. The galaxies are set to collide with each other. Techniques described are an approach to classifying which stars belong to which galaxy after the collision. We also later identify the coordinates of pairs of points that lie within a radius R of each other. Points already lying with radius R in either galaxy are ignored, and the focus is narrowed to points in one galaxy in close proximity to the points in the other galaxy.

We synthesize ~ 80,000 stars in each of two galaxies—a fictitious GFFA ("galaxy far, far away") and a fictitious "Xeno", which is the backdrop for Superman's Krypton. Combined, this is over 100,000 stars, for which we search for neighboring pairs on the order of a few light years apart.

Play the following video which plots and rotates the combined galaxies. It plots a 3D scatter plot of the nearly 200,000 stars in the two galaxies. It gradually reduces the opacity of each star to ultimately reveal the one or two or handfuls of stars that are within a given radius of each other as a result of the collision of the two galaxies. The center bulge in each galaxy is so dense that the red zone stars are not visible until we turn the opacity of the stars down significantly.

<video controls src="Videos/CollidingGalaxiesAnimation.mp4" />
_____no_output_____
# Learning Objectives:

1. Apply Multiple Classification Algorithms with GPU to classify stars belonging to each galaxy within a combined super galaxy to determine most accurate model.
1. Apply Intel® Extension for Scikit-learn* patch and SYCL context to compute on available GPU resource.
1. **Optional**: Use KDtree, also known as k-d tree, to find all stars interacting within a given radius (say two light years) due to the collision of two galaxies - This has been moved to an separate optional notebook in the current folder.

## Library Dependencies:

 - **pip install pickle**
 - also requires these libraries if they are not already installed: **matplotlib, numpy, pandas, time, sklearn**

To synthesized the galaxy data uses a parametric equations described in the following paper [A New Formula Describing the Scaffold Structure of Spiral Galaxies](https://arxiv.org/ftp/arxiv/papers/0908/0908.0892.pdf) regarding the parametric equation of arm: :

$$ r \left( \phi \right) = \frac{A}{log(B \  \ tan(\frac{\phi}{2N}))  } $$

The synthesizer used here, generates an arbitrary number of arms, generates a 3D gauassian distribution of stars around a galactic center, then distributes a gausian distribiutoin  of stars along the length of each arm. In addition, it generates an arbitrary number of "globular clusters" of stars accoring to a 3D gaussian distribution sprinkled out randomly along the arm curves.

We also used rotation matrices  from this blog [3D Rotations and Euler angles in Python](https://www.meccanismocomplesso.org/en/3d-rotations-and-euler-angles-in-python/)

# Fictitious Galaxies:

We create two fictitious galaxies: GFFA ("galaxy far, far away") and Xeno (The purported galaxy for Superman's planet Krypton). We intersect the galaxies and use various classification algorithms to identify the stars in each galaxy.

# k Nearest Neighbors

kNN classification follows the general workflow described in the oneAPI Giyhub repository [Classification Usage Model](https://oneapi-src.github.io/oneDAL/daal/usage/training-and-prediction/classification.html#classification-usage-model).

k-Nearest Neighbors (kNN) classification is a classification (or regression) algorithm. The model of the kNN classifier is based on feature vectors and class labels from the training data set. It uses distances between points as a key element for classifying similar points.

# Random Forest

Random forests or random decision forests are an ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time. For classification tasks, the output of the random forest is the class selected by most trees.

# Define GPU Context and Fit Multiple Sklearn Models Using GPU

We try muliple models and multiple training size values to determine which comboination of models and training size yield optimimum Receiver Operator Characteristic (ROC) scores_____no_output_____# load and run 04_analyzeGalaxyBatch

run the following load command in the next cell.

%load 04_analyzeGalaxyBatch.py_____no_output_____

<code>
%load 04_analyzeGalaxyBatch.py_____no_output_____#!python 04_analyzeGalaxyBatch.py   # works on Windows
! chmod 755 q; chmod 755 run_ModelCompare.sh; if [ -x "$(command -v qsub)" ]; then ./q run_ModelCompare.sh; else ./run_ModelCompare.sh; fi_____no_output_____
</code>

# Plot result of classification

This time a static image of a previous run of the algorithm showing one galaxy edge on

![galaxyRed_BlueEdge.png](Assets/galaxyRed_BlueEdge.png)

# Notices & Disclaimers

Intel technologies may require enabled hardware, software or service activation.

No product or component can be absolutely secure.
Your costs and results may vary.

© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries.
*Other names and brands may be claimed as the property of others.
_____no_output_____