# Notebook from kfollette/AST337-Fall2017
Path: Labs/Lab1/Lab1.ipynb

# AST 337 In-Class Lab #1
### Wednesday, September 6, 2017

In this lab, you'll learn to read in and manipulate tabular data with the python package pandas and plot that data with the plotting module _matplotlib.pyplot_.

On the science end, you will compare H-R diagrams for two different open star clusters, a globular cluster, and a  population of field (non-cluster) stars._____no_output_____

#load packages
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline_____no_output_____

First off, we'll need to read in the data with the pandas function *read_csv*. A basic example is given below:_____no_output_____

pleiades = pd.read_csv('pleiades.csv')_____no_output_____

"pleiades" is now a pandas dataframe object, which is essentially a form of python table. To see what's stored in pleiades, execute the cell below. Anywhere you see ..., that means that there are a number of additional columns or rows that have been hidden. _____no_output_____

pleiades_____no_output_____

Perhaps more useful are the pandas .columns and .dtypes methods. Execute the cells below and then edit the descriptions of what each does in the cell below (double click on this text to get into the markdown cell, where you can type regular text)

(1) The .columns method does....

(2) The .dtypes method does...._____no_output_____

pleiades.columns_____no_output_____pleiades.dtypes_____no_output_____

The two columns that we care about for this lab are the Temperature (Teff) and the Luminosity (Lbol). The units for these two columns are Kelvin and Solar luminosities, respectively. As you will label these later in your plots, I'll note here that there's a special trick for getting the sun symbol in a Markdown cell using the typesetting system LaTeX.

Solar Luminosities = L$_{\odot}$

Double click on this cell to see how I made the symbol above.

Let's create a pandas series that we can manipulate from each of the columns of interest.  _____no_output_____

pleiades_L = pleiades["Lbol"]
pleiades_T = pleiades["Teff"]_____no_output_____

Note though from your .dtypes output above that both of these columns have dtype "object", which is not a data type that will allow us to manipulate them. For example, try executing the cell below, where we attempt to subtract the value 2 from each entry in the "pleiades_L" column. You should get an error..._____no_output_____

pleiades_L = pleiades_L - 2_____no_output_____

So we want to convert the type of these pandas series to be numeric, which we do with pandas.to_numeric, as below:_____no_output_____

pleiades_L_new = pd.to_numeric(pleiades_L, errors='coerce')
pleiades_T_new = pd.to_numeric(pleiades_T, errors='coerce')
# With "coerce", we are telling the to_numeric function to change any invalid entries to NaNs._____no_output_____

Now let's print the first ten elements of each array to verify that nothing weird happened during this conversion. _____no_output_____

pleiades_L[0:10], pleiades_L_new[0:10], pleiades_T[0:10], pleiades_T_new[0:10]_____no_output_____

Now let's plot these two quantities against one another. There are many ways to plot in pyplot, but I'll use the one that I find to work most consistently and intuitively below. _____no_output_____

fig,ax = plt.subplots(figsize=(7,7))
ax.plot(pleiades_T_new, pleiades_L_new)_____no_output_____

Yikes, that's ugly, right? That's because the default plot symbol is a line connecting all the points. In this case what we really want is a so-called scatterplot, which we can do easily by specifying the plotting marker right after the y variable, as below. Below I use 'o', which stands for the circle symbol. For the full list of matplotlib symbols, see [this link](https://matplotlib.org/api/markers_api.html)._____no_output_____

fig,ax = plt.subplots(figsize=(7,7))
ax.plot(pleiades_T_new, pleiades_L_new, 'o')_____no_output_____

Note the default plotting color is blue, but you can change this easily by adding a color shorthand before the marker shorthand. Below I use 'g' for green, but here again, there are lots of options, as outlined at [this link](https://matplotlib.org/api/colors_api.html)._____no_output_____

fig,ax = plt.subplots(figsize=(7,7))
ax.plot(pleiades_T_new, pleiades_L_new, 'go')_____no_output_____

Colors notwithstanding, this plot is still very ugly and should not look much like an H-R diagram to you. For one thing, H-R diagrams usually have log(Luminosity) on the y-axis, which you do as follows:_____no_output_____

fig,ax = plt.subplots(figsize=(7,7))
ax.plot(pleiades_T_new, pleiades_L_new, 'go')
ax.set_yscale('log')_____no_output_____

OK that's much nicer, but should still look backwards to you, because we always draw H-R Diagrams with the Temperature axis running from high to low temperature. This is a pretty easy fix too._____no_output_____

fig,ax = plt.subplots(figsize=(7,7))
ax.plot(pleiades_T_new, pleiades_L_new, 'go')
ax.set_yscale('log')
ax.set_xlim(11000,1000)_____no_output_____

OK! This is starting to look like a good H-R diagram, but without a plot title or axis labels, it's still not very good, so let's add those. _____no_output_____

fig,ax = plt.subplots(figsize=(7,7))
ax.plot(pleiades_T_new, pleiades_L_new, 'go')
ax.set_yscale('log')
ax.set_xlim(11000,1000)
plt.title('H-R Diagram for the Pleiades')
plt.xlabel('Temperature (in K)')
plt.ylabel('log(Luminosity (in L$_{\odot}$))')_____no_output_____

# Exercise 1

Now it's time for you to do some exploring with the rest of the data. For each of the remaining four data files in the Lab 1 directory:

1) read in the data

2) find the columns for luminosity and temperature

(note that some of the raw data has units of log(L) or log(T), which you'll have to "undo" to get the regular units of L and T. Refer to the Homework0 Exercise #1 if you can't remember how)

3) assign the relevant columns to pandas series and convert data types to numeric as necessary

4) make a plot of the data with appropriate axis labels, ranges, a plot title, etc.    _____no_output_____

## Add code here to read in the other three files in the Lab 1 directory, and give them descriptive variable names._____no_output_____## Add code here to identify the column labels for luminosity and temperature. Be careful - columns may not have
## the same names, and be sure to check the units of the quantities._____no_output_____## Convert to pandas series and data types, if necessary._____no_output_____## Plot the data in this cell and the following cells for each sample._____no_output_____

# Exercise 2 (Multi-Panel Plots)

The individual plots for each of these clusters are nice, but really we'd like to be able to compare them side-by-side. You can do this by making either a multi-panel plot, or an overlapping plot. A skeleton outline of how to do each is below. Please use this as a framework to make similar plots with our data._____no_output_____

#some fake data
data_x = np.arange(0,100)
data_y = 3*data_x
data_y2 = data_x**2
data_y3 = data_x + 20
data_y4 = np.sqrt(data_x)_____no_output_____# multipanel plot example
fig,((ax1,ax2),(ax3,ax4)) = plt.subplots(2, 2, figsize=(10,10))
fig.suptitle('This is a title for my multipanel plot')
ax1.plot(data_x, data_y, 'go')
ax1.set_title('Figure 1 Title')
ax1.set_xlabel('x label')
ax1.set_ylabel('y label')
ax2.plot(data_x, data_y2, 'bo')
ax2.set_title('Figure 2 Title')
ax2.set_xlabel('x label')
ax2.set_ylabel('y label')
ax3.plot(data_x, data_y3, 'ro')
ax3.set_title('Figure 3 Title')
ax3.set_xlabel('x label')
ax3.set_ylabel('y label')
ax4.plot(data_x, data_y4, 'mo')
ax4.set_title('Figure 4 Title')
ax4.set_xlabel('x label')
ax4.set_ylabel('y label')_____no_output_____#overlay plot example
fig,ax = plt.subplots(figsize=(10,10))
plt.title('This is a title for my multipanel plot')
ax.plot(data_x, data_y, 'go', label='legend entry 1', alpha=0.5)
ax.plot(data_x, data_y2, 'bo', label='legend entry 2', alpha=0.5)
ax.plot(data_x, data_y3, 'ro', label='legend entry 2', alpha=0.5)
ax.plot(data_x, data_y4, 'mo', label='legend entry 2', alpha=0.5)
ax.set_title('Figure Title')
ax.set_xlabel('x label')
ax.set_ylabel('y label')
plt.legend(numpoints=1)
#TRY EXECUTING WITH AND WITHOUT THE FOLLOWING LINE. HERE AND IN THE DATA YOU'LL BE PLOTTING,
#A SUBJECTIVE DECISION MUST BE MADE ABOUT AXIS RANGES
ax.set_ylim(0,200)_____no_output_____## In this cell, create your own overlay plot showing the different populations. Hint: You may want to plot the
## sample with the most data points first._____no_output_____## In this cell, create a multi-panel plot for the different populations._____no_output_____

# Exercise 3 (Comprehension Questions)

1) Describe the differences between the H-R diagrams. Do this both qualitatively (note differences in their appearance) and quantitatively. You might find methods like .min and .max to be useful here.

2) Which of the four groups of stars has stars that are NOT on the main sequence, and which don't? Why, do you think?

3) Why do you think the low temperature (M-dwarf) end of the main sequence cuts off at such a different place in the four samples? Do you think this is a physical effect or an instrumental one and why?

4) Why do you think there are no white dwarfs in any of the samples besides the field sample?_____no_output_____

## Your answers to each of the four questions here._____no_output_____


# Notebook from wconnell/metrx
Path: notebook/2020.01.26_converging-model.ipynb

import numpy as np
import pandas as pd
np.random.seed(123)

# viz
import matplotlib.pyplot as plt

# notebook settings
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

pd.set_option('display.max_columns', 1000)_____no_output_____

## Sample Prep_____no_output_____

samples = pd.read_csv('../data/TCGA/rna-seq_pan/meta/gdc_sample_sheet.2019-12-12.tsv', sep="\t")
# get file type
samples['data'] = [val[1] for i,val in samples['File Name'].str.split(".").items()]
samples.head()_____no_output_____

Samples with RNAseq adjacent normal tissue_____no_output_____

samples[samples['Sample Type']=='Solid Tissue Normal']['data'].value_counts()_____no_output_____samples['project'] = [val[1] for i,val in samples['Project ID'].str.split("-").items()]
samples['project'].value_counts()_____no_output_____# all cases with adjacent normal tissue
cases = samples[samples['Sample Type']=='Solid Tissue Normal']['Case ID']_____no_output_____# disparity in cases
samples[(samples['Case ID'].isin(cases)) & (samples['Sample Type']=='Primary Tumor')
        & (samples['data']=='FPKM') & (samples['project']=='BRCA')]['Case ID'].nunique()
samples[(samples['Case ID'].isin(cases)) & (samples['Sample Type']=='Solid Tissue Normal')
        & (samples['data']=='FPKM') & (samples['project']=='BRCA')]['Case ID'].nunique()_____no_output_____# divide, join, subset
case_tumor = samples[(samples['Case ID'].isin(cases)) & (samples['Sample Type']=='Primary Tumor') &
                     (samples['data']=='FPKM') & (samples['project']=='BRCA')]
case_norm = samples[(samples['Case ID'].isin(cases)) & (samples['Sample Type']=='Solid Tissue Normal') &
                    (samples['data']=='FPKM') & (samples['project']=='BRCA')]
cases = case_norm[case_norm['Case ID'].isin(case_tumor['Case ID'])]['Case ID']
cases.shape_____no_output_____case_tumor = case_tumor[case_tumor['Case ID'].isin(cases)]
case_norm = case_norm[case_norm['Case ID'].isin(cases)]
cases = pd.concat([case_tumor, case_norm])_____no_output_____case_tumor.shape
case_norm.shape
cases.shape_____no_output_____

## Dataset Prep_____no_output_____

from sklearn.model_selection import train_test_split

target = 'Sample Type'
cases[target] = cases[target].astype('category')

train, test = train_test_split(cases)
train[target].value_counts()
test[target].value_counts()_____no_output_____train[target].cat.categories_____no_output_____import torch
from torch.optim import lr_scheduler
import torch.optim as optim
from torch.autograd import Variable
torch.manual_seed(123)

from trainer import fit
import visualization as vis
import numpy as np
cuda = torch.cuda.is_available()
print("Cuda is available: {}".format(cuda))

classes = {key:val for val,key in enumerate(train[target].cat.categories.values)}
classes_____no_output_____from tcga_datasets import TCGA, SiameseTCGA
root_dir = "../data/TCGA/rna-seq_pan/"
batch_size = 1

train_dataset = TCGA(root_dir, samples=train, train=True, target=target)
test_dataset = TCGA(root_dir, samples=test, train=False, target=target)

kwargs = {'num_workers': 10, 'pin_memory': True} if cuda else {'num_workers': 10}
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True, **kwargs)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=False, **kwargs)_____no_output_____

## Siamese Network_____no_output_____

# Step 1 set up dataloader
root_dir = "../data/TCGA"
siamese_train_dataset = SiameseTCGA(train_dataset) # Returns pairs of images and target same/different
siamese_test_dataset = SiameseTCGA(test_dataset)
batch_size = 8
kwargs = {'num_workers': 10, 'pin_memory': True} if cuda else {}
siamese_train_loader = torch.utils.data.DataLoader(siamese_train_dataset, batch_size=batch_size, shuffle=True, **kwargs)
siamese_test_loader = torch.utils.data.DataLoader(siamese_test_dataset, batch_size=batch_size, shuffle=False, **kwargs)

# Set up the network and training parameters
from tcga_networks import EmbeddingNet, SiameseNet
from losses import ContrastiveLoss
from metrics import AccumulatedAccuracyMetric

# Step 2
embedding_net = EmbeddingNet()
# Step 3
model = SiameseNet(embedding_net)
if cuda:
    model.cuda()

# Step 4
margin = 1.
loss_fn = ContrastiveLoss(margin)
lr = 1e-3
optimizer = optim.Adam(model.parameters(), lr=lr)
scheduler = lr_scheduler.StepLR(optimizer, 8, gamma=0.1, last_epoch=-1)
n_epochs = 10
# print training metrics every log_interval * batch_size
log_interval = 30_____no_output_____train_loss, val_loss = fit(siamese_train_loader, siamese_test_loader, model, loss_fn, optimizer, scheduler,
    n_epochs, cuda, log_interval)Train: [0/170 (0%)]	Loss: 0.268963
Epoch: 1/10. Train set: Average loss: 109.3384
Epoch: 1/10. Validation set: Average loss: 19.4182
Train: [0/170 (0%)]	Loss: 309.796631
Epoch: 2/10. Train set: Average loss: 128.6826
Epoch: 2/10. Validation set: Average loss: 146.4794
Train: [0/170 (0%)]	Loss: 120.472649
Epoch: 3/10. Train set: Average loss: 220.2393
Epoch: 3/10. Validation set: Average loss: 7.1959
Train: [0/170 (0%)]	Loss: 1.989862
Epoch: 4/10. Train set: Average loss: 13.7897
Epoch: 4/10. Validation set: Average loss: 0.8082
Train: [0/170 (0%)]	Loss: 3.589590
Epoch: 5/10. Train set: Average loss: 3.9411
Epoch: 5/10. Validation set: Average loss: 5.1269
Train: [0/170 (0%)]	Loss: 0.450236
Epoch: 6/10. Train set: Average loss: 8.6736
Epoch: 6/10. Validation set: Average loss: 0.8676
Train: [0/170 (0%)]	Loss: 0.245182
Epoch: 7/10. Train set: Average loss: 1.4281
Epoch: 7/10. Validation set: Average loss: 0.2053
Train: [0/170 (0%)]	Loss: 0.009832
Epoch: 8/10. Train set: Average loss: 0.6234
Epoch: 8/10. Validation set: Average loss: 0.1697
Train: [0/170 (0%)]	Loss: 0.110812
Epoch: 9/10. Train set: Average loss: 0.1303
Epoch: 9/10. Validation set: Average loss: 0.1713
Train: [0/170 (0%)]	Loss: 0.057411
Epoch: 10/10. Train set: Average loss: 0.1504
Epoch: 10/10. Validation set: Average loss: 0.1757
plt.plot(range(0, n_epochs), train_loss, 'rx-')
plt.plot(range(0, n_epochs), val_loss, 'bx-')_____no_output_____train_embeddings_cl, train_labels_cl = vis.extract_embeddings(train_loader, model)
vis.plot_embeddings(train_embeddings_cl, train_labels_cl, siamese_train_dataset.labels_dict)_____no_output_____val_embeddings_baseline, val_labels_baseline = vis.extract_embeddings(test_loader, model)
vis.plot_embeddings(val_embeddings_baseline, val_labels_baseline, siamese_test_dataset.labels_dict)_____no_output_____

## Integrated Gradients
Test completeness axiom through comparison of different baselines_____no_output_____

import copy
from captum.attr import LayerActivation
from captum.attr import IntegratedGradients_____no_output_____tmp_model = copy.deepcopy(model)_____no_output_____tmp_model_____no_output_____def attribution_pairs(SiameseTCGA, exp, ctrl):
    # subset different samples
    negative_pairs = np.array(SiameseTCGA.test_pairs)
    negative_pairs = negative_pairs[negative_pairs[:,2] == 0]
    # map labels to integers
    ctrl = siamese_test_dataset.labels_dict[ctrl]
    exp = siamese_test_dataset.labels_dict[exp]
    # ordered indices of samples
    ctrl_data = [idx for pair in negative_pairs[:, :2] for idx in pair if np.isin(idx, SiameseTCGA.label_to_indices[ctrl])]
    exp_data = [idx for pair in negative_pairs[:, :2] for idx in pair if np.isin(idx, SiameseTCGA.label_to_indices[exp])]
    # data
    ctrl_data = Variable(SiameseTCGA.test_data[ctrl_data], requires_grad=True)
    exp_data = Variable(SiameseTCGA.test_data[exp_data], requires_grad=True)
    return ctrl_data, exp_data_____no_output_____ctrl_data, exp_data = attribution_pairs(siamese_test_dataset, exp='Primary Tumor', ctrl='Solid Tissue Normal')_____no_output_____ig = IntegratedGradients(tmp_model.get_embedding)_____no_output_____attr, delta = ig.attribute(exp_data.cuda(), ctrl_data.cuda(), target=0, n_steps=50, return_convergence_delta=True)
attr = attr.cpu().detach().numpy()_____no_output_____attr.shape
feat_imp = pd.DataFrame(data=attr.mean(axis=0), index=train_dataset.data.columns, columns=['Attribution'])
feat_imp.hist(bins=100)
feat_imp.describe()
feat_imp.nlargest(10, columns='Attribution')_____no_output_____

Inspect individual sample attributions_____no_output_____

attr_samp = pd.Series(attr[0], index=train_dataset.data.columns)
attr_samp.nlargest(10)
attr_samp.sum()_____no_output_____for i in range(30):
    attr_samp = pd.Series(attr[i], index=train_dataset.data.columns)
    attr_samp.nlargest(10)_____no_output_____

#### Check embedding of zero-feature vector_____no_output_____

base_zero = torch.zeros(siamese_test_dataset.test_data.shape[1]).cuda()
base_emb = model.get_embedding(base_zero)
base_emb_____no_output_____model.get_l_____no_output_____


# Notebook from akhadangi/EM-stellar
Path: EM_stellar.ipynb

_____no_output_____# **EM-stellar**

This notebook represents the implementation of the [**EM-stellar**](https://www.biorxiv.org/content/10.1101/2020.07.15.203836v1). Deep learning (DL) methods offer an exciting opportunity to automate the segmentation process by learning from manual annotations of a small sample of EM data. While many DL methods are being rapidly adopted to segment EM data no benchmark analysis has been conducted on these methods to date. We present EM-stellar as a platform that can be used to benchmark the performance of a range of state-of-the-art DL methods on user-provided datasets.

---
Papers related to this Notebook:

- **EM-stellar: benchmarking deep learning for electron microscopy image segmentation** by *Afshin Khadangi, Thomas Boudier, Vijay Rajagopal*  (https://www.biorxiv.org/content/10.1101/2020.07.15.203836v1)

- **EM-net: Deep learning for electron microscopy image segmentation** by *Afshin Khadangi, Thomas Boudier, Vijay Rajagopal*  (https://www.biorxiv.org/content/10.1101/2020.02.03.933127v1)

**Please cite** [**this original paper**](https://www.biorxiv.org/content/10.1101/2020.07.15.203836v1) **when using or developing this notebook.**

---_____no_output_____###**Structure of a notebook**

 This cell is adapted from the [ZeroCostDL4Mic](https://github.com/HenriquesLab/ZeroCostDL4Mic).

---
The notebook contains two types of cell:

**Text cells** provide information and can be modified by douple-clicking the cell. You are currently reading the text cell. You can create a new text by clicking `+ Text`.

**Code cells** contain code and the code can be modfied by selecting the cell. To execute the cell, move your cursor on the `[ ]`-mark on the left side of the cell (play button appears). Click to execute the cell. After execution is done the animation of play button stops. You can create a new coding cell by clicking `+ Code`.

---
###**Table of contents, Code snippets** and **Files**

On the top left side of the notebook you find three tabs which contain from top to bottom:

*Table of contents* = contains structure of the notebook. Click the content to move quickly between sections.

*Code snippets* = contain examples how to code certain tasks. You can ignore this when using this notebook.

*Files* = contain all available files. After mounting your google drive (see section 1.) you will find your files and folders here.

**Remember that all uploaded files are purged after changing the runtime.** All files saved in Google Drive will remain. You do not need to use the Mount Drive-button; your Google Drive is connected in section 1.2.

**Note:** The "sample data" in "Files" contains default files. Do not upload anything in here!

---
###**Making changes to the notebook**

**You can make a copy** of the notebook and save it to your Google Drive. To do this click file -> save a copy in drive.

To **edit a cell**, double click on the text. This will show you either the source code (in code cells) or the source text (in text cells).
You can use the `#`-mark in code cells to comment out parts of the code. This allows you to keep the original code piece in the cell as a comment._____no_output_____# **0. Before getting started**
---

 **Important:** Make sure that all of the images are equal in size, both in $x$ and $y$. For example, suppose that you have segmented your test data with [**EM-net**](https://github.com/akhadangi/EM-net) by trying a variety of the networks, and you aim at benchmarking these methods by retrieving a wide range of evaluation metrics. Let's say that you have tried segmenting your test images on U-net, VGG and ResNet with the following image sizes:
*   $256$ x $512$

You have also previously tried EM-net v1 and v2 2x with the following image sizes:
*   $512$ x $512$

Supposing these different patches have overlapping areas, then you will need to extract those areas before using EM-stellar. Let's suppose that these two different datasets share an area of $128$ x $128$. Hence you will need to extract these overlapping patches before using this platform.

 **Important:** Make sure that you have saved your patches in the ***.png** format.
0. Create a folder called "Main".
1. Create two sub folders `test_segmentation_results` and `ground_truth_segmentation_masks` within the "Main" folder in the following format and upload them into your google drive:

some *This is Blue italic.* text

*   **drive**
    - **Main**
      - ***test_segmentation_results***
        - U-net
          - *result_1.png*, *result_2.png*, ...
        - VGG
          - *result_1.png*, *result_2.png*, ...
        - EM-net v1 BN
          - *result_1.png*, *result_2.png*, ...
        - ResNet
          - *result_1.png*, *result_2.png*, ...
        - ...
      - ***ground_truth_segmentation_masks***
        - *mask_1.png*, *mask_2.png*, ...

2. Place the segmentation result images (e.g. predicted by the networks) in the corresponding folder (for example folder `u-net`) and put all these folders to **test_segmentation_results** folder.
3. Place all your ground-truth images in the **ground_truth_segmentation_masks** folder. This folder contains the binary masks that we want to compare the predictions of the methods with. Hence, it is different from the validation data as the validation data is sampled from the training data that you have already performed during the training.
_____no_output_____## **1.1 Change the Runtime**
---

Go to **Runtime -> Change the Runtime type**

**Runtime type: Python 3** *(Python 3 is programming language in which this program is written)*

**Accelator:** *This notebook is accelerator free! Select the `None` as type of the accelerator.*
_____no_output_____## **1.2. Mount your Google Drive**
---
 To use this notebook on the data present in your Google Drive, you need to mount your Google Drive to this notebook.

 Play the cell below to mount your Google Drive and follow the link. In the new browser window, select your drive and select 'Allow', copy the code, paste into the cell and press enter. This will give Colab access to the data on the drive.

 Once this is done, your data are available in the **Files** tab on the top left of notebook._____no_output_____

#@markdown ##Play the cell to connect your Google Drive to Colab

#@markdown * Click on the URL.

#@markdown * Sign in your Google Account.

#@markdown * Copy the authorization code.

#@markdown * Enter the authorization code.

#@markdown * Click on "Files" site on the right. Refresh the site. Your Google Drive folder should now be available here as "drive".

# mount user's Google Drive to Google Colab.
from google.colab import drive
drive.mount('/content/gdrive')Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

Enter your authorization code:
··········
Mounted at /content/gdrive

# **2. Install EM-stellar Dependencies**
---_____no_output_____

%%capture
#@markdown ##Play to install EM-stellar standard dependencies
# import logging, os

# logging.disable(logging.WARNING)

# importing os to set environment variable
import os
!pip install tqdm

# install openjdk for java and set ENV variable
!apt-get install -y openjdk-8-jdk-headless -qq > /dev/null
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"     #set environment variable
print("1/7: Installed Java openjdk 8.")
# install maven
!sudo apt-get install maven
print("2/7: Installed maven.")

# install imglyb
!pip install imglyb==0.3.5
print("3/7: Installed imglyb.")

# install pyimagej
!pip install pyimagej
print("4/7: Installed pyimagej (for FiJi use).")

# install pyjnius (python library to access java classes)
!wget https://anaconda.org/conda-forge/pyjnius/1.2.0/download/linux-64/pyjnius-1.2.0-py36hc2cfd8d_0.tar.bz2 && tar -xjf pyjnius-1.2.0-py36hc2cfd8d_0.tar.bz2 -C /usr/local
print("5/7: Installed pyjnius (python wrapper for Java).")

print("Downloading latest version of FiJi ...")
_____no_output_____%%capture
%%bash
#@markdown ##Play to install FiJi (ImageJ)

wget https://downloads.imagej.net/fiji/latest/fiji-linux64.zip && unzip fiji-linux64.zip

cd Fiji.app/
./ImageJ-linux64 --update add-update-site clij2 https://sites.imagej.net/clij2/
./ImageJ-linux64 --update add-update-site clij https://sites.imagej.net/clij/
./ImageJ-linux64 --update update_____no_output_____

# **3. Select your paths**

---_____no_output_____ **Paths for segmentation results and ground-truth binary masks**

**`segmentation_results`, `ground_truth_binary_masks`:** These are the folders containing your segmentation results (e.g. predictions from networks) and ground-truth binary masks files (manually annotated masks). Enter the path to the source and target images for training. **These should be located in the same parent folder.**

**Make sure the directories exist before entering them!**

 **Select benchmarking metrics**

**metrics**: choose the metrics that you want to use for benchmarking different networks that you have used for segmenting test data.

---
**Caution!** If you run this notebook **for the first time**, you will need to run the following cell twice as some dependency issues of ImageJ with the Colab. Run the following cell and if you encounter an error, please Go to **Runtime -> Restart runtime**. Once it has been restarted, please run the following cell again until you see the output `FiJi has been loaded successfully!`.
You won't need to re-run the previous cells after restarting the runtime as they have been already cached on the Colab.
_____no_output_____

#@markdown ###Path to training images:
# import imagej
import os
os.environ["PYJNIUS_JAR"] = "/usr/local/share/pyjnius/pyjnius.jar"
import imagej;
ij = imagej.init("Fiji.app");
print('FiJi has been loaded successfully!')
from google.colab import drive

drive.mount('/content/gdrive', force_remount=True)
import os
from pathlib import Path
import numpy as np
import pandas as pd
from sklearn.metrics import confusion_matrix, jaccard_score
from skimage.io import imread_collection, imsave
import glob
from PIL import Image
import seaborn as sns

import shutil
segmentation_results = '/content/gdrive/My Drive/Images/Networks outputs' #@param {type:"string"}

#We find the name of the base_directory
path = Path(segmentation_results)
base_directory = '/'+(str(path.parent))

ground_truth_binary_masks = '/content/gdrive/My Drive/Images/Groundtruth' #@param {type:"string"}
#We assign a temporary directory to just include the stacked images
Temp_dir = base_directory + '/Temp'

#Cleaning up pre-existing directories, if any!
if os.path.exists(Temp_dir):
  shutil.rmtree(Temp_dir)

#Creating new directories
os.mkdir(Temp_dir)

#@markdown ###Select the metrics that you want to evaluate inference results with:

V_Rand = True #@param {type:"boolean"}
V_Info = True #@param {type:"boolean"}
Standard_metrics = True #@param {type:"boolean"}FiJi has been loaded successfully!
Mounted at /content/gdrive

# **4. Visualise the metrics**

---_____no_output_____ **Paths for training data and models**

**`Training_source`, `Training_target`:** These are the folders containing your source (e.g. EM images) and target files (segmentation masks). Enter the path to the source and target images for training. **These should be located in the same parent folder.**

**`Patch_width`, `Patch_height`:** Training deep neural networks requires lots of computaional resources including **memory**. Feeding larger image data size to the network can exhaust the resources. Such a limitation can be handled by reducing the input image size. Hence, we have provided **patch training** which enables the user to overcome this limitation. You can simply upload the training images with any size, and set the patch size by controlling these two parameters. **The larger patch size will require higher memory.**

**`model_name`:** Use only my_model -style, not my-model. If you want to use a previously trained model, enter the name of the pretrained model (which should be contained in the trained_model -folder after training).

**`model_path`**: Enter the path of the folder where you want to save your model.

**Make sure the directories exist before entering them!**

 **Select training parameters**

**`epochs`**: Choose more epochs for larger training sets. Observing how much the loss reduces between epochs during training may help determine the optimal value. **Default: 50**

**Advanced Parameters - experienced users only**

**`batch_size`**: This parameter describes the amount of images that are loaded into the network per step. Smaller batchsizes may improve training performance slightly but may increase training time. If the notebook crashes while loading the dataset this can be due to a too large batch size. Decrease the number in this case. **Default: 2**

**`loss_functions`**: Training performance depends strongly on the loss function. To find out more about losses, see: https://keras.io/losses/. **Default: binary_crossentropy**

**`optimizer`**: Training performance depends strongly on the optimiser as well. To find out more about optimizers, see: https://keras.io/optimizers/. **Default: Rectified Adam**

_____no_output_____

#@markdown ##4.1. Play to retrieve the evaluation metrics values
import logging, os
from tqdm.autonotebook import tqdm

for root, dirs, files in os.walk(segmentation_results, topdown=True):
  if (len(dirs)>0):
    childdirs = dirs
parentdir = [ground_truth_binary_masks, segmentation_results]

childdirs_imgs = []
print("iterating the segmentation results directories ...")
for item in tqdm(childdirs):
    tmp = np.asarray(list(imread_collection(parentdir[1]+"/"+item+"/*.png")),dtype='float32')
    if (np.max(tmp)>1):tmp=tmp/255.0
    childdirs_imgs.append(tmp)
    tmp = np.vstack((tmp,tmp))
    imsave(Temp_dir+'/'+item+'.tif',tmp)
Ground_data = np.asarray(list(imread_collection(parentdir[0]+"/*.png")),dtype='float32')
Ground_dataTmp = np.vstack((Ground_data,Ground_data))
zipdata = zip(childdirs, childdirs_imgs)
mydict = dict(zipdata)
if (np.max(Ground_data)>1):Ground_data=Ground_data/255.0
if (np.max(Ground_dataTmp)>1):Ground_dataTmp=Ground_dataTmp/255.0
imsave(Temp_dir+'/Ground.tif',Ground_dataTmp)

Language_extension = "BeanShell"

macroVRand = """
import trainableSegmentation.metrics.*;
#@output String VRand
metric = new RandError( originalLabels, proposedLabels );
maxThres = 1.0;
maxScore = metric.getMaximalVRandAfterThinning( 0.0, maxThres, 0.1, true );
VRand = maxScore;
"""

macroVInfo = """
import trainableSegmentation.metrics.*;
#@output String VInfo
metric = new VariationOfInformation( originalLabels, proposedLabels );
maxThres =1.0;
maxScore = metric.getMaximalVInfoAfterThinning( 0.0, maxThres, 0.1 );
VInfo = maxScore;
"""
current_path = Temp_dir
VRandDf = []
VInfoDf = []
if (V_Rand or V_Info):
  print("Retrieving Rand/Info metrics for the networks ...")
for item in tqdm(childdirs):
  if (V_Rand):
    macroVRandTmp = ('\nimport ij.IJ;\noriginalLabels=IJ.openImage("' + current_path + '/' + 'Ground' + '.tif");' +
                  '\nproposedLabels=IJ.openImage("' + current_path + '/' + item + '.tif");' + macroVRand)
    VRand = float(ij.py.run_script(Language_extension, macroVRandTmp).getOutput('VRand'))
    VRandDf.append(VRand)
  if (V_Info):
    macroVInfoTmp = ('\nimport ij.IJ;\noriginalLabels=IJ.openImage("' + current_path + '/' + 'Ground' + '.tif");' +
                  '\nproposedLabels=IJ.openImage("' + current_path + '/' + item + '.tif");' + macroVInfo)
    VInfo = float(ij.py.run_script(Language_extension, macroVInfoTmp).getOutput('VInfo'))
    VInfoDf.append(VInfo)

if (Standard_metrics):
  print("Retrieving standard metrics for the segmentation results ...")
  Ground = Ground_data.flatten()
  Ground = (Ground > 0.5).astype(np.int_)
  F1score = []
  accuracy = []
  sensitivity = []
  specificity = []
  PPV = []
  NPV = []
  Jaccard = []
  for item in tqdm(mydict):
      itemTmp = mydict[item]
      itemTmp = (itemTmp > 0.5).astype(np.int_)
      itemTmp = itemTmp.flatten()
      tn, fp, fn, tp = confusion_matrix(Ground, itemTmp).ravel()
      accuracy.append((tp+tn)/(tp+tn+fp+fn))
      sn = tp/(tp+fn)
      sensitivity.append(sn)
      specificity.append(tn/(tn+fp))
      pr = tp/(tp+fp)
      PPV.append(pr)
      NPV.append(tn/(tn+fn))
      Jaccard.append(jaccard_score(Ground, itemTmp))
      F1score.append(2*(pr*sn)/(pr+sn))    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:3: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)
  This is separate from the ipykernel package so we can avoid doing imports until

## **4.2. Visualisation**
---_____no_output_____

#@markdown ###4.2.1 Play to visualise the tabular report
import logging, os
if (V_Rand and V_Info and Standard_metrics):
  MyDataFrameDict = {'Network': childdirs,
          'F1-score': F1score,
          'VRand': VRandDf,
          'VInfo': VInfoDf,
          'accuracy': accuracy,
          'sensitivity': sensitivity,
          'specificity': specificity,
          'PPV': PPV,
          'NPV': NPV,
          'Jaccard': Jaccard
          }
elif (V_Rand and V_Info):
  MyDataFrameDict = {'Network': childdirs,
          'VRand': VRandDf,
          'VInfo': VInfoDf
          }
elif (V_Rand and Standard_metrics):
  MyDataFrameDict = {'Network': childdirs,
          'F1-score': F1score,
          'VRand': VRandDf,
          'accuracy': accuracy,
          'sensitivity': sensitivity,
          'specificity': specificity,
          'PPV': PPV,
          'NPV': NPV,
          'Jaccard': Jaccard
          }
elif (V_Info and Standard_metrics):
  MyDataFrameDict = {'Network': childdirs,
          'F1-score': F1score,
          'VInfo': VInfoDf,
          'accuracy': accuracy,
          'sensitivity': sensitivity,
          'specificity': specificity,
          'PPV': PPV,
          'NPV': NPV,
          'Jaccard': Jaccard
          }
elif (Standard_metrics):
  MyDataFrameDict = {'Network': childdirs,
          'F1-score': F1score,
          'accuracy': accuracy,
          'sensitivity': sensitivity,
          'specificity': specificity,
          'PPV': PPV,
          'NPV': NPV,
          'Jaccard': Jaccard
          }
elif (V_Info):
  MyDataFrameDict = {'Network': childdirs,
          'VInfo': VInfoDf
          }
elif (V_Rand):
  MyDataFrameDict = {'Network': childdirs,
          'VRand': VRandDf
          }

df = pd.DataFrame(MyDataFrameDict, index=childdirs, columns = ['F1-score', 'VRand', 'VInfo', 'accuracy', 'sensitivity',
                                                  'specificity', 'PPV', 'NPV', 'Jaccard'])
df = df.round(decimals=5)

cm = sns.diverging_palette(20, 130, n = len(childdirs), l=70, as_cmap=True)

styles = [
    dict(props=[("font-family", "Times New Roman"),
                               ("text-align", "center")])
]
pd.set_option('precision', 5)
df.sort_index(axis=0, inplace = True)
html = (df.style.set_table_styles(styles).background_gradient(cmap=cm))
html_____no_output_____#@markdown ###4.2.2 Play to visualise the heatmap report
import matplotlib.pyplot as plt
plt.figure(figsize=(6, 4), dpi=300)
sns.set(font_scale=0.8, font="sans-serif")
csfont = {'fontname':'sans-serif'}
heat_map = sns.heatmap(df, cmap="Spectral")
plt.suptitle("Heatmap of the segmentation methods relative to different metrics", fontsize=16, y=1)
plt.xlabel("Metrics", fontsize=14, **csfont)
plt.ylabel("Methods", fontsize=14, **csfont)
plt.show()_____no_output_____


# Notebook from hposborn/MonoTools
Path: MonoTools/examples/Search_Example.ipynb

# Using MonoTools search

Here we will go through a few examples of how to fit a mono (one transit) or duo (two transit) transiting planet system.

This will include:
* Accessing lightcurves
* Accessing stellar parameters
* Searching for planets
* Vetting planet candidates using model-comparisons

For info on how to _fit_ for planetary systems with MonoTools see "Fit_Example.ipynb"_____no_output_____

import numpy as np
import pandas as plt
import matplotlib.pyplot as plt
%matplotlib inline

%load_ext autoreload
%autoreload 2

from MonoTools import search, tools, starpars
#Depending on where you run this notebook, you may need MonoTools.MonoTools or MonoToolsThe autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
Created TAP+ (v1.2.1) - Connection:
	Host: gea.esac.esa.int
	Use HTTPS: True
	Port: 443
	SSL Port: 443
Created TAP+ (v1.2.1) - Connection:
	Host: geadata.esac.esa.int
	Use HTTPS: True
	Port: 443
	SSL Port: 443

## 1) Lightcurves

Lightcurves are a dictionary with a few important arrays - `time`, `flux`, `flux_err` and `flux_unit` (in the case of ppt, this would be 0.001)
The following would also be useful and are used by some functions:
* `cadence` - in the form "[mission letter][cadence in nearest minute]", e.g. T2 (for 2-minute TESS data), or K30 (for 30-min Kepler or K2 data). This should ideally be in an array form with matching length to `time`
* `mask` - in the form of a boolean, where those True points are the cadences to be used.
* `bg_flux` - background flux
* `cent_1` and `cent_2` - centroids in x and y
* `jd_base` - zero-point of time in BJD

However, you don't need to worry about this, because you can load lightcurves from using `tools.openLightCurve(ID,mission)`. For TESS this will load in SPOC 2-minute cadence lightcurves and, if those aren't available, search for either SPOC-HLSP, QLP or Eleanor FFI lightcurves (in decreasing order of preference).

One can also use mission-specific lightcurve searches:

for TESS lightcurves only:
`tlc=tools.TESS_lc(7020254)`

for K2 lightcurves only:
`k2lc=tools.K2_lc(211442297)`

for Kepler lightcurves only:
`keplc=tools.getKeplerLC(8462852)
_____no_output_____

#Let's load the data for the case of a multi-transiting & a mono-transiting planet
#We are only loading primary mission data here (specified by the sectors), as the extended mission data "solves" the period - which we can use as a check later on!

lc,hdr = tools.openLightCurve(349488688,'tess',sectors=np.arange(1,26))_____no_output_____print(tools.MonoData_savepath)/Volumes/LUVOIR/MonoToolsData
plt.plot(lc['time'][lc['mask']],lc['flux'][lc['mask']],',')

#Binning the lightcurve to 30mins
lc=tools.lcBin(lc,binsize=30/1440)

plt.plot(lc['bin_time'],lc['bin_flux'],'.')_____no_output_____

[home](#contents)

## 2) Stellar Parameters

In `MonoTools.starpars` are multiple methods for estimating stellar parameters without needing to manually look them up.

The simplest is to run `starpars.getStellarInfoFromCsv`.
This script searches the input catalogues that have been updated with Gaia. There is native support to download/search:
* TICv8 (Stassun et al) using Astroquery
* Kevin Hardegree-Ullman's table for K2 (https://ui.adsabs.harvard.edu/abs/2020ApJS..247...28H/abstract)
* Travis Berger's work for Kepler (Gaia-Kepler Stellar Properties Catalog, https://arxiv.org/abs/2001.07737)

In cases where the star is present in multiple source (e.g. a K2 object in the TIC), we check all possible source and take the value with the best-constrained stellar density.
Similarly, if density is provided from multiple sources within the same table (e.g. a quoted $\rho_s$ value, plus $R_s$ & $M_s$, plus $R_s$ & $\log{\rm g}$), then we compute a weighted average to produce a constrained $\rho_s$.

In cases where the star is NOT present in those catalogues, `starpars.getStellarInfoFromCsv` will then try to scrape the initial input catalogue (e.g. Huber's EPIC and the KIC, both from the exoplanet archive).

Finally, in cases where no info can be found from any source, `starpars.getStellarInfoFromCsv` scrapes survey data for the star from Gaia, APAS, 2MASS, SDSS, etc, and calls a modified version of Dan Huber's `IsoClassify` module with that info to produce stellar parameters. This may requre an RA/Dec (in the form of an astropy `SkyCoord` object to be inputted._____no_output_____

#For K2-115:
info_tic,_,_ = starpars.getStellarInfoFromCsv(7020254,'tess')
info_epic,_,_ = starpars.getStellarInfoFromCsv(211442297,'K2')Breaking because status = COMPLETE
211442297  not in TIC
print([info_tic[c] for c in ['rho','rad','mass','logg','source']],
      [info_epic[c] for c in ['rho','rad','mass','logg','source']])[1.6462452320335064, 0.85554, 1.026, 4.58473, 'TICv8'] [1.290200785393436, 0.868, 0.89, 4.403, 'KHU_EPIC']
# and for our target star:

info,_,_ = starpars.getStellarInfoFromCsv(349488688,'tess')Breaking because status = COMPLETE
[info[c] for c in ['rho','rad','mass','logg','source']]_____no_output_____

## 3) Searching the lightcurve for MonoTransits

Let's use `search.MonoTransitSearch` to try to find any potential transits.

There's quite a few different inputs, but the important ones are:
* Stellar parameters `Rs`, `Ms`, `Teff` - these are used to build transit models. For a quick way of getting stellar parameters, see  7) Stellar Parameters

* `mono_SNR_thresh` - the signal to noise limit to call the detections a candidate or not
* `use_binned` - this will speed up the search
* `use_poly` - co-fits a polynomial with the transit model. If this is False, make sure `use_flat` is True_____no_output_____

detns, outparams, fig_loc = search.MonoTransitSearch(lc, 349488688, 'tess', plot=True,
                                                     Rs=info['rad'],Ms=info['mass'],Teff=info['teff'],
                                                     use_stellar_dens=True,
                                                     mono_SNR_thresh=6.5,use_binned=True,use_poly=True)349488688 - Searching 10671 positions with 5 durations: 0.111,0.207,0.292,0.373,0.45
#Detections are dictionaries of vital information for each monotransiting planet candidate:
print(detns.keys())
print(detns['00'])dict_keys(['00', '01', '02'])
{'llk_trans': 87.40515464312568, 'llk_sin': -211.4933289319617, 'llk_poly': -575.6050678399336, 'BIC_trans': -160.5325752863603, 'BIC_sin': 434.0455160389463, 'BIC_poly': 1168.4960244733106, 'sin_DeltaBIC': -594.5780913253066, 'poly_DeltaBIC': -1329.028599759671, 'tcen': 2002.778531310588, 'period': nan, 'period_err': nan, 'DeltaBIC': -1329.028599759671, 'tdur': 0.29166367411259486, 'depth': 1.5276511173703724, 'orbit_flag': 'mono', 'snr': 9.401514971410847, 'P_min': 6.508630183133164}

## 3) Vetting MonoTransit candidates

In `search` there is a function to quickly fit a monotransit, taking only the lightcurve dict, the time of transit and the duration._____no_output_____

#Here
fitted_cand = search.QuickMonoFit(lc,detns['02']['tcen'],detns['02']['tdur'],fit_poly=True,
                                  polyorder=2, ndurs=4, how='mono')
plt.plot(fitted_cand['monofit_x'], fitted_cand['monofit_y'],'.',alpha=0.4)
plt.plot(fitted_cand['monofit_x'], fitted_cand['monofit_ymodel'],'-k')_____no_output_____

We can also just pass the same info to `VetCand`. This will run:
* `QuickMonoFit`
* A check of the SNR (white) and SNR (red - e.g. using the noise at the specific transit duration).
* `VariabilityCheck` - a model comparison between the transit model, a variability model (e.g. sin wave) and a "Step model" (e.g. two polynomials and a step between).
* `AsteroidCheck` - a Gaussian fit to the background flux, which should spike if the dip is due to an asteroid pasing through the BG aperture
* `CentroidCheck` - a fit to the x and y centroids with the transit model and centroid shifts
* `CheckInstrumentalNoise` - a check of the number of transit-like features detected at this epoch (if in TESS)

In each case where the candidate fails a test, the `flag` field in the resulting dictionary is labelled with the type of false positive (e.g. 'asteroid','EB','instrumental').

If the candidate passes all tests, it will have the flag: `planet`_____no_output_____

search.VetCand?_____no_output_____# We have some ideas where planet candidates are - let's not bias our flattening by including them. Hence:
tr_mask=tools.create_transit_mask(lc['time'],np.array([detns[v]['tcen'] for v in detns]),
                                  np.array([detns[v]['tdur'] for v in detns]))
lc=tools.lcFlatten(lc,transit_mask=tr_mask)_____no_output_____#Copying to remove links:
from copy import deepcopy
vetted_detns=deepcopy(detns)

#Looping over each candidate detection and running vetting:
for det in detns:
    vetted_detns[det], _ = search.VetCand(detns[det], det, 349488688, lc, mission='tess',mono_SNR_r_thresh=3,
                                          Rs=info['rad'],Ms=info['mass'],Teff=info['teff'], plot=True, transit_mask=tr_mask)
print({det:vetted_detns[det]['flag'] for det in vetted_detns})Earthshine model not useful in this case - polynomial model is better by 1.1440094657583208e-05
00 - Checks complete.  SNR: 37.35684  SNR_r: nan  variability: -213.105  centroid: -0.39798 | flag: planet
Earthshine model not useful in this case - polynomial model is better by 2.876530764459774e-05
01 - Checks complete.  SNR: 15.36271  SNR_r: nan  variability: -28.2318  centroid: -0.01096 | flag: planet
Earthshine model not useful in this case - polynomial model is better by 2.7751747208178064e-06
02 - Checks complete.  SNR: 10.96927  SNR_r: nan  variability: -26.2565  centroid: -0.04543 | flag: planet
{'00': 'planet', '01': 'planet', '02': 'planet'}
[detns[k]['flag'] for k in detns]_____no_output_____

## 4) Periodic Planet Search

We also need to search the lightcurve for periodic candidates, as either the detected monos may be consistent with a periodic planet, or there may be another low-SNR planet in the system. We can do that with `PeriodicPlanetSearch`.

This uses `transitleastsquares` to search for periodic signals (e.g. planets with $\geq3$ transits). It then masks the detected signal and continues searching until no candidates with SNR greater than the threshold are found.

The inputs here are:
* lightcurve dict
* ID
* Dictionary of planets already detected (may be `{}`)
* `use_binned` - whether to bin data. TLS can be exceedingly slow with unbinned data. `binsize` sets the bin size.
* `use_flat` - unless the lightcurve is very flat already, we should flatted the lightcurve here
* `n_search_loops` - maximum number of times to mask candidates (if `mask_prev_planets`) and re-search the lightcurve
* `Rs`,`Ms`,`Teff` - stellar parameters used in TLS to create transit shapes
* `multi_SNR_thresh` - the SNR threshold for detection
* `plot` [boolean] and `plot_loc` - whether to plot and where to save that plot_____no_output_____

detns, _ = search.PeriodicPlanetSearch(lc, 349488688, detns, use_binned=False, use_flat=True,
                                    binsize=30/1440.0, n_search_loops=5,
                                    Rs=info['rad'],Ms=info['mass'],Teff=info['teff'], multi_SNR_thresh=7.0,
                                    plot=True, plot_loc=None, mask_prev_planets=True)Using TLS on ID=349488688 to search for multi-transiting planets
after flattening: 17203
Transit Least Squares TLS 1.0.24 (21 November 2019)
Creating model cache for 38 durations
Searching 1172 data points, 1466 periods from 1.1 to 8.553 days
Using 1 of 12 CPU threads

We can also apply `search.VetCand` to periodic objects, although `AsteroidCheck` and `InstrumentalNoise` are not performed:_____no_output_____

#performing a candidate vetting for periodic candidates:
for det in detns:
    if detns[det]['orbit_flag']=='periodic':
        #Only doing the vettng for periodic candidates:
        outdet, _ =search.VetCand(detns[det], det, 349488688, lc, mission='tess',
                                  Rs=info['rad'],Ms=info['mass'],Teff=info['teff'], plot=True)
        vetted_detns[det]=outdet_____no_output_____

## 5) Identifying Confused Candidates

I mean two things by this:
* "Monotransits" which are in fact associated with one of the multi-transiting candidates found in the periodic search
* "Monotransits" which have two such candidates, making them "Duotransits" (but not necessarily with a certain period)

For the former, we can run `CheckPeriodConfusedPlanets`. This takes:
* `lc` - dict of light curve
* `all_dets` - dict of candidates
* `mono_mono` = True - whether to compare multis with each other
* `multi_multi` = True - whether to compare monos with each other
* `mono_multi` = True - whether to compare monos with multis
It produces a new dictionary of candidates, where those lower-SNR candidates which are confused with a higher-SNR sibling are reassigned from `flag='planet'` to `flag='confused'`

And for the latter, we use `CheckMonoPairs`. This loops through each monotransit with each other, comparing durations and depths, as well as seeing if such a pair is possible given the observed lightcurve. It takes as arguments:
* `lc_time` - the time array from the lightcurve
* `all_pls` - the dict of planet candidates
* `prox_thresh` = 3.5 - a rough SNR factor above which we consider two monotransits to be alike_____no_output_____

# Checking for confusion between candidate period/epochs:
#final_detns, monos, multis = search.CheckPeriodConfusedPlanets(deepcopy(lc), deepcopy(vetted_detns))

# Here we also have monotransit pairs, so we must also check if they appear similar enough to constitute a duo with CheckMonoPairs
final_detns = search.CheckMonoPairs(lc['time'], deepcopy(vetted_detns))_____no_output_____#The detections that made it through:
{i:{key:final_detns[i][key] for key in ['flag','orbit_flag','period','tdur','tcen','depth']} for i in final_detns if final_detns[i]['flag']=='planet'}_____no_output_____#And those detections that were removed:
{i:{key:final_detns[i][key] for key in ['flag','orbit_flag','period','tdur','tcen','depth']} for i in final_detns if final_detns[i]['flag']!='planet'}_____no_output_____


# Notebook from calvintirrell/dsc-hypothesis_testing-seattle-012720
Path: hypothesis_testing.ipynb

# Hypothesis testing _____no_output_____## Intuition for Hypothesis Testing Example

Greg has recently claimed that his lucky quarter is actually distinctly different than every other kind of quarter. Due to the unique weight distribution from the quarter's design there is actually a greater chance for the quarter to land tails than other fair coins.

Do we believe him?

I sure don't. But lets be good data scientists and put this claim to the test.

Let's flip the coin once and if it comes up tails then I'll change my mind.

Would you change your mind?

How many tails would I have to flip in order to convince you that this coin actual isn't fair? How many to know for sure that it isn't fair?

What is a reasonable threshold to set?_____no_output_____

# imports
from scipy import stats
import statsmodels.stats as sm
import numpy as np
import matplotlib.pyplot as plt_____no_output_____

### High Level Hypothesis Testing
1. Start with a Scientific Question (yes/no)
2. Take the skeptical stance (Null hypothesis)
3. State the complement (Alternative)
4. Create a model of the situation Assuming the Null Hypothesis is True!
5. Decide how surprised you would need to be in order to change your mind_____no_output_____## Definitions_____no_output_____**What is statistical hypothesis testing?**

When we perform experiments, we typically do not have access to all the members of a population, and need to take **samples** of measurements to make inferences about the population.

A statistical hypothesis test is a method for testing a hypothesis about a parameter in a population using data measured in a sample.

We test a hypothesis by determining the chance of obtaining a sample statistic if the null hypothesis regarding the population parameter is true.

> The goal of hypothesis testing is to make a decision about the value of a population parameter based on sample data.

_____no_output_____**Why do we care about hypothesis testing?**

Scenarios:
* Chemistry - do inputs from two different barley fields produce different yields?
* Astrophysics - do star systems with near-orbiting gas giants have hotter stars?
* Economics - demography, surveys, etc.
* Medicine - BMI vs. Hypertension, etc.
* Business - which ad is more effective given engagement?_____no_output_____**Intuition**

Suppose you have a large dataset for a population. The data is normally distributed with mean 0 and standard deviation 1.

Along comes a new sample with a sample mean of 2.9.

> The idea behind hypothesis testing is a desire to quantify our belief as to whether our sample of observations came from the same population as the original dataset.

According to the empirical (68–95–99.7) rule for normal distributions there is only roughly a 0.003 chance that the sample came from the same population, because it is roughly 3 standard deviations above the mean.

To formalize this intuition, we define an threshold value for deciding whether we believe that the sample is from the same underlying population or not. This threshold is $\alpha$, the **significance threshold**.

This serves as the foundation for hypothesis testing where we will reject or fail to reject the null hypothesis.
_____no_output_____# Hypothesis testing

Regardless of the type of statistical hypothesis test you're performing, there are five main steps to executing them:

1. Set up a null and alternative hypothesis

2. Choose a significance level $\alpha$ (or use the one assigned).

3. Determine the critical test statistic value or p-value. **(Find the rejection region for the null hypothesis.)**

4. Calculate the value of the test statistic.

5. Compare the test statistic value to the critical test statistic value to reject the null hypothesis or not._____no_output__________no_output_____**Decision Rule**:

The decision rule tells us when we can reject the null hypothesis.

It depends on 3 factors:
1. The alternative hypothesis
    * Is this an upper-tailed, lower-tailed, or two-tailed test?
2. The test statistic
3. The level of significance $\alpha$.

Upper-tailed test (right-tailed test):
* The null hypothesis is rejected if the test statistic is greater than the critical value.

Lower-tailed test (left-tailed test):
* The null hypothesis is rejected if the test statistic is smaller than the critical value.

Two-tailed test:
* The null hypothesis is rejected if the test statistic is either larger than an upper critical value or smaller than a lower critical value._____no_output_____# Language of Hypothesis testing

**Significance Level $\alpha$**

The significance level $\alpha$ is the threshold at which you're okay with rejecting the null hypothesis. It is the probability of rejecting the null hypothesis when it is true.

The most commonly used $\alpha$ in science is $\alpha = 0.05$. When you set $\alpha = 0.05$, you're saying "I'm okay with rejecting the null hypothesis if there is less than a 5% chance that the results I am seeing are actually due to randomness".

**p-values**

The p-value is the probability of observing a test statistic at least as large as the one observed, by random chance, assuming that the null hypothesis is true.

If $p \lt \alpha$, we reject the null hypothesis.

If $p \geq \alpha$, we fail to reject the null hypothesis.

> **We do not accept the alternative hypothesis, we only reject or fail to reject the null hypothesis in favor of the alternative.**

**What if the experiment we perform fails to reject the null hypothesis?**

* We do not throw out failed experiments!
* We say "this methodology, with this data, does not produce significant results"
    * Maybe we need more data!_____no_output_____## Type 1 Errors (False Positives) and Type 2 Errors (False Negatives)
Most tests for the presence of some factor are imperfect. And in fact most tests are imperfect in two ways: They will sometimes fail to predict the presence of that factor when it is after all present, and they will sometimes predict the presence of that factor when in fact it is not. Clearly, the lower these error rates are, the better, but it is not uncommon for these rates to be between 1% and 5%, and sometimes they are even higher than that. (Of course, if they're higher than 50%, then we're better off just flipping a coin to run our test!)

Predicting the presence of some factor (i.e. counter to the null hypothesis) when in fact it is not there (i.e. the null hypothesis is true) is called a "false positive". Failing to predict the presence of some factor (i.e. in accord with the null hypothesis) when in fact it is there (i.e. the null hypothesis is false) is called a "false negative".

How does changing our alpha value change the rate of type 1 and type 2 errors?_____no_output_____# Let's continue our discussion of hypothesis tests with an example._____no_output_____Suppose that African elephants have weights distributed normally around a mean of 9000 lbs with a standard deviation of 900 lbs. _Pachyderm Adventures_ has recently measured the weights of **35** Gabonese elephants and has calculated their average weight at 8637 lbs.

Is the average weight of Gabonese elephants different that the average weight of African elephants? Use significance level $\alpha = 0.05$.

**What are the null and alternative hypotheses? What is the significance level of the test?**_____no_output_____* Null hypothesis
    * The average weight of Gabonese elephants is the same as the average weight of African elephants.

* Alternative hypothesis
    * The average weight of Gabonese elephants is different than the average weight of African elephants.

The significance level of our test is $\alpha = 0.05$. _____no_output_____**What should be our test statistic? Are we running an upper, lower, or two-tailed test? Why?**_____no_output_____Since we know the population standard deviation, the size of our sample is greater than 30, and we are comparing the sample mean to the population mean, we are going to run a one-sample z-test.

Since we want to know if the sample mean is **different** from the population mean, we are running a two-tailed test. _____no_output_____**What's the value of the critical test statistic that we should use for our test?**_____no_output_____

# critical z-statistic
alpha = 0.05

# point percent function is the inverse of the cumulative density function which can be understood as the quantile
stats.norm.ppf(alpha/2), stats.norm.ppf(1-alpha/2)_____no_output_____

> Since we are performing a two-tailed one-sample z-test and $\alpha = 0.05$, if the z-score we compute is greater than 1.96 or smaller than -1.96, then we can reject the null hypothesis at significance level 0.05 in favor of the alternative hypothesis. _____no_output_____**Perform the test.**

Compute the relevant test statistic for the sample._____no_output_____Compute the z-statistic for the sample.

$$\text{z-statistic} = \frac{\bar{x} - \mu}{\sigma/\sqrt{n}}, $$ where $\bar x$ is the sample mean, $\mu$ is the population mean, $\sigma$ is the population standard deviation, and $n$ is the sample size. _____no_output_____

n = 35
sigma = 900

x_bar = 8637
mu = 9000

se = sigma/np.sqrt(n)
z = (x_bar - mu)/se
print(z)-2.386152179183512

**Make a decision: do we reject the null hypothesis or not?**_____no_output_____> z = -2.39 is smaller than -1.96, thus we can reject the null hypothesis in favor of the alternative hypothesis at significance level $\alpha = 0.05$.
- - -

Another way of getting to same answer: _____no_output_____

stats.norm.cdf(z)_____no_output_____

> The area of the tail corresponding to this z-score is 0.0085. This is below 0.025. Thus we reject the null hypothesis in favor of the alternative at significance level $\alpha = 0.05$. _____no_output_____**Would we be able to reject the null hypothesis if our significance threshold was $\alpha = 0.01$?**_____no_output_____The area of the tail corresponding to the calculated z-statistic z = -2.386 is `stats.norm.cdf(z) = 0.0085`.

> Since the area of the tail corresponding to the z-score we obtained is 0.0085, which is greater than 0.005, we fail to reject the null hypothesis in favor of the alternative at a significance level of $\alpha = 0.01$. _____no_output_____

# critical z-statistic
alpha = 0.01
stats.norm.ppf(alpha/2), stats.norm.ppf(1-alpha/2)_____no_output_____

> Alternatively, since we are performing a two-tailed one-sample z-test and $\alpha = 0.01$, if the z-score we compute is greater than 2.58 or smaller than -2.58, then we can reject the null hypothesis at significance level 0.05 in favor of the alternative hypothesis.

>Since the calculated z-statistic is -2.386, we fail to reject the null hypothesis in favor of the alternative at a significance level of $\alpha = 0.01$. _____no_output_____# z-tests vs t-tests

According to the **Central Limit Theorem**, the sampling distribution of a statistic, like the sample mean, will follow a normal distribution _as long as the sample size is sufficiently large_.

__What if we don't have large sample sizes?__

When we do not know the population standard deviation or we have a small sample size, the sampling distribution of the sample statistic will follow a t-distribution.
* Smaller sample sizes have larger variance, and t-distributions account for that by having heavier tails than the normal distribution.
* t-distributions are parameterized by degrees of freedom, fewer degrees of freedom fatter tails. Also converges to a normal distribution as dof >> 0_____no_output_____# One-sample z-tests and one-sample t-tests

One-sample z-tests and one-sample t-tests are hypothesis tests for the population mean $\mu$.

How do we know whether we need to use a z-test or a t-test?

_____no_output_____**When we perform a hypothesis test for the population mean, we want to know how likely it is to obtain the test statistic for the sample mean given the null hypothesis that the sample mean and population mean are not different.**

The test statistic for the sample mean summarizes our sample observations. How do test statistics differ for one-sample z-tests and t-tests?

A t-test is like a modified z-test.

* Penalize for small sample size: "degrees of freedom"

* Use sample standard deviation $s$ to estimate the population standard deviation $\sigma$.

_____no_output_____A one-sample t-test estimates the population mean (one parameter). A sample with size $n$ provides $n$ pieces of information, or degrees of freedom, for estimating the population mean and its variability.

One degree of freedom is used to estimate the mean, the remaining $n-1$ degrees of freedom are used to estimate variability.

>The one-sample t-test for samples of size $n$ has $n-1$ degrees of freedom._____no_output_____
_____no_output_____## One-sample z-test

* For large enough sample sizes $n$ with known population standard deviation $\sigma$, the test statistic of the sample mean $\bar x$ is given by the **z-statistic**,
$$Z = \frac{\bar{x} - \mu}{\sigma/\sqrt{n}}$$ where $\mu$ is the population mean.

* Our hypothesis test tries to answer the question of how likely we are to observe a z-statistic as extreme as our sample's given the null hypothesis that the sample and the population have the same mean, given a significance threshold of $\alpha$. This is a one-sample z-test.  _____no_output_____## One-sample t-test

* For small sample sizes or samples with unknown population standard deviation, the test statistic of the sample mean is given by the **t-statistic**,
$$ t = \frac{\bar{x} - \mu}{s/\sqrt{n}} $$ Here, $s$ is the sample standard deviation, which is used to estimate the population standard deviation, and $\mu$ is the population mean.

* Our hypothesis test tries to answer the question of how likely we are to observe a t-statistic as extreme as our sample's given the null hypothesis that the sample and population have the same mean, given a significance threshold of $\alpha$. This is a one-sample t-test._____no_output_____## Compare and contrast z-tests and t-tests.
In both cases, it is assumed that the samples are normally distributed.

A t-test is like a modified z-test:
1. Penalize for small sample size; use "degrees of freedom"
2. Use the _sample_ standard deviation $s$ to estimate the population standard deviation $\sigma$.

T-distributions have more probability in the tails. As the sample size increases, this decreases and the t distribution more closely resembles the z, or standard normal, distribution. By sample size n = 1000 they are virtually indistinguishable from each other. _____no_output_____## Here's an example:

A coffee shop relocates from Manhattan to Brooklyn and wants to make sure that all lattes are consistent before and after their move. They buy a new machine and hire a new barista. In Manhattan, lattes are made with 4 oz of espresso. A random sample of 25 lattes made in their new store in Brooklyn shows a mean of 4.6 oz and standard deviation of 0.22 oz. Are their lattes different now that they've relocated to Brooklyn?

**What's the null and alternative hypothesis to test in this case? What kind of test should we run? Why?** _____no_output_____> $H_0$: Lattes are the same.

> $H_1$: Lattes are different.

>> Should run a one-sample t-test. Unknown population standard deviation. Small sample size. _____no_output_____## Two-sample t-tests

Sometimes, we are interested in determining whether two population means are equal. In this case, we use two-sample t-tests.

There are two types of two-sample t-tests: **paired** and **independent** (unpaired) tests.

What's the difference?

**Paired tests**: How is a sample affected by a certain treatment? The individuals in the sample remain the same and you compare how they change after treatment.

**Independent tests**: When we compare two different, unrelated samples to each other, we use an independent (or unpaired) two-sample t-test._____no_output_____The test statistic for an unpaired two-sample t-test is slightly different than the test statistic for the one-sample t-test.

Assuming equal variances, the test statistic for a two-sample t-test is given by:

$$ t = \frac{\bar{x_1} - \bar{x_2}}{\sqrt{s^2 \left( \frac{1}{n_1} + \frac{1}{n_2} \right)}}$$

where $s^2$ is the pooled sample variance,

$$ s^2 = \frac{\sum_{i=1}^{n_1} \left(x_i - \bar{x_1}\right)^2 + \sum_{j=1}^{n_2} \left(x_j - \bar{x_2}\right)^2 }{n_1 + n_2 - 2} $$

Here, $n_1$ is the sample size of sample 1 and $n_2$ is the sample size of sample 2.

An independent two-sample t-test for samples of size $n_1$ and $n_2$ has $(n_1 + n_2 - 2)$ degrees of freedom. _____no_output_____## Sample problem: Unpaired two-sample t-test

You measure the delivery times of ten different restaurants in two different neighborhoods, A and B. You want to know if restaurants in the different neighborhoods have the same delivery times. It's okay to assume both samples have equal variances.

``` python
delivery_times_A = [28.4, 23.3, 30.4, 28.1, 29.4, 30.6, 27.8, 30.9, 27.0, 32.8]
delivery_times_B = [26.4, 26.3, 27.4, 30.4, 25.1, 28.4, 23.3, 24.7, 31.8, 24.3]
```_____no_output_____# Let's practice solving hypothesis test problems!_____no_output_____## Example 1
Let's revisit our Gabonese elephant weight example.

Suppose that African elephants have weights distributed normally around a mean of 9000 lbs with a standard deviation of 900 lbs. _Pachyderm Adventures_ has recently measured the weights of **35** Gabonese elephants and has calculated their average weight at 8637 lbs.

Is the average weight of Gabonese elephants _less_ than the average weight of African elephants? Use significance level $\alpha = 0.05$.

**What are the null and alternative hypothesis in this case?**

**What kind of test do we need to run?**

**What's the critical test statistic value we should use?**

**Perform the test and make a decision regarding the null hypothesis.**_____no_output_____* Null hypothesis
    * The average weight of Gabonese elephants is the same as the average weight of African elephants.

* Alternative hypothesis
    * The average weight of Gabonese elephants is less than the average weight of African elephants.
- - -

We need to run a lower-tailed one-sample z-test. _____no_output_____## Example 2
Next, let's finish working through our coffee shop example...

A coffee shop relocates from Manhattan to Brooklyn and wants to make sure that all lattes are consistent before and after their move. They buy a new machine and hire a new barista. In Manhattan, lattes are made with 4 oz of espresso. A random sample of 25 lattes made in their new store in Brooklyn shows a mean of 4.6 oz and standard deviation of 0.22 oz. Are their lattes different now that they've relocated to Brooklyn? Use a significance level of $\alpha = 0.01$. _____no_output_____State null and alternative hypothesis
1. Null: the amount of espresso in the lattes is the same as before the move.
2. Alternative: the amount of espresso in the lattes is different before and after the move.

What kind of test?
* two-tailed one-sample t-test
    * small sample size
    * unknown population standard deviation
    * two-tailed because we want to know if amounts are same or different _____no_output_____

x_bar = 4.6
mu = 4
s = 0.22
n = 25

df = n-1

t = (x_bar - mu)/(s/n**0.5)
print("The t-statistic for our sample is {}.".format(round(t, 2)))The t-statistic for our sample is 13.64.
# critical t-statistic values
stats.t.ppf(0.005, df), stats.t.ppf(1-0.005, df)_____no_output_____

Can we reject the null hypothesis?

> Yes. t > |t_critical|. we can reject the null hypothesis in favor of the alternative at $\alpha = 0.01$. _____no_output_____## Example 3

I'm buying jeans from store A and store B. I know nothing about their inventory other than prices.

``` python
store1 = [20,30,30,50,75,25,30,30,40,80]
store2 = [60,30,70,90,60,40,70,40]
```

Should I go just to one store for a less expensive pair of jeans? I'm pretty apprehensive about my decision, so $\alpha = 0.1$. It's okay to assume the samples have equal variances._____no_output_____**State the null and alternative hypotheses**_____no_output_____> Null: Store A and B have the same jean prices.

> Alternative: Store A and B do not have the same jean prices. _____no_output_____**What kind of test should we run? Why?** _____no_output_____> Run a two-tailed two independent sample t-test. Sample sizes are small. _____no_output_____**Perform the test.**_____no_output_____

store1 = [20,30,30,50,75,25,30,30,40,80]
store2 = [60,30,70,90,60,40,70,40]

stats.ttest_ind(store1, store2)_____no_output_____

**Make decision.**_____no_output_____> We fail to reject the null hypothesis at a significance level of $\alpha = 0.1$. We do not have evidence to support that jean prices are different in store A and store B. _____no_output_____## Example 4

Next, let's finish working through the restaurant delivery times problem.

You measure the delivery times of ten different restaurants in two different neighborhoods. You want to know if restaurants in the different neighborhoods have the same delivery times. It's okay to assume both samples have equal variances. Set your significance threshold to 0.05.

``` python
delivery_times_A = [28.4, 23.3, 30.4, 28.1, 29.4, 30.6, 27.8, 30.9, 27.0, 32.8]
delivery_times_B = [26.4, 26.3, 27.4, 30.4, 25.1, 28.4, 23.3, 24.7, 31.8, 24.3]
```_____no_output_____State null and alternative hypothesis. What type of test should we perform? _____no_output_____> Null hypothesis: The delivery times for restaurants in neighborhood A are equal to delivery times for restaurants in neighborhood B.

> Alternative hypothesis: Delivery times for restaurants in neighborhood A are not equal to delivery times for restaurants in neighborhood B.

> Two-sided unpaired two-sample t-test_____no_output_____

delivery_times_A = [28.4, 23.3, 30.4, 28.1, 29.4, 30.6, 27.8, 30.9, 27.0, 32.8]
delivery_times_B = [26.4, 26.3, 27.4, 30.4, 25.1, 28.4, 23.3, 24.7, 31.8, 24.3]_____no_output_____stats.ttest_ind(delivery_times_A, delivery_times_B)_____no_output_____

> We cannot reject the null hypothesis that restaurant A and B have equal delivery times. p-value > $\alpha$. _____no_output_____# Level Up: More practice problems!_____no_output_____A rental car company claims the mean time to rent a car on their website is 60 seconds with a standard deviation of 30 seconds. A random sample of 36 customers attempted to rent a car on the website. The mean time to rent was 75 seconds. Is this enough evidence to contradict the company's claim at a significance level of $\alpha = 0.05$? _____no_output_____Null hypothesis:

Alternative hypothesis:
_____no_output_____

# one-sample z-test
_____no_output_____

Reject?:_____no_output_____Consider the gain in weight (in grams) of 19 female rats between 28 and 84 days after birth.

Twelve rats were fed on a high protein diet and seven rats were fed on a low protein diet.

``` python
high_protein = [134, 146, 104, 119, 124, 161, 107, 83, 113, 129, 97, 123]
low_protein = [70, 118, 101, 85, 107, 132, 94]
```

Is there any difference in the weight gain of rats fed on high protein diet vs low protein diet? It's OK to assume equal sample variances. _____no_output_____Null and alternative hypotheses? _____no_output_____> null:

> alternative: _____no_output_____What kind of test should we perform and why? _____no_output_____> Test:_____no_output_____We fail to reject the null hypothesis at a significance level of $\alpha = 0.05$. _____no_output_____**What if we wanted to test if the rats who ate a high protein diet gained more weight than those who ate a low-protein diet?**_____no_output_____Null:

alternative:_____no_output_____Kind of test? _____no_output_____Critical test statistic value? _____no_output_____Can we reject?_____no_output_____# Summary

Key Takeaways:

* A statistical hypothesis test is a method for testing a hypothesis about a parameter in a population using data measured in a sample.
* Hypothesis tests consist of a null hypothesis and an alternative hypothesis.
* We test a hypothesis by determining the chance of obtaining a sample statistic if the null hypothesis regarding the population parameter is true.
* One-sample z-tests and one-sample t-tests are hypothesis tests for the population mean $\mu$.
* We use a one-sample z-test for the population mean when the population standard deviation is known and the sample size is sufficiently large. We use a one-sample t-test for the population mean when the population standard deviation is unknown or when the sample size is small.
* Two-sample t-tests are hypothesis tests for differences in two population means. _____no_output_____


# Notebook from DarkTitan007/COVID-19-Genome-Analysis-using-ML
Path: COVID_19_Genome_Analysis_using_ML.ipynb

_____no_output_____

!pip install BiopythonCollecting Biopython
  Downloading biopython-1.79-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (2.3 MB)
     |████████████████████████████████| 2.3 MB 5.2 MB/s
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from Biopython) (1.21.5)
Installing collected packages: Biopython
Successfully installed Biopython-1.79
from Bio import Entrez, SeqIO
Entrez.email = ""
handle = Entrez.efetch(db="nucleotide", id="MN908947", rettype="gb", retmode="text")
recs = list(SeqIO.parse(handle, 'gb'))
handle.close()_____no_output_____recs_____no_output_____covid_dna = recs[0].seq_____no_output_____covid_dna_____no_output_____print(f'The genome of Covid-19 consists of {len(covid_dna)} nucleotides.')The genome of Covid-19 consists of 29903 nucleotides.
# molecular weight
from Bio.SeqUtils import molecular_weight
molecular_weight(covid_dna)_____no_output_____# GC content - higher GC content implies more stable molecule due to G and C forming triple hydrogen bonds
from Bio.SeqUtils import GC
GC(covid_dna)_____no_output_____count_nucleotides = {
    'A': covid_dna.count('A'),
    'T': covid_dna.count('T'),
    'C': covid_dna.count('C'),
    'G': covid_dna.count('G')
}_____no_output_____count_nucleotides_____no_output_____import matplotlib.pyplot as plt
width = 0.5
plt.bar(count_nucleotides.keys(), count_nucleotides.values(), width, color=['b', 'r', 'm', 'c'])
plt.xlabel('Nucleotide')
plt.ylabel('Frequency')
plt.title('Nucleotide Frequency')_____no_output_____covid_mrna = covid_dna.transcribe()
covid_mrna_____no_output_____covid_aa = covid_mrna.translate()
covid_aa/usr/local/lib/python3.7/dist-packages/Bio/Seq.py:2983: BiopythonWarning: Partial codon, len(sequence) not a multiple of three. Explicitly trim the sequence or add trailing N before translation. This may become an error in future.
  BiopythonWarning,
#most common amino acids
from collections import Counter
common_amino = Counter(covid_aa)
common_amino.most_common(10)_____no_output_____del common_amino['*']

width = 0.5
plt.bar(common_amino.keys(), common_amino.values(), width, color=['b', 'r', 'm', 'c'])
plt.xlabel('Amino Acid')
plt.ylabel('Frequency')
plt.title('Protein Sequence Frequency')_____no_output_____print(f"Covid-19's genome has {sum(common_amino.values())} amino acids")Covid-19's genome has 9193 amino acids
proteins = covid_aa.split('*')_____no_output_____proteins[:5]_____no_output_____print(f'We have {len(proteins)} amino acids in the covid-19 genome')We have 775 amino acids in the covid-19 genome
for protein in proteins:
    if len(protein) < 20:
        proteins.remove(protein)_____no_output_____print(f'We have {len(proteins)} proteins with more than 20 amino acids in the covid-19 genome')We have 409 proteins with more than 20 amino acids in the covid-19 genome
top_5_proteins = sorted(proteins, key = len)_____no_output_____top_5_proteins[-1]_____no_output_____len(top_5_proteins[-1])_____no_output_____with open("protein_seq.fasta", "w") as file:
    file.write(f">covid protein\n{top_5_proteins[-1]}")_____no_output_____

**Summary of Findings**
*Sequence length*: 29,903 base pairs
*GC content*: 37.97%
**Has high amount of Leucine L and Serine S**
**409 proteins with more than 20 amino acids**
*The largest protein is of length of 2,701 amino acid*._____no_output_____


# Notebook from arnaldoneto01/aceleracao-data-science
Path: data-science-1/main.ipynb

# Desafio 3

Neste desafio, iremos praticar nossos conhecimentos sobre distribuições de probabilidade. Para isso,
dividiremos este desafio em duas partes:

1. A primeira parte contará com 3 questões sobre um *data set* artificial com dados de uma amostra normal e
    uma binomial.
2. A segunda parte será sobre a análise da distribuição de uma variável do _data set_ [Pulsar Star](https://archive.ics.uci.edu/ml/datasets/HTRU2), contendo 2 questões.

> Obs.: Por favor, não modifique o nome das funções de resposta._____no_output_____## _Setup_ geral_____no_output_____

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as sct
import seaborn as sns
from statsmodels.distributions.empirical_distribution import ECDF_____no_output_____# %matplotlib inline

from IPython.core.pylabtools import figsize

figsize(12, 8)

sns.set()_____no_output_____

## Parte 1_____no_output_____### _Setup_ da parte 1_____no_output_____

np.random.seed(42)

dataframe = pd.DataFrame({"normal": sct.norm.rvs(20, 4, size=10000),
                     "binomial": sct.binom.rvs(100, 0.2, size=10000)})_____no_output_____

## Inicie sua análise a partir da parte 1 a partir daqui_____no_output_____

# Sua análise da parte 1 começa aqui.
dataframe.describe()_____no_output_____

## Questão 1

Qual a diferença entre os quartis (Q1, Q2 e Q3) das variáveis `normal` e `binomial` de `dataframe`? Responda como uma tupla de três elementos arredondados para três casas decimais.

Em outra palavras, sejam `q1_norm`, `q2_norm` e `q3_norm` os quantis da variável `normal` e `q1_binom`, `q2_binom` e `q3_binom` os quantis da variável `binom`, qual a diferença `(q1_norm - q1 binom, q2_norm - q2_binom, q3_norm - q3_binom)`?_____no_output_____

def q1():
    q_norm = dataframe['normal'].quantile((0.25, 0.5, 0.75))
    q_binom = dataframe['binomial'].quantile((0.25, 0.5, 0.75))
    return tuple((q_norm - q_binom).round(3))_____no_output_____

Para refletir:

* Você esperava valores dessa magnitude?
Sim
* Você é capaz de explicar como distribuições aparentemente tão diferentes (discreta e contínua, por exemplo) conseguem dar esses valores?
A distribuição binomial se aproxima da distribuição normal quando há quantidade suficiente de observações (e.g. n > 30)_____no_output_____## Questão 2

Considere o intervalo $[\bar{x} - s, \bar{x} + s]$, onde $\bar{x}$ é a média amostral e $s$ é o desvio padrão. Qual a probabilidade nesse intervalo, calculada pela função de distribuição acumulada empírica (CDF empírica) da variável `normal`? Responda como uma único escalar arredondado para três casas decimais._____no_output_____

def q2(spread=1):
    ecdf = ECDF(dataframe['normal'])
    normal_std = dataframe['normal'].std()
    normal_mean = dataframe['normal'].mean()
    upper_p = ecdf(normal_mean+normal_std*spread)
    lower_p = ecdf(normal_mean-normal_std*spread)
    return float((upper_p-lower_p).round(3))
print("{} probalidade para 1s".format(q2()))
print("{} probalidade para 2s".format(q2(spread=2)))
print("{} probalidade para 3s".format(q2(spread=3)))0.684 probalidade para 1s
0.954 probalidade para 2s
0.997 probalidade para 3s

Para refletir:

* Esse valor se aproxima do esperado teórico?
Sim, foi obtido 0.684 e o valor teórico esperado é de 0.682
* Experimente também para os intervalos $[\bar{x} - 2s, \bar{x} + 2s]$ e $[\bar{x} - 3s, \bar{x} + 3s]$.
Os valores foram 0.954 e 0.997 respectivamente para 2s e 3s. _____no_output_____## Questão 3

Qual é a diferença entre as médias e as variâncias das variáveis `binomial` e `normal`? Responda como uma tupla de dois elementos arredondados para três casas decimais.

Em outras palavras, sejam `m_binom` e `v_binom` a média e a variância da variável `binomial`, e `m_norm` e `v_norm` a média e a variância da variável `normal`. Quais as diferenças `(m_binom - m_norm, v_binom - v_norm)`?_____no_output_____

def q3():
    norm_description = np.array([dataframe['normal'].mean(),dataframe['normal'].var()])
    binom_description = np.array([dataframe['binomial'].mean(),dataframe['binomial'].var()])
    return tuple((binom_description-norm_description).round(3))_____no_output_____

Para refletir:

* Você esperava valore dessa magnitude?
Sim
* Qual o efeito de aumentar ou diminuir $n$ (atualmente 100) na distribuição da variável `binomial`?
A alteração no valor de n irá alterar a média e variância da distribuição binomial. Aumentando o valor de n, aumentam-se a média e variância.  _____no_output_____## Parte 2_____no_output_____### _Setup_ da parte 2_____no_output_____

stars = pd.read_csv("pulsar_stars.csv")

stars.rename({old_name: new_name
              for (old_name, new_name)
              in zip(stars.columns,
                     ["mean_profile", "sd_profile", "kurt_profile", "skew_profile", "mean_curve", "sd_curve", "kurt_curve", "skew_curve", "target"])
             },
             axis=1, inplace=True)

stars.loc[:, "target"] = stars.target.astype(bool)_____no_output_____

## Inicie sua análise da parte 2 a partir daqui_____no_output_____

# Sua análise da parte 2 começa aqui.
stars.head()
_____no_output_____

## Questão 4

Considerando a variável `mean_profile` de `stars`:

1. Filtre apenas os valores de `mean_profile` onde `target == 0` (ou seja, onde a estrela não é um pulsar).
2. Padronize a variável `mean_profile` filtrada anteriormente para ter média 0 e variância 1.

Chamaremos a variável resultante de `false_pulsar_mean_profile_standardized`.

Encontre os quantis teóricos para uma distribuição normal de média 0 e variância 1 para 0.80, 0.90 e 0.95 através da função `norm.ppf()` disponível em `scipy.stats`.

Quais as probabilidade associadas a esses quantis utilizando a CDF empírica da variável `false_pulsar_mean_profile_standardized`? Responda como uma tupla de três elementos arredondados para três casas decimais._____no_output_____

def q4():
    false_pulsar_mean_profile = stars[stars['target']==0]
    s1 = false_pulsar_mean_profile['mean_profile']
    false_pulsar_mean_profile_standardized = (s1 - s1.mean()) / s1.std()
    ecdf = ECDF(false_pulsar_mean_profile_standardized)
    z_score = sct.norm.ppf([0.8, 0.90, 0.95])
    print("Quantis teóricos \n{}".format(z_score))
    return tuple(ecdf(z_score).round(3))_____no_output_____

Para refletir:

* Os valores encontrados fazem sentido?
Sim. Esses resultado indicam que a curva tem comportamento de uma distribuição normal
* O que isso pode dizer sobre a distribuição da variável `false_pulsar_mean_profile_standardized`?
A distribuição da variável `false_pulsar_mean_profile_standardized` se comporta como uma distribuição normal_____no_output_____## Questão 5

Qual a diferença entre os quantis Q1, Q2 e Q3 de `false_pulsar_mean_profile_standardized` e os mesmos quantis teóricos de uma distribuição normal de média 0 e variância 1? Responda como uma tupla de três elementos arredondados para três casas decimais._____no_output_____

def q5():
    false_pulsar_mean_profile = stars[stars['target']==0]
    s1 = false_pulsar_mean_profile['mean_profile']
    false_pulsar_mean_profile_standardized = (s1 - s1.mean()) / s1.std()
    q_false_pusar = false_pulsar_mean_profile_standardized.quantile((0.25,0.5,0.75))
    z_score = sct.norm.ppf([0.25, 0.5, 0.75])
    return tuple((q_false_pusar-z_score).round(3))_____no_output_____

Para refletir:

* Os valores encontrados fazem sentido?
Sim
* O que isso pode dizer sobre a distribuição da variável `false_pulsar_mean_profile_standardized`?
A distribuição da variável `false_pulsar_mean_profile_standardized` se comporta como uma distribuição normal
* Curiosidade: alguns testes de hipóteses sobre normalidade dos dados utilizam essa mesma abordagem._____no_output_____


# Notebook from jumelet/path_explain
Path: examples/tabular/pulsar/find_peak_examples.ipynb

<code>
%load_ext autoreload
%autoreload 2_____no_output_____import tensorflow as tf
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix

from path_explain.path_explainer_tf import PathExplainerTF
from path_explain.utils import set_up_environment
from plot.scatter import scatter_plot, _set_axis_config
from plot.summary import summary_plot

from preprocess import pulsar_dataset_____no_output_____x_train, y_train, x_test, y_test, feature_names, x_train_un, x_test_un = pulsar_dataset()_____no_output_____attributions = np.load('attributions.npy')
interactions = np.load('interactions.npy')_____no_output_____feature_names_____no_output_____feature_names = [
    'Mean (IP)',
    'SD (IP)',
    'Kurtosis (IP)',
    'Skewness (IP)',
    'Mean (DM-SNR)',
    'SD (DM-SNR)',
    'Kurtosis (DM-SNR)',
    'Skewness (DM-SNR)'
]_____no_output_____kurtosis_profile_index = 2
sd_dm_snr_index = 5_____no_output_____fig, axs = scatter_plot(attributions,
                        x_test,
                        feature_index='SD (DM-SNR)',
                        interactions=interactions,
                        color_by='Kurtosis (IP)',
                        feature_names=feature_names,
                        scale_y_ind=True)_____no_output_____fig, axs = scatter_plot(attributions,
                        x_test_un.values,
                        feature_index='Kurtosis (IP)',
                        interactions=interactions,
                        color_by='SD (DM-SNR)',
                        feature_names=feature_names,
                        scale_y_ind=True,
                        plot_main=False)
ax = axs[1]
ax.axvline(0.3, c='lime', lw=3)
ax.axvline(1.8, c='lime', lw=3)
plt.savefig('kurtosis_and_sd.pdf')_____no_output_____def customize_axis(ax, xlabel, ylabel, title, ax_below=True):
    _set_axis_config(ax, [0.2, 0.2, 1.0, 1.0])
    ax.grid(linewidth=0.5)
    ax.set_axisbelow(ax_below)
    ax.tick_params(length=6, labelsize=12)
    ax.set_xlabel(xlabel, fontsize=14)
    ax.set_ylabel(ylabel, fontsize=14)
    ax.set_title(title, fontsize=18)_____no_output_____data = pd.DataFrame({
    'Kurtosis (IP)': x_train_un.values[:, kurtosis_profile_index],
    'SD (DM-SNR)': x_train_un.values[:, sd_dm_snr_index],
    'Label': y_train
})
fig, ax = plt.subplots(1, 1, figsize=(8, 5))

for c in [0, 1]:
    select_data = data[data['Label'] == c]
    color = 'firebrick'
    label = 'Pulsar Star'
    if c == 0:
        color = 'navy'
        label = 'Background Noise'

    ax.scatter(x=select_data['Kurtosis (IP)'],
               y=select_data['SD (DM-SNR)'],
               c=color,
               s=6,
               label=label)

lgnd = ax.legend(prop={'size': 12})
lgnd.legendHandles[0]._sizes = [50]
lgnd.legendHandles[1]._sizes = [50]

ax.axvline(0.3, c='lime', lw=3)
ax.axvline(1.8, c='lime', lw=3)
customize_axis(ax, xlabel='Kurtosis of the Integrated Profile', ylabel='Standard Deviation of the DM-SNR Curve', title='Plot of Kurtosis (IP) vs. SD (DM-SNR)')
plt.savefig('pulsar_data_points.pdf')_____no_output_____sns.jointplot(x='x', y='y', data=data[y_train==0], kind='kde', xlim=(-1, 4), ylim=(-1, 4))_____no_output_____sns.jointplot(x='x', y='y', data=data[y_train==1], kind='kde', xlim=(-1, 4), ylim=(-1, 4))_____no_output_____
</code>


# Notebook from lafleur1/isolearn
Path: examples/splicing_hexamer_regression_multicell.ipynb

import pandas as pd

import os
import numpy as np

import scipy.sparse as sp
import scipy.io as spio

import matplotlib.pyplot as plt
import matplotlib.cm as cm

import isolearn.io as isoio
import isolearn.generator as iso

import scipy.optimize as spopt
from scipy.stats import pearsonr

from logistic_regression_helpers import *
_____no_output_____

Load 5' Alternative Splicing Data

- Load a Pandas DataFrame + Matlab Matrix of measured Splicing Sequences

- isolearn.io loads all .csv and .mat files of a directory into memory as a dictionary

- The DataFrame has one column - padded_sequence - containing the splice donor sequence

- The Matrix contains RNA-Seq counts of measured splicing at each position across the sequence

_____no_output_____

#Load Splicing Data

splicing_dict = isoio.load('data/processed_data/splicing_5ss_data/splicing_5ss_data')
_____no_output_____

Create a Training and Test Set

- We create an index containing row numbers corresponding to training and test sequences

- Notice that we do not alter the underlying DataFrame, we only make lists of pointers to rows

_____no_output_____

#Generate training and test set indexes

test_set_size = 0.10

data_index = np.arange(len(splicing_dict['df']), dtype=np.int)

train_index = data_index[:-int(len(data_index) * test_set_size)]
test_index = data_index[train_index.shape[0]:]

print('Training set size = ' + str(train_index.shape[0]))
print('Test set size = ' + str(test_index.shape[0]))Training set size = 238183
Test set size = 26464

Create Data Generators

- In Isolearn, we always build data generators that will encode and feed us the data on the fly

- Here, for example, we create a training and test generator separately (using list comprehension)

- First argument: The list of row indices (of data points) for this generator

- Second argument: Dictionary or data sources

- Third argument: Batch size for the data generator
- Fourth argument: List of inputs, where each input is specified as a dictionary of attributes

- Fifth argument: List of outputs

In this example, we specify slightly more complicated inputs and outputs:

- We specicy two different inputs: Two hexamer encoders (NMerEncoders) for two different parts of the splice donor sequence

- We also use list comprehension to specify 4 different Splice Donor Usage targets (outputs), one for each cell line

_____no_output_____

#Create a hexamer count data generator, to be used for logistic regression of SD1 Usage (HEK cell line)
#Set the batch sizes equal to the size of the train/test sets.

splicing_gens = {
    gen_id : iso.DataGenerator(
        idx,
        {
            'df' : splicing_dict['df'],
            'hek_count' : splicing_dict['hek_count'],
            'hela_count' : splicing_dict['hela_count'],
            'mcf7_count' : splicing_dict['mcf7_count'],
            'cho_count' : splicing_dict['cho_count'],
        },
        batch_size=len(idx),
        inputs = [
            {
                'id' : 'random_region_1',
                'source_type' : 'dataframe',
                'source' : 'df',
                'extractor' : lambda row, index: row['padded_sequence'][122: 122 + 35],
                'encoder' : iso.NMerEncoder(n_mer_len=6, count_n_mers=True),
                'sparse' : True,
                'sparse_mode' : 'col'
            },
            {
                'id' : 'random_region_2',
                'source_type' : 'dataframe',
                'source' : 'df',
                'extractor' : lambda row, index: row['padded_sequence'][165: 165+35],
                'encoder' : iso.NMerEncoder(n_mer_len=6, count_n_mers=True),
                'sparse' : True,
                'sparse_mode' : 'col'
            }
        ],
        outputs = [
            {
                'id' : cell_type + '_sd1_usage',
                'source_type' : 'matrix',
                'source' : cell_type + '_count',
                'extractor' : lambda c, index: np.ravel(c),
                'transformer' : lambda t: t[120] / np.sum(t)
            } for cell_type in ['hek', 'hela', 'mcf7', 'cho']
        ],
        randomizers = [],
        shuffle = False,
        densify_batch_matrices=True
    ) for gen_id, idx in [('all', data_index), ('train', train_index), ('test', test_index)]
}
_____no_output_____

Execute the data generators by indexing them

In this example, we specified the batch size to be the entire train/test sets.

Hence, indexing into the generators at index 0 gives us the entire encoded dataset.
_____no_output_____

#Get training data from generator
[X_up_train, X_dn_train], [y_train_hek, y_train_hela, y_train_mcf7, y_train_cho] = splicing_gens['train'][0]
X_train = sp.hstack([X_up_train, X_dn_train])

#Get test data from generator
[X_up_test, X_dn_test], [y_test_hek, y_test_hela, y_test_mcf7, y_test_cho] = splicing_gens['test'][0]
X_test = sp.hstack([X_up_test, X_dn_test])
 _____no_output_____

Perform Logistic Hexamer Regression

Here we take our X and y matrices and feed them into an optimization algorithm (LM-BFGS with Logistic Regression) in order to learn log odds ratio weights of each of the hexamer counts encoded in matrix X, on the basis of how they influence splice donor usage (y).
_____no_output_____

#Perform hexamer logistic regression on all cell lines

cell_lines = [
    ('hek', (y_train_hek, y_test_hek)),
    ('hela', (y_train_hela, y_test_hela)),
    ('mcf7', (y_train_mcf7, y_test_mcf7)),
    ('cho', (y_train_cho, y_test_cho))
]

for cell_name, [y_train, y_test] in cell_lines :

    print("Starting logistic n-mer regression for cell line " + str(cell_name) + "...")

    w_init = np.zeros(X_train.shape[1] + 1)
    lambda_penalty = 0

    (w_bundle, _, _) = spopt.fmin_l_bfgs_b(log_loss, w_init, fprime=log_loss_gradient, args=(X_train, y_train, lambda_penalty), maxiter = 200)

    print("Regression finished.")

    #Collect weights
    w = w_bundle[1:]
    w_0 = w_bundle[0]

    #Evaluate predictions on test set
    y_test_pred = get_y_pred(X_test, w, w_0)

    r_val, p_val = pearsonr(y_test_pred, y_test)
    print("Test set R^2 = " + str(round(r_val * r_val, 2)) + ", p = " + str(p_val))

    #Plot test set scatter
    f = plt.figure(figsize=(4, 4))

    plt.scatter(y_test_pred, y_test, color='black', s=5, alpha=0.05)

    plt.xticks([0.0, 0.25, 0.5, 0.75, 1.0], fontsize=14)
    plt.yticks([0.0, 0.25, 0.5, 0.75, 1.0], fontsize=14)
    plt.xlabel('Predicted SD1 Usage', fontsize=14)
    plt.ylabel('True SD1 Usage', fontsize=14)
    plt.title(str(cell_name), fontsize=16)

    plt.xlim(0, 1)
    plt.ylim(0, 1)

    plt.tight_layout()
    plt.show()
Starting logistic n-mer regression for cell line hek...
Regression finished.
Test set R^2 = 0.78, p = 0.0