# Notebook from rpatil524/Community-Notebooks
Path: Notebooks/How_to_explore_CPTAC_protein_abundances.ipynb

_____no_output_____# Example notebook exploring CPTAC protein abundances
_____no_output_____Check out more notebooks at our [Community Notebooks Repository!](https://github.com/isb-cgc/Community-Notebooks)

```
Title:   Example notebook exploring CPTAC protein abundances
Author:  Boris Aguilar
Created: 01-19-2021
Purpose: Retrieve and analyze protein abundances from CPTAC
Notes:   This notebook recapitulates the following notebook https://pdc.cancer.gov/API_documentation/PDC_clustergram.html
```
The notebook extracts protein abundances from the CPTAC Clear cell renal cell carcinoma (CCRCC) quant and their associated clinical metadata from the publicly available BigQuery tables that the ISB-CGC project has produced based on CPTAC.
Finally, the notebook clusters and visualizes the data using the Seaborn clustermap package.
_____no_output_____## Modules_____no_output_____

from google.cloud import bigquery
from google.colab import auth
import seaborn as se
import pandas as pd
import pandas_gbq
import matplotlib.pyplot as plt
from scipy.stats import zscore_____no_output_____

## Defining helper functions_____no_output_____

# A color mapping function for the clinical annotations
def get_colors(df, name, color) -> pd.Series:
    s = pd.Series( df[name]  )
    #s = df[name]
    su = s.unique()
    colors = se.light_palette(color, len(su))
    lut = dict(zip(su, colors))
    return s.map(lut)_____no_output_____

## Google Authentication
The first step is to authorize access to BigQuery and the Google Cloud. For more information see ['Quick Start Guide to ISB-CGC'](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/HowToGetStartedonISB-CGC.html) and alternative authentication methods can be found [here](https://googleapis.dev/python/google-api-core/latest/auth.html).

Moreover you need to [create a google cloud](https://cloud.google.com/resource-manager/docs/creating-managing-projects#console) project to be able to run BigQuery queries._____no_output_____

auth.authenticate_user()
my_project_id = "" # write your project id here_____no_output_____

## Fetch the data
The following code obtains protein abundances and clinical metada for the all the cases in the CPTAC CCRCC study. Specifically we join two tables quant_proteome_CPTAC_CCRCC_discovery_study_pdc_current and clinical_CPTAC3_discovery_pdc_current that host protein abundances and clinical metada, respectively.

The results of query is automatically stored in pandas dataframe (quant_data) by the function read_gbq. _____no_output_____

sql = '''
SELECT pg.aliquot_submitter_id, pg.gene_symbol,
       CAST(pg.protein_abundance_log2ratio as FLOAT64) as log2ratio,
       clin.tumor_stage, clin.primary_diagnosis
FROM `isb-cgc-bq.CPTAC.quant_proteome_CPTAC_CCRCC_discovery_study_pdc_current` as pg
JOIN `isb-cgc-bq.CPTAC.clinical_CPTAC3_discovery_pdc_current` as clin
ON pg.case_id = clin.case_id
'''
quant_data = pandas_gbq.read_gbq(sql,project_id=my_project_id )
quant_data
Downloading: 100%|██████████| 1985337/1985337 [01:38<00:00, 20156.62rows/s]

## Analysis
The clustermap module within the Seaborn package does not allow for NaN values. So we must create a mask value that does not interfere much with the clustering and is likely to be unique._____no_output_____

mask_na = 0.000666
quant_data = quant_data.fillna(mask_na)_____no_output_____

We then need to pivot the data. This step transforms the data from tidy format used in BigQuery to the format required by the clustermap function ( aliquot_submitter_id as columns and gene_symbol as rows )._____no_output_____

ga = pd.pivot_table(quant_data,values='log2ratio',
                               index='gene_symbol',
                               columns='aliquot_submitter_id')
print(ga.shape)(9591, 207)

Next we set up colors for the clinical features *tumor_stage* and *primary_diagnosis*_____no_output_____

labels = quant_data[['aliquot_submitter_id','tumor_stage','primary_diagnosis']].drop_duplicates().set_index('aliquot_submitter_id')
stage_col_colors = get_colors(labels, 'tumor_stage', 'red')
diagnosis_col_colors = get_colors(labels, 'primary_diagnosis', 'green')

#combine the two series into a dataframe
color_bars = pd.concat([stage_col_colors, diagnosis_col_colors], axis=1)_____no_output_____

Finally we generate the clustergram_____no_output_____

se.clustermap(ga, metric='euclidean', method='complete', cmap='seismic', mask=ga == 0.000666, center=0.,
              figsize=(10, 10) ,  col_colors = color_bars )
plt.show() #12.5 50/usr/local/lib/python3.7/dist-packages/seaborn/matrix.py:649: UserWarning: Clustering large matrix with scipy. Installing `fastcluster` may give better performance.
  warnings.warn(msg)

The clustergram shows the protein expression data grouped according to aliquot_submitter_id (columns) and gene_symbol (rows).

You could also convert the log2 ratio data to a standard statistic, like z-score. This can help compress the range, accounting for outliers._____no_output_____

zdf = ga.T.apply(zscore, ddof=len(ga.columns)-1)
zdf = zdf.T_____no_output_____

And examine clustering according to that transformation._____no_output_____

se.clustermap(zdf, metric='euclidean', method='complete', cmap='seismic', mask=ga == 0.000666, center=0.,
              figsize=(10, 10) ,  col_colors = color_bars )
plt.show() #12.5 50/usr/local/lib/python3.7/dist-packages/seaborn/matrix.py:649: UserWarning: Clustering large matrix with scipy. Installing `fastcluster` may give better performance.
  warnings.warn(msg)

This clustergram has a better grouping of the data. One can clearly distinguish a cluster of cell lines in the left hand side. Please compare this figure with the one generated by the PDC notebook in the following link https://pdc.cancer.gov/API_documentation/PDC_clustergram.html_____no_output_____


# Notebook from tmoerman/alois
Path: notebooks/01_from_lcpm/07_TDA.ipynb

# 06 TDA

**Objective:**
* test out KeplerMapper on Spatial Transcriptomics data with different lenses_____no_output_____

# %load common.py
import pandas as pd
import numpy as np
import os

import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

sns.set(style="whitegrid")

# Commonly used constants.

slides = [
    'B02_D1', 'B02_E1', 'B03_C2', 'B03_D2', 'B04_D1',
    'B04_E1', 'B05_D2', 'B05_E2', 'B06_E1', 'B07_C2',
    'N02_C1', 'N02_D1', 'N03_C2', 'N03_D2', 'N04_D1',
    'N04_E1', 'N05_C2', 'N05_D2', 'N06_D2', 'N07_C1']

GT = 'GT'
YEN = 'AB1_StdDev_Yen'
REGION = 'Region_predict'
ASTROCYTE = 'astrocyte'
NEURON = 'neuron'
INTERNEURON = 'interneuron'
AGE = 'age'
DAYS = 'age_days'
MONTHS = 'age_months'
AGE_GT ='age_GT'
SAMPLE_ID = 'sampleID'
SPOT_UID = 'spot_UID'
SLIDE_ID = 'slide_ID'

# The merged data file in Parquet format.

parquet = '/media/tmo/data/work/datasets/02_ST/parquet/'
st_full = parquet + 'st_full'

def read_full(path=st_full):
    return pd.read_parquet(st_full)

def enrich(full):
    # Add age column (young, old)
    full[AGE] = np.where(full[MONTHS] < 10, 'young', 'old')

    # Add combined column age_GT.
    full[AGE_GT] = full[[AGE, GT]].apply(lambda x: '_'.join(x), axis=1)

    # parse slide ID
    full['slide_ID'] = full[SPOT_UID].apply(lambda x: str(x).split('__')[0])

    return full

def read_enriched(path=st_full):
    return enrich(read_full(path))

# TODO
# * add 'coarse_region' that joins regions with <500 entries in them into one region OTHER_____no_output_____full = read_enriched()_____no_output_____n_genes = 46454_____no_output_____gene_cols = full.columns[1:n_genes+1]_____no_output_____ex_matrix = full[gene_cols].as_matrix()_____no_output_____ex_matrix.shape_____no_output_____ex_matrix_AB1 = full[full['AB1_StdDev_Yen'] > 0][gene_cols].as_matrix()_____no_output_____ex_matrix_AB1.shape_____no_output_____

---_____no_output_____

import kmapper as km
from kmapper.cover import Cover
from sklearn.cluster import DBSCAN_____no_output_____mapper = km.KeplerMapper(verbose=1)_____no_output_____yen_lens = full[full['AB1_StdDev_Yen'] > 0]['AB1_StdDev_Yen'].values_____no_output_____full[full['AB1_StdDev_Yen'] > 0][['AB1_StdDev_Yen']].sort_values(by='AB1_StdDev_Yen', ascending=False).plot(use_index=False)
plt.show()_____no_output_____%%time
graph = mapper.map(yen_lens,
                   ex_matrix_AB1,
                   clusterer=DBSCAN(n_jobs=-1, eps=0.01),
                   coverer=Cover(nr_cubes=50, overlap_perc=0.33))Mapping on data shaped (5009, 46454) using lens shaped (5009,)

Creating 50 hypercubes.

Created 0 edges and 0 nodes in 0:07:51.347290.
CPU times: user 2min 6s, sys: 5min 29s, total: 7min 35s
Wall time: 7min 51s
mapper.visualize(graph)_____no_output_____from sklearn.manifold import t_sne_____no_output_____t_sne.TSNE?_____no_output_____


# Notebook from jrneeley/Pulsating_Variable_Tools
Path: notebooks/Create_simulated_variables.ipynb

# Create simulated variable stars - example

Below is an example specific to VV124 (data provided in test_data folder), which could be adapted fairly easily for any photometric catalogs from DAOPHOT. This program *does not* include period fitting, but creates light curve files that are compatible with the template fitting IDL program FITLC (Sarajedini et al. 2009). Modifications to variable_simulations.py would be necessary for compatibility with other period fitting software._____no_output_____

from Variable_Star_Analysis import variable_simulations as vs
import numpy as np
import matplotlib.pyplot as plt

%load_ext autoreload
%autoreload 2_____no_output_____

## Setup

You will likely need to write your own function to describe the properties (scatter, photometric uncertainty as a function of magnitude) of your dataset in order to run the best simulations. The functions below are specific to the format of the photometric catalogs in the test data.

*Requirements*

You must create three arrays, which are used to interpolate the proper scatter (between observations of the same star) and photometric uncertainty (of individual measurements) for a given magnitude. Below is a description of the three arrays.

- error_key - Lists the filter/exposure time pairs of your dataset. Example format:
    `np.zeros(nrows, dtype=np.dtype([('filter', 'U5'), ('exptime', float)]))`
    where nrows is the number of filter/exptime pairs in your dataset
- error_array - Contains the typcial photometric uncertainty, and standard deviation of photometric uncertainty for several magnitude bins. Dimensions are (num filter/exptime pairs, num of magnitude bins, 3). Example format:
`np.zeros((nrows,nbins,3))` where nrows is the number of filter/exptime pairs, and nbins is the number of magnitude bins appropriate for you data.
- scatter_array - Contains the typical scatter between observations of the same star, for several magnitude bins, as well as the standard deviation of observed scatters within that bin. Example format:
`np.zeros((n_filters,nbins,3))` where n_filters is the number of filters in your dataset and nbins is the number of magnitude bins (ideally same as in error_array)_____no_output_____

from scipy.stats import binned_statistic

def mad(x):
        med_x = np.nanmedian(x)
        return np.nanmedian(np.abs(x - med_x))

def get_properties(n_epochs):

    n_filters = 2

    multiepoch_data = np.loadtxt('../test_data/vv124/vv124_allcal_12_1line.raw.tran')

    b_mags = multiepoch_data[:,3:3+n_epochs*2:2]
    b_emags = multiepoch_data[:,4:4+n_epochs*2:2]

    i_mags = multiepoch_data[:,3+n_epochs*2:-2:2]
    i_emags = multiepoch_data[:,4+n_epochs*2:-2:2]

    b_mags[b_mags > 50] = np.nan
    b_emags[b_emags > 5] = np.nan
    i_mags[i_mags > 50] = np.nan
    i_emags[i_emags > 5] = np.nan

    b_mags[b_mags == 0.0] = np.nan
    b_emags[b_emags == 0.0] = np.nan
    i_mags[i_mags == 0.0] = np.nan
    i_emags[i_emags == 0.0] = np.nan

    sharp = multiepoch_data[:,-1]

    sharp_repeat = np.zeros((len(sharp), len(b_mags[0])))
    for i in range(len(b_mags[0])):
        sharp_repeat[:,i] = sharp
    sharp_flat = sharp_repeat.flatten()

    low_sharp = np.abs(sharp) < 0.1
    low_sharp_flat = np.abs(sharp_flat) < 0.1

    dt = np.dtype([('filter', 'U5'), ('exptime', float)])
    nrows = 2
    nbins = 15

    error_key = np.zeros(nrows, dtype=dt)

    error_key['filter'][0] = 'F475W'
    error_key['filter'][1] = 'F814W'
    error_key['exptime'][0] = 1400
    error_key['exptime'][1] = 1200

    error_array = np.zeros((nrows,nbins,3))
    scatter_array = np.zeros((n_filters,nbins,3))

    b_mags_flat = b_mags.flatten()
    b_emags_flat = b_emags.flatten()
    i_mags_flat = i_mags.flatten()
    i_emags_flat = i_emags.flatten()

    notnan = ~np.isnan(b_mags_flat) & ~np.isnan(b_emags_flat)
    select = notnan & low_sharp_flat

    phot_unc, edges, _ = binned_statistic(b_mags_flat[select], b_emags_flat[select],
                    'median', bins=nbins)
    sig_phot_unc, edges2, _ = binned_statistic(b_mags_flat[select], b_emags_flat[select],
                    statistic=mad, bins=nbins)

    centers1 = (edges[:-1] + edges[1:])/2.0

    error_array[0,:,0] = centers1
    error_array[0,:,1] = phot_unc
    error_array[0,:,2] = sig_phot_unc

    fig, ax = plt.subplots(1,1)

    ax.scatter(b_mags_flat[select], b_emags_flat[select], s=0.1, color='k', alpha=0.1)
    ax.errorbar(centers1, phot_unc, yerr=sig_phot_unc, fmt='o', color='orange')
    #ax.set_ylim(0,0.4)
    ax.set_xlabel('{} mag'.format('F475W'))
    ax.set_ylabel('$\sigma_i$ {}'.format('F475W'))
    ax.text(0.1, 0.9, 'Exp = {} s'.format(1400), transform=ax.transAxes)

    plt.show()

    # repeat for i band
    notnan = ~np.isnan(i_mags_flat) & ~np.isnan(i_emags_flat)
    select = notnan & low_sharp_flat

    phot_unc, edges, _ = binned_statistic(i_mags_flat[select], i_emags_flat[select],
                    'median', bins=nbins)
    sig_phot_unc, edges2, _ = binned_statistic(i_mags_flat[select], i_emags_flat[select],
                    statistic=mad, bins=nbins)

    centers1 = (edges[:-1] + edges[1:])/2.0

    error_array[1,:,0] = centers1
    error_array[1,:,1] = phot_unc
    error_array[1,:,2] = sig_phot_unc

    fig, ax = plt.subplots(1,1)

    ax.scatter(i_mags_flat[select], i_emags_flat[select], s=0.1, color='k', alpha=0.1)
    ax.errorbar(centers1, phot_unc, yerr=sig_phot_unc, fmt='o', color='orange')
    #ax.set_ylim(0,0.4)
    ax.set_xlabel('{} mag'.format('F814W'))
    ax.set_ylabel('$\sigma_i$ {}'.format('F814W'))
    ax.text(0.1, 0.9, 'Exp = {} s'.format(1200), transform=ax.transAxes)

    plt.show()

    b_mean_mags = np.nanmean(b_mags, axis=1)
    b_stds = np.nanstd(b_mags, axis=1)
    i_mean_mags = np.nanmean(i_mags, axis=1)
    i_stds = np.nanstd(i_mags, axis=1)

    notnan = ~np.isnan(b_mean_mags) & ~np.isnan(b_stds)
    select = notnan & low_sharp
    scatter, edges, _ = binned_statistic(b_mean_mags[select], b_stds[select],
        'median', bins=nbins)
    sig_scatter, edges2, _ = binned_statistic(b_mean_mags[select], b_stds[select],
        statistic=mad, bins=nbins)

    centers2 = (edges[:-1] + edges[1:])/2.0

    scatter_array[0,:,0] = centers2
    scatter_array[0,:,1] = scatter
    scatter_array[0,:,2] = sig_scatter

    fig, ax = plt.subplots(1,1)

    ax.scatter(b_mean_mags[select], b_stds[select], s=0.1, color='k', alpha=0.1)
    ax.errorbar(centers2, scatter, yerr=sig_scatter, fmt='o', color='orange')
    #ax.set_ylim(0,0.4)
    ax.set_xlabel('{} mag'.format('F475W'))
    ax.set_ylabel('$\sigma$ {}'.format('F475W'))

    plt.show()

    notnan = ~np.isnan(i_mean_mags) & ~np.isnan(i_stds)
    select = notnan & low_sharp
    scatter, edges, _ = binned_statistic(i_mean_mags[select], i_stds[select],
        'median', bins=nbins)
    sig_scatter, edges2, _ = binned_statistic(i_mean_mags[select], i_stds[select],
        statistic=mad, bins=nbins)

    centers2 = (edges[:-1] + edges[1:])/2.0

    scatter_array[1,:,0] = centers2
    scatter_array[1,:,1] = scatter
    scatter_array[1,:,2] = sig_scatter

    fig, ax = plt.subplots(1,1)

    ax.scatter(i_mean_mags[select], i_stds[select], s=0.1, color='k', alpha=0.1)
    ax.errorbar(centers2, scatter, yerr=sig_scatter, fmt='o', color='orange')
    #ax.set_ylim(0,0.4)
    ax.set_xlabel('{} mag'.format('F814W'))
    ax.set_ylabel('$\sigma$ {}'.format('F814W'))

    plt.show()

    return error_key, error_array, scatter_array
_____no_output_____error_key, error_array, scatter_array = get_properties(14)_____no_output_____

The dataset property arrays are printed below for clarity on their format. _____no_output_____

print ('Error_key:')
print(error_key)
print('\n\nError Array:')
print(error_array)
print('\n\nScatter Array:')
print(scatter_array)Error_key:
[('F475W', 1400.) ('F814W', 1200.)]

Error Array:
[[[2.03523e+01 1.29000e-02 2.10000e-03]
  [2.11769e+01 1.64000e-02 2.75000e-03]
  [2.20015e+01 1.80000e-02 3.40000e-03]
  [2.28261e+01 2.04000e-02 4.80000e-03]
  [2.36507e+01 2.86000e-02 8.40000e-03]
  [2.44753e+01 3.81000e-02 1.06000e-02]
  [2.52999e+01 5.75000e-02 1.65000e-02]
  [2.61245e+01 9.22000e-02 2.39000e-02]
  [2.69491e+01 1.34000e-01 3.29000e-02]
  [2.77737e+01 2.65900e-01 6.68000e-02]
  [2.85983e+01 4.78900e-01 1.18800e-01]
  [2.94229e+01 8.94300e-01 2.20700e-01]
  [3.02475e+01 1.76505e+00 4.25750e-01]
  [3.10721e+01 3.29855e+00 6.34850e-01]
  [3.18967e+01 4.43720e+00 3.48400e-01]]

 [[1.81056e+01 5.29000e-02 1.25000e-02]
  [1.90068e+01 2.37500e-02 4.35000e-03]
  [1.99080e+01 2.10000e-02 4.40000e-03]
  [2.08092e+01 2.05000e-02 3.70000e-03]
  [2.17104e+01 2.21000e-02 4.40000e-03]
  [2.26116e+01 2.57000e-02 5.60000e-03]
  [2.35128e+01 3.46000e-02 8.00000e-03]
  [2.44140e+01 5.31000e-02 1.37000e-02]
  [2.53152e+01 8.61000e-02 1.98000e-02]
  [2.62164e+01 1.57300e-01 4.08000e-02]
  [2.71176e+01 3.13200e-01 8.00000e-02]
  [2.80188e+01 6.05800e-01 1.48800e-01]
  [2.89200e+01 1.23500e+00 3.03500e-01]
  [2.98212e+01 2.63065e+00 5.99050e-01]
  [3.07224e+01 4.24875e+00 4.37150e-01]]]

Scatter Array:
[[[2.03225762e+01 8.94724446e-03 9.48282828e-04]
  [2.10461571e+01 1.46663574e-02 3.64388504e-03]
  [2.17697381e+01 1.51071249e-02 4.92918147e-03]
  [2.24933190e+01 1.38905636e-02 2.33817054e-03]
  [2.32169000e+01 1.94350983e-02 5.71930998e-03]
  [2.39404810e+01 3.46707315e-02 1.40395571e-02]
  [2.46640619e+01 4.40305146e-02 1.72693830e-02]
  [2.53876429e+01 6.21588237e-02 2.40868159e-02]
  [2.61112238e+01 1.09027095e-01 5.01477193e-02]
  [2.68348048e+01 1.46269678e-01 5.80376377e-02]
  [2.75583857e+01 2.77306910e-01 1.12435160e-01]
  [2.82819667e+01 5.00657792e-01 1.99859134e-01]
  [2.90055476e+01 8.58249418e-01 2.90948858e-01]
  [2.97291286e+01 8.84401227e-01 2.15824762e-01]
  [3.04527095e+01 8.86500577e-01 1.46390129e-01]]

 [[1.82326467e+01 8.07221399e-02 5.57965617e-03]
  [1.90449400e+01 1.55138187e-02 2.08386781e-03]
  [1.98572333e+01 1.36985571e-02 3.24984638e-03]
  [2.06695267e+01 1.50142064e-02 6.21147567e-03]
  [2.14818200e+01 1.78735391e-02 4.71376270e-03]
  [2.22941133e+01 2.03856006e-02 5.72571903e-03]
  [2.31064067e+01 2.76472865e-02 8.91991917e-03]
  [2.39187000e+01 3.96961032e-02 1.36122030e-02]
  [2.47309933e+01 6.93484252e-02 2.71840240e-02]
  [2.55432867e+01 1.09113146e-01 4.25325210e-02]
  [2.63555800e+01 2.05196285e-01 7.81246153e-02]
  [2.71678733e+01 4.01706655e-01 1.52295461e-01]
  [2.79801667e+01 7.86768848e-01 2.76190957e-01]
  [2.87924600e+01 8.69461385e-01 2.03803843e-01]
  [2.96047533e+01 8.44939314e-01 1.36966073e-01]]]

## Generate Simulated star parameters

This step is independent of your dataset, and simply generates the basic parameters
of the simulated stars based on the distributions of variables in the LMC. It is
currently possible to create simulated first overtone/fundamental Classical Cepheids (CEP), first overtone/fundamental RR Lyrae stars (RRL),
first overtone/fundamental Anomalous Cepheids (AC), and BL Her/W Virginis Type II Cepheids (T2C)

The arguments you must give the function `simulate_params` are:
1. array of types of variables you want to generate. Pick from ['CEP', 'RRL', 'AC', 'T2C'].
2. the number of each type of variable/subtype you want to generate.
3. the local path to the OGLEIV LMC data
4. (optional) a seed for the random generators. In practice, you should use seed=0 to get random results, but for repeatability you may explicitly set a seed (as we have done in this notebook). _____no_output_____

# path to where you have downloaded ogle data
ogle4_dir = '/Users/jill/Research/Surveys/OGLE/OGLEIV/LMC/'

# generate parameters (period, amp, I mean mag) of simulated stars
sim_data = vs.simulate_params(['CEP', 'RRL', 'AC', 'T2C'], 125, ogle4_dir, seed=5)

print('Total number of simulated stars:', len(sim_data['id']))
print('sim_data array:')
print(sim_data[0:-1:50])_____no_output_____

## Generate simulated light curves

The following cell generates the simulated light curve files, which are specific to your dataset.

You must provide the function `make_simulated_lcv` the following arguments:
1. data array created above in `simulate_params`
2. array with MJDs of your observations. MJDs can be in any order, but they should correspond to the appropriate filter (next argument).
3. array of filter name of each observation.
4. arrray of exposure times of each observation.
5. approximate distance modulus of the target/variables you are analyzing
6. scatter_array from setup
7. error_key from setup
8. error_array from setup
9. output directory where you want your simulated light curves to be written

Optional keyword arguments:
1. append - Default False. Set to True if you want to create more simulated stars without overwriting previously created ones
2. period_search_range - The range in days over which you want to fit templates. Used in FITLC header. Default 0.1 - 3.0 days
3. rrl_cutoff - Approximate magnitude cutoff to differentiate RRL and Cepheids. Stars with mean mag below cutoff will have a shortened period_search_range of 0.1 - 1.0 days. Stars above this cutoff will use the period_search_range you set.
4. template_file - path/name of the file that has FITLC light curve templates
5. seed - If seed != 0 then a seed will be provided to the random number generators for repeatability. By default set to 0, so output will not be repeatable. _____no_output_____

output_dir = '../test_data/vv124/simulations/lcvs/'
distance = 25.8 # approximate distance modulus

filters1 = np.repeat('B', 14)
filters2 = np.repeat('I', 14)
filters = np.append(filters1, filters2)

# HST filters are not currently supported, so change names to closest ground based filters
error_key['filter'][error_key['filter'] == 'F475W'] = 'B'
error_key['filter'][error_key['filter'] == 'F814W'] = 'I'

mjds = np.array([58490.640099000, 58490.838859000, 58491.771357000, 58491.847075000, 58492.427560000,
                58492.504885000, 58492.824905000, 58492.906455000, 58495.803519000, 58495.885266000,
                58496.332749000, 58496.414009000, 58496.531369000, 58496.612676000, 58490.657217000,
                58490.855977000, 58491.782763000, 58491.823596000, 58492.444678000, 58492.487119000,
                58492.842023000, 58492.888689000, 58495.820637000, 58495.867500000, 58496.349867000,
                58496.396243000, 58496.548487000, 58496.594910000])

exptimes1 = np.repeat(1400, 14)
exptimes2 = np.repeat(1200, 14)
exptimes = np.append(exptimes1, exptimes2)

vs.make_simulated_lcv(sim_data, mjds, filters,
    exptimes, distance, scatter_array, error_key, error_array,
    output_dir, append=False, period_search_range=[0.3, 6.0], rrl_cutoff=25.3,
    template_file='../var_templates.txt', seed=5)_____no_output_____

## Visualizations

In the cells below we show a few visualizations to demonstrate the output of the simulations. First a few example light curves, then the period-luminosity relation and period-color relation of the simulated stars.
_____no_output_____

# Plot a couple lcvs of each type to be sure they look ok

# read parameter file
dt = np.dtype([('id', int), ('type', 'U3'), ('mode', 'U2'), ('template', int), ('period', float),
              ('t0', float), ('mag1', float), ('amp1', float), ('sig1', float), ('mag2', float),
              ('amp2', float), ('sig2', float)])
data = np.loadtxt('../test_data/vv124/simulations/lcvs/true_params.txt', dtype=dt)

# Get template light curves
template_file = '../var_templates.txt'
dt = np.dtype([('phase', float), ('ab1', float), ('ab2', float),
        ('ab3', float), ('ab4', float), ('ab5', float), ('ab6', float),
        ('c', float), ('ab7', float), ('ab8', float)])
templates = np.loadtxt(template_file, dtype=dt)
templates_new = np.c_[templates['ab1'], templates['ab2'], templates['ab3'],
        templates['ab4'], templates['ab5'], templates['ab6'], templates['c'],
        templates['ab7'], templates['ab8']]
templates_mean = np.mean(templates_new, axis=0)

for i in range(4):

    n = 125

    star_indices = [n*2*i, n*2*i+1, n+2*n*i, n+2*n*i+1]

    fig, ax = plt.subplots(1,4, figsize=(15,3))

    for j in range(len(star_indices)):
        xx = star_indices[j]
        t = data['template'][xx]
        mag_temp1 = (templates_new[:,t]-templates_mean[t])*data['amp1'][xx] + data['mag1'][xx]
        ax[j].plot(templates['phase'], mag_temp1, color='xkcd:steel blue')
        mag_temp2 = (templates_new[:,t]-templates_mean[t])*data['amp2'][xx] + data['mag2'][xx]
        ax[j].plot(templates['phase'], mag_temp2, color='xkcd:rose')
        ax[j].set_title('{} {}'.format(data['type'][xx], data['mode'][xx]))
        #ax[j].set_title(var_types[j])
        ax[j].invert_yaxis()

        # read in lcv data
        dt = np.dtype([('filter', int), ('mjd', float), ('mag', float), ('err', float)])
        lcv = np.loadtxt('../test_data/vv124/simulations/lcvs2/sim_{}.fitlc'.format(xx+1), dtype=dt, skiprows=3)
        ph = np.mod((lcv['mjd']-data['t0'][xx])/data['period'][xx], 1)
        f1 = lcv['filter'] == 1
        ax[j].errorbar(ph[f1], lcv['mag'][f1], yerr=lcv['err'][f1], fmt='o', color='k')
        f2 = lcv['filter'] == 0
        ax[j].errorbar(ph[f2], lcv['mag'][f2], yerr=lcv['err'][f2], fmt='o', color='gray')

        ax[j].text(0.05, 0.15, 'amp={:.2f}'.format(data['amp1'][xx]), transform=ax[j].transAxes)
        ax[j].text(0.05, 0.05, '$\sigma=${:.2f}'.format(data['sig1'][xx]), transform=ax[j].transAxes)

    plt.show()_____no_output_____# Plot PL relation of simulated stars
fig, ax = plt.subplots(1,2, figsize=(12,5))

cep = data['type'] == 'CEP'
rrl = data['type'] == 'RRL'
t2c = data['type'] == 'T2C'
ac = data['type'] == 'AC'

ax[0].scatter(np.log10(data['period'][cep]), data['mag2'][cep], s=5)
ax[0].scatter(np.log10(data['period'][rrl]), data['mag2'][rrl], s=5)
ax[0].scatter(np.log10(data['period'][t2c]), data['mag2'][t2c], s=5)
ax[0].scatter(np.log10(data['period'][ac]), data['mag2'][ac], s=5)

ax[0].invert_yaxis()
ax[0].set_xlabel('logP')
ax[0].set_ylabel('I mag')

ax[1].scatter(np.log10(data['period'][cep]), data['mag1'][cep] - data['mag2'][cep], s=5)
ax[1].scatter(np.log10(data['period'][rrl]), data['mag1'][rrl] - data['mag2'][rrl], s=5)
ax[1].scatter(np.log10(data['period'][t2c]), data['mag1'][t2c] - data['mag2'][t2c], s=5)
ax[1].scatter(np.log10(data['period'][ac]), data['mag1'][ac] - data['mag2'][ac], s=5)

ax[1].set_xlabel('logP')
ax[1].set_ylabel('B-I')
_____no_output_____

## Wrap Up

After generating these simulated stars, run them through the period fitting program FITLC (or an equivalent program if you have modified the output light curves appropriately). Create a file called `sim_results.txt` that contains both the simulated and measured parameters for each star. If you are using the same file formats/directory structure as test_data, you can use the function `compile_sims` in `variable_simulations.py` to create the file `sim_results.txt`. Otherwise, copy the format of the file in the test_data foler. The last four columns of this file should be:
- variability flag - 1 if the simulated star would pass the variability identification criteria you set for your data (e.g. a variability index) , 0 otherwise.
- visual flag - by default -1. This column will get filled in as you go through classification
- variable type - by default XXX. This column will get filled in as you go through classification
- variable subtype/mode - by default XX. This column will get filled in as you go through classification

To classify the simulated stars, run command
`python variable_classification.py 'simulations'`_____no_output_____


# Notebook from benvictoria17/Machinelearning
Path: Modeling Earthquake Damage - Ensemble Learning.ipynb

<code>
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import emoji
import folium

from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
import cufflinks as cf
import plotly.graph_objects as go
import plotly.express as px
%matplotlib inline
sns.set_style('darkgrid')/opt/conda/lib/python3.7/site-packages/geopandas/_compat.py:115: UserWarning:

The Shapely GEOS version (3.9.1-CAPI-1.14.2) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.

data = pd.read_csv("https://raw.githubusercontent.com/benvictoria17/DataAnalytics/master/dataset/Earthquakes%20in%20Greece%20(1901-2018)/Major%20EarthQuakes%20in%20Greece.csv")_____no_output_____data.head()_____no_output_____data.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1173 entries, 0 to 1172
Data columns (total 8 columns):
 #   Column               Non-Null Count  Dtype
---  ------               --------------  -----
 0   Year                 1173 non-null   int64
 1   Month                1173 non-null   int64
 2   Date                 1173 non-null   int64
 3   Hours                1173 non-null   int64
 4   Minutes              1173 non-null   int64
 5   LATATITUDE (N)       1173 non-null   float64
 6   LONGITUDE  (E)       1173 non-null   float64
 7   MAGNITUDE (Richter)  1173 non-null   float64
dtypes: float64(3), int64(5)
memory usage: 73.4 KB
data.describe()_____no_output_____data.columns_____no_output_____columns = ['Year', 'Month','Day','Hour', 'Minute', 'Latitude(N)','Longitude(E)','Magnitude(R)']
data.columns = columns_____no_output_____data.head()_____no_output_____data_0 = data[data['Magnitude(R)'] >= 1.0]
annual_nbr = data_0.groupby(by = 'Year').size()
print(annual_nbr)Year
1901     3
1902     4
1903     7
1904    12
1905    13
        ..
2013     8
2014    13
2015     9
2016     8
2017    14
Length: 117, dtype: int64
init_notebook_mode(connected = True)
cf.go_offline()
fig = go.Figure(data=[go.Table(header=dict(values=['Year', 'Number of Earthquakes']),
                 cells=dict(values=[annual_nbr.index, annual_nbr]))
                     ])
fig.update_layout(
    title="Number of Earthquakes, in Greece, per Year (1901 - 2018).")
fig.show()_____no_output_____annual_nbr.iplot(kind = 'bar', theme = 'solar',
                 title = 'Number of Earhtquakes, in Greece, per year (1901 - 2018)',
                 xTitle = 'Year', yTitle = 'Number of earthquakes')_____no_output_____data_0['Magnitude(R)'].iplot(kind = 'histogram', theme = 'solar',
                             title = 'Frequency per level of reported earthquake',
                             xTitle = 'Magnitute (R)',
                             yTitle = "Number of earthquakes reported")_____no_output_____severe_earth = data_0[data_0['Magnitude(R)'] >= 7.1]
severe_earth_0 = severe_earth.sort_values(by='Magnitude(R)', ascending = False)
severe_earth_0.head()_____no_output_____my_map = folium.Map(location=[severe_earth['Latitude(N)'].mean(),
                    severe_earth['Longitude(E)'].mean()], zoom_start = 6,
                    control_scale = True)

folium.Marker(
    location=[36.30, 23.00],
    popup="Monday, 11 August 1903 (04:32) ~ 8.0 R",
    icon=folium.Icon(icon="glyphicon glyphicon-star-empty", color= 'red')
).add_to(my_map)

folium.Marker(
    location=[41.8, 23.1],
    popup="Sunday, 4 April 1904 (10:25) ~ 7.8 R",
    icon=folium.Icon(icon="glyphicon glyphicon-star-empty", color= 'red')
).add_to(my_map)

folium.Marker(
    location=[40.3, 24.4],
    popup="Tuesday, 8 November 1905 (22:06) ~ 7.4 R",
    icon=folium.Icon(icon="glyphicon glyphicon-star-empty", color= 'red')
).add_to(my_map)

folium.Marker(
    location=[40.20, 27.52],
    popup="Wednesday, 18 March 1953 (19:06) ~ 7.4 R",
    icon=folium.Icon(icon="glyphicon glyphicon-star-empty", color= 'red')
).add_to(my_map)

folium.Marker(
    location=[36.64, 25.91],
    popup="Monday, 9 July 1956 (03:11) ~ 7.4 R",
    icon=folium.Icon(icon="glyphicon glyphicon-star-empty", color= 'red')
).add_to(my_map)

my_map_____no_output_____my_map_2 = folium.Map(location=[39.14, 42.65], zoom_start = 4,
                    control_scale = True)

folium.Marker(
    location=[39.14, 42.65],
    popup="Monday, 28 April 1903 (01:30) ~ 8.0 R",
    icon=folium.Icon(icon="glyphicon glyphicon-star-empty", color= 'red')
).add_to(my_map_2)

my_map_2_____no_output_____df = px.data.iris()

fig = px.scatter(data_0, x = 'Month', y = 'Year', color = 'Magnitude(R)', title = 'Magnitude of the reported earthquakes, by Month and by Year (1901 - 2018)' )

fig.show()_____no_output_____
</code>


# Notebook from fhalab/SPGen
Path: remote_generation/signal_peptide/code/1a_analysis.ipynb

Checking to see if protein sequences Zach sent in excel file called "initial_enzymes_1" are in the training set. If so, remove them from the training set.

First, align sequences using biopython module, then compute percentage of similarity

Reference: https://www.biostars.org/p/208540/_____no_output_____

<code>
%matplotlib inline
import pickle
import random
from collections import Counter

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('whitegrid')

import csv

from Bio import pairwise2 as pw2_____no_output_____# load in prot sequences from dataset
df = pd.read_excel('../dataset.xls')
dataset = df['Prot Sequences'].values

# load in prot sequences from Zach's excel
df = pd.read_excel('initial_enzymes_1.xlsx')
prot_excel = df['Protein -met -sigp'].values_____no_output_____len(dataset)_____no_output_____def write_to_csv(filename, query_prot, dataset_prot, sim):
    """ Function that writes out query sequence, dataset sequence, and similarity to csv file. """
    lst = [[query_prot], [dataset_prot], [sim]]
    with open(filename, 'a') as myfile:
        for row in lst:
            for col in row:
                myfile.write(str(col))
            myfile.write('\n')_____no_output_____# Create csv files for each query protein in Zach's excel with the similarity to each protein in dataset--after alignment

for i, prot in enumerate(prot_excel[20:21]):
    filename = "Query protein " + str(i + 21) + ".csv"
    for j, dat in enumerate(dataset):
        global_align = pw2.align.globalxx(prot, dat) # align sequences
        seq_length = min(len(prot), len(dat))
        matches = global_align[0][2]

        percent_match = (matches / seq_length) * 100
        write_to_csv(filename, prot, dat, percent_match)_____no_output_____sim = [] # a list of lists of similarities of each protein sequence in the dataset to each query protein

for i, prot in enumerate(prot_excel): # read each csv made for each query protein
    filename = "Query+protein+" + str(i + 1) + ".csv"
    with open(filename, "r") as f:
        reader = csv.reader(f, delimiter="\t")
        index = 0 # for appending to lists of similarities within the list "sim"
        for j, line in enumerate(reader): # reads in each row of the csv
            if i == 0: # make initial lists of similarities of each protein in dataset
                if (j + 1) % 3 == 0:
                    sim.append([float(line[0])])
            else: # append to lists of similarities of each protein in dataset
                if (j + 1) % 3 == 0:
                    sim[index].append(float(line[0]))
                    index += 1_____no_output_____indices_50 = [] # stores indices of protein sequences in dataset at least 50% similar
indices_75 = [] # stores indices of protein sequences in dataset at least 75% similar
indices_90 = [] # stores indices of protein sequences in dataset at least 90% similar
indices_95 = [] # stores indices of protein sequences in dataset at least 95% similar
indices_99 = [] # stores indices of protein sequences in dataset at least 99% similar

for i, dat in enumerate(dataset):
    if max(sim[i]) >= 50:
        indices_50.append(i)
    if max(sim[i]) >= 75:
        indices_75.append(i)
    if max(sim[i]) >= 90:
        indices_90.append(i)
    if max(sim[i]) >= 95:
        indices_95.append(i)
    if max(sim[i]) >= 99:
        indices_99.append(i)

dataset_50 = np.delete(dataset, indices_50) # remove protein sequences at least 50% similar
dataset_75 = np.delete(dataset, indices_75) # remove protein sequences at least 75% similar
dataset_90 = np.delete(dataset, indices_90) # remove protein sequences at least 90% similar
dataset_95 = np.delete(dataset, indices_95) # remove protein sequences at least 95% similar
dataset_99 = np.delete(dataset, indices_99) # remove protein sequences at least 99% similar_____no_output_____print(len(dataset_75))
print(len(dataset_90))
print(len(dataset_95))
print(len(dataset_99))22120
27687
28218
28546
# write new datasets out to csv

def write_prot_to_csv(filename, prot):
    """ Function that writes protein sequence out to one row in csv """
    with open(filename, 'a', newline='') as myfile:
        wr = csv.writer(myfile)
        wr.writerow([str(prot)])

for i, dataset in enumerate([dataset_50, dataset_75, dataset_90, dataset_95, dataset_99]):
    filename = "dataset_" + str(i + 1) + ".csv"
    for j, prot in enumerate(dataset):
        write_prot_to_csv(filename, prot)_____no_output_____len(dataset)_____no_output_____# function that writes out to csv - similarity
# take out of instance
# c5.xlarge
# top for multiple notebooks, new file for each query (41 files total)_____no_output_____
</code>


# Notebook from ventolab/HGDA
Path: scATACseq_females_0.ipynb

## ATAC + MULTIOME (females)

### Build anndata from cellatac output _____no_output_____

# Load libraries
import pandas as pd
import scanpy as sc
import numpy as np
import scipy.sparse/opt/conda/envs/atac_env/lib/python3.8/site-packages/joblib/_multiprocessing_helpers.py:45: UserWarning: [Errno 28] No space left on device.  joblib will operate in serial mode
  warnings.warn('%s.  joblib will operate in serial mode' % (e,))
# Define variables
outdir = "/nfs/team292/vl6/my_MULTIOME_dir/females_apr2021/"
experiment_prefix = 'females_'
cellatac_outdir = '/lustre/scratch117/cellgen/cellgeni/TIC-atacseq/tic-1004/female/results200k-b-female/'
_____no_output_____input={'cnts': cellatac_outdir + 'peak_matrix/peaks_bc_matrix.mmtx.gz',
        "bcs": cellatac_outdir + 'peak_matrix/bc.txt',
      "peaks": cellatac_outdir + 'peak_matrix/peaks.txt',
      'clusters': cellatac_outdir + 'qc/seurat-clades.tsv'}

output = {
    "cnt_mmtx":outdir + experiment_prefix + '_ATAC_cisTopic.mmtx',
    "cnt_peaks":outdir + experiment_prefix + '_ATAC_cisTopic.peaks.tsv',
    "cnt_cells":outdir + experiment_prefix + '_ATAC_cisTopic.cells.tsv',
    "h5ad":outdir + experiment_prefix + '_ATAC_raw.h5ad',
}_____no_output_____

#### Make anndata object_____no_output_____

adata = sc.read_mtx(input["cnts"]).T

bc = pd.read_table(input["bcs"], header=None)
feat = pd.read_table(input["peaks"], header=None)

adata.obs_names = bc[0]
adata.var_names = feat[0]_____no_output_____adata_____no_output_____# Code from cell barcodes
adata.obs['code'] = [name[0:2] for name in adata.obs_names]
adata.obs['code'].value_counts(dropna = False)_____no_output_____# Sample from code
sample_dictionary = {'01' : 'FCA_GND8046539', '02' : 'FCA_GND8768483', '03' : 'FCA_GND8768484',
                    '04' : 'FCA_GND8768485', '05' : 'HD_F_GON9479871', '06' : 'HD_F_GON9479873',
                     '07' : 'HD_F_GON9479874', '08' : 'HD_F_GON9479875', '09' : 'HD_F_GON9525613',
                     '10' : 'HD_F_GON9525614', '11' : 'HD_F_GON9883959', '12' : 'HD_F_GON9883960'}
adata.obs['sample'] = adata.obs['code'].map(sample_dictionary)
adata.obs['sample'].value_counts(dropna = False)_____no_output_____

#### Load peak annotations (done with R script by Emma)_____no_output_____

peak_anno_df = pd.read_csv(outdir + "ATACpeaks_annotation.csv", index_col=0)
peak_anno_df.index = peak_anno_df["peak_id"]
peak_anno_df.drop("peak_id",1, inplace=True)
adata.var = pd.concat([adata.var, peak_anno_df], 1)_____no_output_____

#### Save binary data to layers_____no_output_____

adata.layers["binary_raw"] = adata.X
adata.layers["binary_raw"][adata.layers["binary_raw"] > 1] = 1
adata_____no_output_____

**Peak filtering**_____no_output_____

adata.var.hist(column = 'peak_width', bins = 200, grid = False, figsize = (25,6), color = '#870052')_____no_output_____var_qc = sc.pp.calculate_qc_metrics(adata, layer = "binary_raw")[1]
adata.var = pd.concat([adata.var, var_qc], 1)
adata/opt/conda/envs/atac_env/lib/python3.8/site-packages/numba/np/ufunc/parallel.py:316: UserWarning: Could not obtain multiprocessing lock due to OS level error: [Errno 28] No space left on device
A likely cause of this problem is '/dev/shm' is missing orread-only such that necessary semaphores cannot be written.
*** The responsibility of ensuring multiprocessing safe access to this initialization sequence/module import is deferred to the user! ***

  warnings.warn(msg % str(e))
adata.var.head()_____no_output_____adata.var.hist(column = 'total_counts', bins = 200, grid = False, figsize = (25,6), color = '#870052')_____no_output_____thirty_percent = len(adata.obs_names) / 100 * 30
point_one_percent = len(adata.obs_names) / 100 * 0.1
print("30% : {}".format(thirty_percent))
print("0.1% : {}".format(point_one_percent))30% : 22385.7
0.1% : 74.61900000000001
# Accessible in at least k cells
adata = adata[:,adata.var.total_counts > point_one_percent]
adata = adata[:,adata.var.total_counts < thirty_percent]
adata_____no_output_____# Remove peaks in ENCODE blacklist
adata = adata[:, adata.var.ENCODE_blacklist == 0]
adata_____no_output_____# Filter by width (remove peaks at the lowest end, closest to min peak width in MACS2)

adata = adata[:, adata.var.peak_width > 210]
adata = adata[:, adata.var.peak_width < 1500]
adata_____no_output_____adata.var.head()_____no_output_____adata.var['annotation'].value_counts()_____no_output_____adata.obs.head()_____no_output_____### Filter peaks that are not accessible in at least 4% of cells from a coarse cluster
min_frac=0.04
# Load cluster information from cellatac outputs
clusters = pd.read_table(input["clusters"], header = None, index_col = 0)
clusters.columns = ["cellatac_clusters"]
adata.obs = clusters.loc[adata.obs_names]
adata_____no_output_____adata.obs.head()_____no_output_____n_clusters = len(np.unique(adata.obs[["cellatac_clusters"]]))
clus_mat = np.empty([adata.n_obs, n_clusters])
for cl in np.unique(adata.obs[["cellatac_clusters"]]):
    clus_mat[np.where(adata.obs['cellatac_clusters']==cl)[0],cl] = 1

clus_mat = scipy.sparse.csr_matrix(clus_mat)
clus_mat[clus_mat != 1 ] = 0/opt/conda/envs/atac_env/lib/python3.8/site-packages/IPython/core/interactiveshell.py:3427: SparseEfficiencyWarning: Comparing a sparse matrix with a nonzero scalar using != is inefficient, try using == instead.
  exec(code_obj, self.user_global_ns, self.user_ns)
/opt/conda/envs/atac_env/lib/python3.8/site-packages/scipy/sparse/_index.py:125: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
  self._set_arrayXarray(i, j, x)
cl_peak_mat = np.dot(clus_mat.T, adata.layers["binary_raw"])_____no_output_____cl_peak_frac = cl_peak_mat/clus_mat.sum(0).T_____no_output_____cl_peak_frac.max(0).shape_____no_output_____bool_matrix = cl_peak_frac.max(0) > min_frac
bool_matrix.shape_____no_output_____bool_vector = np.squeeze(np.asarray(bool_matrix))
bool_vector.shape_____no_output_____adata = adata[:, bool_vector]_____no_output_____adata_____no_output_____sc.pp.calculate_qc_metrics(adata, layer = "binary_raw", inplace = True)Trying to set attribute `.obs` of view, copying.
adata.var.hist(column = 'log1p_total_counts', bins = 200, grid = False, figsize = (25,6), color = '#870052')_____no_output_____adata.obs.hist(column = 'log1p_total_counts', bins = 200, grid = False, figsize = (25,6), color = '#870052')_____no_output_____adata = adata[adata.obs.log1p_total_counts >= 5.5]
adata_____no_output_____## Write output anndata
adata.write_h5ad(output["h5ad"])/opt/conda/envs/atac_env/lib/python3.8/site-packages/anndata/_core/anndata.py:1207: ImplicitModificationWarning: Initializing view as actual.
  warnings.warn(
Trying to set attribute `.var` of view, copying.
... storing 'annotation' as categorical
Trying to set attribute `.var` of view, copying.
... storing 'gene_name' as categorical
Trying to set attribute `.var` of view, copying.
... storing 'gene_id' as categorical

#### End of notebook_____no_output_____


# Notebook from joewandy/omics_integration
Path: graphomics/notebooks/pipeline/deseq2_rpy2.ipynb

# deseq2 pipeline in rpy2_____no_output_____

%load_ext autoreload
%autoreload 2_____no_output_____import os
import uuid
import pylab as plt
import matplotlib
from IPython.display import display, HTML, Image

import numpy as np
import pandas as pd

%matplotlib inline_____no_output_____

Load test data_____no_output_____

count_data = pd.read_pickle('../static/data/debugging/data_df.p')
count_data = count_data.drop('gene_id', axis=1)
count_data.head()_____no_output_____col_data = pd.read_pickle('../static/data/debugging/design_df.p')
col_data_____no_output_____

Run DESeq2_____no_output_____

from rpy2.robjects.packages import importr
from rpy2 import robjects
from rpy2.robjects import Formula
from rpy2.robjects import pandas2ri
pandas2ri.activate()

deseq = importr('DESeq2')
grdevices = importr('grDevices')_____no_output_____def to_pd_df(r_df):
    pd_df = pandas2ri.ri2py_dataframe(r_df)
    pd_df.index = r_df.rownames
    return pd_df_____no_output_____def run_deseq(count_data, col_data, keep_threshold, case, control):
    design = Formula("~ group")
    dds = deseq.DESeqDataSetFromMatrix(countData=count_data, colData=col_data, design=design)
    sv = robjects.StrVector(col_data['group'].values)
    condition = robjects.FactorVector(sv)
    runs = col_data.index
    rstring = """
        function(dds, condition, runs, keepThreshold, case, control) {
            # collapse technical replicates
            dds$condition <- condition
            dds$condition <- relevel(dds$condition, ref=control) # set control
            dds$sample <- runs
            dds$run <- runs
            ddsColl <- collapseReplicates(dds, dds$sample, dds$run)
            # count filter
            keep <- rowSums(counts(ddsColl)) >= keepThreshold
            ddsColl <- ddsColl[keep,]
            # run DESeq2 analysis
            ddsAnalysis <- DESeq(dds)
            res <- results(ddsAnalysis, contrast=c("group", control, case))
            resOrdered <- res[order(res$padj),]  # sort by p-adjusted values
            df = as.data.frame(resOrdered)
            rld <- as.data.frame(assay(rlog(dds, blind=FALSE)))
            list(df, rld, resOrdered)
        }
    """
    rfunc = robjects.r(rstring)
    results = rfunc(dds, condition, runs, keep_threshold, case, control)
    pd_df = to_pd_df(results[0])
    rld_df = to_pd_df(results[1])
    res_ordered = results[2]
    return pd_df, rld_df, res_ordered_____no_output_____pd_df, rld_df, res_ordered = run_deseq(count_data, col_data, 10, 'HK', 'UN')c:\users\joewa\.virtualenvs\web_omics-mv1eiydh\lib\site-packages\rpy2\robjects\pandas2ri.py:191: FutureWarning: from_items is deprecated. Please use DataFrame.from_dict(dict(items), ...) instead. DataFrame.from_dict(OrderedDict(items)) may be used to preserve the key order.
  res = PandasDataFrame.from_items(items)
pd_df.head(20)_____no_output_____

### Make some plots_____no_output_____

def plot_notebook(rfunc, res):
    fn = '{uuid}.png'.format(uuid = uuid.uuid4())
    grdevices.png(fn)
    rfunc(res)
    grdevices.dev_off()
    return Image(filename=fn)_____no_output_____rfunc = robjects.r('function(res) { plotMA(res, ylim=c(-2,2)) }')
plot_notebook(rfunc, res_ordered)_____no_output_____import pandas as pd
import numpy as np
from sklearn.decomposition import PCA_____no_output_____df = rld_df.transpose()_____no_output_____n_components = 10
pca = PCA(n_components=n_components)
X = pca.fit_transform(df)_____no_output_____np.cumsum(pca.explained_variance_ratio_)_____no_output_____X.shape_____no_output_____fig, ax = plt.subplots()
ax.scatter(X[:, 0], X[:, 1])

for i, txt in enumerate(df.index):
    ax.annotate(txt, (X[i, 0], X[i, 1]))

plt.tight_layout()_____no_output_____