# Notebook from chanzuckerberg/cellxgene-vignettes
Path: workflows/scanpy-workflow.ipynb

# Overview
## Purpose
This notebook provides a walkthrough of a "vanilla" analysis of scRNAseq data in scanpy. I've outlined some of the most general questions that a biologist may want to ask of the dataset, with comments for biological context and demo visualizations for each. This is intended to outline the "paths" a biologist may want to take through the cellxgene app, and scope out the most important questions for the tool to be able to address easily.

Importantly, this analysis is replicated [here](https://docs.google.com/document/d/1nJIACHHs3PKB9VJ3T4yia4iqNQOq7yrE-M1uezc8_PU/edit) using cellxgene for as many steps as possible.

## Analytical steps
For each dataset, a standard analytical workflow may look like:
1. Characterize the shape of the raw data
2. Normalize the data
3. Look at the distribution of the dataset overall. For high-dimensional datasets, this includes dimensionality reduction.
4. Look at rows (cells): are there clusters? How defined are they? Do they correspond to "ground truth" characterizations?
5. Look at columns (genes): are there covarying sets of genes? What do they do? Do these groups "make sense" biologically?
6. Look at the relationship between rows (cells) and columns (genes): for a given cluster of cells, what genes make their expression profiles different from other cells? For a given gene, which cells express that gene? _____no_output_____

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import matplotlib.cm as cm
from matplotlib.colors import ListedColormap, LinearSegmentedColormap, to_hex, Normalize
%matplotlib inline
import scanpy.api as sc
import math
import random
from scipy import stats
from timeit import timeit
from itertools import combinations

sns.set(style='whitegrid', palette='Set2', font_scale=1.3)_____no_output_____colors = ["#511EA8", "#482BB6", "#4039C3", "#3F4ACA", "#3E5CD0", "#416CCE", "#447CCD", "#4989C4", "#4E96BC", "#559FB0", "#5DA8A4", "#66AE96", "#6FB388", "#7AB77C", "#85BA6F", "#91BC64", "#9DBE5A", "#AABD53", "#B6BD4B", "#C2BA46", "#CDB642", "#D6B03F", "#DDA83C", "#E29D39", "#E69036", "#E67F33", "#E56D30", "#E2592C", "#DF4428", "#DC2F24"]
cmap = LinearSegmentedColormap.from_list('cmap', colors, N=len(colors))
palette = sns.color_palette(colors)
sns.set(style='whitegrid', font_scale=1.2)

sns.palplot(palette)

def assign_colors(vals, cmap=cmap):
    ''' return a dictionary of { value: (R,G,B,a) } '''
    if type(cmap) == str:
        cmap = cm.get_cmap(cmap)
    vals = set(vals)
    nvals = len(vals)

    colors = [cmap(i/nvals) for i in range(nvals)]

    return { val: colors[i] for i,val in enumerate(vals) }_____no_output_____

## 1. Load & add default annotations to the raw data  _____no_output_____

rawdata = sc.read('../data/Marrow-counts.csv').transpose()
cell_annotation = pd.read_csv('../data/annotations_FACS.csv', index_col='cell')
rawdata.obs['cell_type'] = cell_annotation.cell_ontology_class/Users/sidneybell/Library/Python/3.7/lib/python/site-packages/IPython/core/interactiveshell.py:3020: DtypeWarning: Columns (0,1) have mixed types. Specify dtype option on import or set low_memory=False.
  interactivity=interactivity, compiler=compiler, result=result)

## 2. Normalize the data: filter to informative cells & genes, center & scale data_____no_output_____

%time
'''
Their 'seurat' recipe for count normalization includes:
    pp.filter_cells(adata, min_genes=200) # drop cells with fewer than 200 genes expressed
    pp.filter_genes(adata, min_cells=3) # remove genes expressed in fewer than 3 cells
    pp.normalize_per_cell(adata, counts_per_cell_after=1e4) # each cell sums to 1
    filter_result = pp.filter_genes_dispersion(
        adata.X, min_mean=0.0125, max_mean=3, min_disp=0.5, log=not log) # remove uninformative genes
    adata._inplace_subset_var(filter_result.gene_subset)
    if log: pp.log1p(adata) # log scale
    pp.scale(adata, max_value=10) # center + scale
'''

adata = sc.pp.recipe_seurat(rawdata, copy=True, log=False)CPU times: user 3 µs, sys: 0 ns, total: 3 µs
Wall time: 7.15 µs

## 2.5 Preprocess the data: compute PCs, build neighbor graph, and infer clusters_____no_output_____

## preprocess the data (this is the major timesuck of the whole operation)
%time
sc.pp.pca(adata)
%time
sc.pp.neighbors(adata)

%time
sc.tl.louvain(adata)CPU times: user 4 µs, sys: 2 µs, total: 6 µs
Wall time: 11 µs
CPU times: user 25 µs, sys: 3 µs, total: 28 µs
Wall time: 7.87 µs
CPU times: user 3 µs, sys: 0 ns, total: 3 µs
Wall time: 34.1 µs

## This brings us to the point where cellxgene intercepts the user.

## 3. Look at the distribution of the dataset overall  _____no_output_____

df = pd.DataFrame(adata.X, index=adata.obs_names, columns=adata.var_names)

nonzero_finite = df.mask((df.values > 0) & (np.isfinite(df.values)), other=1) # anything with finite value > 0 becomes 1
zero_nonfinite = df.mask((df.values < 1) | (~np.isfinite(df.values)), other=1) # any value < 1 or with non-finite value becomes 1

fig, ax = plt.subplots(ncols=3,nrows=1, figsize=(16,4))

reads_per_cell = df.sum(axis=1)
plt.sca(ax[0])
plt.hist(reads_per_cell.values, bins=20)
ax[0].set_title('Reads per cell:\nraw data')
ax[0].set_xlabel('N cells')
ax[0].set_ylabel('N reads')

genes_per_cell = nonzero_finite.sum(axis=1)
plt.sca(ax[1])
plt.hist(genes_per_cell.values, bins=20)
ax[1].set_title('Genes expressed per cell:\nraw data')
ax[1].set_xlabel('N cells')
ax[1].set_ylabel('N genes')

pCells_per_gene = nonzero_finite.sum(axis=0)/len(nonzero_finite.index.values)
plt.sca(ax[2])
plt.hist(pCells_per_gene.values, bins=30)
ax[2].set_title('Proportion of cells\nexpressing each gene:\nraw data')
ax[2].set_ylabel('N genes')
ax[2].set_xlabel('Proportion of cells expressing')

plt.tight_layout()
plt.show()_____no_output_____

## 4. Look at rows (cells)
### Are there clusters? How defined are they? Do they correspond to "ground truth" characterizations?_____no_output_____

ax = sns.countplot(adata.obs['louvain'], order = adata.obs['louvain'].value_counts().index, color = cmap(.2))
ax.set_xlabel('Cluster ID')
ax.set_title('Cluster assignments in original data')
plt.show()
plt.clf()_____no_output_____def plot_tsne(adata, colorby, title=None, cmap=cmap):
    ''' Plot tSNE colored by metadata field indicated in colorby.
        Expects to find tSNE coordinates in adata.obsm['X_tsne']
        and metadata in adata.obs[colorby]'''

    sns.set(style='whitegrid', font_scale=1.2)
    fig, ax = plt.subplots()
    x = adata.obsm['X_tsne'][:,0]
    y = adata.obsm['X_tsne'][:,1]

    if colorby:
        if colorby in adata.obs:
            color_map = assign_colors(pd.unique(adata.obs[colorby]), cmap)
            colors = [color_map[x] if x in color_map else (.211,.211,.211,0.1) for x in adata.obs[colorby].values ]
            legend_handles = []
            for val, color in color_map.items():
                patch = mpatches.Patch(color=color, label=val)
                legend_handles.append(patch)
            plt.legend(loc=(1,.3),handles=legend_handles)

        else:
            gene_idx = adata.var_names.tolist().index(colorby)
            gene_expr_vals = adata.X[:,gene_idx]
            colors = [ cm.viridis_r(i / max(gene_expr_vals)) for i in gene_expr_vals ]
            norm = Normalize(vmin=min(gene_expr_vals),vmax=max(gene_expr_vals))
            sm = cm.ScalarMappable(cmap='viridis_r', norm=norm)
            sm.set_array([])
            plt.colorbar(sm)
    else:
        colors = ['lightgray' for x in adata.obs_names.values]
    ax.scatter(x, y, c=colors, alpha=0.3)
    ax.set_xlabel('tSNE 1')
    ax.set_ylabel('tSNE 2')

    if title:
        ax.set_title(title)
    plt.show()

sc.tl.tsne(adata, perplexity=len(pd.unique(adata.obs['cell_type'])),learning_rate=1000)
plot_tsne(adata, colorby='cell_type', title='Cell Type', cmap=cmap)
plot_tsne(adata, colorby='louvain', title='Cluster', cmap=cmap)_____no_output_____# def calc_mutual_information(clusters, classes):
#     assert clusters.index.tolist() == classes.index.tolist()
#     I = 0. # mutual information, summed over all clusters + classes
#     N = len(clusters) # N observations

#     K = pd.unique(clusters.values) # cluster bins
#     J = pd.unique(classes.values) # class bins (e.g., cell types)
#     for (k, j) in list(combinations(K,J,2)):
#         k_cells = set(clusters.loc[clusters.values == k].index) # cells in k bin
#         j_cells = set(classes.loc[classes.values==j].index) # cells in j bin

#         p_kj = len(k_cells.intersection(j_cells)) / N # rel freq of a cell being in both class k and cluster j
#         p_k = len(k_cells) / N # rel freq of a cell being in k
#         p_j = len(j_cells) / N # rel freq of a cell being in j

#         I_kj = p_kj * np.log(p_kj / (p_k*p_j)) # mutual information between cluster k and class j
#         I += I_kj # add to running total
#     return I

# def calc_entropy(labels):
#     H = 0.
#     N = len(labels)

#     for k in labels:
#         k_cells = set(labels.loc[labels.values == k].index)
#         p_k = len(k_cells) / N
#         H_k = -1.*p_k*np.log(p_k)
#         H += H_k
#     return H

# def calc_normalized_mutual_information(clusters, classes):
#     mutual_information = calc_mutual_information(clusters, classes)
#     entropy_norm = (calc_entropy(clusters) + calc_entropy(classes)) / 2
#     return mutual_information / entropy_norm

# def calc_cluster_purity(cluster_val, clusters, classes):
#     assert clusters.index.tolist() == classes.index.tolist()

#     cluster = clusters.loc[clusters.values == cluster_val]
#     cluster_idx = set(cluster.index.values)

#     class_distrib = pd.value_counts(classes[cluster_idx])
#     most_common_class = class_distrib.index[0]

#     p_k =

# clusters = demo.obs['louvain']
# classes = demo.obs['cell_type']
# calc_cluster_purity('0', clusters, classes)_____no_output_____

## 5. Look at columns (genes)
### Are there covarying sets of genes? What do they do? Do these groups "make sense" biologically?

#### [Add this later]_____no_output_____### 6. Look at the relationship between columns (genes) and rows (cells)
#### Which genes are differentially expressed in a given cell cluster?
#### Which cells express my-favorite-gene?

#### Scanpy does not provide a method for computing differential expression.
Ambrose recommends [MAST](https://bioconductor.org/packages/release/bioc/html/MAST.html), but notes this is a very slow computation. In the mean time, the following is a nice ttest implementation pulled from Charlotte's cellxgene-rest-api code  _____no_output_____

def diffexp(expression_1, expression_2, genes, pval=0.001, num_genes=20):
    """
    get top expressed genes from two different cell sets (uses t-test)
    :param expression_1: numpy expression array cell set 1
    :param expression_2: numpy expression array cell set 2
    :param pval: stats limit
    :param num_genes: number of genes to limit results to
    :return: Top genes and mean expression, pvalue, and average difference
     between cell set1 to cell set 2 for each gene in both cell sets
     sorted by t-test value
    """
    diff_exp = stats.ttest_ind(expression_1, expression_2)
    set1 = np.logical_and(diff_exp.pvalue < pval, diff_exp.statistic > 0)
    set2 = np.logical_and(diff_exp.pvalue < pval, diff_exp.statistic < 0)
    stat1 = diff_exp.statistic[set1]
    stat2 = diff_exp.statistic[set2]
    sort_set1 = np.argsort(stat1)[::-1]
    sort_set2 = np.argsort(stat2)
    pval1 = diff_exp.pvalue[set1][sort_set1]
    pval2 = diff_exp.pvalue[set2][sort_set2]
    mean_ex1_set1 = np.mean(expression_1[:, set1], axis=0)[sort_set1]
    mean_ex2_set1 = np.mean(expression_2[:, set1], axis=0)[sort_set1]
    mean_ex1_set2 = np.mean(expression_1[:, set2], axis=0)[sort_set2]
    mean_ex2_set2 = np.mean(expression_2[:, set2], axis=0)[sort_set2]
    mean_diff1 = mean_ex1_set1 - mean_ex2_set1
    mean_diff2 = mean_ex1_set2 - mean_ex2_set2
    genes_cellset_1 = np.array([genes[idx] for idx, val in enumerate(set1) if val])[sort_set1]
    genes_cellset_2 = np.array([genes[idx] for idx, val in enumerate(set2) if val])[sort_set2]
    return {
        "celllist1": {
            "topgenes": genes_cellset_1.tolist()[:num_genes],
            "mean_expression_cellset1": mean_ex1_set1.tolist()[:num_genes],
            "mean_expression_cellset2": mean_ex2_set1.tolist()[:num_genes],
            "pval": pval1.tolist()[:num_genes],
            "ave_diff": mean_diff1.tolist()[:num_genes]
        },
        "celllist2": {
            "topgenes": genes_cellset_2.tolist()[:num_genes],
            "mean_expression_cellset1": mean_ex1_set2.tolist()[:num_genes],
            "mean_expression_cellset2": mean_ex2_set2.tolist()[:num_genes],
            "pval": pval2.tolist()[:num_genes],
            "ave_diff": mean_diff2.tolist()[:num_genes]
        },
    }_____no_output_____def compare_cell_sets(adata, cluster1_label, cluster2_label=None, cluster_key='louvain'):
    idx1 = adata.obs[cluster_key] == cluster1_label
    cluster1 = adata.X[idx1]

    if not cluster2_label:
        idx2 = adata.obs[cluster_key] != cluster1_label ## If no second set provided, compare to all others
        cluster2_label='all others'
    else:
        idx2 = adata.obs[cluster_key] == cluster2_label
    cluster2 = adata.X[idx2]

    genes = adata.var_names
    full_de = diffexp(cluster1, cluster2,genes=genes, num_genes=5)
    de = pd.DataFrame(full_de['celllist1']).append(pd.DataFrame(full_de['celllist2']))
    de.rename(columns={'mean_expression_cellset1': cluster1_label,
                       'mean_expression_cellset2': cluster2_label},
             inplace=True)
    de = de.set_index('topgenes')

    fig, ax = plt.subplots(figsize=(6, 5))
    pval_cutoff = 0.001

    colors = assign_colors(set(de.index.values), 'Set2')
    for gene, vals in de.iterrows():
        c = colors[gene]
        x1, x2 = 0.25, 0.75
        y1, y2 = vals[cluster1_label], vals[cluster2_label]
        ax.scatter(x1, y1, s=60, c=c)
        ax.scatter(x2, y2, s=60, c=c)
        ax.plot([0.25, 0.75], [vals[cluster1_label], vals[cluster2_label]], c=c, label=gene)

    ax.text(0.25, -0.1, cluster1_label,transform=ax.transAxes, ha='center')
    ax.text(0.75,-0.1, cluster2_label, transform=ax.transAxes, ha='center')
    ax.set_ylabel('Mean expression')
    plt.legend(loc=(1,.4), bbox_transform=ax.transAxes,)

    ax.set_xlim(0,1)
    plt.tight_layout()
    plt.show()
    return full_de

# compare_cell_sets(adata, '0', None)

B_T_cell = compare_cell_sets(adata, 'B cell', 'T cell', cluster_key='cell_type')
B_cell = compare_cell_sets(adata, 'B cell', cluster_key='cell_type')_____no_output_____## Eventually, add a cell here showing gene ontology terms for enriched genes.

## Perhaps also try a GO enrichment analysis on the counts matrix (i.e., which processes are upregulated in cell type X)_____no_output_____


# Notebook from johnlees/genome-sketching
Path: notebooks/r3.1.Set-similarity-with-MinHash.ipynb

# 3.1. Set similarity with MinHash_____no_output_____## Recap

As we covered in the review, there are several different *flavours* of MinHash (e.g. KMV, Bottom-k). We will use **KMV MinHash** for now, which populates a sketch using the Minimum Values from K random permutations of the input data.

It gets a bit confusing when we start talking about k-mers as well as KMV sketches, so from now on we will only use k to refer to k-mers.

This notebook will:

* demonstrate set similarity
 - Jaccard similarity

* create the MinHash data structure using a Python library
 - add elements from a set
 - output a MinHash sketch of the set
 - estimate Jaccard similarity between sets using MinHash sketches

* adjust underlying properties of the MinHash data structure
 - vary the base hash function and the number of set permutations
 - compare Jaccard estimates from MinHash to the actual Jaccard set similarity

To begin, import the Python modules we will need:_____no_output_____

from matplotlib_venn import venn2, venn3
from datasketch import MinHash # a MinHash library by Eric Zhu_____no_output_____

We'll also import the functions that we covered in the [Background](Background.ipynb) workbook. This will make things like decomposing sequences to k-mers easier._____no_output_____

import nbimporter
import Background as utils_____no_output_____

***

## A scenario

To demonstrate set similarity and MinHash for genomic data, let's use a simple scenario. We'll revisit this scenario in the other workbooks, and the [example workflows](r4.1.Workflows-for-genomics.ipynb) are based on this too.

### Antimicrobial Resistance

**Antimicrobial resistance** (AMR) is a major public health concern and a massive research area, in which genomics plays a big role.

The term AMR refers to the ability of a microbe to resist the effects of antimicrobial drugs, which would usually kill or inactivate the microbe. Within AMR, there is antibiotic resistance, which specifically refers to bacteria that are resistant to antibiotic drugs. These resistant bacteria are harder to treat, needing higher doses or different antibiotics. Consequently, treatment regimes can be longer, more expensive or ineffective.

We will focus on antibiotic resistance and how we can use genomics and data sketching to research and combat this problem.

### Carbapenem resistance

Carbapenems are a group of antibiotic drug, belonging to the **β-lactam** class of antibiotics. They are a very effective antibiotic and are usually reserved for treating multidrug-resistant bacterial infections.

Resistance to carbapenems is particularly worrying as there are few drugs that can treat infections caused by carbapenem resistant bacteria.

Carbapenem resistance is mediated by a particular type of enzyme that is called a **β-lactamase**. This type of enzyme can break down the structure of the antibiotic, inactivating it.

Carbapenems were typically stable against β-lactamases but this is no longer the case, with several classes of β-lactamases emerging that can break down carbapenems - these are called **carbapenemases**.

### Carbapenemases

Hopefully you are still with me - one more definition to go.... Within the carbapenemase group of enzymes, we have several classes (groan). We won't go into this really, but the important thing is that one of these classes contains **IMP-type carbapenemases** (metallo-β-lactamases).

These carbapenemase enzymes are encoded by IMP genes and, importantly, are **plasmid-mediated**. So they can be transferred between bacteria, resulting in the spread of AMR.

### TL;DR

> IMP genes confer resistance to antibiotics that are reserved for treating multi-drug resistant infections. IMP genes are mobile and contibute to the spread of antibiotic resistance.

Let's use set similarity and data sketching to look at IMP genes and see if we can differentiate between IMP genes and non-carbapenemase β-lactamases.

## Set similarity

Okay. Let's start by taking two IMP genes and seeing how similar they are, based on their k-mer composition._____no_output_____* get the k-mers (k=7) for each gene:_____no_output_____

# These AMR genes are downloaded from the ResFinder database (accessed 05/2017):
## blaIMP-1_1_DQ522237
geneA = "ATGAGCAAGTTATCTGTATTCTTTATATTTTTGTTTTGCAGCATTGCTACCGCAGCAGAGTCTTTGCCAGATTTAAAAATTGAAAAGCTTGATGAAGGCGTTTATGTTCATACTTCGTTTAAAGAAGTTAACGGGTGGGGCGTTGTTCCTAAACATGGTTTGGTGGTTCTTGTAAATGCTGAGGCTTACCTAATTGACACTCCATTTACGGCTAAAGATACTGAAAAGTTAGTCACTTGGTTTGTGGAGCGTGGCTATAAAATAAAAGGCAGCATTTCCTCTCATTTTCATAGCGACAGCACGGGCGGAATAGAGTGGCTTAATTCTCGATCTATCCCCACGTATGCATCTGAATTAACAAATGAACTGCTTAAAAAAGACGGTAAGGTTCAAGCCACAAATTCATTTAGCGGAGTTAACTATTGGCTAGTTAAAAATAAAATTGAAGTTTTTTATCCAGGCCCGGGACACACTCCAGATAACGTAGTGGTTTGGTTGCCTGAAAGGAAAATATTATTCGGTGGTTGTTTTATTAAACCGTACGGTTTAGGCAATTTGGGTGACGCAAATATAGAAGCTTGGCCAAAGTCCGCCAAATTATTAAAGTCCAAATATGGTAAGGCAAAACTGGTTGTTCCAAGTCACAGTGAAGTTGGAGACGCATCACTCTTGAAACTTACATTAGAGCAGGCGGTTAAAGGGTTAAACGAAAGTAAAAAACCATCAAAACCAAGCAACTAA"

## blaIMP-2_1_AJ243491
geneB = "ATGAAGAAATTATTTGTTTTATGTGTATGCTTCCTTTGTAGCATTACTGCCGCGGGAGCGCGTTTGCCTGATTTAAAAATCGAGAAGCTTGAAGAAGGTGTTTATGTTCATACATCGTTCGAAGAAGTTAACGGTTGGGGTGTTGTTTCTAAACACGGTTTGGTGGTTCTTGTAAACACTGACGCCTATCTGATTGACACTCCATTTACTGCTACAGATACTGAAAAGTTAGTCAATTGGTTTGTGGAGCGCGGCTATAAAATCAAAGGCACTATTTCCTCACATTTCCATAGCGACAGCACAGGGGGAATAGAGTGGCTTAATTCTCAATCTATTCCCACGTATGCATCTGAATTAACAAATGAACTTCTTAAAAAAGACGGTAAGGTGCAAGCTAAAAACTCATTTAGCGGAGTTAGTTATTGGCTAGTTAAAAATAAAATTGAAGTTTTTTATCCCGGCCCGGGGCACACTCAAGATAACGTAGTGGTTTGGTTACCTGAAAAGAAAATTTTATTCGGTGGTTGTTTTGTTAAACCGGACGGTCTTGGTAATTTGGGTGACGCAAATTTAGAAGCTTGGCCAAAGTCCGCCAAAATATTAATGTCTAAATATGTTAAAGCAAAACTGGTTGTTTCAAGTCATAGTGAAATTGGGGACGCATCACTCTTGAAACGTACATGGGAACAGGCTGTTAAAGGGCTAAATGAAAGTAAAAAACCATCACAGCCAAGTAACTAA"

# get the canonical k-mers for each gene
kmersA = utils.getKmers(geneA, 7)
kmersB = utils.getKmers(geneB, 7)

# sanity check and just print a few k-mers out from each gene
print(kmersA[0:10])
print(kmersB[0:10])

# how many k-mers do we have
print("blaIMP-1 is {} bases long and has {} k-mers (k=7)" .format(len(geneA), len(kmersA)))
print("blaIMP-2 is {} bases long and has {} k-mers (k=7)" .format(len(geneA), len(kmersB)))

_____no_output_____

* now we have the two genes decomposed to sets of k-mers, let's see how similar they are

* to do this, we will calculate the Jaccard similarity:_____no_output_____

# this step converts the Python lists of k-mers into sets, which isn't important for the example but just let's us use some built in functions
setA = set(kmersA)
setB = set(kmersB)

# get the intersection and union of the k-mer sets
intersection = len(setA.intersection(setB))
union = len(setA.union(setB))

# calculate the Jaccard similarity
actualJaccardSimilarity = float(intersection)/float(union)

# print the values
print("The intersection of the k-mer sets = {}\nThe union of the k-mer sets = {}\n" .format(intersection, union) )
print("The Jaccard simularity between genes A and B = {}\n" .format(actualJaccardSimilarity))

# visualise with a Venn diagram
print("Venn diagram showing the intersection of the carbapenemase genes:")
venn2([setA, setB], set_labels = ('blaIMP-1', 'blaIMP-2'))_____no_output_____

* great, so now we know that are two IMP genes share 299 k-mers and the Jaccard similarity is 0.29

* now, how does a non-carbapenemase β-lactamase gene compare?_____no_output_____

# CTX-M β-lactamases are also plasmid mediated but they can't break down carbapenemases
## blaCTX-M-1_6_DQ915955
geneC = "ATGGTTAAAAAATCACTGCGTCAGTTCACGCTGATGGCGACGGCAACCGTCACGCTGTTGTTAGGAAGTGTGCCGCTGTATGCGCAAACGGCGGACGTACAGCAAAAACTTGCCGAATTAGAGCGGCAGTCGGGAGGAAGACTGGGTGTGGCATTGATTAACACAGCAGATAATTCGCAAATACTTTATCGTGCTGATGAGCGCTTTGCGATGTGCAGCACCAGTAAAGTGATGGCCGTGGCCGCGGTGCTGAAGAAAAGTGAAAGCGAACCGAATCTGTTAAATCAGCGAGTTGAGATCAAAAAATCTGACTTGGTTAACTATAATCCGATTGCGGAAAAGCACGTCGATGGGACGATGTCACTGGCTGAGCTTAGCGCGGCCGCGCTACAGTACAGCGATAACGTGGCGATGAATAAGCTGATTTCTCACGTTGGCGGCCCGGCTAGCGTCACCGCGTTCGCCCGACAGCTGGGAGACGAAACGTTCCGTCTCGACCGTACCGAGCCGACGTTAAACACCGCCATTCCGGGCGATCCGCGTGATACCACTTCACCTCGGGCAATGGCGCAAACTCTGCGTAATCTGACGCTGGGTAAAGCATTGGGTGACAGCCAACGGGCGCAGCTGGTGACATGGATGAAAGGCAATACCACCGGTGCAGCGAGCATTCAGGCTGGACTGCCTGCTTCCTGGGTTGTGGGGGATAAAACCGGCAGCGGTGACTATGGCACCACCAACGATATCGCGGTGATCTGGCCAAAAGATCGTGCGCCGCTGATTCTGGTCACTTACTTCACCCAGCCTCAACCTAAGGCAGAAAGCCGTCGCGATGTATTAGCGTCGGCGGCTAAAATCGTCACCAACGGTTTGTAA"
kmersC = utils.getKmers(geneC, 7)
setC = set(kmersC)
# visualise with a Venn diagram
print("Venn diagram showing the intersection of the β-lactamase genes:")
venn3([setA, setB, setC], set_labels = ('blaIMP-1', 'blaIMP-2', 'blaCTX-M-1'))
_____no_output_____

* we can see that the blaIMP genes share a lot more k-mers than either do with the blaCTX-M gene, which was to be expected!

* now let's try approximating these similarities using MinHash_____no_output_____## Estimating set similarity with MinHash

We'll use a MinHash library to save us coding up the data structure. This library means we don't have to define our own hash functions (you can do if you want though). All we need to do is specify how many set permutations we want to sample from and then we just provide it the k-mers.

* create the MinHash data structure: _____no_output_____

# define the number of set permutations to apply to our k-mer sets
permutations=100

# construct a MinHash data structure for each bla-IMP gene
mh1 = MinHash(num_perm=permutations)
mh2 = MinHash(num_perm=permutations)_____no_output_____

* now we add the k-mers from each gene to their respective MinHash:_____no_output_____

# define a helper function to add set elements to a MinHash data structure
def add2MinHash(mh, data):
    # loop through the list of k-mers
    for d in data:
        # here, we use the library's update function
        # this is where each k-mer is hashed, with a different hash for each set permutation
        # if the k-mer hash is the lowest value seen to date, it is added to the MinHash sketch
        mh.update(d.encode('utf8'))
    return mh

# add the k-mers
mh1 = add2MinHash(mh1, kmersA)
mh2 = add2MinHash(mh2, kmersB)_____no_output_____

* now we have MinHash sketches, one for each gene

* to compare the sketches and get a Jaccard similarity estimate, we just use the jaccard method from the library!_____no_output_____

print("Estimated Jaccard similarity for blaIMP-1 and blaIMP-2 is", mh1.jaccard(mh2))_____no_output_____

* not bad, our actual Jaccard similarity between blaIMP-1 and blaIMP-2 was 0.29 and our estimate is 0.24
* this estimate is based on only 100 k-mers per sketch, compared to the 735 k-mers per gene for the actual similarity
* for the sake of completeness, let's sketch the blaCTX-M gene_____no_output_____

mh3 = MinHash(num_perm=permutations)
mh3 = add2MinHash(mh3, kmersC)
print("Estimated Jaccard similarity for blaIMP-1 and blaCTX-M-1 is", mh1.jaccard(mh3))
print("Estimated Jaccard similarity for blaIMP-2 and blaCTX-M-1 is", mh2.jaccard(mh3))
_____no_output_____

## MinHash a genomic data stream

So MinHash is pretty good at approximating similarity between genes. We haven't really seen the benefits though, as all we did was essential reduce the number of k-mers from 735 to 100, which doesn't save us much in terms of time or space.

Let's finish up this notebook by showing how we can apply MinHash to something more useful.

A bacteria that we look at in the example workflow is [Enterobacter cloacae](https://en.wikipedia.org/wiki/Enterobacter_cloacae), which is a nosocomial pathogen. Let's use MinHash to compare how similar the reference genome is for this bacteria to another species from the same genus, [Enterobacter mori](https://microbe.directory/species/Enterobacter+mori), which is a plant pathogen.

* let's simulate a genomic data stream from an E.cloacae download and MinHash it:_____no_output_____

# stream k-mers from E. cloacae
dataStream = utils.streamGenome('../data/GCF_000025565.1_ASM2556v1_genomic.fna.gz', 7)

# set up the MinHash
cloacaeSketch = MinHash(num_perm=128)

# sketch! this will take a couple of minutes
for kmer in dataStream:
    cloacaeSketch.update(kmer.encode('utf8'))_____no_output_____

* now do the same for E.mori:_____no_output_____

# stream k-mers from E. cloacae
dataStream = utils.streamGenome('../data/GCF_000211415.1_ASM21141v1_genomic.fna.gz', 7)

# set up the MinHash
moriSketch = MinHash(num_perm=128)
for kmer in dataStream:
    moriSketch.update(kmer.encode('utf8'))_____no_output_____

* and compare!_____no_output_____

print("Estimated Jaccard similarity for E.cloacae and E.mori reference genomes is", cloacaeSketch.jaccard(moriSketch))_____no_output_____

So, these genomes are estimated to be 93% similar, based on k-mer composition. We have used only 128 hashes to approximate each genome, and each hash is derived from a k-mer which is 7 bases long.

A good exercise is to try the above again, this time increasing the size of k. Hopefully you should see that the longer the k, the less similar the genomes are. This is because the longer the k-mer, the more species specific it is. So, increase the k-mer length to 31 here and you should be able to differentiate between these two enterobacter species quite nicely.

You can also try changing the number of values in the sketches - long sketches give better estimates!

We'll revisit MinHash later, this time using some cool bioinformatics software which can do the above more quickly and output more information etc._____no_output_____***

Return to [this notebook](./notebooks/Background.ipynb) to access the table of contents and select another notebook.

***_____no_output_____


# Notebook from oliverob/Cirq
Path: docs/gates.ipynb

# Gates

A `Gate` is an effect that can be applied to a collection of
qubits (objects with a ``Qid``).  `Gates` can be applied
to qubits by calling their ``on`` method, or, alternatively
calling the gate on the qubits.  The object created by such calls
is an ``Operation``.  Alternatively, a `Gate` can be thought of
as a [factory](https://en.wikipedia.org/wiki/Factory_method_pattern)
that, given input qubits, generates an associated
`GateOperation` object._____no_output_____## Gates versus Operations

![Circuits and Moments](images/moments.png)

The above example shows the first half of a quantum teleportation circuit,
found in many quantum computation textbooks.  This example uses three different
gates: a Hadamard (H), controlled-Not (CNOT), and measurement.  These are
represented in cirq by ```cirq.H```, ```cirq.CNOT```, and ``cirq.measure``,
respectively.

In this example, a Hadamard is an example of a `Gate` object that can be applied
in many different circumstances and to many different qubits.  Note that the
above example has two instances of an H gate but applied to different qubits.
This is an example of one `Gate` type with two `Operation` instances, one
applied to the qubit '|b⟩' and the other applied to qubit '|a⟩'.

Gates can generally be applied to any type of qubit (``NamedQubit``,
``LineQubit``, ``GridQubit``, etc) to create an Operation.
However, depending on the application, you may prefer a specific type
of qubit.  For instance, [Google devices](google/devices.md) generally use
``GridQubits``.  Other [devices](devices.md) may have connectivity constraints
that further restrict the set of qubits that can be used, especially in multi-
qubit operations.

The following example shows how to construct each of these gates and operations.
_____no_output_____

import cirq

# This examples uses named qubits to remain abstract.
# However, we can also use LineQubits or GridQubits to specify a geometry
a = cirq.NamedQubit('a')
b = cirq.NamedQubit('b')
c = cirq.NamedQubit('c')

# Example Operations, that correspond to the moments above
print(cirq.H(b))
print(cirq.CNOT(b, c))
print(cirq.CNOT(a, b))
print(cirq.H(a))
print(cirq.measure(a,b))
# prints
# H(b)
# CNOT(b, c)
# CNOT(a, b)
# H(a)
# cirq.MeasurementGate(2, 'a,b', ())(a, b)H(b)
CNOT(b, c)
CNOT(a, b)
H(a)
cirq.MeasurementGate(2, 'a,b', ())(a, b)

This would create the operations needed to comprise the circuit from the above
diagram.  The next step would be composing these operations into moments and
circuits.  For more on those types, see the documentation on
[Circuits](circuits.ipynb)._____no_output_____## Other gate features

Most ``Gate``s operate on a specific number of qubits, which can be accessed
by the ``num_qubits()`` function.  One notable exception is the
``MeasurementGate`` which can be applied to a variable number of qubits.

Most gates also have a unitary matrix representation, which can be accessed
by ``cirq.unitary(gate)``.

Not all `Gate`s correspond to unitary evolution. They may represent a
probabilistic mixture of unitaries, or a general quantum channel. The component
unitaries and associated probabilities of a mixture can be accessed by
`cirq.mixture(gate)`. The Kraus operator representation of a channel can be
accessed by `cirq.channel(gate)`. Non-unitary gates are often used in the
simulation of noise. See [noise documentation](noise.ipynb) for more details.

Many arithmetic operators will work in the expected way when applied to
gates.  For instance, ``cirq.X**0.5`` represents a square root of X gate.
These can also be applied to Operators for a more compact representation, such
as ``cirq.X(q1)**0.5`` will be a square root of X gate applied to the q1
qubit.  This functionality depends on the "magic methods" of the gate
being defined (see below for details).

Gates can be converted to a controlled version by using `Gate.controlled()`.
In general, this returns an instance of a `ControlledGate`.  However, for
certain special cases where the controlled version of the gate is also a
known gate, this returns the instance of that gate.
For instance, `cirq.X.controlled()` returns a `cirq.CNOT` gate.
Operations have similar functionality `Operation.controlled_by()`, such as
`cirq.X(q0).controlled_by(q1)`._____no_output_____## Common gates

Cirq supports a number of gates natively, with the opportunity to extend these
gates for more advanced use cases._____no_output_____### Measurement gate

**cirq.MeasurementGate** This is a measurement in the computational basis.
This gate can be applied to a variable number of qubits.  The function
`cirq.measure(q0, q1, ...)` can also be used as a short-hand to create a
`MeasurementGate` ._____no_output_____### Single qubit gates

Most single-qubit gates can be thought of as rotation around an axis in the
[Bloch Sphere](https://en.wikipedia.org/wiki/Bloch_sphere) representation and
are usually referred to by their axis of rotation.  Some operators use the notation of a 'half-turn' which is defined as a 180 degree (pi radians) rotation around the axis.

**cirq.X / cirq.Y / cirq.Z** The [Pauli gates](https://en.wikipedia.org/wiki/Quantum_logic_gate#Pauli-X_gate) X, Y, and Z which rotate the state around the  associated axis by one half-turn.

**cirq.rx(rads)** A rotation about the Pauli 'X' axis in terms of radians.  This is equivalent to `exp(-i X rads / 2) = cos(rads/2) I - i sin(rads/2) X`

**cirq.ry(rads)** A rotation about the Pauli 'Y' axis in terms of radians.  This is equivalent to `exp(-i Y rads / 2) = cos(rads/2) I - i sin(rads/2) Y`

**cirq.rz(rads)** A rotation about the Pauli 'Z' axis in terms of radians.  This is equivalent to `exp(-i Z rads / 2) = cos(rads/2) I - i sin(rads/2) Z`

**cirq.XPowGate(exponent=t)** Rotations about the Pauli ``X`` axis, equivalent to ``cirq.X**t``.  See [XPowGate](https://cirq.readthedocs.io/en/latest/generated/cirq.XPowGate.html) for its unitary matrix.  Note that this has a global phase of e^{i·π·t/2} versus the traditionally defined rotation matrix, which can be modified by the optional parameter `global_shift`.

**cirq.YPowGate(exponent=t)** Rotations about the Pauli ``Y`` axis, equivalent to ``cirq.Y**t``.  See [YPowGate](https://cirq.readthedocs.io/en/latest/generated/cirq.YPowGate.html) for its unitary matrix.  Note that this has a global phase of e^{i·π·t/2} versus the traditionally defined rotation matrix, which can be modified by the optional parameter `global_shift`.

**cirq.ZPowGate(exponent=t)** Rotations about the Pauli ``Z`` axis, equivalent to ``cirq.Z**t``.  See [ZPowGate](https://cirq.readthedocs.io/en/latest/generated/cirq.ZPowGate.html) for its unitary matrix.  Note that this has a global phase of e^{i·π·t/2} versus the traditionally defined rotation matrix, which can be modified by the optional parameter `global_shift`.

**cirq.PhasedXPowGate**
This gate is a rotation about an axis in the XY plane of the Bloch sphere.
The ``PhasedXPowGate`` takes two parameters, ``exponent`` and ``phase_exponent``.
The gate is equivalent to the circuit `───Z^-p───X^t───Z^p───` where `p` is the
`phase_exponent` and `t` is the `exponent`.

**cirq.H / cirq.HPowGate** The Hadamard gate is a rotation around the X+Z axis.
`cirq.HPowGate(exponent=t)` is a variable rotation of t turns around this
axis. `cirq.H` is a π rotation and is equivalent to
`cirq.HPowGate(exponent=1)`

**S** The square root of Z gate, equivalent to `cirq.Z**0.5`

**T** The fourth root of Z gate, equivalent to `cirq.Z**0.25`.
_____no_output_____### Two qubit gates

**cirq.CZ / cirq.CZPowGate** The controlled-Z gate.  A two qubit gate that
phases the |11⟩ state.  `cirq.CZPowGate(exponent=y)` is equivalent to
`cirq.CZ**t` and has a matrix representation of ``exp(i pi |11⟩⟨11| t)``.

**cirq.CNOT / cirq.CNotPowGate** The controlled-X gate.  This gate swaps the
|11⟩ and |10⟩ states.  `cirq.CNotPowGate(exponent=t)` is equivalent
to `cirq.CNOT**t` .

**cirq.SWAP / cirq.SwapPowGate** The swap gate swaps the |01⟩ and |10⟩ states.
`cirq.SWAP**t` is the same as `cirq.SwapPowGate(exponent = t)`

**cirq.ISWAP / cirq.ISwapPowGate**  The iSwap gate swaps the |01⟩ and |10⟩
states and adds a relative phase of i.  `cirq.ISWAP**t` is the same as
`cirq.ISwapPowGate(exponent = t)`

**Parity gates**: The gates cirq.XX, cirq.YY, and cirq.ZZ are equivalent to
performing the equivalent one-qubit Pauli gates on both qubits.  The gates
cirq.XXPowGate, cirq.YYPowGate, and cirq.ZZPowGate are the powers of
these gates.
_____no_output_____### Other Gates

**cirq.MatrixGate**: A gate defined by its unitary matrix in the form of a
numpy ndarray.

**cirq.WaitGate**:  This gate does nothing for a specified `cirq.Duration`
amount of time.  This is useful for conducting T1 and T2 decay experiments.

**cirq.CCNOT, cirq.CCX, cirq.TOFFOLI, cirq.CCXPowGate**: Three qubit gates
representing the controlled-controlled-X gates.

**cirq.CCZ, cirq.CCZPowGate**: Three qubit gates representing a
controlled-controlled-Z gate.

**CSWAP, CSwapGate, FREDKIN**: Three qubit gates representing a controlled-SWAP
gate._____no_output_____## Advanced: Creating your own gates

If the above gates are not sufficient for your use case, it is fairly simple
to create your own gate.   In order to do so, you can define your class and
inherit the `cirq.Gate` class and define the functionality in your class.

Much of cirq relies on "magic methods", which are methods prefixed with one or
two underscores and used by cirq's protocols or built-in python methods.
For instance,  python translates `cirq.Z**0.25` into
`cirq.Z.__pow__(0.25)`.  Other uses are specific to cirq and are found in the
protocols subdirectory.  They are defined below.

At minimum, you will need to define either the ``_num_qubits_`` or
``_qid_shape_`` magic method to define the number of qubits (or qudits) used
in the gate.  For convenience one can use the ``SingleQubitGate``,
``TwoQubitGate``, and ``ThreeQubitGate`` classes for these common gate sizes._____no_output_____### Magic Methods_____no_output_____
#### Standard python magic methods

There are many standard magic methods in python.  Here are a few of the most
important ones used in cirq:
  * `__str__` for user-friendly string output and  `__repr__` is the python-friendly string output, meaning that `eval(repr(y))==y` should always be true.
  * `__eq__` and `__hash__` which define whether objects are equal or not.  You
  can also use `cirq.value.value_equality` for objects that have a small list
  of sub-values that can be compared for equality.
  * Arithmetic functions such as `__pow__`, `__mul__`, `__add__` define the
  action of `**`, `*`, and `+` respectively.

#### `cirq.num_qubits` and `def _num_qubits_`

A `Gate` must implement the `_num_qubits_` (or `_qid_shape_`) method.
This method returns an integer and is used by `cirq.num_qubits` to determine
how many qubits this gate operates on.

#### `cirq.qid_shape` and `def _qid_shape_`

A qudit gate or operation must implement the `_qid_shape_` method that returns a
tuple of integers.  This method is used to determine how many qudits the gate or
operation operates on and what dimension each qudit must be.  If only the
`_num_qubits_` method is implemented, the object is assumed to operate only on
qubits. Callers can query the qid shape of the object by calling
`cirq.qid_shape` on it. See [qudit documentation](qudits.ipynb) for more
information.

#### `cirq.unitary` and `def _unitary_`

When an object can be described by a unitary matrix, it can expose that unitary
matrix by implementing a `_unitary_(self) -> np.ndarray` method.
Callers can query whether or not an object has a unitary matrix by calling
`cirq.unitary` on it.
The `_unitary_` method may also return `NotImplemented`, in which case
`cirq.unitary` behaves as if the method is not implemented.

#### `cirq.decompose` and `def _decompose_`

Operations and gates can be defined in terms of other operations by implementing
a `_decompose_` method that returns those other operations. Operations implement
`_decompose_(self)` whereas gates implement `_decompose_(self, qubits)`
(since gates don't know their qubits ahead of time).

The main requirements on the output of `_decompose_` methods are:

1. DO NOT CREATE CYCLES. The `cirq.decompose` method will iterative decompose until it finds values satisfying a `keep` predicate. Cycles cause it to enter an infinite loop.
2. Head towards operations defined by Cirq, because these operations have good decomposition methods that terminate in single-qubit and two qubit gates.
These gates can be understood by the simulator, optimizers, and other code.
3. All that matters is functional equivalence.
Don't worry about staying within or reaching a particular gate set; it's too hard to predict what the caller will want. Gate-set-aware decomposition is useful, but *this is not the protocol that does that*.
Gate-set-aware decomposition may be added in the future, but doesn't exist within Cirq at the moment.

For example, `cirq.CCZ` decomposes into a series of `cirq.CNOT` and `cirq.T` operations.
This allows code that doesn't understand three-qubit operation to work with `cirq.CCZ`; by decomposing it into operations they do understand.
As another example, `cirq.TOFFOLI` decomposes into a `cirq.H` followed by a `cirq.CCZ` followed by a `cirq.H`.
Although the output contains a three qubit operation (the CCZ), that operation can be decomposed into two qubit and one qubit operations.
So code that doesn't understand three qubit operations can deal with Toffolis by decomposing them, and then decomposing the CCZs that result from the initial decomposition.

In general, decomposition-aware code consuming operations is expected to recursively decompose unknown operations until the code either hits operations it understands or hits a dead end where no more decomposition is possible.
The `cirq.decompose` method implements logic for performing exactly this kind of recursive decomposition.
Callers specify a `keep` predicate, and optionally specify intercepting and fallback decomposers, and then `cirq.decompose` will repeatedly decompose whatever operations it was given until the operations satisfy the given `keep`.
If `cirq.decompose` hits a dead end, it raises an error.

Cirq doesn't make any guarantees about the "target gate set" decomposition is heading towards.
`cirq.decompose` is not a method
Decompositions within Cirq happen to converge towards X, Y, Z, CZ, PhasedX, specified-matrix gates, and others.
But this set will vary from release to release, and so it is important for consumers of decompositions to look for generic properties of gates,
such as "two qubit gate with a unitary matrix", instead of specific gate types such as CZ gates.

#### `cirq.inverse` and `__pow__`

Gates and operations are considered to be *invertable* when they implement a `__pow__` method that returns a result besides `NotImplemented` for an exponent of -1.
This inverse can be accessed either directly as `value**-1`, or via the utility method `cirq.inverse(value)`.
If you are sure that `value` has an inverse, saying `value**-1` is more convenient than saying `cirq.inverse(value)`.
`cirq.inverse` is for cases where you aren't sure if `value` is invertable, or where `value` might be a *sequence* of invertible operations.

`cirq.inverse` has a `default` parameter used as a fallback when `value` isn't invertable.
For example, `cirq.inverse(value, default=None)` returns the inverse of `value`, or else returns `None` if `value` isn't invertable.
(If no `default` is specified and `value` isn't invertible, a `TypeError` is raised.)

When you give `cirq.inverse` a list, or any other kind of iterable thing, it will return a sequence of operations that (if run in order) undoes the operations of the original sequence (if run in order).
Basically, the items of the list are individually inverted and returned in reverse order.
For example, the expression `cirq.inverse([cirq.S(b), cirq.CNOT(a, b)])` will return the tuple `(cirq.CNOT(a, b), cirq.S(b)**-1)`.

Gates and operations can also return values beside `NotImplemented` from their `__pow__` method for exponents besides `-1`.
This pattern is used often by Cirq.
For example, the square root of X gate can be created by raising `cirq.X` to 0.5:
_____no_output_____

import cirq
print(cirq.unitary(cirq.X))
# prints
# [[0.+0.j 1.+0.j]
#  [1.+0.j 0.+0.j]]

sqrt_x = cirq.X**0.5
print(cirq.unitary(sqrt_x))
# prints
# [[0.5+0.5j 0.5-0.5j]
#  [0.5-0.5j 0.5+0.5j]][[0.+0.j 1.+0.j]
 [1.+0.j 0.+0.j]]
[[0.5+0.5j 0.5-0.5j]
 [0.5-0.5j 0.5+0.5j]]

The Pauli gates included in Cirq use the convention ``Z**0.5 ≡ S ≡ np.diag(1, i)``, ``Z**-0.5 ≡ S**-1``, ``X**0.5 ≡ H·S·H``, and the square root of ``Y`` is inferred via the right hand rule.
_____no_output_____
#### `_circuit_diagram_info_(self, args)` and `cirq.circuit_diagram_info(val, [args], [default])`

Circuit diagrams are useful for visualizing the structure of a `Circuit`.
Gates can specify compact representations to use in diagrams by implementing a `_circuit_diagram_info_` method.
For example, this is why SWAP gates are shown as linked '×' characters in diagrams.

The `_circuit_diagram_info_` method takes an `args` parameter of type `cirq.CircuitDiagramInfoArgs` and returns either
a string (typically the gate's name), a sequence of strings (a label to use on each qubit targeted by the gate), or an
instance of `cirq.CircuitDiagramInfo` (which can specify more advanced properties such as exponents and will expand
in the future).

You can query the circuit diagram info of a value by passing it into `cirq.circuit_diagram_info`._____no_output_____


# Notebook from Amadeus-System/00.Crystal_Graph_Convolutional_Neural_Network
Path: 00. Original CGCNN code.ipynb

# Google Drive Mount_____no_output_____

from google.colab import drive
drive.mount("/content/drive")Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
cd drive/My Drive[Errno 2] No such file or directory: 'drive/My Drive'
/content/drive/My Drive
ls'읽을 예정인 논문 모음'/  '01. Backup Files'/   cgcnn/      GraphGAN/
'00. Github'/             '02. 데이터'/         CGCNN-HD/

# CGCNN github clone_____no_output_____

#!git clone https://github.com/txie-93/cgcnn_____no_output_____

# Pymatgen Install_____no_output_____

!pip install pymatgen==2020.11.11Requirement already satisfied: pymatgen==2020.11.11 in /usr/local/lib/python3.7/dist-packages (2020.11.11)
Requirement already satisfied: plotly>=4.5.0 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2020.11.11) (4.14.3)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pymatgen==2020.11.11) (2.23.0)
Requirement already satisfied: spglib>=1.9.9.44 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2020.11.11) (1.16.1)
Requirement already satisfied: ruamel.yaml>=0.15.6 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2020.11.11) (0.16.13)
Requirement already satisfied: monty>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2020.11.11) (2021.3.3)
Requirement already satisfied: matplotlib>=1.5 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2020.11.11) (3.2.2)
Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from pymatgen==2020.11.11) (1.1.5)
Requirement already satisfied: networkx>=2.2 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2020.11.11) (2.5)
Requirement already satisfied: uncertainties>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2020.11.11) (3.1.5)
Requirement already satisfied: sympy in /usr/local/lib/python3.7/dist-packages (from pymatgen==2020.11.11) (1.7.1)
Requirement already satisfied: scipy>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2020.11.11) (1.6.1)
Requirement already satisfied: numpy>=1.14.3 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2020.11.11) (1.19.5)
Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from pymatgen==2020.11.11) (0.8.9)
Requirement already satisfied: palettable>=3.1.1 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2020.11.11) (3.3.0)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from plotly>=4.5.0->pymatgen==2020.11.11) (1.15.0)
Requirement already satisfied: retrying>=1.3.3 in /usr/local/lib/python3.7/dist-packages (from plotly>=4.5.0->pymatgen==2020.11.11) (1.3.3)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2020.11.11) (1.24.3)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2020.11.11) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2020.11.11) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2020.11.11) (2020.12.5)
Requirement already satisfied: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.10" in /usr/local/lib/python3.7/dist-packages (from ruamel.yaml>=0.15.6->pymatgen==2020.11.11) (0.2.2)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen==2020.11.11) (1.3.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen==2020.11.11) (2.4.7)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen==2020.11.11) (0.10.0)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen==2020.11.11) (2.8.1)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->pymatgen==2020.11.11) (2018.9)
Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.7/dist-packages (from networkx>=2.2->pymatgen==2020.11.11) (4.4.2)
Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from uncertainties>=3.1.4->pymatgen==2020.11.11) (0.16.0)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy->pymatgen==2020.11.11) (1.2.1)

## cgcnn/cgcnn/data.py_____no_output_____

from __future__ import print_function, division

import csv
import functools
import json
import os
import random
import warnings

import numpy as np
import torch
from pymatgen.core.structure import Structure
from torch.utils.data import Dataset, DataLoader
from torch.utils.data.dataloader import default_collate
from torch.utils.data.sampler import SubsetRandomSampler_____no_output_____def get_train_val_test_loader(dataset, collate_fn = default_collate,
                              batch_size = 64, train_ratio = None,
                              val_ratio = 0.1, test_ratio = 0.1, return_test = False,
                              num_workers = 1, pin_memory = False, **kwargs):
    """
    Utility function for dividing a dataset to train, val, test datasets.
    !!! The dataset needs to be shuffled before using the function !!!

    Parameters
    ----------
    dataset: torch.utils.data.Dataset
             The full dataset to be divided.
    collate_fn: torch.utils.data.DataLoader
    batch_size: int
    train_ratio: float
    val_ratio: float
    test_ratio: float
    return_test: bool
        Whether to return the test dataset loader. If False, the last test_size
        data will be hidden.
    num_workers: int
    pin_memory: bool

    Returns
    -------
    train_loader: torch.utils.data.DataLoader
        DataLoader that random samples the training data.
    val_loader: torch.utils.data.DataLoader
        DataLoader that random samples the validation data.
    (test_loader): torch.utils.data.DataLoader
        DataLoader that random samples the test data, returns if
        return_test = True.
    """
    total_size = len(dataset)

    if train_ratio is None:
        assert val_ratio + test_ratio < 1
        train_ratio = 1 - val_ratio - test_ratio
        print('[Warning] train_ratio is None, using all training data.')
    else:
        assert train_ratio + val_ratio + test_ratio <= 1

    indices = list(range(total_size))

    if kwargs['train_size']:
        train_size = kwargs['train_size']
    else:
        train_size = int(train_ratio * total_size)

    if kwargs['test_size']:
        test_size = kwargs['test_size']
    else:
        test_size = int(test_ratio * total_size)

    if kwargs['val_size']:
        valid_size = kwargs['val_size']
    else:
        valid_size = int(val_ratio * total_size)

    train_sampler = SubsetRandomSampler(indices[:train_size])
    val_sampler = SubsetRandomSampler(
        indices[-(valid_size + test_size):-test_size])

    if return_test:
        test_sampler = SubsetRandomSampler(indices[-test_size:])

    train_loader = DataLoader(dataset, batch_size = batch_size,
                              sampler = train_sampler,
                              num_workers = num_workers,
                              collate_fn = collate_fn, pin_memory = pin_memory)
    val_loader = DataLoader(dataset, batch_size = batch_size,
                            sampler = val_sampler,
                            num_workers = num_workers,
                            collate_fn = collate_fn, pin_memory = pin_memory)
    if return_test:
        test_loader = DataLoader(dataset, batch_size = batch_size,
                                 sampler = test_sampler,
                                 num_workers = num_workers,
                                 collate_fn = collate_fn, pin_memory = pin_memory)
    if return_test:
        return train_loader, val_loader, test_loader
    else:
        return train_loader, val_loader
        _____no_output_____def collate_pool(dataset_list):
    """
    Collate a list of data and return a batch for predicting crystal properties.

    Parameters
    ----------
    dataset_list: list of tuples for each data point.
        (atom_fea, nbr_fea, nbr_fea_idx, target)

        atom_fea: torch.Tensor shape (n_i, atom_fea_len)
        nbr_fea: torch.Tensor shape (n_i, M, nbr_fea_len)
        nbr_fea_idx: torch.LongTensor shape (n_i, M)
        target: torch.Tensor shape (1, )
        cif_id: str or int

    Returns
    -------
    N = sum(n_i); N0 = sum(i)

    batch_atom_fea: torch.Tensor shape (N, orig_atom_fea_len)
        Atom features from atom type
    batch_nbr_fea: torch.Tensor shape (N, M, nbr_fea_len)
        Bond features of each atom's M neighbors
    batch_nbr_fea_idx: torch.LongTensor shape (N, M)
        Indices of M neighbors of each atom
    crystal_atom_idx: list of torch.LongTensor of length N0
        Mapping from the crystal idx to atom idx
    target: torch.Tensor shape (N, 1)
        Target value for prediction
    batch_cif_ids: list
    """
    batch_atom_fea, batch_nbr_fea, batch_nbr_fea_idx = [], [], []
    crystal_atom_idx, batch_target = [], []
    batch_cif_ids = []
    base_idx = 0

    for i, ((atom_fea, nbr_fea, nbr_fea_idx), target, cif_id) in enumerate(dataset_list):

        n_i = atom_fea.shape[0]  # number of atoms for this crystal

        batch_atom_fea.append(atom_fea)
        batch_nbr_fea.append(nbr_fea)
        batch_nbr_fea_idx.append(nbr_fea_idx + base_idx)

        new_idx = torch.LongTensor(np.arange(n_i) + base_idx)

        crystal_atom_idx.append(new_idx)
        batch_target.append(target)
        batch_cif_ids.append(cif_id)

        base_idx += n_i

    return (torch.cat(batch_atom_fea, dim = 0),
            torch.cat(batch_nbr_fea, dim = 0),
            torch.cat(batch_nbr_fea_idx, dim = 0),
            crystal_atom_idx), \
            torch.stack(batch_target, dim = 0), \
            batch_cif_ids
_____no_output_____class GaussianDistance(object):
    """
    Expands the distance by Gaussian basis.

    Unit: angstrom
    """
    def __init__(self, dmin, dmax, step, var = None):
        """
        Parameters
        ----------
        dmin: float
            Minimum interatomic distance
        dmax: float
            Maximum interatomic distance
        step: float
            Step size for the Gaussian filter
        """
        assert dmin < dmax
        assert dmax - dmin > step
        self.filter = np.arange(dmin, dmax + step, step)
        if var is None:
            var = step
        self.var = var

    def expand(self, distances):
        """
        Apply Gaussian distances filter to a numpy distance array

        Parameters
        ----------
        distance: np.array shape n-d array
            A distance matrix of any shape

        Returns
        -------
        expanded_distance: shape (n+1)-d array
            Expanded distance matrix with the last dimension of length
            len(self.filter)
        """
        return np.exp(-(distances[..., np.newaxis] - self.filter)**2 / self.var**2)
        _____no_output_____class AtomInitializer(object):
    """
    Base class for initializing the vector representation for atoms.
    !!! Use one AtomInitializer per dataset !!!
    """
    def __init__(self, atom_types):
        self.atom_types = set(atom_types)
        self._embedding = {}

    def get_atom_fea(self, atom_type):
        assert atom_type in self.atom_types
        return self._embedding[atom_type]

    def load_state_dict(self, state_dict):
        self._embedding = state_dict
        self.atom_types = set(self._embedding.keys())
        self._decodedict = {idx: atom_type for atom_type, idx in
                            self._embedding.items()}

    def state_dict(self):
        return self._embedding

    def decode(self, idx):
        if not hasattr(self, '_decodedict'):
            self._decodedict = {idx: atom_type for atom_type, idx in
                                self._embedding.items()}
        return self._decodedict[idx]
        _____no_output_____class AtomCustomJSONInitializer(AtomInitializer):
    """
    Initialize atom feature vectors using a JSON file, which is a python
    dictionary mapping from element number to a list representing the
    feature vector of the element.

    Parameters
    ----------
    elem_embedding_file: str
        The path to the .json file
    """
    def __init__(self, elem_embedding_file):
        with open(elem_embedding_file) as f:
            elem_embedding = json.load(f)
        elem_embedding = {int(key): value for key, value
                          in elem_embedding.items()}
        atom_types = set(elem_embedding.keys())
        super(AtomCustomJSONInitializer, self).__init__(atom_types)
        for key, value in elem_embedding.items():
            self._embedding[key] = np.array(value, dtype = float)
            _____no_output_____class CIFData(Dataset):
    """
    The CIFData dataset is a wrapper for a dataset where the crystal structures
    are stored in the form of CIF files. The dataset should have the following
    directory structure:

    root_dir
    ├── id_prop.csv
    ├── atom_init.json
    ├── id0.cif
    ├── id1.cif
    ├── ...

    id_prop.csv: a CSV file with two columns. The first column recodes a
    unique ID for each crystal, and the second column recodes the value of
    target property.

    atom_init.json: a JSON file that stores the initialization vector for each
    element.

    ID.cif: a CIF file that recodes the crystal structure, where ID is the
    unique ID for the crystal.

    Parameters
    ----------
    root_dir: str
        The path to the root directory of the dataset
    max_num_nbr: int
        The maximum number of neighbors while constructing the crystal graph
    radius: float
        The cutoff radius for searching neighbors
    dmin: float
        The minimum distance for constructing GaussianDistance
    step: float
        The step size for constructing GaussianDistance
    random_seed: int
        Random seed for shuffling the dataset

    Returns
    -------
    atom_fea: torch.Tensor shape (n_i, atom_fea_len)
    nbr_fea: torch.Tensor shape (n_i, M, nbr_fea_len)
    nbr_fea_idx: torch.LongTensor shape (n_i, M)
    target: torch.Tensor shape (1, )
    cif_id: str or int
    """
    def __init__(self, root_dir, max_num_nbr = 12, radius = 8, dmin = 0, step = 0.2,
                 random_seed = 123):
        self.root_dir = root_dir
        self.max_num_nbr, self.radius = max_num_nbr, radius
        assert os.path.exists(root_dir), 'root_dir does not exist!'
        id_prop_file = os.path.join(self.root_dir, 'id_prop.csv')
        assert os.path.exists(id_prop_file), 'id_prop.csv does not exist!'
        with open(id_prop_file) as f:
            reader = csv.reader(f)
            self.id_prop_data = [row for row in reader]
        random.seed(random_seed)
        random.shuffle(self.id_prop_data)
        atom_init_file = os.path.join(self.root_dir, 'atom_init.json')
        assert os.path.exists(atom_init_file), 'atom_init.json does not exist!'

        self.ari = AtomCustomJSONInitializer(atom_init_file)
        self.gdf = GaussianDistance(dmin = dmin, dmax = self.radius, step = step)

    def __len__(self):
        return len(self.id_prop_data)

    @functools.lru_cache(maxsize = None)  # Cache loaded structures
    def __getitem__(self, idx):
        cif_id, target = self.id_prop_data[idx]
        crystal = Structure.from_file(os.path.join(self.root_dir, cif_id + '.cif'))

        atom_fea = np.vstack([self.ari.get_atom_fea(crystal[i].specie.number)
                              for i in range(len(crystal))])
        atom_fea = torch.Tensor(atom_fea)

        all_nbrs = crystal.get_all_neighbors(self.radius, include_index = True)
        all_nbrs = [sorted(nbrs, key = lambda x: x[1]) for nbrs in all_nbrs]

        nbr_fea_idx, nbr_fea = [], []
        for nbr in all_nbrs:
            if len(nbr) < self.max_num_nbr:
                warnings.warn('{} not find enough neighbors to build graph. '
                              'If it happens frequently, consider increase '
                              'radius.'.format(cif_id))
                nbr_fea_idx.append(list(map(lambda x: x[2], nbr)) +
                                   [0] * (self.max_num_nbr - len(nbr)))
                nbr_fea.append(list(map(lambda x: x[1], nbr)) +
                               [self.radius + 1.] * (self.max_num_nbr - len(nbr)))
            else:
                nbr_fea_idx.append(list(map(lambda x: x[2],
                                            nbr[:self.max_num_nbr])))
                nbr_fea.append(list(map(lambda x: x[1],
                                        nbr[:self.max_num_nbr])))

        nbr_fea_idx, nbr_fea = np.array(nbr_fea_idx), np.array(nbr_fea)
        nbr_fea = self.gdf.expand(nbr_fea)
        atom_fea = torch.Tensor(atom_fea)
        nbr_fea = torch.Tensor(nbr_fea)
        nbr_fea_idx = torch.LongTensor(nbr_fea_idx)
        target = torch.Tensor([float(target)])

        return (atom_fea, nbr_fea, nbr_fea_idx), target, cif_id
_____no_output_____

## cgcnn/cgcnn/model.py_____no_output_____

from __future__ import print_function, division

import torch
import torch.nn as nn_____no_output_____class ConvLayer(nn.Module):
    """
    Convolutional operation on graphs
    """
    def __init__(self, atom_fea_len, nbr_fea_len):
        """
        Initialize ConvLayer.

        Parameters
        ----------
        atom_fea_len: int
            Number of atom hidden features.
        nbr_fea_len: int
            Number of bond features.
        """
        super(ConvLayer, self).__init__()
        self.atom_fea_len = atom_fea_len
        self.nbr_fea_len = nbr_fea_len
        self.fc_full = nn.Linear(2 * self.atom_fea_len + self.nbr_fea_len,
                                 2 * self.atom_fea_len)
        self.sigmoid = nn.Sigmoid()
        self.softplus1 = nn.Softplus()
        self.bn1 = nn.BatchNorm1d(2 * self.atom_fea_len)
        self.bn2 = nn.BatchNorm1d(self.atom_fea_len)
        self.softplus2 = nn.Softplus()

    def forward(self, atom_in_fea, nbr_fea, nbr_fea_idx):
        """
        Forward pass

        N: Total number of atoms in the batch
        M: Max number of neighbors

        Parameters
        ----------
        atom_in_fea: Variable(torch.Tensor) shape (N, atom_fea_len)
            Atom hidden features before convolution
        nbr_fea: Variable(torch.Tensor) shape (N, M, nbr_fea_len)
            Bond features of each atom's M neighbors
        nbr_fea_idx: torch.LongTensor shape (N, M)
            Indices of M neighbors of each atom

        Returns
        -------
        atom_out_fea: nn.Variable shape (N, atom_fea_len)
            Atom hidden features after convolution
        """
        # TODO will there be problems with the index zero padding?
        N, M = nbr_fea_idx.shape

        # convolution
        atom_nbr_fea = atom_in_fea[nbr_fea_idx, :]
        total_nbr_fea = torch.cat(
            [atom_in_fea.unsqueeze(1).expand(N, M, self.atom_fea_len),
             atom_nbr_fea, nbr_fea], dim = 2)

        total_gated_fea = self.fc_full(total_nbr_fea)
        total_gated_fea = self.bn1(total_gated_fea.view(
            -1, self.atom_fea_len * 2)).view(N, M, self.atom_fea_len * 2)

        nbr_filter, nbr_core = total_gated_fea.chunk(2, dim = 2)
        nbr_filter = self.sigmoid(nbr_filter)
        nbr_core = self.softplus1(nbr_core)
        nbr_sumed = torch.sum(nbr_filter * nbr_core, dim = 1)
        nbr_sumed = self.bn2(nbr_sumed)
        out = self.softplus2(atom_in_fea + nbr_sumed)

        return out_____no_output_____class CrystalGraphConvNet(nn.Module):
    """
    Create a crystal graph convolutional neural network for predicting total
    material properties.
    """
    def __init__(self, orig_atom_fea_len, nbr_fea_len,
                 atom_fea_len = 64, n_conv = 3, h_fea_len = 128, n_h = 1,
                 classification = False):
        """
        Initialize CrystalGraphConvNet.

        Parameters
        ----------
        orig_atom_fea_len: int
            Number of atom features in the input.
        nbr_fea_len: int
            Number of bond features.
        atom_fea_len: int
            Number of hidden atom features in the convolutional layers
        n_conv: int
            Number of convolutional layers
        h_fea_len: int
            Number of hidden features after pooling
        n_h: int
            Number of hidden layers after pooling
        """
        super(CrystalGraphConvNet, self).__init__()
        self.classification = classification
        self.embedding = nn.Linear(orig_atom_fea_len, atom_fea_len)
        self.convs = nn.ModuleList([ConvLayer(atom_fea_len = atom_fea_len,
                                              nbr_fea_len = nbr_fea_len)
                                              for _ in range(n_conv)])
        self.conv_to_fc = nn.Linear(atom_fea_len, h_fea_len)
        self.conv_to_fc_softplus = nn.Softplus()

        if n_h > 1:
            self.fcs = nn.ModuleList([nn.Linear(h_fea_len, h_fea_len)
                                      for _ in range(n_h - 1)])
            self.softpluses = nn.ModuleList([nn.Softplus()
                                             for _ in range(n_h - 1)])

        if self.classification:
            self.fc_out = nn.Linear(h_fea_len, 2)
        else:
            self.fc_out = nn.Linear(h_fea_len, 1)

        if self.classification:
            self.logsoftmax = nn.LogSoftmax(dim = 1)
            self.dropout = nn.Dropout()

    def forward(self, atom_fea, nbr_fea, nbr_fea_idx, crystal_atom_idx):
        """
        Forward pass

        N: Total number of atoms in the batch
        M: Max number of neighbors
        N0: Total number of crystals in the batch

        Parameters
        ----------
        atom_fea: Variable(torch.Tensor) shape (N, orig_atom_fea_len)
            Atom features from atom type
        nbr_fea: Variable(torch.Tensor) shape (N, M, nbr_fea_len)
            Bond features of each atom's M neighbors
        nbr_fea_idx: torch.LongTensor shape (N, M)
            Indices of M neighbors of each atom
        crystal_atom_idx: list of torch.LongTensor of length N0
            Mapping from the crystal idx to atom idx

        Returns
        -------
        prediction: nn.Variable shape (N, )
            Atom hidden features after convolution
        """
        atom_fea = self.embedding(atom_fea)
        for conv_func in self.convs:
            atom_fea = conv_func(atom_fea, nbr_fea, nbr_fea_idx)

        crys_fea = self.pooling(atom_fea, crystal_atom_idx)
        crys_fea = self.conv_to_fc(self.conv_to_fc_softplus(crys_fea))
        crys_fea = self.conv_to_fc_softplus(crys_fea)

        if self.classification:
            crys_fea = self.dropout(crys_fea)

        if hasattr(self, 'fcs') and hasattr(self, 'softpluses'):
            for fc, softplus in zip(self.fcs, self.softpluses):
                crys_fea = softplus(fc(crys_fea))

        out = self.fc_out(crys_fea)

        if self.classification:
            out = self.logsoftmax(out)

        return out

    def pooling(self, atom_fea, crystal_atom_idx):
        """
        Pooling the atom features to crystal features

        N: Total number of atoms in the batch
        N0: Total number of crystals in the batch

        Parameters
        ----------
        atom_fea: Variable(torch.Tensor) shape (N, atom_fea_len)
            Atom feature vectors of the batch
        crystal_atom_idx: list of torch.LongTensor of length N0
            Mapping from the crystal idx to atom idx
        """
        assert sum([len(idx_map) for idx_map in crystal_atom_idx]) == \
            atom_fea.data.shape[0]
        summed_fea = [torch.mean(atom_fea[idx_map], dim = 0, keepdim = True)
                      for idx_map in crystal_atom_idx]
        return torch.cat(summed_fea, dim = 0)
_____no_output_____

## cgcnn/main.py_____no_output_____

import argparse
import os
import shutil
import sys
import time
import warnings
from random import sample

import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from sklearn import metrics
from torch.autograd import Variable
from torch.optim.lr_scheduler import MultiStepLR

#from cgcnn.data import CIFData
#from cgcnn.data import collate_pool, get_train_val_test_loader
#from cgcnn.model import CrystalGraphConvNet_____no_output__________no_output__________no_output_____class Normalizer(object):
    """ Normalize a Tensor and restore it later. """

    def __init__(self, tensor):
        """ tensor is taken as a sample to calculate the mean and std """
        self.mean = torch.mean(tensor)
        self.std = torch.std(tensor)

    def norm(self, tensor):
        return (tensor - self.mean) / self.std

    def denorm(self, normed_tensor):
        return normed_tensor * self.std + self.mean

    def state_dict(self):
        return {'mean': self.mean,
                'std': self.std}

    def load_state_dict(self, state_dict):
        self.mean = state_dict['mean']
        self.std = state_dict['std']
        _____no_output_____def mae(prediction, target):
    """
    Computes the mean absolute error between prediction and target

    Parameters
    ----------
    prediction: torch.Tensor (N, 1)
    target: torch.Tensor (N, 1)
    """
    return torch.mean(torch.abs(target - prediction))_____no_output_____def class_eval(prediction, target):

    prediction = np.exp(prediction.numpy())
    target = target.numpy()

    pred_label = np.argmax(prediction, axis = 1)
    target_label = np.squeeze(target)

    if not target_label.shape:
        target_label = np.asarray([target_label])

    if prediction.shape[1] == 2:
        precision, recall, fscore, _ = metrics.precision_recall_fscore_support(
            target_label, pred_label, average = 'binary')
        auc_score = metrics.roc_auc_score(target_label, prediction[:, 1])
        accuracy = metrics.accuracy_score(target_label, pred_label)
    else:
        raise NotImplementedError
    return accuracy, precision, recall, fscore, auc_score_____no_output_____class AverageMeter(object):
    """ Computes and stores the average and current value """

    def __init__(self):
        self.reset()

    def reset(self):
        self.val = 0
        self.avg = 0
        self.sum = 0
        self.count = 0

    def update(self, val, n = 1):
        self.val = val
        self.sum += val * n
        self.count += n
        self.avg = self.sum / self.count_____no_output_____def save_checkpoint(state, is_best, filename="checkpoint.pth.tar"):
    torch.save(state, filename)
    if is_best:
        shutil.copyfile(filename, 'model_best.pth.tar')_____no_output_____def adjust_learning_rate(optimizer, epoch, k):
    """ Sets the learning rate to the initial LR decayed by 10 every k epochs """
    assert type(k) is int
    lr = args.lr * (0.1 ** (epoch // k))
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr
        _____no_output_____def train(train_loader, model, criterion, optimizer, epoch, normalizer):

    batch_time = AverageMeter()
    data_time = AverageMeter()
    losses = AverageMeter()

    if args.task == 'regression':
        mae_errors = AverageMeter()
    else:
        accuracies = AverageMeter()
        precisions = AverageMeter()
        recalls = AverageMeter()
        fscores = AverageMeter()
        auc_scores = AverageMeter()

    # switch to train mode
    model.train()

    end = time.time()

    for i, (input, target, _) in enumerate(train_loader):

        # measure data loading time
        data_time.update(time.time() - end)

        if args.cuda:
            input_var = (Variable(input[0].cuda(non_blocking = True)),
                         Variable(input[1].cuda(non_blocking = True)),
                         input[2].cuda(non_blocking = True),
                         [crys_idx.cuda(non_blocking = True) for crys_idx in input[3]])
        else:
            input_var = (Variable(input[0]),
                         Variable(input[1]),
                         input[2],
                         input[3])

        # normalize target
        if args.task == 'regression':
            target_normed = normalizer.norm(target)
        else:
            target_normed = target.view(-1).long()

        if args.cuda:
            target_var = Variable(target_normed.cuda(non_blocking = True))
        else:
            target_var = Variable(target_normed)

        # compute output
        output = model(*input_var)
        loss = criterion(output, target_var)

        # measure accuracy and record loss
        if args.task == 'regression':
            mae_error = mae(normalizer.denorm(output.data.cpu()), target)
            losses.update(loss.data.cpu(), target.size(0))
            mae_errors.update(mae_error, target.size(0))
        else:
            accuracy, precision, recall, fscore, auc_score = \
                class_eval(output.data.cpu(), target)
            losses.update(loss.data.cpu().item(), target.size(0))
            accuracies.update(accuracy, target.size(0))
            precisions.update(precision, target.size(0))
            recalls.update(recall, target.size(0))
            fscores.update(fscore, target.size(0))
            auc_scores.update(auc_score, target.size(0))

        # compute gradient and do SGD step
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # measure elapsed time
        batch_time.update(time.time() - end)
        end = time.time()

        if i % args.print_freq == 0:
            if args.task == 'regression':
                print('Epoch: [{0}][{1}/{2}]\t'
                      'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
                      'Data {data_time.val:.3f} ({data_time.avg:.3f})\t'
                      'Loss {loss.val:.4f} ({loss.avg:.4f})\t'
                      'MAE {mae_errors.val:.3f} ({mae_errors.avg:.3f})'.format(
                          epoch, i, len(train_loader), batch_time = batch_time,
                          data_time = data_time, loss = losses, mae_errors = mae_errors))
            else:
                print('Epoch: [{0}][{1}/{2}]\t'
                      'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
                      'Data {data_time.val:.3f} ({data_time.avg:.3f})\t'
                      'Loss {loss.val:.4f} ({loss.avg:.4f})\t'
                      'Accu {accu.val:.3f} ({accu.avg:.3f})\t'
                      'Precision {prec.val:.3f} ({prec.avg:.3f})\t'
                      'Recall {recall.val:.3f} ({recall.avg:.3f})\t'
                      'F1 {f1.val:.3f} ({f1.avg:.3f})\t'
                      'AUC {auc.val:.3f} ({auc.avg:.3f})'.format(
                          epoch, i, len(train_loader), batch_time = batch_time,
                          data_time = data_time, loss = losses, accu = accuracies,
                          prec = precisions, recall = recalls, f1 = fscores,
                          auc = auc_scores))
_____no_output_____def validate(val_loader, model, criterion, normalizer, test = False):

    batch_time = AverageMeter()
    losses = AverageMeter()

    if args.task == 'regression':
        mae_errors = AverageMeter()
    else:
        accuracies = AverageMeter()
        precisions = AverageMeter()
        recalls = AverageMeter()
        fscores = AverageMeter()
        auc_scores = AverageMeter()

    if test:
        test_targets = []
        test_preds = []
        test_cif_ids = []

    # switch to evaluate mode
    model.eval()

    end = time.time()

    for i, (input, target, batch_cif_ids) in enumerate(val_loader):

        if args.cuda:
            with torch.no_grad():
                input_var = (Variable(input[0].cuda(non_blocking = True)),
                             Variable(input[1].cuda(non_blocking = True)),
                             input[2].cuda(non_blocking = True),
                             [crys_idx.cuda(non_blocking = True) for crys_idx in input[3]])
        else:
            with torch.no_grad():
                input_var = (Variable(input[0]),
                             Variable(input[1]),
                             input[2],
                             input[3])
        if args.task == 'regression':
            target_normed = normalizer.norm(target)
        else:
            target_normed = target.view(-1).long()

        if args.cuda:
            with torch.no_grad():
                target_var = Variable(target_normed.cuda(non_blocking = True))
        else:
            with torch.no_grad():
                target_var = Variable(target_normed)

        # compute output
        output = model(*input_var)
        loss = criterion(output, target_var)

        # measure accuracy and record loss
        if args.task == 'regression':
            mae_error = mae(normalizer.denorm(output.data.cpu()), target)
            losses.update(loss.data.cpu().item(), target.size(0))
            mae_errors.update(mae_error, target.size(0))

            if test:
                test_pred = normalizer.denorm(output.data.cpu())
                test_target = target
                test_preds += test_pred.view(-1).tolist()
                test_targets += test_target.view(-1).tolist()
                test_cif_ids += batch_cif_ids
        else:
            accuracy, precision, recall, fscore, auc_score = \
                class_eval(output.data.cpu(), target)

            losses.update(loss.data.cpu().item(), target.size(0))
            accuracies.update(accuracy, target.size(0))
            precisions.update(precision, target.size(0))
            recalls.update(recall, target.size(0))
            fscores.update(fscore, target.size(0))
            auc_scores.update(auc_score, target.size(0))

            if test:
                test_pred = torch.exp(output.data.cpu())
                test_target = target
                assert test_pred.shape[1] == 2
                test_preds += test_pred[:, 1].tolist()
                test_targets += test_target.view(-1).tolist()
                test_cif_ids += batch_cif_ids

        # measure elapsed time
        batch_time.update(time.time() - end)
        end = time.time()

        if i % args.print_freq == 0:
            if args.task == 'regression':
                print('Test: [{0}/{1}]\t'
                      'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
                      'Loss {loss.val:.4f} ({loss.avg:.4f})\t'
                      'MAE {mae_errors.val:.3f} ({mae_errors.avg:.3f})'.format(
                          i, len(val_loader), batch_time = batch_time, loss = losses,
                          mae_errors = mae_errors))
            else:
                print('Test: [{0}/{1}]\t'
                      'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
                      'Loss {loss.val:.4f} ({loss.avg:.4f})\t'
                      'Accu {accu.val:.3f} ({accu.avg:.3f})\t'
                      'Precision {prec.val:.3f} ({prec.avg:.3f})\t'
                      'Recall {recall.val:.3f} ({recall.avg:.3f})\t'
                      'F1 {f1.val:.3f} ({f1.avg:.3f})\t'
                      'AUC {auc.val:.3f} ({auc.avg:.3f})'.format(
                          i, len(val_loader), batch_time = batch_time, loss = losses,
                          accu = accuracies, prec = precisions, recall = recalls,
                          f1 = fscores, auc = auc_scores))
    if test:
        star_label = '**'
        import csv
        with open('test_results.csv', 'w') as f:
            writer = csv.writer(f)

            for cif_id, target, pred in zip(test_cif_ids, test_targets, test_preds):
                writer.writerow((cif_id, target, pred))
    else:
        star_label = '*'

    if args.task == 'regression':
        print(' {star} MAE {mae_errors.avg:.3f}'.format(star = star_label,
                                                        mae_errors = mae_errors))
        return mae_errors.avg
    else:
        print(' {star} AUC {auc.avg:.3f}'.format(star = star_label,
                                                 auc = auc_scores))
        return auc_scores.avg
        _____no_output__________no_output__________no_output__________no_output__________no_output__________no_output__________no_output__________no_output__________no_output__________no_output_____


# Notebook from wckdouglas/cfNA
Path: plots/peak_analysis.ipynb

<code>
%matplotlib inline
%load_ext autoreload
%autoreload 2

import pandas as pd
import numpy as np
import os
import matplotlib.pyplot as plt
import matplotlib.lines as mlines
import seaborn as sns
from sequencing_tools.viz_tools import okabeito_palette, color_encoder, simpsons_palette
from sequencing_tools.stats_tools import p_adjust
from scipy.special import ndtr
from collections import defaultdict
from sequencing_tools.fastq_tools import reverse_complement
from sequencing_tools.bam_tools import get_strand
import RNA
from multiprocessing import Pool
import mygene
import random
import pysam
import glob
import re
#import genomeview
from pybedtools import BedTool, set_tempdir
import mappy as mp
from plotting_utils import figure_path
from matplotlib import rcParams
from peak_utils import *
from tblout_parser import read_tbl
import sys
sys.path.insert(0,'/stor/home/cdw2854/cfNA/peak_callings')
from structural_peaks import PeakAnalyzer, mRNAFilter
from exon_coverage import ExonFilter
plt.rc('axes', labelsize=20)
plt.rc('xtick', labelsize = 20)
plt.rc('ytick', labelsize = 20)
plt.rc('font', **{'family':'sans-serif',
                  'sans-serif':'Arial'})
Peak_analyzer = PeakAnalyzer()
mrna_filter = mRNAFilter()
exon_filter = ExonFilter()The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
project_path = '/stor/work/Lambowitz/cdw2854/cfNA/tgirt_map'
project_path = '/stor/work/Lambowitz/yaojun/Work/cfNA/tgirt_map'
set_tempdir(project_path)
peak_path = project_path + '/bed_files/merged_bed/MACS2/annotated'
#peak_path = project_path + '/CLAM//BED_files/peaks/annotation'
peak_tsv = peak_path + '/unfragmented.filtered.tsv'
print(peak_tsv)
peak_df = load_peaks(peak_tsv)  \
    .assign(sense_gtype = lambda d: np.where(d.sense_gtype == ".", 'Unannotated', d.sense_gtype))\
    .assign(antisense_gtype = lambda d: np.where(d.antisense_gtype == ".", 'Unannotated', d.antisense_gtype)) \
    .sort_values('pileup', ascending=False)
peak_df.head()/stor/work/Lambowitz/yaojun/Work/cfNA/tgirt_map/bed_files/merged_bed/MACS2/annotated/unfragmented.filtered.tsv
fig = plt.figure(figsize=(10,10))
size_ax = fig.add_subplot(221)
cov_ax = fig.add_subplot(222)
number_ax = fig.add_subplot(224)
dist_cov_ax = fig.add_subplot(223)
#peak_annotation_ax = fig.add_subplot(224)

plot_peak_size(peak_df, ax = size_ax)
size_ax.legend(fontsize=15, frameon=False)
plot_peak_coverage(peak_df, cov_ax, log=True)
#plot_cov_density(peak_df, dist_cov_ax)
plot_peak_cum_cov(peak_df, dist_cov_ax)
plot_peak_number(peak_df, number_ax, ce)
#### add hepG2
#combined_peaks = pd.concat([peak_df.assign(annotation = 'K562'),
#                        hep_peak_df.assign(annotation = 'K562 + HepG2')])
#plot_peak_bar(peak_annotation_ax, combined_peaks)
fig.tight_layout()

figurename = figure_path + '/peak_qc.pdf'
fig.savefig(figurename, bbox_inches = 'tight')
print('Plotted: ', figurename)/stor/work/Lambowitz/cdw2854/src/miniconda3/lib/python3.6/site-packages/scipy/stats/stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval
/stor/work/Lambowitz/cdw2854/src/miniconda3/lib/python3.6/site-packages/numpy/core/_methods.py:140: RuntimeWarning: Degrees of freedom <= 0 for slice
  keepdims=keepdims)
/stor/work/Lambowitz/cdw2854/src/miniconda3/lib/python3.6/site-packages/numpy/core/_methods.py:132: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
/stor/home/cdw2854/cfNA/plots/peak_utils.py:584: UserWarning: Attempted to set non-positive left xlim on a log-scaled axis.
Invalid limit will be ignored.
  ax.set_xlim(0, 1e4)
peak_df = peak_df\
    .query('sample_count >= %i & pileup >= %i' %(sample_cutoff, pileup_cutoff))\
    .assign(transcriptome = lambda d: list(map(transcriptome_map, d.chrom, d.start, d.end, d.strand)))\
    .assign(is_exon = lambda d: [mrna_filter.search(chrom, start, end, attribute = 'exon') for chrom, start, end in zip(d.chrom, d.start, d.end)]) \
    #.assign(is_transcriptome_peak = lambda d: [mrna_filter.search(chrom, start, end, attribute = 'transcriptome') for chrom, start, end in zip(d.chrom, d.start, d.end)])\
    #.pipe(lambda d: d[(d.is_exon=="no") | ((d.is_exon=="yes") & (d.is_transcriptome_peak=="yes"))])
peak_df.shape_____no_output_____peak_df.assign(l = lambda d: d.end-d.start).describe()#query('l == 998')_____no_output_____rbps = set()
for x in peak_df.query('sense_gtype=="RBP"').sense_gname.tolist():
    for rbp in x.split(','):
        rbps.add(rbp)
len(rbps)_____no_output_____peak_df\
    .query('sense_gtype=="Long RNA"')\
    .sort_values('log10p', ascending=False)\
    .filter(['chrom','start','end','sense_gname','peakname'])\
    .head(20)_____no_output_____exon_filter = ExonFilter()
needed_columns = ['chrom','start','end','peakname','score','strand']
needed_columns.extend(list(set(peak_df.columns.tolist()) - set(needed_columns)))
exon_filtered_peaks = peak_df\
    .filter(needed_columns)
exon_filtered_peaks = exon_filter.filter(exon_filtered_peaks, f = 0.1)
exon_filtered_peaks.head()Filtered out 355 peaks
fig = plt.figure(figsize=(10,10))

strand_ax = fig.add_axes([-0.1, 0.52, 0.45, 0.45])
pie_ax = fig.add_axes([0.55, 0.52, 0.5, 0.5])
#rbp_ax = fig.add_axes([-0.1, 0, 0.35, 0.5])
#long_ax = fig.add_axes([0.38, 0, 0.35, 0.5])
rbp_ax = fig.add_axes([-0.1, -0.15, 0.45, 0.45])
long_ax = fig.add_axes([0.55, -0.15, 0.45, 0.45])

#misc_ax = fig.add_axes([0.84, 0, 0.35, 0.5])

top_n = 15
plot_peak_strand(peak_df, strand_ax)
sense_peaks = peak_df.query('is_sense == "Sense"')
plot_peak_pie(sense_peaks, pie_ax, ce)
#plot_repeats_RNA(sense_peaks, misc_ax, ce, rnatype='Repeats', top_n = top_n)
rbp_df = plot_rbp(sense_peaks, rbp_ax, ce, top_n = top_n)
plot_long_RNA_peak(exon_filtered_peaks, long_ax, ce, top_n = top_n, y_val = 'log10p')
used = set()
for xt in  long_ax.get_xticklabels():
    if xt.get_text() in {'AB019441.29','RPS2P55','RPL13AP25'}:
        rt = 'Pseudogene'
    elif xt.get_text() in 'DAPK1':
        rt = 'miRNA'
    elif re.search('PKD|ARHG',xt.get_text()):
        rt = 'Full-length intron'
    else:
        rt = 'Within intron'
    xt.set_color(peak_type_ce.encoder[rt])
    used.add(rt)
cc_ce = color_encoder()
cc_ce.encoder = {k:v for k,v in peak_type_ce.encoder.items() if k in used}
cc_ce.show_legend(ax = long_ax, frameon=False, fontsize=20)
sns.despine()

fig.text(-0.09,0.32, 'RBP-binding-site peaks', fontsize=20)
fig.text(0.56,0.32, 'Intron peaks', fontsize=20)
#fig.text(0.85,0.5, 'Repeats', fontsize=18)

figure_name = figure_path + '/peak_figure.pdf'
fig.savefig(figure_name, bbox_inches = 'tight')
print('Saved:', figure_name)No handles with labels found to put in legend.
No handles with labels found to put in legend.
***** WARNING: File /stor/work/Lambowitz/ref/hg19_ref/genes/introns.gencode.bed.gz has inconsistent naming convention for record:
GL000192.1	430271	438553	ENST00000621489.1_1	2	-	intron

***** WARNING: File /stor/work/Lambowitz/ref/hg19_ref/genes/introns.gencode.bed.gz has inconsistent naming convention for record:
GL000192.1	430271	438553	ENST00000621489.1_1	2	-	intron

/stor/work/Lambowitz/cdw2854/src/miniconda3/lib/python3.6/site-packages/pybedtools/bedtool.py:3439: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
  return pandas.read_table(self.fn, *args, **kwargs)
No handles with labels found to put in legend.
exon_filtered_peaks\
    .query('sense_gtype=="Long RNA"')\
    .sort_values('log10p', ascending=False)\
    .filter(['chrom','start','end','sense_gname','peakname'])\
    .head(20)_____no_output_____ax = plt.subplot()
sense_peaks = exon_filtered_peaks.query('is_sense == "Sense"')
plot_peak_pie(sense_peaks, ax, ce)_____no_output_____fig = plt.figure(figsize=(8,8))

#strand_ax = fig.add_axes([-0.1, 0.52, 0.45, 0.45])
#pie_ax = fig.add_axes([0.55, 0.52, 0.5, 0.5])
#rbp_ax = fig.add_axes([-0.1, 0, 0.35, 0.5])
#long_ax = fig.add_axes([0.38, 0, 0.35, 0.5])
#rbp_ax = fig.add_axes([-0.1, -0.15, 0.45, 0.45])
long_ax = fig.add_subplot(111)#fig.add_axes([0.55, -0.15, 0.45, 0.45])

#misc_ax = fig.add_axes([0.84, 0, 0.35, 0.5])

top_n = 15
#plot_peak_strand(exon_filtered_peaks, strand_ax)
sense_peaks = exon_filtered_peaks.query('is_sense == "Sense"')
#plot_peak_pie(exon_filtered_peaks, pie_ax, ce)
#plot_repeats_RNA(sense_peaks, misc_ax, ce, rnatype='Repeats', top_n = top_n)
#rbp_df = plot_rbp(exon_filtered_peaks, rbp_ax, ce, top_n = top_n)
plot_long_RNA_peak(peak_df, long_ax, ce, top_n = top_n, y_val = 'log10p')
used=set()
for xt in  long_ax.get_xticklabels():
    if xt.get_text() in {'HBB','HBA2','HBA1','PPBP','PF4','FTL','NRGN','ACTB'}:
        rt = 'mRNA'
    elif xt.get_text() in {'TMSB4XP8','TMSL6','AC016739.2','RP11-543P15.1',
                          'RP11-3P17.3','FGF7P6','RPL13AP25'}:
        rt = 'Pseudogene'
    elif xt.get_text() in 'DAPK1':
        rt='miRNA'
    else:
        rt = 'Within introns'
    xt.set_color(peak_type_ce.encoder[rt])
    used.add(rt)
cc_ce = color_encoder()
cc_ce.encoder = {k:v for k,v in peak_type_ce.encoder.items() if k in used}
cc_ce.show_legend(ax = long_ax, frameon=False, fontsize=20)
sns.despine()

figure_name = figure_path + '/peak_figure_filtered.pdf'
fig.savefig(figure_name, bbox_inches = 'tight')
print('Saved:', figure_name)***** WARNING: File /stor/work/Lambowitz/ref/hg19_ref/genes/introns.gencode.bed.gz has inconsistent naming convention for record:
GL000192.1	430271	438553	ENST00000621489.1_1	2	-	intron

***** WARNING: File /stor/work/Lambowitz/ref/hg19_ref/genes/introns.gencode.bed.gz has inconsistent naming convention for record:
GL000192.1	430271	438553	ENST00000621489.1_1	2	-	intron

/stor/work/Lambowitz/cdw2854/src/miniconda3/lib/python3.6/site-packages/pybedtools/bedtool.py:3439: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
  return pandas.read_table(self.fn, *args, **kwargs)
No handles with labels found to put in legend.

</code>