# Notebook from andim/transitions-paper
Path: iidswitching.ipynb

# Figure 4: Transitions between switching and non-switching strategies depend on environmental selectivity and environmental frequencies

Goal: show analytical results on when switching is optimal in temporally uncorrelated environments_____no_output_____

<code>
import numpy as np

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use(['transitions.mplstyle'])
import matplotlib
colors = matplotlib.rcParams['axes.prop_cycle'].by_key()['color']
black = matplotlib.rcParams['text.color']
width = matplotlib.rcParams['axes.linewidth']
from matplotlib import colors as mplcolors

import sys
sys.path.append('lib/')
import plotting
import evolimmune_____no_output_____def func(pienv, s):
    return np.clip((pienv*(2.0-s) + s-1.0)/s, 0, 1)

s = np.linspace(0, 1)
fig, axes = plt.subplots(figsize=(6.6, 2.7), ncols=2)

ax = axes[0]
cmap = mplcolors.LinearSegmentedColormap.from_list('mycmap', [colors[0], colors[1]])
CS = ax.contour(func(s[:,None], s[None,:]), extent=(0, 1, 0, 1), cmap=cmap, levels=np.arange(0.2, 1.0, 0.2))
ax.clabel(CS, inline=1, fmt='%g', manual=[(0.9, 0.2), (0.9, 0.4), (0.9, 0.6), (0.9, 0.8)])
ax.plot(s, (1 - s)/(2 - s), color=black, lw=1.8)
ax.plot(s, 1/(2 - s), color=black, lw=1.8)
textkwargs = dict(ha='center', va='center')
ax.text(0.35, 0.85, r'\bf only $\bm{\sigma=2}$', color=colors[1], **textkwargs)
ax.text(0.35, 0.16, r'\bf only $\bm{\sigma=1}$', color=colors[0], **textkwargs)
ax.text(0.7, 0.5, r'\bf switching', color=black, **textkwargs)
arrowprops = dict(edgecolor=black, facecolor=black, arrowstyle='-|>', lw=width)
ax.annotate("", xy=(0.99, 0.9), xytext=(0.99, 0.1), arrowprops=arrowprops)
ax.text(1.03, 0.5, 'frequency of phenotype 2,\n'+r'$\pi^\star(\sigma=2)$', rotation=90,
        va='center', ha='left', multialignment='center')
ax.set_xlabel('selection coefficient,\n $1-w$')
ax.set_ylabel('frequency of environment 2,\n$p(x=2)$')

ax = axes[1]
s0 = np.logspace(-3, 0, 100)
ps = [0.2, 0.5, 0.8]
for i, p in enumerate(ps):
    l, = ax.plot(s0, s0/(s0+p/(1.0-p)), label='%g'%p, c=colors[i+3])
    ax.plot(s0, (s0/(1.0-s0))*((1.0-p)/p), c=l.get_color())
    ax.fill_between(s0, s0/(s0+p/(1.0-p)), (s0/(1.0-s0))*((1.0-p)/p), color=l.get_color(), alpha=.5)
ax.set_xscale('log')
ax.set_yscale('log')
ax.set_xlim(0.01, 1)
ax.set_ylim(0.01, 1)
ax.legend(title='$p(x=2)$', loc='upper left', ncol=1)
ax.set_xlabel('selection coefficient env. 1,\n$1-w(x=1)$')
ax.set_ylabel('selection coefficient env. 2,\n$1-w(x=2)$')

for ax in axes:
    plotting.despine(ax)
plotting.label_axes(axes, xy=(-0.2, 0.975))
fig.tight_layout(w_pad=4.0)
fig.savefig('svgs/iidswitching.svg')/home/andreas/miniconda2/envs/transitions/lib/python2.7/site-packages/ipykernel/__main__.py:2: RuntimeWarning: divide by zero encountered in divide
  from ipykernel import kernelapp as app
/home/andreas/miniconda2/envs/transitions/lib/python2.7/site-packages/ipykernel/__main__.py:29: RuntimeWarning: divide by zero encountered in divide
/home/andreas/miniconda2/envs/transitions/lib/python2.7/site-packages/ipykernel/__main__.py:30: RuntimeWarning: divide by zero encountered in divide

</code>

**Transitions between switching and non-switching strategies depend on environmental selectivity and environmental frequencies.** In a temporally uncorrelated environment changing randomly between two states, $1$ and $2$, a population of organisms is adapted optimally by either being in a single phenotypic state or by having a mixture of phenotypes (bet-hedging) depending on the statistics of the environment and the degree to which the phenotypes are specialized. In environment $x=1$ ($2$) phenotype $2$ ($1$) has replication rate $w_x$ relative to the other phenotype.
(A) Transitions as a function of specialization level and environmental frequency in the symmetric case, $w_1 = w_2 = w$. The black lines mark the transition from single-phenotype to bet-hedging strategies: above the upper (lower) line the entire population optimally has phenotype 2 (1), between the two lines phenotypic diversification provides an advantage. The optimal fraction of phenotype $2$ in the bet-hedging region is shown by the colored lines.
(B) Regions of selection factors in which bet-hedging is the preferred strategy (shaded areas) for environments with different frequencies of being in state $2$. Either strong selection or a precise mapping between the relative selection factors and the relative environmental frequencies are needed to make bet-hedging optimal. _____no_output_____


# Notebook from Famingzhao/scMVP
Path: demos/manuscript_analysis/paired_cellline_demo.ipynb

<code>
%matplotlib inline

import os
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
from scMVP.dataset import LoadData,GeneExpressionDataset, CellMeasurement
from scMVP.models import VAE_Attention, Multi_VAE_Attention, VAE_Peak_SelfAttention
from scMVP.inference import UnsupervisedTrainer
from scMVP.inference import MultiPosterior, MultiTrainer
import torch

import scanpy as sc
import anndata

import scipy.io as sp_io
from scipy.sparse import csr_matrix, issparse_____no_output_____torch.set_num_threads(40)_____no_output_____input_path = "./paired_cellline/"
output_path = "./paired_cellline/scMVP_output"

paired_cellline_dataset = {
                "gene_names": 'paired_cellline_scale_gene.txt',
                "gene_expression": 'paired_cellline_rna_normalize_count.mtx',
                "gene_barcodes": 'paired_cellline_cell_barcode.txt',
                "atac_names": 'paired_cellline_peak.txt',
                "atac_expression": 'paired_cellline_atac_normalize_count.mtx',
                "atac_barcodes": 'paired_cellline_cell_barcode.txt'
                }
dataset = LoadData(dataset=paired_cellline_dataset,data_path=input_path,
                       dense=False,gzipped=False, atac_threshold=0.001,
                       cell_threshold=1)[2021-06-28 19:09:04,815] INFO - scMVP.dataset.scMVP_dataloader | Preprocessing joint profiling dataset.
[2021-06-28 19:09:07,180] INFO - scMVP.dataset.scMVP_dataloader | Finished preprocessing dataset
[2021-06-28 19:09:07,299] INFO - scMVP.dataset.dataset | Remapping labels to [0,N]
[2021-06-28 19:09:07,301] INFO - scMVP.dataset.dataset | Remapping batch_indices to [0,N]
[2021-06-28 19:09:07,729] INFO - scMVP.dataset.dataset | Computing the library size for the new data
[2021-06-28 19:09:07,787] INFO - scMVP.dataset.dataset | Downsampled from 3256 to 3256 cells
#####
lr = 5e-3
use_batches = False
use_cuda = True
n_alfa = 1.0
n_centroids=10

# joint RNA and ATAC embedding
multi_vae = Multi_VAE_Attention(dataset.nb_genes, len(dataset.atac_names), n_batch=0, n_latent=20, n_centroids=n_centroids, n_alfa = n_alfa, mode="mm-vae") # should provide ATAC num, alfa, mode and loss type
trainer = MultiTrainer(
    multi_vae,
    dataset,
    train_size=0.9,
    use_cuda=use_cuda,
    frequency=5,
)

if os.path.exists('%s/multi_vae_trainer.pkl' % output_path):
    trainer.model.load_state_dict(torch.load('%s/multi_vae_trainer.pkl' % output_path))
else:
    trainer.train(n_epochs=20, lr=lr)
    torch.save(trainer.model.state_dict(), '%s/multi_vae_trainer.pkl' % output_path)

trainer.model.eval()

full = trainer.create_posterior(trainer.model, dataset, indices=np.arange(len(dataset)),type_class=MultiPosterior)
latent, latent_rna, latent_atac, cluster_gamma, cluster_index, batch_indices, labels = full.sequential().get_latent()
batch_indices = batch_indices.ravel()
imputed_values = full.sequential().imputation()_____no_output_____# visulization
prior_adata = anndata.AnnData(X=latent)
prior_adata.obsm["X_multi_vi"] = latent
prior_adata.obs['cell_type'] = torch.tensor(labels.reshape(-1,1))
sc.pp.neighbors(prior_adata, use_rep="X_multi_vi", n_neighbors=30)
sc.tl.umap(prior_adata, min_dist=0.3)
sc.tl.louvain(prior_adata)
sc.pl.umap(prior_adata, color=['louvain'])
plt.show()

# color with kmeans cluster
from sklearn.cluster import KMeans
latent_code = prior_adata.obsm["X_umap"]

kmeans = KMeans(n_clusters=3, random_state=0).fit(latent_code)
prior_adata.obs['kmeans'] = kmeans.labels_.astype(str)
sc.pl.umap(prior_adata, color=['kmeans'])_____no_output_____
# save file
df = pd.DataFrame(data=prior_adata.obsm["X_multi_vi"],  index=dataset.barcodes)
df.to_csv(os.path.join(output_path,"multivae_latent_imputation.csv"))

df = pd.DataFrame(data=prior_adata.obsm["X_umap"],  columns=["umap_dim1","umap_dim2"] , index=dataset.barcodes)
df.insert(0,"kmeans",prior_adata.obs['kmeans'].values)
df.to_csv(os.path.join(output_path,"multivae_umap.csv"))

df = pd.DataFrame(data=imputed_values[1].T, columns=dataset.barcodes, index=dataset.atac_names)
df.to_csv(os.path.join(output_path,"atac_multivae_imputation.csv"))

df = pd.DataFrame(data=imputed_values[0].T, columns=dataset.barcodes, index=dataset.gene_names)
df.to_csv(os.path.join(output_path,"gene_multivae_imputation.csv"))_____no_output_____
</code>


# Notebook from mniehus/QMlabs
Path: Lab 7 - Time Evolution.ipynb

# Lab 7 - Time Evolution
Exploring time evolution of quantum states. Run the usual imports, and use the spin-1/2 states as previously defined:_____no_output_____

import matplotlib.pyplot as plt
from numpy import sqrt,pi,arange,cos,sin
from qutip import *_____no_output_____%matplotlib inline_____no_output_____pz = Qobj([[1],[0]])
mz = Qobj([[0],[1]])
px = Qobj([[1/sqrt(2)],[1/sqrt(2)]])
mx = Qobj([[1/sqrt(2)],[-1/sqrt(2)]])
py = Qobj([[1/sqrt(2)],[1j/sqrt(2)]])
my = Qobj([[1/sqrt(2)],[-1j/sqrt(2)]])
Sx = 1/2.0*sigmax()
Sy = 1/2.0*sigmay()
Sz = 1/2.0*sigmaz()_____no_output_____

Define the Hamiltonian:
$$H= - \mathbf{\mu}\cdot \mathbf{B} =-\gamma S_z B$$
$$\hat{H} = -\Omega \hat{S}_z$$_____no_output_____

Omega = 5
H = -Omega*Sz_____no_output_____t = arange(0,4*pi/Omega,0.05)_____no_output_____

The next line calls a Schrödinger equation solver (sesolve). It's arguments are the Hamiltonian, the starting state $\lvert+x\rangle$ (px), the time values, and a list of operators. `sesolve` returns many things, but the `expect` method is most useful, it gives the expectation values of the three operators in the operator list._____no_output_____

expect_ops = [Sx,Sy,Sz]
result1 = sesolve(H, px, t, expect_ops)_____no_output_____expect_ops[0] # TODO get name of variable to use in label_____no_output_____labels = ['x','y','z']
for r,l in zip(result1.expect,labels):
    plt.plot(result1.times*Omega/pi, r, label="$\langle S_%c \\rangle $" % l)

plt.xlabel("Time ($\Omega t/\pi$)", size=18)
plt.legend(fontsize=16) #TODO fix legend text size_____no_output_____

Now what if the system starts in $\lvert+z\rangle$?_____no_output_____

result2 = sesolve(H, pz, t, [Sx,Sy,Sz])_____no_output_____for r,l in zip(result2.expect,labels):
    plt.plot(result2.times*Omega/pi, r, label="$\langle S_%c \\rangle $" % l)

plt.xlabel("Time ($\Omega t/\pi$)", size=18)
plt.legend(fontsize=16) #TODO fix legend text size_____no_output_____

Spin-up stays spin-up (i.e. no prescession)_____no_output_____## Two particle systems:
$\lvert\psi\rangle = \frac{1}{\sqrt{2}} \lvert+z,-z\rangle + \frac{1}{\sqrt{2}} \lvert-z,+z\rangle$

Use the `tensor` QuTiP function to form multi-particle states_____no_output_____

psi = 1/sqrt(2)*tensor(pz, mz) + 1/sqrt(2)*tensor(mz, pz)_____no_output_____

Hamiltonian is the same for both particles so we use the tensor to form $\hat{H}$ from individual operators_____no_output_____

omega = 5
H = -omega*tensor(Sz,Sz)_____no_output_____expect_op = tensor(mz,pz)*tensor(mz,pz).dag()
result3 = sesolve(H, psi, t, expect_op)_____no_output_____for r,l in zip(result3.expect,labels):
    plt.plot(result3.times*omega/pi, r, label="$\langle -z,+z\\rangle$")

plt.xlabel("Time ($\Omega t/\pi$)", size=18)
plt.legend(fontsize=16) #TODO fix legend text size_____no_output_____

The value is constant since the state is initially in an eigenstate of $\hat{H}$._____no_output_____## What if the magnetic field is not along an axis?_____no_output_____Notice the Hamiltonian has an $x$ and a $z$ component:_____no_output_____

omega=2
H = -omega/sqrt(2)*(Sz + Sx)
t = arange(0,2*pi/omega,0.05)_____no_output_____result4 = sesolve(H, px, t, [Sx, Sy, Sz])_____no_output_____for r,l in zip(result4.expect,labels):
    plt.plot(result4.times*omega/pi, r, label="$\langle S_%c \\rangle $" % l)

plt.xlabel("Time ($\Omega t/\pi$)", size=18)
plt.legend(fontsize=16) #TODO fix legend text size_____no_output_____

Harder to interpret, so we'll use the Bloch sphere:_____no_output_____

sx, sy, sz = result4.expect_____no_output_____b = Bloch()
b.add_points([sx,sy,sz])
b.zlabel = ['$\\left|+z\\right>$', '$\\left|-z\\right>$']
b.view = [-45,20]
b.add_vectors([1/sqrt(2),0,1/sqrt(2)])
b.show()_____no_output_____

## Time-dependent Hamiltonian:
We'll explore the parameters of a spin in a time-varying magnetic field. This system is relevant to nuclear magnetic resonance (NMR) which is used in chemistry and as Magnetic Resonance Imaging (MRI) in medicine.

Following Compliment 9.A the Hamiltonian is:
$$\hat{H}= - \Omega_0 \hat{S}_z - \Omega_1 cos(\omega t)\hat{S}_x$$
We then solve for a certain amount of time after the state starts in $|\psi(0)\rangle = |+z\rangle$

We also use the definition of the Rabi frequency: $\Omega_R = \sqrt{(\omega - \Omega_0)^2 + (\Omega_1/2)^2}$ as in (9.A.28)

Note that the time span is 3 units of $\frac{2\pi}{\Omega_R}$. Leave the scaling in place, but to plot a longer time period, change 3.0 to something larger. This lets us match the units in Fig. 9.A.1._____no_output_____

omega0 = 2.0 * 2 * pi   # pick a nice value for a frequency, note this is 1 Hz
omega1 = 0.25 * 2 * pi  # 25% of omega0
w = 2.0 * 2 * pi        # the driving frequency

H0 = - omega0 * Sz      # the first term in H
H1 = - omega1 * Sx      # the second term in H

omegaR = sqrt((w - omega0)**2 + (omega1/2.0)**2)
t = arange(0,3.0 * 2 * pi / omegaR,0.02)   # scale the time by omegaR, plot 3 units of 2pi/omegaR

args = [H0, H1, w]      # parts of the Hamiltonian

def H1_coeff(t, args):
    return cos(w * t)

H = [H0, [H1, H1_coeff]]_____no_output_____

The next line calls a Schrödinger equation solver (`sesolve`). The arguments are the Hamiltonian, the starting state $\lvert+z\rangle$ (`pz`), the time values, a list of operators, and the arguments to the function `H_t`. `sesolve` returns many things, but the `expect` method is most useful, it gives the expectation values of the four operators in the operator list. Notice the fourth operator is the $\lvert-z\rangle$ projection operator. It's expectation value is $P(\lvert-z\rangle,t)$_____no_output_____

result5 = sesolve(H, pz, t, [Sx, Sy, Sz, mz*mz.dag()],args)
sx, sy, sz, Pmz = result5.expect_____no_output_____

Look at the Bloch sphere for this system:_____no_output_____

b2 = Bloch()
b2.add_points([sx,sy,sz])
b2.zlabel = ['$\\left|+z\\right>$', '$\\left|-z\\right>$']
b2.show()_____no_output_____

Make a plot analogous to Fig 9.A.1:_____no_output_____

plt.tick_params(labelsize=18)
plt.plot(result5.times*omegaR/pi,Pmz)
plt.xlabel("Time ($\Omega_R t/\pi$)", size=18)
plt.ylabel("$P(-z, t)$", size=18)_____no_output_____

### Q) What happens in each unit of time ($\frac{2\pi}{\Omega_R}$)? Look at the plot of $P(-z,t)$ to interpret this. How is your figure different from the on in Fig. 9.A.1?_____no_output_____### Q) How does the evolution change if you double $\Omega_0$?_____no_output_____### Q) After doubling $\Omega_0$ what if you double the driving frequency ($\omega$) also? Interpret this observation in terms of Fig. 9.A.2. In practice, what experimental parameter changes $\Omega_0$?_____no_output_____### Q) How does $\Omega_1$ influence the dynamics? (Be careful reading the plots since the units are scaled by $\Omega_R$)._____no_output_____### Advanced topic: we can change the Hamiltonian so the applied field turns off at a certain time, and it is possible to get the spin to stay in a particular state. This is very useful in quantum optics where certain operations change the atomic state in a very specific way._____no_output_____

omega0 = 1.0 * 2 * pi   # pick a nice value for a frequency, note this is 1 Hz
omega1 = 0.05 * 2 * pi  # 25% of omega0
w = 1.0 * 2 * pi        # the driving frequency

H0 = - omega0 * Sz      # the first term in H
H1 = - omega1 * Sx      # the second term in H

omegaR = sqrt((w - omega0)**2 + (omega1/2.0)**2)
t = arange(0,3.0 * 2 * pi / omegaR,0.05)   # scale the time by omegaR, plot 3 units of 2pi/omegaR

def H1_coeff2(t, args):       # this function calculates H at each time step t
    if t < 2*pi/omegaR * 0.5:  # only add the H1 piece for the first chunk of time.
        coeff = cos(w * t)
    else:
        coeff = 0
    return coeff

H = [H0, [H1, H1_coeff2]]_____no_output_____result6 = sesolve(H, pz, t, [Sx, Sy, Sz, mz*mz.dag()],args)
sx, sy, sz, Pz = result6.expect_____no_output_____plt.plot(result6.times,Pz)
plt.ylim(-0.1,1.1)
plt.xlim(-5,125)
plt.xlabel("Time ($\Omega_R t/\pi$)", size=18)
plt.ylabel("$P(-z, t)$", size=18)_____no_output_____


# Notebook from philipmac/SimpleITK-Notebooks
Path: Python/01_Image_Basics.ipynb

# Welcome to the first SimpleITK Notebook demo:
# SimpleITK Image Basics

This document will give a brief orientation to the SimpleITK Image class.

First we import the SimpleITK Python module. By convention our module is imported into the shorter and more Pythonic "sitk" local name.
_____no_output_____

from __future__ import print_function
%matplotlib inline
import matplotlib.pyplot as plt
import SimpleITK as sitk
_____no_output_____

## Image Construction

There are a variety of ways to create an image. All images' initial value is well defined as zero._____no_output_____

image = sitk.Image(256, 128, 64, sitk.sitkInt16)
image_2D = sitk.Image(64, 64, sitk.sitkFloat32)
image_2D = sitk.Image([32,32], sitk.sitkUInt32)
image_RGB = sitk.Image([128,128], sitk.sitkVectorUInt8, 3)_____no_output_____

### Pixel Types

The pixel type is represented as an enumerated type. The following is a table of the enumerated list.

sitkUInt8	Unsigned 8 bit integer

sitkInt8	Signed 8 bit integer

sitkUInt16	Unsigned 16 bit integer

sitkInt16	Signed 16 bit integer

sitkUInt32	Unsigned 32 bit integer

sitkInt32	Signed 32 bit integer

sitkUInt64	Unsigned 64 bit integer

sitkInt64	Signed 64 bit integer

sitkFloat32	32 bit float

sitkFloat64	64 bit float

sitkComplexFloat32	complex number of 32 bit float

sitkComplexFloat64	complex number of 64 bit float

sitkVectorUInt8	Multi-component of unsigned 8 bit integer

sitkVectorInt8	Multi-component of signed 8 bit integer

sitkVectorUInt16	Multi-component of unsigned 16 bit integer

sitkVectorInt16	Multi-component of signed 16 bit integer

sitkVectorUInt32	Multi-component of unsigned 32 bit integer

sitkVectorInt32	Multi-component of signed 32 bit integer

sitkVectorUInt64	Multi-component of unsigned 64 bit integer

sitkVectorInt64	Multi-component of signed 64 bit integer

sitkVectorFloat32	Multi-component of 32 bit float

sitkVectorFloat64	Multi-component of 64 bit float

sitkLabelUInt8	RLE label of unsigned 8 bit integers

sitkLabelUInt16	RLE label of unsigned 16 bit integers

sitkLabelUInt32	RLE label of unsigned 32 bit integers

sitkLabelUInt64	RLE label of unsigned 64 bit integers

There is also `sitkUnknown`, which is used for undefined or erroneous pixel ID's. It has a value of -1.

The 64-bit integer types are not available on all distributions. When not available the value is `sitkUnknown`.
_____no_output_____### More Information about the Image class be obtained in the Docstring

SimpleITK classes and functions have the Docstrings derived from the C++ definitions and the Doxygen documentation. _____no_output_____

help(image)_____no_output_____

## Accessing Attributes

If you are familiar with ITK, then these methods will follow your expectations:_____no_output_____

print(image.GetSize())
print(image.GetOrigin())
print(image.GetSpacing())
print(image.GetDirection())
print(image.GetNumberOfComponentsPerPixel())_____no_output_____

Note: The starting index of a SimpleITK Image is always 0. If the output of an ITK filter has non-zero starting index, then the index will be set to 0, and the origin adjusted accordingly.

The size of the image's dimensions have explicit accessors:_____no_output_____

print(image.GetWidth())
print(image.GetHeight())
print(image.GetDepth())_____no_output_____

Since the dimension  and pixel type of a SimpleITK image is determined at run-time accessors are needed. _____no_output_____

print(image.GetDimension())
print(image.GetPixelIDValue())
print(image.GetPixelIDTypeAsString())_____no_output_____

What is the depth of a 2D image?_____no_output_____

print(image_2D.GetSize())
print(image_2D.GetDepth())_____no_output_____

What is the dimension and size of a Vector image?_____no_output_____

print(image_RGB.GetDimension())
print(image_RGB.GetSize())_____no_output_____print(image_RGB.GetNumberOfComponentsPerPixel())_____no_output_____

For certain file types such as DICOM, additional information about the image is contained in the meta-data dictionary._____no_output_____

for key in image.GetMetaDataKeys():
        print("\"{0}\":\"{1}\"".format(key, image.GetMetaData(key)))_____no_output_____

## Accessing Pixels

There are the member functions ``GetPixel`` and ``SetPixel`` which provides an ITK-like interface for pixel access._____no_output_____

help(image.GetPixel)_____no_output_____print(image.GetPixel(0, 0, 0))
image.SetPixel(0, 0, 0, 1)
print(image.GetPixel(0, 0, 0))_____no_output_____print(image[0,0,0])
image[0,0,0] = 10
print(image[0,0,0])_____no_output_____

## Conversion between numpy and SimpleITK_____no_output_____

nda = sitk.GetArrayFromImage(image)
print(nda)_____no_output_____help(sitk.GetArrayFromImage)_____no_output_____# Get a view of the image data as a numpy array, useful for display
nda = sitk.GetArrayViewFromImage(image)_____no_output_____nda = sitk.GetArrayFromImage(image_RGB)
img = sitk.GetImageFromArray(nda)
img.GetSize()_____no_output_____help(sitk.GetImageFromArray)_____no_output_____img = sitk.GetImageFromArray(nda, isVector=True)
print(img)_____no_output_____

## The order of index and dimensions need careful attention during conversion

ITK's Image class does not have a bracket operator. It has a GetPixel which takes an ITK Index object as an argument, which is ordered as ``(x,y,z)``. This is the convention that SimpleITK's Image class uses for the GetPixel method and slicing operator as well. In numpy, an array is indexed in the **opposite** order ``(z,y,x)``. Also note that the access to channels is different. In SimpleITK you do not access the channel directly, rather the pixel value representing all channels for the specific pixel is returned and you then access the channel for that pixel. In the numpy array you are accessing the channel directly. _____no_output_____

import numpy as np

multi_channel_3Dimage = sitk.Image([2,4,8], sitk.sitkVectorFloat32, 5)
x = multi_channel_3Dimage.GetWidth() - 1
y = multi_channel_3Dimage.GetHeight() - 1
z = multi_channel_3Dimage.GetDepth() - 1
multi_channel_3Dimage[x,y,z] = np.random.random(multi_channel_3Dimage.GetNumberOfComponentsPerPixel())

nda = sitk.GetArrayFromImage(multi_channel_3Dimage)

print("Image size: " + str(multi_channel_3Dimage.GetSize()))
print("Numpy array size: " + str(nda.shape))

# Notice the index order and channel access are different:
print("First channel value in image: " + str(multi_channel_3Dimage[x,y,z][0]))
print("First channel value in numpy array: " + str(nda[z,y,x,0]))_____no_output_____

### Are we still dealing with Image, because I haven't seen one yet...

While SimpleITK does not do visualization, it does contain a built in ``Show`` method. This function writes the image out to disk and than launches a program for visualization. By default it is configured to use ImageJ, because it is readily supports all the image types which SimpleITK has and load very quickly. However, it's easily customizable by setting environment variables._____no_output_____

sitk.Show(image)_____no_output_____sitk.Show?_____no_output_____

By converting into a numpy array, matplotlib can be used for visualization for integration into the scientific python environment._____no_output_____

%matplotlib inline
import matplotlib.pyplot as plt_____no_output_____z = 0
slice = sitk.GetArrayViewFromImage(image)[z,:,:]
plt.imshow(slice)_____no_output_____


# Notebook from cbreenmachine/wgbs-load
Path: analysis/2022-01-DSS/run-DSS.ipynb

# run-DSS.R
# Runs Dispersion Shrinkage Estimation method on methylation data (one chromosome at a time)
library(data.table)
library(tidyverse)
library(argparse)
library(DSS)── Attaching packages ────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.3.1 ──

✔ ggplot2 3.3.5     ✔ purrr   0.3.4
✔ tibble  3.1.6     ✔ dplyr   1.0.7
✔ tidyr   1.1.4     ✔ stringr 1.4.0
✔ readr   2.1.1     ✔ forcats 0.5.1

── Conflicts ───────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::between()   masks data.table::between()
✖ dplyr::filter()    masks stats::filter()
✖ dplyr::first()     masks data.table::first()
✖ dplyr::lag()       masks stats::lag()
✖ dplyr::last()      masks data.table::last()
✖ purrr::transpose() masks data.table::transpose()

Loading required package: Biobase

Loading required package: BiocGenerics

Loading required package: parallel

Attaching package: ‘BiocGenerics’

The following objects are masked from ‘package:parallel’:

    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
    clusterExport, clusterMap, parApply, parCapply, parLapply,
    parLapplyLB, parRapply, parSapply, parSapplyLB

The following objects are masked from ‘package:dplyr’:

    combine, intersect, setdiff, union

The following objects are masked from ‘package:stats’:

    IQR, mad, sd, var, xtabs

The following objects are masked from ‘package:base’:

    anyDuplicated, append, as.data.frame, basename, cbind, colnames,
    dirname, do.call, duplicated, eval, evalq, Filter, Find, get, grep,
    grepl, intersect, is.unsorted, lapply, Map, mapply, match, mget,
    order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
    rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply,
    union, unique, unsplit, which.max, which.min

Welcome to Bioconductor

    Vignettes contain introductory material; view with
    'browseVignettes()'. To cite Bioconductor, see
    'citation("Biobase")', and for packages 'citation("pkgname")'.

Loading required package: BiocParallel

Loading required package: bsseq

Loading required package: GenomicRanges

Loading required package: stats4

Loading required package: S4Vectors

Attaching package: ‘S4Vectors’

The following objects are masked from ‘package:dplyr’:

    first, rename

The following object is masked from ‘package:tidyr’:

    expand

The following objects are masked from ‘package:data.table’:

    first, second

The following object is masked from ‘package:base’:

    expand.grid

Loading required package: IRanges

Attaching package: ‘IRanges’

The following objects are masked from ‘package:dplyr’:

    collapse, desc, slice

The following object is masked from ‘package:purrr’:

    reduce

The following object is masked from ‘package:data.table’:

    shift

Loading required package: GenomeInfoDb

Loading required package: SummarizedExperiment

Loading required package: MatrixGenerics

Loading required package: matrixStats

Attaching package: ‘matrixStats’

The following objects are masked from ‘package:Biobase’:

    anyMissing, rowMedians

The following object is masked from ‘package:dplyr’:

    count

Attaching package: ‘MatrixGenerics’

The following objects are masked from ‘package:matrixStats’:

    colAlls, colAnyNAs, colAnys, colAvgsPerRowSet, colCollapse,
    colCounts, colCummaxs, colCummins, colCumprods, colCumsums,
    colDiffs, colIQRDiffs, colIQRs, colLogSumExps, colMadDiffs,
    colMads, colMaxs, colMeans2, colMedians, colMins, colOrderStats,
    colProds, colQuantiles, colRanges, colRanks, colSdDiffs, colSds,
    colSums2, colTabulates, colVarDiffs, colVars, colWeightedMads,
    colWeightedMeans, colWeightedMedians, colWeightedSds,
    colWeightedVars, rowAlls, rowAnyNAs, rowAnys, rowAvgsPerColSet,
    rowCollapse, rowCounts, rowCummaxs, rowCummins, rowCumprods,
    rowCumsums, rowDiffs, rowIQRDiffs, rowIQRs, rowLogSumExps,
    rowMadDiffs, rowMads, rowMaxs, rowMeans2, rowMedians, rowMins,
    rowOrderStats, rowProds, rowQuantiles, rowRanges, rowRanks,
    rowSdDiffs, rowSds, rowSums2, rowTabulates, rowVarDiffs, rowVars,
    rowWeightedMads, rowWeightedMeans, rowWeightedMedians,
    rowWeightedSds, rowWeightedVars

The following object is masked from ‘package:Biobase’:

    rowMedians

Filter first!!!_____no_output_____

parser <- ArgumentParser()
parser$add_argument("--chr", default= "chr22", help='Chromosome to run DSS on')
parser$add_argument("--covariates", default= "NK,CD4T")
parser$add_argument("--num_pcs", default= "2", help = 'number of principal components (precomputed)')
args <- parser$parse_known_args()

# Setup output directory, will save called regions, figures?
odir <- paste0("result-", args[[1]]$chr)
dir.create(odir, showWarning = FALSE)_____no_output_____pc.df <- read_csv(file.path("../../data/prin-comps-array-samples/", paste0(args[[1]]$chr , ".csv")),
                  col_types = cols())_____no_output_____ss.df <- read_csv("../../data/meta/phenos-cleaned.csv", col_types = cols())
DT <- fread(file.path("../../data/cov-meth/", paste0(args[[1]]$chr, ".tsv")))_____no_output_____make_tall_matrix <- function(DT, value.var) {
    # Turn to wide format with samples indexing columns
    # and position / locus indexing row
    out <- dcast(DT, pos~sample, value.var=value.var)
    #rownames(out) <- out$pos
    #out[ ,c("pos") := NULL]

    return(out)
}

drop_null_positions <- function(X, p=0.5){
    # Drops positions in X with too many nulls,
    # X is from `make_tall_matrix` and p is a percentage from 0 to 1
    stopifnot(p >= 0, p <= 1)

    # Cut point as float
    cut <- (ncol(X) - 1) * p
    num_null <- rowSums(is.na(X))

    return(X[num_null <= cut, ])
}

_____no_output_____M <- make_tall_matrix(DT, "methylated") %>% drop_null_positions
Cov <- make_tall_matrix(DT, "coverage") %>% drop_null_positions_____no_output_____valid_samples <- intersect(intersect(colnames(M), ss.df$sample), pc.df$sample)

filt.df <- ss.df %>%
            inner_join(pc.df, by = "sample") %>%
            dplyr::filter(sample %in% valid_samples)

# pack years smoking needs to be not na
filt.df$pack_years[is.na(filt.df$pack_years)] <- 0

drop_cols <- as.character(setdiff(colnames(M), valid_samples))[-1] # dont remove pos
M[, (drop_cols):=NULL]
Cov[, (drop_cols):=NULL]

length(valid_samples)_____no_output_____load.samples <- filt.df %>% filter(cohort == "AD") %>% pull(sample) %>% as.character
ctrl.samples <- filt.df %>% filter(cohort == "CONTROL") %>% pull(sample) %>% as.character_____no_output_____get_na_mask <- function(X, col.names){
    mask <- is.na(X)
    cn <- colnames(mask)
    for (j in 1:ncol(mask)){
        if ((cn[j] %in% col.names)) {
            mask[, j] <- FALSE
        }
    }
    return(mask)
}

impute_by_group <- function(X, col.names, round.mean = TRUE){
    # The imputed value is the position mean
    position.means <- rowMeans(X[ , col.names, with = FALSE], na.rm = TRUE)

    # If
    if (round.mean) {position.means <- round(position.means)}
    mask <- get_na_mask(X, col.names)

    # Imputation step
    # With data.tables, for loops much master than vectorization
    for (i in 1:nrow(X)){
        for (j in which(mask[i, ])){
            set(X, i, j, value = position.means[i])
        }
    }
}_____no_output_____impute_by_group(M, load.samples)
_____no_output_____impute_by_group(M, ctrl.samples)
impute_by_group(Cov, load.samples)
impute_by_group(Cov, ctrl.samples)_____no_output_____head(M)_____no_output_____M <- drop_null_positions(M, 0)
Cov <- drop_null_positions(Cov, 0)_____no_output_____# Order needs to be correct!!!
valid_pos <- intersect(M$pos, Cov$pos)
M <- M[pos %in% valid_pos , c("pos", filt.df$sample), with = FALSE]
Cov <- Cov[pos %in% valid_pos, c("pos", filt.df$sample), with = FALSE]_____no_output_____# create bs seq object, needs chromosome identiifer, methylated reads, and unmethylated reads
bs <- BSseq(chr = rep(DT$chr[1], nrow(M)), pos = M$pos,
            M = as.matrix(M[ , -c("pos"), with=FALSE]),
            Cov = as.matrix(Cov[, -c("pos"), with=FALSE]),
            sampleNames = names(M)[-1])
_____no_output_____# Print this check in the future!!!!
all( filt.df$sample == colnames(bs) )_____no_output_____#TODO: formula from input flags
dml.fit <- DMLfit.multiFactor(bs, design = filt.df, smoothing = TRUE, smoothing.span = 200,
            formula = ~cohort + PC1 + PC2 + Gran + CD8T + CD4T + NK + Bcell + bmi + age + sex)Fitting DML model for CpG site: 100000 , 200000 , 300000 , 400000 , 500000 , colnames(dml.fit$X)
test.cohort <- DMLtest.multiFactor(dml.fit, coef = 2)_____no_output_____table(is.na(test.cohort$stat))_____no_output_____methylation <- M / Cov
methylation$pos <- M$pos
fwrite(methylation, file.path(odir, "methylation.csv"))_____no_output_____save(list = c("test.cohort"), file= file.path(odir, "test-values.RData"))
fwrite(M, file.path(odir, "M.csv"))
fwrite(Cov, file.path(odir, "Cov.csv"))
_____no_output_____dmrs <- callDMR(test.cohort, p.threshold=0.01, minCG = 5 )_____no_output_____dmrs_____no_output_____


# Notebook from mathieukaltschmidt/CompQD
Path: Exercise sheet 3/exercise3_Mathieu-Kaltschmidt_Carl-von-Randow.ipynb

# Programing exercise 3: Coupled oscillators

Due on Monday, 12.11.2018, 14.15h

### Total: 100 points

__This exercise sheet has been solved by:__ Mathieu Kaltschmidt and Carl von Randow_____no_output_____## The problem

Consider two coupled harmonic oscillators described by the Hamiltonian
$$
H=H_1 + H_2 + V =\frac{1}{2m}(p_1^2 + p_2^2) + \frac{1}{2}k(x_1^2 + x_2^2) + \frac{1}{2}\lambda(x_1 - x_2)^2
$$
We want to calculate the eigenvalues and eigenfunctions of this Hamiltonian and compare the dynamics of the corresponding classical problem. This is exactly solvable by transforming into center of mass and relatice coordinates. However, we also want to treat the coupling term lambda as a perturbation and solve the problem by expanding into eigenfunctions of the 2D harmonic oscillator and solve the resulting problem, both, perturbatively for small $\lambda$ and by exact diagonalization._____no_output_____

# load standard libraries
import numpy as np
import numpy.linalg as LA
import matplotlib.pyplot as plt
%matplotlib inline

#for 3D plotting
from mpl_toolkits.mplot3d import Axes3D

#Plot settings
plt.style.use('seaborn-talk')
plt.rcParams["font.family"]='serif'
plt.rcParams["figure.figsize"][0] = 16
plt.rcParams["figure.figsize"][1] = 10_____no_output_____

$$\newcommand{\braket}[3]{\left\langle{#1}\middle|{#2}\middle|{#3}\right\rangle}$$_____no_output_____### Exercise 1 (20 points)

Transform the problem into center of mass and relative coordinates using
$$
x_{CM}=(x_1+x_2)/2 \\
x_{rel}=(x_1-x_2)
$$
Write a function that returns the classical trajectory for initial conditions $\{x_1(t=0)=x_{10}, x_2(t=0)=x_{20}, \dot{x}_1(t=0)=v_{10}, \dot{x}_2(t=0)=v_{20}\}$.

Make a density plot of the potential and plot the classical trajectory for $m=k=1,\lambda=0.2$ and ${x_{10}=1,x_{20}=v_{10}=v_{20}=0}$ up to a time $t_f$ into it. Also, plot the positions of the two particles as a function of time. Observe the beating between center-of-mass and relative frequency.
_____no_output_____Very instructive 2D plotting tips:
https://jakevdp.github.io/PythonDataScienceHandbook/04.04-density-and-contour-plots.html_____no_output_____### Analytical solution of the problem
To change our old coordinates to the new ones we have to use:

\begin{align}
x_{1}&=x_{CM}+\frac{1}{2}x_{rel} \\
x_{2}&=x_{CM}-\frac{1}{2}x_{rel}
\end{align}

This yields the following Hamiltonian:

\begin{align}
H=m(\dot{x}_{CM}^2 + \frac{1}{4}\dot{x}{rel}^2)+\frac{1}{2} k(\frac{1}{2}x_{rel}^2+x_{CM}^2)+\frac{1}{2} \lambda x_{rel}^2
\end{align}

$H$ can be split into two independent Hamiltonians $H_{rel}$ and $H_{CM}$:

\begin{align}
H_{CM}&=m\dot{x}_{CM}^2+\frac{1}{2} k x_{CM}^2 \\
H_{rel}&=m\frac{1}{4}\dot{x}_{rel}^2+x_{rel}^2(\frac{1}{4}k+\frac{1}{2} \lambda)
\end{align}

By solving the Euler-Lagrange equations we obtain:

\begin{align}
x_{rel}(t) &= S_{rel} sin(\omega_{rel}t) + C_{rel} cos(\omega_{rel}t)\\
x_{CM}(t) &= S_{CM} sin(\omega_{CM}t) + C_{CM} cos(\omega_{CM}t)\\
\end{align}

$\omega_{rel}$ and $\omega_{CM}$ are given by:

\begin{align}
\omega_{rel} &= \sqrt{\frac{k+2\lambda}{m}}\\
\omega_{CM} &= \sqrt{\frac{k}{2m}}
\end{align}

To transform this solution back to our old coordinates we use the first two equations in this exercise and get the following result:

\begin{align}
x_1(t) &= C_{CM} cos(\omega_{CM}t) + S_{CM} sin(\omega_{CM}t)+\frac{1}{2} \left[ C_{rel} cos(\omega_{rel}t) + S_{rel} sin(\omega_{rel}t) \right] \\
x_2(t) &= C_{CM} cos(\omega_{CM}t) + S_{CM} sin(\omega_{CM}t)-\frac{1}{2} \left[ C_{rel} cos(\omega_{rel}t) + S_{rel} sin(\omega_{rel}t) \right]
\end{align}

The initial conditions lead to the following coefficients:

\begin{align}
C_{rel} &= x_{10}-x_{20}\\
S_{rel} &= \frac{1}{\omega_{rel}} \left( v_{10}-v_{20} \right)\\
C_{CM} &= \frac{1}{2} \left( x_{10}+x_{20} \right) \\
S_{CM} &= \frac{1}{2 \omega_{CM} } \left( v_{10}+v_{20} \right)
\end{align}_____no_output_____

#Defining spatial grid structure
x1=np.linspace(-2,2,100)
x2=np.linspace(-2,2,100)

#some parameters
(m, k, hbar, omega, const)  = (1, 1, 1, 1, 0.2)

#initial conditions
(x10, v10) = (1, 0)
(x20, v20) = (0, 0)_____no_output_____#Implementing the solution for the trajectories
def trajectories(x1,v1,x2,v2,t):
    x_rel = (x1 - x2)
    omega_rel = np.sqrt((k + 2*const)/(2*m))
    omega_cm = np.sqrt(k/(2*m))

    #defining the known prefactors
    C_rel = x_rel
    S_rel =(v1 - v2) / omega_rel

    C_CM = 0.5 * (x1 + x2)
    S_CM =(v1 + v2) / (2 * omega_cm)

    x1=C_CM*np.cos(omega_cm*t) + S_CM*np.sin(omega_cm*t) + ((C_rel*np.cos(omega_rel*t) + S_rel*np.sin(omega_rel*t)))/2
    x2=C_CM*np.cos(omega_cm*t) + S_CM*np.sin(omega_cm*t) - ((C_rel*np.cos(omega_rel*t) + S_rel*np.sin(omega_rel*t)))/2

    return x1, x2_____no_output_____#Defining the harmonic oscillator potential
def oscillator(x1,x2):
    return 0.5*(x1**2 + x2**2)

#Defining the given coupling potential
def coupling(x1,x2,const):
    return 0.5*const*(x1-x2)**2

#Defining the total potential (oscillator + coupling)
def V_total(x1,x2,const):
    return oscillator(x1,x2) + coupling(x1,x2,const)

#Creating the grid structure + density plots
def density_plot(x1,x2,potential,const):
    x,y = np.meshgrid(x1,x2)
    z = potential(x,y,const)
    plt.contour(x,y,z,20,cmap = 'plasma')
    plt.colorbar()_____no_output_____#Calcultating the trajectory
time = np.linspace(0,40,1000)
particle1, particle2 = trajectories(x10,v10,x20,v20,time)_____no_output_____#Plotting the trajectories in the total potential
plt.plot(particle1, particle2, color='teal', linewidth = 4, label = 'Trajectory of the coupled system')
density_plot(x1,x2,V_total,const)
plt.title('Trajectory of a system of two particles in a coupled oscillator potential \n Initial conditions: $x_{1,0} = 1, \ x_{2,0} = v_{1,0} = v_{2,0} = 0$', size = 20)
plt.xlabel('Coordinate $x_1$', size = 18)
plt.ylabel('Coordinate $x_2$', size = 18)
plt.legend(frameon=True,fontsize='large', borderpad=1.5, borderaxespad=1.5,loc='lower right')_____no_output_____#Plotting both trajectories as a function of time
plt.plot(time, particle1, label="Particle 1", color = 'darkred')
plt.plot(time ,particle2, label="Particle 2", color = 'darkblue')
plt.ylabel("Position", size = 22)
plt.xlabel("Time $t$", size = 22)
plt.title("Trajectories of both particles over time", size = 24)
plt.legend(frameon=True,fontsize='large', borderpad=1.5, borderaxespad=1.5,loc='best')_____no_output_____

### Exercise 2 (10 points)

What are the exact quantum mechanical eigenenergies?
Calculate the ground state energy perturbatively to second order in $\lambda$. Use the decomposition into ladder operators
$$
x/x_{HO} = (a^\dagger + a)/\sqrt{2}
$$
Show that the same result is obtained by Taylor expanding the exact ground state energy $E_0 = \hbar\omega/2(1+\sqrt{1+2\lambda/k}) $ to second order in $\lambda/k$.

Plot the exact and perturbative ground state energy as a function of $\lambda/k$._____no_output_____### Result from Taylor expanding around small $\frac{\lambda}{k}$

In the following calculation we substitute $\frac{\lambda}{k} =: \alpha$ and calculate the Taylor expansion around $\alpha_0 \approx 0$.

The ground state energy is defined as: $E_0 = \frac{\hbar\omega}{2}\left(1+\sqrt{\frac{2\lambda}{k}}\right)$.

We get the following result for the expansion to second order in $\alpha$:

\begin{align}
E_0\kern-0.3em\mid\kern-0.3em\small_{\alpha_0 = 0} &\approx \frac{\hbar\omega}{2} \left( 1 + \sqrt{2\cdot\alpha_0}\cdot\alpha^{0}
     + \frac{1}{1!}\cdot\frac{1}{2}\frac{2}{\sqrt{1+2\alpha_0}}\cdot \alpha^1 + \frac{1}{2!}\cdot\left(-\frac{1}{2}\right)\frac{2}{\sqrt{1+2\alpha_0}^3}\cdot \alpha^2 + \mathcal{O}\left(\alpha^3\right) \right) \\
     &=  \frac{\hbar\omega}{2} \left( 2 + \alpha - \frac{1}{2}\alpha^2 + \mathcal{O}\left(\alpha^3\right) \right) = \hbar\omega \left( 1 + \frac{\alpha}{2} - \frac{\alpha^2}{4}\right) + \mathcal{O}\left(\alpha^3\right)
\end{align}

The final result of the expansion, reads:

$$
E_0 \approx \hbar\omega \left( 1 + \frac{\lambda}{2k} - \frac{\lambda^2}{4k^2}\right) + \mathcal{O}\left(\frac{\lambda^3}{k^3}\right)
$$_____no_output_____### Result using perturbation theory_____no_output_____The idea of perturbation theory is to seperate the general Hamiltonian into the two unperturbed parts for both particles and a small perturbated part, e.g. :

$$
\mathcal{H} = \underbrace{\mathcal{H}_1 + \mathcal{H}_2}_{:= \ \mathcal{H}_0} + V = \mathcal{H}_0 + \lambda\cdot\mathcal{H}_1
$$
The potential $V=\frac{1}{2}\left(x_1-x_2\right)^2$ corresponds to the harmonic oscillator problem, in this case for a system of two particles.

#### Finding the correct eigeneenergies

One computes the eigenenergies $E_n$ by finding the correction terms from perturbation theory:

$$
E_n = E_n^0 + \lambda \ E_n^1 + \lambda^2 \ E_n^2 +\mathcal{O}\left(\lambda^3\right)
$$

#### Lowest order in unperturbed Hamiltonian

For this case we only have to consider the general solution of the harmonic oscillator problem and assume both particles occupying the groundstate $n=0$:

$$
E_0^0 = \hbar\omega\left( n_1 + n_2 + 1\right) = \hbar\omega
$$

#### First order perturbation theory

\begin{align}
E^1_0 &= \langle n_1,n_2| \ \mathcal{H}_1 \ |n_1,n_2\rangle = \langle n_1,n_2|\frac12 ( x_1-x_2)^2|n_1,n_2\rangle
      = \frac{\hbar}{4m\omega}\langle n_1,n_2| \left(a^\dagger_1+a_1-(a^\dagger_2+a_2)\right)^2|n_1,n_2\rangle \\
      &= \frac{\hbar}{4m\omega} \langle n_1,n_2|\left(a^\dagger_1+a_1-(a^\dagger_2+a_2)\right)\left(\sqrt{n_1+1}|n_1+1,n_2\rangle+\sqrt{n_1}|n_1-1,n_2\rangle-\sqrt{n_2+1}|n_1,n_2+1\rangle-\sqrt{n_2}|n_1,n_2-1\rangle\right)
\end{align}

For the ground states we know that $n_1=n_2=0$:

$$
E^1_0=
\frac{\hbar}{4m\omega} \langle 0,0|\left(a^\dagger_1+a_1-a^\dagger_2-a_2\right)\left(|1,0\rangle-|0,1\rangle\right)=
\frac{\hbar}{2m\omega}=
\frac{\hbar\omega}{2k}
$$

#### Second order perturbation theory:
\begin{align}
E^2_{0,0}&=\sum_{m_1,m_2\neq 0}\frac{\left|\langle m^0_1,m^0_2|\ \mathcal{H}_1 \ |0,0\rangle\right|^2}{E_{0,0}^0-E_{m_1,m_2}^0} \\
&=\sum_{m_1,m_2\neq 0}\frac{\left|\langle0,0|\frac{\hbar}{4m\omega}\left(a_1-a_2\right)|m_1, m_2\rangle\right|^2}{-\hbar\omega(m_1+m_2)} \\
&=\left(\frac{\hbar}{4m\omega}\right)^2\left[\frac{|\sqrt{2}|^2}{-2\hbar\omega}+\frac{|-1|^2}{-2\hbar\omega}+\frac{|-1|^2}{-2\hbar\omega}+\frac{|\sqrt{2}|^2}{-2\hbar\omega}\right] \\
&=\frac{-3\hbar}{16m^2\omega^3}=
-\frac{3\hbar\omega}{16k^2}
\end{align}

We approximate the last term and get:

$$
E_0 \approx \hbar\omega \left( 1 + \frac{\lambda}{2k} - \frac{\lambda^2}{4k^2}\right)
$$

#### Conclusion
Therefore we found (almost) the same result for both methods._____no_output_____### Plotting the result _____no_output_____

testrange = np.linspace(0,2,100)

#Exact round state energy
def E0(const):
    return (hbar*omega)/2*(1+np.sqrt(1+2*const/k))
#Result from perturbation theory
def E0_PT(const):
    return hbar*omega*(1+const/(2*k)-const**2/(4*k**2))

plt.plot(testrange, E0(testrange), label="Exact ground state energy", color = 'darkred')
plt.plot(testrange, E0_PT(testrange), label="Ground state energy from perturbation theory", color = 'darkblue')
plt.title("Comparison of the exact ground state energy with the result from perturbation theory", size = 20)
plt.ylabel("Ground State Energy [$\hbar\omega$]", size = 22)
plt.xlabel("Order of perturbation $\ \lambda/k$", size = 22)

plt.grid(True)
plt.legend(frameon=True,fontsize='large', borderpad=1.5, borderaxespad=1.5,loc='best')_____no_output_____

#### Conclusion
As one can see, the calculated approximation using perturbation theory holds for small $\frac{\lambda}{k}$._____no_output_____### Exercise 3 (40 points)

Expand the Hamiltonian in the eigenbasis of the 2D harminic oscillators (use the same cutoff $N_1=N_2=N_{max}$ for both oscillator modes 1 and 2).
For this, one can expand the perturbation term $\frac{1}{2}\lambda(x_1 - x_2)^2$ into ladder operators and then apply it to the unperturbed basis states.
How many non-zero elements will there be at most in each row?

There are many ways of building the Hamiltonian matrix in the chosen basis. First, make sure you understand what the basis states are and how one would order them in a reasonable way.
A failsafe way to build up the Hamiltonian matrix is to iterate over all states of the bare basis and determine the matrix-elments it couples to. Another, arguably more elegant, way is to build the matrix representing the anihilation operator on the Hilbert space of one of the oscillators. The creation operator is then simply the transpose of that and the position operator can be built by adding the two. We have a 2D problem so we work in a Hilbert space that is a tensor product of the two Hilbert spaces of the oscillators. Thus, to represent the position operator $\hat{x}_1$ acting on oscillator 1 in the full Hilbert space  we have to use $\hat{x}_1 \otimes \mathbf{1}$, where $\mathbf{1}$ is the identity acting on the second subspace (see the numpy "eye" function). This can be achieved by using the Kronecker product. For multiplying or squaring operators you can use the numpy dot product function.
The second method might be less efficient if you want to exploit the sparsity of the problem but it is very handy since it requires only to build one operator explicitly and the rest is matrix algebra (or two if you use different basis sizes for the two oscillators).

Diagonalize the Hamiltonian and check your result by comparing the ground state energy to the exact result. How small can you make the basis such that the ground sate energy still converged? (You should find that $N_1=N_2=5$, i.e. 25 basis states, is sufficient to get the ground state with good precision up to $\lambda=2$. Note that using a simple spatial grid to represent the wave function we would need much larger basis sizes!) Also look at some excited states, at least the lowest 6, make sure that your diagonalization results agree with the exact one, and understand the structure of the spectrum.

Bonus (not graded): This is of course still a problem that can be solved analytically, and thus we are complicating it by doing the basis set expansion. However, one could now use this method to apply it to non-integrable cases like a quartic (non-linear) coupling or even the quantum version of the Henon-Heiles problem, which does not separate and is classically chaotoic!
If you want to go beyond, try other potentials!_____no_output_____### Solution:
#### Non-zero elements in each row
With this method one finds at most $3$ non-zero elements in each row. This can easily be seen by writing down a general matrix multiplication of the form of our problem. These non-zero entries are located on the main diagonal and the respective second next diagonals.
_____no_output_____

#Implementation of x1- and x2-operators with N_cut = N
def x_ops(N):
    #Creating the identity Matrix with dim = N x N
    identity = np.eye(N)
    #Matrix representation of the ladder operators
    values = np.sqrt(np.linspace(1,N-1,N-1))
    a = np.diag(values,1)
    a_dagger = np.diag(values,-1)
    #Computing final result using the tensor product
    x1 = 1/np.sqrt(2)*(np.kron(a,identity)+np.kron(a_dagger,identity))
    x2 = 1/np.sqrt(2)*(np.kron(identity,a)+np.kron(identity,a_dagger))
    return x1, x2

def perturbation(N,const):
    x1, x2 = x_ops(N)
    H1 = (const/2) * (np.dot(x1,x1)-np.dot(x1,x2)-np.dot(x2,x1)+np.dot(x2,x2))
    return H1, x1, x2

#Now we want to compute the eigenenergies
def eigenenergies(N,const):
    H1, x1, x2 = perturbation(N,const)
    #Unperturbed Hamiltonian H_0
    H0 = np.diag(np.linspace((hbar*omega)/2,hbar*omega*(N-1/2),N))
    identity = np.eye(N)
    #Applying the tensor product again
    H0_tensor = np.kron(H0,identity)+np.kron(identity,H0)
    eigvals, eigvecs = LA.eig(H0_tensor + H1)
    return eigvals,eigvecs_____no_output_____#Comparing some values for N
Nvals = np.linspace(2,10,9, dtype=int)
const = 2

#We directly compare the theoretical with the numerical values for all considered values for N
print("Energy for the ground state (theoretical):", E0(const))
print()
print('Numerical values for different N:')

for N in Nvals:
    vals, vecs = eigenenergies(N, const)
    print('For N =', N, ': ', sorted(vals)[0], ' , Difference to theoretical value = ', np.abs(sorted(vals)[0]-E0(const)) )Energy for the ground state (theoretical): 1.61803398875

Numerical values for different N:
For N = 2 :  1.58578643763  , Difference to theoretical value =  0.032247551123
For N = 3 :  1.5748352925  , Difference to theoretical value =  0.0431986962489
For N = 4 :  1.61723525671  , Difference to theoretical value =  0.000798732040295
For N = 5 :  1.61663936756  , Difference to theoretical value =  0.0013946211869
For N = 6 :  1.6180001152  , Difference to theoretical value =  3.38735513465e-05
For N = 7 :  1.61798832975  , Difference to theoretical value =  4.5658998066e-05
For N = 8 :  1.61803186541  , Difference to theoretical value =  2.12333610228e-06
For N = 9 :  1.61803237262  , Difference to theoretical value =  1.61613160099e-06
For N = 10 :  1.61803385578  , Difference to theoretical value =  1.32965591426e-07

#### Conclusion
We found out that we have to choose at least $N=4$ to get a sufficient precision for the ground state energy.

#### Excitation states_____no_output_____

#Testing some exitation states
excitation = np.linspace(1,6,6, dtype=int)
for state in excitation:
    print(state,'. excited state: E = ',sorted(vals)[state])1 . excited state: E =  2.61802723699
2 . excited state: E =  3.61807359778
3 . excited state: E =  3.85409571552
4 . excited state: E =  4.61725673509
5 . excited state: E =  4.85407373673
6 . excited state: E =  5.62139155209

We chose the ordering of the excitation states depending on the value of their respective energy level._____no_output_____### Exercise 4 (30 points)

Calculate the time evolution for different initial conditions: 1) oscillator 1 in the first excited state, oscillator two in its ground state $|0\rangle$; 2) oscillator 1 in $|\psi_0\rangle = (|0\rangle+|1\rangle)/\sqrt(2)$, oscillator two in state $|0\rangle$;  3) oscillator 1 in a coherent state (e.g. with $\langle n \rangle = |\alpha|^2 = 2$) and oscillator 2 in $|0\rangle$. Example parameters: $\lambda=0.2$, $N_1=N_2=10$, $t_f = 40$.

Monitor the dynamics by calculating $\langle x_1 \rangle$ and $\langle x_2 \rangle$, the energy expectation value of each oscillator, as well as the probabilities for each oscillator to be in state $|n\rangle$. Compare your observations to the classical expectation: propagate a classical particle with $x_{i0}=\langle x_i(t=0) \rangle$ and $p_{i0}=\langle p_i(t=0) \rangle$. Describe and interpret your observations.

Bonus (not graded): Represent the wave function on a 2D spatial grid using the exact harmonic oscillator eigenfunctions. Animate the time-dependence of the wave packet._____no_output_____### Analytical solution_____no_output_____As usual the time evolutions are given by:

1)
\begin{align}
\left| \ \Psi_1(t) \ \right\rangle = e^{-i E_{1,0}t}(\left| 1 \right\rangle \otimes \left| 0 \right\rangle )\\
\end{align}

2)
\begin{align}
\left| \ \Psi_2(t) \ \right\rangle = \frac{1}{\sqrt{2}} \left(  e^{-i E_{0,0}t}(\left| 0 \right\rangle \otimes \left| 0 \right\rangle) + e^{-i E_{1,0}t}(\left| 1 \right\rangle \otimes \left| 0 \right\rangle)\right)
\end{align}

3)
The coherent state is described by:

\begin{align}
\left| \ \Psi_3(t=0) \ \right\rangle = e^{-\frac{1}{2}|z|^2}e^{za^{\dagger{}}}\left| 0 \right\rangle
\end{align}

Since we want $\left\langle n \right\rangle = 2 $, we get $z = \sqrt{2}$.

#### x expectation values:
For the first case we have no time dependency for $\left\langle x_i \right\rangle$, as $\left\langle \ \Psi(t) \ \middle| \ x\ \middle| \ \Psi(t) \ \right\rangle $ (with $x$ being either $\hat{x}_1 \otimes \mathbf{1}$ or $\mathbf{1} \otimes \hat{x}_2$) makes the complex rotation go away. Since we have an orthogonal basis, which means that the scalar products vanish after applying the $x$-Operators, we obtain an expectation value of 0 in both instances. For the second particle this is also true in all further systems.

This means for the first system:

\begin{align}
\left\langle x_1 \right\rangle =0
\end{align}

And in all systems:

\begin{align}
\left\langle x_2 \right\rangle =0
\end{align}

After applying $\hat{x}_1 \otimes \mathbf{1}$ we get the following for the time evolution of system two:

\begin{align}
x\left| \ \Psi_2(t) \ \right\rangle =  \frac{1}{2} \left( e^{-i E_{1,0}t}((\left| 0 \right\rangle + \sqrt{2} \left| 2 \right\rangle) \otimes \left| 0 \right\rangle) + e^{-i E_{0,0}t}(\left| 1 \right\rangle \otimes \left| 0 \right\rangle) \right)
\end{align}

This results in an expectation value of:

\begin{align}
\frac{1}{2\sqrt{2}} (e^{it(E_{1,0}-E_{0,0})}+e^{it(E_{0,0}-E_{1,0})}) =
\frac{1}{\sqrt{2}} cos((E_{1,0}-E_{0,0})t)
\end{align}

Also in the last system we have an oscillation of the expectation value with:

\begin{align}
\sqrt{2} \ Re \left\{ \sqrt{2}e^{-iEt} \right\}
\end{align}

#### Energy expectation values:

The first two systems are formulated in the energy eigenbasis. Therefore the expectation value for their respective energies is constant in time.

For the first system we find: $\langle E \rangle = E_{1,0}$

The second system is a bit more complicated: $\langle E \rangle =  \frac{1}{2} \cdot (E_{0,0} + E_{1,0})$

The coherent state also has a constant energy expectation value, e.g. $\langle E \rangle = E_{0,0} + \hbar\omega\mid\kern-0.2em\alpha\kern-0.2em\mid^2  = E_{0,0} + 2\hbar\omega$

__We were not sure, which parts of this exercise were supposed to be solved numerically and which parts were supposed to be solved analytically._______no_output_____### Dynamics_____no_output_____#### Second example_____no_output_____

#Calculating important values
N=5
vals2, vecs2 = eigenenergies(N,const)
vals2 = sorted(vals2)
E00 = vals2[0]
E10 = vals2[1]

def expval_x(t,E00,E10):
    return 1/np.sqrt(2)*np.cos((E10-E00)*t)

timesteps = np.linspace(0,20,100)

#Plotting the result
plt.plot(timesteps ,expval_x(timesteps,E00,E10), color = 'darkblue')
plt.title('Expectation value for $x$ in the second example system', size = 20)
plt.xlabel('Time  $t$', size = 18)
plt.ylabel(r'$ \langle x \rangle $', size = 18)
plt.xlim(0,20)
plt.grid(True)_____no_output_____

#### Interpretation of the probabilities for states $|n\rangle$

The probability to detect particle two in state $|0\rangle$ is always 1.

In the first system, we detect $x_1$ with a probabilty of 1 ( = 100% ) in the first excitation state and in the second system with a probability of $\frac{1}{2}$ in either state of the superposition, e.g  $ |0\rangle$ and $ |1\rangle$._____no_output_____


# Notebook from sanju1066/EasyMath
Path: math_ds_predict.ipynb

# Math Dataset prediction sample

> You can contact me on twitter as [@mandubian](http://twitter.com/mandubian)

This notebook demonstrates some predictions using a pre-trained Transformer on Mathematical Reasoning Dataset v1.0 (arithmeric/add_sub/easy) as explained in this very cool paper:

> Analysing Mathematical Reasoning Abilities of Neural Models
>
> David Saxton, Edward Grefenstette, Felix Hill, Pushmeet Kohli
>
> (Submitted on 2 Apr 2019)
>
>http://arxiv.org/abs/1904.01557

**On top of paper, the current project provides a few Numpy/Pytorch helpers to manipulate Mathematical Reasoning Dataset in a simpler way and train very fast different kinds of models on it. Without loading everything in memory, it tries to manage data in a streamed and lazy-loading way (as much as Python allows it).**

The idea of the paper is to provide a robust toolkit (https://github.com/deepmind/mathematics_dataset) to randomly and heterogenously generate mathematical datasets among multiple math problem categories:
    - algebra
    - numbers
    - polynomials
    - arithmetic
    - measurement
    - comparison
    - probability
    - calculus

All problem are constituted of a textual question like `what is 30 + 535?` (max 160 chars) and a textual response `565` (max 30 chars). So it's not an abstract representation of math problem but a very human one and it mixes Natural Language Processing with math concepts.

For each of those categories, it provides multiple operation modules. For ex, in `algebra` category:
    - mul
    - add_or_sub_in_base
    - simplify_surd
    - mul_div_multiple
    - mixed
    - nearest_integer_root
    - div
    - add_or_sub
    - add_sub_multiple
    - add_sub_multiple_longer
    - mul_div_multiple_longer
    - div_big
    - mul_big
    - mixed_longer
    - add_or_sub_big

Problem can be generated with different difficulties:
    - train-easy
    - train-medium
    - train-hard

It also provides test datasets for `interpolation` tests mixing all kinds of problem per category and `extrapolation` tests to measure generalization capabilities of models.

>An implementation of dataset generator is provided at >https://github.com/deepmind/mathematics_dataset and a v1.0 of the dataset is >available there

The main purpose of this dataset generator is to provide a playground to people who want to study how neural networks can learn to solve mathematical problems or even better learn math abstractions.
_____no_output_____

import math
import numpy as np
import torch
from torch.utils import data
import torch.optim as optim
import torch.nn.functional as F
from tensorboard_utils import Tensorboard

from math_dataset import (
    LazyFileMathDataset,
    MathDatasetManager,
    question_answer_to_position_batch_collate_fn
)

import model_process
import utils
import checkpoints

from tensorboard_utils import Tensorboard
from tensorboard_utils import tensorboard_event_accumulator

import matplotlib
import numpy as np
import matplotlib.pyplot as plt
%matplotlib notebook

print("Torch Version", torch.__version__)

%reload_ext autoreload
%autoreload 2Torch Version 1.0.0.dev20190402

## Initialization_____no_output_____### Initialize Cuda device_____no_output_____

device = torch.device("cuda")
print("device", device)device cuda

### Build Math-Dataset manager_____no_output_____

mdsmgr = MathDatasetManager(
  "/home/mandubian/datasets/mathematics_dataset/v1.0"
)initialized MultiFilesMathDataset with categories ['algebra', 'numbers', 'polynomials', 'arithmetic', 'measurement', 'comparison', 'probability', 'calculus'] and types ['train-easy', 'train-medium', 'train-hard', 'interpolate', 'extrapolate']

### Restore Transformer trained arithmetic/add_or_sub/easy_____no_output_____

exp_name = "math_ds_arithmetic_add_or_sub_easy"
unique_id = "2019-04-22T13:32:24"_____no_output_____

### Restore Transformer model for this experience_____no_output_____

# build default transformer model
model = utils.build_transformer()
# restore model from checkpoint
_ = checkpoints.restore_best_checkpoint(exp_name, unique_id, "validation", model)
Extracting state from checkpoints/math_ds_arithmetic_add_or_sub_easy_2019-04-22T13:32:24_validation_best.pth
Loading model state_dict from state found in checkpoints/math_ds_arithmetic_add_or_sub_easy_2019-04-22T13:32:24_validation_best.pth

### Plotting training of this model_____no_output_____

valid_ea = tensorboard_event_accumulator(
    f"./runs/{exp_name}_{unique_id}_eval/events.out.tfevents.1555941107.c77692173f6a"
)

interpolate_ea = tensorboard_event_accumulator(
    f"./runs/{exp_name}_{unique_id}_interpolate/events.out.tfevents.1555941116.c77692173f6a"
)

valid_accuracy = valid_ea.Scalars("epoch/accuracy")
valid_loss_per_char = valid_ea.Scalars("epoch/loss_per_char")

interpolate_accuracy = interpolate_ea.Scalars("epoch/accuracy")
interpolate_loss_per_char = interpolate_ea.Scalars("epoch/loss_per_char")
_____no_output_____plt.rcParams['figure.figsize'] = [10, 6]

fig, ax = plt.subplots()

ax.plot(
    list(map(lambda l: l.step, valid_accuracy)),
    list(map(lambda l: l.value, valid_accuracy)),
    marker='+', label='Validation Accuracy'
)
ax.plot(
    list(map(lambda l: l.step, interpolate_accuracy)),
    list(map(lambda l: l.value, interpolate_accuracy)),
    marker='+', label='Interpolate Accuracy'
)
plt.title('Algebra/Linear_1d Accuracy')
ax.legend(loc='upper left', frameon=False)
plt.xticks(np.arange(0, 20, step=1.0))
plt.yticks(np.arange(0.3, 1.0, step=0.1))
plt.show()_____no_output_____

> We see that in terms of validation, training reaches 94-95% on validation dataset and 60% for interpolate dataset._____no_output_____## Single Prediction (the 5 best predictions)_____no_output_____

model_process.predict_single(
    "what is 32 + 445?",
    model, device, n_best=5
)
_____no_output_____

## Multiple Predictions_____no_output_____

questions = [
    "what is 10 + 32?",
    "34 + 77?",
    "compute -23 + 129"
]

model_process.predict_multiple(questions, model, device)_____no_output_____

### All your base are belong to 111_____no_output_____

questions = [
    "What is 34 + 77",
    "What is 34 + 87",
]

print(model_process.predict_multiple(questions, model, device))

questions = [
    "What is 34 + 78",
    "What is 34 + 88",
]

print(model_process.predict_multiple(questions, model, device))[{'resp': '111', 'score': 0.0}, {'resp': '111', 'score': 0.0}]
[{'resp': '112', 'score': 0.0}, {'resp': '122', 'score': 0.0}]

> Ok it still has a few weird issues :p_____no_output_____### Find the top worst cases_____no_output_____

mdsmgr = MathDatasetManager(
  "/home/mandubian/datasets/mathematics_dataset/v1.0"
)

ds = mdsmgr.build_dataset_from_module('arithmetic', 'add_or_sub', 'train-easy')
print("size", len(ds))

ds_loader = data.DataLoader(
    ds, batch_size=128, shuffle=False, num_workers=4,
    collate_fn=question_answer_to_position_batch_collate_fn)
initialized MultiFilesMathDataset with categories ['algebra', 'numbers', 'polynomials', 'arithmetic', 'measurement', 'comparison', 'probability', 'calculus'] and types ['train-easy', 'train-medium', 'train-hard', 'interpolate', 'extrapolate']
size 666666
import math_dataset
import collections

# Very unefficient Works case computer
worst_k = collections.deque(maxlen=10)

def build_worst_k(batch_idx, all_hyp, all_scores):
    for i, idx_seqs in enumerate(all_hyp):
        r = math_dataset.np_decode_string(np.array(idx_seqs[0]))
        s = all_scores[i][0].cpu().item()
        if len(worst_k) == 0 or s < worst_k[0][2]:
            worst_k.appendleft((batch_idx + i, r, s))

model_process.predict_dataset(
    ds_loader, model, device,
    callback=build_worst_k,
    #max_batches=5
)
_____no_output_____
for (idx, r, s) in worst_k:
    q = ds[idx]["q"]
    a = ds[idx]["a"]
    print("-> question:", q)
    print("   predicted:", r, "(", a, ")")
    print("   solution:", a)
    print("   score", s)
-> question: What is 0.2 plus 0.1973?
   predicted: -5.0896 ( 0.3973 )
   solution: 0.3973
   score -1.60174560546875
-> question: What is -3.4 less than -2.9?
   predicted: 21.278 ( 0.5 )
   solution: 0.5
   score -1.5341033935546875
-> question: Put together 4 and 191647.
   predicted: -2.438 ( 191651 )
   solution: 191651
   score -1.3736724853515625
-> question: Put together 12193 and -7.
   predicted: -2155 ( 12186 )
   solution: 12186
   score -1.3134536743164062
-> question: -14649+2
   predicted: 2.754 ( -14647 )
   solution: -14647
   score -0.994903564453125
-> question: 2683+2
   predicted: 0.082 ( 2685 )
   solution: 2685
   score -0.789306640625
-> question: What is 209 - -0.8?
   predicted: 1146.84 ( 209.8 )
   solution: 209.8
   score -0.76519775390625
-> question: -9259432 + 1
   predicted: -63.5854 ( -9259431 )
   solution: -9259431
   score -0.63372802734375
-> question: -0.3 + 6703
   predicted: 904.269 ( 6702.7 )
   solution: 6702.7
   score -0.5973968505859375
-> question: What is -99634 plus 98?
   predicted: -99556 ( -99536 )
   solution: -99536
   score -0.5908584594726562

As you can see, there are completely absurd answers like:

```
-0.3 + 6703 => 904.269  (should be 6702.7)
```

and others that aren't so far:

```
-99634 plus 98 => -99536 (should be -99536)
```

It's quite hard to know what our transformer learnt. Certainly, not the math laws, neither any abstraction but it can perform a lot of additions in
_____no_output_____### Check Transformer Model trained on arithmetic/add_or_sub/hard_____no_output_____

exp_name = "math_ds_arithmetic_add_or_sub_hard"
unique_id = "20190423_1800"_____no_output_____

#### Reload model_____no_output_____

# build default transformer model
model = utils.build_transformer()
# restore model from checkpoint
_ = checkpoints.restore_best_checkpoint(exp_name, unique_id, "validation", model)
Extracting state from checkpoints/math_ds_arithmetic_add_or_sub_hard_20190423_1800_validation_best.pth
Loading model state_dict from state found in checkpoints/math_ds_arithmetic_add_or_sub_hard_20190423_1800_validation_best.pth

#### Plot accuracy evolution during training_____no_output_____

valid_ea = tensorboard_event_accumulator(
    f"./runs/{exp_name}_{unique_id}_eval/events.out.tfevents.1556039027.c77692173f6a"
)

interpolate_ea = tensorboard_event_accumulator(
    f"./runs/{exp_name}_{unique_id}_interpolate/events.out.tfevents.1556039036.c77692173f6a"
)

valid_accuracy = valid_ea.Scalars("epoch/accuracy")
valid_loss_per_char = valid_ea.Scalars("epoch/loss_per_char")

interpolate_accuracy = interpolate_ea.Scalars("epoch/accuracy")
interpolate_loss_per_char = interpolate_ea.Scalars("epoch/loss_per_char")
_____no_output_____plt.rcParams['figure.figsize'] = [10, 6]

fig, ax = plt.subplots()

ax.plot(
    list(map(lambda l: l.step, valid_accuracy)),
    list(map(lambda l: l.value, valid_accuracy)),
    marker='+', label='Validation Accuracy'
)
ax.plot(
    list(map(lambda l: l.step, interpolate_accuracy)),
    list(map(lambda l: l.value, interpolate_accuracy)),
    marker='+', label='Interpolate Accuracy'
)
plt.title('Algebra/Linear_1d Accuracy')
ax.legend(loc='upper left', frameon=False)
plt.xticks(np.arange(0, 20, step=1.0))
plt.yticks(np.arange(0.3, 1.0, step=0.1))
plt.show()_____no_output_____

Compared to model trainined Easy mode, it reaches 85% instead of 95% on validation dataset but it's much more efficient on interpolate dataset (almost same as validation dataset).

So it's much more able to manage diversity of interpolate dataset.

If we test on _____no_output_____

questions = [
    "-709581481585 - 0.144?"
]

print(model_process.predict_multiple(questions, model, device))
# solution is -709581481585.144[{'resp': '-709581481585.144', 'score': 0.0}]

Hey, it's not bad, right?

Test it on very simple additions:_____no_output_____

questions = [
    "What is 1 + 1",
    "What is 5 + 5",
]

print(model_process.predict_multiple(questions, model, device))[{'resp': '11.3', 'score': -0.00078582763671875}, {'resp': '51.5', 'score': -0.08533477783203125}]

Not very good in simple additions :D

```
1 + 1 => 11.3
```

So, it's not really computing the additions, it's more about learning to associate characters together in a sequence.

So, it's not yet going to replace our little calculators or even our brains but is it possible to push our relational models (or other kinds of models) so far that it understands the laws of additions? That's the stake for my next experiments ;)

> Next Steps:
> - test funnier models
> - test extrapolate models
> - train skynet ;)_____no_output_____


# Notebook from fjug/tutorials
Path: notebooks/1_-_Using_ImageJ/ImageJ_Ops/06_-_Evaluating_Expressions.ipynb

# Evaluating expressions_____no_output_____ImageJ Ops offers a powerful expression evaluation op, built on SciJava's [Parsington](https://github.com/scijava/parsington) library:_____no_output_____Let's get ImageJ ready and prepare an image for demonstration. _____no_output_____

#@ImageJ ij
import net.imglib2.FinalInterval

image = ij.io().open("http://imagej.net/images/clown.png")
w = image.dimension(0); h = image.dimension(1)
slice = FinalInterval.createMinSize(0, 0, 0, w, h, 1)

// crop to only red channel left
grayImage = ij.op().run("crop", image, slice, true)_____no_output_____ij.op().help("eval")_____no_output_____

Operators in the expression map to ops as follows:_____no_output_____

import net.imagej.ops.eval.OpEvaluator
new OpEvaluator().getOpMap()_____no_output_____

You can also call any op in an `eval` statement as a function, using familiar function syntax.

Here is an example of the `eval` op being used to compute a [Difference of Gaussians](https://en.wikipedia.org/wiki/Difference_of_Gaussians):_____no_output_____

image32 = ij.op().convert().float32(grayImage)
dogFormula = "gauss(image, [10, 10, 1]) - gauss(image, [5, 5, 1])"
dog = ij.op().eval(dogFormula, ["image": image32])_____no_output_____


# Notebook from Pegasus-01/Machine-Learning-Basics
Path: 02-Pandas/07-mergingData&operations.ipynb

import numpy as np
import pandas as pd_____no_output_____mydata = {'me':[3,5,np.NaN],'mf':[7,np.NaN,np.NaN],'mof':[3,5,4]}_____no_output_____mydataframe = pd.DataFrame(mydata)_____no_output_____mydataframe_____no_output_____mydataframe.dropna()_____no_output_____mydataframe.fillna(value= mydataframe['me'].mean())_____no_output_____

# Part 2_____no_output_____

newdata = {
    'Movies':['star-wars','star-wars','batman','batman','deadpool','deadpool'],
    'Viewers':['me','mf','mof','bobby','dean','sammy'],
    'Ratings':[10,7,6,9,8,5]
}_____no_output_____movieframe = pd.DataFrame(newdata)_____no_output_____movieframe_____no_output_____movieframe.groupby('Viewers').mean()_____no_output_____movieframe.groupby('Movies').mean().loc['deadpool']_____no_output_____


# Notebook from BrownDwarf/welter
Path: notebooks/welter_issue018-07_More_Photometry.ipynb

# Welter
## Issue 18- LkCa 4 photometry analysis

July, 2016

## Other data

There is more photometry out there:

|Study|band|
|---|---|
|Integral-OMC |$V-$band|
|TrES|$r-$band|
|NSVS|unfiltered|
|Herbst et al.| UBVR|
|ASAS3| $V$|
|SuperWASP| "WASP V" 400 to 700 nm|
|Coyote I| UBVRI|

_____no_output_____

import warnings
warnings.filterwarnings("ignore")
import numpy as np
from astropy.io import fits
import matplotlib.pyplot as plt_____no_output_____% matplotlib inline
% config InlineBackend.figure_format = 'retina'
import seaborn as sns
sns.set_context('paper', font_scale=1.4)
sns.set_style('ticks')_____no_output_____import pandas as pd_____no_output_____

## Integral-OMC data_____no_output_____

ff = fits.open('../data/photometry/IOMC_1827000018.fits')_____no_output_____hdu1 = ff[1]_____no_output_____hdu1.data.shape_____no_output_____tt = hdu1.data['BARYTIME']+2451544.5_____no_output_____fl = hdu1.data['MAG_V']_____no_output_____plt.plot(tt, fl, '.')_____no_output_____plt.plot(np.mod(tt, 3.375), fl, '.')
plt.ylim(13.3, 12.5)_____no_output_____

OK, so 138 extra $V$ band photometric data points spread out over 2-3 years.  Not bad._____no_output_____

OMC_out = pd.DataFrame({'HJD':hdu1.data['BARYTIME']+2451544.5,
                        'Vmag':hdu1.data['MAG_V'],
                        'Verr':hdu1.data['ERRMAG_V']}, columns=['HJD', 'Vmag', 'Verr'])_____no_output_____OMC_out.to_csv('../data/photometry/LkCa4_Integral_OMC.csv', index=False)_____no_output_____

## NSVS
This data comes from ROTSE.  Unfortunately it is *un*filtered, which makes it hard to compare directly to the other $V-$band data.  Still, maybe it's worth a peak.
I accessed the data from: http://skydot.lanl.gov/nsvs/star.php?num=6759899&mask=32004
It's in a pesky HTML format, so I just copied and pasted the output, rather than mess around with HTML tags._____no_output_____

nsvs_raw = pd.read_csv('../data/photometry/NSVS_LkCa4_unfiltered.tsv', delim_whitespace=True)_____no_output_____#nsvs_raw.tail()_____no_output_____plt.plot(nsvs_raw['MJD-50000'], nsvs_raw.mag, '.')_____no_output_____plt.plot(np.mod(nsvs_raw['MJD-50000'], 3.375), nsvs_raw.mag, '.')
plt.ylim(12.4, 11.8)_____no_output_____

Cool, but not very helpful since the filter isn't standardized._____no_output_____## Herbst & Shevchenko 1999
I downloaded the data from Vizier: http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=J/AJ/118/1043
It has these notes:
>#Note (1): JD may be negative (for dates before 2440000)
#Note (2): *: the V-R magnitudes given were made in the Johnson system

**This data originally appeared in**:
- Vrba et al. 1993
- Bouvier et al. 1993

and later appeared in:
- Grankin et al. 2008_____no_output_____

#! head ../data/photometry/J_AJ_118_1043_wtts_lkca4.dat.csv_____no_output_____hs99 = pd.read_csv('../data/photometry/J_AJ_118_1043_wtts_lkca4.dat.csv', delim_whitespace=True, skiprows=[1,2])_____no_output_____hs99['JD'] = hs99['JD']++2440000.0_____no_output_____

Convert 99.99's to `NaN`s._____no_output_____

bi = hs99 == 99.99
hs99[bi] = np.NaN_____no_output_____hs99['Bmag'] = hs99['B-V']+hs99['Vmag']
hs99['Rmag'] = hs99['Vmag']-hs99['V-R']
hs99['Imag'] = hs99['Rmag']-hs99['R-I']
hs99['Umag'] = hs99['U-B']+hs99['Bmag']_____no_output_____#ad = pd.read_csv('../data/photometry/LkCa4_grankin.dat')

#plt.plot(hs99.JD, hs99.Vmag, 'o')
#plt.plot(ad.HJD, ad.Vmag, '.')
#plt.vlines(2449000.0, 12.9, 12.2)
#plt.ylim(12.9, 12.2)_____no_output_____

Drop the photometry after JD $\sim2449000.0$, since it's repeated with Grankin data._____no_output_____

bi2 = hs99.JD > 2449000.0
hs99 = hs99.drop(hs99.index[bi2], axis=0)_____no_output_____ad = pd.read_csv('../data/photometry/LkCa4_grankin.dat')

plt.plot(hs99.JD, hs99.Vmag, 'o')
plt.plot(ad.HJD, ad.Vmag, '.')
plt.vlines(2449000.0, 12.9, 12.2)
plt.ylim(12.9, 12.2)_____no_output_____vid = hs99['JD'] < 2447000.0 #Vrba 1993
boid = hs99['JD'] > 2447000.0 #Vrba 1993
hs99['source'] = np.NaN
hs99.source[vid] = 'Vrba et al. 1993'
hs99.source[boid] = 'Bouvier et al. 1993'_____no_output_____hs99.to_csv('../data/photometry/LkCa4_herbst99.dat', index=False)_____no_output_____

## Bouvier et al. 1993 (Coyote)
Data available here:
ftp://cdsarc.u-strasbg.fr/pub/cats/J/A+AS/101/485/table1

**This data is included in the Herbst 1999 reanalysis data!**_____no_output_____

#! curl ftp://cdsarc.u-strasbg.fr/pub/cats/J/A+AS/101/485/table1 > ../data/photometry/bouvier1993_tbl1.tsv_____no_output_____#! head ../data/photometry/bouvier1993_tbl1.tsv_____no_output_____#bv_cols = ['name1', 'junk', 'HBC_num', 'JD-2400000', 'Vmag', 'B-V', 'U-B', 'V-R', 'V-I']
#bvd = pd.read_fwf('../data/photometry/bouvier1993_tbl1.tsv', usecols=[0,2,3,4,5,6,7,8],
#                  header=None, index_col=False, names=bv_cols)_____no_output_____#bvd_lkca4 = bvd[bvd.name1 == 'LkCa-4']_____no_output_____#plt.plot(np.mod(bvd_lkca4['JD-2400000'], 3.375), bvd_lkca4.Vmag, '.')
#plt.ylim(12.7, 12.4)_____no_output_____

## ASAS3_____no_output_____> Each data row consists of the following fields:
> -  HJD-2450000
> -  magnitudes (one for each aperture)
> -  frame errors describing average photometric quality of the frame (for each >aperture)
> -  frame number
> -  grade :
>    A - best data, no 29.999 (not measured) indication
>    B - mean data, no 29.999 (not measured) indication
>    C - A and B with 29.999 (not measured) indication
>    D - worst data, probably useless
_____no_output_____

#! curl http://www.astrouw.edu.pl/cgi-asas/asas_cgi_get_data?041628+2807.6,asas3 > ../data/photometry/ASAS3_LkCa4.dat_____no_output_____#! head ../data/photometry/ASAS3_LkCa4.dat_____no_output_____as3_names = ['HJD','MAG_0','MAG_1','MAG_2','MAG_3','MAG_4','MER_0','MER_1','MER_2','MER_3','MER_4','GRADE','FRAME']
as3 = pd.read_csv('../data/photometry/ASAS3_LkCa4.dat', comment='#', delim_whitespace=True, names=as3_names)_____no_output_____

Drop the overtly bad quality data.  Hopefully the errors are estimated correctly for everything else (they seem reasonable...)_____no_output_____

gi = (as3.MAG_0 < 20) & (as3.GRADE != 'D')
as3 = as3[gi]_____no_output_____#plt.plot(as3.HJD+2450000, as3.MAG_0, '.')
#plt.ylim(13.3, 12.4)_____no_output_____#plt.plot(np.mod(as3.HJD+2450000, 3.375), as3.MAG_0, '.')
#plt.ylim(13.3, 12.4)_____no_output_____

A bit noisy because I'm smearing over 2 years of data._____no_output_____

as3['JD_like'] = as3.HJD + 2450000.0
as3['date_type'] = 'HJD'_____no_output_____

Just use "MAG_0", they're all fairly similar..._____no_output_____

as3['Vmag'], as3['Verr']= as3.MAG_0, as3.MER_0_____no_output_____as3.columns_____no_output_____as3.drop(['HJD','MAG_0', 'MAG_1', 'MAG_2', 'MAG_3', 'MAG_4', 'MER_0', 'MER_1',
       'MER_2', 'MER_3', 'MER_4', 'GRADE', 'FRAME'], axis=1, inplace=True)_____no_output_____as3['source'] = 'ASAS3'_____no_output_____as3.to_csv('../data/photometry/LkCa4_asas3.csv', index=False)_____no_output_____

## SuperWASP
Unfortunately SuperWASP uses broadband filters.  See Figure 3 of Pollacco et al._____no_output_____

#! head ../data/photometry/1SWASP_J041628.10+280735.5.csv_____no_output_____sw_cols = ['HJD','camera','mag','mag_err']
sw = pd.read_csv('../data/photometry/1SWASP_J041628.10+280735.5.csv', names=sw_cols, skiprows=1)_____no_output_____#sw.head()_____no_output_____plt.plot(sw.HJD, sw.mag, '.')
plt.ylim(13.5, 12.0)_____no_output_____plt.plot(np.mod(sw.HJD, 3.375), sw.mag, '.', alpha=0.3)
plt.ylim(13.5, 12.0)_____no_output_____

## TrES
From Devor et al. 2008, from a website that differed from the one in the footnote of the 2008 paper.
Unfortunately this is $r-$band photometry_____no_output_____

#! head ../data/photometry/TrES_rband_T-Tau0-01262.lc_____no_output_____cols = ['HJD-2400000', 'r_band', 'r_unc']
TrES = pd.read_csv('../data/photometry/TrES_rband_T-Tau0-01262.lc', names=cols, delim_whitespace=True)_____no_output_____plt.plot(TrES['HJD-2400000'], TrES.r_band, '.')_____no_output_____

The data is "normalized", is that just mean-subracted?  _____no_output_____

plt.plot(np.mod(TrES['HJD-2400000'], 3.375), TrES.r_band, '.')
plt.ylim(0.6, -0.5)_____no_output_____

## APASS
Not many data points, only 4 I think...
From this query:
http://tombstone.physics.mcmaster.ca/APASS/conesearch_offset.php?radeg=64.11708&decdeg=28.12658&raddeg=0.005_____no_output_____

#! head ../data/photometry/APASS_LkCa4.dat_____no_output_____#ap_cols = ['JD_offset', 'mag', 'unc']
#ap_filter = pd.read_csv('../data/photometry/APASS_LkCa4.dat',
#                        delim_whitespace=True, skiprows=1, names='ABCDE', usecols='C')
#ap = pd.read_csv('../data/photometry/APASS_LkCa4.dat', sep=',', names=ap_cols, skiprows=1, comment='#')
#ap['band'] = ap_filter.C
#ap.head()_____no_output_____

Looks like APASS data is included in the AAVSO data anyways._____no_output_____## AAVSO data
Need to search for V1068 Tau_____no_output_____

#! curl https://www.aavso.org/sites/default/files/tmp/aavsodata_578db6103872a.txt > ../data/photometry/AAVSO_LkCa4.dat_____no_output_____#! head ../data/photometry/AAVSO_LkCa4.dat_____no_output_____aavso = pd.read_csv('../data/photometry/AAVSO_LkCa4.dat')_____no_output_____aavso.columns_____no_output_____aavso.Band.unique()_____no_output_____

Throw away the unknown photometric bands._____no_output_____

gi = (aavso.Band != 'Vis.') & (aavso.Band != 'CV')_____no_output_____aavso = aavso[gi]_____no_output_____aa_new = aavso[aavso.columns.values[[0,1,2,4]]].pivot(index='JD', columns='Band')_____no_output_____aa_new.to_csv('../data/photometry/LkCa4_aavso.csv')_____no_output_____

Seems legit._____no_output_____## All together_____no_output_____ASASSN_____no_output_____

asassn = pd.read_csv('../data/photometry/LkCa4_ASASSN_aggregated.dat')_____no_output_____plt.figure(figsize=(14, 6))
plt.plot(hs99.JD+2440000.0, hs99.Vmag, 'o', label='Herbst & Shevchenko 1999')
plt.plot(ad.HJD, ad.Vmag, '.', label='Grankin et al. 2008')
#plt.plot(bvd_lkca4['JD-2400000']+2440000.0, bvd_lkca4.Vmag, '.', label='Bouvier et al. 1993')
plt.plot(as3.HJD+2450000, as3.MAG_0, '.', label='ASAS3')
plt.plot(tt, fl, '.', label='Integral-OMC')
plt.plot(asassn.JD, asassn.mean_mag, '.', label='ASASSN')
plt.plot(aavso.JD[gi], aavso.Magnitude[gi], '.', label='AAVSO')
plt.vlines(2454826.0, 13.3, 12.2, linestyles='dashed',label='Low Res Double Spec and Triple Spec')
plt.vlines(2450781.0, 13.3, 12.2, label='2MASS')
plt.legend(loc='best')
plt.title('$V-$band data of LkCa 4')
plt.ylim(13.3, 12.0)

#plt.xlim(2446357.4-50, 2446519.2+50)_____no_output_____plt.figure(figsize=(14, 6))
plt.plot()
plt.plot(sw.HJD, sw.mag, '.', label ='SuperWASP: "WASP V')
plt.plot(TrES['HJD-2400000']+2400000.0, TrES.r_band+12.0, '.', label='TrES: $r-$band')
plt.plot(nsvs_raw['MJD-50000']+2400000.5+50000, nsvs_raw.mag, '.', label= 'NSVS: unfiltered')

plt.vlines(2454826.0, 13.3, 12.2)

plt.title('Photometry in filters other than $V$')
plt.legend(loc='best')
plt.ylim(14.3, 10.0)
plt.xlim(2446000, 2458000)_____no_output_____

## The end._____no_output_____


# Notebook from misialq/paycheck
Path: eval/figure3.ipynb

<code>
from os.path import join
import json
import csv
import glob
import os

import rpy2.rinterface
from pandas import read_csv, DataFrame, Series
from qiime2 import Artifact
from statsmodels.stats.weightstats import DescrStatsW
from scipy.stats import t, sem
from IPython.display import Image_____no_output_____%reload_ext rpy2.ipython_____no_output_____%%R
library(ggplot2)_____no_output_____empo3_dir = '/Users/benkaehler/Data/empo_3/'_____no_output_____columns = ['sample type', 'sample', 'fold']
data = {c:[] for c in columns}
for type_dir in glob.glob(join(empo3_dir, '*')):
    if not os.path.exists(join(type_dir, 'results', 'weights.qza')):
        continue
    type_ = os.path.basename(type_dir)
    if type_ in ('sterile-water-blank', 'single-strain', 'mock-community', 'nick', 'plant-surface'):
        continue
    for fold_dir in glob.glob(join(type_dir, 'tmp', 'fold-*')):
        _, fold = fold_dir.rsplit('-', 1)
        with open(join(fold_dir, 'sample_test.json')) as fh:
            try:
                samples = json.load(fh)
            except UnicodeDecodeError:
                print(join(fold_dir, 'sample_test.json'), 'is corrupted')
        data['sample'].extend(samples)
        data['sample type'].extend([type_]*len(samples))
        data['fold'].extend([fold]*len(samples))
folds = DataFrame(data)_____no_output_____abundances = read_csv(join(empo3_dir, 'abundances.tsv'), sep='\t')
abundances = abundances.set_index(['sample type', 'sample'])
prf = read_csv(join(empo3_dir, 'eval_taxa_prf.tsv'), sep='\t')
prf = prf[prf['sample type']  != 'plant-surface']
prf = prf.join(abundances, ['sample type', 'sample'])_____no_output_____corrupted_samples = list(set(prf['sample']) - set(folds['sample']))
corrupted_samples = DataFrame({'sample': corrupted_samples})
corrupted_samples['sample type'] = 'animal-corpus'
corrupted_samples['fold'] = '0'
folds = folds.append(corrupted_samples)
folds = folds.set_index(['sample type', 'sample'])
prf = prf.join(folds, ['sample type', 'sample'])_____no_output_____prf = prf[(prf['sample type'] != 'single-strain') &
          (prf['sample type'] != 'sterile-water-blank') &
          (prf['sample type'] != 'mock-community') &
          (prf['sample type'] != 'plant-surface')]
prf = prf[(prf['class weights'] == 'bespoke70') |
          (prf['class weights'] == 'uniform70')]
prf = prf[prf['method'] == 'weighted']
level7 = prf[((prf['level'] == 7) & (prf['class weights'] == 'bespoke70')) |
             ((prf['level'] == 6) & (prf['class weights'] == 'uniform70'))]_____no_output_____bespoke = level7[level7['class weights'] == 'bespoke70']
bespoke.rename(columns={'F-measure':'bespoke'}, inplace=True)
uniform = level7[level7['class weights'] == 'uniform70']
uniform.rename(columns={'F-measure':'uniform'}, inplace=True)
uniform = uniform.set_index(['sample type', 'sample', 'fold'])
for_diff = bespoke.join(uniform, ['sample type', 'sample', 'fold'], rsuffix='_uniform')_____no_output_____grouped = for_diff.groupby(['sample type', 'class weights', 'fold'])
def weighted_stats(x):
    d1 = DescrStatsW(x['bespoke'] - x['uniform'], weights=x['abundance'])
    return Series([d1.mean], index=['mean'])
figure1 = grouped.apply(weighted_stats)
figure1.reset_index(inplace=True)_____no_output_____grouped = figure1.groupby(['sample type', 'class weights'])
def weighted_stats(x):
    mu = x['mean'].mean()
    se = sem(x['mean'])
    lower, upper = t.interval(0.95, 5, mu, se)
    return Series([mu, lower, upper], index=['mean', 'lower', 'upper'])
figure1 = grouped.apply(weighted_stats)
figure1.reset_index(inplace=True)_____no_output_____old_labels = 'uniform70', 'bespoke70'
new_labels = 'Uniform Genus Level', 'Bespoke Species Level'
for old, new in zip(old_labels, new_labels):
    figure1.loc[figure1['class weights'] == old, 'class weights'] = new
old_labels = [
    'animal-corpus',
    'animal-distal-gut',
    'animal-proximal-gut',
    'animal-secretion',
    'animal-surface',
    'plant-corpus',
    'plant-rhizosphere',
    'plant-surface',
    'sediment-non-saline',
    'soil-non-saline',
    'surface-non-saline',
    'water-non-saline',
    'sediment-saline',
    'surface-saline',
    'water-saline'
]
new_labels = [
    'Animal corpus',
    'Animal distal gut',
    'Animal proximal gut',
    'Animal secretion',
    'Animal surface',
    'Plant corpus',
    'Plant rhizosphere',
    'Plant surface',
    'Sediment (non-saline)',
    'Soil (non-saline)',
    'Surface (non-saline)',
    'Water (non-saline)',
    'Sediment (saline)',
    'Surface (saline)',
    'Water (saline)'
]
for old, new in zip(old_labels, new_labels):
    figure1.loc[figure1['sample type'] == old, 'sample type'] = new
figure1['direction'] = ['none']*len(figure1)
figure1.loc[figure1['mean'] >= 0., 'direction'] = 'Bespoke Species Level'
figure1.loc[figure1['mean'] < 0., 'direction'] = 'Uniform Genus Level'_____no_output_____%%R -i figure1
figure1$sample.type = factor(figure1$sample.type, levels=c(
    'Animal corpus',
    'Animal distal gut',
    'Animal proximal gut',
    'Animal secretion',
    'Animal surface',
    'Plant corpus',
    'Plant rhizosphere',
    'Plant surface',
    'Sediment (non-saline)',
    'Soil (non-saline)',
    'Surface (non-saline)',
    'Water (non-saline)',
    'Sediment (saline)',
    'Surface (saline)',
    'Water (saline)'))
figure1$class.weights = factor(figure1$class.weights, levels=c(
    'Uniform Genus Level', 'Bespoke Species Level'
))
cbPalette <- c("#009E73", "#E69F00")
ggplot(data=figure1, aes(x=sample.type, y=mean, fill=direction)) +
  geom_bar(stat="identity", position="identity") +
  coord_flip() +
  theme_bw() +
  theme(legend.position="bottom",
        legend.direction="horizontal") +
  labs(x='EMPO3 Habitat', y='Unfair F-measure Difference') +
  geom_errorbar(aes(ymin=lower, ymax=upper), position=position_dodge()) +
  scale_fill_manual(values=cbPalette) +
  guides(fill=guide_legend(title="More Accurate", nrow=2, byrow=TRUE))
ggsave(file="figure3.png", width=5, height=5, dpi=300)_____no_output_____Image("figure3.png")_____no_output_____max(figure1['upper'] - figure1['lower'])_____no_output_____
</code>


# Notebook from FZJ-INM1-BDA/siibra-tutorials
Path: 02-DataFeatures.ipynb

# Extracting multimodal data features

`siibra` provides access to data features of different modalities using the `get_features` method, which accepts a feature modality and is sensitive to the selections configured in the atlas (parcellation, region). If not particular selection is made, `get_features` considers all brain regions of the current parcellation._____no_output_____

import siibra_____no_output_____

### Prerequisite: Allow  `siibra` to connect to the EBRAINS Knowledge Graph

Most of the features linked to atlas regions are curated datasets in the [EBRAINS knowledge graph](https://kg.ebrainseu) (KG). To fetch data from the KG, you need to pass an EBRAINS KG authentication token to siibra. For being able to generate such tokens, you need to follow three steps:

1. Register for an EBRAINS account, as described [here](https://ebrains.eu/register).
2. Enable API access for your EBRAINS account, as described [here](https://kg.humanbrainproject.eu/develop.html)).
3. Generate a token by visiting the [KG authorization endpoint](https://nexus-iam.humanbrainproject.org/v0/oauth2/authorize).

Step 1. and 2. need to be performed only once. Step 3 has to be performed every once in a while, since the KG does not (yet) provide persistent tokens.

Just copy the token text from the resulting page, and run the next cell to paste it in._____no_output_____

siibra.set_ebrains_token(input('Paste in EBRAINS KG access token: '))_____no_output_____

### Densities of neurotransmitter receptors

Transmitter receptor density fingerprints are linked to brain regions by their name in the EBRAINS Knowledge Graph. Like any data feature, they are accessed using the `get_features` method of the atlas, which makes use of the current selection in the atlas. The `get_features` method knows from the specified data modality that the match is determined from the brain region identified. Receptor densities come as a nicely structured datatype. Amongst other things, they can visualize themselves in a plot.

If we don't specify a particular selection, the atlas will return all available receptor density features linked to the parcellation map:_____no_output_____

atlas = siibra.atlases['human']
julichbrain = atlas.get_parcellation("julich")
for f in siibra.get_features(julichbrain,"receptor"):
    print(f.matched_region.name, f)_____no_output_____

If we select a region, the returned list is filtered accordingly. _____no_output_____

import siibra
atlas = siibra.atlases['human']
v1 = atlas.get_region('v1',parcellation="julich 2.9")
r, = siibra.get_features(v1,"receptor")
fig = r.plot(r.matched_region.name)_____no_output_____

### Layer-specific cell distributions in cortical regions

This version of siibra provides a pre-release preview of layer-specific cell distributions in different areas of BigBrain._____no_output_____

features = siibra.get_features(v1,"celldistribution")
for index in [0,-1]:
    # we plot only the first and last feature here
    fig = features[index].plot(features[index].matched_region.name)_____no_output_____

### Gene Expressions from the Allen Atlas

The atlas client can make calls to gene expression data from the Allen atlas and evaluate them in the ICBM space to find regional gene expression levels. It also has a list of available gene names for convenient selection. Gene expressions are linked to atlas regions by coordinates of their probes in MNI space. The `get_features` method detects this from the feature modality, and applies the mask of the regions that are currently selected in the atlas to filter the probes. We can visualize these filtered locations._____no_output_____

features = siibra.get_features(v1,"gene",gene="GABARAPL2")
print(features[0])

# plot
from nilearn import plotting
all_coords = [tuple(g.location) for g in features]
mask = v1.build_mask("mni152")
display = plotting.plot_roi(mask)
display.add_markers(all_coords,marker_size=5) _____no_output_____

### Connectivity matrices
For `siibra`, a connectivity matrix is a data feature like the others below, and can be found using the same `get_features` function just by choosing another modality. `siibra` knows from the modality type that this type of data does not match to the selected brain region in the atlas, but to the selected parcellation. Therefore, `get_features` return connectivity datasets that are defined for the selected parcellation.

To learn about the nature of the provided connectivity, the `src_info` attribute provides a detailed description of the dataset. _____no_output_____

# Get the first four connectivity matrices available for the parcellation
features = siibra.get_features(julichbrain,"connectivitymatrix")

# format dataset names for use as figure titles
from textwrap import wrap
titleformat = lambda text : "\n".join(wrap(text.replace('_',' '),20))

# plot the matrices
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
fig = plt.figure()
for i,feature in enumerate(features):
    ax = fig.add_subplot(1,len(features),i+1)
    ax.imshow(np.log(feature.array),cmap=plt.cm.hot)
    ax.set_title(titleformat(feature.name),size=10)
fig.show()_____no_output_____

### Locations of iEEG recordings

iEEG data is queried per session/subject. The data is organized into multiple electrodes per session, and multiple contact points per electrode. Iterating over a session yields its electrodes. Iterating over each electrode yields its contact points. Each of these items are themselves of type `SpatialFeature`, so they have a `.location` and can be matched against an atlas. Below, we use this to visualize MNI coordinates of contact points which correspond to electrodes touching the selected region. We highlight the contact points inside the selected atlas region in red._____no_output_____

sessions = siibra.get_features(v1,"ieeg")

coords = [pt
          for session in sessions
          for electrode in session
          for pt in electrode.location
          if electrode.match(v1)]

from nilearn import plotting
mask = v1.build_mask("mni152")
plotting.view_markers(
    [tuple(pt) for pt in coords],
    ["red" if pt.intersects_mask(mask) else "blue"
    for pt in coords ],
    marker_size=3)_____no_output_____

### Neuron morphologies

`siibra` has an interface to the [NeuroMorpho](https://neuromorpho.org) repository of single neuronal cell reconstructions._____no_output_____

import warnings
neurons = siibra.get_features(atlas.get_region('ifg'),'morphology')
for n in neurons[:2]:
    n.plot()_____no_output_____

### Other regional datasets_____no_output_____

for f in siibra.get_features(v1,"ebrains"):
    print(f"{f.name}\n{f.url}\n")_____no_output_____