# Notebook from bugkjw/bis438-bioinformatics
Path: Project#2/project2_ELAVL1.ipynb

In this project, we will predict RBP-binding RNA sequences using a neural network model based on CNN (B. Alipanahi, et al. "Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning", Nature Biotechnology, 33, pages831–838 (2015)).

1) Build CNN models for predicting the RBP-binding RNA sequences (ALKBH5 and ELAVL1).

2) Train your model using the training dataset (and optimize your model using validation set).

3) Report the ROC-AUC score of the test dataset using the trained model.

4) Use google colab (or whatever you want)

5) Upload  jupyter notebook (".ipynb" file) or link to your google colab code

Due data: Oct/11th 23:59pm

Grading criteria (10pt total):
- It runs anyway... --> 5pt
- Properly optimize the models --> 2pt
- Good performance --> 2pt
- Nice and proper comments --> 1pt
- It's a beautiful code! --> +3pt

Project #2에서는 RBP-binding RNA sequence를 예측하는 것인데, 이를 확장해서 binding motif도 역시 구하는 extended project를 수행함

프로젝트 수행 수준에 따라, 자네의 최종 점수는 중간고사의 class 평균점수 + alpha (물론 여기서 alpha는 negative number도 가능하다)_____no_output_____# Prediction of ELAVL1 binding sequences
In this notebook, we **predict ELAVL1 binding RNA sequences with a CNN model.**

Then as an extention, we also **analyze learned CNN filters to find local binding motif sequences.**

To effectively detect binding motifs of different lengths, we used an adapted version of **inception module** used for GoogleNet in image classification task.

The notebook is written as follows:

**Basic project**

- Module / data import

- Train / validation set split

- CNN architecture design

- Hyperparameter setting & model optimization

- Model evaluation on test dataset with ROC-AUC score

**Extended project**

- **Analysis of learned filters for local RNA sequence motifs**

References:

[1] B. Alipanahi, et al. "Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning", Nature Biotechnology, 33, pages831–838 (2015)

[2] Szegedy et al., "Going Deeper with Convolutions", CVPR (2015)

[3] Chung et al., "Prediction of binding property of RNA-binding proteins using multi-sized filters and multi-model deep convolutional network", PLOS ONE

[4] Zhang et al., "A deep learning framework for modeling structural
features of RNA-binding protein targets", Nucleic Acids Research_____no_output_____#Module / data import_____no_output_____

#### project2_pseudo_code.py

# import
import os
import numpy as np
import random

import keras
import tensorflow as tf
###################
import tensorflow.keras as keras
from keras import Model
from keras.backend import one_hot
from keras.layers import Activation, Dense, Dropout, Input, BatchNormalization
from keras.layers import Conv1D, Concatenate, MaxPooling1D, AveragePooling1D, Flatten
from keras.regularizers import l2
from keras.optimizers import adam
from keras.utils import np_utils
###################_____no_output_____# import my google drive
from google.colab import drive
drive.mount('/content/gdrive')_____no_output_____# read fasta files
fasta_file_path='gdrive/My Drive/data sample/'
rbp_name = 'ELAVL1'
print('list of files', os.listdir(fasta_file_path))_____no_output_____# loading training set
train_fasta = list()
train_label = list()
for single_file in [x for x in os.listdir(fasta_file_path) if rbp_name in x and 'train' in x]:
  print('Processing file...', single_file)
  with open(fasta_file_path + single_file) as f:
    for line in f.readlines():
      # get fasta sequence
      if '>' in line:
        continue
      else:
        train_fasta.append(line.strip())
      # get positive negative label
      if 'positives' in single_file:
        train_label.append(1)
      else:
        train_label.append(0)_____no_output_____# convert sequence file to one-hot encoding representation
# function: fasta to onehot representation
def convert2onehot(sequence_list):
  map = {
      'A':[1,0,0,0],
      'U':[0,1,0,0],
      'T':[0,1,0,0],
      'G':[0,0,1,0],
      'C':[0,0,0,1]
  }

  onehot = []
  for single_sequence in sequence_list:
    single_onehot = []
    for x in single_sequence:
      single_onehot.append(map[x.upper()])
    onehot.append(single_onehot)

  return np.asarray(onehot, dtype=np.float32)

data_input = convert2onehot(train_fasta)
data_label = keras.utils.to_categorical(train_label, 2)_____no_output_____

#Train / validation set split

80% training set / 20% validation set_____no_output_____

# Install sklearn package
!pip install imblearn_____no_output_____# split training set into training set and validation set
# random shuffling of training data
###################
# random shuffle indices for training data
arr = np.arange(data_input.shape[0])
np.random.shuffle(arr)

# apply same shuffling to data and label
data_input = data_input[arr,:,:]
data_label = data_label[arr,:]

# split training and validation dataset from raw dataset (9:1)
from sklearn.model_selection import train_test_split
train_input, validation_input, train_label, validation_label = train_test_split(data_input, data_label, test_size=0.1, random_state=1)
###################
print('Dataset preparation done... train_input, train_label, validation_input, validation_label')
print('Size of each set...', train_input.shape, train_label.shape, validation_input.shape, validation_label.shape)_____no_output_____

#CNN architecture design

**Design purpose: Build a model that predicts from & learns various lengths of sequence motifs.**

Inception module from GoogleNet does this (concatenation of different sized filters, [2]).

![inception_module](https://d2mxuefqeaa7sj.cloudfront.net/s_8C760A111A4204FB24FFC30E04E069BD755C4EEFD62ACBA4B54BBA2A78E13E8C_1490879611424_inception_module.png)

We can adapt this architecture to sequence model simply by changing 2d filters to 1d.

At the same time, we want to analyze learned filters in 1st inception module to look for RBP binding sequence motifs.

From the requirements, we designed the model as follows:

- Use simplified inception modules without 1-convolution sublayers

- Instead of small (1-, 3-, 5-) convolutional filters, use large ones (8-, 16-, 32-) for later analysis of filters in the first inception module [3]

- 3 inception modules -> Input size reduction by max-pooling -> FC layers for classification

#Optimization / regularization techniques

Below are some additional techniques we applied for optimization and regularization:

- Adam optimizer

- L2 weight regularization

- Dropout before prediction layer

- Early stopping

#Model training curve

![training_curve](https://drive.google.com/uc?export=download&id=1xgGiz_9_ECWh5_p8EaiL_V-wRw2jEeJN)_____no_output_____

train_mode = True_____no_output_____# model building
###################
# hyperparameters
lr = 3e-4
reg = 1e-2
p = 1/2

# design purpose: build a model that predicts from / learns various lengths of sequence motifs.
# GoogleNet inception module-based CNN architecture
# use modified inception module instead of plain convolutional layers

# inception module: detects motifs with different kernel sizes
# concatenation of activation maps from filters of variying lengths

# use 1d convolution layers, starting from 4-channel input initially
inputs = Input(shape=(200,4,))

# inception module 1
x1 = Conv1D(2, 8, strides=1, padding='same', data_format = 'channels_last')(inputs)
x2 = Conv1D(2, 16, strides=1, padding='same', data_format = 'channels_last')(inputs)
x3 = Conv1D(2, 32, strides=1, padding='same', data_format = 'channels_last')(inputs)
# concatenate motif scans along channel dimension
x = Concatenate(axis=2)([x1, x2, x3])
# ReLU activation
x = Activation('relu')(x)
# max-pooling
x = MaxPooling1D(pool_size=2, padding='valid', data_format='channels_last')(x)

# inception module 2
x1 = Conv1D(2, 8, strides=1, padding='same', data_format = 'channels_last')(x)
x2 = Conv1D(2, 16, strides=1, padding='same', data_format = 'channels_last')(x)
x3 = Conv1D(2, 32, strides=1, padding='same', data_format = 'channels_last')(x)
# concatenate motif scans along channel dimension
x = Concatenate(axis=2)([x1, x2, x3])
# ReLU activation
x = Activation('relu')(x)
# max-pooling
x = MaxPooling1D(pool_size=2, padding='valid', data_format='channels_last')(x)

# inception module 3
x1 = Conv1D(2, 8, strides=1, padding='same', data_format = 'channels_last')(x)
x2 = Conv1D(2, 16, strides=1, padding='same', data_format = 'channels_last')(x)
x3 = Conv1D(2, 32, strides=1, padding='same', data_format = 'channels_last')(x)
# concatenate motif scans along channel dimension
x = Concatenate(axis=2)([x1, x2, x3])
# ReLU activation
x = Activation('relu')(x)
# max-pooling
x = MaxPooling1D(pool_size=2, padding='valid', data_format='channels_last')(x)

# reducing input size with 1-convolution and max pooling
x = Conv1D(6, 1, strides=1, data_format='channels_last')(x)
x = MaxPooling1D(pool_size=5, padding='valid', data_format='channels_last')(x)
x = Activation('relu')(x)
x = Conv1D(6, 1, strides=1, data_format='channels_last')(x)
x = MaxPooling1D(pool_size=5, padding='valid', data_format='channels_last')(x)
x = Activation('relu')(x)

# flattening before FC layers
x = Flatten()(x)
# plain FC layers
x = Dense(64, kernel_regularizer=l2(reg))(x)
x = Activation('relu')(x)
x = Dropout(rate=p)(x)
# prediction layer
predictions = Dense(2, activation='softmax')(x)

# model compilation
model = Model(inputs=inputs, outputs=predictions)
model.compile(optimizer=adam(lr=lr,decay=1e-5),
                 loss='categorical_crossentropy',
                 metrics=['accuracy'])
model.summary() # print summary of model
###################

# model training
###################
if train_mode:
  # if training mode, train
  # early stopping with validation dataset
  from keras.callbacks import EarlyStopping
  from keras.callbacks import ModelCheckpoint

  es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=5)
  mc = ModelCheckpoint(fasta_file_path + 'ELAVL1_best.h5', monitor='val_acc', mode='max', verbose=1, save_best_only=True)

  # training the model using training and validation set
  epochs = 50
  # train AR-classification task
  hist = model.fit(train_input, train_label,
                   validation_data=(validation_input, validation_label),
                   batch_size=8, epochs=epochs, verbose=0, shuffle=True, callbacks=[es, mc])

  # visualize training curve
  import matplotlib.pyplot as plt
  fig, axs = plt.subplots(1,2, constrained_layout=True)

  axs[0].plot(hist.history['loss'], label='train loss')
  axs[0].plot(hist.history['val_loss'], label='val loss')
  axs[0].legend(loc='upper left')
  axs[0].set_title('loss')

  axs[1].plot(hist.history['acc'], label='train acc')
  axs[1].plot(hist.history['val_acc'], label='val acc')
  axs[1].legend(loc='upper left')
  axs[1].set_title('acc')

else:
  # if not training mode, load pre-trained weights
  model.load_weights(fasta_file_path + 'ELAVL1_best.h5')
  print("Model weights loaded")
###################_____no_output_____

#Model evaluation on test dataset with ROC-AUC score

With trained model, we report the performance on test set as follows:

Classification accuracy: 93%

AUC-ROC score: 0.97

![roc_curve](https://drive.google.com/uc?export=download&id=1igHW3s2RsPE3m_y0ZA2ILKZ_RxJj1jON)_____no_output_____

# evaluation: calculate AUC score for the test set
# loading test sets
###################
# loading training set
test_fasta = list()
test_label = list()
for single_file in [x for x in os.listdir(fasta_file_path) if rbp_name in x and 'test' in x]:
  print('Processing file...', single_file)
  with open(fasta_file_path + single_file) as f:
    for line in f.readlines():
      # get fasta sequence
      if '>' in line:
        continue
      else:
        test_fasta.append(line.strip())
      # get positive negative label
      if 'positives' in single_file:
        test_label.append(1)
      else:
        test_label.append(0)
###################

test_input = convert2onehot(test_fasta)
test_label = keras.utils.to_categorical(test_label, 2)
print('TEST set prepared...', test_input.shape, test_label.shape)

# calculate AUC score for test sets
###################
# evaluate the prediction model using test dataset.
from sklearn.metrics import classification_report, roc_curve, auc
import matplotlib.pyplot as plt

test_loss, test_acc = model.evaluate(test_input, test_label)
test_pred = model.predict(test_input)

# 0: negative, 1: positive
target_names = ['0','1']
print("classification report")
print(classification_report(test_label[:,1]==1, test_pred[:,1]>0.5, target_names=target_names))

# compute and draw ROC curve
fpr, tpr, thresholds = roc_curve(test_label[:,1], test_pred[:,1])
roc_auc = auc(fpr, tpr)

fig = plt.figure()
plt.plot(fpr, tpr, "b")
plt.plot([0, 0], [0, 1], "k--")
plt.plot([0, 1], [1, 1], "k--")
plt.plot([0, 1], [0, 1], "k--")
plt.title("AUC = %0.2f" % roc_auc)
plt.xlabel("FP rate (1 - specificity)")
plt.ylabel("TP rate (sensitivity)")
###################_____no_output_____

#Extended project: Analysis of learned filters for local RNA sequence motifs

In this section, we
(1) analyze learned filters of various lengths in our inception-like CNN,
(2) try to find RNA sequence motifs important for RBP binding and
(3) assess their biological plausibility.

Convolution filters, or motif detectors learned from RBP data, are shown below as 4 x length colormaps (each columns were mean-subtracted before visualization).

We can see that, for example, the first 8-length filter detects U-rich local region and the first 16-length filter specifically detects "CUCCUCCAUGG...".

![f8](https://drive.google.com/uc?export=download&id=1fnsSsxMJ0UATQquwDsx1zaQlN0RY3UST)

![f16](https://drive.google.com/uc?export=download&id=1TXNbLotXTMkSEFLTOYWjTQ9ezySponHH)

![f32](https://drive.google.com/uc?export=download&id=1uKDrw86-gsX_IrnYZOVb_ZFPcJZiV2lz)_____no_output_____

# load convolution filters from learned model
conv1 = model.layers[1].get_weights()
conv2 = model.layers[2].get_weights()
conv3 = model.layers[3].get_weights()
# load w and b
w1 = conv1[0]; b1 = conv1[1]
w2 = conv2[0]; b2 = conv2[1]
w3 = conv3[0]; b3 = conv3[1]

# function for visualization of learned filters
def vis_filter(length, w):
  fig = plt.figure()
  plt.title("filter length = "+str(length))
  plt.axis('off')
  for ii in range(2):
    ax = fig.add_subplot(2,1,ii+1)
    temp_img = w[:,:,ii].transpose() # filter image
    ax.imshow(temp_img-np.mean(temp_img,0),cmap='bwr') # mean subtraction for each position
    ax.set_xticks([],[])
    plt.yticks((0,1,2,3), ('A','U','C','G'))

# plot w of 8-length filter as a heatmap
vis_filter(8, w1)
# plot w of 16-length filter as a heatmap
vis_filter(16, w2)
# plot w of 32-length filter as a heatmap
vis_filter(32, w3)_____no_output_____

# (source code for sequence logo)
modified from: https://github.com/azofeifa/PWM_logo/tree/master/src

_____no_output_____

from math import log
import numpy as np
import math
import matplotlib.pyplot as plt

def despine(ax,right=True, top=True, left=False,bottom=False):
    if left:
        ax.spines['left'].set_visible(False)
    if bottom:
        ax.spines['bottom'].set_visible(False)
    if right:
        ax.spines['right'].set_visible(False)
    if top:
        ax.spines['top'].set_visible(False)

def draw_A(width, height, x, y, ax, color="blue",lw=2.0,alpha=0.5):
    ax.plot([x, x+width*0.5], [y, y+height],lw=lw, color=color ,alpha=alpha)
    ax.plot([x+width*0.5, x+width], [y+height, y],lw=lw, color=color ,alpha=alpha)
    ax.plot([x+width*0.25,x+width*0.75], [height*0.5, height*0.5], color=color, lw=lw,alpha=alpha)
    return y+height

def draw_U(width, height,x,y, ax, color="red",lw=2.0,alpha=0.5):
    ax.plot([x, x], [y+height/2, y+height],lw=lw, color=color,alpha=alpha)
    ax.plot([x+width, x+width], [y+height/2, y+height],lw=lw, color=color,alpha=alpha)
    x += width/2
    y += height/2
    ts = np.linspace(math.pi, math.pi*2, 100)
    xs = [math.cos(t)*width/2+x for t in ts]
    ys = [math.sin(t)*height/2+y for t in ts]
    ax.plot(xs,ys,lw=lw, color=color,alpha=alpha)
    return y + height/2

def draw_C(width, height,x,y, ax, color="green",lw=2.0,alpha=0.5):
    x += width/2
    y += height/2
    ts = np.linspace(3.85*math.pi/2., math.pi/6., 100)
    xs = [math.cos(t)*width/2+x for t in ts]
    ys = [math.sin(t)*height/2+y for t in ts]
    ax.plot(xs,ys,lw=lw, color=color,alpha=alpha)
    return y + height/2

def draw_G(width, height,x,y, ax, color="orange",lw=2.0,alpha=0.5):
    x += width/2
    y += height/2
    ts = np.linspace(3.85*math.pi/2., math.pi/6., 100)
    xs = [math.cos(t)*width/2+x for t in ts]
    ys = [math.sin(t)*height/2+y for t in ts]
    ax.plot(xs,ys,lw=lw, color=color,alpha=alpha)
    ptl = x, y-height/2
    ptu = xs[0],ys[0]
    ax.plot([ptu[0],ptu[0]], [ptu[1], ptl[1]],lw=lw, color=color,alpha=alpha)
    ax.plot([ptu[0]-width*0.75/2,ptu[0]], [ptu[1], ptu[1]],lw=lw, color=color,alpha=alpha)
    return y + height/2

def draw_seq_log(PSSM,title, AX=None,lw=1.0,alpha=1):
    ax = AX
    if AX is None:
        F = plt.figure(facecolor="white")
        ax = plt.gca()
    ax.set_title(title, fontsize=10)
    A = np.array(PSSM)
    for i in range(A.shape[0]):
        A[i,:] /= sum(A[i,:])
        methods = (draw_A, draw_U, draw_G, draw_C)
        x = i
        width = 0.9
        y = 0.0
        I = 2+sum([A[i,l]*math.log(A[i,l],2)  for l in range(4)])
        for j in range(4):
            d = methods[j]
            h = I*A[i,j]
            #print(h)
            y = d(width, h, x, y, ax, lw=lw, alpha=alpha)
    ax.set_xlim(0,A.shape[0])
    ax.set_xticks([])
    ax.set_xlabel("Position",fontsize=10)
    ax.set_ylabel("bits",fontsize=10)
    despine(ax,bottom=True)
    ax.yaxis.grid(True)
    ax.set_ylim(0,2)
    if AX is None:
        plt.show()

class logo:
    def __init__(self, matrix, name=""):
        self.m      = matrix
        self.name   = name
    def draw(self ,ax=None):
        draw_seq_log(self.m,self.name, AX=ax)

def main():
    A=[[ 0.356611,0.085184,0.455087,0.103118],
       [ 0.217352,0.033832,0.616781,0.132036],
       [ 0.411544,0.053398,0.486295,0.048762],
       [ 0.026776,0.902577,0.020027,0.050619],
       [ 0.773107,0.016504,0.055813,0.154576],
       [ 0.121956,0.027309,0.046496,0.804239],
       [ 0.010707,0.015609,0.947804,0.025881],
       [ 0.031773,0.310079,0.022036,0.636112],
       [ 0.055142,0.622140,0.063964,0.258755],
       [ 0.176130,0.035488,0.131170,0.657211],
       [ 0.095022,0.092511,0.793152,0.019314],
       [ 0.190134,0.058570,0.593589,0.157707],
       [ 0.456692,0.015034,0.473979,0.054296],
       [ 0.008153,0.976192,0.001984,0.013671],
       [ 0.767356,0.030153,0.037623,0.164869],
       [ 0.170081,0.032247,0.065448,0.732224],
       [ 0.036761,0.017177,0.926087,0.019975],
       [ 0.059032,0.416386,0.015859,0.508723],
       [ 0.114088,0.584113,0.060812,0.240987],
       [ 0.224960,0.355487,0.071233,0.348320]]
    #L        = logo(A, name="P53")
    #F        = plt.figure(figsize=(12,10),facecolor="white")
    #ax       = plt.gca()
    #L.draw(ax=ax)
    #plt.show()
if __name__ == "__main__":
    main()_____no_output_____

#Extended project: Binding motifs & sequence logos

To visualize the patterns learned by each 1st layer filters, we generate a PPM (position probability matrix) derived from the filter's response to actual sequences[1].

1. All positive sequences from training and test set are fed through the convolutional and rectifying (ReLU) operations of the model.

2. Then, all the sequences that passed the ReLU threshold (activation>0 for one or more positions in activation map) are aligned.

3. From the aligned sequences (assembly), a PPM is computed by counting the occurrences of bases in each position, which is then transformed to a sequence logo.

As a result, we get 1 sequence motif per 1 convolutional filter.

Below are sequence logos of learned filters:

**Filter length 8**

8-1

![1](https://drive.google.com/uc?export=download&id=11_VIrpsoGi8LXwViwcdMgf7Hy0NaXOn6)

8-2

![2](https://drive.google.com/uc?export=download&id=1N52MgzFdRzbKx4Eee9q5osiAWf4ewXSK)

**Filter length 16**

16-1

![3](https://drive.google.com/uc?export=download&id=1HVqxLAKE9_uJPE15ENLhvzGNZXK6oR4I)

16-2

![4](https://drive.google.com/uc?export=download&id=1x21aEjqTAhXar1cFfeppG3WIwbjewqIo)

**Filter length 32**

32-1

![5](https://drive.google.com/uc?export=download&id=1CEuQhbOtgghPifaBNDOUojuCjPDDJYK7)

32-2

![6](https://drive.google.com/uc?export=download&id=1NjuAwQbJy4Jfx1f8psVQApu3mDONAttN)

Let's compare the sequences to RBP-binding motifs of ELAVL1 known in literature.

**Known motif**

CISBP-RNA database

![literature](https://drive.google.com/uc?export=download&id=1d0ckS1s8Tu9Nowgo39NXCDpygQPUwoBA)

HITS-CLIP database

![literature2](https://drive.google.com/uc?export=download&id=1eWzhflUmpcPfSihwTvaM-ghwLHyLqpdL)

As seen from sequences above, the CNN filters learn to detect short sequences with frequent occurence of U / AU neucleotides (8-1, 8-2, 16-1, 16-2, 32-1), which is in fact similar to RBP-binding motif of ELAVL1 known from experiment.

This implies that filters of CNN optimized for RBP-binding prediction is able to capture biologically important sequence motifs in RBP-binding process.

**Biological meaning of this binding motif is discussed later in project2_ALKBH5.ipynb!**_____no_output_____

# gather positive RBP sequences
positive = np.reshape(np.asarray(np.where(test_label[:,1])),-1) # use only positive test cases
bind_input = test_input[positive,:,:]
positive = np.reshape(np.asarray(np.where(train_label[:,1])),-1) # use only positive train cases
bind_input = np.concatenate((bind_input,train_input[positive,:,:]),axis=0)

# function for PPM computation and sequence logo generation
def gen_logo(length, w, b, logo_width):
  # generating sequence logos
  # n: filter length
  # w: filters as weight matrices
  # b: filter biases
  # logo_width: sequence logo width
  n = 2 # number of filters per each length
  n_test = bind_input.shape[0] # number of test data
  thr = 0
  #thr = np.sum(np.max(w,1),0)*0.7

  w_2d = np.reshape(w,(-1,n))
  seq = np.zeros((n_test,length,4,n)) # subsequences yielding max response for a filter
  keep = np.zeros((n_test,n)) # sequences to keep / discard

  # Align sequences for each filters
  for ii in range(n_test):
    img = bind_input[ii,:,:] # an input image, 200*4
    s = np.zeros((200-length,n)) # sliding activation map, slide*n
    for jj in range(200-length): # sliding convolution, 'valid'
      img_1d = np.reshape(img[jj:jj+length,:],(1,-1)) # 1*24
      s[jj,:] = np.matmul(img_1d,w_2d) # convolution operation
      s[jj,:] += b # bias addition

    max_s = np.max(s,0) # compute maximum activation of each filters
    keep[ii,:] = max_s>thr # determine exclusion from assembly
    # select max scoring subsequence for each filter
    max_t = np.argmax(s,0)
    for ff in range(n): seq[ii,:,:,ff] = img[max_t[ff]:max_t[ff]+length,:]

  # Select positive-responding sequences and create PPM of each filter
  for ff in range(n):
    # compute PPM of each filter
    tmp = np.reshape(np.asarray(np.where(keep[:,ff])),-1) # sequences to exclude / include
    ppm = np.mean(seq[tmp,:,:,ff],0) # compute PPM from aligned assembly
    ppm[np.where(ppm==0)] += 1e-10 # for numerical stability

    # generate sequence logo
    L = logo(ppm, name=str(ff+1))
    F = plt.figure(figsize=(logo_width,3),facecolor="white")
    ax = plt.gca()
    L.draw(ax=ax)
    plt.show()_____no_output_____gen_logo(length=8, w=w1, b=b1, logo_width = 5)
gen_logo(length=16, w=w2, b=b2, logo_width = 10)
gen_logo(length=32, w=w3, b=b3, logo_width = 15)_____no_output__________no_output_____


# Notebook from yaromar/project1
Path: pipeline/.ipynb_checkpoints/ORIGINAL-checkpoint.ipynb

<code>
#!/usr/bin/env python 3
import re

PATH = "../data/Interfaces/"
#PATH = "/net/pan1/interactomes/pipeline/Interactome/Workflow/Interfaces/"
CHAIN_FILE = "text.tsv"
PDB_LIST = "pdbList.txt"_____no_output_____#PARAMETERS:
#fils is a string with path to the file to be checked

#RESULTS:
#Returns 1 or 0 depending on file existence

def file_check(file):

    try:
        open(file, "r")
        return 1

    except IOError:
        print("Error: " + file + " does not appear to exist.")
        return 0_____no_output_____#PARAMETERS:
#name is a string with the name of chain
#typeCount is a 2-element list with the first one being an empty string and the second - 0 (iteger)

#RESULTS:
#Updates the typeCount array: the first element = general histone type(s); the second element = (should be) number of histones in chain

def is_histone(name, typeCount):

    if(not re.search(r'chaperone|ase|binding|p53 peptide|non-histone|jmjc|rna|synth', name, re.I)):
        if(re.search(r'histone.*h?\d|h?\d.*histone|h?\d.*histone-like|histone-like.*h?\d|histone macro.*h?\d|h?\d.*histone macro|h?\d.*\speptide|\speptide.*h?\d|h3k4me0|h3(1-9)k4me3|$h\d^|archaeal histone|histone peptide', name, re.I)):
            typeCount[1] = 1 #adds the number of histones in chain  (should be changed to actual number of histones in chain!!!)

            if(re.search(r'h2a', name, re.I)):
                typeCount[0] += 'h2a|'

            elif(re.search(r'h2b', name, re.I)):
                typeCount[0] += 'h2b|'

            elif(re.search(r'h3', name, re.I)):
                typeCount[0] += 'h3|'

            elif(re.search(r'h4', name, re.I)):
                typeCount[0] += 'h4|'

            elif(re.search(r'h1', name, re.I)):
                typeCount[0] += 'h1|'

            elif(re.search(r'h5', name, re.I)):
                typeCount[0] += 'h5|'

            else:
                typeCount[0] += 'some histone|'_____no_output_____#PARAMETERS:
#pdbList is a text file with a header and one column PDB
#files is a list
#parameter is a string, either 'mapping' or 'interface' depending on desired results

#RESULTS:
#A list of absolute paths to either mapping files or interface files as it is stored on local NCBI machines

def get_files(pdbList, files, parameter):

    with open(pdbList, 'r') as pfh:
        pfh.readline()

        if(parameter == 'mapping'):

            for line in pfh:
                line = line.strip()
                folder = line[1] + line[2]
                files.append(PATH + folder + '/' + line + '_chain_protein_mapping.tab')

        elif(parameter == 'interface'):

            for line in pfh:
                line = line.strip()
                folder = line[1] + line[2]
                files.append(PATH + folder + '/' + line + '_atomic_contacts_5.0A.tab')_____no_output_____#PARAMETERS:
#pdb is a string with pdb id
#parameter is a string, either 'mapping' or 'interface' depending on desired results

#RESULTS:
#An absolute paths to either mapping file or interface file as it is stored on local NCBI machines

def get_file(pdb, parameter):

    if(parameter == 'mapping'):
        folder = pdb[1] + pdb[2]
        file = (PATH + folder + '/' + pdb + '_chain_protein_mapping.tab')
        return file

    elif(parameter == 'interface'):
        folder = pdb[1] + pdb[2]
        file = (PATH + folder + '/' + pdb + '_atomic_contacts_5.0A.tab')
        return file_____no_output_____#PARAMETERS:
#cFile is tab-separated file with a header and 4 columns: pdb, chain, uniprot, name
#dictionary is nested with the innermost dict being dictionary['pdb'] = {}

#RESULTS:
#The format of the end-product dictionary is: {pdb : {AlexChain: myChain|UNIPROT|name|type|process|function|organism|nucleosome(bool)|bindingPartner(bool)}}
#Example: {1alq : {'G': 'E|P02302|Histone H3.3C|H3|1212|4141|human|1|0'}}

def get_chain_dictionaries(cFile, dictionary):

    with open(cFile, 'r') as cfh:
        cfh.readline()

        histoneCount = {} #is used to count number of histones in a structure!!!!!!!

        tempDict = {}
        tempDict['pdb'] = {}

        mappingFiles = []
        get_files(PDB_LIST, mappingFiles, 'mapping')

        for file in mappingFiles:

            try: #adds a pdb entry to the dict only if mapping file exists

                with open(file, 'r') as mfh:
                    mfh.readline() #skips header
                    pdb = file.split('/')[-1].split('_', 1)[0]

                    for mLine in mfh:
                        chainPair = mLine.split('\t', 2)
                        alexChain = chainPair[0]
                        myChain = chainPair[1]

                        if(pdb in tempDict):
                            tempDict[pdb][alexChain] = myChain

                        else:
                            tempDict[pdb] = {alexChain : myChain}

            except IOError:
                pass
                #print("Error: " + mappingFile + " does not appear to exist.")

        for cLine in cfh:
            fields = cLine.strip().split('\t')

            pdb = fields[0]

            if(pdb in tempDict): #continues only if a mapping file exists
                chain = fields[1]
                organism = fields[2]
                process = fields[3]
                function = fields[4]
                uniprot = fields[5]
                name = fields[6]

                histoneTypeAndCount = ['', 0]

                is_histone(name, histoneTypeAndCount) #checks whether the name looks like a histone!!!!!!!!!

                tempType = histoneTypeAndCount[0]
                tempCount = histoneTypeAndCount[1]

                #######################
                if(tempCount): #if the chain is a [part of a] histone

                    if(pdb in histoneCount):

                        if(tempType not in histoneCount[pdb]):
                            histoneCount[pdb].append(tempType) #!!!!!!

                    else:
                        histoneCount[pdb] = [tempType]

                try: #adds a chain entry to the dict only if there exists a corresponding chain in the mapping file
                    alexChain = list(tempDict[pdb].keys())[list(tempDict[pdb].values()).index(chain)]

                    if(pdb in dictionary):

                        if(tempCount): #checks if chain is a histone!!!!
                            dictionary[pdb][alexChain] = str(tempDict[pdb][alexChain]) + '|' + uniprot + '|' + name + '|' + tempType + '|' + process + '|' + function + '|' + organism #!!!!

                        else: #!!!!
                            dictionary[pdb][alexChain] = str(tempDict[pdb][alexChain]) + '|' + uniprot + '|' + name + '|' + 'other|' + '|' + process + '|' + function + '|' + organism#!!!!

                    else:

                        if(tempCount): #checks if chain is a histone!!!!
                            dictionary[pdb] = {alexChain : str(tempDict[pdb][alexChain]) + '|' + uniprot + '|' + name + '|' + tempType + '|' + process + '|' + function + '|' + organism} #!!!!

                        else: #!!!!
                            dictionary[pdb] = {alexChain : str(tempDict[pdb][alexChain]) + '|' + uniprot + '|' + name + '|' + 'other|' + '|' + process + '|' + function + '|' + organism} #!!!!

                except ValueError:
                    #print("Error: " + str(ValueError) + ", in " + pdb)
                    pass

        for structure in list(dictionary):

            if(structure in histoneCount):
                uniqueHistoneNum = len(histoneCount[structure])
                partnerFlag = 0

                if(uniqueHistoneNum > 2): #checks if pdb has at least a 3 different histones ~ is a nucleosome!!!!!!!

                    for chain in dictionary[structure]:
                        chainFields = dictionary[structure][chain].split('|')

                        chainType = chainFields[3]##

                        dictionary[structure][chain] += 'nucleosome:1|' #!!!!!!

                        if(partnerFlag == 0 and chainType == 'other'):
                            partnerFlag = 1

                    if(partnerFlag == 0):

                        for chain in dictionary[structure]:
                            dictionary[structure][chain] += 'bp:0|'

                    else:

                        for chain in dictionary[structure]:
                            dictionary[structure][chain] += 'bp:1|'

                else: #!!!!!!

                    for chain in dictionary[structure]: #!!!!!
                        chainFields = dictionary[structure][chain].split('|')

                        chainType = chainFields[3]

                        dictionary[structure][chain] += 'nucleosome:0|' #!!!!!!

                        if(partnerFlag == 0 and chainType == 'other'):
                            partnerFlag = 1

                    if(partnerFlag == 0):

                        for chain in dictionary[structure]:
                            dictionary[structure][chain] += 'bp:0|'

                    else:

                        for chain in dictionary[structure]:
                            dictionary[structure][chain] += 'bp:1|'

            else:
                del dictionary[structure]_____no_output_____#PARAMETERS:
#interfaceFiles is a list of strings containing names of interface files
#chainDictionary is a dictionary produced by 'get_chain_dictionaries'
#interfaceDictonary is a nested dictionary with the innermost dict being interfaceDictionary['uniprotPair'] = {}, where the values are lists of the next form [chain pair data, residue number, pdb IDs]
#Example: 'P84233@P62799': {'44': ['A|P84233|Histone H3.2|h3|1@B|P62799|Histone H4|h4|1$E|P84233|Histone H3.2|h3|1@F|P62799|Histone H4|h4|1', 17, '1zla$q5cl']
#note that data fields of a chain is separated by |, @ separates chain from binding partner chain, $ separates chain pairs and pdb structures

def residue_count(interfaceFiles, chainDictionary, interfaceDictionary):

    for file in interfaceFiles:
        pdb = file.split('/')[-1].split('_', 1)[0]

        try:

            with open (file, 'r') as ifh:
                ifh.readline()

                atomList1 = [] #atomic count
                atomList2 = [] #atomic count

                for line in ifh:
                    lineFields = line.split('\t')

                    chain1 = lineFields[0].split('_', 1)[0] # the split part treats biological assembly chains as separate chains ???
                    chain2 = lineFields[4].split('_', 1)[0]

                    residue1 = lineFields[2]
                    residue2 = lineFields[6]

                    atom1 = lineFields[3] #atomic count
                    atom2 = lineFields[7] #atomic count

                    atomEntry1 = chain1 + residue1 + atom1 #atomic count
                    atomEntry2 = chain2 + residue2 + atom2 #atomic count

                    fields1 = chainDictionary[pdb][chain1].split('|', 5)
                    fields2 = chainDictionary[pdb][chain2].split('|', 5)

                    uniprot1 = fields1[1]
                    uniprot2 = fields2[1]

                    type1 = fields1[3]
                    type2 = fields2[3]

                    uniprotPair1 = uniprot1 + '@' + uniprot2
                    chainPair1 = chainDictionary[pdb][chain1] + '@' + chainDictionary[pdb][chain2]

                    uniprotPair2 = uniprot2 + '@' + uniprot1
                    chainPair2 = chainDictionary[pdb][chain2] + '@' + chainDictionary[pdb][chain1]

                    if(type1 != 'other'):

                        if(uniprotPair1 in interfaceDictionary):

                            if(residue1 in interfaceDictionary[uniprotPair1]):
                                #'interfaceDictionary[uniprotPair1][residue1][1] += 1  #

                                if(atomEntry1 not in atomList1): #atomic count
                                    interfaceDictionary[uniprotPair1][residue1][1] += 1
                                    atomList1.append(atomEntry1) #atomic count

                                if(chainPair1 not in interfaceDictionary[uniprotPair1][residue1][0]):
                                    interfaceDictionary[uniprotPair1][residue1][0] += '$' + chainPair1

                                    if(pdb not in interfaceDictionary[uniprotPair1][residue1][2]):
                                        interfaceDictionary[uniprotPair1][residue1][2] += '$' + pdb

                            else:
                                interfaceDictionary[uniprotPair1][residue1] = [chainPair1, 1, pdb]    #format of the innermost dict
                                atomList1.append(atomEntry1) #atomic count

                        elif(uniprotPair2 not in interfaceDictionary):
                            interfaceDictionary[uniprotPair1] = {residue1 : [chainPair1, 1, pdb]}
                            atomList1.append(atomEntry1)#atomic count

                        else:

                            if(residue2 in interfaceDictionary[uniprotPair2]):

                                if(atomEntry2 not in atomList2): #atomic count
                                    interfaceDictionary[uniprotPair2][residue2][1] += 1
                                    atomList2.append(atomEntry2) #atomic count

                                if(chainPair2 not in interfaceDictionary[uniprotPair2][residue2][0]):
                                    interfaceDictionary[uniprotPair2][residue2][0] += '$' + chainPair2

                                    if(pdb not in interfaceDictionary[uniprotPair2][residue2][2]):
                                        interfaceDictionary[uniprotPair2][residue2][2] += '$' + pdb

                            else:
                                interfaceDictionary[uniprotPair2][residue2] = [chainPair2, 1, pdb]    #format of the innermost dict
                                atomList2.append(atomEntry2) #atomic count

                    elif(uniprotPair1 not in interfaceDictionary):

                        if(uniprotPair2 in interfaceDictionary):

                            if(residue2 in interfaceDictionary[uniprotPair2]):

                                if(atomEntry2 not in atomList2): #atomic count
                                    interfaceDictionary[uniprotPair2][residue2][1] += 1
                                    atomList2.append(atomEntry2) #atomic count

                                if(chainPair2 not in interfaceDictionary[uniprotPair2][residue2][0]):
                                    interfaceDictionary[uniprotPair2][residue2][0] += '$' + chainPair2

                                    if(pdb not in interfaceDictionary[uniprotPair2][residue2][2]):
                                        interfaceDictionary[uniprotPair2][residue2][2] += '$' + pdb

                            else:
                                interfaceDictionary[uniprotPair2][residue2] = [chainPair2, 1, pdb]    #format of the innermost dict
                                atomList2.append(atomEntry2) #atomic count

                        else:
                            interfaceDictionary[uniprotPair2] = {residue2 : [chainPair2, 1, pdb]}
                            atomList2.append(atomEntry2) #atomic count

                    else:

                        if(residue1 in interfaceDictionary[uniprotPair1]):

                            if(atomEntry1 not in atomList1): #atomic count
                                interfaceDictionary[uniprotPair1][residue1][1] += 1
                                atomList1.append(atomEntry1) #atomic count

                            if(chainPair1 not in interfaceDictionary[uniprotPair1][residue1][0]):
                                interfaceDictionary[uniprotPair1][residue1][0] += '$' + chainPair1

                                if(pdb not in interfaceDictionary[uniprotPair1][residue1][2]):
                                    interfaceDictionary[uniprotPair1][residue1][2] += '$' + pdb

                        else:
                            interfaceDictionary[uniprotPair1][residue1] = [chainPair1, 1, pdb]    #format of the innermost dict
                            atomList1.append(atomEntry1) #atomic count
#CHECK FOR REDUNDANCY!!!

#                     if(uniprotPair1 in interfaceDictionary):

#                         if(residue1 in interfaceDictionary[uniprotPair1]):
#                             interfaceDictionary[uniprotPair1][residue1][1] += 1

#                             if(chainPair1 not in interfaceDictionary[uniprotPair1][residue1][0]):
#                                 interfaceDictionary[uniprotPair1][residue1][0] += '$' + chainPair1

#                                 if(pdb not in interfaceDictionary[uniprotPair1][residue1][2]):
#                                     interfaceDictionary[uniprotPair1][residue1][2] += '$' + pdb

#                         else:
#                             interfaceDictionary[uniprotPair1][residue1] = [chainPair1, 1, pdb]    #format of the innermost dict

#                     elif(uniprotPair2 in interfaceDictionary):

#                         if(residue2 in interfaceDictionary[uniprotPair2]):
#                             interfaceDictionary[uniprotPair2][residue2][1] += 1

#                             if(chainPair2 not in interfaceDictionary[uniprotPair2][residue2][0]):
#                                 interfaceDictionary[uniprotPair2][residue2][0] += '$' + chainPair2

#                                 if(pdb not in interfaceDictionary[uniprotPair2][residue2][2]):
#                                     interfaceDictionary[uniprotPair2][residue2][2] += '$' + pdb

#                         else:
#                             interfaceDictionary[uniprotPair2][residue2] = [chainPair2, 1, pdb]

#                     else:
#                          interfaceDictionary[uniprotPair1] = {residue1 : [chainPair1, 1, pdb]}

        except (IOError, KeyError) as e:
            #print("Error: " + interfaceFile + " does not appear to exist.")
            pass_____no_output_____def normalize_count(interfaceDictionary):

    for pair in interfaceDictionary:

        for residue in interfaceDictionary[pair]:
            pdbCount = len(interfaceDictionary[pair][residue][2].split('$'))
            interfaceDictionary[pair][residue].append(interfaceDictionary[pair][residue][1] / pdbCount) #[3] is normalized by uniprot pair_____no_output_____def average_histones(interfaceDictionary):

    avgDict = {}
    avgDict['type'] = {}

    for pair in interfaceDictionary:

        for residue in interfaceDictionary[pair]:
            targetFields = interfaceDictionary[pair][residue][0].split('@')[0].split('|')
            sourceFields = interfaceDictionary[pair][residue][0].split('@')[1].split('|')

            if(targetFields[3] != 'other'):
                histoneType = targetFields[3]
                normalizedCount = interfaceDictionary[pair][residue][3]

                if(histoneType in avgDict):

                    if(residue in avgDict[histoneType]):
                        avgDict[histoneType][residue][0] += normalizedCount
                        avgDict[histoneType][residue][1] += 1 #for normalization

                    else:
                        avgDict[histoneType][residue] = [normalizedCount, 1]

                else:
                    avgDict[histoneType] = {residue : [normalizedCount, 1]}

            elif(sourceFields[3] != 'other'):
                histoneType = sourceFields[3]
                normalizedCount = interfaceDictionary[pair][residue][3]

                if(histoneType in avgDict):

                    if(residue in avgDict[histoneType]):
                        avgDict[histoneType][residue][0] += normalizedCount
                        avgDict[histoneType][residue][1] += 1 #for normalization

                    else:
                        avgDict[histoneType][residue] = [normalizedCount, 1]

                else:
                    avgDict[histoneType] = {residue : [normalizedCount, 1]}

    avgDict2 = {}
    avgDict2['type'] = {}

    for histoneType in avgDict:

        for residue in avgDict[histoneType]:

            if(histoneType in avgDict2):
                avgDict2[residue] = avgDict[histoneType][residue][0] / avgDict[histoneType][residue][1]

            else:
                avgDict2[histoneType] = {residue : avgDict[histoneType][residue][0] / avgDict[histoneType][residue][1]}

    return avgDict2_____no_output_____def sum_contacts(interfaceDictionary):

    sumDict = {}
    histoneDict = {}

    for pair in interfaceDictionary:

        if(pair != 'uniprotPair'):

            targetFields = []
            sourceFields = []

            nucleosomeFlag = 0
            for residue in interfaceDictionary[pair]:
                pairs = interfaceDictionary[pair][residue][0].split('$')

                for instance in pairs:
                    targetFields = instance.split('@')[0].split('|')
                    sourceFields = instance.split('@')[1].split('|')

                    if(targetFields[7].split(':')[1] == '1' or sourceFields[7].split(':')[1] == '1'):
                        nucleosomeFlag = 1
                        break

                if(nucleosomeFlag):
                    break

            #randomResidue = list(interfaceDictionary[pair].keys())[0]
            #randomPair = interfaceDictionary[pair][randomResidue][0].split('$')[0]

            #targetFields = randomPair.split('@')[0].split('|')
            #sourceFields = randomPair.split('@')[1].split('|')

            pdbList = []

            for residue in interfaceDictionary[pair]:
                pdbIDs = interfaceDictionary[pair][residue][2].split('$')

                for pdb in pdbIDs:

                    if(pdb not in pdbList):
                        pdbList.append(pdb)

            if(targetFields[3] != 'other'):
                histoneType = targetFields[3]

                newPair = histoneType + '@'

                if(sourceFields[3] != 'other'):
                    histoneType2 = sourceFields[3]

                    newPair += histoneType2

                else:

                    for field in sourceFields:
                        if(field == sourceFields[len(sourceFields) - 1]):
                            newPair += field
                        else:
                            newPair += field + '|'

                totalCount = 0

                for residue in interfaceDictionary[pair]:
                    normalizedCount = interfaceDictionary[pair][residue][3]
                    totalCount += normalizedCount

                if(newPair in sumDict):
                    sumDict[newPair][0] += totalCount

                else:
                    sumDict[newPair] = [totalCount, pdbList]

                if(newPair in histoneDict):
                    histoneDict[newPair] += 1

                else:
                    histoneDict = {newPair : 1}

            elif(sourceFields[3] != 'other'):
                histoneType = sourceFields[3]

                newPair = histoneType + '@'

                for field in targetFields:
                    if(field == targetFields[len(targetFields) - 1]):
                        newPair += field
                    else:
                        newPair += field + '|'

                totalCount = 0

                for residue in interfaceDictionary[pair]:
                    normalizedCount = interfaceDictionary[pair][residue][3]
                    totalCount += normalizedCount

                if(newPair in sumDict):
                    sumDict[newPair][0] += totalCount

                else:
                    sumDict[newPair] = [totalCount, pdbList]

                if(newPair in histoneDict):
                    histoneDict[newPair] += 1

                else:
                    histoneDict = {newPair : 1}

            else:
                newPair = ''

                for field in targetFields:
                    if(field == targetFields[len(targetFields) - 1]):
                        newPair += field
                    else:
                        newPair += field + '|'

                newPair += '@'

                for field in sourceFields:
                    if(field == sourceFields[len(sourceFields) - 1]):
                        newPair += field
                    else:
                        newPair += field + '|'

                totalCount = 0

                for residue in interfaceDictionary[pair]:
                    normalizedCount = interfaceDictionary[pair][residue][3]
                    totalCount += normalizedCount

                if(newPair in sumDict):
                    sumDict[newPair][0] += totalCount

                else:
                    sumDict[newPair] = [totalCount, pdbList]

                if(newPair in histoneDict):
                    histoneDict[newPair] += 1

                else:
                    histoneDict = {newPair : 1}

            for pair in histoneDict:
                sumDict[pair][0] = sumDict[pair][0] / histoneDict[pair]
            #####Have to account for the case when an interface between two non-histone chains is already in the dictionary, but is stored in a reverse order!!!!

    return sumDict_____no_output_____def main():
    chainDictionary = {}
    chainDictionary['chain'] = {}
    get_chain_dictionaries(CHAIN_FILE, chainDictionary)
#     for pdb in chainDictionary:
#         for chain in chainDictionary[pdb]:
#             print(pdb + '\t' + chain + '\t' + str(chainDictionary[pdb][chain]))

    interfaceFiles = []
    get_files(PDB_LIST, interfaceFiles, 'interface')

    interfaceDictionary = {}
    interfaceDictionary['uniprotPair'] = {}
    residue_count(interfaceFiles, chainDictionary, interfaceDictionary)
#     for pair in interfaceDictionary:
#         for a in interfaceDictionary[pair]:
#             print(pair + '\t' + a + '\t' + str(interfaceDictionary[pair][a]))

    normalize_count(interfaceDictionary)
#     for pair in interfaceDictionary:
#         for residue in interfaceDictionary[pair]:
#             uniprots = pair.split('@')
#             chains = str(interfaceDictionary[pair][residue][0]).split('@')
#             name1 = chains[0].split('|')[2]
#             name2 = chains[-1].split('|')[2]
#             count = str(interfaceDictionary[pair][residue][3])
#             print(uniprots[0] + '@' + name1 + '@' + uniprots[1] + '@' + name2 + '@' + residue + '@' + count)

#     avgDict = average_histones(interfaceDictionary)
#     for entry in avgDict:
#         print(entry + '\t' + str(avgDict[entry]))

    sumDict = sum_contacts(interfaceDictionary)
    for pair in sumDict:
        chains = pair.split('@')
        target = chains[0]
        source = chains[1]

        pdbIDs = ''

        for pdb in sumDict[pair][1]:

            if(pdb == sumDict[pair][1][len(sumDict[pair][1]) - 1]):
                pdbIDs += pdb

            else:
                pdbIDs += pdb + '|'

            targetFields = target.split('|')
            sourceFields = source.split('|')
            contacts = sumDict[pair][0]
         ######
        if(len(targetFields) > 1 and targetFields[6].split(':')[1] == '1'):
            if(len(sourceFields) > 1):
                print(sourceFields[2] + ';' + sourceFields[0] + ';' + sourceFields[1] + ';' + sourceFields[4] + ';' + sourceFields[5] + ';' + sourceFields[6].split('nucleosome')[0] + ';' + targetFields[2] + ';' + targetFields[0] + ';' + targetFields[1] + ';' + targetFields[4] + ';' + targetFields[5] + ';' + targetFields[6].split('nucleosome')[0] + ';' + pdbIDs + ';' + 'nucleosome' + ';' + str(contacts))
            else:
                print(source + ';' + ';' + ';' + ';' + ';' + ';' + targetFields[2] + ';' + targetFields[0] + ';' + targetFields[1] + ';' + targetFields[4] + ';' + targetFields[5] + ';' + targetFields[6].split('nucleosome')[0] + ';' + pdbIDs + ';' + 'nucleosome' + ';' + str(contacts))
        elif(len(sourceFields) > 1 and sourceFields[6].split(':')[1] == '1'):
            if(len(targetFields) > 1):
                print(sourceFields[2] + ';' + sourceFields[0] + ';' + sourceFields[1] + ';' + sourceFields[4] + ';' + sourceFields[5] + ';' + sourceFields[6].split('nucleosome')[0] + ';' + targetFields[2] + ';' + targetFields[0] + ';' + targetFields[1] + ';' + targetFields[4] + ';' + targetFields[5] + ';' + targetFields[6].split('nucleosome')[0] + ';' + pdbIDs + ';' + 'nucleosome' + ';' + str(contacts))
            else:
                print(target + ';' + ';' + ';' + ';' + ';' + ';' + sourceFields[2] + ';' + sourceFields[0] + ';' + sourceFields[1] + ';' + sourceFields[4] + ';' + sourceFields[5] + ';' + sourceFields[6].split('nucleosome')[0] + ';' + pdbIDs + ';' + 'nucleosome' + ';' + str(contacts))
        elif(len(sourceFields) == 1 and len(targetFields) == 1):
            print(target + ';' + ';' + ';' + ';' + ';' + ';' + source + ';' + ';' + ';' + ';' + ';' + ';' + pdbIDs + ';' + 'NA' + ';' + str(contacts))
        else:
            if(len(targetFields) > 1 and len(sourceFields) > 1):
                print(sourceFields[2] + ';' + sourceFields[0] + ';' + sourceFields[1] + ';' + sourceFields[4] + ';' + sourceFields[5] + ';' + sourceFields[6].split('nucleosome')[0] + ';' + targetFields[2] + ';' + targetFields[0] + ';' + targetFields[1] + ';' + targetFields[4] + ';' + targetFields[5] + ';' + targetFields[6].split('nucleosome')[0] + ';' + pdbIDs + ';' + 'histone' + ';' + str(contacts))
            elif(len(targetFields) > 1):
                print(source + ';' + ';' + ';' + ';' + ';' + ';' + targetFields[2] + ';' + targetFields[0] + ';' + targetFields[1] + ';' + targetFields[4] + ';' + targetFields[5] + ';' + targetFields[6].split('nucleosome')[0] + ';' + pdbIDs + ';' + 'histone' + ';' + str(contacts))
            else:
                print(target + ';' + ';' + ';' + ';' + ';' + ';' + sourceFields[2] + ';' + sourceFields[0] + ';' + sourceFields[1] + ';' + sourceFields[4] + ';' + sourceFields[5] + ';' + sourceFields[6].split('nucleosome')[0] + ';' + pdbIDs + ';' + 'histone' + ';' + str(contacts))_____no_output_____if __name__ == "__main__":
    main()_____no_output_____
</code>


# Notebook from check-spelling/jdaviz
Path: notebooks/concepts/imviz_simple_aper_phot.ipynb

# Imviz simple aperture photometry

This is a proof-of-concept showing how to use Imviz to perform simple aperture photometry using hand-drawn aperture on a single object._____no_output_____

import warnings

from astropy.utils.data import download_file

from jdaviz import Imviz_____no_output_____with warnings.catch_warnings():
    warnings.simplefilter('ignore')  # Hide import warnings
    imviz = Imviz(verbosity='warning')  # Hide info messages_____no_output_____

We can load JWST images with ASDF and GWCS._____no_output_____

jwf277w = download_file('https://stsci.box.com/shared/static/iao1zxtigyrhq7k3wtu5nchrxzlhj9kv.fits', cache=True)
jwf444w = download_file('https://stsci.box.com/shared/static/rey83o5wq6g7qd7xym6r1jq9wlsxaqnt.fits', cache=True)
my_cen = (1002, 1154)
my_radius = 20
my_zoom = 4

imviz.load_data(jwf277w, data_label='JWST_F277W')
imviz.load_data(jwf444w, data_label='JWST_F444W')_____no_output_____

Or we can load HST/ACS images with FITS WCS._____no_output_____

Or we can load plain Numpy array without WCS or unit._____no_output_____

Display the app._____no_output_____

imviz.app_____no_output_____

Now, we would zoom in on a star and draw an aperture._____no_output_____

viewer = imviz.default_viewer
viewer.cuts = '95%'
viewer.center_on(my_cen)
viewer.zoom_level = my_zoom

# Click on image to finalize selection.
imviz._apply_interactive_region(
    'bqplot:circle', (my_cen[0] - my_radius, my_cen[1] - my_radius),
                     (my_cen[0] + my_radius, my_cen[1] + my_radius))_____no_output_____

Now, use the "Imviz Simple Aperture Photometry" plugin.

Once photometry is done, we would do the following to extract the data from Imviz back to notebook for further processing._____no_output_____

results = imviz.get_aperture_photometry_results()
results_____no_output_____


# Notebook from EvanZ/ppl-pm
Path: xpm.ipynb

# Bayesian Inference of NBA Player Ratings using the Pyro Probabilistic Programming Language_____no_output_____If you're reading this, you are probably already familiar with many of the various forms of player ratings that all get labeled some for of "+/-", including APM, RAPM, RPM, BPM, PIPM, etc. The basic idea has been described ad nauseum by this point, but briefly, we define a "stint" as a pair of home and away 5-player matchups that play consecutive possessions before a substitution is made. A stint can last anywhere from a single possession to a dozen or more. The outcome of a stint is typically the adjusted net rating (the difference in points normalized by 100 possessions). This is the very basic formulation, but depending on the priors and the target variable (for example, one can do rebound +/- just as easily), we get the many different rating systems. While I'm not going to re-invent the wheel here, I do want to show how we can formulate models in a very flexible and reasoned way using a probabilistic programming language, and specifically, [Uber's Pyro library](https://eng.uber.com/pyro/). I'm also going to show how we can use different priors to begin to explore wider possibilites for player ratings. Let's do it!_____no_output_____As always, we're going to first import a bunch of libraries and packages._____no_output_____

import pandas as pd
import numpy as np
import bz2
import json
import logging
import os

import torch
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from torch.distributions import constraints
from IPython import display
import pyro
import pyro.distributions as dist
import pyro.optim as optim
from pyro.infer.autoguide import AutoDiagonalNormal
from pyro.infer import SVI, Trace_ELBO

from sklearn.feature_extraction import DictVectorizer
from sklearn.model_selection import train_test_split

pyro.set_rng_seed(1)
assert pyro.__version__.startswith('1.5.0')
%matplotlib inline_____no_output_____

Next, we're going to load 3 seasons worth of play-by-play data that I collected for [nbawowy.com](http://nbawowy.com) during the 2016-18 seasons. _____no_output_____

stints = []
with bz2.open('data/matchups-2016-2018.json.bz2', 'r') as f:
    for line in f:
        stints.append(json.loads(line))
print(stints[0][stints[0]['home']])
print(stints[0][stints[0]['away']]){'on': ['George Hill', 'Monta Ellis', 'Paul George', 'Jordan Hill', 'Ian Mahinmi'], 'entered': [], 'exited': [], 'stats': {'poss': 6, 'time': 0, 'pts': 5, 'oreb': 1, 'dreb': 2, 'orebx': 3, 'drebx': 2, 'tov': 0, 'team_tov': 0, 'non_steal_tov': 0, 'fta': 1, 'ftm': 1, 'foul': 2, 'fgm': 2, 'fgx': 5, 'fg2m': 2, 'fg3m': 0}}
{'on': ['Mike Conley', 'Courtney Lee', 'Tony Allen', 'Zach Randolph', 'Marc Gasol'], 'entered': [], 'exited': [], 'stats': {'poss': 5, 'time': 0, 'pts': 9, 'oreb': 2, 'dreb': 3, 'orebx': 2, 'drebx': 1, 'tov': 0, 'team_tov': 0, 'non_steal_tov': 0, 'fta': 1, 'ftm': 1, 'foul': 3, 'fgm': 4, 'fgx': 4, 'fg2m': 4, 'fg3m': 0}}

You can see a stint is defined as a list of players for each team along with various stats aggregated over however many possessions were played before a substitution was made. Once we have the data loaded, the next step is to calculate the ratings and setup or feature matrix. So far this is exactly the same code that I have used previously to calculate [RAPM](https://github.com/EvanZ/nba-rapm/blob/master/rapm_splits_players.ipynb). _____no_output_____

matchups = []
ratings = []
weights = []

for stint in stints:
    home = stint['home']
    away = stint['away']
    poss = (stint[home]['stats']['poss'] + stint[away]['stats']['poss'])/2

    home_unit = {"{}".format(name): 1 for name in stint[home]['on']}
    away_unit = {"{}".format(name): -1 for name in stint[away]['on']}

    matchup = home_unit.copy()
    matchup.update(away_unit)

    if poss >= 1:  # to avoid some ill-conditioning we only use stints that have possessions >= 1
        delta = 100*(stint[home]['stats']['pts']-stint[away]['stats']['pts'])/poss
        matchups.append(matchup)
        ratings.append(delta)
        weights.append(poss)

dv = DictVectorizer(sparse=False)
features = dv.fit_transform(matchups)
players = dv.get_feature_names()
train_features, test_features, train_ratings, test_ratings, train_weights, test_weights = train_test_split(features, ratings, weights, test_size=0.15)
print(matchups[0],ratings[0],weights[0]){'George Hill': 1, 'Monta Ellis': 1, 'Paul George': 1, 'Jordan Hill': 1, 'Ian Mahinmi': 1, 'Mike Conley': -1, 'Courtney Lee': -1, 'Tony Allen': -1, 'Zach Randolph': -1, 'Marc Gasol': -1} -72.72727272727273 5.5

Before we move on to the fun stuff, let's look at the distribution of ratings. Keep in mind each stint can have a varying amount of possessions._____no_output_____

fig = sns.displot(data=np.array(ratings),kind='kde')
fig.set(title='Distribution of Net Ratings')_____no_output_____

Likewise, here is the distribution of stint length. Notice that most stints are actually fairly short with very few ever going more than 15 or so possessions. Consequently, the net ratings over each stint are exceedingly noisy._____no_output_____

fig = sns.displot(data=np.array(weights),kind='kde')
fig.set(title='Distribution of Possessions')_____no_output_____

Now that we have all the data, it's time to build the models and do some inference. First let's start with the "obvious" basic model. We assume that the ratings for each stint are distributed normally with a parameter $\mu$ and $\sigma$. Furthermore, we assume that there is a homecourt advantage $HCA$ which is positive and put a $Uniform(0,10)$ prior on it (based on my vast knowledge of NBA homecourt advantage!). For the ratings $\beta_i$, we will assign to each player a $Normal(0,1)$ prior. We'll use a $Uniform(0,200)$ prior for $sigma$. The deterministic parameter $\mu$ is then simply the sum of the $HCA$ latent variable and the player ratings $\beta$ dotted with the highly sparse feature matrix. This is all setup in the fairly simple code below._____no_output_____

# Basic model with normally distributed player ratings and stint outcomes.

def model(features, ratings):
    hca = pyro.sample("HCA", dist.Uniform(0,10))
    beta = pyro.sample("beta", dist.Normal(torch.zeros(features.shape[-1]),torch.ones(features.shape[-1])))
    sigma = pyro.sample("sigma", dist.Uniform(0,200))
    for i in pyro.plate("data", len(features), subsample_size=1000):
        x = torch.Tensor(features[i])
        mu = hca + torch.dot(x, beta)
        pyro.sample(f"result_{i}", dist.Normal(mu, sigma), obs=ratings[i])_____no_output_____

Now we'll do inference using the stochastic variational inference mode in Pyro. Note that I've split the dataset into training and test. We'll use the test set later to measure out-of-sample loss._____no_output_____

guide = AutoDiagonalNormal(model)
adam = pyro.optim.Adam({"lr": 0.03})
svi = SVI(model=model, guide=guide, optim=adam, loss=Trace_ELBO())

pyro.clear_param_store()
iters = 300
losses = []
for j in range(iters):
    # calculate the loss and take a gradient step
    loss = svi.step(train_features, train_ratings)
    losses.append(loss)
sns.set_theme(style='whitegrid')
sns.lineplot(data=losses)_____no_output_____

Note that the loss was minimized very rapidly in this case. That is not always true. Let's see how the top 25 rankings look according to this model. In the list below you'll see 3 columns, the player, his rating, and the variance. If you haven't seen a +/- model with variances, you're not alone. It's not typically given, although I have seen it worked out previously. I encourage you to read this [blog post from Justin Jacobs on the matter](https://squared2020.com/2018/12/24/regularized-adjusted-plus-minus-part-iii-what-had-really-happened-was/). _____no_output_____

guide.requires_grad_(False)

beta_mu = pyro.param('AutoDiagonalNormal.loc')[1:]
beta_sigma = pyro.param('AutoDiagonalNormal.scale')[1:]

xpm = sorted(list(zip(beta_mu.numpy(),beta_sigma.numpy(),beta_mu.numpy()-2*beta_sigma.numpy(), beta_mu.numpy()+2*beta_sigma.numpy(),players)),key=lambda x: -x[0])
ranked_players = []
for idx, obj in enumerate(xpm):
    ranked_players.append(obj[4])
    if idx+1 <= 25:
        print(f"{idx+1}.{obj[4]} {obj[0]:0.3} ({obj[1]:.2})")1.Stephen Curry 2.83 (0.75)
2.Draymond Green 2.3 (0.72)
3.Chris Paul 1.9 (0.7)
4.Klay Thompson 1.85 (1.2)
5.Kawhi Leonard 1.81 (0.87)
6.Russell Westbrook 1.77 (0.82)
7.Kyle Lowry 1.72 (0.72)
8.Kevin Durant 1.71 (0.86)
9.LeBron James 1.68 (1.1)
10.Eric Gordon 1.54 (0.69)
11.James Harden 1.51 (1.0)
12.Andre Iguodala 1.5 (0.86)
13.Kemba Walker 1.48 (0.92)
14.Nene 1.32 (0.91)
15.Joel Embiid 1.32 (1.1)
16.Manu Ginobili 1.31 (0.82)
17.Patrick Patterson 1.26 (0.88)
18.Rudy Gobert 1.23 (1.2)
19.Paul Millsap 1.21 (0.7)
20.Kyle Korver 1.16 (0.92)
21.J.J. Redick 1.14 (0.72)
22.Nikola Jokic 1.1 (1.4)
23.Serge Ibaka 1.09 (1.4)
24.Fred VanVleet 1.04 (0.89)
25.Davis Bertans 1.01 (0.61)

We can visualize the distributions via sampling._____no_output_____

samples = []
for i in range(82): # 82 games!
    sample = list(zip(players,guide.sample_latent()[1:].numpy()))
    for player, value in sample:
        samples.append((player, value))

df = pd.DataFrame.from_records(samples,columns=['player','value'])
fig, ax = plt.subplots(figsize=(10,10))
sns.violinplot(data=df.where(df.player.isin(ranked_players[:25])), x='value', y='player',palette="light:g", inner="points", orient="h",order=ranked_players[:25])_____no_output_____

We are also interested in the distributions for $HCA$ and $\sigma$ (the stint "noise"). Let's look at them together in a bivariate distribution plot (fancy!)._____no_output_____

from pyro.infer import Predictive

predictive = Predictive(model, guide=guide, num_samples=82, return_sites=("HCA","sigma"))
samples = predictive(train_features,torch.tensor(train_ratings))_____no_output_____sns.displot(data=pd.DataFrame.from_dict(samples),x='HCA',y='sigma',kind='kde')_____no_output_____

Well, that makes perfect sense. We know from experience that $HCA~3$ and $\sigma$ being around 115 should not be too surprising given the distribution of ratings we saw earlier. Before moving on to other models, the final piece of the puzzle is to examine the loss or error in our test data. We can then use this as a baseline for comparison. To do this we use the guide function to sample from the posterior distribution and calculate the RMSE._____no_output_____

losses = []
for i in range(len(test_features)):
    sample = guide.forward()
    mu = np.dot(test_features[i],sample['beta'])+sample['HCA']
    norm = pyro.distributions.Normal(mu, sample['sigma'])
    losses.append(np.power(norm.sample()-test_ratings[i],2))
print(f'Test Loss (RMSE) = {np.sqrt(np.mean(np.array(losses))):0.3f}')Test Loss (RMSE) = 160.355

Ok, now let's have some fun and test some of our assumptions in the model. We assumed a normal distribution for the sampling of stint ratings. The ratings distribution given earlier seems really "pointy" at 0, reminiscent of a Laplace distribution ("double exponential"). What if we try that? See the last `pyro.sample` statement in the following code._____no_output_____

def model(features, ratings):
    hca = pyro.sample("HCA", dist.Uniform(0,10))
    beta = pyro.sample("beta", dist.Normal(torch.zeros(features.shape[-1]),torch.ones(features.shape[-1])))
    sigma = pyro.sample("sigma", dist.Uniform(0,200))
    for i in pyro.plate("data", len(features), subsample_size=1000):
        x = torch.Tensor(features[i])
        mu = hca + torch.dot(x, beta)
        pyro.sample(f"result_{i}", dist.Laplace(mu, sigma), obs=ratings[i]) # <-----------Replaced Normal with Laplace

guide = AutoDiagonalNormal(model)
adam = pyro.optim.Adam({"lr": 0.03})
svi = SVI(model=model, guide=guide, optim=adam, loss=Trace_ELBO())

pyro.clear_param_store()
iters = 500
losses = []
for j in range(iters):
    # calculate the loss and take a gradient step
    loss = svi.step(train_features, train_ratings)
    losses.append(loss)
    if j % 50 == 0:
        pass
#         print("[iteration %04d] loss: %.4f" % (j, loss/1000))
sns.set_theme(style='whitegrid')
sns.lineplot(data=losses)_____no_output_____guide.requires_grad_(False)

beta_mu = pyro.param('AutoDiagonalNormal.loc')[1:]
beta_sigma = pyro.param('AutoDiagonalNormal.scale')[1:]

xpm = sorted(list(zip(beta_mu.numpy(),beta_sigma.numpy(),beta_mu.numpy()-2*beta_sigma.numpy(), beta_mu.numpy()+2*beta_sigma.numpy(),players)),key=lambda x: -x[0])
ranked_players = []
for idx, obj in enumerate(xpm):
    ranked_players.append(obj[4])
    if idx+1 <= 25:
        print(f"{idx+1}.{obj[4]} {obj[0]:0.3} ({obj[1]:.2})")1.Chris Paul 1.5 (0.96)
2.LeBron James 1.4 (0.49)
3.Stephen Curry 1.39 (0.67)
4.Kawhi Leonard 1.27 (0.52)
5.Kyle Lowry 1.2 (0.56)
6.Nikola Jokic 1.18 (0.57)
7.Draymond Green 1.12 (0.75)
8.Mike Conley 1.02 (0.44)
9.Rudy Gobert 0.996 (0.62)
10.Kevin Durant 0.954 (0.78)
11.James Harden 0.89 (0.41)
12.Russell Westbrook 0.881 (0.67)
13.Jayson Tatum 0.875 (0.77)
14.Victor Oladipo 0.866 (0.58)
15.Joel Embiid 0.841 (0.86)
16.Dejounte Murray 0.822 (0.92)
17.Nene 0.803 (0.66)
18.Klay Thompson 0.796 (0.63)
19.Kristaps Porzingis 0.794 (0.6)
20.Jrue Holiday 0.767 (0.5)
21.Kemba Walker 0.753 (0.54)
22.Serge Ibaka 0.736 (0.57)
23.Andrew Bogut 0.724 (0.72)
24.Channing Frye 0.7 (0.62)
25.Michael Kidd-Gilchrist 0.699 (0.86)
samples = []
for i in range(82): # 82 games!
    sample = list(zip(players,guide.sample_latent()[1:].numpy()))
    for player, value in sample:
        samples.append((player, value))

df = pd.DataFrame.from_records(samples,columns=['player','value'])
fig, ax = plt.subplots(figsize=(10,10))
sns.violinplot(data=df.where(df.player.isin(ranked_players[:25])), x='value', y='player',palette="light:g", inner="points", orient="h",order=ranked_players[:25])_____no_output_____

Looking at the player rankings this model gives, we certainly see some differences compared to the first model, but it's not crazy. Generally speaking, we still see the best players in the league. So just looking at the rankings isn't going to give us the answer. Let's look at the test loss again._____no_output_____

losses = []
for i in range(len(test_features)):
    sample = guide.forward()
    mu = np.dot(test_features[i],sample['beta'])+sample['HCA']
    laplace = pyro.distributions.Laplace(mu, sample['sigma']) # <-----------------sample from Laplace distribution
    losses.append(np.power(laplace.sample()-test_ratings[i],2))
print(f'Test Loss (RMSE) = {np.sqrt(np.mean(np.array(losses))):0.3f}')Test Loss (RMSE) = 169.383

Ok, well that's about 169 which is greater than the 161 we got from the original model. Let's ditch the Laplace distribution and try something a bit more "out there". So far we haven't utilized the stint lengths in our model. Just for kicks, let's try replacing the normal sampling with a Student-T distribution which uses the stint length as the degrees of freedom parameter and see what we get. _____no_output_____

def model(features, ratings, weights): # <---------------------Notice we added weights as a third argument
    hca = pyro.sample("HCA", dist.Uniform(0,10))
    beta = pyro.sample("beta", dist.Normal(torch.zeros(features.shape[-1]),torch.ones(features.shape[-1])))
    sigma = pyro.sample("sigma", dist.Uniform(0,200))
    for i in pyro.plate("data", len(features), subsample_size=1000):
        x = torch.Tensor(features[i])
        mu = hca + torch.dot(x, beta)
        pyro.sample(f"result_{i}", dist.StudentT(weights[i], mu, sigma), obs=ratings[i]) # <------------swap in Student-T distribution

guide = AutoDiagonalNormal(model)
adam = pyro.optim.Adam({"lr": 0.03})
svi = SVI(model=model, guide=guide, optim=adam, loss=Trace_ELBO())

pyro.clear_param_store()
iters = 500
losses = []
for j in range(iters):
    # calculate the loss and take a gradient step
    loss = svi.step(train_features, train_ratings, train_weights) # <-------------Notice we added train_weights here
    losses.append(loss)
    if j % 50 == 0:
        pass
#         print("[iteration %04d] loss: %.4f" % (j, loss/1000))
sns.set_theme(style='whitegrid')
sns.lineplot(data=losses)_____no_output_____guide.requires_grad_(False)

beta_mu = pyro.param('AutoDiagonalNormal.loc')[1:]
beta_sigma = pyro.param('AutoDiagonalNormal.scale')[1:]

xpm = sorted(list(zip(beta_mu.numpy(),beta_sigma.numpy(),beta_mu.numpy()-2*beta_sigma.numpy(), beta_mu.numpy()+2*beta_sigma.numpy(),players)),key=lambda x: -x[0])
ranked_players = []
for idx, obj in enumerate(xpm):
    ranked_players.append(obj[4])
    if idx+1 <= 25:
        print(f"{idx+1}.{obj[4]} {obj[0]:0.3} ({obj[1]:.2})")1.Chris Paul 3.07 (0.96)
2.Stephen Curry 2.99 (0.78)
3.LeBron James 2.5 (0.81)
4.Kevin Durant 2.47 (0.88)
5.Draymond Green 2.33 (0.65)
6.Kawhi Leonard 2.23 (1.2)
7.Nikola Jokic 2.08 (1.1)
8.Russell Westbrook 2.06 (0.69)
9.Klay Thompson 1.83 (1.3)
10.James Harden 1.83 (0.62)
11.Victor Oladipo 1.73 (0.93)
12.Rudy Gobert 1.52 (0.77)
13.Serge Ibaka 1.45 (0.8)
14.Andre Iguodala 1.43 (0.81)
15.Kyle Lowry 1.42 (0.98)
16.Gordon Hayward 1.36 (0.77)
17.Eric Gordon 1.3 (0.81)
18.J.J. Redick 1.3 (0.7)
19.Andre Roberson 1.28 (0.82)
20.Darren Collison 1.28 (0.75)
21.LaMarcus Aldridge 1.27 (0.86)
22.Kemba Walker 1.26 (0.59)
23.Kyle Korver 1.26 (0.88)
24.Mike Conley 1.25 (0.81)
25.Steven Adams 1.24 (1.1)
samples = []
for i in range(82): # 82 games!
    sample = list(zip(players,guide.sample_latent()[1:].numpy()))
    for player, value in sample:
        samples.append((player, value))

df = pd.DataFrame.from_records(samples,columns=['player','value'])
fig, ax = plt.subplots(figsize=(10,10))
sns.violinplot(data=df.where(df.player.isin(ranked_players[:25])), x='value', y='player',palette="light:g", inner="points", orient="h",order=ranked_players[:25])_____no_output_____

Ok once again, the rankings don't seem obviously broken. But what is the loss?_____no_output_____

losses = []
for i in range(len(test_features)):
    sample = guide.forward()
    mu = np.dot(test_features[i],sample['beta'])+sample['HCA']
    student_t = pyro.distributions.StudentT(test_weights[i], mu, sample['sigma']) # <-------sample from Student-T distribution and use stint length
    losses.append(np.power(student_t.sample()-test_ratings[i],2))
print(f'Test Loss (RMSE) = {np.sqrt(np.mean(np.array(losses))):0.3f}')Test Loss (RMSE) = 21883.504

Wow, really bad. The fat tails of a Student-T doesn't seem to be a good idea, although it would be hard to notice by looking at the rankings. Ok, let's look at one more model variation. This time we will change our assumption that $\beta$ is normally distributed and instead experiment with a really fat-tailed Cauchy prior. I've never seen a +/- model that makes this assumption, so it should be interesting._____no_output_____

def model(features, ratings):
    hca = pyro.sample("HCA", dist.Uniform(0,10))
    beta = pyro.sample("beta", dist.Cauchy(torch.zeros(features.shape[-1]),torch.ones(features.shape[-1]))) # <-----Cauchy prior
    sigma = pyro.sample("sigma", dist.Uniform(0,200))
    for i in pyro.plate("data", len(features), subsample_size=1000):
        x = torch.Tensor(features[i])
        mu = hca + torch.dot(x, beta)
        pyro.sample(f"result_{i}", dist.Normal(mu, sigma), obs=ratings[i])

guide = AutoDiagonalNormal(model)
adam = pyro.optim.Adam({"lr": 0.03})
svi = SVI(model=model, guide=guide, optim=adam, loss=Trace_ELBO())

pyro.clear_param_store()
iters = 300
losses = []
for j in range(iters):
    # calculate the loss and take a gradient step
    loss = svi.step(train_features, train_ratings)
    losses.append(loss)
    if j % 50 == 0:
        pass
#         print("[iteration %04d] loss: %.4f" % (j, loss/1000))
sns.set_theme(style='whitegrid')
sns.lineplot(data=losses)_____no_output_____guide.requires_grad_(False)

beta_mu = pyro.param('AutoDiagonalNormal.loc')[1:]
beta_sigma = pyro.param('AutoDiagonalNormal.scale')[1:]

xpm = sorted(list(zip(beta_mu.numpy(),beta_sigma.numpy(),beta_mu.numpy()-2*beta_sigma.numpy(), beta_mu.numpy()+2*beta_sigma.numpy(),players)),key=lambda x: -x[0])
ranked_players = []
for idx, obj in enumerate(xpm):
    ranked_players.append(obj[4])
    if idx+1 <= 25:
        print(f"{idx+1}.{obj[4]} {obj[0]:0.3} ({obj[1]:.2})")1.Chris Paul 4.06 (3.9)
2.Stephen Curry 3.93 (2.4)
3.LeBron James 3.33 (0.73)
4.Kyle Lowry 3.31 (2.0)
5.Draymond Green 3.29 (3.1)
6.Kawhi Leonard 3.23 (1.5)
7.Rudy Gobert 2.65 (1.4)
8.J.J. Redick 2.53 (1.6)
9.Kevin Durant 2.44 (3.3)
10.Klay Thompson 2.21 (1.7)
11.Eric Gordon 2.16 (1.3)
12.James Harden 2.01 (1.1)
13.Andre Iguodala 1.94 (1.6)
14.Russell Westbrook 1.94 (1.6)
15.Blake Griffin 1.72 (2.0)
16.Manu Ginobili 1.68 (1.5)
17.Cody Zeller 1.66 (1.5)
18.Nene 1.59 (1.2)
19.Nikola Jokic 1.52 (1.3)
20.Fred VanVleet 1.51 (2.3)
21.Kyle Korver 1.5 (2.2)
22.Delon Wright 1.47 (1.4)
23.Serge Ibaka 1.46 (1.3)
24.Boris Diaw 1.43 (0.99)
25.Mike Conley 1.4 (1.2)
samples = []
for i in range(82): # 82 games!
    sample = list(zip(players,guide.sample_latent()[1:].numpy()))
    for player, value in sample:
        samples.append((player, value))

df = pd.DataFrame.from_records(samples,columns=['player','value'])
fig, ax = plt.subplots(figsize=(10,10))
sns.violinplot(data=df.where(df.player.isin(ranked_players[:25])), x='value', y='player',palette="light:g", inner="points", orient="h",order=ranked_players[:25])_____no_output_____losses = []
for i in range(len(test_features)):
    sample = guide.forward()
    mu = np.dot(test_features[i],sample['beta'])+sample['HCA']
    norm = pyro.distributions.Normal(mu, sample['sigma'])
    losses.append(np.power(norm.sample()-test_ratings[i],2))
print(f'Test Loss (RMSE) = {np.sqrt(np.mean(np.array(losses))):0.3f}')Test Loss (RMSE) = 161.811

Well, hey, whaddya know. The loss is slightly higher than our original model, but not by much. It's actually a bit of a surprising result to me._____no_output_____## Conclusion_____no_output_____Well hopefully you got something out of all this. I think the take home message from my perspective is that a PPL such as Pyro really gives you the flexibility and convenience to try out all your crazy and not-so-crazy ideas. I've only scratched the surface here. Please let me know if you have any questions or comments. You can catch me on Twitter (@thecity2)._____no_output_____### If you enjoyed this post please leave a Github Star!_____no_output_____


# Notebook from marimigliorini/AceleraDev-DS
Path: data-science-1/main.ipynb

# Desafio 3

Neste desafio, iremos praticar nossos conhecimentos sobre distribuições de probabilidade. Para isso,
dividiremos este desafio em duas partes:

1. A primeira parte contará com 3 questões sobre um *data set* artificial com dados de uma amostra normal e
    uma binomial.
2. A segunda parte será sobre a análise da distribuição de uma variável do _data set_ [Pulsar Star](https://archive.ics.uci.edu/ml/datasets/HTRU2), contendo 2 questões.

> Obs.: Por favor, não modifique o nome das funções de resposta._____no_output_____## _Setup_ geral_____no_output_____

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as sct
import seaborn as sns
from statsmodels.distributions.empirical_distribution import ECDF_____no_output_____#%matplotlib inline

from IPython.core.pylabtools import figsize

figsize(12, 8)

sns.set()_____no_output_____

## Parte 1_____no_output_____### _Setup_ da parte 1_____no_output_____

np.random.seed(42)

dataframe = pd.DataFrame({"normal": sct.norm.rvs(20, 4, size=10000),
                     "binomial": sct.binom.rvs(100, 0.2, size=10000)})_____no_output_____

## Inicie sua análise a partir da parte 1 a partir daqui_____no_output_____

# Sua análise da parte 1 começa aqui.
dataframe.head()_____no_output_____

## Questão 1

Qual a diferença entre os quartis (Q1, Q2 e Q3) das variáveis `normal` e `binomial` de `dataframe`? Responda como uma tupla de três elementos arredondados para três casas decimais.

Em outra palavras, sejam `q1_norm`, `q2_norm` e `q3_norm` os quantis da variável `normal` e `q1_binom`, `q2_binom` e `q3_binom` os quantis da variável `binom`, qual a diferença `(q1_norm - q1 binom, q2_norm - q2_binom, q3_norm - q3_binom)`?_____no_output_____

def q1():
    # Retorne aqui o resultado da questão 1.
    q1_diff = dataframe['normal'].quantile(0.25) - dataframe['binomial'].quantile(0.25)
    q2_diff = dataframe['normal'].quantile(0.50) - dataframe['binomial'].quantile(0.50)
    q3_diff = dataframe['normal'].quantile(0.75) - dataframe['binomial'].quantile(0.75)
    return (q1_diff.round(3), q2_diff.round(3), q3_diff.round(3))_____no_output_____

Para refletir:

* Você esperava valores dessa magnitude?

* Você é capaz de explicar como distribuições aparentemente tão diferentes (discreta e contínua, por exemplo) conseguem dar esses valores?_____no_output_____## Questão 2

Considere o intervalo $[\bar{x} - s, \bar{x} + s]$, onde $\bar{x}$ é a média amostral e $s$ é o desvio padrão. Qual a probabilidade nesse intervalo, calculada pela função de distribuição acumulada empírica (CDF empírica) da variável `normal`? Responda como uma único escalar arredondado para três casas decimais._____no_output_____

def q2():
    # Retorne aqui o resultado da questão 2.
    ecdf = ECDF(dataframe['normal'])
    prob = ecdf(dataframe['normal'].mean()+dataframe['normal'].std()) - ecdf(dataframe['normal'].mean()-dataframe['normal'].std())
    return float(prob.round(3))_____no_output_____ecdf = ECDF(dataframe['normal'])
plt.plot(ecdf.x, ecdf.y)
plt.show()_____no_output_____

Para refletir:

* Esse valor se aproxima do esperado teórico?
* Experimente também para os intervalos $[\bar{x} - 2s, \bar{x} + 2s]$ e $[\bar{x} - 3s, \bar{x} + 3s]$._____no_output_____## Questão 3

Qual é a diferença entre as médias e as variâncias das variáveis `binomial` e `normal`? Responda como uma tupla de dois elementos arredondados para três casas decimais.

Em outras palavras, sejam `m_binom` e `v_binom` a média e a variância da variável `binomial`, e `m_norm` e `v_norm` a média e a variância da variável `normal`. Quais as diferenças `(m_binom - m_norm, v_binom - v_norm)`?_____no_output_____

def q3():
    # Retorne aqui o resultado da questão 3.
    return ((dataframe['binomial'].mean()-dataframe['normal'].mean()).round(3), (dataframe['binomial'].var()-dataframe['normal'].var()).round(3))_____no_output_____

Para refletir:

* Você esperava valore dessa magnitude?
* Qual o efeito de aumentar ou diminuir $n$ (atualmente 100) na distribuição da variável `binomial`?_____no_output_____## Parte 2_____no_output_____### _Setup_ da parte 2_____no_output_____

stars = pd.read_csv("pulsar_stars.csv")

stars.rename({old_name: new_name
              for (old_name, new_name)
              in zip(stars.columns,
                     ["mean_profile", "sd_profile", "kurt_profile", "skew_profile", "mean_curve", "sd_curve", "kurt_curve", "skew_curve", "target"])
             },
             axis=1, inplace=True)

stars.loc[:, "target"] = stars.target.astype(bool)_____no_output_____

## Inicie sua análise da parte 2 a partir daqui_____no_output_____

# Sua análise da parte 2 começa aqui.
stars.head()_____no_output_____

## Questão 4

Considerando a variável `mean_profile` de `stars`:

1. Filtre apenas os valores de `mean_profile` onde `target == 0` (ou seja, onde a estrela não é um pulsar).
2. Padronize a variável `mean_profile` filtrada anteriormente para ter média 0 e variância 1.

Chamaremos a variável resultante de `false_pulsar_mean_profile_standardized`.

Encontre os quantis teóricos para uma distribuição normal de média 0 e variância 1 para 0.80, 0.90 e 0.95 através da função `norm.ppf()` disponível em `scipy.stats`.

Quais as probabilidade associadas a esses quantis utilizando a CDF empírica da variável `false_pulsar_mean_profile_standardized`? Responda como uma tupla de três elementos arredondados para três casas decimais._____no_output_____

def q4():
    # Retorne aqui o resultado da questão 4.
    false_pulsar_mean_profile_standardized = (stars.query("target == 0")['mean_profile'] - stars.query("target == 0")['mean_profile'].mean())/stars.query("target == 0")['mean_profile'].std()
    ecdf = ECDF(false_pulsar_mean_profile_standardized)
    return tuple(ecdf(sct.norm.ppf([0.80, 0.90, 0.95])).round(3))_____no_output_____

Para refletir:

* Os valores encontrados fazem sentido?
* O que isso pode dizer sobre a distribuição da variável `false_pulsar_mean_profile_standardized`?_____no_output_____## Questão 5

Qual a diferença entre os quantis Q1, Q2 e Q3 de `false_pulsar_mean_profile_standardized` e os mesmos quantis teóricos de uma distribuição normal de média 0 e variância 1? Responda como uma tupla de três elementos arredondados para três casas decimais._____no_output_____

def q5():
    # Retorne aqui o resultado da questão 5.
    false_pulsar_mean_profile_standardized = (stars.query("target == 0")['mean_profile'] - stars.query("target == 0")['mean_profile'].mean())/stars.query("target == 0")['mean_profile'].std()
    q_diff = false_pulsar_mean_profile_standardized.quantile([0.25,0.5,0.75]) - sct.norm.ppf([0.25, 0.50, 0.75])
    return tuple(q_diff.round(3))_____no_output_____

Para refletir:

* Os valores encontrados fazem sentido?
* O que isso pode dizer sobre a distribuição da variável `false_pulsar_mean_profile_standardized`?
* Curiosidade: alguns testes de hipóteses sobre normalidade dos dados utilizam essa mesma abordagem._____no_output_____


# Notebook from dongxulee/lifeCycle
Path: 20210516/.ipynb_checkpoints/simulationV_gamma4-time-higherRent-higherBuycost-checkpoint.ipynb

%pylab inline
from jax.scipy.ndimage import map_coordinates
from constant import *
import warnings
from jax import jit, partial, random, vmap
from tqdm import tqdm
warnings.filterwarnings("ignore")
np.printoptions(precision=2)Populating the interactive namespace from numpy and matplotlib
nX = Xs.shape[0]
nA = As.shape[0]
Xs.shape, As.shape
Vgrid = np.load("Value.npy")_____no_output_____#Define the earning function, which applies for both employment status and 8 econ states
@partial(jit, static_argnums=(0,))
def y(t, x):
    '''
        x = [w,n,m,s,e,o]
        x = [0,1,2,3,4,5]
    '''
    if t <= T_R:
        return detEarning[t] * (1+gGDP[jnp.array(x[3], dtype = jnp.int8)]) * x[4] + (1-x[4]) * welfare
    else:
        return detEarning[-1]

#Earning after tax and fixed by transaction in and out from 401k account
@partial(jit, static_argnums=(0,))
def yAT(t,x):
    yt = y(t, x)
    if t <= T_R:
        # yi portion of the income will be put into the 401k if employed
        return (1-tau_L)*(yt * (1-yi))*x[4] + (1-x[4])*yt
    else:
        # t > T_R, n/discounting amount will be withdraw from the 401k
        return (1-tau_R)*yt + x[1]*Dn[t]

#Define the evolution of the amount in 401k account
@partial(jit, static_argnums=(0,))
def gn(t, x, r = r_bar):
    if t <= T_R:
        # if the person is employed, then yi portion of his income goes into 401k
        n_cur = x[1] + y(t, x) * yi * x[4]
    else:
        # t > T_R, n*Dn amount will be withdraw from the 401k
        n_cur = x[1] - x[1]*Dn[t]
        # the 401 grow with the rate r
    return (1+r)*n_cur

#Define the utility function
@jit
def u(c):
    return (jnp.power(c, 1-gamma) - 1)/(1 - gamma)

#Define the bequeath function, which is a function of bequeath wealth
@jit
def uB(tb):
    return B*u(tb)

#Reward function depends on the housing and non-housing consumption
@jit
def R(x,a):
    '''
    Input:
        x = [w,n,m,s,e,o]
        x = [0,1,2,3,4,5]
        a = [c,b,k,h,action]
        a = [0,1,2,3,4]
    '''
    c = a[:,0]
    h = a[:,3]
    C = jnp.power(c, alpha) * jnp.power(h, 1-alpha)
    return u(C)

@partial(jit, static_argnums=(0,))
def feasibleActions(t, x):
    # owner
    sell = As[:,2]
    budget1 = yAT(t,x) + x[0] - (1-sell)*(x[2]*(1+rh)-x[2]*Dm[t]) + sell*(H*pt - x[2] - c_s) + (1-sell)*(((t<=T_R)*tau_L + (t>T_R)*tau_R)*x[2]*rh)
    # last term is the tax deduction of the interest portion of mortgage payment
    h = jnp.ones(nA)*H*(1+kappa)*(1-sell) + sell*jnp.clip(budget1*As[:,0]*(1-alpha)/pr, a_max = 500)
    c = budget1*As[:,0]*(1-sell) + sell*(budget1*As[:,0] - h*pr)
    budget2 = budget1*(1-As[:,0])
    k = budget2*As[:,1]*(1-Kc)
    b = budget2*(1-As[:,1])
    owner_action = jnp.column_stack((c,b,k,h,sell))
    # renter
    buy = As[:,2]
    budget1 = yAT(t,x) + x[0] - buy*(H*pt*0.2 + c_h)
    h = jnp.clip(budget1*As[:,0]*(1-alpha)/pr, a_max = 500)*(1-buy) + buy*jnp.ones(nA)*H*(1+kappa)
    c = (budget1*As[:,0] - h*pr)*(1-buy) + buy*budget1*As[:,0]
    budget2 = budget1*(1-As[:,0])
    k = budget2*As[:,1]*(1-Kc)
    b = budget2*(1-As[:,1])
    renter_action = jnp.column_stack((c,b,k,h,buy))

    actions = x[5]*owner_action + (1-x[5])*renter_action
    return actions

@partial(jit, static_argnums=(0,))
def transition(t,a,x):
    '''
        Input:
            x = [w,n,m,s,e,o]
            x = [0,1,2,3,4,5]
            a = [c,b,k,h,action]
            a = [0,1,2,3,4]
        Output:
            w_next
            n_next
            m_next
            s_next
            e_next
            o_next

            prob_next
    '''
    nA = a.shape[0]
    s = jnp.array(x[3], dtype = jnp.int8)
    e = jnp.array(x[4], dtype = jnp.int8)
    # actions taken
    b = a[:,1]
    k = a[:,2]
    action = a[:,4]
    w_next = ((1+r_b[s])*b + jnp.outer(k,(1+r_k)).T).T.flatten().repeat(nE)
    n_next = gn(t, x)*jnp.ones(w_next.size)
    s_next = jnp.tile(jnp.arange(nS),nA).repeat(nE)
    e_next = jnp.column_stack((e.repeat(nA*nS),(1-e).repeat(nA*nS))).flatten()
    # job status changing probability and econ state transition probability
    pe = Pe[s, e]
    ps = jnp.tile(Ps[s], nA)
    prob_next = jnp.column_stack(((1-pe)*ps,pe*ps)).flatten()

    # owner
    m_next_own = ((1-action)*x[2]*Dm[t]).repeat(nS*nE)
    o_next_own = (x[5] - action).repeat(nS*nE)
    # renter
    m_next_rent = (action*H*pt*0.8*Dm[t]).repeat(nS*nE)
    o_next_rent = action.repeat(nS*nE)

    m_next = x[5] * m_next_own + (1-x[5]) * m_next_rent
    o_next = x[5] * o_next_own + (1-x[5]) * o_next_rent
    return jnp.column_stack((w_next,n_next,m_next,s_next,e_next,o_next,prob_next))

# used to calculate dot product
@jit
def dotProduct(p_next, uBTB):
    return (p_next*uBTB).reshape((p_next.shape[0]//(nS*nE), (nS*nE))).sum(axis = 1)

# define approximation of fit
@jit
def fit(v, xp):
    return map_coordinates(v,jnp.vstack((xp[:,0]/scaleW,
                                                      xp[:,1]/scaleN,
                                                      xp[:,2]/scaleM,
                                                      xp[:,3],
                                                      xp[:,4],
                                                      xp[:,5])),
                                                     order = 1, mode = 'nearest')

@partial(jit, static_argnums=(0,))
def V(t,V_next,x):
    '''
    x = [w,n,m,s,e,o]
    x = [0,1,2,3,4,5]
    xp:
        w_next    0
        n_next    1
        m_next    2
        s_next    3
        e_next    4
        o_next    5
        prob_next 6
    '''
    actions = feasibleActions(t,x)
    xp = transition(t,actions,x)
    # bequeath utility
    TB = xp[:,0]+x[1]*(1+r_bar)+xp[:,5]*(H*pt-x[2]*(1+rh)-c_s)
    bequeathU = uB(TB)
    if t == T_max-1:
        Q = R(x,actions) + beta * dotProduct(xp[:,6], bequeathU)
    else:
        Q = R(x,actions) + beta * dotProduct(xp[:,6], Pa[t]*fit(V_next, xp) + (1-Pa[t])*bequeathU)
    Q = jnp.nan_to_num(Q, nan = -100)
    v = Q.max()
    cbkha = actions[Q.argmax()]
    return v, cbkha.reshape((1,-1))_____no_output_____

Probability of transition between employment and unemployment states is know: $P(0, 1 | S)$, $P(1, 0 | S)$, $P(1, 1 | S)$, $P(0, 0 | S)$

The unconditional probability is:
$$P(0,1) = \Sigma_{s = 1}^{s = 8}P(0, 1 | S = s)P(S=s)$$_____no_output_____

S_distribution = jnp.ones(nS)/nS
for i in range(100):
    S_distribution = jnp.matmul(S_distribution, Ps)
S_distribution_____no_output_____#P(0,1)
P01 = jnp.dot(Pe[:,0],S_distribution)
#P(1,0)
P10 = jnp.dot(Pe[:,1],S_distribution)
jnp.array([[1-P01, P01],[P10, 1-P10]])_____no_output_____E_distribution = jnp.ones(2)/2
for i in range(100):
    E_distribution = jnp.matmul(E_distribution, jnp.array([[1-P01, P01],[P10, 1-P10]]))
E_distribution_____no_output_____num = 10000
'''
    x = [w,n,m,s,e,o]
    x = [5,0,0,0,0,0]
'''
from jax import random

def simulation(key):
    initE = random.choice(a = nE, p=E_distribution, key = key)
    initS = random.choice(a = nS, p=S_distribution, key = key)
    x = [5, 0, 0, initS, initE, 0]
    path = []
    move = []
    for t in range(T_min, T_max-1):
        _, key = random.split(key)
        _,a = V(t,Vgrid[:,:,:,:,:,:,t+1],x)
        xp = transition(t,a,x)
        p = xp[:,-1]
        x_next = xp[:,:-1]
        x = x_next[random.choice(a = nS*nE, p=p, key = key)]
        path.append(x)
        move.append(a[0])
    return jnp.array(path), jnp.array(move)_____no_output_____%%time
# simulation part
keys = vmap(random.PRNGKey)(jnp.arange(num))
Paths, Moves = vmap(simulation)(keys)CPU times: user 4h 27min 47s, sys: 26min 44s, total: 4h 54min 31s
Wall time: 13min 39s
# x = [w,n,m,s,e,o]
# x = [0,1,2,3,4,5]
ws = Paths[:,:,0].T
ns = Paths[:,:,1].T
ms = Paths[:,:,2].T
ss = Paths[:,:,3].T
es = Paths[:,:,4].T
os = Paths[:,:,5].T
cs = Moves[:,:,0].T
bs = Moves[:,:,1].T
ks = Moves[:,:,2].T
hs = Moves[:,:,3].T_____no_output_____plt.figure(figsize = [16,8])
plt.title("The mean values of simulation")
plt.plot(range(21, T_max-1 + 20),jnp.mean(ws + H*pt*os - ms,axis = 1)[:-1], label = "wealth + home equity")
plt.plot(range(21, T_max-1 + 20),jnp.mean(ws,axis = 1)[:-1], label = "wealth")
plt.plot(range(21, T_max-1 + 20),jnp.mean(cs,axis = 1)[:-1], label = "consumption")
plt.plot(range(21, T_max-1 + 20),jnp.mean(bs,axis = 1)[:-1], label = "bond")
plt.plot(range(21, T_max-1 + 20),jnp.mean(ks,axis = 1)[:-1], label = "stock")
# plt.plot((hs*pr).mean(axis = 1)[:-1], label = "housing")
plt.legend()_____no_output_____plt.title("housing consumption")
plt.plot(range(21, T_max-1 + 20),(hs).mean(axis = 1)[:-1], label = "housing")_____no_output_____plt.title("house owner percentage in the population")
plt.plot(range(21, T_max-1 + 20),(os).mean(axis = 1)[:-1], label = "owning")_____no_output_____plt.title("401k")
plt.plot(range(21, T_max-1 + 20),(ns).mean(axis = 1)[:-1], label = "housing")_____no_output_____# agent number, x = [w,n,m,s,e,o]
agentNum = 134
plt.figure(figsize = [16,8])
plt.plot(range(21, T_max-1 + 20),ws[:,agentNum][:-1], label = "wealth")
plt.plot(range(21, T_max-1 + 20),ns[:,agentNum][:-1], label = "401k")
plt.plot(range(21, T_max-1 + 20),ms[:,agentNum][:-1], label = "mortgage")
plt.plot(range(21, T_max-1 + 20),cs[:,agentNum][:-1], label = "consumption")
plt.plot(range(21, T_max-1 + 20),bs[:,agentNum][:-1], label = "bond")
plt.plot(range(21, T_max-1 + 20),ks[:,agentNum][:-1], label = "stock")
plt.legend()_____no_output_____


# Notebook from single-cell-genetics/singlecell_neuroseq_paper
Path: differentiation_prediction_model/predict_neuronal_differentiation_success.ipynb

# README

This script calculates model scores for the neuronal differentiation classification model described in Jerber et al, 2020.

Note that this model was trained on Z-score normalised iPSC RNA-seq data from one project (the HipSci project). This has the following implications:

- The model cannot be applied at all to a single sample (for which the Z-score is undefined), and will not produce  stable predictions for small numbers of samples.
- While the Z-score normalization was intended to improve portability of the model across iPSC RNA-seq samples from other projects, this is untested. In particular, the model score threshold applied in the paper to generate high-precision predictions may not be well calibrated in another dataset. Therefore, a more appropriate application of the model score in that context would be to rank and prioritize cell lines that are the most likely to differentiate well.
_____no_output_____

<code>
import pandas as pd
import numpy as np
import os_____no_output_____# specify input files
t_file = './example_expression_data.tsv'
coefficient_file = './model_coefficients.tsv'
out_file = './example_output_file.tsv'_____no_output_____# load bulk RNA-seq data

t_df = pd.read_csv(t_file, sep='\t', index_col=0)

# load model coefficients

coef_df = pd.read_csv(coefficient_file, sep='\t', index_col=0)_____no_output_____# read gene expression table and check for missing data

feature_list = t_df.index.intersection(coef_df.index)

model_feature_list = [x for x in coef_df.index if x!='intercept']

# check for model features that are not in the input transcript data
missing_features = [x for x in model_feature_list if x not in feature_list]

if len(missing_features)>0:
    print('Warning: {} genes missing from the input gene expression table:'.format(len(missing_features)))
    for gene in missing_features:
        print(gene)
_____no_output_____# get model coefficients in appropriate form for NumPy

coefficients = coef_df.loc[feature_list, 'coefficient'].values.reshape((1, -1))

intercept = coef_df.loc['intercept','coefficient']_____no_output_____# transform input expression data into z-score normalized NumPy array

t_df = t_df.loc[feature_list, :]

# apply z-score transformation to the data
t_df = t_df.apply(lambda x: (x-x.mean())/x.std(), axis=1)

X = t_df.transpose().values_____no_output_____# compute model scores for logistic regression model

linear_scores = list((np.matmul(coefficients, X.transpose()) + intercept)[0])
prob_scores = [1/(1+np.exp(-x)) for x in linear_scores]

out_df = pd.DataFrame({'cell_line_id':t_df.columns,
                       'model_score':prob_scores})

out_df_____no_output_____# output to file

out_df.to_csv(out_file, sep='\t', index=False)_____no_output_____
</code>