# Notebook from creageng2016/document_classification_dl
Path: NLPClassifier/NLP_Ngram_Multichannel_CNN.ipynb

# N-gram Multichannel CNN and CNN + LSTM

This exercise develop a multichannel convolutional neural network using Keras to predict movie sentiment. The exercise showed how to:

    1. Develop a multichannel CNN for text classification in Keras.
    2. Develop a multichannel CNN and LSTM for text classification in Keras

### How to modify this code for other dataset

To classify multiple classes of text data, change dense layer's loss function from *binary_crossentropy* to *categorical_crossentropy*

To output probability of each class, instead of 1 or 0, change dense layer's activation function from *sigmoid* to *softmax*

### Reference

https://machinelearningmastery.com/develop-n-gram-multichannel-convolutional-neural-network-sentiment-analysis/

https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/

https://www.researchgate.net/publication/321259272_Multi-channel_LSTM-CNN_model_for_Vietnamese_sentiment_analysis

https://keras.io/layers/merge/
https://keras.io/getting-started/functional-api-guide/

http://ruder.io/word-embeddings-2017/

_____no_output_____## Movie Review Dataset

The dataset is comprised of 1,000 positive and 1,000 negative movie reviews drawn from an archive of the rec.arts.movies.reviews newsgroup hosted at imdb.com. The authors refer to this dataset as the “polarity dataset.”

The data has been cleaned up somewhat; for example:

The dataset is comprised of only English reviews.
All text has been converted to lowercase.
There is white space around punctuation like periods, commas, and brackets.
Text has been split into one sentence per line.

You can download the dataset from here:

Movie Review Polarity Dataset (review_polarity.tar.gz, 3MB)
https://www.cs.cornell.edu/people/pabo/movie-review-data/review_polarity.tar.gz_____no_output_____## Data Preparation
In this section, we will look at 3 things:

    Separation of data into training and test sets.
    Loading and cleaning the data to remove punctuation and numbers.
    Prepare all reviews and save to file.
_____no_output_____

<code>
from nltk.corpus import stopwords
import string
from string import punctuation
from os import listdir
from pickle import dump
_____no_output_____data_dir = 'data/review_polarity/'
output_dir = 'output/multichannel_cnn/'_____no_output_____# load doc into memory
def load_doc(filename):
    file = open(filename, 'r')
    text = file.read()
    file.close()
    return text_____no_output_____#convert text to token
def clean_doc(doc):
    #split into tokens by white space
    tokens = doc.split()
    # remove puncturation from each token
    table = str.maketrans('', '', string.punctuation)
    tokens = [w.translate(table) for w in tokens]

    #remove remaining tokens that are not alphabetic
    tokens = [word for word in tokens if word.isalpha()]

    # filter out stop words
    stop_words = set(stopwords.words('english'))
    tokens = [w for w in tokens if not w in stop_words]

    # filter out short tokens
    tokens = [word for word in tokens if len(word) > 1]

    #note: the following does not work
    #token = [w for w in tokens if ((not w in stop_words) and (len(w) >1))]

    tokens = ' '.join(tokens)
    return tokens
     _____no_output_____filename = 'data/review_polarity/txt_sentoken/pos/cv000_29590.txt'
text = load_doc(filename)
print(text)films adapted from comic books have had plenty of success , whether they're about superheroes ( batman , superman , spawn ) , or geared toward kids ( casper ) or the arthouse crowd ( ghost world ) , but there's never really been a comic book like from hell before .
for starters , it was created by alan moore ( and eddie campbell ) , who brought the medium to a whole new level in the mid '80s with a 12-part series called the watchmen .
to say moore and campbell thoroughly researched the subject of jack the ripper would be like saying michael jackson is starting to look a little odd .
the book ( or " graphic novel , " if you will ) is over 500 pages long and includes nearly 30 more that consist of nothing but footnotes .
in other words , don't dismiss this film because of its source .
if you can get past the whole comic book thing , you might find another stumbling block in from hell's directors , albert and allen hughes .
getting the hughes brothers to direct this seems almost as ludicrous as casting carrot top in , well , anything , but riddle me this : who better to direct a film that's set in the ghetto and features really violent street crime than the mad geniuses behind menace ii society ?
the ghetto in question is , of course , whitechapel in 1888 london's east end .
it's a filthy , sooty place where the whores ( called " unfortunates " ) are starting to get a little nervous about this mysterious psychopath who has been carving through their profession with surgical precision .
when the first stiff turns up , copper peter godley ( robbie coltrane , the world is not enough ) calls in inspector frederick abberline ( johnny depp , blow ) to crack the case .
abberline , a widower , has prophetic dreams he unsuccessfully tries to quell with copious amounts of absinthe and opium .
upon arriving in whitechapel , he befriends an unfortunate named mary kelly ( heather graham , say it isn't so ) and proceeds to investigate the horribly gruesome crimes that even the police surgeon can't stomach .
i don't think anyone needs to be briefed on jack the ripper , so i won't go into the particulars here , other than to say moore and campbell have a unique and interesting theory about both the identity of the killer and the reasons he chooses to slay .
in the comic , they don't bother cloaking the identity of the ripper , but screenwriters terry hayes ( vertical limit ) and rafael yglesias ( les mis ? rables ) do a good job of keeping him hidden from viewers until the very end .
it's funny to watch the locals blindly point the finger of blame at jews and indians because , after all , an englishman could never be capable of committing such ghastly acts .
and from hell's ending had me whistling the stonecutters song from the simpsons for days ( " who holds back the electric car/who made steve guttenberg a star ? " ) .
don't worry - it'll all make sense when you see it .
now onto from hell's appearance : it's certainly dark and bleak enough , and it's surprising to see how much more it looks like a tim burton film than planet of the apes did ( at times , it seems like sleepy hollow 2 ) .
the print i saw wasn't completely finished ( both color and music had not been finalized , so no comments about marilyn manson ) , but cinematographer peter deming ( don't say a word ) ably captures the dreariness of victorian-era london and helped make the flashy killing scenes remind me of the crazy flashbacks in twin peaks , even though the violence in the film pales in comparison to that in the black-and-white comic .
oscar winner martin childs' ( shakespeare in love ) production design turns the original prague surroundings into one creepy place .
even the acting in from hell is solid , with the dreamy depp turning in a typically strong performance and deftly handling a british accent .
ians holm ( joe gould's secret ) and richardson ( 102 dalmatians ) log in great supporting roles , but the big surprise here is graham .
i cringed the first time she opened her mouth , imagining her attempt at an irish accent , but it actually wasn't half bad .
the film , however , is all good .
2 : 00 - r for strong violence/gore , sexuality , language and drug content

tokens = clean_doc(text)
print(tokens)films adapted comic books plenty success whether theyre superheroes batman superman spawn geared toward kids casper arthouse crowd ghost world theres never really comic book like hell starters created alan moore eddie campbell brought medium whole new level mid series called watchmen say moore campbell thoroughly researched subject jack ripper would like saying michael jackson starting look little odd book graphic novel pages long includes nearly consist nothing footnotes words dont dismiss film source get past whole comic book thing might find another stumbling block hells directors albert allen hughes getting hughes brothers direct seems almost ludicrous casting carrot top well anything riddle better direct film thats set ghetto features really violent street crime mad geniuses behind menace ii society ghetto question course whitechapel londons east end filthy sooty place whores called unfortunates starting get little nervous mysterious psychopath carving profession surgical precision first stiff turns copper peter godley robbie coltrane world enough calls inspector frederick abberline johnny depp blow crack case abberline widower prophetic dreams unsuccessfully tries quell copious amounts absinthe opium upon arriving whitechapel befriends unfortunate named mary kelly heather graham say isnt proceeds investigate horribly gruesome crimes even police surgeon cant stomach dont think anyone needs briefed jack ripper wont go particulars say moore campbell unique interesting theory identity killer reasons chooses slay comic dont bother cloaking identity ripper screenwriters terry hayes vertical limit rafael yglesias les mis rables good job keeping hidden viewers end funny watch locals blindly point finger blame jews indians englishman could never capable committing ghastly acts hells ending whistling stonecutters song simpsons days holds back electric carwho made steve guttenberg star dont worry itll make sense see onto hells appearance certainly dark bleak enough surprising see much looks like tim burton film planet apes times seems like sleepy hollow print saw wasnt completely finished color music finalized comments marilyn manson cinematographer peter deming dont say word ably captures dreariness victorianera london helped make flashy killing scenes remind crazy flashbacks twin peaks even though violence film pales comparison blackandwhite comic oscar winner martin childs shakespeare love production design turns original prague surroundings one creepy place even acting hell solid dreamy depp turning typically strong performance deftly handling british accent ians holm joe goulds secret richardson dalmatians log great supporting roles big surprise graham cringed first time opened mouth imagining attempt irish accent actually wasnt half bad film however good strong violencegore sexuality language drug content
# load all docs in a directory
def process_docs(directory, is_trian):
	documents = list()
	# walk through all files in the folder
	for filename in listdir(directory):
		# skip any reviews in the test set
		if is_trian and filename.startswith('cv9'):
			continue
		if not is_trian and not filename.startswith('cv9'):
			continue
		# create the full path of the file to open
		path = directory + '/' + filename
		# load the doc
		doc = load_doc(path)
		# clean doc
		tokens = clean_doc(doc)
		# add to list
		documents.append(tokens)
	return documents

# save a dataset to file
def save_dataset(dataset, filename):
	dump(dataset, open(filename, 'wb'))
	print('Saved: %s' % filename)

# load all training reviews

negative_docs = process_docs(data_dir+ 'txt_sentoken/neg', True)
positive_docs = process_docs(data_dir+ 'txt_sentoken/pos', True)
trainX = negative_docs + positive_docs
trainy = [0 for _ in range(900)] + [1 for _ in range(900)]
save_dataset([trainX,trainy], data_dir+'train.pkl')

# load all test reviews
negative_docs = process_docs(data_dir+'txt_sentoken/neg', False)
positive_docs = process_docs(data_dir+'txt_sentoken/pos', False)
testX = negative_docs + positive_docs
testY = [0 for _ in range(100)] + [1 for _ in range(100)]
save_dataset([testX,testY], data_dir+'test.pkl')Saved: data/review_polarity/train.pkl
Saved: data/review_polarity/test.pkl
from pickle import load
from numpy import array
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.utils.vis_utils import plot_model
from keras.models import Model
from keras.layers import Input
from keras.layers import Dense
from keras.layers import Flatten
from keras.layers import Dropout
from keras.layers import Embedding
from keras.layers.convolutional import Conv1D
from keras.layers.convolutional import MaxPooling1D
from keras.layers.merge import concatenate

# load a clean dataset
def load_dataset(filename):
	return load(open(filename, 'rb'))

# fit a tokenizer
def create_tokenizer(lines):
	tokenizer = Tokenizer()
	tokenizer.fit_on_texts(lines)
	return tokenizer

# calculate the maximum document length
def max_length(lines):
	return max([len(s.split()) for s in lines])

# encode a list of lines
def encode_text(tokenizer, lines, length):
	# integer encode
	encoded = tokenizer.texts_to_sequences(lines)
	# pad encoded sequences
	padded = pad_sequences(encoded, maxlen=length, padding='post')
	return padded

# define the model
def define_model(length, vocab_size):
	# channel 1
	inputs1 = Input(shape=(length,))
	embedding1 = Embedding(vocab_size, 100)(inputs1)
	conv1 = Conv1D(filters=32, kernel_size=4, activation='relu')(embedding1)
	drop1 = Dropout(0.5)(conv1)
	pool1 = MaxPooling1D(pool_size=2)(drop1)
	flat1 = Flatten()(pool1)
	# channel 2
	inputs2 = Input(shape=(length,))
	embedding2 = Embedding(vocab_size, 100)(inputs2)
	conv2 = Conv1D(filters=32, kernel_size=6, activation='relu')(embedding2)
	drop2 = Dropout(0.5)(conv2)
	pool2 = MaxPooling1D(pool_size=2)(drop2)
	flat2 = Flatten()(pool2)
	# channel 3
	inputs3 = Input(shape=(length,))
	embedding3 = Embedding(vocab_size, 100)(inputs3)
	conv3 = Conv1D(filters=32, kernel_size=8, activation='relu')(embedding3)
	drop3 = Dropout(0.5)(conv3)
	pool3 = MaxPooling1D(pool_size=2)(drop3)
	flat3 = Flatten()(pool3)
	# merge
	merged = concatenate([flat1, flat2, flat3])
	# interpretation
	dense1 = Dense(10, activation='relu')(merged)
	outputs = Dense(1, activation='sigmoid')(dense1)
	model = Model(inputs=[inputs1, inputs2, inputs3], outputs=outputs)
	# compile
	model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
	# summarize
	print(model.summary())
	plot_model(model, show_shapes=True, to_file='output/multichannel_cnn/cnn.png')
	return model

# load training dataset
trainLines, trainLabels = load_dataset(data_dir+'train.pkl')
# create tokenizer
tokenizer = create_tokenizer(trainLines)
# calculate max document length
length = max_length(trainLines)
# calculate vocabulary size
vocab_size = len(tokenizer.word_index) + 1
print('Max document length: %d' % length)
print('Vocabulary size: %d' % vocab_size)
# encode data
trainX = encode_text(tokenizer, trainLines, length)
print(trainX.shape)

# define model
model = define_model(length, vocab_size)
# fit model
model.fit([trainX,trainX,trainX], array(trainLabels), epochs=10, batch_size=16)
# save the model
model.save(output_dir+'model.h5')Max document length: 1380
Vocabulary size: 44277
(1800, 1380)
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_8 (InputLayer)            (None, 1380)         0
__________________________________________________________________________________________________
input_9 (InputLayer)            (None, 1380)         0
__________________________________________________________________________________________________
input_10 (InputLayer)           (None, 1380)         0
__________________________________________________________________________________________________
embedding_8 (Embedding)         (None, 1380, 100)    4427700     input_8[0][0]
__________________________________________________________________________________________________
embedding_9 (Embedding)         (None, 1380, 100)    4427700     input_9[0][0]
__________________________________________________________________________________________________
embedding_10 (Embedding)        (None, 1380, 100)    4427700     input_10[0][0]
__________________________________________________________________________________________________
conv1d_8 (Conv1D)               (None, 1377, 32)     12832       embedding_8[0][0]
__________________________________________________________________________________________________
conv1d_9 (Conv1D)               (None, 1375, 32)     19232       embedding_9[0][0]
__________________________________________________________________________________________________
conv1d_10 (Conv1D)              (None, 1373, 32)     25632       embedding_10[0][0]
__________________________________________________________________________________________________
dropout_8 (Dropout)             (None, 1377, 32)     0           conv1d_8[0][0]
__________________________________________________________________________________________________
dropout_9 (Dropout)             (None, 1375, 32)     0           conv1d_9[0][0]
__________________________________________________________________________________________________
dropout_10 (Dropout)            (None, 1373, 32)     0           conv1d_10[0][0]
__________________________________________________________________________________________________
max_pooling1d_8 (MaxPooling1D)  (None, 688, 32)      0           dropout_8[0][0]
__________________________________________________________________________________________________
max_pooling1d_9 (MaxPooling1D)  (None, 687, 32)      0           dropout_9[0][0]
__________________________________________________________________________________________________
max_pooling1d_10 (MaxPooling1D) (None, 686, 32)      0           dropout_10[0][0]
__________________________________________________________________________________________________
flatten_7 (Flatten)             (None, 22016)        0           max_pooling1d_8[0][0]
__________________________________________________________________________________________________
flatten_8 (Flatten)             (None, 21984)        0           max_pooling1d_9[0][0]
__________________________________________________________________________________________________
flatten_9 (Flatten)             (None, 21952)        0           max_pooling1d_10[0][0]
__________________________________________________________________________________________________
concatenate_2 (Concatenate)     (None, 65952)        0           flatten_7[0][0]
                                                                 flatten_8[0][0]
                                                                 flatten_9[0][0]
__________________________________________________________________________________________________
dense_3 (Dense)                 (None, 10)           659530      concatenate_2[0][0]
__________________________________________________________________________________________________
dense_4 (Dense)                 (None, 1)            11          dense_3[0][0]
==================================================================================================
Total params: 14,000,337
Trainable params: 14,000,337
Non-trainable params: 0
__________________________________________________________________________________________________
None
Epoch 1/10
1800/1800 [==============================] - 52s 29ms/step - loss: 0.6977 - acc: 0.5406
Epoch 2/10
1800/1800 [==============================] - 51s 29ms/step - loss: 0.3116 - acc: 0.8933
Epoch 3/10
1800/1800 [==============================] - 50s 28ms/step - loss: 0.0088 - acc: 0.9994
Epoch 4/10
1800/1800 [==============================] - 50s 28ms/step - loss: 0.0021 - acc: 1.0000
Epoch 5/10
1800/1800 [==============================] - 48s 27ms/step - loss: 0.0012 - acc: 1.0000
Epoch 6/10
1800/1800 [==============================] - 49s 27ms/step - loss: 9.2009e-04 - acc: 1.0000
Epoch 7/10
1800/1800 [==============================] - 48s 27ms/step - loss: 6.6605e-04 - acc: 1.0000
Epoch 8/10
1800/1800 [==============================] - 51s 28ms/step - loss: 4.6665e-04 - acc: 1.0000
Epoch 9/10
1800/1800 [==============================] - 49s 27ms/step - loss: 4.4235e-04 - acc: 1.0000
Epoch 10/10
1800/1800 [==============================] - 48s 27ms/step - loss: 4.0575e-04 - acc: 1.0000

</code>

### Plot the Model_____no_output_____<img src='output/multichannel_cnn/cnn.png'/>_____no_output_____## Evaluate Model

Evaluate the trained model performance by using both training and test dataset_____no_output_____

<code>
from pickle import load
from numpy import array
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.models import load_model

# load a clean dataset
def load_dataset(filename):
	return load(open(filename, 'rb'))

# fit a tokenizer
def create_tokenizer(lines):
	tokenizer = Tokenizer()
	tokenizer.fit_on_texts(lines)
	return tokenizer

# calculate the maximum document length
def max_length(lines):
	return max([len(s.split()) for s in lines])

# encode a list of lines
def encode_text(tokenizer, lines, length):
	# integer encode
	encoded = tokenizer.texts_to_sequences(lines)
	# pad encoded sequences
	padded = pad_sequences(encoded, maxlen=length, padding='post')
	return padded

# load datasets
trainLines, trainLabels = load_dataset(data_dir+'train.pkl')
testLines, testLabels = load_dataset(data_dir+'test.pkl')

# create tokenizer
tokenizer = create_tokenizer(trainLines)
# calculate max document length
length = max_length(trainLines)
# calculate vocabulary size
vocab_size = len(tokenizer.word_index) + 1
print('Max document length: %d' % length)
print('Vocabulary size: %d' % vocab_size)
# encode data
trainX = encode_text(tokenizer, trainLines, length)
testX = encode_text(tokenizer, testLines, length)
print(trainX.shape, testX.shape)

# load the model
model = load_model(output_dir+'model.h5')

# evaluate model on training dataset
loss, acc = model.evaluate([trainX,trainX,trainX], array(trainLabels), verbose=0)
print('Train Accuracy: %f' % (acc*100))

# evaluate model on test dataset dataset
loss, acc = model.evaluate([testX,testX,testX],array(testLabels), verbose=0)
print('Test Accuracy: %f' % (acc*100))Max document length: 1380
Vocabulary size: 44277
(1800, 1380) (200, 1380)
Train Accuracy: 100.000000
Test Accuracy: 89.000000

</code>

## Multichannle CNN + LSTM in parallel_____no_output_____

<code>
from pickle import load
from numpy import array
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.utils.vis_utils import plot_model
from keras.models import Model
from keras.layers import Input
from keras.layers import Dense
from keras.layers import Flatten
from keras.layers import Dropout
from keras.layers import Embedding
from keras.layers import LSTM
from keras.layers.convolutional import Conv1D
from keras.layers.convolutional import MaxPooling1D
from keras.layers.merge import concatenate

# load a clean dataset
def load_dataset(filename):
	return load(open(filename, 'rb'))

# fit a tokenizer
def create_tokenizer(lines):
	tokenizer = Tokenizer()
	tokenizer.fit_on_texts(lines)
	return tokenizer

# calculate the maximum document length
def max_length(lines):
	return max([len(s.split()) for s in lines])

# encode a list of lines
def encode_text(tokenizer, lines, length):
	# integer encode
	encoded = tokenizer.texts_to_sequences(lines)
	# pad encoded sequences
	padded = pad_sequences(encoded, maxlen=length, padding='post')
	return padded

# define the model
def define_model(length, vocab_size):
	# channel 1
	inputs1 = Input(shape=(length,),dtype='int32')
	embedding1 = Embedding(vocab_size, 100)(inputs1)
	conv1 = Conv1D(filters=32, kernel_size=4, activation='relu')(embedding1)
	drop1 = Dropout(0.5)(conv1)
	pool1 = MaxPooling1D(pool_size=2)(drop1)
	flat1 = Flatten()(pool1)
	# channel 2
	inputs2 = Input(shape=(length,),dtype='int32')
	embedding2 = Embedding(vocab_size, 100)(inputs2)
	conv2 = Conv1D(filters=32, kernel_size=6, activation='relu')(embedding2)
	drop2 = Dropout(0.5)(conv2)
	pool2 = MaxPooling1D(pool_size=2)(drop2)
	flat2 = Flatten()(pool2)
	# channel 3
	inputs3 = Input(shape=(length,),dtype='int32')
	embedding3 = Embedding(vocab_size, 100)(inputs3)
	conv3 = Conv1D(filters=32, kernel_size=8, activation='relu')(embedding3)
	drop3 = Dropout(0.5)(conv3)
	pool3 = MaxPooling1D(pool_size=2)(drop3)
	flat3 = Flatten()(pool3)

	# cnn + lstm
	inputs4 = Input(shape=(length,),dtype='int32')
	embedding4 = Embedding(vocab_size, 100)(inputs4)
	conv4 = Conv1D(filters=32, kernel_size=3, padding='same', activation='relu')(embedding4)
	drop4 = Dropout(0.2)(conv4)
	pool4 = MaxPooling1D(pool_size=2)(drop4)
	lstm4 = LSTM(100, dropout=0.2, recurrent_dropout=0.2)(pool4)
	drop_lstm4 = Dropout(0.2)(lstm4)
	#lstm4 = LSTM(100, dropout=0.2, recurrent_dropout=0.2, input_shape=(pool4.shape[1], pool4.shape[2]), return_sequences=True)(pool4)

	# merge
	merged = concatenate([flat1, flat2, flat3, lstm4])

	# interpretation
	#dense1 = Dense(10, activation='relu')(merged)
	#outputs = Dense(1, activation='sigmoid')(dense1)  #model.add(Dense(num_class,activation='softmax'))

	outputs = Dense(1, activation='sigmoid')(merged)  #model.add(Dense(num_class,activation='softmax'))

	model = Model(inputs=[inputs1, inputs2, inputs3, inputs4], outputs=outputs)
	# compile
	model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) #model.compile(loss = 'categorical_crossentropy', optimizer='adam',metrics = ['accuracy'])
	# summarize
	print(model.summary())
	plot_model(model, show_shapes=True, to_file='output/multichannel_cnn/cnn_lstm.png')
	return model

# load training dataset
trainLines, trainLabels = load_dataset(data_dir+'train.pkl')
# create tokenizer
tokenizer = create_tokenizer(trainLines)
# calculate max document length
length = max_length(trainLines)
# calculate vocabulary size
vocab_size = len(tokenizer.word_index) + 1
print('Max document length: %d' % length)
print('Vocabulary size: %d' % vocab_size)
# encode data
trainX = encode_text(tokenizer, trainLines, length)
print(trainX.shape)

# define model
model = define_model(length, vocab_size)
# fit model
model.fit([trainX,trainX,trainX,trainX], array(trainLabels), epochs=5, batch_size=16)
# save the model
model.save(output_dir+'multichannel_cnn_lstm_model.h5')Max document length: 1380
Vocabulary size: 44277
(1800, 1380)
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_19 (InputLayer)           (None, 1380)         0
__________________________________________________________________________________________________
input_20 (InputLayer)           (None, 1380)         0
__________________________________________________________________________________________________
input_21 (InputLayer)           (None, 1380)         0
__________________________________________________________________________________________________
input_22 (InputLayer)           (None, 1380)         0
__________________________________________________________________________________________________
embedding_19 (Embedding)        (None, 1380, 100)    4427700     input_19[0][0]
__________________________________________________________________________________________________
embedding_20 (Embedding)        (None, 1380, 100)    4427700     input_20[0][0]
__________________________________________________________________________________________________
embedding_21 (Embedding)        (None, 1380, 100)    4427700     input_21[0][0]
__________________________________________________________________________________________________
embedding_22 (Embedding)        (None, 1380, 100)    4427700     input_22[0][0]
__________________________________________________________________________________________________
conv1d_19 (Conv1D)              (None, 1377, 32)     12832       embedding_19[0][0]
__________________________________________________________________________________________________
conv1d_20 (Conv1D)              (None, 1375, 32)     19232       embedding_20[0][0]
__________________________________________________________________________________________________
conv1d_21 (Conv1D)              (None, 1373, 32)     25632       embedding_21[0][0]
__________________________________________________________________________________________________
conv1d_22 (Conv1D)              (None, 1380, 32)     9632        embedding_22[0][0]
__________________________________________________________________________________________________
dropout_19 (Dropout)            (None, 1377, 32)     0           conv1d_19[0][0]
__________________________________________________________________________________________________
dropout_20 (Dropout)            (None, 1375, 32)     0           conv1d_20[0][0]
__________________________________________________________________________________________________
dropout_21 (Dropout)            (None, 1373, 32)     0           conv1d_21[0][0]
__________________________________________________________________________________________________
dropout_22 (Dropout)            (None, 1380, 32)     0           conv1d_22[0][0]
__________________________________________________________________________________________________
max_pooling1d_19 (MaxPooling1D) (None, 688, 32)      0           dropout_19[0][0]
__________________________________________________________________________________________________
max_pooling1d_20 (MaxPooling1D) (None, 687, 32)      0           dropout_20[0][0]
__________________________________________________________________________________________________
max_pooling1d_21 (MaxPooling1D) (None, 686, 32)      0           dropout_21[0][0]
__________________________________________________________________________________________________
max_pooling1d_22 (MaxPooling1D) (None, 690, 32)      0           dropout_22[0][0]
__________________________________________________________________________________________________
flatten_17 (Flatten)            (None, 22016)        0           max_pooling1d_19[0][0]
__________________________________________________________________________________________________
flatten_18 (Flatten)            (None, 21984)        0           max_pooling1d_20[0][0]
__________________________________________________________________________________________________
flatten_19 (Flatten)            (None, 21952)        0           max_pooling1d_21[0][0]
__________________________________________________________________________________________________
lstm_3 (LSTM)                   (None, 100)          53200       max_pooling1d_22[0][0]
__________________________________________________________________________________________________
concatenate_4 (Concatenate)     (None, 66052)        0           flatten_17[0][0]
                                                                 flatten_18[0][0]
                                                                 flatten_19[0][0]
                                                                 lstm_3[0][0]
__________________________________________________________________________________________________
dense_7 (Dense)                 (None, 1)            66053       concatenate_4[0][0]
==================================================================================================
Total params: 17,897,381
Trainable params: 17,897,381
Non-trainable params: 0
__________________________________________________________________________________________________
None
Epoch 1/5
1800/1800 [==============================] - 128s 71ms/step - loss: 0.7096 - acc: 0.5311
Epoch 2/5
1800/1800 [==============================] - 121s 67ms/step - loss: 0.3130 - acc: 0.9367
Epoch 3/5
1800/1800 [==============================] - 121s 67ms/step - loss: 0.0202 - acc: 0.9978
Epoch 4/5
1800/1800 [==============================] - 123s 69ms/step - loss: 0.0031 - acc: 1.0000
Epoch 5/5
1800/1800 [==============================] - 130s 72ms/step - loss: 0.0016 - acc: 1.0000
from pickle import load
from numpy import array
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.models import load_model

# load a clean dataset
def load_dataset(filename):
	return load(open(filename, 'rb'))

# fit a tokenizer
def create_tokenizer(lines):
	tokenizer = Tokenizer()
	tokenizer.fit_on_texts(lines)
	return tokenizer

# calculate the maximum document length
def max_length(lines):
	return max([len(s.split()) for s in lines])

# encode a list of lines
def encode_text(tokenizer, lines, length):
	# integer encode
	encoded = tokenizer.texts_to_sequences(lines)
	# pad encoded sequences
	padded = pad_sequences(encoded, maxlen=length, padding='post')
	return padded

# load datasets
trainLines, trainLabels = load_dataset(data_dir+'train.pkl')
testLines, testLabels = load_dataset(data_dir+'test.pkl')

# create tokenizer
tokenizer = create_tokenizer(trainLines)
# calculate max document length
length = max_length(trainLines)
# calculate vocabulary size
vocab_size = len(tokenizer.word_index) + 1
print('Max document length: %d' % length)
print('Vocabulary size: %d' % vocab_size)
# encode data
trainX = encode_text(tokenizer, trainLines, length)
testX = encode_text(tokenizer, testLines, length)
print(trainX.shape, testX.shape)

# load the model
model = load_model(output_dir+'multichannel_cnn_lstm_model.h5')

# evaluate model on training dataset
loss, acc = model.evaluate([trainX,trainX,trainX, trainX], array(trainLabels), verbose=0)
print('Train Accuracy: %f' % (acc*100))

# evaluate model on test dataset dataset
loss, acc = model.evaluate([testX,testX,testX,testX],array(testLabels), verbose=0)
print('Test Accuracy: %f' % (acc*100))Max document length: 1380
Vocabulary size: 44277
(1800, 1380) (200, 1380)
Train Accuracy: 100.000000
Test Accuracy: 85.500000

</code>

<img src='output/multichannel_cnn/cnn_lstm.png'/>_____no_output_____### Use GridSV to find the best model
https://github.com/keras-team/keras/blob/master/examples/mnist_sklearn_wrapper.py_____no_output_____


# Notebook from ojhall94/malatium
Path: code/tests/old_builds/pymc3_pb_smoothness.ipynb

# Properly inferring the mode frequencies from the data
I'm going to add a proper treatment of the mode frequencies. The remaining caveats are:

- I will not impose a complex prior on linewidth
- I will not impose a complex prior on mode heights
- I am not accounting for any asphericities due to near-surface magnetic fields

The expected effect of this will be, in order:

- Increased uncertainty on linewidths
- Increased runtime as the mode heights are less constrained
- Possible linewidth broadening or shifting of mode frequencies if there are significant frequency shifts

Things that I've written but not implemented in this example (for speed purposes):

- A comprehensive background treatment

Instead we are using:

- Dividing the background out and just using a single parameter._____no_output_____

import numpy as np
import matplotlib.pyplot as plt

import lightkurve as lk
from astropy.units import cds
from astropy import units as u
import seaborn as sns
import mystyle as ms

import corner
import pystan
import pandas as pd
import pickle
import glob
from astropy.io import ascii
import os

from utils import *

import pymc3 as pm
import arvizWARNING:theano.tensor.blas:Using NumPy C-API based implementation for BLAS functions.
target = 3632418
f, p, ff, pp, star, modelocs, elocs, modeids, overtones = read_data(target, 0, -1)_____no_output_____kic = star.KIC
dnu_ = star.dnu
numax_ = star.numax_____no_output_____f, p = rebin(f, p, 50)Length: 105910
Length: 2118
backdir = glob.glob('/home/oliver/PhD/mnt/RDS/malatium/backfit/'
                +str(kic)+'/*_fit.pkl')[0]
with open(backdir, 'rb') as file:
    backfit = pickle.load(file)

labels=['loga','logb','logc','logd','logj','logk','white','scale','nyq']
res = np.array([np.median(backfit[label]) for label in labels])
res[0:6] = 10**res[0:6]

phi_ = np.array([np.median(backfit[label]) for label in labels])
phi_sigma = pd.DataFrame(backfit)[labels].cov()
phi_cholesky = np.linalg.cholesky(phi_sigma)

mm = get_background(ff, *res)
m = get_background(f, *res)
p /= m
pp /= mm_____no_output_____# Read in the fit data
cad = pd.read_csv('../../data/cadmium.csv', index_col=0)
cad = cad.loc[cad.KIC == target]_____no_output_____pg = lk.Periodogram(ff*u.microhertz, pp*(cds.ppm**2 / u.microhertz))
ax = pg.plot(alpha=.5)
ax.plot(f, p)
ax.plot(ff, mm)
plt.scatter(modelocs, [15]*len(modelocs),c=modeids, s=50, edgecolor='w',zorder=100)
ax.set_xlim(500,2000)
ax.set_ylim(0, 50)_____no_output_____pg = lk.periodogram.SNRPeriodogram(f*u.microhertz, p*(cds.ppm**2/u.microhertz))
ax = pg.plot(alpha=.5)
# pg.smooth(filter_width=3.).plot(ax=ax, linewidth=2)
ax.plot(f, m)
plt.scatter(modelocs, [15]*len(modelocs),c=modeids, s=20, edgecolor='k')
plt.show()_____no_output_____

## Finding the mode frequencies through the asymptotic relation_____no_output_____We have good constraints on the mode frequencies from previous studies, but since we are using the same data, we don't want to use their posteriors as priors on our frequencies. Instead we want to find a way to include the prior knowledge from previous studies without making our study dependent on them.

The locations of the radial $l = 0$ modes can be predicted from the asymptotic relation. Radial modes of consecutive overtones are separated by the large frequency separation $\Delta\nu$, in principle. However in practice, these mode frequencies can be subject to some curvature, as well as noise from glitches in the sound speed profile in the stellar interior.

We're going to omit a detailed treatment of glitches for now, but we *will* include a curvature term, using the astymptotic relation presented in [Vrard et al. 2015](https://ui.adsabs.harvard.edu/abs/2015A%26A...579A..84V/abstract), which goes as

$\large\nu_{l=0} = (\bar{n} + \epsilon + (\frac{\alpha}{2}(n_{\rm max} - \bar{n})^2)) \Delta\nu + \Delta$

where $\nu_{\rm l=0}$ is the frequency locations of all l=0 modes at overtones $\bar{n}$, $\epsilon$ is the phase offset, $\alpha$ determines the curvature, $\Delta\nu$ is the large separation, and $n_{\rm max}$ is the overtone closest to $\nu_{\rm max}$, round which the curvature is centered, and is given by

$n_{\rm max} = \frac{\nu_{\rm max}}{\Delta\nu} - \epsilon$,

and $\Delta$ is the noise on the frequency positions. Including this noise term allows us to formalize the 'smoothness condition' ([Davies et al. 2016](https://ui.adsabs.harvard.edu/abs/2016MNRAS.456.2183D/abstract)), where we specify that the difference in large frequency separation between subsequent radial modes should be close to zero, with some scatter. We therefore specify $\Delta$ as:

$\Delta = \mathcal{N}(0, \sigma_\Delta)$,

where $\sigma_\Delta$ is a free parameter. I guess eventually we should really upgrade this to a Gaussian Process periodic Kernel, so that we take care of glitch patterns.

The positions of the dipole and octopole $l = 1, 2$ modes are then determined from the radial frequencies, as

$\nu_{l=1} = \nu_{l=0} + \delta\nu_{01} + \mathcal{N}(0, \sigma_{01})$

$\nu_{l=2} = \nu_{l=0} -  \delta\nu_{02} + \mathcal{N}(0, \sigma_{02})$

where $\delta\nu_{01}$ and $\delta\nu_{02}$ are the small separations between the radial frequency and the dipole and octopole frequencies of the same radial degree. $\sigma_{01}$ and $\sigma_{02}$ are the uncertainties on these separations. All are free parameters, and as before we're adding on noise.

So we know have a complex hierarchical system where the mode frequencies are latent parameters, and we have a bunch of hyperparameters controlling them, giving them noise and curvature. These hyperparameters are where we include our *prior* information from the Kages and LEGACY papers, as first guesses and as means on the distributions from which they are drawn.

$\epsilon \sim \mathcal{N}(\epsilon_{\rm prior}, 1.)$

$\alpha \sim \mathcal{N_{\rm log}}(\alpha_{\rm prior}, 1.)$

$\Delta\nu \sim \mathcal{N}(\Delta\nu_{\rm prior}, \Delta\nu_{\rm prior}*0.1)$

$\nu_{\rm max} \sim \mathcal{N}(\nu_{\rm max, \rm prior}, \nu_{\rm max, \rm prior}*0.1)$

$\delta\nu_{01} \sim \mathcal{N}(\delta\nu_{01, \rm prior}, \Delta\nu_{\rm prior}*0.1)$

$\delta\nu_{02} \sim \mathcal{N}(\delta\nu_{02, \rm prior}, 3.)$

$\epsilon$, $\alpha$ and the small separations will be determined from a fit to the mode frequencies of each star. The remainder are taken as reported in the literature.

The noise terms $\sigma_\Delta$, $\sigma_{01}$ and $\sigma_{02}$ will all be drawn from lognormal distributions, to ensure they don't go too close to zero.

Notice that I've flipped the sign in the equation for $\nu_{l=2}$. In a typical $l=0,2$ pair, the octopole mode is one overtone lower than the radial mode. When passing the overtone numbers into the model, we simply add $+1$ to those of the octopole modes to make this equation work and maintain our traditional measure of $\delta\nu_{\rm 02}$.

We use the overtone numbers $n$ reported in LEGACY and Kages, and also do not fit for any modes of oscillation not reported in those papers._____no_output_____# Build the model_____no_output_____

class model():
    def __init__(self, f, n0, n1, n2, f0_, f1_, f2_):
        self.f = f
        self.n0 = n0
        self.n1 = n1
        self.n2 = n2
        self.npts = len(f)
        self.M = [len(f0_), len(f1_), len(f2_)]

    def epsilon(self, i, l, m):
    #We use the prescriptions from Gizon & Solanki 2003 and Handberg & Campante 2012
        if l == 0:
            return 1
        if l == 1:
            if m == 0:
                return np.cos(i)**2
            if np.abs(m) == 1:
                return 0.5 * np.sin(i)**2
        if l == 2:
            if m == 0:
                return 0.25 * (3 * np.cos(i)**2 - 1)**2
            if np.abs(m) ==1:
                return (3/8)*np.sin(2*i)**2
            if np.abs(m) == 2:
                return (3/8) * np.sin(i)**4
        if l == 3:
            if m == 0:
                return (1/64)*(5*np.cos(3*i) + 3*np.cos(i))**2
            if np.abs(m) == 1:
                return (3/64)*(5*np.cos(2*i) + 3)**2 * np.sin(i)**2
            if np.abs(m) == 2:
                return (15/8) * np.cos(i)**2 * np.sin(i)**4
            if np.abs(m) == 3:
                return (5/16)*np.sin(i)**6

    def lor(self, freq, h, w):
        return h / (1.0 + 4.0/w**2*(self.f - freq)**2)

    def mode(self, l, freqs, hs, ws, i, split=0):
        for idx in range(self.M[l]):
            for m in range(-l, l+1, 1):
                self.modes += self.lor(freqs[idx] + (m*split),
                                     hs[idx] * self.epsilon(i, l, m),
                                     ws[idx])

    def model(self, p):
        f0, f1, f2, g0, g1, g2, h0, h1, h2, split, i, b = p

        # Calculate the modes
        self.modes = np.zeros(self.npts)
        self.mode(0, f0, h0, g0, i)
        self.mode(1, f1, h1, g1, i, split)
        self.mode(2, f2, h2, g2, i, split)

        #Create the model
        self.mod = self.modes + b
        return self.mod

    def asymptotic(self, n, numax, deltanu, alpha, epsilon):
        nmax = (numax / deltanu) - epsilon
        over = (n + epsilon + ((alpha/2)*(nmax - n)**2))
        return over * deltanu

    def f0(self, p):
        numax, deltanu, alpha, epsilon, d01, d02 = p

        return self.asymptotic(self.n0, numax, deltanu, alpha, epsilon)

    def f1(self, p):
        numax, deltanu, alpha, epsilon, d01, d02 = p

        f0 = self.asymptotic(self.n1, numax, deltanu, alpha, epsilon)
        return f0 + d01

    def f2(self, p):
        numax, deltanu, alpha, epsilon, d01, d02 = p

        f0 = self.asymptotic(self.n2+1, numax, deltanu, alpha, epsilon)
        return f0 - d02_____no_output_____f0_ = modelocs[modeids==0]
f1_ = modelocs[modeids==1]
f2_ = modelocs[modeids==2]
f0_e = elocs[modeids==0]
f1_e = elocs[modeids==1]
f2_e = elocs[modeids==2]
n0 = overtones[modeids==0]
n1 = overtones[modeids==1]
n2 = overtones[modeids==2]

alpha_ = cad.alpha.values[0]
alpha_e = cad.e_alpha.values[0]
epsilon_ = cad.epsilon.values[0]
epsilon_e = cad.e_epsilon.values[0]
d01_ = cad.d01.values[0]
d01_e = cad.e_d01.values[0]
d02_ = cad.d02.values[0]
d02_e = cad.e_d02.values[0]
numax_ = star.numax
numax_e = star.enumax
deltanu_ = star.dnu
deltanu_e = star.ednu_____no_output_____

Do some first guesses for height_____no_output_____

init_m =[f0_,                         # l0 modes
       f1_,                         # l1 modes
       f2_,                         # l2 modes
       np.ones(len(f0_)) * 2.0,     # l0 widths
       np.ones(len(f1_)) * 2.0,     # l1 widths
       np.ones(len(f2_)) * 2.0,     # l2 widths
       np.sqrt(gaussian(f0_, 0, numax_, 15.) * 2.0 * np.pi / 2.0) ,# l0 heights
       np.sqrt(gaussian(f1_, 1, numax_, 15.) * 2.0 * np.pi / 2.0) ,# l1 heights
       np.sqrt(gaussian(f2_, 2, numax_, 15.) * 2.0 * np.pi / 2.0) ,# l2 heights
       1.0 * np.sin(np.pi/2),       # projected splitting
       np.pi/2.,                    # inclination angle
       1.                           # background parameters
        ]

init_f =[numax_,                       # numax
       dnu_,                         # deltanu
       alpha_,                      # curvature term
       epsilon_,                    # phase term
       d01_ ,                       # small separation l=0,1
       d02_                        # small separation l=0,2
        ]

mod = model(f, n0, n1, n2, f0_, f1_, f2_)
with plt.style.context(lk.MPLSTYLE):
    fig, ax = plt.subplots()
    ax.plot(f, p)
    ax.plot(f, mod.model(init_m), lw=2)
    ax.scatter(modelocs, [15]*len(modelocs),c=modeids, s=50, edgecolor='grey', zorder=100)
    plt.show()

    fig, ax = plt.subplots()
    ax.errorbar(f0_%dnu_, n0, xerr=f0_e, fmt='^',label='0')
    ax.errorbar(f1_%dnu_, n1, xerr=f1_e, fmt='>',label='1')
    ax.errorbar(f2_%dnu_, n2, xerr=f2_e, fmt='o',label='2')

    ax.plot(mod.f0(init_f)%dnu_, n0, label='0')
    ax.plot(mod.f1(init_f)%dnu_, n1, label='1')
    ax.plot(mod.f2(init_f)%dnu_, n2, label='2')

    ax.legend()
    plt.show()_____no_output_____

# Build the priors in PyMC3_____no_output_____

pm_model = pm.Model()

with pm_model:
    # Frequency Hyperparameters
    alpha   = pm.TruncatedNormal('alpha',   mu=alpha_,   sigma=alpha_e,   testval=alpha_, lower=0.)
    epsilon = pm.TruncatedNormal('epsilon', mu=epsilon_, sigma=epsilon_e, testval=epsilon_, lower=0.)
    d01     = pm.TruncatedNormal('d01',     mu=d01_,     sigma=d01_e,     testval=d01_, lower=0.)
    d02     = pm.TruncatedNormal('d02',     mu=d02_,     sigma=d02_e,     testval=d02_, lower=0.)
    numax   = pm.TruncatedNormal('numax',   mu=numax_,   sigma=numax_e,   testval=numax_, lower=0.)
    deltanu = pm.TruncatedNormal('deltanu', mu=deltanu_, sigma=deltanu_e, testval=deltanu_, lower=0.)

    # Noise hyperparameters and latent parameters
    sigma0 =  pm.Gamma('sigma0', alpha=2.0, beta=0.5, testval=2.)
    sigma01 = pm.Gamma('sigma01', alpha=2.0, beta=0.5, testval=2.)
    sigma02 = pm.Gamma('sigma02', alpha=2.0, beta=0.5, testval=2.)

    Delta0 = pm.Normal('Delta0', mu=0., sigma=1., shape=len(init_m[0]))
    Delta1 = pm.Normal('Delta1', mu=0., sigma=1., shape=len(init_m[1]))
    Delta2 = pm.Normal('Delta2', mu=0., sigma=1., shape=len(init_m[2]))

    #Frequencies
    f0 = pm.Deterministic('f0', mod.f0([numax_, deltanu_, alpha_, epsilon_, d01_, d02_]) + Delta0*sigma0)
    f1 = pm.Deterministic('f1', mod.f1([numax_, deltanu_, alpha_, epsilon_, d01_, d02_]) + Delta1*sigma01)
    f2 = pm.Deterministic('f2', mod.f2([numax_, deltanu_, alpha_, epsilon_, d01_, d02_]) + Delta2*sigma02)

    # Mode linewidths
    g0 = pm.HalfNormal('g0', sigma=2.0, testval=init_m[3], shape=len(init_m[3]))
    g1 = pm.HalfNormal('g1', sigma=2.0, testval=init_m[4], shape=len(init_m[4]))
    g2 = pm.HalfNormal('g2', sigma=2.0, testval=init_m[5], shape=len(init_m[5]))

    # Mode amplitudes
    a0 = pm.HalfNormal('a0', sigma=20., testval=init_m[6], shape=len(init_m[6]))
    a1 = pm.HalfNormal('a1', sigma=20., testval=init_m[7], shape=len(init_m[7]))
    a2 = pm.HalfNormal('a2', sigma=20., testval=init_m[8], shape=len(init_m[8]))

    # Mode heights (determined by amplitude and linewidth)
    h0 = pm.Deterministic('h0', 2*a0**2/np.pi/g0)
    h1 = pm.Deterministic('h1', 2*a1**2/np.pi/g1)
    h2 = pm.Deterministic('h2', 2*a2**2/np.pi/g2)

    # Rotation and inclination parameterizations
    xsplit = pm.HalfNormal('xsplit', sigma=2.0, testval=init_m[10])
    cosi = pm.Uniform('cosi', 0., 1.)

    # Detangled inclination and splitting
    i = pm.Deterministic('i', np.arccos(cosi))
    split = pm.Deterministic('split', xsplit/pm.math.sin(i))

    # Background terms
    b = pm.TruncatedNormal('b', mu=1., sigma=.1, testval=1., lower=0.)

    #Model
    fit = mod.model([f0, f1, f2, g0, g1, g2, h0, h1, h2, split, i, b])

#     like = pm.Gamma('like', alpha=1, beta=1.0/fit, observed=p)
    like = pm.Normal('like', mu=fit, sigma=2, observed=p)_____no_output_____init = 1000
with pm_model:
    trace = pm.sample(chains=4, tune=int(init/2), draws=int(init/2))INFO:pymc3:Auto-assigning NUTS sampler...
INFO:pymc3:Initializing NUTS using jitter+adapt_diag...
pm.traceplot(trace, var_names=['alpha','epsilon','numax','deltanu','d01','d02',
                             'f0','f1','f2','Delta0','Delta01','Delta02','sigma0', 'sigma01', 'sigma02',
                            'h0','h1','h2','g0','g1','g2','a0','a1','a2','xsplit','cosi','i','split','b'])
plt.show()_____no_output_____pm.summary(trace, var_names=['alpha','epsilon','numax','deltanu','d01','d02',
                             'f0','f1','f2','Delta0','Delta01','Delta02','sigma0', 'sigma01', 'sigma02',
                            'h0','h1','h2','g0','g1','g2','a0','a1','a2','xsplit','cosi','i','split','b'])_____no_output_____

# Now lets plot some diagnostics..._____no_output_____

labels=['xsplit','cosi','i','split','b','numax','deltanu','alpha','epsilon','d01','d02']
chain = np.array([trace[label] for label in labels])
verbose = [r'$\delta\nu_s^*$',r'$\cos(i)$',r'$i$',r'$\delta\nu_{\rm s}$',
          r'$\nu_{\rm max}$', r'$\Delta\nu$', r'$\alpha$',r'$\epsilon$',r'$\delta\nu_{01}$',r'$\delta\nu_{02}$']
corner.corner(chain.T, labels=verbose, quantiles=[0.16, 0.5, 0.84]
              ,show_titles=True)
plt.show()_____no_output_____for idx in range(len(trace['h0'].T)):
    chain = np.array([trace['a0'].T[idx], trace['h0'].T[idx], trace['g0'].T[idx], trace['f0'].T[idx]])
    corner.corner(chain.T, labels=['A','H','g','f'],
                  quantiles=[0.16, 0.5, 0.84],show_titles=True)
    plt.show()_____no_output_____

Looks like all the background parameters have been tightened up or remained within the priors. Always good to check!_____no_output_____### Now let's plot some output evaluation:_____no_output_____

with plt.style.context(lk.MPLSTYLE):
    fig, ax = plt.subplots()
    ax.plot(f, p)
    labels=['f0','f1','f2','g0','g1','g2','h0','h1','h2','split','i', 'phi']
    res = np.array([np.median(trace[label],axis=0) for label in labels])
    ax.plot(f, mod.model(init_m), lw=2)
    plt.show()

    fig, ax = plt.subplots()

    res = [np.median(trace[label]) for label in ['numax','deltanu','alpha','epsilon','d01','d02']]
    resls = [np.median(trace[label],axis=0) for label in['f0','f1','f2']]
    stdls = [np.std(trace[label],axis=0) for label in['f0','f1','f2']]

    ax.plot(mod.f0(res)%res[1], n0, label='0 asy')
    ax.plot(mod.f1(res)%res[1], n1, label='1 asy')
    ax.plot(mod.f2(res)%res[1], n2, label='2 asy')

    ax.errorbar(resls[0]%res[1], n0, xerr=stdls[0], fmt='^',label='0 mod')
    ax.errorbar(resls[1]%res[1], n1, xerr=stdls[1], fmt='>',label='1 mod')
    ax.errorbar(resls[2]%res[1], n2, xerr=stdls[2], fmt='o',label='2 mod')

    ax.legend()_____no_output_____labels=['f0','f1','f2','g0','g1','g2','h0','h1','h2','split','i', 'phi']
res = np.array([np.median(trace[label],axis=0) for label in labels])
residual = p/mod.model(res)
sns.distplot(residual, label='Model')
sns.distplot(np.random.chisquare(2, size=10000)/2, label=r'Chi22')
sns.distplot(np.random.normal(size=10000)*np.std(residual) + np.mean(residual), label='Gaussian')
plt.legend()_____no_output_____

The model doesn't fit the $\chi_2^2$ noise, probably because its binned? _____no_output_____# Verdict:

I'm confident in the completion of this addition, but it takes too long a time to use when testing other targets. I'll save it for now, run a longer test overnight and move on to GP testing._____no_output_____

import sys
sys.exit()_____no_output_____

# Leftovers_____no_output_____

res = [np.median(trace[label]) for label in ['numax','deltanu','alpha','epsilon','d01','d02']]

resls = [np.median(trace[label],axis=0) for label in['f0','f1','f2']]
stdls = [np.std(trace[label],axis=0) for label in['f0','f1','f2']]

with plt.style.context(ms.ms):
    fig, ax = plt.subplots()

    ax.plot(mod.f0(res)%res[1], n0, label='0 asy')
    ax.plot(mod.f1(res)%res[1], n1, label='1 asy')
    ax.plot(mod.f2(res)%res[1], n2, label='2 asy')

    ax.errorbar(resls[0]%res[1], n0, xerr=stdls[0], fmt='^',label='0 mod')
    ax.errorbar(resls[1]%res[1], n1, xerr=stdls[1], fmt='>',label='1 mod')
    ax.errorbar(resls[2]%res[1], n2, xerr=stdls[2], fmt='o',label='2 mod')

    ax.errorbar(f0_%res[1], n0, xerr=f0_e, fmt='^',label='0 lit')
    ax.errorbar(f1_%res[1], n1, xerr=f1_e, fmt='>',label='1 lit')
    ax.errorbar(f2_%res[1], n2, xerr=f2_e, fmt='o',label='2 lit')

    ax.legend()_____no_output_____


# Notebook from Kaushal1011/machine-learning
Path: src/analytics/pandas-primer.ipynb

## pandas
---
Basics guide to understand *pandas Python Data Analysis Library*_____no_output_____

import numpy as np
import pandas as pd
from numpy import nan as NA
from pandas import DataFrame, Series_____no_output_____np.__version_______no_output_____pd.__version_______no_output_____

### Series
---
A Series is a one-dimensional array-like object containing an array of data
(of any NumPy data type) and an associated array of data labels, called its *index*._____no_output_____

obj = Series([4, 7, -5, 3])_____no_output_____obj_____no_output_____obj.values_____no_output_____obj.index_____no_output_____obj2 = Series([4, 7, -5, 3], index=['d', 'b', 'a', 'c'])_____no_output_____obj2_____no_output_____obj2.index_____no_output_____obj2['a']_____no_output_____obj2[['c', 'a', 'd']]_____no_output_____

By default, the index is from $0$ to $len - 1$_____no_output_____

obj2[obj2 > 0]_____no_output_____obj2 * 2_____no_output_____np.exp(obj2)_____no_output_____

Integer indexing may generate errors on using -1 as callable index

Another way to think about a Series is as a fixed-length OrderedDict,
as it is a mapping of index values to data values_____no_output_____

'b' in obj2_____no_output_____'e' in obj2_____no_output_____sdata = {
    'Ohio': 35,
    'Texas': 71,
    'Oregon': 16,
    'Utah': 5
}_____no_output_____obj3 = Series(sdata)_____no_output_____obj3_____no_output_____states = ['California', 'Ohio', 'Oregon', 'Texas']_____no_output_____obj4 = Series(sdata, index=states)_____no_output_____obj4_____no_output_____obj4.isnull()_____no_output_____obj4.notnull()_____no_output_____

Series automatically aligns differently indexed data in arithmetic operations_____no_output_____

obj3 + obj4_____no_output_____

Series object and its index both have *name* attribute_____no_output_____

obj4.name = 'population'_____no_output_____obj4.index.name = 'state'_____no_output_____obj4_____no_output_____obj.index = ['Bob', 'Steve', 'Jeff', 'Ryan']_____no_output_____obj_____no_output_____

### DataFrame
---
A DataFrame represents a tabular, spreadsheet-like data structure
containing an ordered collection of columns. The DataFrame has both a row and column index.
It can be thought of as a dict of Series._____no_output_____

data = {
    'state': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada'],
    'year': [2000, 2001, 2002, 2001, 2002],
    'pop': [1.5, 1.7, 3.6, 2.4, 2.9]
}_____no_output_____frame = DataFrame(data)_____no_output_____frame_____no_output_____frame2 = DataFrame(data,
                   columns=['year', 'state', 'pop', 'debt'],
                   index=['one', 'two', 'three', 'four', 'five'])_____no_output_____frame2_____no_output_____frame2.columns_____no_output_____frame2['state']_____no_output_____frame2.state_____no_output_____frame2.iloc[0]_____no_output_____frame2.loc['four']_____no_output_____frame2['debt'] = 16.5_____no_output_____frame2_____no_output_____frame2['debt'] = np.arange(5)_____no_output_____frame2_____no_output_____

When assigning lists or arrays to a column, the value’s length must match the length
of the DataFrame. If a Series is assigned, it will be instead conformed exactly to the
DataFrame’s index, inserting missing values in any holes_____no_output_____

val = Series([-1.2, -1.5, -1.7], index=['two', 'four', 'five'])_____no_output_____frame2['debt'] = val_____no_output_____frame2_____no_output_____

Assigning a column that doesn’t exist will create a new column. The `del` keyword will
delete columns as with a dict_____no_output_____

frame2['eastern'] = frame2.state == 'Ohio'_____no_output_____frame2_____no_output_____del frame2['eastern']_____no_output_____frame2.columns_____no_output_____

If a nested dict of dicts is passed to a DataFrame,
it will interpret the outer dict keys as the columns
and the inner keys as the row indices_____no_output_____

pop = {
    'Nevada': {
        2001: 2.4,
        2002: 2.9
    },
    'Ohio': {
        2000: 1.5,
        2001: 1.7,
        2002: 3.6
    }
}_____no_output_____frame3 = DataFrame(pop)_____no_output_____frame3_____no_output_____frame3.T_____no_output_____

dict of Series can also be passed_____no_output_____

pdata = {'Ohio': frame3['Ohio'][:-1], 'Nevada': frame3['Nevada'][:2]}_____no_output_____DataFrame(pdata)_____no_output_____frame3.index.name = 'year'
frame3.columns.name = 'state'_____no_output_____frame3_____no_output_____frame3.values_____no_output_____frame2.values_____no_output_____

Data inputs to DataFrame constructor:
- 2D ndarray
    - A matrix of data, passing optional row and column labels
- dict of arrays, lists, or tuples
    - Each sequence becomes a column in the DataFrame. All sequences must be the same length
- NumPy structured/record array
    - Treated as the “dict of arrays” case
- dict of Series
    - Each value becomes a column. Indexes from each Series are unioned together to form the result’s row index if no explicit index is passed
- dict of dicts
    - Each inner dict becomes a column. Keys are unioned to form the row index as in the “dict of Series” case
- list of dicts or Series
    - Each item becomes a row in the DataFrame. Union of dict keys or Series indexes become the DataFrame’s column labels
- list of lists or tuples
    - Treated as the “2D ndarray” case
- DataFrame
    - The DataFrame’s indexes are used unless different ones are passed
- NumPy MaskedArray
    - Like the “2D ndarray” case except masked values become NaN/missing in the DataFrame result_____no_output_____### Index objects
---
pandas’s Index objects are responsible for holding the axis labels and other metadata
(like the axis name or names). Any array or other sequence of labels used when
constructing a Series or DataFrame is internally converted to an Index_____no_output_____

obj = Series(range(3), index=['a', 'b', 'c'])_____no_output_____obj_____no_output_____obj.index_____no_output_____obj.index[1:]_____no_output_____

Index objects are immutable so they can be shared safely among data_____no_output_____

index = pd.Index(np.arange(3))_____no_output_____obj2 = Series([1.5, -2.5, 0], index=index)_____no_output_____obj2.index is index_____no_output_____

Index can also behave as a fixed-size set_____no_output_____

frame3_____no_output_____'Ohio' in frame3.columns_____no_output_____2003 in frame3.index_____no_output_____

### Reindexing
---_____no_output_____

obj = Series([4, 7, -5, 3], index=['d', 'b', 'a', 'c'])_____no_output_____obj_____no_output_____obj2 = obj.reindex(['a', 'b', 'c', 'd', 'e'])_____no_output_____obj2_____no_output_____obj.reindex(['a', 'b', 'c', 'd', 'e'], fill_value=0)_____no_output_____obj3 = Series(['blue', 'purple', 'yellow'], index=[0, 2, 4])_____no_output_____obj3_____no_output_____obj3.reindex(range(6), method='ffill')_____no_output_____obj3.reindex(range(6), method='bfill')_____no_output_____

reindex methods

        Argument
        Description

        ffill
        Fill values forward

        bfill
        Fill values backward

_____no_output_____

frame = DataFrame(np.arange(9).reshape((3, 3)),
                  index=['a', 'c', 'd'],
                  columns=['Ohio', 'Texas', 'California'])_____no_output_____frame_____no_output_____frame2 = frame.reindex(['a', 'b', 'c', 'd'])_____no_output_____frame2_____no_output_____states = ['Texas', 'Utah', 'California']_____no_output_____frame.reindex(columns=states)_____no_output_____

### Dropping
---_____no_output_____

obj = Series(np.arange(5.), index=['a', 'b', 'c', 'd', 'e'])_____no_output_____obj_____no_output_____obj.drop(['c', 'd'])_____no_output_____data = DataFrame(np.arange(16).reshape((4, 4)),
                 index=['Ohio', 'Colorado', 'Utah', 'New York'],
                 columns=['one', 'two', 'three', 'four'])_____no_output_____data.drop(['Colorado', 'Ohio'])_____no_output_____data.drop('two', axis=1)_____no_output_____

### Indexing, selection && filtering_____no_output_____

obj = Series(np.arange(4), index=['a', 'b', 'c', 'd'])_____no_output_____obj['b']_____no_output_____obj[0]_____no_output_____obj[2:4]_____no_output_____obj[['b', 'a', 'd']]_____no_output_____obj[[1, 3]]_____no_output_____obj[obj < 2]_____no_output_____

In slicing with labels, the endpoint is inclusive_____no_output_____

obj['b':'c']_____no_output_____obj['b':'c'] = 5_____no_output_____obj_____no_output_____data = DataFrame(np.arange(16).reshape((4, 4)),
                 index=['Ohio', 'Colorado', 'Utah', 'New York'],
                 columns=['one', 'two', 'three', 'four'])_____no_output_____data_____no_output_____data[['three', 'one']]_____no_output_____data['two']_____no_output_____data[:2]_____no_output_____data[2:]_____no_output_____data[data['three'] > 5]_____no_output_____data < 5_____no_output_____data[data < 5] = 0_____no_output_____data_____no_output_____data.loc['Colorado', ['two', 'three']]_____no_output_____data[['four', 'one']].loc[['Colorado', 'Utah']]_____no_output_____data.iloc[:, [3, 0, 1]].loc[['Colorado', 'Utah']]_____no_output_____data.loc[['Colorado', 'Utah']].iloc[:, [3, 0, 1]]_____no_output_____

### Arithmetic and data alignment
---
When adding together objects, if any index pairs are not the same,
the respective index in the result will be the union of the index pairs._____no_output_____

s1 = Series([7.3, -2.5, 3.4, 1.5], index=['a', 'c', 'd', 'e'])_____no_output_____s2 = Series([-2.1, 3.6, -1.5, 4, 3.1], index=['a', 'c', 'e', 'f', 'g'])_____no_output_____s1_____no_output_____s2_____no_output_____s1 + s2_____no_output_____

Missing values propagate in arithmetic computations.

In the case of DataFrame, alignment is performed on both the rows and the columns._____no_output_____

df1 = DataFrame(np.arange(9).reshape((3, 3)),
                columns=list('bcd'),
                index=['Ohio', 'Texas', 'Colorado'])_____no_output_____df2 = DataFrame(np.arange(12).reshape((4, 3)),
                columns=list('bde'),
                index=['Utah', 'Ohio', 'Texas', 'Oregon'])_____no_output_____df1_____no_output_____df2_____no_output_____df1 + df2_____no_output_____df1 = DataFrame(np.arange(12).reshape((3, 4)), columns=list('abcd'))
df2 = DataFrame(np.arange(20).reshape((4, 5)), columns=list('abcde'))_____no_output_____df1_____no_output_____df2_____no_output_____df1 + df2_____no_output_____df1.add(df2, fill_value=0)_____no_output_____df1.reindex(columns=df2.columns, fill_value=0)_____no_output_____

### Operations between DataFrame and Series
---_____no_output_____In NumPy arrays, there is the *broadcasting* property_____no_output_____

arr = np.arange(12).reshape((3, 4))_____no_output_____arr_____no_output_____arr[0]_____no_output_____arr - arr[0]_____no_output_____

Similarly in pandas DataFrame && Series,_____no_output_____

frame = DataFrame(np.arange(12.).reshape((4, 3)),
                  columns=list('bde'),
                  index=['Utah', 'Ohio', 'Texas', 'Oregon'])_____no_output_____frame_____no_output_____series = frame.iloc[0]_____no_output_____series_____no_output_____frame - series_____no_output_____

By default, arithmetic broadcasts down the rows.
If an index value is not found in either of DataFrame's column or Series' index,
the objects will be reindexed to form an union_____no_output_____

series2 = Series(range(3), index=['b', 'e', 'f'])_____no_output_____frame + series2_____no_output_____

**To broadcast over columns instead of rows, we use the arithmetic methods instead of arithmetic operators**_____no_output_____

series3 = frame['d']_____no_output_____frame_____no_output_____series3_____no_output_____frame.sub(series3, axis=0)_____no_output_____

### Function application and mapping
---
NumPy ufuncs (element-wise array methods) work fine with pandas objects_____no_output_____

frame = DataFrame(np.random.randn(4, 3),
                  columns=list('bde'),
                  index=['Utah', 'Ohio', 'Texas', 'Oregon'])_____no_output_____frame_____no_output_____np.abs(frame)_____no_output_____frame.apply(lambda x: x.max() - x.min())_____no_output_____frame.apply(lambda x: x.max() - x.min(), axis=1)_____no_output_____def f(x: Series) -> Series:
    return Series([x.max(), x.min()], index=['max', 'min'])_____no_output_____frame.apply(f, axis='index')_____no_output_____frame.apply(f, axis='columns')_____no_output_____frame.applymap(lambda x: '{:.2f}'.format(x))_____no_output_____frame['e'].map(lambda x: '{:.2f}'.format(x))_____no_output_____

### Sorting and ranking
---_____no_output_____

obj = Series(range(4), index=['d', 'a', 'b', 'c'])_____no_output_____obj.sort_index()_____no_output_____obj.sort_values()_____no_output_____frame = DataFrame(np.arange(8).reshape((2, 4)),
                  index=['three', 'one'],
                  columns=['d', 'a', 'b', 'c'])_____no_output_____frame_____no_output_____frame.sort_index()_____no_output_____frame.sort_index(axis=1)_____no_output_____frame = DataFrame({'b': [4, 7, -3, 2], 'a': [0, 1, 0, 1]})_____no_output_____frame_____no_output_____frame.sort_index()_____no_output_____frame.sort_values(by='a')_____no_output_____frame.sort_values(by=['a', 'b'])_____no_output_____obj.rank()_____no_output_____frame.rank(method='first')_____no_output_____

### Axis indexes with duplicate values
---_____no_output_____

obj = Series(range(5), index=['a', 'a', 'b', 'b', 'c'])_____no_output_____obj_____no_output_____obj.index.is_unique_____no_output_____obj['a']_____no_output_____obj['c']_____no_output_____df = DataFrame(np.random.randn(4, 3), index=['a', 'a', 'b', 'b'])_____no_output_____df_____no_output_____df.loc['a']_____no_output_____

### Summarizing and Computing Descriptive Statistics
---_____no_output_____

df = DataFrame([[1.4, np.nan], [7.1, -4.5], [np.nan, np.nan], [0.75, -1.3]],
               index=['a', 'b', 'c', 'd'],
               columns=['one', 'two'])_____no_output_____df_____no_output_____

Statistical methods exclude missing data during computation_____no_output_____

df.sum()_____no_output_____df.sum(axis=1)_____no_output_____

NaN values are excluded unless entire slice is NaN.
This can be disabled._____no_output_____

df.mean(axis=1, skipna=False)_____no_output_____df.idxmax()_____no_output_____df.idxmin()_____no_output_____df.cumsum()_____no_output_____df.describe()_____no_output_____

describe methods also works on non-numeric data_____no_output_____

obj = Series(['a', 'a', 'b', 'c'] * 4)_____no_output_____obj.describe()_____no_output_____frame.head()_____no_output_____frame.tail()_____no_output_____frame.corr()_____no_output_____frame.cov()_____no_output_____obj.unique()_____no_output_____obj.value_counts()_____no_output_____obj.value_counts(sort=True)_____no_output_____data = DataFrame({
    'Qu1': [1, 3, 4, 3, 4],
    'Qu2': [2, 3, 1, 2, 3],
    'Qu3': [1, 5, 2, 4, 4]
})_____no_output_____data_____no_output_____data.apply(pd.value_counts)_____no_output_____

### Handling Missing Data
---_____no_output_____

string_data = Series(['aardvark', 'artichoke', np.nan, 'avocado'])_____no_output_____string_data_____no_output_____

*Python3's inbuilt None type is also considered NaN by pandas*_____no_output_____

string_data.isnull()_____no_output_____string_data[0] = None_____no_output_____string_data.isnull()_____no_output_____data = Series([1, NA, 3.5, NA, 7])_____no_output_____data_____no_output_____data.dropna()_____no_output_____data[data.notnull()]_____no_output_____data = DataFrame([[1., 6.5, 3.], [1., NA, NA], [NA, NA, NA], [NA, 6.5, 3.]])_____no_output_____data_____no_output_____cleaned = data.dropna()_____no_output_____cleaned_____no_output_____data.dropna(how='all')_____no_output_____data[3] = NA_____no_output_____data_____no_output_____data.dropna(axis=1, how='all')_____no_output_____df = DataFrame(np.random.randn(7, 3))_____no_output_____df_____no_output_____df.iloc[:4, 1] = NA
df.iloc[:2, 2] = NA_____no_output_____df_____no_output_____df.dropna(thresh=3)_____no_output_____df.fillna(0)_____no_output_____df.fillna({1: 0.5, 2: -1})_____no_output_____df.fillna(0, inplace=True)_____no_output_____df_____no_output_____

Interpolation can also be used_____no_output_____

df = DataFrame(np.random.randn(6, 3))_____no_output_____df.iloc[2:, 1] = NA
df.iloc[4:, 2] = NA_____no_output_____df_____no_output_____df.fillna(method='ffill')_____no_output_____df.fillna(method='ffill', limit=2)_____no_output_____df.fillna(data.mean())_____no_output_____

### Hierarchical indexing
---_____no_output_____

data = Series(np.random.randn(10),
              index=[['a', 'a', 'a', 'b', 'b', 'b', 'c', 'c', 'd', 'd'],
                     [1, 2, 3, 1, 2, 3, 1, 2, 2, 3]])_____no_output_____data_____no_output_____data.index_____no_output_____data['b']_____no_output_____data.loc[['b', 'd']]_____no_output_____data[:, 2]_____no_output_____data.unstack()_____no_output_____data.unstack().stack()_____no_output_____frame = DataFrame(np.arange(12).reshape((4, 3)),
                  index=[['a', 'a', 'b', 'b'], [1, 2, 1, 2]],
                  columns=[['Ohio', 'Ohio', 'Colorado'],
                           ['Green', 'Red', 'Green']])_____no_output_____frame_____no_output_____frame.index.names = ['key1', 'key2']
frame.columns.names = ['state', 'color']_____no_output_____frame_____no_output_____frame['Ohio', 'Green']_____no_output_____frame['Ohio']_____no_output_____frame['Ohio', 'Green']['a', 1]_____no_output_____frame_____no_output_____frame.swaplevel('key1', 'key2')_____no_output_____frame.sort_index(level=1)_____no_output_____frame.swaplevel(0, 1).sort_index(level=0)_____no_output_____frame.sum(level=0, axis=1)_____no_output_____frame.sum(level=0, axis=0)_____no_output_____frame.sum(level=1, axis=0)_____no_output_____frame = DataFrame({
    'a': range(7),
    'b': range(7, 0, -1),
    'c': ['one', 'one', 'one', 'two', 'two', 'two', 'two'],
    'd': [0, 1, 2, 0, 1, 2, 3]
})_____no_output_____frame_____no_output_____frame2 = frame.set_index(['c', 'd'])_____no_output_____frame2_____no_output_____frame.set_index(['c', 'd'], drop=False)_____no_output_____frame2.reset_index()_____no_output_____

### Read and Write
---_____no_output_____

df = pd.read_csv('ex.csv')_____no_output_____df_____no_output_____pd.read_table('ex.csv', sep=',')_____no_output_____pd.read_csv('ex.csv', header=None)_____no_output_____df.to_csv('ex.csv')_____no_output_____

### Combining and merging datasets
---_____no_output_____

df1 = DataFrame({
    'key': ['b', 'b', 'a', 'c', 'a', 'a', 'b'],
    'data1': range(7)
})_____no_output_____df2 = DataFrame({'key': ['a', 'b', 'd'], 'data2': range(3)})_____no_output_____df1_____no_output_____df2_____no_output_____

This is an example of a many-to-one merge situation; the data in df1 has multiple rows
labeled a and b, whereas df2 has only one row for each value in the key column._____no_output_____

pd.merge(df1, df2)_____no_output_____pd.merge(df1, df2, on='key')_____no_output_____df3 = DataFrame({
    'lkey': ['b', 'b', 'a', 'c', 'a', 'a', 'b'],
    'data1': range(7)
})_____no_output_____df4 = DataFrame({'rkey': ['a', 'b', 'd'], 'data2': range(3)})_____no_output_____df3_____no_output_____df4_____no_output_____pd.merge(df3, df4, left_on='lkey', right_on='rkey')_____no_output_____pd.merge(df1, df2, how='outer')_____no_output_____

There may also be many-to-many merges_____no_output_____

df1 = DataFrame({'key': ['b', 'b', 'a', 'c', 'a', 'b'], 'data1': range(6)})_____no_output_____df2 = DataFrame({'key': ['a', 'b', 'a', 'b', 'd'], 'data2': range(5)})_____no_output_____df1_____no_output_____df2_____no_output_____pd.merge(df1, df2, on='key', how='left')_____no_output_____pd.merge(df1, df2, on='key', how='right')_____no_output_____pd.merge(df1, df2, on='key', how='inner')_____no_output_____

To merge with multiple keys, pass a list of column names_____no_output_____

left = DataFrame({
    'key1': ['foo', 'foo', 'bar'],
    'key2': ['one', 'two', 'one'],
    'lval': [1, 2, 3]
})

right = DataFrame({
    'key1': ['foo', 'foo', 'bar', 'bar'],
    'key2': ['one', 'one', 'one', 'two'],
    'rval': [4, 5, 6, 7]
})_____no_output_____left_____no_output_____right_____no_output_____pd.merge(left, right, on=['key1', 'key2'], how='inner')_____no_output_____pd.merge(left, right, on=['key1', 'key2'], how='outer')_____no_output_____

When joining columns-on-columns, the indexes on the passed DataFrame objects are discarded._____no_output_____

pd.merge(left, right, on='key1')_____no_output_____pd.merge(left, right, on='key1', suffixes=('_left', '_right'))_____no_output_____left1 = DataFrame({'key': ['a', 'b', 'a', 'a', 'b', 'c'], 'value': range(6)})
right1 = DataFrame({'group_val': [3.5, 7]}, index=['a', 'b'])_____no_output_____left1_____no_output_____right1_____no_output_____

In some cases, the merge key or keys in a DataFrame will be found in its index._____no_output_____

pd.merge(left1, right1, left_on='key', right_index=True)_____no_output_____

With hierarchically-indexed data,_____no_output_____

lefth = DataFrame({
    'key1': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada'],
    'key2': [2000, 2001, 2002, 2001, 2002],
    'data': np.arange(5.)
})

righth = DataFrame(np.arange(12).reshape((6, 2)),
                   index=[['Nevada', 'Nevada', 'Ohio', 'Ohio', 'Ohio', 'Ohio'],
                          [2001, 2000, 2000, 2000, 2001, 2002]],
                   columns=['event1', 'event2'])_____no_output_____lefth_____no_output_____righth_____no_output_____pd.merge(lefth, righth, left_on=['key1', 'key2'], right_index=True)_____no_output_____pd.merge(lefth,
         righth,
         left_on=['key1', 'key2'],
         right_index=True,
         how='outer')_____no_output_____left2 = DataFrame([[1, 2], [3, 4], [5, 6]],
                  index=['a', 'c', 'e'],
                  columns=['Ohio', 'Nevada'])

right2 = DataFrame([[7, 8], [9, 10], [11, 12], [13, 14]],
                   index=['b', 'c', 'd', 'e'],
                   columns=['Missouri', 'Alabama'])_____no_output_____left2_____no_output_____right2_____no_output_____pd.merge(left2, right2, how='outer', left_index=True, right_index=True)_____no_output_____left2.join(right2, how='outer')_____no_output_____

### Concatenating Along an Axis
---_____no_output_____

arr = np.arange(12).reshape((3, 4))_____no_output_____arr_____no_output_____np.concatenate([arr, arr])_____no_output_____np.concatenate([arr, arr], axis=1)_____no_output_____s1 = Series([0, 1], index=['a', 'b'])
s2 = Series([2, 3, 4], index=['c', 'd', 'e'])
s3 = Series([5, 6], index=['f', 'g'])_____no_output_____pd.concat([s1, s2, s3])_____no_output_____pd.concat([s1, s2, s3], axis=1, sort=False)_____no_output_____s4 = pd.concat([s1 * 5, s3])_____no_output_____s4_____no_output_____pd.concat([s1, s4], axis=1, sort=False, join='inner')_____no_output_____pd.concat([s1, s4], axis=1, sort=False).reindex(index=['a', 'c', 'b', 'e'])_____no_output_____result = pd.concat([s1, s1, s3], keys=['one', 'two', 'three'])_____no_output_____result_____no_output_____result.unstack()_____no_output_____df1 = DataFrame(np.random.randn(3, 4), columns=['a', 'b', 'c', 'd'])
df2 = DataFrame(np.random.randn(2, 3), columns=['b', 'd', 'a'])_____no_output_____df1_____no_output_____df2_____no_output_____pd.concat([df1, df2], ignore_index=True, sort=False)_____no_output_____data = DataFrame(np.arange(6).reshape((2, 3)),
                 index=pd.Index(['Ohio', 'Colorado'], name='state'),
                 columns=pd.Index(['one', 'two', 'three'], name='number'))_____no_output_____data_____no_output_____data.stack()_____no_output_____data.unstack()_____no_output_____

### Data Transformation
---_____no_output_____

data = DataFrame({
    'k1': ['one'] * 3 + ['two'] * 4,
    'k2': [1, 1, 2, 3, 3, 4, 4]
})_____no_output_____data_____no_output_____data.duplicated()_____no_output_____data.drop_duplicates()_____no_output_____data = DataFrame({
    'food': [
        'bacon', 'pulled pork', 'bacon', 'Pastrami', 'corned beef', 'Bacon',
        'pastrami', 'honey ham', 'nova lox'
    ],
    'ounces': [4, 3, 12, 6, 7.5, 8, 3, 5, 6]
})_____no_output_____data_____no_output_____meat_to_animal = {
    'bacon': 'pig',
    'pulled pork': 'pig',
    'pastrami': 'cow',
    'corned beef': 'cow',
    'honey ham': 'pig',
    'nova lox': 'salmon'
}_____no_output_____data['food'].map(lambda x: meat_to_animal[x.lower()])_____no_output_____

### Discretization and Binning
---
Continuous data is often discretized or otherwised separated into “bins” for analysis._____no_output_____

ages = [20, 22, 25, 27, 21, 23, 37, 31, 61, 45, 41, 32]
bins = [18, 25, 35, 60, 100]_____no_output_____cats = pd.cut(ages, bins)_____no_output_____cats_____no_output_____pd.value_counts(cats)_____no_output_____

### Detecting and Filtering Outliers
---_____no_output_____

data = DataFrame(np.random.randn(1000, 4))_____no_output_____data.describe()_____no_output_____data[np.abs(data) > 3] = np.sign(data) * 3_____no_output_____data_____no_output_____

### Permutation and Random Sampling
---_____no_output_____

df = DataFrame(np.arange(5 * 4).reshape(5, 4))_____no_output_____sampler = np.random.permutation(5)_____no_output_____sampler_____no_output_____df_____no_output_____df.take(sampler)_____no_output_____df.take(np.random.permutation(len(df))[:3])_____no_output_____bag = np.array([5, 7, -1, 6, 4])
sampler = np.random.randint(0, len(bag), size=10)_____no_output_____sampler_____no_output_____draws = bag.take(sampler)_____no_output_____draws_____no_output_____

### Computing Indicator/Dummy Variables
---_____no_output_____

df = DataFrame({'key': ['b', 'b', 'a', 'c', 'a', 'b'], 'data1': range(6)})_____no_output_____pd.get_dummies(df['key'])_____no_output_____dummies = pd.get_dummies(df['key'], prefix='key')_____no_output_____df_with_dummy = df[['data1']].join(dummies)_____no_output_____df_with_dummy_____no_output_____

---_____no_output_____


# Notebook from mlovci/tmpnb_PAG2014
Path: Introduction.ipynb

flotilla — An open-source toolkit for single-cell RNA-seq data analysis

2015 Plant and Animal Genomes Conference

Michael T Lovci Ph.D.

@YeoLab

_____no_output_____

import css_styling_____no_output_____

Outline

  * Why single cells?

  * Goals of `flotilla`

  * Test-drive

  _____no_output_____
Single-cell Transcriptomes

Why you should care.

_____no_output_____  * Single cell transcriptomics stand to change the way biologists look at heterogeneity in gene expression and the types of questions they will want to ask.

    * How is gene expression varied and how does gene expression co-vary?

    * Are there cell sub-populations?

    * What genes create the largest diffence between two groups of cells?

_____no_output_____  * As computation and large-scale biology becomes more demanded, more than ever bioinformaticians will be called on to interpret and describe to the data that biologists create.

  * With this increased demand for bioinformaticians' time, there must be reliable and scalable solutions for processing and sharing data, analysis and methods._____no_output_____
Goals of `flotilla`

Data-driven conversations about _biology_
_____no_output_____  * Interactive data exploration.

  * Reduce the time from idea to answers.

  * "Let biologists do biology."

  * Reproducibility.

  * Transparency.

  * Fight data senescence (genomics data have long-term value).

  * Community-driven standard_____no_output_____
Applications of `flotilla`

Why is `flotilla` useful for bioinformatics experts?
_____no_output_____  * Live-coding in meetings and presentations.

  * Data sharing.

  * Replication in multiple computation environments._____no_output_____##Data Model

Object-oriented design makes adding new featues across mulitple datatypes easy.

_____no_output_____#
How to get `flotilla`

 There are several routes to obtaining `flotilla`.

 Depending on your interests pursue the following:_____no_output_____###Docker Hub

The `docker` Virtual Machine is re-built on Docker Hub with every change to `flotilla`. Start a container with `flotilla` installed and an iPython notebook running with:

    sudo docker run -v ${HOME}/flotilla_projects:/root/flotilla_projects \
                    -v ${HOME}/flotilla_notebooks:/root/home/ipython \
                    -it -P -p 8888 \
                    mlovci/flotilla:dev # ( or mlovci/flotilla:latest )

Where `-v` mounts host directories in the VM for read/write.

There is also a web-start tool for Mac OS X users that have already [installed docker](https://docs.docker.com/installation/#installation):

    curl https://raw.githubusercontent.com/\
         YeoLab/flotilla/dev/docker/start_docker.py |python

 _____no_output_____###PyPI

PyPI is updated with every versioned release of `flotilla`

    pip install flotilla
_____no_output_____###GitHub

New features are available here, community interaction (request new features, watch our progress), report bugs (of course there are none), contribute new features, browse code.

    git clone https://github.com/YeoLab/flotilla.git
    cd flotilla
    pip install .
_____no_output_____Now, proceed to the [examples](examples/) for demonstrations.

 ###**github.com/mlovci/tmpnb_pag2015**
 ###**hub.docker.com/u/mlovci/tmpnb-pag2015**

 1. [Demo](examples/PAG_Demo.ipynb)

 _____no_output_____


# Notebook from khiron/epithelial
Path: buildITP/.ipynb_checkpoints/OPPE-checkpoint.ipynb

<code>
import datetime
lipidname = "OPPE"
tail = "CCCC CDCC"
link = "G G"
head = "E P"

description = ";   A general model Phosphosatidylethanolamine (PE) lipid \n;      C18:1(9c) oleic acid, and C18:0 palmitic acid\n"
modeledOn=";   This topology follows the standard Martini 2.0 lipid definitions and building block rules.\n; Reference(s): \n;   S.J. Marrink, A.H. de Vries, A.E. Mark. Coarse grained model for semi-quantitative lipid simulations. JPC-B, 108:750-760, \n;   2004. doi:10.1021/jp036508g \n;   S.J. Marrink, H.J. Risselada, S. Yefimov, D.P. Tieleman, A.H. de Vries. The MARTINI force field: coarse grained model for \n;   biomolecular simulations. JPC-B, 111:7812-7824, 2007. doi:10.1021/jp071097f \n;   T.A. Wassenaar, H.I. Ingolfsson, R.A. Bockmann, D.P. Tieleman, S.J. Marrink. Computational lipidomics with insane: a versatile \n;   tool for generating custom membranes for molecular simulations. JCTC, 150410125128004, 2015. doi:10.1021/acs.jctc.5b00209\n; Created: "

now = datetime.datetime.now()
membrane="testmembrane"
insane="../insane+SF.py"
mdparams="../test.mdp"
martinipath="../martini.ff/"
ITPCatalogue="./epithelial.cat"
ITPMasterFile="martini_v2_epithelial.itp"

modeledOn+= now.strftime("%Y.%m.%d")+"\n"

# Cleaning up intermediate files from previous runs
!rm -f *#*
!rm -f *step*
!rm -f {membrane}*_____no_output_____import fileinput
import os.path

print("Create itp")
!python {martinipath}/lipid-martini-itp-v06.py -o {lipidname}.itp -alname {lipidname} -name {lipidname} -alhead '{head}' -allink '{link}' -altail '{tail}'

#update description and parameters
with fileinput.FileInput(lipidname+".itp", inplace=True) as file:
    for line in file:
        if line == ";   This is a ...\n":
            print(description, end='')
        elif line == ";   Was modeled on ...\n":
            print(modeledOn, end='')
        else:
            print(line, end='')

Create itp
The Martini lipid itp generator version 0.6  Args are: -o OPPE.itp -alname OPPE -alhead 'E P' -allink 'G G' -altail 'CCCC CDCC'
#Add this ITP file to the catalogue file
if not os.path.exists(ITPCatalogue):
    ITPCatalogueData = []
else:
    with open(ITPCatalogue, 'r') as file :
        ITPCatalogueData = file.read().splitlines()

    ITPCatalogueData = [x for x in ITPCatalogueData if not x==lipidname+".itp"]

ITPCatalogueData.append(lipidname+".itp")

with open(ITPCatalogue, 'w') as file :
    file.writelines("%s\n" % item for item in ITPCatalogueData)

#build ITPFile
with open(martinipath+ITPMasterFile, 'w') as masterfile:
    for ITPfilename in ITPCatalogueData:
        with open(ITPfilename, 'r') as ITPfile :
            for line in ITPfile:
                masterfile.write(line)

print("Done")Done
# build a simple membrane to visualize this species
!python2 {insane} -o {membrane}.gro -p {membrane}.top -d 0 -x 3 -y 3 -z 3 -sol PW -center -charge 0 -orient -u {lipidname}:1 -l {lipidname}:1 -itpPath {martinipath}; X: 3.000 (4 lipids) Y: 3.000 (4 lipids)
; 16 lipids in upper leaflet, 16 lipids in lower leaflet
; NDX Solute 1 0
; Charge of protein: 0.000000
; NDX Membrane 1 384
; Charge of membrane: 0.000000
; Total charge: 0.000000
; NDX Solvent 385 0
; NDX System 1 384
import os #Operating system specific commands
import re #Regular expression library

print("Test")
print("Grompp")
grompp = !gmx grompp -f {mdparams} -c {membrane}.gro -p {membrane}.top -o  {membrane}.tpr
success=True
for line in grompp:
    if re.search("ERROR", line):
        success=False
    if re.search("Fatal error", line):
        success=False
    #if not success:
    print(line)

if success:
    print("Run")
    !export GMX_MAXCONSTRWARN=-1
    !export GMX_SUPPRESS_DUMP=1
    run = !gmx mdrun -v -deffnm {membrane}
    summary=""
    logfile = membrane+".log"
    if not os.path.exists(logfile):
        print("no log file")
        print("== === ====")
        for line in run:
            print(line)
    else:
        try:
            file = open(logfile, "r")
            fe = False
            for line in file:
                if fe:
                    success=False
                    summary=line
                elif re.search("^Steepest Descents.*converge", line):
                    success=True
                    summary=line
                    break
                elif re.search("Fatal error", line):
                    fe = True
        except IOError as exc:
            sucess=False;
            summary=exc;
    if success:
        print("Success")
    else:
        print(summary)Test
Grompp
                      :-) GROMACS - gmx grompp, 2018.1 (-:

                            GROMACS is written by:
     Emile Apol      Rossen Apostolov      Paul Bauer     Herman J.C. Berendsen
    Par Bjelkmar    Aldert van Buuren   Rudi van Drunen     Anton Feenstra
  Gerrit Groenhof    Aleksei Iupinov   Christoph Junghans   Anca Hamuraru
 Vincent Hindriksen Dimitrios Karkoulis    Peter Kasson        Jiri Kraus
  Carsten Kutzner      Per Larsson      Justin A. Lemkul    Viveca Lindahl
  Magnus Lundborg   Pieter Meulenhoff    Erik Marklund      Teemu Murtola
    Szilard Pall       Sander Pronk      Roland Schulz     Alexey Shvetsov
   Michael Shirts     Alfons Sijbers     Peter Tieleman    Teemu Virolainen
 Christian Wennberg    Maarten Wolf
                           and the project leaders:
        Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel

Copyright (c) 1991-2000, University of Groningen, The Netherlands.
Copyright (c) 2001-2017, The GROMACS development team at
Uppsala University, Stockholm University and
the Royal Institute of Technology, Sweden.
check out http://www.gromacs.org for more information.

GROMACS is free software; you can redistribute it and/or modify it
under the terms of the GNU Lesser General Public License
as published by the Free Software Foundation; either version 2.1
of the License, or (at your option) any later version.

GROMACS:      gmx grompp, version 2018.1
Executable:   /usr/bin/gmx
Data prefix:  /usr
Working dir:  /home/richard/projects/epithelial/buildITP
Command line:
  gmx grompp -f ../test.mdp -c testmembrane.gro -p testmembrane.top -o testmembrane.tpr

Ignoring obsolete mdp entry 'title'
Setting the LD random seed to -1610087321
Generated 1 of the 741 non-bonded parameter combinations
Excluding 1 bonded neighbours molecule type 'OPPE'
Excluding 1 bonded neighbours molecule type 'OPPE'
Excluding 1 bonded neighbours molecule type 'PW'
Removing all charge groups because cutoff-scheme=Verlet
Number of degrees of freedom in T-Coupling group System is 1149.00

GROMACS reminds you: "Carbohydrates is all they groove" (Frank Zappa)

Analysing residue names:
There are:    32      Other residues
Analysing residues not classified as Protein/DNA/RNA/Water and splitting into groups...
This run will generate roughly 0 Mb of data
Run
Success

</code>