# Notebook from jemalicisou/Mineirando_dados
Path: me-data-analysis-1.ipynb

# Mentoria Evolution - Data Analysis_____no_output_____ Minerando Dados

www.minerandodados.com.br_____no_output_____**Importante**: Antes de executar as seguintes células verifique se **todos os arquivos** estão no mesmo diretório_____no_output_____**Importe o Pandas**_____no_output_____

import pandas as pd_____no_output_____

**Ler a base de dados em memória**_____no_output_____

dataset = pd.read_csv('kc_house_data.csv', sep=',')_____no_output_____

** Tipo: DataFrame**_____no_output_____

type(dataset)_____no_output_____

**Imprime informações do Dataframe**_____no_output_____

dataset.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 21613 entries, 0 to 21612
Data columns (total 21 columns):
id               21613 non-null int64
date             21613 non-null object
price            21613 non-null float64
bedrooms         21609 non-null float64
bathrooms        21613 non-null float64
sqft_living      21613 non-null int64
sqft_lot         21613 non-null int64
floors           21612 non-null float64
waterfront       21613 non-null int64
view             21613 non-null int64
condition        21613 non-null int64
grade            21613 non-null int64
sqft_above       21613 non-null int64
sqft_basement    21613 non-null int64
yr_built         21613 non-null int64
yr_renovated     21613 non-null int64
zipcode          21613 non-null int64
lat              21613 non-null float64
long             21613 non-null float64
sqft_living15    21613 non-null int64
sqft_lot15       21613 non-null int64
dtypes: float64(6), int64(14), object(1)
memory usage: 3.5+ MB

## Mapeamento SQL para Pandas_____no_output_____

dataset.head(10)_____no_output_____from IPython.display import Image
Image("tabela-sql-pandas.png")_____no_output_____

**Retorna todas os registros do dataframe**_____no_output_____

dataset_____no_output_____

**Retorna os top 10 registros** _____no_output_____

dataset.head(10)_____no_output_____

**Retorna os imóveis com 3 quartos**_____no_output_____

dataset.loc[dataset['bedrooms']==3]_____no_output_____

**Retorna imóveis únicos na base de dados**_____no_output_____

dataset.id.unique()_____no_output_____dataset.bedrooms.unique()_____no_output_____dataset.bathrooms.head(10)_____no_output_____dataset.bathrooms.mean()_____no_output_____

**Retorna a contagem de todos os registros por colunas**_____no_output_____

dataset.count()_____no_output_____

**Imprime o nome das colunas do dataframe**_____no_output_____

dataset.columns_____no_output_____

**Informações estatisticas da base de dados**_____no_output_____

dataset.describe()_____no_output_____

## Fazendo Querys no Dataframe_____no_output_____**Lista imóveis com 3 quartos e com banheiros maior igual a 2**_____no_output_____

dataset.loc[(dataset['bedrooms']==3) & (dataset['bathrooms'] > 2)]_____no_output_____

** Conta a quantidade de imóveis com 4 quartos**_____no_output_____

dataset[dataset['bedrooms']==4].count()_____no_output_____

** Ordena Dataframe pela coluna preço por ordem decrescente**_____no_output_____

dataset.sort_values(by='price', ascending=False)_____no_output_____

**Agrupa e conta quantidade de imóveis por tamanho de quartos**_____no_output_____

dataset.bedrooms.value_counts()_____no_output_____dataset.bathrooms.value_counts()_____no_output_____

## Consulta os dados em mais de um dataset_____no_output_____* Utiliza o método merge() para união dos dataframes
* União do tipo **inner join**, **left join** e **right join**
* Especifica a coluna chave para união_____no_output_____**Carregando o dataset de pedidos**_____no_output_____

orders = pd.read_csv('olist_orders_dataset.csv')_____no_output_____orders.head()_____no_output_____

- Carregando o dataset de itens pedidos_____no_output_____

orders_items = pd.read_csv('olist_order_items_dataset.csv')_____no_output_____orders_items.head()_____no_output_____

** Consultando os dados nos dois datasets e juntando através da chave order_id**
- Selecionando os atributos do dataset **orders (pedidos)**
> - order_id (id do pedido)
> - order_status (status do pedido)
> - order_approved_at (data e hora da aprovação do pedido)

- Selecionando os atributos do dataset **orders_items (itens do pedidos)**
> - product_id (id do produto)
> - seller_id (id do vendedor)
> - price (preço do produto)
> - freight_value (valor do frete)_____no_output_____

df_query = pd.merge(orders[['order_id','order_status','order_approved_at']],
                 orders_items[['order_id','product_id','seller_id','price','freight_value']],
                 on='order_id')_____no_output_____df_query.head()_____no_output_____

**Left Join**_____no_output_____

df_query = pd.merge(orders[['order_id','order_status','order_approved_at']],
                 orders_items[['order_id','product_id','seller_id','price','freight_value']],
                 on='order_id', how='left')_____no_output_____df_query.head()_____no_output_____

**Right Join**_____no_output_____

df_query = pd.merge(orders[['order_id','order_status','order_approved_at']],
                 orders_items[['order_id','product_id','seller_id','price','freight_value']],
                 , how='right',right_on='id_pedido' )_____no_output_____df_query.head()_____no_output_____

* Pratique o que foi aprendido refazendo todos os passos
* Consulte a documentação para aprender mais sobre os métodos e recursos utilizados.
* **Dúvidas?** Mande um e-mail para mim em contato@minerandodados.com.br_____no_output_____


# Notebook from Yunrou/introtodeeplearning
Path: lab1/Part1_TensorFlow.ipynb

      Visit MIT Deep Learning

        Run in Google Colab

        View Source on GitHub

# Copyright Information
_____no_output_____

# Copyright 2021 MIT 6.S191 Introduction to Deep Learning. All Rights Reserved.
#
# Licensed under the MIT License. You may not use this file except in compliance
# with the License. Use and/or modification of this code outside of 6.S191 must
# reference:
#
# © MIT 6.S191: Introduction to Deep Learning
# http://introtodeeplearning.com
#_____no_output_____

# Lab 1: Intro to TensorFlow and Music Generation with RNNs

In this lab, you'll get exposure to using TensorFlow and learn how it can be used for solving deep learning tasks. Go through the code and run each cell. Along the way, you'll encounter several ***TODO*** blocks -- follow the instructions to fill them out before running those cells and continuing.

# Part 1: Intro to TensorFlow

## 0.1 Install TensorFlow

TensorFlow is a software library extensively used in machine learning. Here we'll learn how computations are represented and how to define a simple neural network in TensorFlow. For all the labs in 6.S191 2021, we'll be using the latest version of TensorFlow, TensorFlow 2, which affords great flexibility and the ability to imperatively execute operations, just like in Python. You'll notice that TensorFlow 2 is quite similar to Python in its syntax and imperative execution. Let's install TensorFlow and a couple of dependencies.
_____no_output_____

%tensorflow_version 2.x
import tensorflow as tf

# Download and import the MIT 6.S191 package
!pip install mitdeeplearning
import mitdeeplearning as mdl

import numpy as np
import matplotlib.pyplot as pltCollecting mitdeeplearning
  Downloading https://files.pythonhosted.org/packages/9d/ad/650eb53c0d9d1213536fe94bc150f89b564ff5ee784bd662272584bb091b/mitdeeplearning-0.2.0.tar.gz (2.1MB)
     |████████████████████████████████| 2.1MB 5.4MB/s
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from mitdeeplearning) (1.19.5)
Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from mitdeeplearning) (2019.12.20)
Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from mitdeeplearning) (4.41.1)
Requirement already satisfied: gym in /usr/local/lib/python3.7/dist-packages (from mitdeeplearning) (0.17.3)
Requirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from gym->mitdeeplearning) (1.3.0)
Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from gym->mitdeeplearning) (1.5.0)
Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from gym->mitdeeplearning) (1.4.1)
Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym->mitdeeplearning) (0.16.0)
Building wheels for collected packages: mitdeeplearning
  Building wheel for mitdeeplearning (setup.py) ... done
  Created wheel for mitdeeplearning: filename=mitdeeplearning-0.2.0-cp37-none-any.whl size=2115442 sha256=d4adaa9b237117ae9da4634fbd210d6547284cdf7c3ca1c99bd8a654b0c911d7
  Stored in directory: /root/.cache/pip/wheels/af/dc/2a/5c3633135e7e4ef4fd31463cfa1942cb1bae7486ab94e7a2ad
Successfully built mitdeeplearning
Installing collected packages: mitdeeplearning
Successfully installed mitdeeplearning-0.2.0

## 1.1 Why is TensorFlow called TensorFlow?

TensorFlow is called 'TensorFlow' because it handles the flow (node/mathematical operation) of Tensors, which are data structures that you can think of as multi-dimensional arrays. Tensors are represented as n-dimensional arrays of base dataypes such as a string or integer -- they provide a way to generalize vectors and matrices to higher dimensions.

The ```shape``` of a Tensor defines its number of dimensions and the size of each dimension. The ```rank``` of a Tensor provides the number of dimensions (n-dimensions) -- you can also think of this as the Tensor's order or degree.

Let's first look at 0-d Tensors, of which a scalar is an example:_____no_output_____

sport = tf.constant("Tennis", tf.string)
number = tf.constant(1.41421356237, tf.float64)

print("`sport` is a {}-d Tensor".format(tf.rank(sport).numpy()))
print("`number` is a {}-d Tensor".format(tf.rank(number).numpy()))_____no_output_____

Vectors and lists can be used to create 1-d Tensors:_____no_output_____

sports = tf.constant(["Tennis", "Basketball"], tf.string)
numbers = tf.constant([3.141592, 1.414213, 2.71821], tf.float64)

print("`sports` is a {}-d Tensor with shape: {}".format(tf.rank(sports).numpy(), tf.shape(sports)))
print("`numbers` is a {}-d Tensor with shape: {}".format(tf.rank(numbers).numpy(), tf.shape(numbers)))_____no_output_____

Next we consider creating 2-d (i.e., matrices) and higher-rank Tensors. For examples, in future labs involving image processing and computer vision, we will use 4-d Tensors. Here the dimensions correspond to the number of example images in our batch, image height, image width, and the number of color channels._____no_output_____

### Defining higher-order Tensors ###

'''TODO: Define a 2-d Tensor'''
matrix = # TODO

assert isinstance(matrix, tf.Tensor), "matrix must be a tf Tensor object"
assert tf.rank(matrix).numpy() == 2_____no_output_____'''TODO: Define a 4-d Tensor.'''
# Use tf.zeros to initialize a 4-d Tensor of zeros with size 10 x 256 x 256 x 3.
#   You can think of this as 10 images where each image is RGB 256 x 256.
images = # TODO

assert isinstance(images, tf.Tensor), "matrix must be a tf Tensor object"
assert tf.rank(images).numpy() == 4, "matrix must be of rank 4"
assert tf.shape(images).numpy().tolist() == [10, 256, 256, 3], "matrix is incorrect shape"_____no_output_____

As you have seen, the ```shape``` of a Tensor provides the number of elements in each Tensor dimension. The ```shape``` is quite useful, and we'll use it often. You can also use slicing to access subtensors within a higher-rank Tensor:_____no_output_____

row_vector = matrix[1]
column_vector = matrix[:,2]
scalar = matrix[1, 2]

print("`row_vector`: {}".format(row_vector.numpy()))
print("`column_vector`: {}".format(column_vector.numpy()))
print("`scalar`: {}".format(scalar.numpy()))_____no_output_____

## 1.2 Computations on Tensors

A convenient way to think about and visualize computations in TensorFlow is in terms of graphs. We can define this graph in terms of Tensors, which hold data, and the mathematical operations that act on these Tensors in some order. Let's look at a simple example, and define this computation using TensorFlow:

![alt text](https://raw.githubusercontent.com/aamini/introtodeeplearning/master/lab1/img/add-graph.png)_____no_output_____

# Create the nodes in the graph, and initialize values
a = tf.constant(15)
b = tf.constant(61)

# Add them!
c1 = tf.add(a,b)
c2 = a + b # TensorFlow overrides the "+" operation so that it is able to act on Tensors
print(c1)
print(c2)_____no_output_____

Notice how we've created a computation graph consisting of TensorFlow operations, and how  the output is a Tensor with value 76 -- we've just created a computation graph consisting of operations, and it's executed them and given us back the result.

Now let's consider a slightly more complicated example:

![alt text](https://raw.githubusercontent.com/aamini/introtodeeplearning/master/lab1/img/computation-graph.png)

Here, we take two inputs, `a, b`, and compute an output `e`. Each node in the graph represents an operation that takes some input, does some computation, and passes its output to another node.

Let's define a simple function in TensorFlow to construct this computation function:_____no_output_____

### Defining Tensor computations ###

# Construct a simple computation function
def func(a,b):
  '''TODO: Define the operation for c, d, e (use tf.add, tf.subtract, tf.multiply).'''
  c = # TODO
  d = # TODO
  e = # TODO
  return e_____no_output_____

Now, we can call this function to execute the computation graph given some inputs `a,b`:_____no_output_____

# Consider example values for a,b
a, b = 1.5, 2.5
# Execute the computation
e_out = func(a,b)
print(e_out)_____no_output_____

Notice how our output is a Tensor with value defined by the output of the computation, and that the output has no shape as it is a single scalar value._____no_output_____## 1.3 Neural networks in TensorFlow
We can also define neural networks in TensorFlow. TensorFlow uses a high-level API called [Keras](https://www.tensorflow.org/guide/keras) that provides a powerful, intuitive framework for building and training deep learning models.

Let's first consider the example of a simple perceptron defined by just one dense layer: $ y = \sigma(Wx + b)$, where $W$ represents a matrix of weights, $b$ is a bias, $x$ is the input, $\sigma$ is the sigmoid activation function, and $y$ is the output. We can also visualize this operation using a graph:

![alt text](https://raw.githubusercontent.com/aamini/introtodeeplearning/master/lab1/img/computation-graph-2.png)

Tensors can flow through abstract types called [```Layers```](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer) -- the building blocks of neural networks. ```Layers``` implement common neural networks operations, and are used to update weights, compute losses, and define inter-layer connectivity. We will first define a ```Layer``` to implement the simple perceptron defined above._____no_output_____

### Defining a network Layer ###

# n_output_nodes: number of output nodes
# input_shape: shape of the input
# x: input to the layer

class OurDenseLayer(tf.keras.layers.Layer):
  def __init__(self, n_output_nodes):
    super(OurDenseLayer, self).__init__()
    self.n_output_nodes = n_output_nodes

  def build(self, input_shape):
    d = int(input_shape[-1])
    # Define and initialize parameters: a weight matrix W and bias b
    # Note that parameter initialization is random!
    self.W = self.add_weight("weight", shape=[d, self.n_output_nodes]) # note the dimensionality
    self.b = self.add_weight("bias", shape=[1, self.n_output_nodes]) # note the dimensionality

  def call(self, x):
    '''TODO: define the operation for z (hint: use tf.matmul)'''
    z = # TODO

    '''TODO: define the operation for out (hint: use tf.sigmoid)'''
    y = # TODO
    return y

# Since layer parameters are initialized randomly, we will set a random seed for reproducibility
tf.random.set_seed(1)
layer = OurDenseLayer(3)
layer.build((1,2))
x_input = tf.constant([[1,2.]], shape=(1,2))
y = layer.call(x_input)

# test the output!
print(y.numpy())
mdl.lab1.test_custom_dense_layer_output(y)_____no_output_____

Conveniently, TensorFlow has defined a number of ```Layers``` that are commonly used in neural networks, for example a [```Dense```](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense?version=stable). Now, instead of using a single ```Layer``` to define our simple neural network, we'll use the  [`Sequential`](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/Sequential) model from Keras and a single [`Dense` ](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/Dense) layer to define our network. With the `Sequential` API, you can readily create neural networks by stacking together layers like building blocks. _____no_output_____

### Defining a neural network using the Sequential API ###

# Import relevant packages
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense

# Define the number of outputs
n_output_nodes = 3

# First define the model
model = Sequential()

'''TODO: Define a dense (fully connected) layer to compute z'''
# Remember: dense layers are defined by the parameters W and b!
# You can read more about the initialization of W and b in the TF documentation :)
# https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense?version=stable
dense_layer = # TODO

# Add the dense layer to the model
model.add(dense_layer)
_____no_output_____

That's it! We've defined our model using the Sequential API. Now, we can test it out using an example input:_____no_output_____

# Test model with example input
x_input = tf.constant([[1,2.]], shape=(1,2))

'''TODO: feed input into the model and predict the output!'''
model_output = # TODO
print(model_output)_____no_output_____

In addition to defining models using the `Sequential` API, we can also define neural networks by directly subclassing the [`Model`](https://www.tensorflow.org/api_docs/python/tf/keras/Model?version=stable) class, which groups layers together to enable model training and inference. The `Model` class captures what we refer to as a "model" or as a "network". Using Subclassing, we can create a class for our model, and then define the forward pass through the network using the `call` function. Subclassing affords the flexibility to define custom layers, custom training loops, custom activation functions, and custom models. Let's define the same neural network as above now using Subclassing rather than the `Sequential` model._____no_output_____

### Defining a model using subclassing ###

from tensorflow.keras import Model
from tensorflow.keras.layers import Dense

class SubclassModel(tf.keras.Model):

  # In __init__, we define the Model's layers
  def __init__(self, n_output_nodes):
    super(SubclassModel, self).__init__()
    '''TODO: Our model consists of a single Dense layer. Define this layer.'''
    self.dense_layer = '''TODO: Dense Layer'''

  # In the call function, we define the Model's forward pass.
  def call(self, inputs):
    return self.dense_layer(inputs)_____no_output_____

Just like the model we built using the `Sequential` API, let's test out our `SubclassModel` using an example input.

_____no_output_____

n_output_nodes = 3
model = SubclassModel(n_output_nodes)

x_input = tf.constant([[1,2.]], shape=(1,2))

print(model.call(x_input))_____no_output_____

Importantly, Subclassing affords us a lot of flexibility to define custom models. For example, we can use boolean arguments in the `call` function to specify different network behaviors, for example different behaviors during training and inference. Let's suppose under some instances we want our network to simply output the input, without any perturbation. We define a boolean argument `isidentity` to control this behavior:_____no_output_____

### Defining a model using subclassing and specifying custom behavior ###

from tensorflow.keras import Model
from tensorflow.keras.layers import Dense

class IdentityModel(tf.keras.Model):

  # As before, in __init__ we define the Model's layers
  # Since our desired behavior involves the forward pass, this part is unchanged
  def __init__(self, n_output_nodes):
    super(IdentityModel, self).__init__()
    self.dense_layer = tf.keras.layers.Dense(n_output_nodes, activation='sigmoid')

  '''TODO: Implement the behavior where the network outputs the input, unchanged,
      under control of the isidentity argument.'''
  def call(self, inputs, isidentity=False):
    x = self.dense_layer(inputs)
    '''TODO: Implement identity behavior'''_____no_output_____

Let's test this behavior:_____no_output_____

n_output_nodes = 3
model = IdentityModel(n_output_nodes)

x_input = tf.constant([[1,2.]], shape=(1,2))
'''TODO: pass the input into the model and call with and without the input identity option.'''
out_activate = # TODO
out_identity = # TODO

print("Network output with activation: {}; network identity output: {}".format(out_activate.numpy(), out_identity.numpy()))_____no_output_____

Now that we have learned how to define `Layers` as well as neural networks in TensorFlow using both the `Sequential` and Subclassing APIs, we're ready to turn our attention to how to actually implement network training with backpropagation._____no_output_____## 1.4 Automatic differentiation in TensorFlow

[Automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation)
is one of the most important parts of TensorFlow and is the backbone of training with
[backpropagation](https://en.wikipedia.org/wiki/Backpropagation). We will use the TensorFlow GradientTape [`tf.GradientTape`](https://www.tensorflow.org/api_docs/python/tf/GradientTape?version=stable) to trace operations for computing gradients later.

When a forward pass is made through the network, all forward-pass operations get recorded to a "tape"; then, to compute the gradient, the tape is played backwards. By default, the tape is discarded after it is played backwards; this means that a particular `tf.GradientTape` can only
compute one gradient, and subsequent calls throw a runtime error. However, we can compute multiple gradients over the same computation by creating a ```persistent``` gradient tape.

First, we will look at how we can compute gradients using GradientTape and access them for computation. We define the simple function $ y = x^2$ and compute the gradient:_____no_output_____

### Gradient computation with GradientTape ###

# y = x^2
# Example: x = 3.0
x = tf.Variable(3.0)

# Initiate the gradient tape
with tf.GradientTape() as tape:
  # Define the function
  y = x * x
# Access the gradient -- derivative of y with respect to x
dy_dx = tape.gradient(y, x)

assert dy_dx.numpy() == 6.0_____no_output_____

In training neural networks, we use differentiation and stochastic gradient descent (SGD) to optimize a loss function. Now that we have a sense of how `GradientTape` can be used to compute and access derivatives, we will look at an example where we use automatic differentiation and SGD to find the minimum of $L=(x-x_f)^2$. Here $x_f$ is a variable for a desired value we are trying to optimize for; $L$ represents a loss that we are trying to  minimize. While we can clearly solve this problem analytically ($x_{min}=x_f$), considering how we can compute this using `GradientTape` sets us up nicely for future labs where we use gradient descent to optimize entire neural network losses._____no_output_____

### Function minimization with automatic differentiation and SGD ###

# Initialize a random value for our initial x
x = tf.Variable([tf.random.normal([1])])
print("Initializing x={}".format(x.numpy()))

learning_rate = 1e-2 # learning rate for SGD
history = []
# Define the target value
x_f = 4

# We will run SGD for a number of iterations. At each iteration, we compute the loss,
#   compute the derivative of the loss with respect to x, and perform the SGD update.
for i in range(500):
  with tf.GradientTape() as tape:
    '''TODO: define the loss as described above'''
    loss = # TODO

  # loss minimization using gradient tape
  grad = tape.gradient(loss, x) # compute the derivative of the loss with respect to x
  new_x = x - learning_rate*grad # sgd update
  x.assign(new_x) # update the value of x
  history.append(x.numpy()[0])

# Plot the evolution of x as we optimize towards x_f!
plt.plot(history)
plt.plot([0, 500],[x_f,x_f])
plt.legend(('Predicted', 'True'))
plt.xlabel('Iteration')
plt.ylabel('x value')_____no_output_____

`GradientTape` provides an extremely flexible framework for automatic differentiation. In order to back propagate errors through a neural network, we track forward passes on the Tape, use this information to determine the gradients, and then use these gradients for optimization using SGD._____no_output_____


# Notebook from saketkc/EE-546-project
Path: notebooks/22-MultiSpecies-SRP007412.ipynb

%pylab inline
%load_ext autoreload
%autoreload 2

from collections import Counter, defaultdict
import pandas as pd
import glob
import os
import itertools
from matplotlib.legend import Legend
from matplotlib.lines import Line2D
from gcca import GCCA
import logging
import seaborn as sns
sns.set_context('poster')#, font_scale=2)
sns.set_style('white')
from sklearn.decomposition import PCA
from sklearn.cross_decomposition import CCA
from ee546_utils import *
from scipy.stats import hypergeom
#MARKERS = list(Line2D.filled_markers)
MARKERS = ['o',  'x', '^', '*','>', '<', 's',  'H',  '8', 'p', 'h',  'D', 'd', 'P']
TISSUES_SHORT_FORM_MAP = {'lv': 'liver',
                          'skm': 'muscle',
                          'br': 'brain',
                          'ht': 'heart',
                          'kd': 'kidney',
                          'cb': 'cerebellum',
                          'lv': 'liver',
                          'ts': 'testes'
                         }

SPECIES_COMMON_NAME_MAP = {"bos_taurus": "Cow",
                           "rattus_norvegicus": "Rat",
                           "pongo_abelii": "Orangutan",
                           "monodelphis_domestica": "Opossum",
                           "macaca_mulatta": "Rhesus",
                           "pan_troglodytes": "Chimp",
                           "mus_musculus": "Mouse",
                           "homo_sapiens": "Human",
                           "gallus_gallus": "Chicken",
                           "ornithorhynchus_anatinus":  "Platypus",
                           "gorilla_gorilla": "Gorilla",
                           "pan_paniscus": "Bonobo"}

SPECIES_COMMON_NAME_MAP_SHORT = {"btaurus": "Cow",
                                 "rnorvegicus": "Rat",
                                 "pabelii": "Orangutan",
                                 "mdomestica": "Opposum",
                                 "mmulatta": "Rhesus",
                                 "ptroglodytes": "Chimp",
                                 "mmusculus": "Mouse",
                                 "hsapiens": "Human",
                                 "ggallus": "Chicken",
                                 "oanatinus":  "Platypus",
                                 "ggorilla": "Gorrila",
                                 "ppaniscus": "Bonobo"}
SRP007412_cols_to_use = [#'cb_M_1', 'lv_F_1', 'ts_M_1', 'target_id',
                         'kd_F_1', 'ht_M_1',
                         'ht_F_1', 'cb_F_1', 'br_M_1',
                         'br_F_1', 'kd_M_1', 'lv_M_1']

MARKERS = ['o',  'x', '^', '*', 'H', '<', 's',  '>', '8', 'p', 'h',  'D', 'd', 'P']
def species_short(name):
    return name[0] + name.split('_')[1]

def _get_PCA_colors_markers(target_names):
    colors = {}
    indexes = defaultdict(list)
    all_species = {}
    row_colors = []
    for index, target_name in enumerate(target_names):
        species = target_name.split('_')[0]
        tissue = target_name.split('_')[1]
        if tissue not in colors.keys():
            colors[tissue] = sns.color_palette('tab10')[len(colors)]
        row_colors.append(colors[tissue])
        indexes[tissue].append(index)
        all_species[index] = species

    colors_dict = {}
    colors_list = []
    indexes_x = defaultdict(list)
    indexes_y = defaultdict(list)
    markers_dict = {}
    markers_list = []
    for index, target_name in enumerate(target_names):
        species = target_name.split('_')[0]
        tissue = target_name.split('_')[1]
        if tissue not in colors_dict.keys():
            colors_dict[tissue] = sns.color_palette('tab10')[len(colors_dict)]
        if species not in markers_dict.keys():
            markers_dict[species] = MARKERS[len(markers_dict)]
        markers_list.append(markers_dict[species])
        colors_list.append(colors_dict[tissue])
        indexes[tissue].append(index)
    return indexes, markers_list, colors_list, colors_dict, markers_dict

def do_PCA(X, target_names, ax=None):
    sns.set_context('poster')#, font_scale=1.75)
    sns.set_style('white')
    pca = PCA(n_components=min(X.shape))
    X_r = pca.fit(X).transform(X)
    if ax is None:
        fig, ax = plt.subplots(figsize=(12, 12))
    else:
        fig = ax.get_figure()
    colors = sns.color_palette('Paired')
    lw = 2
    indexes, markers_list, colors_list, colors_dict, markers_dict = _get_PCA_colors_markers(target_names)

    for tissue, values in indexes.items():
        for value in values:
            ax.scatter(X_r[value, 0],
                       X_r[value, 1],
                       alpha=0.4,
                       color=colors_dict[tissue],
                       marker=markers_list[value],
                       lw=5)

    scatter_points = []
    annotations = []
    for tissue, color in colors_dict.items():
        p = ax.scatter([], [], color=color, label=tissue)
        if tissue in TISSUES_SHORT_FORM_MAP.keys():
            tissue = TISSUES_SHORT_FORM_MAP[tissue]
        annotations.append(tissue)
        scatter_points.append(p)

    leg = Legend(ax, scatter_points, annotations, title='Tissue',
                 bbox_to_anchor=(0., 1.02, 1., .02), loc=0,
                 fancybox=True, shadow=True,borderaxespad=0,
                 ncol=5, mode="expand")
    ax.add_artist(leg)

    scatter_points = []
    annotations = []
    for species, marker in markers_dict.items():
        p = ax.scatter([], [], marker=marker, color='black', label=species)
        annotations.append(SPECIES_COMMON_NAME_MAP_SHORT[species])
        scatter_points.append(p)

    #leg = Legend(ax, scatter_points, annotations, title='Species',
    #             bbox_to_anchor=(1, 1), loc='upper right',borderaxespad=0)
    leg = Legend(ax, scatter_points, annotations, title='Species', ncol=3,
             bbox_to_anchor=(0.5, -0.4), loc='lower center',)

    ax.add_artist(leg)

    #ax.set_title('PCA')
    ax.set_xlabel('PC1 ({:.2f} %)'.format(pca.explained_variance_ratio_[0]*100))
    ax.set_ylabel('PC2 ({:.2f} %)'.format(pca.explained_variance_ratio_[1]*100))
    #fig.tight_layout()

    return X_r, fig

def _get_CCA_colors_markers(target_names):
    colors_dict = {}
    colors_list = []
    indexes_x = defaultdict(list)
    indexes_y = defaultdict(list)
    markers_dict = {}
    markers_list = []
    for index, target_name in enumerate(target_names):
        species = target_name.split('_')[0]
        tissue = target_name.split('_')[1]
        if tissue not in colors_dict.keys():
            colors_dict[tissue] = sns.color_palette('tab10')[len(colors_dict)]
        if species not in markers_dict.keys():
            markers_dict[species] = MARKERS[len(markers_dict)]
        markers_list.append(markers_dict[species])
        colors_list.append(colors_dict[tissue])
        if index < X.shape[0]:
            indexes_x[tissue].append(index)
        else:
            indexes_y[tissue].append(index-X.shape[0])
    return indexes_x, indexes_y, markers_list, colors_list, colors_dict, markers_dict

def do_CCA(X, Y, target_names, ax=None):

    sns.set_context('poster')#, font_scale=1.7)
    sns.set_style('white')
    cca = CCA(n_components=2)
    cca.fit(X, Y)
    X_c, Y_c = cca.transform(X, Y)
    if ax is None:
        fig, ax = plt.subplots(figsize=(12, 12))
    else:
        fig = ax.get_figure()    #assert
    indexes_x, indexes_y, markers_list, colors_list, colors_dict, markers_dict = _get_CCA_colors_markers(target_names)
    for tissue, values in indexes_x.items():
        for value in values:
            ax.scatter(X_c[value, 0],
                       X_c[value, 1],
                       alpha=0.4,
                       color=colors_dict[tissue],
                       marker=markers_list[value],
                       lw=5)
    for tissue, values in indexes_y.items():
        for value in values:
            ax.scatter(Y_c[value, 0],
                       Y_c[value, 1],
                       alpha=0.4,
                       color=colors_dict[tissue],
                       marker=markers_list[value+X_c.shape[0]],
                       lw=5)
    scatter_points = []
    annotations = []
    for tissue, color in colors_dict.items():
        p = ax.scatter([], [], color=color, label=tissue)
        if tissue in TISSUES_SHORT_FORM_MAP.keys():
            tissue = TISSUES_SHORT_FORM_MAP[tissue]
        annotations.append(tissue)
        scatter_points.append(p)

    leg = Legend(ax, scatter_points, annotations, title='Tissue',
                 bbox_to_anchor=(0., 1.02, 1., .02), loc=0,
                 fancybox=True, shadow=True,borderaxespad=0,
                 ncol=3, mode="expand")
    ax.add_artist(leg)

    scatter_points = []
    annotations = []
    for species, marker in markers_dict.items():
        p = ax.scatter([], [], marker=marker, color='black', label=species)
        annotations.append(SPECIES_COMMON_NAME_MAP_SHORT[species])
        scatter_points.append(p)

    leg = Legend(ax, scatter_points, annotations, title='Species', ncol=3,
             bbox_to_anchor=(0.5, -0.35), loc='lower center',)
    ax.add_artist(leg)
    ax.set_xlabel('CC1')
    ax.set_ylabel('CC2')
    return fig

def normalize_df_rowwise(df):
    df_centered = df.sub(df.mean(axis=1), axis=0)
    df_centered = df_centered.div(df_centered.std(axis=1), axis=0)
    return df_centered

def normalize_df_colwise(df):
    df_centered = df.sub(df.mean(axis=0), axis=1)
    df_centered = df_centered.div(df_centered.std(axis=0), axis=1)
    return df_centeredPopulating the interactive namespace from numpy and matplotlib
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

# 01. Read Data_____no_output_____

ortho_pairwise = {}
for f in glob.glob('../ortholog_tsv/*.tsv'):
    if 'ortholog_one2one' in f:
        continue
    second_species = f.split('/')[-1].replace('.tsv', '').split('_')[-1]
    print(f)
    ortho_pairwise[f.split('/')[-1].replace('.tsv', '')] = pd.read_table(f)[['ensembl_gene_id', '{}_homolog_ensembl_gene'.format(second_species), '{}_homolog_orthology_type'.format(second_species)]].set_index('ensembl_gene_id')../ortholog_tsv/mmusculus_btaurus.tsv
../ortholog_tsv/mmusculus_ggallus.tsv
../ortholog_tsv/mmusculus_ggorilla.tsv
../ortholog_tsv/mmusculus_hsapiens.tsv
../ortholog_tsv/mmusculus_mdomestica.tsv
../ortholog_tsv/mmusculus_mmulatta.tsv
../ortholog_tsv/mmusculus_oanatinus.tsv
../ortholog_tsv/mmusculus_pabelii.tsv
../ortholog_tsv/mmusculus_ppaniscus.tsv
../ortholog_tsv/mmusculus_ptroglodytes.tsv
../ortholog_tsv/mmusculus_rnorvegicus.tsv
ONE2ONE_orthologs = pd.read_table('../ortholog_tsv/ortholog_one2one.tsv')_____no_output_____

# 02. Read kallisto tables SRP007412_____no_output_____

kallisto_counts = {}
srp_species =  defaultdict(list)
for f in glob.glob('../cross-species-data/SRP007412-*_kallisto_gene_tables.tsv'):
    sample_name = f.split('/')[-1].replace('_kallisto_gene_tables.tsv', '')
    srp, species = sample_name.split('-')
    species = species[0] + species.split('_')[-1]
    kallisto_counts['{}_{}'.format(srp, species)] = pd.read_table(f)
    srp_species[srp].append(species)_____no_output_____srp = 'SRP007412'
pairwise_kallisto_counts = {}
pairwise_kallisto_counts_normalized = {}
pairwise_kallisto_counts_normalized_individual = defaultdict(dict)
for species1, species2 in itertools.combinations(srp_species[srp], 2):
    # load counts
    counts1 = kallisto_counts['{}_{}'.format(srp, species1)]
    counts2 = kallisto_counts['{}_{}'.format(srp, species2)]

    if species1 ==  'mmusculus':
        species1, species2 = species2, species1
        counts1, counts2 = counts2, counts1
        #cols_to_use = counts1.columns.tolist()
        # Only use these columns in the second dataframe
        #counts2 = counts2.loc[:, cols_to_use]
    if species2 ==  'mmusculus':
        cols_to_use = counts2.columns.tolist()
        # Only use these columns in the second dataframe
        counts1 = counts1.loc[:, cols_to_use]
    intersecting_columns = set(counts1.columns.tolist()).intersection(set(counts2.columns.tolist()))
    intersecting_columns = list(sorted(intersecting_columns))

    counts1 = counts1.loc[:, intersecting_columns]
    counts2 = counts2.loc[:, intersecting_columns]

    if species2 != 'mmusculus':
        ortho1 = ortho_pairwise['mmusculus_{}'.format(species1)]
        ortho1 = ortho1.loc[ortho1['{}_homolog_orthology_type'.format(species1)]=='ortholog_one2one']
        ortho1 = ortho1.drop(columns=['{}_homolog_orthology_type'.format(species1)])

        ortho2 = ortho_pairwise['mmusculus_{}'.format(species2)]
        ortho2 = ortho2.loc[ortho2['{}_homolog_orthology_type'.format(species2)]=='ortholog_one2one']
        ortho2 = ortho2.drop(columns=['{}_homolog_orthology_type'.format(species2)])

        ortho_12 = ortho1.join(ortho2, how='outer').dropna().reset_index(drop=True)\
                    .set_index('{}_homolog_ensembl_gene'.format(species1))
        species2_lookup_col = '{}_homolog_ensembl_gene'.format(species2)
        species1_ortho_genes  = ortho_12.index
        species2_ortho_genes  = ortho_12['{}_homolog_ensembl_gene'.format(species2)]

        counts1_trimmed = counts1.loc[species1_ortho_genes].dropna()
        counts2_trimmed = counts2.loc[species2_ortho_genes].dropna()

        new_index_genes = counts1_trimmed.index
        reduced_set = ortho_12.loc[new_index_genes]
        reduced_set = reduced_set.loc[reduced_set[species2_lookup_col].isin(counts2_trimmed.index)]
        counts1_trimmed = counts1_trimmed.loc[reduced_set.index]
        counts2_trimmed = counts2_trimmed.loc[reduced_set[species2_lookup_col]]
        del ortho_12
        del reduced_set
        del ortho1
        del ortho2
    else:

        ortho = ortho_pairwise['mmusculus_{}'.format(species1)]
        ortho = ortho.loc[ortho['{}_homolog_orthology_type'.format(species1)]=='ortholog_one2one']
        ortho = ortho.drop(columns=['{}_homolog_orthology_type'.format(species1)])
        species2_lookup_col = '{}_homolog_ensembl_gene'.format(species1)

        species1_ortho_genes  = ortho['{}_homolog_ensembl_gene'.format(species1)]
        species2_ortho_genes  = ortho.index.tolist()

        counts1_trimmed = counts1.loc[species1_ortho_genes].dropna()
        counts2_trimmed = counts2.loc[species2_ortho_genes].dropna()

        new_index_genes = counts2_trimmed.index
        reduced_set = ortho.loc[new_index_genes]
        reduced_set = reduced_set.loc[reduced_set[species2_lookup_col].isin(counts1_trimmed.index)]
        counts1_trimmed = counts1_trimmed.loc[reduced_set[species2_lookup_col]]
        counts2_trimmed = counts2_trimmed.loc[reduced_set.index]
        del ortho
    if len(counts1.columns)!= len(counts2.columns):
        print('Skipping {}, {}'.format(species1, species2))
        continue
    assert (counts1.columns == counts2.columns).all()

    counts1_trimmed.columns =  ['{}_{}'.format(species1, x) for x in counts1_trimmed.columns]
    counts2_trimmed.columns =  ['{}_{}'.format(species2, x) for x in counts2_trimmed.columns]

    del counts1
    del counts2
    counts2_trimmed.index = counts1_trimmed.index.tolist()
    counts_concat = pd.concat([counts1_trimmed.T, counts2_trimmed.T], ignore_index=False)
    pairwise_kallisto_counts['{}_{}'.format(species1, species2)] = counts_concat

    counts_concat_centered = counts_concat.sub(counts_concat.mean(axis=1), axis=0)
    counts_concat_centered = counts_concat_centered.div(counts_concat_centered.std(axis=1), axis=0)

    pairwise_kallisto_counts_normalized['{}_{}'.format(species1, species2)] = counts_concat_centered
    pairwise_kallisto_counts_normalized_individual['{}_{}'.format(species1, species2)][species1] = counts_concat_centered.iloc[:len(counts1_trimmed.T.index),]
    pairwise_kallisto_counts_normalized_individual['{}_{}'.format(species1, species2)][species2] = counts_concat_centered.iloc[len(counts1_trimmed.T.index):,]/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/scanpy/lib/python3.6/site-packages/ipykernel/__main__.py:42: FutureWarning:
Passing list-likes to .loc or [] with any missing label will raise
KeyError in the future, you can use .reindex() as an alternative.

See the documentation here:
https://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike
/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/scanpy/lib/python3.6/site-packages/ipykernel/__main__.py:43: FutureWarning:
Passing list-likes to .loc or [] with any missing label will raise
KeyError in the future, you can use .reindex() as an alternative.

See the documentation here:
https://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike
/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/scanpy/lib/python3.6/site-packages/pandas/core/indexing.py:1472: FutureWarning:
Passing list-likes to .loc or [] with any missing label will raise
KeyError in the future, you can use .reindex() as an alternative.

See the documentation here:
https://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike
  return self._getitem_tuple(key)
/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/scanpy/lib/python3.6/site-packages/ipykernel/__main__.py:64: FutureWarning:
Passing list-likes to .loc or [] with any missing label will raise
KeyError in the future, you can use .reindex() as an alternative.

See the documentation here:
https://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike
/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/scanpy/lib/python3.6/site-packages/ipykernel/__main__.py:65: FutureWarning:
Passing list-likes to .loc or [] with any missing label will raise
KeyError in the future, you can use .reindex() as an alternative.

See the documentation here:
https://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike
for species_pair, counts in pairwise_kallisto_counts_normalized.items():
    X = counts.values
    target_names = counts.index
    X_r, fig = do_PCA(X, target_names)
    species1, species2 = species_pair.split('_')
    fig.savefig('../{}_plots/pca_normalized_{}_{}.pdf'.format(srp, species, species2),
                bbox_inches='tight')
    X = pairwise_kallisto_counts_normalized_individual[species_pair][species1]
    Y = pairwise_kallisto_counts_normalized_individual[species_pair][species2]
    fig = do_CCA(X, Y, target_names)
    fig.savefig('../{}_plots/cca_normalized_{}_{}.pdf'.format(srp, species, species2),
                bbox_inches='tight')
    X = X_r[:X.shape[0],]
    Y = X_r[X.shape[0]:,]
    fig = do_CCA(X, Y, target_names)
    fig.savefig('../{}_plots/cca_on_pca_{}_{}.pdf'.format(srp, species, species2),
                bbox_inches='tight')
_____no_output_____

## Load All orthologous genes at once_____no_output_____

species_wise_tissues = defaultdict(list)

for f in glob.glob('../cross-species-data/SRP007412-*_kallisto_gene_tables.tsv'):
    sample_name = f.split('/')[-1].replace('_kallisto_gene_tables.tsv', '')
    srp, species = sample_name.split('-')

    #species = species[0] + species.split('_')[-1]
    df = pd.read_table(f)
    tissues = sorted(set([x.split('_')[1] for x in df.columns]))
    for index, tissue in enumerate(tissues):
        if tissue in TISSUES_SHORT_FORM_MAP.keys():
            tissues[index] = TISSUES_SHORT_FORM_MAP[tissue]

    species_wise_tissues[species]+=list(tissues)
srp = 'SRP007412'
ortho_pairwise = {}
all_orthologs = pd.DataFrame()
for f in ['../ortholog_tsv/mmusculus_{}.tsv'.format(x) for x in map(lambda s: species_short(s), species_wise_tissues.keys())]:
    if 'mmusculus_mmusculus' in f:
        continue
    second_species = f.split('/')[-1].replace('.tsv', '').split('_')[-1]
    df = pd.read_table(f)[['ensembl_gene_id', '{}_homolog_ensembl_gene'.format(second_species), '{}_homolog_orthology_type'.format(second_species)]].set_index('ensembl_gene_id')
    df = df[df['{}_homolog_orthology_type'.format(second_species)]=='ortholog_one2one'].drop(columns=['{}_homolog_orthology_type'.format(second_species)])
    all_orthologs = all_orthologs.join(df, how='outer')_____no_output_____SPECIES_FOR_SRP = ['mus_musculus', 'gallus_gallus', 'ornithorhynchus_anatinus', 'macaca_mulatta', 'gorilla_gorilla', 'pan_paniscus', 'pan_troglodytes', 'pongo_abelii', 'homo_sapiens',  'monodelphis_domestica']
_____no_output_____all_orthologs_filtered = all_orthologs.copy()
for species in species_wise_tissues.keys():
    species = species_short(species)
    count = kallisto_counts['{}_{}'.format(srp, species)]
    if species == 'mmusculus':
        all_orthologs_filtered = all_orthologs_filtered[all_orthologs_filtered.index.isin(count.index)]
    else:
        all_orthologs_filtered = all_orthologs_filtered[all_orthologs_filtered['{}_homolog_ensembl_gene'.format(species)].isin(count.index)]

        _____no_output_____all_orthologs_filtered.shape_____no_output_____all_orthologs_filtered.head()_____no_output_____all_species_combined_counts_df = pd.DataFrame()
specie_wise_counts = []
specie_wise_index_tracker = []
specie_wise_target_names = {}
for index, species in enumerate(SPECIES_FOR_SRP):
    species = species_short(species)
    count = kallisto_counts['{}_{}'.format(srp, species)]
    col_key = '{}_homolog_ensembl_gene'.format(species)
    print(species)
    if index == 0:
        count = count.loc[all_orthologs_filtered.index, ]
        column_order = SRP007412_cols_to_use
        count = count.loc[:, column_order]
        count.columns = ['{}_{}'.format(species, x) for x in count.columns]
        all_species_combined_counts_df = count

    else:
        count = count.loc[all_orthologs_filtered[col_key], column_order]
        count.columns = ['{}_{}'.format(species, x) for x in count.columns]
        count.index = all_species_combined_counts_df.index.tolist()
        #assert (count.columns == column_order).all()
        all_species_combined_counts_df = all_species_combined_counts_df.join(count, how='outer')

    specie_wise_counts.append(count)
    specie_wise_target_names[species] =     count.columns.tolist()
    specie_wise_index_tracker.append(count.shape[1])mmusculus
ggallus
oanatinus
mmulatta
ggorilla
ppaniscus
ptroglodytes
pabelii
hsapiens
mdomestica
all_species_combined_counts_df_____no_output_____specie_wise_index_tracker_____no_output_____all_species_combined_counts_df_T = all_species_combined_counts_df.T_____no_output_____all_species_combined_counts_df_normalized = normalize_df_rowwise(all_species_combined_counts_df_T)_____no_output_____logging.root.setLevel(level=logging.INFO)
gcca = GCCA()

gcca.fit(*specie_wise_counts)

gcca.transform(*specie_wise_counts)

X = all_species_combined_counts_df_normalized.values
target_names = all_species_combined_counts_df_normalized.index
X_r, fig = do_PCA(X, target_names)
#fig.tight_layout()
fig.savefig('../SRP007412_plots/pca_normalized_all.pdf',
            bbox_inches='tight')2018-11-26 00:00:19,707 : gcca.gcca : INFO : data num is 10
2018-11-26 00:00:19,708 : gcca.gcca : INFO : data shape x_0: (6054, 8)
2018-11-26 00:00:19,708 : gcca.gcca : INFO : data shape x_1: (6054, 8)
2018-11-26 00:00:19,708 : gcca.gcca : INFO : data shape x_2: (6054, 8)
2018-11-26 00:00:19,709 : gcca.gcca : INFO : data shape x_3: (6054, 8)
2018-11-26 00:00:19,709 : gcca.gcca : INFO : data shape x_4: (6054, 8)
2018-11-26 00:00:19,709 : gcca.gcca : INFO : data shape x_5: (6054, 8)
2018-11-26 00:00:19,710 : gcca.gcca : INFO : data shape x_6: (6054, 8)
2018-11-26 00:00:19,710 : gcca.gcca : INFO : data shape x_7: (6054, 8)
2018-11-26 00:00:19,710 : gcca.gcca : INFO : data shape x_8: (6054, 8)
2018-11-26 00:00:19,711 : gcca.gcca : INFO : data shape x_9: (6054, 8)
2018-11-26 00:00:19,711 : gcca.gcca : INFO : normalizing
2018-11-26 00:00:19,731 : gcca.gcca : INFO : calc variance & covariance matrix
2018-11-26 00:00:19,747 : gcca.gcca : INFO : adding regularization term
2018-11-26 00:00:19,747 : gcca.gcca : INFO : calculating generalized eigenvalue problem ( A*u = (lambda)*B*u )
2018-11-26 00:00:19,749 : gcca.gcca : INFO : solving eig
2018-11-26 00:00:19,749 : gcca.gcca : INFO : left shape: (80, 80)
2018-11-26 00:00:19,750 : gcca.gcca : INFO : right shape: (80, 80)
2018-11-26 00:00:19,750 : gcca.gcca : INFO : calculating eigen dimension
2018-11-26 00:00:19,751 : gcca.gcca : INFO : eig_dim: 80
2018-11-26 00:00:19,752 : gcca.gcca : INFO : calculating eigenvalues & eigenvector
2018-11-26 00:00:19,758 : gcca.gcca : INFO : sorting eigenvalues & eigenvector
2018-11-26 00:00:19,759 : gcca.gcca : INFO : normalization
2018-11-26 00:00:19,760 : gcca.gcca : INFO : normalization
2018-11-26 00:00:19,760 : gcca.gcca : INFO : normalization
2018-11-26 00:00:19,761 : gcca.gcca : INFO : normalization
2018-11-26 00:00:19,761 : gcca.gcca : INFO : normalization
2018-11-26 00:00:19,761 : gcca.gcca : INFO : normalization
2018-11-26 00:00:19,762 : gcca.gcca : INFO : normalization
2018-11-26 00:00:19,762 : gcca.gcca : INFO : normalization
2018-11-26 00:00:19,763 : gcca.gcca : INFO : normalization
2018-11-26 00:00:19,763 : gcca.gcca : INFO : normalization
2018-11-26 00:00:19,764 : gcca.gcca : INFO : data num is 10
2018-11-26 00:00:19,764 : gcca.gcca : INFO : data shape x_0: (6054, 8)
2018-11-26 00:00:19,765 : gcca.gcca : INFO : data shape x_1: (6054, 8)
2018-11-26 00:00:19,765 : gcca.gcca : INFO : data shape x_2: (6054, 8)
2018-11-26 00:00:19,765 : gcca.gcca : INFO : data shape x_3: (6054, 8)
2018-11-26 00:00:19,766 : gcca.gcca : INFO : data shape x_4: (6054, 8)
2018-11-26 00:00:19,766 : gcca.gcca : INFO : data shape x_5: (6054, 8)
2018-11-26 00:00:19,766 : gcca.gcca : INFO : data shape x_6: (6054, 8)
2018-11-26 00:00:19,767 : gcca.gcca : INFO : data shape x_7: (6054, 8)
2018-11-26 00:00:19,767 : gcca.gcca : INFO : data shape x_8: (6054, 8)
2018-11-26 00:00:19,767 : gcca.gcca : INFO : data shape x_9: (6054, 8)
2018-11-26 00:00:19,768 : gcca.gcca : INFO : normalizing
2018-11-26 00:00:19,780 : gcca.gcca : INFO : transform matrices by GCCA
fig , ax  = plt.subplots(figsize=(12, 12))
X_r = gcca.z_list
target_names = all_species_combined_counts_df_normalized.index

indexes, markers_list, colors_list, colors_dict, markers_dict = _get_PCA_colors_markers(target_names)

scatter_points = []
annotations = []
for index, species in enumerate(SPECIES_FOR_SRP):
    target_names = specie_wise_target_names[species_short(species)]
    indexes, _, _, _ , _ = _get_PCA_colors_markers(target_names)
    z_list = X_r[index]
    for tissue, values in indexes.items():
        for value in values:
            ax.scatter(z_list[value, 0],
                       z_list[value, 1],
                       alpha=0.4,
                       color=colors_dict[tissue],
                       marker=markers_dict[species_short(species)],
                       lw=5)

target_names = all_species_combined_counts_df_normalized.index
indexes, markers_list, colors_list, colors_dict, markers_dict = _get_PCA_colors_markers(target_names)
for tissue, color in colors_dict.items():
    p = ax.scatter([], [], color=color, label=tissue)
    if tissue in TISSUES_SHORT_FORM_MAP.keys():
        tissue = TISSUES_SHORT_FORM_MAP[tissue]
    annotations.append(tissue)
    scatter_points.append(p)

leg = Legend(ax, scatter_points, annotations, title='Tissue',
             bbox_to_anchor=(0., 1.02, 1., .02), loc=0,
             fancybox=False, shadow=False,borderaxespad=0,
             ncol=5, mode="expand")
ax.add_artist(leg)

scatter_points = []
annotations = []
for species, marker in markers_dict.items():
    p = ax.scatter([], [], marker=marker, color='black', label=species)
    annotations.append(SPECIES_COMMON_NAME_MAP_SHORT[species])
    scatter_points.append(p)

#leg = Legend(ax, scatter_points, annotations, title='Species',
#             bbox_to_anchor=(1.04, 0), loc="lower left",)# frameon=False)
#leg = Legend(ax, scatter_points, annotations, title='Species',
#             bbox_to_anchor=(1, 1), loc='upper right',borderaxespad=0,
#             #bbox_to_anchor=(0.5, -0.05), loc=0, ncol=5,
#             )# frameon=False)
leg = Legend(ax, scatter_points, annotations, title='Species', ncol=3,
             bbox_to_anchor=(0.5, -0.4), loc='lower center',)
ax.add_artist(leg)
ax.set_xlabel('CC1')
ax.set_ylabel('CC2')
#fig.tight_layout()
fig.savefig('../SRP007412_plots/cca_normalized_all.pdf', bbox_inches='tight')
#fig.tight_layout(rect=[0,0,0.8,1])
_____no_output_____all_species_combined_counts_df_normalized.T_____no_output_____specie_wise_index_tracker_____no_output_____fig , ax  = plt.subplots(figsize=(12, 12))
gcca = GCCA()
PCA_transformed = PCA(n_components=min(all_species_combined_counts_df_normalized.shape)).fit(all_species_combined_counts_df_normalized).transform(all_species_combined_counts_df_normalized)

a = PCA_transformed[:specie_wise_index_tracker[0],:]
b = PCA_transformed[specie_wise_index_tracker[0]:sum(specie_wise_index_tracker[:2]), :]
c = PCA_transformed[sum(specie_wise_index_tracker[:2]):sum(specie_wise_index_tracker[:3]), :]
d = PCA_transformed[sum(specie_wise_index_tracker[:3]):sum(specie_wise_index_tracker[:4]), :]
e = PCA_transformed[sum(specie_wise_index_tracker[:4]):sum(specie_wise_index_tracker[:5]), :]
f = PCA_transformed[sum(specie_wise_index_tracker[:5]):sum(specie_wise_index_tracker[:6]), :]
g = PCA_transformed[sum(specie_wise_index_tracker[:6]):sum(specie_wise_index_tracker[:7]), :]
h = PCA_transformed[sum(specie_wise_index_tracker[:7]):sum(specie_wise_index_tracker[:8]), :]
i = PCA_transformed[sum(specie_wise_index_tracker[:8]):sum(specie_wise_index_tracker[:9]), :]
j = PCA_transformed[sum(specie_wise_index_tracker[:9]):sum(specie_wise_index_tracker[:10]), :]

gcca.fit(a,b,c,d,e,f,g,h,i,j)
gcca.transform(a,b,c,d,e,f,g,h,i,j)
X_r = gcca.z_list
target_names = all_species_combined_counts_df_normalized.index

indexes, markers_list, colors_list, colors_dict, markers_dict = _get_PCA_colors_markers(target_names)

scatter_points = []
annotations = []
for index, species in enumerate(SPECIES_FOR_SRP):
    target_names = specie_wise_target_names[species_short(species)]
    indexes, _, _, _ , _ = _get_PCA_colors_markers(target_names)
    z_list = X_r[index]
    for tissue, values in indexes.items():
        for value in values:
            ax.scatter(z_list[value, 0],
                       z_list[value, 1],
                       alpha=0.4,
                       color=colors_dict[tissue],
                       marker=markers_dict[species_short(species)],
                       lw=5)

target_names = all_species_combined_counts_df_normalized.index
indexes, markers_list, colors_list, colors_dict, markers_dict = _get_PCA_colors_markers(target_names)
for tissue, color in colors_dict.items():
    p = ax.scatter([], [], color=color, label=tissue)
    if tissue in TISSUES_SHORT_FORM_MAP.keys():
        tissue = TISSUES_SHORT_FORM_MAP[tissue]
    annotations.append(tissue)
    scatter_points.append(p)

leg = Legend(ax, scatter_points, annotations, title='Tissue',
             bbox_to_anchor=(0., 1.02, 1., .02), loc=0,
             fancybox=True, shadow=True,borderaxespad=0,
             ncol=5, mode="expand")
ax.add_artist(leg)

scatter_points = []
annotations = []
for species, marker in markers_dict.items():
    p = ax.scatter([], [], marker=marker, color='black', label=species)
    annotations.append(SPECIES_COMMON_NAME_MAP_SHORT[species])
    scatter_points.append(p)

#leg = Legend(ax, scatter_points, annotations, title='Species',
#             bbox_to_anchor=(1.04, 0), loc="lower left",)# frameon=False)
#leg = Legend(ax, scatter_points, annotations, title='Species',
#             bbox_to_anchor=(1, 1), loc='upper right',borderaxespad=0, shadow=False,
             #bbox_to_anchor=(0.5, -0.05), loc=0, ncol=5,
#            )# frameon=False)
leg = Legend(ax, scatter_points, annotations, title='Species', ncol=3,
             bbox_to_anchor=(0.5, -0.4), loc='lower center',)
ax.add_artist(leg)
ax.set_xlabel('CC1')
ax.set_ylabel('CC2')
fig.savefig('../SRP007412_plots/cca_on_pca_all.pdf',bbox_inches='tight')2018-11-25 23:27:56,205 : gcca.gcca : INFO : data num is 10
2018-11-25 23:27:56,206 : gcca.gcca : INFO : data shape x_0: (8, 80)
2018-11-25 23:27:56,206 : gcca.gcca : INFO : data shape x_1: (8, 80)
2018-11-25 23:27:56,206 : gcca.gcca : INFO : data shape x_2: (8, 80)
2018-11-25 23:27:56,208 : gcca.gcca : INFO : data shape x_3: (8, 80)
2018-11-25 23:27:56,208 : gcca.gcca : INFO : data shape x_4: (8, 80)
2018-11-25 23:27:56,209 : gcca.gcca : INFO : data shape x_5: (8, 80)
2018-11-25 23:27:56,209 : gcca.gcca : INFO : data shape x_6: (8, 80)
2018-11-25 23:27:56,210 : gcca.gcca : INFO : data shape x_7: (8, 80)
2018-11-25 23:27:56,210 : gcca.gcca : INFO : data shape x_8: (8, 80)
2018-11-25 23:27:56,210 : gcca.gcca : INFO : data shape x_9: (8, 80)
2018-11-25 23:27:56,211 : gcca.gcca : INFO : normalizing
2018-11-25 23:27:56,212 : gcca.gcca : INFO : calc variance & covariance matrix
2018-11-25 23:27:56,215 : gcca.gcca : INFO : adding regularization term
2018-11-25 23:27:56,216 : gcca.gcca : INFO : calculating generalized eigenvalue problem ( A*u = (lambda)*B*u )
2018-11-25 23:27:56,223 : gcca.gcca : INFO : solving eig
2018-11-25 23:27:56,224 : gcca.gcca : INFO : left shape: (800, 800)
2018-11-25 23:27:56,225 : gcca.gcca : INFO : right shape: (800, 800)
2018-11-25 23:27:56,225 : gcca.gcca : INFO : calculating eigen dimension
2018-11-25 23:27:56,516 : gcca.gcca : INFO : eig_dim: 70
2018-11-25 23:27:56,516 : gcca.gcca : INFO : calculating eigenvalues & eigenvector

# HO-SVD_____no_output_____

from collections import OrderedDict
common_tissue_counts = OrderedDict()
common_tissue_counts_normalized = OrderedDict()
for index, species in enumerate(SPECIES_FOR_SRP):
    species = species_short(species)
    count = kallisto_counts['{}_{}'.format(srp, species)]
    col_key = '{}_homolog_ensembl_gene'.format(species)
    print(species)
    column_order = SRP007412_cols_to_use
    if index == 0:
        count = count.loc[:, column_order]
    else:
        count = count.loc[:, column_order]

    count = count.loc[:, column_order]
    common_tissue_counts[species] = count
    common_tissue_counts_normalized[species] = normalize_df_colwise(count)mmusculus
ggallus
oanatinus
mmulatta
ggorilla
ppaniscus
ptroglodytes
pabelii
hsapiens
mdomestica
U, sigmas, V, eigen_values = ho_SVD(common_tissue_counts_normalized.values())_____no_output_____eigen_values_____no_output_____sigmas[1]_____no_output_____sigmas[0].shape_____no_output_____angles[0].shape_____no_output_____angles_____no_output_____angles = np.arctan2(sigmas[3], sigmas[4]) - np.pi/4
fig, ax = plt.subplots(figsize=(8,8))
sns.set_color_codes("muted")

sns.barplot(x=np.arange(1, len(angles)+1), y=angles, ax=ax, color="b")
ax.axhline(y=np.pi/8)
ax.axhline(y=-np.pi/8)
ax.set_ylim(-np.pi/4, np.pi/4)
_____no_output_____fig, ax = plt.subplots(figsize=(8,8))
sns.set_color_codes("muted")
sns.barplot(x=np.arange(1, len(eigen_values)+1), y=1/eigen_values, ax=ax, color="b")
ax.axhline(y=0.5)
_____no_output_____eigen_indexes = np.where(1/eigen_values >= 0.5)
eigen_indexes = eigen_indexes[0]
eigen_indexes =[5,6,7]_____no_output_____common_tissue_counts_normalized.keys()_____no_output_____# For these inidexes find the top 10%
# and bottom 10% genes in the projection
species_wise_genes = defaultdict(list)#OrderedDict()
species_wise_totalgenes = OrderedDict()
for species_index, species in enumerate(common_tissue_counts_normalized.keys()):
    if species == 'mmusculus':
        orthologous_genelist = all_orthologs_filtered.index.dropna().tolist()
    else:
        orthologous_genelist = all_orthologs_filtered['{}_homolog_ensembl_gene'.format(species)].dropna().tolist()

    for eigen_index in eigen_indexes:
        projection = np.dot(U[species_index], V)[:, eigen_index]
        total_genes = len(projection)
        #n_genes_to_select = int(0.1 * total_genes)
        n_genes_to_select = int(np.floor(0.5*len(orthologous_genelist)))
        bottom10_genes = np.argsort(projection)[:n_genes_to_select]
        top10_genes = np.argsort(projection)[::-1][:n_genes_to_select]
        counts =  kallisto_counts['{}_{}'.format(srp, species)]
        bottom10_genes = counts.iloc[bottom10_genes,:].index.tolist()
        top10_genes = counts.iloc[top10_genes,:].index.tolist()
        species_wise_genes[species] += bottom10_genes + top10_genes
    species_wise_genes[species] = set(species_wise_genes[species])
    species_wise_totalgenes[species] = len(counts.index)_____no_output_____from scipy.stats import hypergeom
for species, genelist in species_wise_genes.items():
    if species == 'mmusculus':
        genelist2 = all_orthologs_filtered.index.dropna().tolist()
        intersection = len(set(genelist).intersection(genelist2))
    else:
        genelist2 = all_orthologs_filtered['{}_homolog_ensembl_gene'.format(species)].dropna().tolist()
        intersection = len(set(genelist).intersection(genelist2))
    M = species_wise_totalgenes[species]
    n = len(genelist2)
    N = len(genelist)
    k = intersection
    p = hypergeom.sf(k, M, n, N)
    print(species, len(genelist), len(genelist2), intersection/len(all_orthologs_filtered.index), p)mmusculus 9580 6054 0.5378262305913446 0.0
ggallus 9411 6054 0.5393128510075983 7.529831236144314e-08
oanatinus 8732 6054 0.5412950115626033 8.12765343859351e-168
mmulatta 8334 6054 0.5518665345226297 1.8480527701955077e-202
ggorilla 9597 6054 0.49008919722497524 1.2247509040852862e-28
ppaniscus 9710 6054 0.44482986455236206 0.597386172133308
ptroglodytes 9492 6054 0.533201189296333 1.4279886249418067e-140
pabelii 8782 6054 0.5379914106375949 1.741796239467308e-126
hsapiens 8446 6054 0.4796828543111992 0.0
mdomestica 8343 6054 0.5622728774364056 2.266259994833407e-258
species_wise_totalgenes[species]_____no_output_____

# Hypergeometric test_____no_output_____#pmf(k, M, n, N) = choose(n, k) * choose(M - n, N - k) / choose(M, N),
#                               for max(0, N - (M-n)) <= k <= min(n, N)
# k, M, n, N,
# M = population size
# N = sample size
# k = number of successes in sample
# n = number of successes in population

{\displaystyle p_{X}(k)=\Pr(X=k)={\frac {{\binom {K}{k}}{\binom {N-K}{n-k}}}{\binom {N}{n}}},}
where

{\displaystyle N} N is the population size,
{\displaystyle K} K is the number of success states in the population,
{\displaystyle n} n is the number of draws (i.e. quantity drawn in each trial),
{\displaystyle k} k is the number of observed successes,

{\textstyle \textstyle {a \choose b}} {\textstyle \textstyle {a \choose b}} is a binomial coefficient.
prb = hypergeom.sf(_____no_output_____

U[1][:, 7]_____no_output_____# For these inidexes find the top 10%
# and bottom 10% genes in the projection
species_wise_genes = defaultdict(list)#OrderedDict()
species_wise_totalgenes = OrderedDict()
species_wise_new_counts = OrderedDict()
species_wise_new_counts_df = pd.DataFrame()
n_genes_to_select = 2000
minimum_common_n_genes = 9999999
species_wise_genes[species]
for species_index, species in enumerate(common_tissue_counts_normalized.keys()):
    if species == 'mmusculus':
        orthologous_genelist = all_orthologs_filtered.index.dropna().tolist()
    else:
        orthologous_genelist = all_orthologs_filtered['{}_homolog_ensembl_gene'.format(species)].dropna().tolist()

    for eigen_index in eigen_indexes:
        projection = np.dot(U[species_index], V)[:, eigen_index]
        total_genes = len(projection)
        bottom10_genes = np.argsort(projection)[:n_genes_to_select]
        top10_genes = np.argsort(projection)[::-1][:n_genes_to_select]
        counts =  kallisto_counts['{}_{}'.format(srp, species)]
        bottom10_genes = counts.iloc[bottom10_genes,:].index.tolist()
        top10_genes = counts.iloc[top10_genes,:].index.tolist()
        species_wise_genes[species] += bottom10_genes + top10_genes
    species_wise_genes[species] = set(species_wise_genes[species])
    species_wise_totalgenes[species] = len(counts.index)
    count = counts.loc[species_wise_genes[species], SRP007412_cols_to_use]
    count.columns = ['{}_{}'.format(species, x) for x in count.columns]
    species_wise_new_counts[species] = count
    if count.shape[0] < minimum_common_n_genes:
        minimum_common_n_genes= count.shape[0]_____no_output_____species_wise_new_counts_df = pd.DataFrame()
for species_index, species in enumerate(common_tissue_counts_normalized.keys()):
    count  = species_wise_new_counts[species].iloc[:minimum_common_n_genes,:]

    if species_index == 0:
        index_replacer = count.index
        species_wise_new_counts_df = count.T
    else:
        count.index = index_replacer

    species_wise_new_counts_df = pd.concat([species_wise_new_counts_df, count.T])
    _____no_output_____species_wise_new_counts_df_____no_output_____species_wise_new_counts_df_normalized = normalize_df_rowwise(species_wise_new_counts_df)_____no_output_____X = species_wise_new_counts_df_normalized.values
target_names = species_wise_new_counts_df_normalized.index
X_r, fig = do_PCA(X, target_names)
_____no_output_____fig , ax  = plt.subplots(figsize=(12, 12))

gcca = GCCA()
PCA_transformed = PCA(n_components=min(species_wise_new_counts_df_normalized.shape)).fit(species_wise_new_counts_df_normalized).transform(species_wise_new_counts_df_normalized)

a = PCA_transformed[:specie_wise_index_tracker[0],:]
b = PCA_transformed[specie_wise_index_tracker[0]:sum(specie_wise_index_tracker[:2]), :]
c = PCA_transformed[sum(specie_wise_index_tracker[:2]):sum(specie_wise_index_tracker[:3]), :]
d = PCA_transformed[sum(specie_wise_index_tracker[:3]):sum(specie_wise_index_tracker[:4]), :]
e = PCA_transformed[sum(specie_wise_index_tracker[:4]):sum(specie_wise_index_tracker[:5]), :]
f = PCA_transformed[sum(specie_wise_index_tracker[:5]):sum(specie_wise_index_tracker[:6]), :]
g = PCA_transformed[sum(specie_wise_index_tracker[:6]):sum(specie_wise_index_tracker[:7]), :]
h = PCA_transformed[sum(specie_wise_index_tracker[:7]):sum(specie_wise_index_tracker[:8]), :]
i = PCA_transformed[sum(specie_wise_index_tracker[:8]):sum(specie_wise_index_tracker[:9]), :]
j = PCA_transformed[sum(specie_wise_index_tracker[:9]):sum(specie_wise_index_tracker[:10]), :]

gcca.fit(a,b,c,d,e,f,g,h,i,j)
gcca.transform(a,b,c,d,e,f,g,h,i,j)
X_r = gcca.z_list
target_names = all_species_combined_counts_df_normalized.index

indexes, markers_list, colors_list, colors_dict, markers_dict = _get_PCA_colors_markers(target_names)

scatter_points = []
annotations = []
for index, species in enumerate(SPECIES_FOR_SRP):
    target_names = specie_wise_target_names[species_short(species)]
    indexes, _, _, _ , _ = _get_PCA_colors_markers(target_names)
    z_list = X_r[index]
    for tissue, values in indexes.items():
        for value in values:
            ax.scatter(z_list[value, 0],
                       z_list[value, 1],
                       alpha=0.4,
                       color=colors_dict[tissue],
                       marker=markers_dict[species_short(species)],
                       lw=5)

target_names = species_wise_new_counts_df_normalized.index
indexes, markers_list, colors_list, colors_dict, markers_dict = _get_PCA_colors_markers(target_names)
for tissue, color in colors_dict.items():
    p = ax.scatter([], [], color=color, label=tissue)
    if tissue in TISSUES_SHORT_FORM_MAP.keys():
        tissue = TISSUES_SHORT_FORM_MAP[tissue]
    annotations.append(tissue)
    scatter_points.append(p)

leg = Legend(ax, scatter_points, annotations, title='Tissue',
             bbox_to_anchor=(0., 1.02, 1., .02), loc=0,
             fancybox=True, shadow=True,borderaxespad=0,
             ncol=5, mode="expand")
ax.add_artist(leg)

scatter_points = []
annotations = []
for species, marker in markers_dict.items():
    p = ax.scatter([], [], marker=marker, color='black', label=species)
    annotations.append(SPECIES_COMMON_NAME_MAP_SHORT[species])
    scatter_points.append(p)

#leg = Legend(ax, scatter_points, annotations, title='Species',
#             bbox_to_anchor=(1.04, 0), loc="lower left",)# frameon=False)
#leg = Legend(ax, scatter_points, annotations, title='Species',
#             bbox_to_anchor=(1, 1), loc='upper right',borderaxespad=0, shadow=False,
             #bbox_to_anchor=(0.5, -0.05), loc=0, ncol=5,
#            )# frameon=False)
leg = Legend(ax, scatter_points, annotations, title='Species', ncol=3,
             bbox_to_anchor=(0.5, -0.4), loc='lower center',)
ax.add_artist(leg)
ax.set_xlabel('CC1')
ax.set_ylabel('CC2')
fig.savefig('../SRP007412_plots/cca_on_pca_shortlisted_genes.pdf',
            bbox_inches='tight')2018-11-27 02:16:14,296 : gcca.gcca : INFO : data num is 10
2018-11-27 02:16:14,297 : gcca.gcca : INFO : data shape x_0: (8, 88)
2018-11-27 02:16:14,300 : gcca.gcca : INFO : data shape x_1: (8, 88)
2018-11-27 02:16:14,300 : gcca.gcca : INFO : data shape x_2: (8, 88)
2018-11-27 02:16:14,303 : gcca.gcca : INFO : data shape x_3: (8, 88)
2018-11-27 02:16:14,304 : gcca.gcca : INFO : data shape x_4: (8, 88)
2018-11-27 02:16:14,306 : gcca.gcca : INFO : data shape x_5: (8, 88)
2018-11-27 02:16:14,307 : gcca.gcca : INFO : data shape x_6: (8, 88)
2018-11-27 02:16:14,309 : gcca.gcca : INFO : data shape x_7: (8, 88)
2018-11-27 02:16:14,310 : gcca.gcca : INFO : data shape x_8: (8, 88)
2018-11-27 02:16:14,313 : gcca.gcca : INFO : data shape x_9: (8, 88)
2018-11-27 02:16:14,313 : gcca.gcca : INFO : normalizing
2018-11-27 02:16:14,316 : gcca.gcca : INFO : calc variance & covariance matrix
2018-11-27 02:16:14,321 : gcca.gcca : INFO : adding regularization term
2018-11-27 02:16:14,323 : gcca.gcca : INFO : calculating generalized eigenvalue problem ( A*u = (lambda)*B*u )
2018-11-27 02:16:14,332 : gcca.gcca : INFO : solving eig
2018-11-27 02:16:14,332 : gcca.gcca : INFO : left shape: (880, 880)
2018-11-27 02:16:14,336 : gcca.gcca : INFO : right shape: (880, 880)
2018-11-27 02:16:14,336 : gcca.gcca : INFO : calculating eigen dimension
2018-11-27 02:16:14,849 : gcca.gcca : INFO : eig_dim: 70
2018-11-27 02:16:14,852 : gcca.gcca : INFO : calculating eigenvalues & eigenvector


# Notebook from daniel-koehn/Differential-equations-earth-system
Path: 05_Diffusion_1D/03_Heat_Equation_1D_periodic_heating.ipynb

###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c) 2019 Daniel Koehn, based on (c)2018 L.A. Barba, G.F. Forsyth [CFD Python](https://github.com/barbagroup/CFDPython#cfd-python), (c)2014 L.A. Barba, I. Hawke, B. Knaepen [Practical Numerical Methods with Python](https://github.com/numerical-mooc/numerical-mooc#practical-numerical-methods-with-python), also under CC-BY._____no_output_____

from IPython.core.display import HTML
css_file = '../style/custom.css'
HTML(open(css_file, 'r').read())_____no_output_____

# 1D Heat Conduction Example I: Diurnal periodic heating of a half-space_____no_output_____In the last notebook we dicussed different types of boundary conditions, like the Dirichlet condition (fixed values at the boundaries) or the Neumann condition (fixed gradient values at the boundaries). In this exercise we will apply a diurnal periodic heating as boundary condition to compute the temperature distribution in the near surface soil down to a depth of 1 m._____no_output_____## Diurnal periodic heating of a half-space

The surface temperature of the earth regularly changes over time due to the day-night cycle and seasonal changes.

In this notebook, we use the 1D heat conduction equation

$$
\begin{equation}
\frac{\partial T}{\partial t} = \alpha \frac{\partial^2 T}{\partial z^2} \notag
\end{equation}
$$

with the thermal diffusivity $\alpha$ and temperature $T$, to compute the influence of diurnal periodic heating of the surface

$$
\begin{equation}
T_{surf} = T_{avg} + \Delta T cos(\omega t) \tag{1}
\end{equation}
$$

on the temperature distribution in the near surface soil. Here, $T_{avg}$ denotes the average temperature over the day, $\Delta T$ the amplitude of the periodic temperature variation and the circular frequency $\omega$ related to the period $\tau$:

$$
\begin{equation}
\omega = \frac{2 \pi}{\tau} \notag
\end{equation}
$$

Let's take a deeper look into the problem._____no_output_____### Setting up the problem_____no_output_____Lets assume that we have an average surface temperature $T_{avg} = 10 ^oC$, temperature variations of $\Delta T = 5^oC$ and a diurnal period $\tau = 24\; h$. The half-space extends down to a depth of $L = 2\; m$ with an initial temperature of $T_{init} = 3\; ^oC$.

Additionally, the [thermal diffusivity](http://en.wikipedia.org/wiki/Thermal_diffusivity) of the mantle rock is $\alpha=1\times10^{-6} {\rm m}^2/{\rm s}$. As usual, start by importing some libraries and setting up the discretization._____no_output_____

import numpy
from matplotlib import pyplot
%matplotlib inline_____no_output_____# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16_____no_output_____

We'll begin by defining a few parameters ...

- depth of the half-space $L = 1\; \text{m}$
- a spatial grid with $nz = 100\; \text{points}$ in z-direction
- thermal diffusivity $\alpha = 1\times10^{-6}\; m^2/s$
- initial temperature of the half-space $T_{init} = 3^oC$
- average surface temperature $T_{avg} = 10 ^oC$
- temperature variations of $\Delta T = 5^oC$
- diurnal period $\tau = 24\; h$_____no_output_____

# Set parameters
L =         # DEFINE DEPTH OF HALF-SPACE HERE [m]!
nz =        # DEFINE NUMBER OF GRIDPOINTS NZ HERE!
dz = L / (nz - 1)  # spatial distance between two consecutive locations [m]
alpha =     #  DEFINE THERMAL DIFFUSIVITY OF ROCK HERE [m^2/s]!

# Define the depth locations [m]
z = numpy.linspace(0.0, L, num=nz)

# Set the initial temperature in the half-space
T_init =     # INITIAL TEMPERATURE OF THE HALF-SPACE [°C]
T0 = numpy.ones(nz)
T0 = T0 * T_init

# DEFINE PARAMETERS FOR PERIODIC DIURNAL BOUNDARY CONDITION AT THE SURFACE HERE!
T_avg =       # average temperature [°C]
DT =          # amplitude of periodic temperature variations [°C]
tau =         # period of periodic temperature variations [s]_____no_output_____

To solve the 1D heat conduction equation, we are using the forward-time, centered-space **FTCS** discretization. You should work it out on a piece of paper yourself (if you can't do it without looking it up, it means you need to do this more!).

$$
\begin{equation}
\frac{T_{i}^{n+1}-T_{i}^{n}}{\Delta t}=\alpha\frac{T_{i+1}^{n}-2T_{i}^{n}+T_{i-1}^{n}}{\Delta z^2} \notag
\end{equation}
$$

To obtain the temperature at the next time step, $T^{n+1}_i$, from the known information at the current time step, we compute

$$
\begin{equation}
T_{i}^{n+1}=T_{i}^{n}+\frac{\alpha\Delta t}{\Delta z^2}(T_{i+1}^{n}-2T_{i}^{n}+T_{i-1}^{n}) \notag
\end{equation}
$$

In the [first lesson of module 5](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/02_spacetime/02_03_1DDiffusion.ipynb), we discretized the diffusion equation  with a forward-time, centered-space scheme, subject to the following stability constraint:

$$
\begin{equation}
\alpha \frac{\Delta t}{(\Delta z)^2} \leq \frac{1}{2} \notag
\end{equation}
$$

##### Exercise 1

Modify the following **FCTS** FD code to implement the periodic boundary condition based on eq. (1) at the surface for given parameters $T_{avg},\;\Delta T,\;\tau$:_____no_output_____

def ftcs_cool_diurnal(T0, nt, nz, dt, dz, alpha, T_avg, DT, tau):
    """
    Computes and returns the depth-temperature distrubtion (geotherm)
    for the half-space cooling problem, according to a provided number
    of time steps, a given initial temperature and thermal diffusivity.
    The diffusion equation is integrated using forward differencing in
    time and central differencing in space.

    Parameters
    ----------
    T0 : numpy.ndarray
        The initial temperature as a 1D array of floats.
    nt : integer
        The number of time steps to compute.
    nz : integer
        The number of spatial grid points.
    dt : float
        The time-step size to integrate.
    dz : float
        The distance between two consecutive locations.
    alpha : float
        The thermal diffusivity of the rock.

    Returns
    -------
    Tstore : numpy.ndarray
        The temperature evolution [°C] as a 2D array of floats.
    time : numpy.ndarray
        time [s] as a 1D array of floats.
    """

    T = T0.copy()
    sigma = alpha * dt / dz**2

    # Initialize array to store T-model at each time step n ...
    Tstore = numpy.zeros([nz,nt])

    # ... and time
    time = numpy.zeros(nt)

    # DEFINE CIRCULAR FREQUENCY OMEGA HERE!
    omega =

    for n in range(nt): # loop over time steps

        Tn = T.copy()
        for i in range(1,nz-1): # loop over spatial grid

            T[i] = (Tn[i] + sigma * (Tn[i+1] - 2.0 * Tn[i] + Tn[i-1]))

        # APPLY DIURNAL PERIODIC BOUNDARY CONDITION EQ. (1) HERE!
        T[0] =

        # Store temperature model at time step n in 2D array Tstore ...
        Tstore[:,n] = T

        # ... and time
        time[n] = n * dt

    return Tstore, time_____no_output_____

We are all set to run! Lets compute the temperature evolution during one diurnal cycle of 24 hours. First, you have to define the maximum cool time **Tmax = 24 h**. To get correct results, do not forget to convert the units of Tmax from [h] to [s].

Next, we define a time step `dt` that satisfies the stability constraint and calculate the number of time steps `nt` for the given `Tmax` and `dt`. Compute the temperature evolution during 24 hours using the FD code `ftcs_cool_diurnal` _____no_output_____

# DEFINE MAXIMUM COOLING TIME in [h]
Tmax =

# CONVERT Tmax UNITS [h] -> [s] HERE!
Tmax =

# Set the time-step size based on CFL limit.
sigma = 0.5
dt = sigma * dz**2 / alpha  # time-step size
nt = (int)(Tmax/dt)         # number of time steps

# Compute the temperature profile in the lithosphere and asthenosphere
Tstore, time = ftcs_cool_diurnal(T0, nt, nz, dt, dz, alpha, T_avg, DT, tau)_____no_output_____

Next, we plot the temperature evolution as image plot ..._____no_output_____

# Plot the temperature evolution of the half-space after Tmax = 24 hours
pyplot.figure(figsize=(15.0, 5.))

pyplot.imshow(numpy.flipud(Tstore),extent=[0,Tmax/(60*60),0,L],aspect=10,cmap='magma')
cb = pyplot.colorbar()
pyplot.ylabel('Depth [m]')
pyplot.xlabel('Time [h]')
cb.set_label(label='Temperature [°C]',weight='bold')
pyplot.gca().invert_yaxis()_____no_output_____

... and as line plots at $\text{depth} = 1\; \text{cm}, 2\; \text{cm}, 5\; \text{cm}, 10\; \text{cm}, 20\; \text{cm}, 40\; \text{cm}, 80\; \text{cm}$, respectively:_____no_output_____

# Plot the temperature evolution of the half-space in 1 cm, 2 cm, 5 cm ,
# 10 cm, 20 cm, 40 cm and 80 cm, respectively.
pyplot.figure(figsize=(10.0, 6.0))
pyplot.xlabel('Time [h]')
pyplot.ylabel('Temperature [°C]')
pyplot.grid()

# Temperature evolution in 1 cm depth
n1cm = (int)(0.01/dz)
pyplot.plot(time/(60.*60.),Tstore[n1cm,:], color='C0', linestyle='-', linewidth=3, label='1 cm depth')

# Temperature evolution in 2 cm depth
n2cm = (int)(0.02/dz)
pyplot.plot(time/(60.*60.),Tstore[n2cm,:], color='C1', linestyle='-', linewidth=3, label='2 cm depth')

# Temperature evolution in 5 cm depth
n5cm = (int)(0.05/dz)
pyplot.plot(time/(60.*60.),Tstore[n5cm,:], color='C2', linestyle='-', linewidth=3, label='5 cm depth')

# Temperature evolution in 10 cm depth
n10cm = (int)(0.1/dz)
pyplot.plot(time/(60.*60.),Tstore[n10cm,:], color='C3', linestyle='-', linewidth=3, label='10 cm depth')

# Temperature evolution in 20 cm depth
n20cm = (int)(0.2/dz)
pyplot.plot(time/(60.*60.),Tstore[n20cm,:], color='C4', linestyle='-', linewidth=3, label='20 cm depth')

# Temperature evolution in 40 cm depth
n40cm = (int)(0.4/dz)
pyplot.plot(time/(60.*60.),Tstore[n40cm,:], color='C5', linestyle='-', linewidth=3, label='40 cm depth')

# Temperature evolution in 80 cm depth
n80cm = (int)(0.8/dz)
pyplot.plot(time/(60.*60.),Tstore[n80cm,:], color='C6', linestyle='-', linewidth=3, label='80 cm depth')

pyplot.legend()
pyplot.xlim(0,Tmax/(60.*60.))_____no_output_____

Describe and discuss the results._____no_output_____## What we learned:

- How to implement a periodic diurnal boundary condition at the surface
- Time evolution of the temperature field in the upper 1 m of the soil_____no_output_____


# Notebook from FadouaKhm/CSE-590---Spring-2021---Intro-to-ML-and-DL-Tutorials
Path: [Intro_to_ML]Spam_Classification.ipynb

_____no_output_____## Problem 2 - Classification_____no_output_____### Load Training and Testing Data_____no_output_____

import pandas as pd
import numpy as np_____no_output_____#Load training data
train_data = pd.read_csv('datasets/classification/spam_train.csv')
train_data.head()_____no_output_____#Load testing data
test_data = pd.read_csv('datasets/classification/spam_test.csv')
test_data.head()_____no_output_____print('Training data shape =',train_data.shape,'\n')
print('Testing data shape =',test_data.shape)Training data shape = (3450, 59)

Testing data shape = (1151, 59)
# Separate features and labels and remove row indices
#Traing data
X_train = train_data.iloc[:,1:-1].values
y_train = train_data.iloc[:,-1].values

#Testing data
X_test = test_data.iloc[:, 1:-1].values
y_test = test_data.iloc[:,-1].values_____no_output_____

### Explore the data Data_____no_output_____

X_train_____no_output_____y_train_____no_output_____X_train.shape, y_train.shape_____no_output_____y_train.shape_____no_output_____X_test_____no_output_____y_test_____no_output_____X_test.shape, y_test.shape_____no_output_____

# a- KNN Binary Classifier_____no_output_____

## Build and fit KNN Classifier
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=10)
knn.fit(X_train,y_train)_____no_output_____## Compute score (R^2) on both training and testing sets
print('Training score: ',knn.score(X_train,y_train))
print('Testing score: ', knn.score(X_test,y_test))Training score:  0.828695652173913
Testing score:  0.7706342311033884
## Track the evolution of performances for different K
def evaluate_knn(range_K, X_train,y_train, X_test,y_test):
    train_score = []
    test_score = []
    for k in range(1,range_K):
        knn = KNeighborsClassifier(n_neighbors=k)
        knn.fit(X_train,y_train)
        #R^2
        train_score.append(knn.score(X_train,y_train))
        test_score.append(knn.score(X_test,y_test))
    return train_score, test_score_____no_output_____# Evaluate for different Ks
range_K = 30
train_score, test_score= evaluate_knn(range_K, X_train,y_train, X_test,y_test)_____no_output_____# Plot accuracy evolution
import matplotlib.pyplot as plt
plt.figure(figsize=(12,5))
plt.plot(range(1, range_K), train_score,'<-', label='Training accuracy')
plt.plot(range(1, range_K), test_score,'*-', label='Testing accuracy')
plt.title('KNN - Training and testing scores')
plt.xlabel('Number of neighbors, K')
plt.ylabel('Accuracy')
plt.grid()
plt.legend()
plt.show()_____no_output_____

### Comments:
It can be noted that for K = 1, the model fits perfectly the training data by scoring 100%
accuracy. It was also able to generalize to the test data by scoring the best accuracy around 80%.
However, there is a risk that the model is overfitting the data for K=1.
As the value K of neighbours increases, the accuracy on the training set decreases relatively._____no_output_____# b- Logistic Regression classifier_____no_output_____

#Build and fit a logistic regression model
from sklearn.linear_model import LogisticRegression
LR = LogisticRegression()
LR.fit(X_train,y_train)_____no_output_____## Track the evolution of performances for different C
def evaluate_LR(range_C, X_train,y_train, X_test,y_test):
    train_score = []
    test_score = []
    for c in range_C:
        LR = LogisticRegression( C = c)
        LR.fit(X_train,y_train)
        #Accuracy
        train_score.append(LR.score(X_train,y_train))
        test_score.append(LR.score(X_test,y_test))
    return train_score, test_score_____no_output_____# Evaluate for different Cs
range_C = np.arange(0.001, 5, 0.5)
train_score, test_score= evaluate_LR(range_C, X_train,y_train, X_test,y_test)_____no_output_____# Plot accuracy evolution
import matplotlib.pyplot as plt
plt.figure(figsize=(15,5))
plt.plot(range(1, len(range_C)+1), train_score,'<-', label='Training accuracy')
plt.plot(range(1, len(range_C)+1), test_score,'*-', label='Testing accuracy')
plt.title('Logistic Regression - Training and testing scores')
plt.xticks([])
plt.xticks(range(1, len(range_C)+1),range_C.astype('float32'))
plt.xlabel('C')
plt.ylabel('Accuracy')
plt.grid()
plt.legend()
plt.show()_____no_output_____

### Comments:
- For low C values, the regularization is strong, ==> low accuracies on both the training and testing sets.
- For C=0.001, the model scores less than 89% on both the train set and the test set.

- As C increases (the regularization decreases), the performance of the model increases. The accuracy on the train set keeps on increasing slowly with C, while the performance on the test set reaches its maximum for C between 2 and 3  then stays almost constant.
- The optimal C value for the logistic regression classifier on this dataset can be between 2 and 3 where the model reaches 92.4% on the test set and around 93.6% on the train set._____no_output_____# c- SVM_____no_output_____

# Build and train SVM model
from sklearn import svm
SVM = svm.SVC()
SVM.fit(X_train,y_train)_____no_output_____## Track the evolution of performances for different C
def evaluate_SVM(range_C, X_train,y_train, X_test,y_test):
    train_score = []
    test_score = []
    for c in range_C:
        SVM = svm.SVC( C =c)
        SVM.fit(X_train,y_train)
        #Accuracy
        train_score.append(SVM.score(X_train,y_train))
        test_score.append(SVM.score(X_test,y_test))
    return train_score, test_score_____no_output_____# Evaluate for different Cs
range_C = np.arange(0.001, 6, 0.5)
train_score, test_score= evaluate_SVM(range_C, X_train,y_train, X_test,y_test)_____no_output_____# Plot accuracy evolution
import matplotlib.pyplot as plt
plt.figure(figsize=(15,5))
plt.plot(range(1, len(range_C)+1), train_score,'<-', label='Training accuracy')
plt.plot(range(1, len(range_C)+1), test_score,'*-', label='Testing accuracy')
plt.title('SVM - Training and testing scores')
plt.xticks([])
plt.xticks(range(1, len(range_C)+1),range_C.astype('float32'))
plt.xlabel('C')
plt.ylabel('Accuracy')
plt.grid()
plt.legend()
plt.show()_____no_output_____

### Comments:
- For low C values (strong regularization), the model reaches low accuracies on both the training and testing sets. For example, for C = 0.001, the model scores less than 65% on both the training and testing. Therefore, the model is underfitting the dataset.

- As C increases (the regularization decreases), the performance of the model increases. The accuracy on the train set keeps on increasing with C, while the performance on the test set reaches its maximum for C=2.5 then stays almost constant.

- There is a risk of overfitting for high C values (weak regularization).
_____no_output_____

_____no_output_____


# Notebook from Rezwanul-Haque/recommendation_system_projects
Path: Content-based-Recommenders/Metadata-Based-Recommender.ipynb

# Problem: IMDB Dataset Recommender
### Problem class: Metadata-Based recommender system
### Problem dataset link: https://bit.ly/33IAohl
### Problem description:
     Creating a Metadata based recommendation system using IMDB dataset.

### Problem Task:
     Recommend the top 10 most similar movies._____no_output_____# Importing libraries_____no_output_____

import warnings
warnings.filterwarnings('ignore')

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

import seaborn as sns
sns.set()_____no_output_____

# Load the dataset into a pandas dataframe_____no_output_____

# Import the original file
original_df = pd.read_csv("data/movies_metadata.csv", low_memory=False)

df.head()_____no_output_____# import data from the clean file
df = pd.read_csv("data/metadata_clean.csv", low_memory=False)

# Add the useful features into the cleaned dataframe
df['overview'], df['id'] = original_df['overview'], original_df['id']

credit_df = pd.read_csv("data/credits.csv", low_memory=False) # https://www.kaggle.com/rounakbanik/the-movies-dataset/downloads/credits.csv/7
keywords_df = pd.read_csv("data/keywords.csv", low_memory=False)

# show the head of the cleaned dataframe
df.head()_____no_output_____df.dtypes_____no_output_____# Convert the id of the df into int
# df['id'] = df['id'].astype('int') ## will arise value error_____no_output_____# Function to convert all non-integer IDs to NaN
def clean_ids(x):
    try:
        return int(x)
    except:
        return np.nan_____no_output_____# Clean the ids of df
df['id'] = df['id'].apply(clean_ids)

# filter all rows that have a null ID
df = df[df['id'].notnull()]_____no_output_____# Convert IDs into int
df['id'] = df['id'].astype('int')
keywords_df['id'] = keywords_df['id'].astype('int')
credit_df['id'] = credit_df['id'].astype('int')_____no_output_____# Merge Keywords and credits into the metadata df
df = df.merge(credit_df, on='id')
df = df.merge(keywords_df, on='id')

df.head()_____no_output_____

## Wrangling Keywords, cast and crew column_____no_output_____

# TODO:
    ## include only the top three keywords
    ## include the top three stars in our cast.
    ## Convert crew into director.extract only the director of the movie and ignore all other crew members.

# Convert the stringified objects into the native python objects
from ast import literal_eval

features = ['cast', 'crew', 'keywords', 'genres']
for feature in features:
    df[feature] = df[feature].apply(literal_eval)_____no_output_____# print the first cast member of the movie in df
df.iloc[0]['crew'][0]_____no_output_____# Extract the director's name. if director is not listed, return  NaN
def get_director(x):
    for crew_member in x:
        if crew_member['job'] == 'Director':
            return crew_member['name']
    return np.nan_____no_output_____# Define the new director feature
df['director'] = df['crew'].apply(get_director)

# Print the directors of the frist 5 movies
df['director'].head()_____no_output_____# Return the list of top 3 elements or entrie list; whichever is more.
def generate_list(x:list):
    if isinstance(x, list):
        names = [ele['name'] for ele in x]
        # Check if more than 3 elements exists. if yes, return only first three.
        if len(names) > 3:
            names = names[:3]
        return names
    # return empty list in case of missing/malformed data
    return []_____no_output_____# Apply the generate_list function to cast and keywords
df['cast'] = df['cast'].apply(generate_list)
df['keywords'] = df['keywords'].apply(generate_list)

# Only consider a maximum of 3 genres
df['genres'] = df['genres'].apply(lambda x: x[:3])_____no_output_____# Print the new features of the first 5 movies along with title
df[['title', 'cast', 'director', 'keywords', 'genres']].head(5)_____no_output_____# Building a vectorizer to build document vectors.
## Reason:
# If two actors had the same first name (say, Ryan Reynolds and Ryan Gosling), the vectorizer will treat
# both Ryans as the same, although they are clearly different entities. This will impact the quality of
# the recommendations we receive. If a person likes Ryan Reynolds' movies, it doesn't imply that they like
# movies by all Ryans.

# Function to sanitize data to prevent ambiguity
# Removes spaces and converts to lowercase
def sanitize(x):
    if isinstance(x, list):
        # strip spaces and convert to lowercase
        return [str.lower(i.replace(" ", '')) for i in x]
    else:
        # Check if director exists. If not, return empty string
        if isinstance(x, str):
            return str.lower(x.replace(" ", ""))
        else:
            return ''_____no_output_____# Apply the sanitize function to cast, keywords, director and genres
features = ['cast', 'director', 'keywords', 'genres']
for feature in features:
    df[feature] = df[feature].apply(sanitize)_____no_output_____

### Creating the metadata soup_____no_output_____

# Function that creates a soup out of the desired metadata
def create_soup(x):
    return ' '.join(x['keywords']) + ' ' + ' '.join(x['cast']) + ' ' + x['director'] + ' ' + ' '.join(x['genres'])_____no_output_____# Create the new soup feature
df['soup'] = df.apply(create_soup, axis=1)_____no_output_____# Display the soup of the first movie
df.iloc[0]['soup']_____no_output_____

# CountVectorizer_____no_output_____

# CountVectorizer is the simplest type of vectorizer
# Example:
#     we have three documents, A, B, and C, which are as follows:
#     A: The sun is a star.
#     B: My love is like a red, red rose
#     C: Mary had a little lamb

# step 1: convert these documents into their vector forms using CountVectorizer
# step 2: compute the size of the vocabulary. The vocabulary is the number of unique words present across all documents
#         The vocabulary for this set of three documents is as follows: the, sun, is, a, star, my, love, like,
#         red, rose, mary, had, little, lamb. Consequently, the size of the vocabulary is 14
# step 3: eliminating the stop words
#         V: like, little, lamb, love, mary, red, rose, sun, star
# The size of our vocabulary is now nine. Therefore, our documents will be represented as ninedimensional vectors,
# and each dimension here will represent the number of times a particular wordoccurs in a document

## Applying CountVectorizer in A B C
#     A: (0, 0, 0, 0, 0, 0, 0, 1, 1)
#     B: (1, 0, 0, 1, 0, 2, 1, 0, 0)
#     C: (0, 1, 1, 0, 1, 0, 0, 0, 0)_____no_output_____# Reason to use CountVectorizer
# Instead of using TF-IDFVectorizer, we will be using CountVectorizer. This is because using TFIDFVectorizer will accord less weight to actors and directors
# who have acted and directed in a relatively larger number of movies_____no_output_____# Define a new CountVectorizer object and create vectors for the soup
from sklearn.feature_extraction.text import CountVectorizer
count = CountVectorizer(stop_words='english')
count_matrix = count.fit_transform(df['soup'])_____no_output_____# Import cosine_similarity function
from sklearn.metrics.pairwise import cosine_similarity

# Compute the cosine similarity score (equivalent to dot product for tf-idf vactors)
cosine_sim = cosine_similarity(count_matrix, count_matrix)_____no_output_____# Reset index of the df and construct reverse mapping again
df = df.reset_index()
indices = pd.Series(df.index, index=df['title'])_____no_output_____# Function that takes in movie title as input and gives recommendations
def content_based_recommnder(title:str, cosine_sim:bool=cosine_similarity, df:pd.DataFrame=df, indices:pd.Series=indices):
    # Obtain the index of the movie that matches the title
    idx = indices[title]

    # Get the pairwise similarity scores of all movies with that movie
    # And convert it inot a list of tuples as described above
    similarity_scores = list(enumerate(cosine_similarity[idx]))

    # Sort the movies based on the cosine similarity scores
    similarity_scores = sorted(similarity_scores, key=lambda x: x[1], reverse=True)

    # Get the scores of the 10 most similar movies. Ignore the first movie.
    similarity_scores = similarity_scores[1:11]

    # Get the movie indices
    movie_indices = [i[0] for i in similarity_scores]

    # Return the top 10 most similar movies
    return df['title'].iloc[movie_indices]_____no_output_____#Get recommendations for The Lion King
content_based_recommnder('The Lion King')_____no_output_____