# Notebook from francois-rozet/info8010-deep-learning
Path: homeworks/homework2.ipynb

# INFO8010: Homework 2

In the previous homework, you learned how to program your first neural network starting from the very first principles of deep learning. If you managed to solve last assignment without any problems **congratulations!** If that was not the case **don't worry**, here's a second assignment for you which you can use to get better at deep learning.

In this homework we will see some slighly more complicated deep learning concepts: we will start by taking a look at some of PyTorch's functionalities that are necessary for training deep networks efficiently. We will then train our first neural networks for tackling different image classification tasks, learn to build custom datasets and explore how to train a CNN.

The strucutre of the notebook is identical to the one of the previous homework. Similarly to last time, you have to submit the notebook **with your solutions** to the exercises. When you encounter a `# your code` comment, you have to write some code yourself and you have to discuss the code/results when you see the instruction

> your discussion

Without further ado let's start by importing the libraries we will need throughout this assignment!_____no_output_____

import matplotlib.pyplot as plt
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
import torch.utils.data as data

from PIL import Image
from torchvision import datasets, transforms, utils_____no_output_____# As of 2022/02/23, the CIFAR10 dataset SSL certificate is outdated which prevents its download.
# The following deactivates the verification of the SSL certificates, but
# never reproduce this unless you absolutely trust the source.
import ssl
ssl._create_default_https_context = ssl._create_unverified_context_____no_output_____

## 1. Dataloaders

Today's first concept are PyTorch's dataloaders. As you have seen during the theoretical lectures, one of the main ingredients for successfully training deep learning models is data, **lots of data**.

As you can easily imagine, it is not possible to load datasets of millions of images into the memory of your machine. Furthermore, these images come in a form that does not make it possible to exploit the tensor operations we have seen in the previous assignment.

To deal with these issues (and many more of them) we can use [dataloaders](https://pytorch.org/docs/stable/data.html), a data loading utility that allows us to deal with large datasets efficiently. In what follows, you are given your first example of dataloader which will use the popular [CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset. _____no_output_____

transform = transforms.Compose([transforms.ToTensor()])
trainset = datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
testset = datasets.CIFAR10(root='./data', train=False, transform=transform)_____no_output_____

Let's explain what we just did. Thanks to PyTorch's [torchvision](https://pytorch.org/vision/stable/index.html) sub-library, we just downloaded the CIFAR10 dataset on our machine. The dataset was stored in the `./data` folder and comes in two different forms thanks to the use of the `train` flag: a version that can be used as training set, and a version that can be used as testing set. These two datasets are subclasses of `torch`'s `data.Dataset` class. We will see later what this `data.Dataset` class consists in exactly. Torchvision also allows us to define a set of image transformations which we have defined at the beginning of this cell: in this case we would like to convert our images to tensors, see the [documentation](https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.ToTensor) for an exact description of this transformation.

Now that we have defined which dataset we would like to use, and the form in which we would like to have our images, we can create our first data loader. Data loaders are objects over which you can iterate and that load, transform and return mini-batches of inputs/targets at each iteration. The advantage of data loaders is that they (can) perform pre-processing of the data in parallel, i.e. in several concurrent worker pools.

Here, we create two data loaders that return mini-batches of 4 elements at each iteration. When using stochastic gradient descent (SGD), the training data loader should shuffle the training dataset. _____no_output_____

trainloader = data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)
testloader = data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)_____no_output_____

Before training anything, let's take a look at the images we just downloaded._____no_output_____

classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

def show_images(img):
    plt.imshow(transforms.functional.to_pil_image(img))
    plt.show()

images, labels = next(iter(trainloader))
show_images(utils.make_grid(images))
print(*[classes[l] for l in labels])_____no_output_____

The `transforms` module comes also in as very handy for performing other type of data transformations: here's an example which transforms the CIFAR10 images into gray scaled images._____no_output_____

transform = transforms.Compose([transforms.Grayscale(), transforms.ToTensor()])
gray_scaled_trainset = datasets.CIFAR10(root='./data', train=True, transform=transform)
gray_scaled_trainloader = data.DataLoader(gray_scaled_trainset, batch_size=4, shuffle=True, num_workers=2)

images, labels = next(iter(gray_scaled_trainloader))
show_images(utils.make_grid(images))
print(*[classes[l] for l in labels])_____no_output_____

### 1.1 Transforms

Al remembered from the theoretical lectures that one way to make neural networks converge faster is to **normalize** the pixel values. He wrote the following code snippet to normalize his training set, but he encountered an error._____no_output_____

transform = transforms.Compose([
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
    transforms.ToTensor(),
])
bugged_trainset = datasets.CIFAR10(root='./data', train=True, transform=transform)
bugged_trainloader = data.DataLoader(bugged_trainset, batch_size=4, shuffle=True, num_workers=2)

images, labels = next(iter(bugged_trainloader))
show_images(utils.make_grid(images))  # should look weird due to normalization
print(*[classes[l] for l in labels])_____no_output_____

Fix his mistake._____no_output_____

# your code_____no_output_____

Al also remembers that, with image datasets, a common practice to increase the robustness of neural networks is **data augmentation**. He wants to apply random flips (vertical and horizontal) and random color changes to his training set, but he does not know how to. Could you help him?_____no_output_____

# your code_____no_output_____

### 1.2 Running operations on a GPU

As you may know, one important aspect of deep learning is that large models can be trained efficiently on specialized hardwares such as Graphical Processing Units (GPUs) or Tensorial Processing Units (TPUs). PyTorch allows you to perform operations on GPUs very easily by transferring the concerned models and/or tensors to GPUs.

However, to do so, you need a CUDA compatible GPU._____no_output_____

torch.cuda.is_available()_____no_output_____

If the result of the previous cell is `True`, everything is ready to run on the GPU and you can continue. Otherwise it means you do not have any GPU that is compatible with the `torch` version installed on your machine. In this case, we invite you to use [Google Colab](https://colab.research.google.com/) to do the rest of this homework. Do not forget to ask Colab for a GPU (in Runtime > Change runtime type > Hardware accelerator)._____no_output_____

device = 'cuda'_____no_output_____

Let's compare the speed of tensor operations on GPU and CPU. _____no_output_____

A = torch.randn(1000, 100000)
B = torch.randn(100000, 1)

# on CPU
%timeit A @ B_____no_output_____A = torch.randn((1000, 100000), device=device)
B = torch.randn((100000, 1), device=device)

# on GPU
%timeit A @ B_____no_output_____

Instead of directly creating a tensor on the GPU you may also transfer a model or a tensor on the GPU, for example we can transfer a simple MLP on the GPU and then back to the CPU as follows._____no_output_____

# create MLP on CPU
mlp = nn.Sequential(
    nn.Linear(3, 512),
    nn.ReLU(),
    nn.Linear(512, 512),
    nn.ReLU(),
    nn.Linear(512, 512),
    nn.ReLU(),
    nn.Linear(512, 1),
    nn.Sigmoid(),
)

# forward pass on CPU
x = torch.randn(256, 3)
%timeit mlp(x)

# transfer MLP to GPU (in-place)
mlp.to(device)

# forward pass on GPU
x = x.to(device)
%timeit mlp(x)

# release the GPU memory
mlp.to('cpu')
x = x.to('cpu')_____no_output_____

As you may notice, computations are much faster on the GPU. However, data transfer between GPU and CPU (and vice-versa) is usually very slow. We recommend to reduce the transfers of data between GPU and CPU as much as possible. For example when you want to save your loss after each iteration, in order to avoid a memory leak, you should prefer doing `.detach()` rather than `.cpu()` or `.item()`._____no_output_____## 2.  Classifying the CIFAR10 dataset with an MLP

Now that you know how to handle datasets, we are ready to properly train today's first deep learning model on the CIFAR10 dataset. Before we dive into it, **do not underestimate** the importance of properly pre-processing the data before training neural networks. This step is as important as defining the neural architectures themselves, but is very often overlooked.

In this exercise you are provided with an already defined multi-layer perceptron that you can train to classify CIFAR10 images. The structure of the network is already defined, yet some crucial hyperparameters are missing. It is your job to fill them in and successfully train the network. As part of the exercise, you are also required to monitor the evolution of training: this usually consists in checking how the training and testing losses evolve during training and keeping track of the model's accuracy on the testing set. Report these statistics with some plots. In addition, transfer the network and the mini-batches on GPU to speed up training.

Fill in the code below, discuss your choices and your results. Are you satisfied with the final accuracy?_____no_output_____

input_features =
output_features =
hidden_features = # your code
learning_rate =
num_epochs =

class MLP(nn.Sequential):
    def __init__(self, input_features, output_features, hidden_features):
        super().__init__(
            nn.Flatten(),
            nn.Linear(input_features, hidden_features),
            nn.ReLU(),
            nn.Linear(hidden_features, hidden_features),
            nn.ReLU(),
            nn.Linear(hidden_features, hidden_features),
            nn.ReLU(),
            nn.Linear(hidden_features, output_features),
        )

network = MLP(input_features, output_features, hidden_features)

# your code

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(network.parameters(), lr=learning_rate)
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
])

trainset = datasets.CIFAR10(root='./data', train=True, transform=transform)
testset = datasets.CIFAR10(root='./data', train=False, transform=transform)

trainloader = torch.utils.data.DataLoader(trainset, batch_size=100, shuffle=True, num_workers=2)
testloader = torch.utils.data.DataLoader(testset, batch_size=100, shuffle=False, num_workers=2)

def train(num_epochs):
    train_avg_loss = []
    test_avg_loss = []
    test_accuracy = []

    for i in range(num_epochs):
        train_losses = []
        test_losses = []

        for x, y in trainloader:
            # your code

            pred = network(x)
            loss = criterion(pred, y)
            train_losses.append(loss.detach())

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        with torch.no_grad():
            correct = 0

            for x, y in testloader:
                # your code

                pred = network(x)
                loss = criterion(pred, y)
                test_losses.append(loss)

                y_pred = pred.argmax(dim=-1)
                correct = correct + (y_pred == y).sum()

            accuracy = correct / len(testset)

        # your code

    return # your code_____no_output_____train_avg_loss, test_avg_loss, test_accuracy = train(num_epochs)_____no_output_____

Plot the statistics below and discuss your hyperparameter choices._____no_output_____

# your code_____no_output_____

> your discussion_____no_output_____## 3.  Create a custom dataset

Sometimes you would like to train a model on your own dataset, which will very likely not be part of `torchvision`. To overcome this you can create a custom dataset class which will handle the data for you. This can be done by inheriting from `torch`'s `data.Dataset` class and defining the methods `__len__` and `__getitem__` (see the [documentation](https://pytorch.org/docs/stable/data.htm)).

In this exercise your goal is to program a custom dataset class which you will later use for training a CNN. We will use the Kaggle Cats and Dogs dataset which you can download from [here](https://www.microsoft.com/en-us/download/details.aspx?id=54765).

When programming a custom dataset class, you have to start by defining the constructor, which will get as input the location of your dataset, whether the images that will be returned will serve for training or testing, and some other potential attributes. For this exercise we will be using 20000 images for training and 5000 images for testing. For the `__getitem__` function you may find the `PIL.Image.open` useful. Do not forget to transform the images into tensors and return the image labels as well ($0$ or $1$)._____no_output_____

# your code

class CatAndDogsDataset(data.Dataset):
    def __init__(self, root_dir, train=True):
        """Initializes a dataset containing images and labels."""
        super().__init__()
        # your code

    def __len__(self):
        """Returns the size of the dataset."""
        # your code

    def __getitem__(self, index):
        """Returns the index-th data item of the dataset."""
        # your code_____no_output_____

Let us have a quick look at these samples._____no_output_____

my_dataset = CatAndDogsDataset('kagglecatsanddogs_3367a/PetImages/', train=True)
my_loader = data.DataLoader(my_dataset, batch_size=4, shuffle=True, num_workers=2)

# your code_____no_output_____

## 4. Classifying the Cats and Dogs dataset with a CNN

As we have seen in class, classifying images with a multi-layer perceptron isn't really a good idea. Convolutional Neural Networks (CNN) are in fact a much better option for this task. It is now your job to create your custom CNN and train it on the Cats and Dogs Dataset.

Similarly to what you have done when classifying the CIFAR10 dataset you are again required to report and discuss the performance of your model._____no_output_____

# your code_____no_output_____

> your discussion_____no_output_____## Feedback

Now that you are done with this final deep-learning assignment here are some final questions about the exercises you were required to solve:_____no_output_____How much time did you spend on this homework?_____no_output_____Do you feel confortable with what it means to define a neural network and train it?_____no_output_____Do you think you now have enough preliminary knowledge for successfully starting to work on your course final project?_____no_output_____If you had to go through the two homeworks again, is there something you would have liked to explore more or explained more into detail?_____no_output_____


# Notebook from ventolab/HGDA
Path: TFs1_germcells.ipynb

# Germ lineage both sex - TFs intersection_____no_output_____

library(Seurat)
library(dplyr)
library(RColorBrewer)
library(destiny, quietly = TRUE)
library(gam)
library(harmony)
library(ggplot2)
library(pheatmap)
library(cowplot)
library(reshape2)
library(SoupX)
# source('G_lib_TFacts.r')

luz_colors = c('#c466b0',
               '#779eed', '#aad3f2', '#71a2c7', '#60bddb',   '#70ccbe',
               '#7b9e99', '#91bd80', '#bdb380',  '#d4db81',  '#dbc381', '#ebd3c3',
               '#e08b8b', '#edb7b7', '#d9abb7', '#d96884', '#e64e74', '#d9a5c3', '#cc8fdb')

colors = c(luz_colors,
           brewer.pal(n=12, name = 'Paired'),
           brewer.pal(n=8, name = 'Set2'),
          brewer.pal(n=8, name = 'Set3'),
          brewer.pal(n=8, name = 'Accent'),
          brewer.pal(n=8, name = 'Pastel1'))

def_colors = list( reds = luz_colors[13:length(luz_colors) ],
                 yellows = luz_colors[7:12],
                 blues = luz_colors[2:6],
                 greys = c('#e8e8e8', '#b5b5b5', '#a6a6a6') )

col_dot = c('grey90',  brewer.pal(n = 9, name = 'Reds')[7])

downsample_data = function(anndataO, var = 'annotated_clusters', N=200){
    samples = list()
    for (cl in unique(anndataO@meta.data[,var]) ){
        cells = colnames(anndataO)[ anndataO@meta.data[,var] == cl ]
        samples[[cl]] = cells
        if( length(cells) > N )
            samples[[cl]] = sample(cells, size = N, replace = F)
    }
    anndataO = subset(anndataO, cells = unlist(samples))

    gene_counts = tabulate(anndataO@assays$RNA@counts@i + 1)
    anndataO = subset(anndataO, features = names(gene_counts)[ gene_counts > 3 ] )

    return(anndataO)
}
_____no_output_____sessionInfo()_____no_output_____

# Load RNA derived TFs info_____no_output_____

TFact = read.csv('figures-germcells/0_FS1_table_TFacts.csv', stringsAsFactors = F)
# TFact$cluster[ TFact$cluster == 'Sertoli' ] = 'Sertoli_NR4A1'
TFact$TF = strsplit(TFact$gene, split = '-[A-E]') %>% sapply(., head, 1)
TFact$ID = paste0(TFact$cluster, '__', TFact$TF)
TFact$scRNA_TFact = ifelse(TFact$p_val_adj < 0.05 & TFact$avg_log2FC > 0.1 , 1, 0)

DEGs = read.csv('figures-germcells/0_FS1_table_DEGs.csv', stringsAsFactors = F)
# DEGs = subset(DEGs, gene %in% TFact$TF)
DEGs$TF = DEGs$gene
DEGs$ID = paste0(DEGs$cluster, '__', DEGs$TF)
# DEGs$avg_log2FC = DEGs$avg_logFC
DEGs$scRNA_DEGs = ifelse(DEGs$p_val_adj < 0.05 & DEGs$avg_log2FC > 0.1, 1, 0) _____no_output_____

# Load ATAC derived TFs info_____no_output_____

TFmotifs = read.csv('/nfs/team292/vl6/my_MULTIOME_dir/germcells_apr2021/TFs/DARs/all_DA_peaks_LR.csv', stringsAsFactors = F)

TFmotifs$cluster = gsub('-', '_', TFmotifs$cluster)
TFmotifs$cluster = gsub('oogonia_meiotic', 'oogonia_meiosis', TFmotifs$cluster)
TFmotifs$cluster = gsub('pre_spermatogonia', 'pre-spermatogonia', TFmotifs$cluster)
TFmotifs$TF = strsplit(TFmotifs$X, split = '\\.') %>% sapply(., head, 1)
TFmotifs$TF = gsub('-HUMAN', '', TFmotifs$TF)
TFmotifs = subset(TFmotifs, TF %in% c(TFact$TF, DEGs$TF) )
TFmotifs$ID = paste0(TFmotifs$cluster, '__', TFmotifs$TF)
# TFmotifs = subset(TFmotifs, TF %in% TFact$TF)
TFmotifs = TFmotifs[ order(TFmotifs$avg_log2FC, decreasing = T) , ]
TFmotifs = TFmotifs[ ! duplicated(TFmotifs$ID) , ]

TFmotifs$snATAC_TFact = ifelse(TFmotifs$p_val_adj < 0.05 & TFmotifs$avg_log2FC > 0.1 , 1, 0) _____no_output_____

# Intersect cluster names_____no_output_____

unique(TFmotifs$cluster)
unique(TFact$cluster)
unique(DEGs$cluster)

intersect(TFact$cluster,TFmotifs$cluster)_____no_output_____

## Merge for plotting_____no_output_____

cnames = intersect(colnames(TFact), colnames(DEGs)) %>% intersect(., colnames(TFmotifs))
df2plot = data.frame(TFact[,cnames], type='scRNA_TFactivities', stringsAsFactors = F)
df2plot = rbind(df2plot,
               data.frame(DEGs[, cnames], type='scRNA_Expression', stringsAsFactors = F))
df2plot = rbind(df2plot,
               data.frame(TFmotifs[, cnames], type='snATA_TFmotif_access', stringsAsFactors = F))_____no_output_____

# Intersect TFs_____no_output_____

names(TFact)[c(1:2,5)] = paste0('scRNA_TFact_', names(TFact)[c(1:2,5)])
names(DEGs)[c(1:5)] = paste0('scRNA_DEGs_', names(DEGs)[c(1:5)])
names(TFmotifs)[c(2:3,6)] = paste0('snATAC_TFact_', names(TFmotifs)[c(2:3,6)])
names(TFmotifs)[1] = 'snATAC_PWM' _____no_output_____df = merge(DEGs, TFact, by = 'ID', all = T)
df = df[, grep('y$', names(df), invert = T) ] # remove unwanted columns
df = merge(df, TFmotifs, by = 'ID', all = T)
df = df[, grep('y$', names(df), invert = T) ] # remove unwanted columns
df = df[, grep('x$', names(df), invert = T) ] # remove unwanted columns
head(df)_____no_output_____df = df[ , ! names(df) %in% c('L1' ) ]
df$cluster = strsplit(df$ID, '__') %>% sapply(., head, 1)
df$TF = strsplit(df$ID, '__') %>% sapply(., tail, 1)
df = df[, sort(names(df)) ]
df$N_agreement = apply(df[, c('scRNA_DEGs', 'scRNA_TFact', 'snATAC_TFact') ], 1, function(x){
    x[ is.na(x) ] = 0
    sum(x)
})
df = df[, -1]
head(df)_____no_output_____df = df[ order(df$N_agreement, decreasing = T) , ]
df = df[ , c(1,19, 20,  14,  3:(ncol(df)-2)) ]
write.csv(df, file = 'figures-germcells/0_FS1_table_TFs_intersection.csv', row.names=F)
head(df)_____no_output_____

# Plot - shared_____no_output_____

# Define clusters to plot
CLofinterest = c( 'PGC', 'PGC_mitotic',
                 'oogonia_STRA8',
                 'oogonia_meiosis', 'oocyte',  'pre-spermatogonia')

# and order accordingly
idx_cl = sapply(CLofinterest, function(x) which(df$cluster == x)) %>% unlist(.)
df =  df[idx_cl, ]

# Define clusters to plot
idx_tf = which(df$N_agreement == 3 |
               (df$N_agreement == 2 & df$scRNA_DEGs == 1 & df$scRNA_DEGs_avg_logFC > 0.5 ) |
               (df$N_agreement == 2 & df$scRNA_DEGs == 1 & df$snATAC_TFact_avg_log2FC > 1 ) |
               (df$N_agreement == 2 & df$scRNA_DEGs == 1 & df$scRNA_TFact_avg_log2FC > 1 )
              )
TFofinterest = unique(df[idx_tf, ]$TF  )
length(TFofinterest)
write.csv(df[idx_tf, ], file = 'figures-germcells/0_FS1_table_TFs_intersection_SELECTED.csv', row.names=F)
# TFofinterest = c('POU5F1',  'KLF4', 'TFAP2C', 'ETV5', 'NANOG',  'SOX15',  'SOX17',
#                  'ESX1', 'ZGLP1', 'RARA',  'HOXA2', 'HOXA4', 'HOXA5', 'HOXB3','RFX6',
#                  'DMRTB1', 'DMRTC2', 'HMGA1',
#                  'FIGLA', 'NOBOX', 'SOHLH1',   'SALL3', 'ZFPM2', 'TP63',
#                 'MSX2', 'HOXC10', 'HOXD11', 'HSF1', 'HSF2', 'SMAD1', 'SIX1')_____no_output_____

## as heatmaps_____no_output_____

# Load TF act Dorothea matrix
x_TFact = read.csv('figures-germcells/0_FS1_table_TFact_percluster.csv', stringsAsFactors = F, row.names = 1)
colnames(x_TFact) = strsplit(colnames(x_TFact), split = '\\.') %>% sapply(., head, 1)
x_TFact = x_TFact[ rownames(x_TFact) %in% CLofinterest , colnames(x_TFact) %in%  TFofinterest ]
rn = rownames(x_TFact)
x_TFact = apply(x_TFact, 2, scale)
rownames(x_TFact) = rn

# Add labels
x_TFact_Lab = x_TFact
x_TFact_Lab[] = ''
for (i in rownames(x_TFact_Lab) )
    for (j in colnames(x_TFact_Lab) ){
        x = subset(df, cluster == i & TF == j)
        if( nrow(x) == 1 )
             if( x[3] > 0 & ! is.na(x[12])){
                 if(x[12] == 1 )
                         x_TFact_Lab[i,j] = 'a'
                 if(x[3] == 3 )
                         x_TFact_Lab[i,j] = '*'
             }
#              if( x[3] > 1 & ! is.na(x[12])){
#                 if(x[3] == 3)
#                     x_TFact_Lab[i,j] = '*'
#                  if ( x[3] == 2 & x[12] == 1 ){
#                     if( ! is.na( x[17]))
#                         if(x[17] == 1)
#                          x_TFact_Lab[i,j] = 'o'
#                     if( ! is.na( x[6]))
#                         if(x[6] == 1 )
#                          x_TFact_Lab[i,j] = 'e'
#                     }

#             }
    }

# Load TFmotifs matrix
x_TFmotifs = read.csv('/nfs/team292/vl6/my_MULTIOME_dir/germcells_apr2021/germcells__chromvar_celltype_zscore.csv', stringsAsFactors = F, row.names = 1)
x_TFmotifs = x_TFmotifs[ rownames(x_TFmotifs) %in% CLofinterest , ]

idx = colnames(x_TFmotifs) %in% TFmotifs$X
x_TFmotifs = x_TFmotifs[ , idx ] # remove duplicated motifs to match TFmotifs
colnames(x_TFmotifs) = strsplit(colnames(x_TFmotifs), split = '\\.') %>% sapply(., head, 1)

# Load Expr matrix
x_Expre = read.csv('figures_manual_annotation/data_utils/average_log.csv', stringsAsFactors = F, row.names = 1) %>% t(.)
rownames(x_Expre) = gsub('pre.spermatogonia', 'pre-spermatogonia', rownames(x_Expre))
x_Expre = x_Expre[ rownames(x_Expre) %in% CLofinterest ,  colnames(x_Expre) %in%  TFofinterest  ]
rn = rownames(x_Expre)
x_Expre = apply(x_Expre, 2, scale)
rownames(x_Expre) = rn

# Add labels
x_Expre_Lab = x_Expre
x_Expre_Lab[] = ''
for (i in rownames(x_Expre_Lab) )
    for (j in colnames(x_Expre_Lab) ){
        x = subset(df, cluster == i & TF == j)
        if( nrow(x) == 1 )
             if( x[3] > 1 & ! is.na(x[6]) ){
                if(x[3] == 3)
                    x_Expre_Lab[i,j] = '*'
                 if ( x[3] == 2 & x[6] == 1 ){
                    if( ! is.na( x[17]))
                        if(x[17] == 1)
                         x_Expre_Lab[i,j] = 'o'
                    if( ! is.na( x[12]))
                        if(x[12] == 1 )
                         x_Expre_Lab[i,j] = 'a'
                    }

            }
    }
_____no_output_____myHeatmap = function(X, TFs, cl, pdf_file, display_numbers = NULL){
    print(cl)
    tfs = intersect(TFs, colnames(X))
    cls =  intersect(cl, rownames(X))
    X = X[ cls,  tfs ]
    print(dim(X))

    paletteLength <- 50
    myColor <- colorRampPalette(c(brewer.pal(n = 5, name = 'Blues')[4], "white", brewer.pal(n = 5, name = 'Reds')[4]))(paletteLength)
    # length(breaks) == length(paletteLength) + 1
    # use floor and ceiling to deal with even/odd length pallettelengths
    myBreaks <- c(seq(min(X), 0, length.out=ceiling(paletteLength/2) + 1),
                  seq(max(X)/paletteLength, max(X), length.out=floor(paletteLength/2)))
    if( is.null(display_numbers) ){
        pheatmap(X, cellheight = 10, cellwidth = 10, cluster_rows = F, cluster_cols = F,
                 color=myColor, breaks=myBreaks,
                filename = pdf_file)
    }else{
        display_numbers = display_numbers[ cls,  tfs ]
        print(dim(display_numbers))
        pheatmap(X, cellheight = 10, cellwidth = 10, cluster_rows = F, cluster_cols = F, display_numbers = display_numbers,
                 color=myColor, breaks=myBreaks,
                filename = pdf_file)
        }
}
myHeatmap(x_TFact, TFofinterest, CLofinterest, 'figures-germcells/0_FS1_heatmap_TFintersection_scRNA_TFact.pdf')
myHeatmap(x_Expre, TFofinterest, CLofinterest, 'figures-germcells/0_FS1_heatmap_TFintersection_scRNA_DEGs.pdf', display_numbers = x_Expre_Lab)
# myHeatmap(x_TFmotifs, TFofinterest, CLofinterest, 'figures-germcells/0_FS1_heatmap_TFintersection_snATAC_TFact.pdf')[1] "PGC"               "PGC_mitotic"       "oogonia_STRA8"
[4] "oogonia_meiosis"   "oocyte"            "pre-spermatogonia"
[1]    6 1376
[1] "PGC"               "PGC_mitotic"       "oogonia_STRA8"
[4] "oogonia_meiosis"   "oocyte"            "pre-spermatogonia"
[1]     6 14791
# plot cellphoneDB TFs
cellphoneTFs = c('ELK1', 'STAT1', 'STAT3', 'SMAD1', 'SMAD5', 'ZGLP1', 'RXRA', 'ESR2')
CLofinterest = c( 'PGC',
                 'oogonia_STRA8',
                 'oogonia_meiosis', 'oocyte')
#                  ,  'pre-spermatogonia')
myHeatmap(x_TFact, cellphoneTFs,
          rev(CLofinterest), 'figures-germcells/0_FS1_heatmap_TFact_cellphoneDB.pdf', display_numbers = x_TFact_Lab)

myHeatmap(x_TFact, cellphoneTFs, rev(CLofinterest), 'figures-germcells/0_FS1_heatmap_TFact_cellphoneDBII.pdf')

# # myHeatmap(x_Expre, cellphoneTFs,
# #           CLofinterest, 'figures-germcells/0_FS1_heatmap_TFexp_cellphoneDB.pdf', display_numbers = x_Expre_Lab)

# # myHeatmap(x_Expre, cellphoneTFs, CLofinterest, 'figures-germcells/0_FS1_heatmap_TFexp_cellphoneDBII.pdf')[1] "oocyte"          "oogonia_meiosis" "oogonia_STRA8"   "PGC"
[1] 4 8
[1] 4 8
[1] "oocyte"          "oogonia_meiosis" "oogonia_STRA8"   "PGC"
[1] 4 8

## what are the Dorothea targets of the new TFs_____no_output_____

# newTFs = c('SOX4', 'HOXA2', 'HOXA4', 'HOXA5', 'HOXB5', 'MXD4', 'DMRTB1', 'TP63')
# source('G_lib_TFacts.r')
# dorothea_regulon$TF_name = strsplit(dorothea_regulon$tf, '_') %>% sapply(., head, 1 )_____no_output_____# markers_crosspecie = c('DPPA4', 'ETV4', 'IFITM1', 'KIT', 'MYBL2', 'NANOG', 'NANOS3', 'POU5F1', 'PRDM1', 'RCC2', 'SALL4', 'SOX15', 'SOX17', 'TEAD4', 'TFAP2C', 'XIST', 'GATA3', 'PPARA',
#     'DAZL', 'DDX4', 'MAEL', 'ESX1', 'MSX1', 'RFX6', 'STRA8', 'ZGLP1',
#     'CHMP2A', 'DMC1', 'DMRTC2', 'INCA1', 'MEIKIN', 'MEIOB', 'MSH4', 'RAD51AP2', 'SPO11', 'SYCP1', 'SYCP3', 'TEX11', 'ZNF711', 'MLH3', 'ZNF131', 'ASTL', 'FIGLA', 'GDF9', 'JAG1', 'NOBOX', 'OOSP2', 'SOHLH1', 'SOHLH2', 'YBX2', 'ZAR1', 'ZP3','BMP15','FOXO3')

# subset(dorothea_regulon, TF_name %in% newTFs &
#        target %in% DEGs$gene[ DEGs$scRNA_DEGs_p_val_adj < 0.1] &
#       target %in% markers_crosspecie )

# subset(dorothea_regulon, TF_name %in% 'MXD4'  &
#        target %in% DEGs$gene[ DEGs$scRNA_DEGs_p_val_adj < 0.001 & DEGs$scRNA_DEGs_avg_log2FC > 0.5 & DEGs$cluster == 'oogonia_STRA8' ])

# subset(dorothea_regulon, TF_name %in% 'ZGLP1'  &
#        target %in% DEGs$gene[ DEGs$scRNA_DEGs_p_val_adj < 0.001 & DEGs$scRNA_DEGs_avg_log2FC > 0.5 & DEGs$cluster == 'oogonia_STRA8' ])_____no_output_____