# Notebook from CyberFlameGO/MadeWithML
Path: notebooks/13_Recurrent_Neural_Networks.ipynb

 Made With ML

Applied ML · MLOps · Production

Join 30K+ developers in learning how to responsibly deliver value with ML.

    🔥  Among the top ML repositories on GitHub

_____no_output_____# Recurrent Neural Networks (RNN)

In this lesson we will learn how to process sequential data (sentences, time-series, etc.) with recurrent neural networks (RNNs)._____no_output_____

_____no_output_____So far we've processed inputs as whole (ex. applying filters across the entire input to extract features) but we can also process our inputs sequentially. For example we can think of each token in our text as an event in time (timestep). We can process each timestep, one at a time, and predict the class after the last timestep (token) has been processed. This is very powerful because the model now has a meaningful way to account for the sequential order of tokens in our sequence and predict accordingly. _____no_output_____# Overview_____no_output_____

RNN forward pass for a single time step $X_t$:

$h_t = tanh(W_{hh}h_{t-1} + W_{xh}X_t+b_h)$

*where*:
* $W_{hh}$ = hidden units weights| $\in \mathbb{R}^{HXH}$ ($H$ is the hidden dim)
* $h_{t-1}$ = previous timestep's hidden state $\in \mathbb{R}^{NXH}$
* $W_{xh}$ = input weights| $\in \mathbb{R}^{EXH}$
* $X_t$ = input at time step t | $\in \mathbb{R}^{NXE}$ ($N$ is the batch size, $E$ is the embedding dim)
* $b_h$ = hidden units bias $\in \mathbb{R}^{HX1}$
* $h_t$ = output from RNN for timestep $t$_____no_output_____* **Objective:**  Process sequential data by accounting for the currend input and also what has been learned from previous inputs.
* **Advantages:**
    * Account for order and previous inputs in a meaningful way.
    * Conditioned generation for generating sequences.
* **Disadvantages:**
    * Each time step's prediction depends on the previous prediction so it's difficult to parallelize RNN operations.
    * Processing long sequences can yield memory and computation issues.
    * Interpretability is difficult but there are few [techniques](https://arxiv.org/abs/1506.02078) that use the activations from RNNs to see what parts of the inputs are processed.
* **Miscellaneous:**
    * Architectural tweaks to make RNNs faster and interpretable is an ongoing area of research._____no_output_____# Set up_____no_output_____

import numpy as np
import pandas as pd
import random
import torch
import torch.nn as nn_____no_output_____SEED = 1234_____no_output_____def set_seeds(seed=1234):
    """Set seeds for reproducibility."""
    np.random.seed(seed)
    random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed) # multi-GPU# Set seeds for reproducibility
set_seeds(seed=SEED)_____no_output_____# Set seeds for reproducibility
set_seeds(seed=SEED)_____no_output_____# Set device
cuda = True
device = torch.device('cuda' if (
    torch.cuda.is_available() and cuda) else 'cpu')
torch.set_default_tensor_type('torch.FloatTensor')
if device.type == 'cuda':
    torch.set_default_tensor_type('torch.cuda.FloatTensor')
print (device)cuda

## Load data_____no_output_____We will download the [AG News dataset](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html), which consists of 120K text samples from 4 unique classes (`Business`, `Sci/Tech`, `Sports`, `World`)_____no_output_____

import numpy as np
import pandas as pd
import re
import urllib_____no_output_____# Load data
url = "https://raw.githubusercontent.com/GokuMohandas/MadeWithML/main/datasets/news.csv"
df = pd.read_csv(url, header=0) # load
df = df.sample(frac=1).reset_index(drop=True) # shuffle
df.head()_____no_output_____

## Preprocessing_____no_output_____We're going to clean up our input data first by doing operations such as lower text, removing stop (filler) words, filters using regular expressions, etc._____no_output_____

import nltk
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer
import re_____no_output_____nltk.download('stopwords')
STOPWORDS = stopwords.words('english')
print (STOPWORDS[:5])
porter = PorterStemmer()[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.
['i', 'me', 'my', 'myself', 'we']
def preprocess(text, stopwords=STOPWORDS):
    """Conditional preprocessing on our text unique to our task."""
    # Lower
    text = text.lower()

    # Remove stopwords
    pattern = re.compile(r'\b(' + r'|'.join(stopwords) + r')\b\s*')
    text = pattern.sub('', text)

    # Remove words in paranthesis
    text = re.sub(r'\([^)]*\)', '', text)

    # Spacing and filters
    text = re.sub(r"([-;;.,!?<=>])", r" \1 ", text)
    text = re.sub('[^A-Za-z0-9]+', ' ', text) # remove non alphanumeric chars
    text = re.sub(' +', ' ', text)  # remove multiple spaces
    text = text.strip()

    return text_____no_output_____# Sample
text = "Great week for the NYSE!"
preprocess(text=text)_____no_output_____# Apply to dataframe
preprocessed_df = df.copy()
preprocessed_df.title = preprocessed_df.title.apply(preprocess)
print (f"{df.title.values[0]}\n\n{preprocessed_df.title.values[0]}")Sharon Accepts Plan to Reduce Gaza Army Operation, Haaretz Says

sharon accepts plan reduce gaza army operation haaretz says

## Split data_____no_output_____

import collections
from sklearn.model_selection import train_test_split_____no_output_____TRAIN_SIZE = 0.7
VAL_SIZE = 0.15
TEST_SIZE = 0.15_____no_output_____def train_val_test_split(X, y, train_size):
    """Split dataset into data splits."""
    X_train, X_, y_train, y_ = train_test_split(X, y, train_size=TRAIN_SIZE, stratify=y)
    X_val, X_test, y_val, y_test = train_test_split(X_, y_, train_size=0.5, stratify=y_)
    return X_train, X_val, X_test, y_train, y_val, y_test_____no_output_____# Data
X = preprocessed_df["title"].values
y = preprocessed_df["category"].values_____no_output_____# Create data splits
X_train, X_val, X_test, y_train, y_val, y_test = train_val_test_split(
    X=X, y=y, train_size=TRAIN_SIZE)
print (f"X_train: {X_train.shape}, y_train: {y_train.shape}")
print (f"X_val: {X_val.shape}, y_val: {y_val.shape}")
print (f"X_test: {X_test.shape}, y_test: {y_test.shape}")
print (f"Sample point: {X_train[0]} → {y_train[0]}")X_train: (84000,), y_train: (84000,)
X_val: (18000,), y_val: (18000,)
X_test: (18000,), y_test: (18000,)
Sample point: china battles north korea nuclear talks → World

## LabelEncoder_____no_output_____Next we'll define a `LabelEncoder` to encode our text labels into unique indices_____no_output_____

import itertools_____no_output_____class LabelEncoder(object):
    """Label encoder for tag labels."""
    def __init__(self, class_to_index={}):
        self.class_to_index = class_to_index
        self.index_to_class = {v: k for k, v in self.class_to_index.items()}
        self.classes = list(self.class_to_index.keys())

    def __len__(self):
        return len(self.class_to_index)

    def __str__(self):
        return f"<LabelEncoder(num_classes={len(self)})>"

    def fit(self, y):
        classes = np.unique(y)
        for i, class_ in enumerate(classes):
            self.class_to_index[class_] = i
        self.index_to_class = {v: k for k, v in self.class_to_index.items()}
        self.classes = list(self.class_to_index.keys())
        return self

    def encode(self, y):
        encoded = np.zeros((len(y)), dtype=int)
        for i, item in enumerate(y):
            encoded[i] = self.class_to_index[item]
        return encoded

    def decode(self, y):
        classes = []
        for i, item in enumerate(y):
            classes.append(self.index_to_class[item])
        return classes

    def save(self, fp):
        with open(fp, 'w') as fp:
            contents = {'class_to_index': self.class_to_index}
            json.dump(contents, fp, indent=4, sort_keys=False)

    @classmethod
    def load(cls, fp):
        with open(fp, 'r') as fp:
            kwargs = json.load(fp=fp)
        return cls(**kwargs)_____no_output_____# Encode
label_encoder = LabelEncoder()
label_encoder.fit(y_train)
NUM_CLASSES = len(label_encoder)
label_encoder.class_to_index_____no_output_____# Convert labels to tokens
print (f"y_train[0]: {y_train[0]}")
y_train = label_encoder.encode(y_train)
y_val = label_encoder.encode(y_val)
y_test = label_encoder.encode(y_test)
print (f"y_train[0]: {y_train[0]}")y_train[0]: World
y_train[0]: 3
# Class weights
counts = np.bincount(y_train)
class_weights = {i: 1.0/count for i, count in enumerate(counts)}
print (f"counts: {counts}\nweights: {class_weights}")counts: [21000 21000 21000 21000]
weights: {0: 4.761904761904762e-05, 1: 4.761904761904762e-05, 2: 4.761904761904762e-05, 3: 4.761904761904762e-05}

## Tokenizer_____no_output_____We'll define a `Tokenizer` to convert our text input data into token indices._____no_output_____

import json
from collections import Counter
from more_itertools import take_____no_output_____class Tokenizer(object):
    def __init__(self, char_level, num_tokens=None,
                 pad_token='<PAD>', oov_token='<UNK>',
                 token_to_index=None):
        self.char_level = char_level
        self.separator = '' if self.char_level else ' '
        if num_tokens: num_tokens -= 2 # pad + unk tokens
        self.num_tokens = num_tokens
        self.pad_token = pad_token
        self.oov_token = oov_token
        if not token_to_index:
            token_to_index = {pad_token: 0, oov_token: 1}
        self.token_to_index = token_to_index
        self.index_to_token = {v: k for k, v in self.token_to_index.items()}

    def __len__(self):
        return len(self.token_to_index)

    def __str__(self):
        return f"<Tokenizer(num_tokens={len(self)})>"

    def fit_on_texts(self, texts):
        if not self.char_level:
            texts = [text.split(" ") for text in texts]
        all_tokens = [token for text in texts for token in text]
        counts = Counter(all_tokens).most_common(self.num_tokens)
        self.min_token_freq = counts[-1][1]
        for token, count in counts:
            index = len(self)
            self.token_to_index[token] = index
            self.index_to_token[index] = token
        return self

    def texts_to_sequences(self, texts):
        sequences = []
        for text in texts:
            if not self.char_level:
                text = text.split(' ')
            sequence = []
            for token in text:
                sequence.append(self.token_to_index.get(
                    token, self.token_to_index[self.oov_token]))
            sequences.append(np.asarray(sequence))
        return sequences

    def sequences_to_texts(self, sequences):
        texts = []
        for sequence in sequences:
            text = []
            for index in sequence:
                text.append(self.index_to_token.get(index, self.oov_token))
            texts.append(self.separator.join([token for token in text]))
        return texts

    def save(self, fp):
        with open(fp, 'w') as fp:
            contents = {
                'char_level': self.char_level,
                'oov_token': self.oov_token,
                'token_to_index': self.token_to_index
            }
            json.dump(contents, fp, indent=4, sort_keys=False)

    @classmethod
    def load(cls, fp):
        with open(fp, 'r') as fp:
            kwargs = json.load(fp=fp)
        return cls(**kwargs)_____no_output_____# Tokenize
tokenizer = Tokenizer(char_level=False, num_tokens=5000)
tokenizer.fit_on_texts(texts=X_train)
VOCAB_SIZE = len(tokenizer)
print (tokenizer)<Tokenizer(num_tokens=5000)>
# Sample of tokens
print (take(5, tokenizer.token_to_index.items()))
print (f"least freq token's freq: {tokenizer.min_token_freq}") # use this to adjust num_tokens[('<PAD>', 0), ('<UNK>', 1), ('39', 2), ('b', 3), ('gt', 4)]
least freq token's freq: 14
# Convert texts to sequences of indices
X_train = tokenizer.texts_to_sequences(X_train)
X_val = tokenizer.texts_to_sequences(X_val)
X_test = tokenizer.texts_to_sequences(X_test)
preprocessed_text = tokenizer.sequences_to_texts([X_train[0]])[0]
print ("Text to indices:\n"
    f"  (preprocessed) → {preprocessed_text}\n"
    f"  (tokenized) → {X_train[0]}")Text to indices:
  (preprocessed) → china battles north korea nuclear talks
  (tokenized) → [  16 1491  285  142  114   24]

## Padding_____no_output_____We'll need to do 2D padding to our tokenized text._____no_output_____

def pad_sequences(sequences, max_seq_len=0):
    """Pad sequences to max length in sequence."""
    max_seq_len = max(max_seq_len, max(len(sequence) for sequence in sequences))
    padded_sequences = np.zeros((len(sequences), max_seq_len))
    for i, sequence in enumerate(sequences):
        padded_sequences[i][:len(sequence)] = sequence
    return padded_sequences_____no_output_____# 2D sequences
padded = pad_sequences(X_train[0:3])
print (padded.shape)
print (padded)(3, 6)
[[1.600e+01 1.491e+03 2.850e+02 1.420e+02 1.140e+02 2.400e+01]
 [1.445e+03 2.300e+01 6.560e+02 2.197e+03 1.000e+00 0.000e+00]
 [1.200e+02 1.400e+01 1.955e+03 1.005e+03 1.529e+03 4.014e+03]]

## Datasets_____no_output_____We're going to create Datasets and DataLoaders to be able to efficiently create batches with our data splits._____no_output_____

class Dataset(torch.utils.data.Dataset):
    def __init__(self, X, y,):
        self.X = X
        self.y = y

    def __len__(self):
        return len(self.y)

    def __str__(self):
        return f"<Dataset(N={len(self)})>"

    def __getitem__(self, index):
        X = self.X[index]
        y = self.y[index]
        return [X, len(X), y]

    def collate_fn(self, batch):
        """Processing on a batch."""
        # Get inputs
        batch = np.array(batch, dtype=object)
        X = batch[:, 0]
        seq_lens = batch[:, 1]
        y = np.stack(batch[:, 2], axis=0)

        # Pad inputs
        X = pad_sequences(sequences=X)

        # Cast
        X = torch.LongTensor(X.astype(np.int32))
        seq_lens = torch.LongTensor(seq_lens.astype(np.int32))
        y = torch.LongTensor(y.astype(np.int32))

        return X, seq_lens, y

    def create_dataloader(self, batch_size, shuffle=False, drop_last=False):
        return torch.utils.data.DataLoader(
            dataset=self, batch_size=batch_size, collate_fn=self.collate_fn,
            shuffle=shuffle, drop_last=drop_last, pin_memory=True)_____no_output_____# Create datasets
train_dataset = Dataset(X=X_train, y=y_train)
val_dataset = Dataset(X=X_val, y=y_val)
test_dataset = Dataset(X=X_test, y=y_test)
print ("Datasets:\n"
    f"  Train dataset:{train_dataset.__str__()}\n"
    f"  Val dataset: {val_dataset.__str__()}\n"
    f"  Test dataset: {test_dataset.__str__()}\n"
    "Sample point:\n"
    f"  X: {train_dataset[0][0]}\n"
    f"  seq_len: {train_dataset[0][1]}\n"
    f"  y: {train_dataset[0][2]}")Datasets:
  Train dataset:<Dataset(N=84000)>
  Val dataset: <Dataset(N=18000)>
  Test dataset: <Dataset(N=18000)>
Sample point:
  X: [  16 1491  285  142  114   24]
  seq_len: 6
  y: 3
# Create dataloaders
batch_size = 64
train_dataloader = train_dataset.create_dataloader(
    batch_size=batch_size)
val_dataloader = val_dataset.create_dataloader(
    batch_size=batch_size)
test_dataloader = test_dataset.create_dataloader(
    batch_size=batch_size)
batch_X, batch_seq_lens, batch_y = next(iter(train_dataloader))
print ("Sample batch:\n"
    f"  X: {list(batch_X.size())}\n"
    f"  seq_lens: {list(batch_seq_lens.size())}\n"
    f"  y: {list(batch_y.size())}\n"
    "Sample point:\n"
    f"  X: {batch_X[0]}\n"
    f" seq_len: {batch_seq_lens[0]}\n"
    f"  y: {batch_y[0]}")Sample batch:
  X: [64, 14]
  seq_lens: [64]
  y: [64]
Sample point:
  X: tensor([  16, 1491,  285,  142,  114,   24,    0,    0,    0,    0,    0,    0,
           0,    0], device='cpu')
 seq_len: 6
  y: 3

## Trainer_____no_output_____Let's create the `Trainer` class that we'll use to facilitate training for our experiments._____no_output_____

class Trainer(object):
    def __init__(self, model, device, loss_fn=None, optimizer=None, scheduler=None):

        # Set params
        self.model = model
        self.device = device
        self.loss_fn = loss_fn
        self.optimizer = optimizer
        self.scheduler = scheduler

    def train_step(self, dataloader):
        """Train step."""
        # Set model to train mode
        self.model.train()
        loss = 0.0

        # Iterate over train batches
        for i, batch in enumerate(dataloader):

            # Step
            batch = [item.to(self.device) for item in batch]  # Set device
            inputs, targets = batch[:-1], batch[-1]
            self.optimizer.zero_grad()  # Reset gradients
            z = self.model(inputs)  # Forward pass
            J = self.loss_fn(z, targets)  # Define loss
            J.backward()  # Backward pass
            self.optimizer.step()  # Update weights

            # Cumulative Metrics
            loss += (J.detach().item() - loss) / (i + 1)

        return loss

    def eval_step(self, dataloader):
        """Validation or test step."""
        # Set model to eval mode
        self.model.eval()
        loss = 0.0
        y_trues, y_probs = [], []

        # Iterate over val batches
        with torch.no_grad():
            for i, batch in enumerate(dataloader):

                # Step
                batch = [item.to(self.device) for item in batch]  # Set device
                inputs, y_true = batch[:-1], batch[-1]
                z = self.model(inputs)  # Forward pass
                J = self.loss_fn(z, y_true).item()

                # Cumulative Metrics
                loss += (J - loss) / (i + 1)

                # Store outputs
                y_prob = F.softmax(z).cpu().numpy()
                y_probs.extend(y_prob)
                y_trues.extend(y_true.cpu().numpy())

        return loss, np.vstack(y_trues), np.vstack(y_probs)

    def predict_step(self, dataloader):
        """Prediction step."""
        # Set model to eval mode
        self.model.eval()
        y_probs = []

        # Iterate over val batches
        with torch.no_grad():
            for i, batch in enumerate(dataloader):

                # Forward pass w/ inputs
                inputs, targets = batch[:-1], batch[-1]
                z = self.model(inputs)

                # Store outputs
                y_prob = F.softmax(z).cpu().numpy()
                y_probs.extend(y_prob)

        return np.vstack(y_probs)

    def train(self, num_epochs, patience, train_dataloader, val_dataloader):
        best_val_loss = np.inf
        for epoch in range(num_epochs):
            # Steps
            train_loss = self.train_step(dataloader=train_dataloader)
            val_loss, _, _ = self.eval_step(dataloader=val_dataloader)
            self.scheduler.step(val_loss)

            # Early stopping
            if val_loss < best_val_loss:
                best_val_loss = val_loss
                best_model = self.model
                _patience = patience  # reset _patience
            else:
                _patience -= 1
            if not _patience:  # 0
                print("Stopping early!")
                break

            # Logging
            print(
                f"Epoch: {epoch+1} | "
                f"train_loss: {train_loss:.5f}, "
                f"val_loss: {val_loss:.5f}, "
                f"lr: {self.optimizer.param_groups[0]['lr']:.2E}, "
                f"_patience: {_patience}"
            )
        return best_model_____no_output_____

# Vanilla RNN_____no_output_____Inputs to RNNs are sequential like text or time-series._____no_output_____

BATCH_SIZE = 64
EMBEDDING_DIM = 100_____no_output_____# Input
sequence_size = 8 # words per input
x = torch.rand((BATCH_SIZE, sequence_size, EMBEDDING_DIM))
seq_lens = torch.randint(high=sequence_size, size=(1, BATCH_SIZE))
print (x.shape)
print (seq_lens.shape)torch.Size([64, 8, 100])
torch.Size([1, 64])

RNN forward pass for a single time step $X_t$:

$h_t = tanh(W_{hh}h_{t-1} + W_{xh}X_t+b_h)$

*where*:
* $W_{hh}$ = hidden units weights| $\in \mathbb{R}^{HXH}$ ($H$ is the hidden dim)
* $h_{t-1}$ = previous timestep's hidden state $\in \mathbb{R}^{NXH}$
* $W_{xh}$ = input weights| $\in \mathbb{R}^{EXH}$
* $X_t$ = input at time step t | $\in \mathbb{R}^{NXE}$ ($N$ is the batch size, $E$ is the embedding dim)
* $b_h$ = hidden units bias $\in \mathbb{R}^{HX1}$
* $h_t$ = output from RNN for timestep $t$

> At the first time step, the previous hidden state $h_{t-1}$ can either be a zero vector (unconditioned) or initialized (conditioned). If we are conditioning the RNN, the first hidden state $h_0$ can belong to a specific condition or we can concat the specific condition to the randomly initialized hidden vectors at each time step. More on this in the subsequent notebooks on RNNs._____no_output_____

RNN_HIDDEN_DIM = 128
DROPOUT_P = 0.1
RNN_DROPOUT_P = 0.1_____no_output_____# Initialize hidden state
hidden_t = torch.zeros((BATCH_SIZE, RNN_HIDDEN_DIM))
print (hidden_t.size())torch.Size([64, 128])

We'll show how to create an RNN cell using PyTorch's [`RNNCell`](https://pytorch.org/docs/stable/generated/torch.nn.RNNCell.html#torch.nn.RNNCell) and the more abstracted [`RNN`](https://pytorch.org/docs/stable/generated/torch.nn.RNN.html#torch.nn.RNN)._____no_output_____

# Initialize RNN cell
rnn_cell = nn.RNNCell(EMBEDDING_DIM, RNN_HIDDEN_DIM)
print (rnn_cell)RNNCell(100, 128)
# Forward pass through RNN
x = x.permute(1, 0, 2) # RNN needs batch_size to be at dim 1

# Loop through the inputs time steps
hiddens = []
for t in range(sequence_size):
    hidden_t = rnn_cell(x[t], hidden_t)
    hiddens.append(hidden_t)
hiddens = torch.stack(hiddens)
hiddens = hiddens.permute(1, 0, 2) # bring batch_size back to dim 0
print (hiddens.size())torch.Size([64, 8, 128])
# We also could've used a more abstracted layer
x = torch.rand((BATCH_SIZE, sequence_size, EMBEDDING_DIM))
rnn = nn.RNN(EMBEDDING_DIM, RNN_HIDDEN_DIM, batch_first=True)
out, h_n = rnn(x) # h_n is the last hidden state
print ("out: ", out.shape)
print ("h_n: ", h_n.shape)out:  torch.Size([64, 8, 128])
h_n:  torch.Size([1, 64, 128])
# The same tensors
print (out[:,-1,:])
print (h_n.squeeze(0))tensor([[-0.5056,  0.6157,  0.4275,  ...,  0.1804, -0.1480, -0.4822],
        [-0.1490,  0.6549,  0.3184,  ...,  0.2831, -0.3557, -0.5438],
        [-0.5290,  0.4321,  0.0885,  ...,  0.4848, -0.2672, -0.2660],
        ...,
        [-0.3273,  0.6155, -0.2170,  ...,  0.1718, -0.1623, -0.3876],
        [-0.3860,  0.3749,  0.0142,  ...,  0.6179, -0.3790, -0.2459],
        [-0.0464,  0.4893, -0.2189,  ...,  0.3532, -0.3793, -0.5216]],
       grad_fn=<SliceBackward>)
tensor([[-0.5056,  0.6157,  0.4275,  ...,  0.1804, -0.1480, -0.4822],
        [-0.1490,  0.6549,  0.3184,  ...,  0.2831, -0.3557, -0.5438],
        [-0.5290,  0.4321,  0.0885,  ...,  0.4848, -0.2672, -0.2660],
        ...,
        [-0.3273,  0.6155, -0.2170,  ...,  0.1718, -0.1623, -0.3876],
        [-0.3860,  0.3749,  0.0142,  ...,  0.6179, -0.3790, -0.2459],
        [-0.0464,  0.4893, -0.2189,  ...,  0.3532, -0.3793, -0.5216]],
       grad_fn=<SqueezeBackward1>)

In our model, we want to use the RNN's output after the last relevant token in the sentence is processed. The last relevant token doesn't refer the `<PAD>` tokens but to the last actual word in the sentence and its index is different for each input in the batch. This is why we included a `seq_lens` tensor in our batches._____no_output_____

def gather_last_relevant_hidden(hiddens, seq_lens):
    """Extract and collect the last relevant
    hidden state based on the sequence length."""
    seq_lens = seq_lens.long().detach().cpu().numpy() - 1
    out = []
    for batch_index, column_index in enumerate(seq_lens):
        out.append(hiddens[batch_index, column_index])
    return torch.stack(out)_____no_output_____# Get the last relevant hidden state
gather_last_relevant_hidden(hiddens=out, seq_lens=seq_lens).squeeze(0).shape_____no_output_____

There are many different ways to use RNNs. So far we've processed our inputs one timestep at a time and we could either use the RNN's output at each time step or just use the final input timestep's RNN output. Let's look at a few other possibilities.

_____no_output_____## Model_____no_output_____

import torch.nn.functional as F_____no_output_____HIDDEN_DIM = 100_____no_output_____class RNN(nn.Module):
    def __init__(self, embedding_dim, vocab_size, rnn_hidden_dim,
                 hidden_dim, dropout_p, num_classes, padding_idx=0):
        super(RNN, self).__init__()

        # Initialize embeddings
        self.embeddings = nn.Embedding(
            embedding_dim=embedding_dim, num_embeddings=vocab_size,
            padding_idx=padding_idx)

        # RNN
        self.rnn = nn.RNN(embedding_dim, rnn_hidden_dim, batch_first=True)

        # FC weights
        self.dropout = nn.Dropout(dropout_p)
        self.fc1 = nn.Linear(rnn_hidden_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, num_classes)

    def forward(self, inputs):
        # Embed
        x_in, seq_lens = inputs
        x_in = self.embeddings(x_in)

        # Rnn outputs
        out, h_n = self.rnn(x_in)
        z = gather_last_relevant_hidden(hiddens=out, seq_lens=seq_lens)

        # FC layers
        z = self.fc1(z)
        z = self.dropout(z)
        z = self.fc2(z)
        return z_____no_output_____# Simple RNN cell
model = RNN(
    embedding_dim=EMBEDDING_DIM, vocab_size=VOCAB_SIZE,
    rnn_hidden_dim=RNN_HIDDEN_DIM, hidden_dim=HIDDEN_DIM,
    dropout_p=DROPOUT_P, num_classes=NUM_CLASSES)
model = model.to(device) # set device
print (model.named_parameters)<bound method Module.named_parameters of RNN(
  (embeddings): Embedding(5000, 100, padding_idx=0)
  (rnn): RNN(100, 128, batch_first=True)
  (dropout): Dropout(p=0.1, inplace=False)
  (fc1): Linear(in_features=128, out_features=100, bias=True)
  (fc2): Linear(in_features=100, out_features=4, bias=True)
)>

## Training_____no_output_____

from torch.optim import Adam_____no_output_____NUM_LAYERS = 1
LEARNING_RATE = 1e-4
PATIENCE = 10
NUM_EPOCHS = 50_____no_output_____# Define Loss
class_weights_tensor = torch.Tensor(list(class_weights.values())).to(device)
loss_fn = nn.CrossEntropyLoss(weight=class_weights_tensor)_____no_output_____# Define optimizer & scheduler
optimizer = Adam(model.parameters(), lr=LEARNING_RATE)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
    optimizer, mode='min', factor=0.1, patience=3)_____no_output_____# Trainer module
trainer = Trainer(
    model=model, device=device, loss_fn=loss_fn,
    optimizer=optimizer, scheduler=scheduler)_____no_output_____# Train
best_model = trainer.train(
    NUM_EPOCHS, PATIENCE, train_dataloader, val_dataloader)Epoch: 1 | train_loss: 1.25500, val_loss: 1.12003, lr: 1.00E-04, _patience: 10
Epoch: 2 | train_loss: 1.03130, val_loss: 0.97659, lr: 1.00E-04, _patience: 10
Epoch: 3 | train_loss: 0.89955, val_loss: 0.87245, lr: 1.00E-04, _patience: 10
Epoch: 4 | train_loss: 0.79928, val_loss: 0.79484, lr: 1.00E-04, _patience: 10
Epoch: 5 | train_loss: 0.72322, val_loss: 0.73841, lr: 1.00E-04, _patience: 10
Epoch: 6 | train_loss: 0.66636, val_loss: 0.69743, lr: 1.00E-04, _patience: 10
Epoch: 7 | train_loss: 0.62149, val_loss: 0.66580, lr: 1.00E-04, _patience: 10
Epoch: 8 | train_loss: 0.58615, val_loss: 0.64251, lr: 1.00E-04, _patience: 10
Epoch: 9 | train_loss: 0.55711, val_loss: 0.62356, lr: 1.00E-04, _patience: 10
Epoch: 10 | train_loss: 0.53208, val_loss: 0.60801, lr: 1.00E-04, _patience: 10
Epoch: 11 | train_loss: 0.51060, val_loss: 0.59539, lr: 1.00E-04, _patience: 10
Epoch: 12 | train_loss: 0.49146, val_loss: 0.58582, lr: 1.00E-04, _patience: 10
Epoch: 13 | train_loss: 0.47394, val_loss: 0.57700, lr: 1.00E-04, _patience: 10
Epoch: 14 | train_loss: 0.45890, val_loss: 0.57153, lr: 1.00E-04, _patience: 10
Epoch: 15 | train_loss: 0.44470, val_loss: 0.56508, lr: 1.00E-04, _patience: 10
Epoch: 16 | train_loss: 0.43173, val_loss: 0.56183, lr: 1.00E-04, _patience: 10
Epoch: 17 | train_loss: 0.41952, val_loss: 0.55895, lr: 1.00E-04, _patience: 10
Epoch: 18 | train_loss: 0.40817, val_loss: 0.55745, lr: 1.00E-04, _patience: 10
Epoch: 19 | train_loss: 0.39774, val_loss: 0.55570, lr: 1.00E-04, _patience: 10
Epoch: 20 | train_loss: 0.38761, val_loss: 0.55598, lr: 1.00E-04, _patience: 9
Epoch: 21 | train_loss: 0.37759, val_loss: 0.55710, lr: 1.00E-04, _patience: 8
Epoch: 22 | train_loss: 0.36872, val_loss: 0.55861, lr: 1.00E-04, _patience: 7
Epoch: 23 | train_loss: 0.35990, val_loss: 0.55922, lr: 1.00E-05, _patience: 6
Epoch: 24 | train_loss: 0.33898, val_loss: 0.53767, lr: 1.00E-05, _patience: 10
Epoch: 25 | train_loss: 0.33417, val_loss: 0.53845, lr: 1.00E-05, _patience: 9
Epoch: 26 | train_loss: 0.33196, val_loss: 0.53936, lr: 1.00E-05, _patience: 8
Epoch: 27 | train_loss: 0.33048, val_loss: 0.54046, lr: 1.00E-05, _patience: 7
Epoch: 28 | train_loss: 0.32941, val_loss: 0.54174, lr: 1.00E-06, _patience: 6
Epoch: 29 | train_loss: 0.32617, val_loss: 0.53706, lr: 1.00E-06, _patience: 10
Epoch: 30 | train_loss: 0.32539, val_loss: 0.53703, lr: 1.00E-06, _patience: 10
Epoch: 31 | train_loss: 0.32537, val_loss: 0.53706, lr: 1.00E-06, _patience: 9
Epoch: 32 | train_loss: 0.32483, val_loss: 0.53709, lr: 1.00E-06, _patience: 8
Epoch: 33 | train_loss: 0.32509, val_loss: 0.53717, lr: 1.00E-07, _patience: 7
Epoch: 34 | train_loss: 0.32449, val_loss: 0.53715, lr: 1.00E-07, _patience: 6
Epoch: 35 | train_loss: 0.32475, val_loss: 0.53718, lr: 1.00E-07, _patience: 5
Epoch: 36 | train_loss: 0.32441, val_loss: 0.53720, lr: 1.00E-07, _patience: 4
Epoch: 37 | train_loss: 0.32452, val_loss: 0.53722, lr: 1.00E-08, _patience: 3
Epoch: 38 | train_loss: 0.32467, val_loss: 0.53722, lr: 1.00E-08, _patience: 2
Epoch: 39 | train_loss: 0.32426, val_loss: 0.53723, lr: 1.00E-08, _patience: 1
Stopping early!

## Evaluation_____no_output_____

import json
from sklearn.metrics import precision_recall_fscore_support_____no_output_____def get_performance(y_true, y_pred, classes):
    """Per-class performance metrics."""
    # Performance
    performance = {"overall": {}, "class": {}}

    # Overall performance
    metrics = precision_recall_fscore_support(y_true, y_pred, average="weighted")
    performance["overall"]["precision"] = metrics[0]
    performance["overall"]["recall"] = metrics[1]
    performance["overall"]["f1"] = metrics[2]
    performance["overall"]["num_samples"] = np.float64(len(y_true))

    # Per-class performance
    metrics = precision_recall_fscore_support(y_true, y_pred, average=None)
    for i in range(len(classes)):
        performance["class"][classes[i]] = {
            "precision": metrics[0][i],
            "recall": metrics[1][i],
            "f1": metrics[2][i],
            "num_samples": np.float64(metrics[3][i]),
        }

    return performance_____no_output_____# Get predictions
test_loss, y_true, y_prob = trainer.eval_step(dataloader=test_dataloader)
y_pred = np.argmax(y_prob, axis=1)_____no_output_____# Determine performance
performance = get_performance(
    y_true=y_test, y_pred=y_pred, classes=label_encoder.classes)
print (json.dumps(performance['overall'], indent=2)){
  "precision": 0.8211429212272771,
  "recall": 0.8212777777777778,
  "f1": 0.8208202838924475,
  "num_samples": 18000.0
}

# Gated RNNs: LSTMs & GRUs_____no_output_____While our simple RNNs so far are great for sequentially processing our inputs, they have quite a few disadvantages. They commonly suffer from exploding or vanishing gradients as a result using the same set of weights ($W_{xh}$ and $W_{hh}$) with each timestep's input. During backpropagation, this can cause gradients to explode (>1) or vanish (<1). If you multiply any number greater than 1 with itself over and over, it moves towards infinity (exploding gradients) and similarly,  If you multiply any number less than 1 with itself over and over, it moves towards zero (vanishing gradients). To mitigate this issue, gated RNNs were devised to selectively retrain information. If you're interested in learning more of the specifics, this [post](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) is a must-read.

There are two popular types of gated RNNs: Long Short-term Memory (LSTMs) units and Gated Recurrent Units (GRUs).

Understanding LSTM Networks - Chris Olah

> When deciding between LSTMs and GRUs, empirical performance is the best factor but in genreal GRUs offer similar performance with less complexity (less weights).

_____no_output_____

# Input
sequence_size = 8 # words per input
x = torch.rand((BATCH_SIZE, sequence_size, EMBEDDING_DIM))
print (x.shape)torch.Size([64, 8, 100])
# GRU
gru = nn.GRU(input_size=EMBEDDING_DIM, hidden_size=RNN_HIDDEN_DIM, batch_first=True)_____no_output_____# Forward pass
out, h_n = gru(x)
print (f"out: {out.shape}")
print (f"h_n: {h_n.shape}")out: torch.Size([64, 8, 128])
h_n: torch.Size([1, 64, 128])

## Bidirectional RNN_____no_output_____We can also have RNNs that process inputs from both directions (first token to last token and vice versa) and combine their outputs. This architecture is known as a bidirectional RNN. _____no_output_____

# GRU
gru = nn.GRU(input_size=EMBEDDING_DIM, hidden_size=RNN_HIDDEN_DIM,
             batch_first=True, bidirectional=True)_____no_output_____# Forward pass
out, h_n = gru(x)
print (f"out: {out.shape}")
print (f"h_n: {h_n.shape}")out: torch.Size([64, 8, 256])
h_n: torch.Size([2, 64, 128])

Notice that the output for each sample at each timestamp has size 256 (double the `RNN_HIDDEN_DIM`). This is because this includes both the forward and backward directions from the BiRNN. _____no_output_____## Model_____no_output_____

class GRU(nn.Module):
    def __init__(self, embedding_dim, vocab_size, rnn_hidden_dim,
                 hidden_dim, dropout_p, num_classes, padding_idx=0):
        super(GRU, self).__init__()

        # Initialize embeddings
        self.embeddings = nn.Embedding(embedding_dim=embedding_dim,
                                       num_embeddings=vocab_size,
                                       padding_idx=padding_idx)

        # RNN
        self.rnn = nn.GRU(embedding_dim, rnn_hidden_dim,
                          batch_first=True, bidirectional=True)

        # FC weights
        self.dropout = nn.Dropout(dropout_p)
        self.fc1 = nn.Linear(rnn_hidden_dim*2, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, num_classes)

    def forward(self, inputs):
        # Embed
        x_in, seq_lens = inputs
        x_in = self.embeddings(x_in)

        # Rnn outputs
        out, h_n = self.rnn(x_in)
        z = gather_last_relevant_hidden(hiddens=out, seq_lens=seq_lens)

        # FC layers
        z = self.fc1(z)
        z = self.dropout(z)
        z = self.fc2(z)
        return z_____no_output_____# Simple RNN cell
model = GRU(
    embedding_dim=EMBEDDING_DIM, vocab_size=VOCAB_SIZE,
    rnn_hidden_dim=RNN_HIDDEN_DIM, hidden_dim=HIDDEN_DIM,
    dropout_p=DROPOUT_P, num_classes=NUM_CLASSES)
model = model.to(device) # set device
print (model.named_parameters)<bound method Module.named_parameters of GRU(
  (embeddings): Embedding(5000, 100, padding_idx=0)
  (rnn): GRU(100, 128, batch_first=True, bidirectional=True)
  (dropout): Dropout(p=0.1, inplace=False)
  (fc1): Linear(in_features=256, out_features=100, bias=True)
  (fc2): Linear(in_features=100, out_features=4, bias=True)
)>

## Training_____no_output_____

# Define Loss
class_weights_tensor = torch.Tensor(list(class_weights.values())).to(device)
loss_fn = nn.CrossEntropyLoss(weight=class_weights_tensor)_____no_output_____# Define optimizer & scheduler
optimizer = Adam(model.parameters(), lr=LEARNING_RATE)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
    optimizer, mode='min', factor=0.1, patience=3)_____no_output_____# Trainer module
trainer = Trainer(
    model=model, device=device, loss_fn=loss_fn,
    optimizer=optimizer, scheduler=scheduler)_____no_output_____# Train
best_model = trainer.train(
    NUM_EPOCHS, PATIENCE, train_dataloader, val_dataloader)Epoch: 1 | train_loss: 1.16930, val_loss: 0.94210, lr: 1.00E-04, _patience: 10
Epoch: 2 | train_loss: 0.82127, val_loss: 0.72819, lr: 1.00E-04, _patience: 10
Epoch: 3 | train_loss: 0.65862, val_loss: 0.64288, lr: 1.00E-04, _patience: 10
Epoch: 4 | train_loss: 0.58088, val_loss: 0.60078, lr: 1.00E-04, _patience: 10
Epoch: 5 | train_loss: 0.53120, val_loss: 0.57543, lr: 1.00E-04, _patience: 10
Epoch: 6 | train_loss: 0.49535, val_loss: 0.55925, lr: 1.00E-04, _patience: 10
Epoch: 7 | train_loss: 0.46603, val_loss: 0.54962, lr: 1.00E-04, _patience: 10
Epoch: 8 | train_loss: 0.44215, val_loss: 0.54387, lr: 1.00E-04, _patience: 10
Epoch: 9 | train_loss: 0.42093, val_loss: 0.53825, lr: 1.00E-04, _patience: 10
Epoch: 10 | train_loss: 0.40259, val_loss: 0.53758, lr: 1.00E-04, _patience: 10
Epoch: 11 | train_loss: 0.38635, val_loss: 0.53765, lr: 1.00E-04, _patience: 9
Epoch: 12 | train_loss: 0.37150, val_loss: 0.53935, lr: 1.00E-04, _patience: 8
Epoch: 13 | train_loss: 0.35735, val_loss: 0.54147, lr: 1.00E-04, _patience: 7
Epoch: 14 | train_loss: 0.34519, val_loss: 0.54368, lr: 1.00E-05, _patience: 6
Epoch: 15 | train_loss: 0.31900, val_loss: 0.53039, lr: 1.00E-05, _patience: 10
Epoch: 16 | train_loss: 0.31445, val_loss: 0.53071, lr: 1.00E-05, _patience: 9
Epoch: 17 | train_loss: 0.31230, val_loss: 0.53166, lr: 1.00E-05, _patience: 8
Epoch: 18 | train_loss: 0.31042, val_loss: 0.53303, lr: 1.00E-05, _patience: 7
Epoch: 19 | train_loss: 0.30883, val_loss: 0.53416, lr: 1.00E-06, _patience: 6
Epoch: 20 | train_loss: 0.30481, val_loss: 0.53245, lr: 1.00E-06, _patience: 5
Epoch: 21 | train_loss: 0.30435, val_loss: 0.53248, lr: 1.00E-06, _patience: 4
Epoch: 22 | train_loss: 0.30458, val_loss: 0.53250, lr: 1.00E-06, _patience: 3
Epoch: 23 | train_loss: 0.30449, val_loss: 0.53258, lr: 1.00E-07, _patience: 2
Epoch: 24 | train_loss: 0.30425, val_loss: 0.53252, lr: 1.00E-07, _patience: 1
Stopping early!

## Evaluation_____no_output_____

from pathlib import Path_____no_output_____# Get predictions
test_loss, y_true, y_prob = trainer.eval_step(dataloader=test_dataloader)
y_pred = np.argmax(y_prob, axis=1)_____no_output_____# Determine performance
performance = get_performance(
    y_true=y_test, y_pred=y_pred, classes=label_encoder.classes)
print (json.dumps(performance['overall'], indent=2)){
  "precision": 0.8206922733621068,
  "recall": 0.8209444444444445,
  "f1": 0.8205583307860935,
  "num_samples": 18000.0
}
# Save artifacts
dir = Path("gru")
dir.mkdir(parents=True, exist_ok=True)
label_encoder.save(fp=Path(dir, 'label_encoder.json'))
tokenizer.save(fp=Path(dir, 'tokenizer.json'))
torch.save(best_model.state_dict(), Path(dir, 'model.pt'))
with open(Path(dir, 'performance.json'), "w") as fp:
    json.dump(performance, indent=2, sort_keys=False, fp=fp)_____no_output_____

## Inference_____no_output_____

def get_probability_distribution(y_prob, classes):
    """Create a dict of class probabilities from an array."""
    results = {}
    for i, class_ in enumerate(classes):
        results[class_] = np.float64(y_prob[i])
    sorted_results = {k: v for k, v in sorted(
        results.items(), key=lambda item: item[1], reverse=True)}
    return sorted_results_____no_output_____# Load artifacts
device = torch.device("cpu")
label_encoder = LabelEncoder.load(fp=Path(dir, 'label_encoder.json'))
tokenizer = Tokenizer.load(fp=Path(dir, 'tokenizer.json'))
model = GRU(
    embedding_dim=EMBEDDING_DIM, vocab_size=VOCAB_SIZE,
    rnn_hidden_dim=RNN_HIDDEN_DIM, hidden_dim=HIDDEN_DIM,
    dropout_p=DROPOUT_P, num_classes=NUM_CLASSES)
model.load_state_dict(torch.load(Path(dir, 'model.pt'), map_location=device))
model.to(device)_____no_output_____# Initialize trainer
trainer = Trainer(model=model, device=device)_____no_output_____# Dataloader
text = "The final tennis tournament starts next week."
X = tokenizer.texts_to_sequences([preprocess(text)])
print (tokenizer.sequences_to_texts(X))
y_filler = label_encoder.encode([label_encoder.classes[0]]*len(X))
dataset = Dataset(X=X, y=y_filler)
dataloader = dataset.create_dataloader(batch_size=batch_size)['final tennis tournament starts next week']
# Inference
y_prob = trainer.predict_step(dataloader)
y_pred = np.argmax(y_prob, axis=1)
label_encoder.decode(y_pred)_____no_output_____# Class distributions
prob_dist = get_probability_distribution(y_prob=y_prob[0], classes=label_encoder.classes)
print (json.dumps(prob_dist, indent=2)){
  "Sports": 0.9944021105766296,
  "World": 0.004813326057046652,
  "Sci/Tech": 0.0007053817971609533,
  "Business": 7.924934470793232e-05
}

> We will learn how to create more context-aware representations and a little bit of interpretability with RNNs in the next lesson on attention._____no_output_____


# Notebook from sundyCoder/STPD
Path: exp2_targeted_data_collection.ipynb

<code>
# Python Libraries
%matplotlib inline
import pickle
import numpy as np
import pandas as pd
import matplotlib
from keras.datasets import cifar10
from keras import backend as K

# Custom Networks
from networks.lenet import LeNet
from networks.pure_cnn import PureCnn
from networks.network_in_network import NetworkInNetwork
from networks.resnet import ResNet
from networks.densenet import DenseNet
from networks.wide_resnet import WideResNet
from networks.capsnet import CapsNet

# Helper functions
from differential_evolution import differential_evolution
import helper
from scipy.misc import imsave
import scipy.misc

matplotlib.style.use('ggplot')
np.random.seed(100)_____no_output_____(x_train, y_train), (x_test, y_test) = cifar10.load_data()
class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
_____no_output_____def perturb_image(xs, img):
    # If this function is passed just one perturbation vector,
    # pack it in a list to keep the computation the same
    if xs.ndim < 2:
        xs = np.array([xs])

    # Copy the image n == len(xs) times so that we can
    # create n new perturbed images
    tile = [len(xs)] + [1]*(xs.ndim+1)
    imgs = np.tile(img, tile)

    # Make sure to floor the members of xs as int types
    xs = xs.astype(int)

    for x,img in zip(xs, imgs):
        # Split x into an array of 5-tuples (perturbation pixels)
        # i.e., [[x,y,r,g,b], ...]
        pixels = np.split(x, len(x) // 5)
        for pixel in pixels:
            # At each pixel's x,y position, assign its rgb value
            x_pos, y_pos, *rgb = pixel
            img[x_pos, y_pos] = rgb

    return imgs_____no_output_____K.tensorflow_backend._get_available_gpus()
#lenet = LeNet()
#nin = NetworkInNetwork()
#resnet = ResNet()
densenet = DenseNet()

models = [densenet]
_____no_output_____def predict_classes(xs, img, target_class, model, minimize=True):
    # Perturb the image with the given pixel(s) x and get the prediction of the model
    imgs_perturbed = perturb_image(xs, img)
    predictions = model.predict(imgs_perturbed)[:,target_class]
    # This function should always be minimized, so return its complement if needed
    return predictions if minimize else 1 - predictions_____no_output_____def attack_success(x, img, target_class, model, targeted_attack=False, verbose=False):
    # Perturb the image with the given pixel(s) and get the prediction of the model
    attack_image = perturb_image(x, x_test[img])

    confidence = model.predict(attack_image)[0]
    predicted_class = np.argmax(confidence)

    # If the prediction is what we want (misclassification or
    # targeted classification), return True
    if (verbose):
        print('Confidence:', confidence[target_class])
    if ((targeted_attack and predicted_class == target_class) or
        (not targeted_attack and predicted_class != target_class)):
        return True
    # NOTE: return None otherwise (not False), due to how Scipy handles its callback function_____no_output_____count = 0
import os
def attack(img, model,cls_id, case_path, target=None, pixel_count=1,
           maxiter=75, popsize=400,verbose=False):
    # Change the target class based on whether this is a targeted attack or not
    targeted_attack = target is not None
    target_class = target if targeted_attack else y_test[img,0]

    # Define bounds for a flat vector of x,y,r,g,b values
    # For more pixels, repeat this layout
    bounds = [(0,32), (0,32), (0,256), (0,256), (0,256)] * pixel_count

    # Population multiplier, in terms of the size of the perturbation vector x
    popmul = max(1, popsize // len(bounds))

    # Format the predict/callback functions for the differential evolution algorithm
    predict_fn = lambda xs: predict_classes(
        xs, x_test[img], target_class, model, target is None)
    callback_fn = lambda x, convergence: attack_success(
        x, img, target_class, model, targeted_attack, verbose)

    # Call Scipy's Implementation of Differential Evolution
    attack_result = differential_evolution(
        predict_fn, bounds, maxiter=maxiter, popsize=popmul,
        recombination=1, atol=-1, callback=callback_fn, polish=False)

    # Calculate some useful statistics to return from this function
    attack_image = perturb_image(attack_result.x, x_test[img])[0]
    prior_probs = model.predict_one(x_test[img])
    predicted_probs = model.predict_one(attack_image)
    predicted_class = np.argmax(predicted_probs)
    actual_class = y_test[img,0]
    success = predicted_class != actual_class

#     if(success):
#         #count += 1
#         name = 'horrse_attacked_'+str(img)+'_'+str(actual_class) +'_'+str(predicted_class)+'.png'
#         save_success(attack_image,name)

    cdiff = prior_probs[actual_class] - predicted_probs[actual_class]
    import scipy.misc
    if(predicted_probs[actual_class] < 0.5):
    # Show the best attempt at a solution (successful or not)
        helper.plot_image(attack_image, actual_class, class_names, predicted_class)
        #saved

        cls_name = case_path + str(cls_id)+'_'+class_names[cls_id]
        ori_name = cls_name + '/original/'+str(img) + '_' + str(actual_class) + '.png'
        ori_path = cls_name + '/original/'
        if not os.path.exists(ori_path):
            #os.makedirs(Annotations_path)
            os.system('mkdir -p %s' % (ori_path))
        scipy.misc.imsave(ori_name, x_test[img])
        at_name = cls_name + '/attacked/'+str(img) +'_'+str(actual_class) +'_'+str(predicted_class)+'.png'
        at_path = cls_name + '/attacked/'
        if not os.path.exists(at_path):
            #os.makedirs(Annotations_path)
            os.system('mkdir -p %s' %(at_path))
        scipy.misc.imsave(at_name, attack_image)
        #np.savetxt('horse_cor_'+str(img)+'.txt', attack_result.x,delimiter=',')
        #np.savetxt('test.out', x, delimiter=',')
        print("success:", prior_probs[actual_class], predicted_probs[actual_class])
    else:
        ok_cls_name =  case_path+str(cls_id)+'_'+class_names[cls_id]
        ok_name = ok_cls_name + '/OK/'+str(img) +'_' + str(actual_class)+ '.png'
        ok_path = ok_cls_name + '/OK/'
        if not os.path.exists(ok_path):
            #os.makedirs(Annotations_path)
            os.system('mkdir -p %s' %(ok_path))
        scipy.misc.imsave(ok_name, x_test[img])
    # Show the best attempt at a solution (successful or not)
    helper.plot_image(attack_image, actual_class, class_names, predicted_class)

    return [model.name, pixel_count, img, actual_class, predicted_class, success, cdiff, prior_probs, predicted_probs, attack_result.x]_____no_output_____def attack_all(models, samples=500, pixels=(1), targeted=False, maxiter=75, popsize=400, verbose=False):
    results = []
    for model in models:
        model_results = []
        valid_imgs = correct_imgs[correct_imgs.name == model.name].img
        img_samples = np.random.choice(valid_imgs, samples, replace=False)

        #for pixel_count in pixels:
        for i,img in enumerate(img_samples):
            print(model.name, '- image', img, '-', i+1, '/', len(img_samples))
            targets = [None] if not targeted else range(10)

            for target in targets:
                if (targeted):
                    print('Attacking with target', class_names[target])
                    if (target == y_test[img,0]):
                        continue
                result = attack(img, model, target, pixel_count,
                                maxiter=maxiter, popsize=popsize,
                                verbose=verbose)
                model_results.append(result)

        results += model_results
        helper.checkpoint(results, targeted)
    return results_____no_output_____#l(models, samples=10, targeted=True)

image = 108
target_class = 1 # Integer in range 0-9
pixels = 1
model = lenet

print('Attacking with target', class_names[target_class])
_ = attack(image, model, target_class, pixel_count=pixels, verbose=True)
_____no_output_____
</code>


# Notebook from yelabucsf/scrna-parameter-estimation
Path: analysis/ifn_hbec/version3/coexpression/corr_in_stim.ipynb

# Coexpression of ISGs under stimulation_____no_output_____

import scanpy as sc
import seaborn as sns
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats
from pybedtools import BedTool
import pickle as pkl
%matplotlib inline
import itertools
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)_____no_output_____import sys
sys.path.append('/home/ssm-user/Github/scrna-parameter-estimation/dist/memento-0.0.6-py3.8.egg')
sys.path.append('/home/ssm-user/Github/misc-seq/miscseq')
import encode
import memento_____no_output_____data_path = '/data_volume/memento/hbec/'
fig_path = '/data/home/Github/scrna-parameter-estimation/figures/fig6/'_____no_output_____import matplotlib
matplotlib.rcParams['pdf.fonttype'] = 42
matplotlib.rcParams['ps.fonttype'] = 42_____no_output_____import matplotlib.pylab as pylab
params = {'legend.fontsize': 'medium',
         'axes.labelsize': 'medium',
         'axes.titlesize':'medium',
         'figure.titlesize':'medium',
         'xtick.labelsize':'small',
         'ytick.labelsize':'small'}
pylab.rcParams.update(params)_____no_output_____

### Read the processed RNA data

Focus on the club and bc/club cells and type I interferons for now.

Encode the timestamps to integers._____no_output_____

adata_processed = sc.read(data_path + 'HBEC_type_I_processed_deep.h5ad')_____no_output_____adata = sc.read(data_path + 'HBEC_type_I_filtered_counts_deep.h5ad')_____no_output_____adata = adata[:, ~adata.var.index.str.startswith('MT-')].copy()
# adata.obs['cell_type'] = adata.obs['cell_type'].apply(lambda x: x if x != 'basal/club' else 'bc')
# adata.obs['cell_type'] = adata.obs['cell_type'].apply(lambda x: x if x != 'ionocyte/tuft' else 'ion-tuft')/home/ssm-user/anaconda3/envs/single_cell/lib/python3.8/site-packages/pandas/core/arrays/categorical.py:2487: FutureWarning: The `inplace` parameter in pandas.Categorical.remove_unused_categories is deprecated and will be removed in a future version.
  res = method(*args, **kwargs)

adata.shape_____no_output_____converter = {'basal/club':'BC', 'basal':'B', 'ciliated':'C', 'goblet':'G', 'ionocyte/tuft':'IT', 'neuroendo':'N'}_____no_output_____adata.obs['ct'] = adata.obs['cell_type'].apply(lambda x: converter[x])
adata_processed.obs['ct'] = adata_processed.obs['cell_type'].apply(lambda x: converter[x])_____no_output_____

### Define ISG classes_____no_output_____

isg_classes = pd.read_csv('../isg_classes.csv')_____no_output_____t1_isg = isg_classes.query('overall_type =="type1"').gene.tolist()
t2_isg = isg_classes.query('overall_type =="type2"').gene.tolist()
shared_isg = isg_classes.query('overall_type == "shared"').gene.tolist()
all_isg = isg_classes[isg_classes.sum(axis=1) > 0].gene.tolist()_____no_output_____tf_df = pd.read_csv('../baseline/human_tf.txt', sep='\t')
tf_list = tf_df['Symbol'].tolist()
tf_list += ['CIITA', 'NLRC5']_____no_output_____# cytokines = pd.read_csv('../mart_export.txt')['Gene name'].tolist()
# cytokines = v_df.query('cytokine').gene.tolist()_____no_output_____

### Setup memento_____no_output_____

def assign_q(batch):

    if batch == 0:
        return 0.387*0.25
    elif batch == 1:
        return 0.392*0.25
    elif batch == 2:
        return 0.436*0.25
    else:
        return 0.417*0.25_____no_output_____adata.obs['q'] = adata.obs['batch'].apply(assign_q)_____no_output_____memento.setup_memento(adata, q_column='q', trim_percent=0.1)Version 0.0.6
# mhc_1 = []
# mhc_2 = []
# for g in included_de_genes:
#     if 'HLA' in g:
#         if len(g.split('-')[1]) > 1:
#             mhc_2.append(g)
#         else:
#             mhc_1.append(g)_____no_output_____

### STAT1 differential correlation_____no_output_____

to_test = []
for g in available_genes:
    if 'STAT' in g or 'IRF' in g:
        to_test.append(g)_____no_output_____cts = ['C', 'B', 'BC']
tps = ['6']
ct = ['C']
stims = ['gamma', 'beta']
for tp in tps:
    for stim in stims:

        adata_stim = adata.copy()[

            adata.obs.ct.isin(ct) & \
            adata.obs.stim.isin([stim, 'control']) & \
            adata.obs.time.isin(['0', tp])].copy()

        time_converter={0:0, int(tp):1}
        adata_stim.obs['time_step'] = adata_stim.obs['time'].astype(int).apply(lambda x: time_converter[x])
        adata_stim.obs['group'] = 1

        memento.create_groups(adata_stim, label_columns=['donor', 'time_step'])
        memento.compute_1d_moments(adata_stim, min_perc_group=.9)

        available_genes = adata_stim.var.index.tolist()
#         to_test = []
#         for g in available_genes:
#             if 'STAT' in g or 'IRF' in g:
#                 to_test.append(g)

#         available_genes = list(   set(available_genes)  - set(to_test) )

        memento.compute_2d_moments(adata_stim, list(itertools.product(['STAT1', 'STAT2', 'IRF9'], available_genes)))

        moment_df = memento.get_2d_moments(adata_stim, groupby='time_step')

        candidates = moment_df.query('time_step_0 > 0.3 | time_step_1 > 0.3')
        print(stim, tp, candidates.shape)
        memento.compute_2d_moments(adata_stim, list(zip(candidates['gene_1'], candidates['gene_2']))  )

        memento.ht_2d_moments(
            adata_stim,
            formula_like='1 + donor + time_step',
            treatment_col='time_step',
            num_boot=10000,
            verbose=1,
            num_cpus=94,
            resampling='permutation',
            approx=False)

        adata_stim.write(data_path + 'differential_utilization/{}_{}.h5ad'.format(stim, tp))/home/ssm-user/anaconda3/envs/single_cell/lib/python3.8/site-packages/pandas/core/arrays/categorical.py:2487: FutureWarning: The `inplace` parameter in pandas.Categorical.remove_unused_categories is deprecated and will be removed in a future version.
  res = method(*args, **kwargs)
/home/ssm-user/Github/scrna-parameter-estimation/dist/memento-0.0.6-py3.8.egg/memento/main.py:614: RuntimeWarning: invalid value encountered in true_divide
  groupby_corr[key] /= groupby_corr_count[key]
to_test = list(set(tf_list) & set(available_genes))_____no_output_____# First, get correlations without adjustment
stim = 'gamma'
ct = ['C']
adata_stim = adata.copy()[

    adata.obs.ct.isin(ct) & \
    adata.obs.stim.isin([stim, 'control']) & \
    adata.obs.time.isin(['0', '3'])].copy()
time_converter={0:0, int('3'):1}
adata_stim.obs['time_step'] = adata_stim.obs['time'].astype(int).apply(lambda x: time_converter[x])
adata_stim.obs['group'] = 1

memento.create_groups(adata_stim, label_columns=['donor', 'time_step'])
memento.compute_1d_moments(adata_stim, min_perc_group=.3)

available_genes = adata_stim.var.index.tolist()

# memento.compute_2d_moments(adata_stim, list(itertools.product(['STAT1'], available_genes)))
# moment_1d_df = memento.get_1d_moments(adata_stim, groupby='time_step')
# candidates = moment_1d_df.query('time_step_0 > 0.35 | time_step_1 > 0.35')
memento.compute_2d_moments(adata_stim, list(itertools.product(['STAT1', 'STAT2', 'IRF9'], available_genes)))
c_df = memento.get_2d_moments(adata_stim, groupby='time_step')
c_df['es'] = c_df['time_step_1'] - c_df['time_step_0']
c_df = c_df.merge(isg_classes, left_on='gene_2', right_on='gene')

sns.boxplot(x='gene_1', y='es', hue='overall_type',data=c_df.query('overall_type != "shared"'))
memento.ht_2d_moments(
    adata_stim,
    formula_like='1 + donor + time_step',
    treatment_col='time_step',
    num_boot=10000,
    verbose=1,
    num_cpus=94,
    resampling='permutation',
    approx=False)

rho_adj = memento.get_2d_ht_result(adata_stim)

# adata_stim.write(data_path + '{}_6_dc.h5ad'.format(stim))/home/ssm-user/anaconda3/envs/single_cell/lib/python3.8/site-packages/pandas/core/arrays/categorical.py:2487: FutureWarning: The `inplace` parameter in pandas.Categorical.remove_unused_categories is deprecated and will be removed in a future version.
  res = method(*args, **kwargs)
adata_stim.write(data_path + 'gamma_6_dc.h5ad')... storing 'memento_group' as categorical
moment_1d_df = memento.get_2d_moments(adata_stim, groupby='time_step')
candidates = moment_1d_df.query('time_step_0 > 0.3 | time_step_1 > 0.3')_____no_output_____len(candidates)_____no_output_____corr_df = memento.get_2d_ht_result(adata_stim)
# corr_df = corr_df.merge(candidates[['gene_1', 'gene_2']], on=['gene_1', 'gene_2'])
corr_df['dc_fdr'] = memento.util._fdrcorrect(corr_df['corr_pval'])
isg_corr_df = corr_df.merge(isg_classes[['gene', 'overall_type']], left_on='gene_2', right_on='gene')_____no_output_____sns.boxplot(x='overall_type', y='corr_coef', data=isg_corr_df)_____no_output_____isg_corr_df.shape_____no_output_____dc_genes = isg_corr_df.query('dc_fdr < 0.1 & corr_coef > 0').gene_2.tolist()_____no_output_____corr_df.shape_____no_output_____'IFIT1' in dc_genes_____no_output_____len(dc_genes)_____no_output_____dc_genes_____no_output_____

### Heatmap for correlations at stim_____no_output_____

ct = 'C'
stim = adata[(adata.obs.stim=='gamma') & adata.obs.ct.isin([ct])].copy().copy()
stim.obs['group'] = 1
memento.create_groups(stim, label_columns=['group', 'ct'])
memento.compute_1d_moments(stim, min_perc_group=.9) _____no_output_____included_de_genes = list(set(dc_genes) & set(stim.var.index))_____no_output_____memento.compute_2d_moments(stim, gene_pairs=list(itertools.combinations(included_de_genes,2)))
c_df = memento.get_2d_moments(stim, groupby='ct')_____no_output_____ct_c_df = c_df[['gene_1', 'gene_2', 'ct_C']]
ct_heatmap = pd.DataFrame(np.zeros((len(included_de_genes), len(included_de_genes))), columns=included_de_genes, index=included_de_genes)
for idx, row in ct_c_df.iterrows():

    ct_heatmap.loc[row['gene_1'], row['gene_2']] = row['ct_C']
    ct_heatmap.loc[row['gene_2'], row['gene_1']] = row['ct_C']_____no_output_____# Manual construction of linkage matrix
from scipy.spatial import distance
from scipy.cluster import hierarchy
from scipy.cluster.hierarchy import fcluster

row_linkage = hierarchy.linkage(distance.pdist(ct_heatmap), method='average')
cluster_assignments = fcluster(row_linkage, t=10, criterion='maxclust')
num_clusters = 5
top_clusters= dict(zip(pd.Series(cluster_assignments).value_counts().head(num_clusters).index, range(num_clusters)))
top_clusters_reverse= dict(zip(range(num_clusters), pd.Series(cluster_assignments).value_counts().head(num_clusters).index, ))

coarse_cluster_assignments = np.array([top_clusters[i] if i in top_clusters else -1 for i in cluster_assignments])_____no_output_____print(ct_heatmap.shape)
cg = sns.clustermap(ct_heatmap.fillna(0.0), center=0, vmax=0.3, metric='euclidean', row_linkage=row_linkage, col_linkage=row_linkage); plt.close()
row_inds = cg.dendrogram_row.reordered_ind
col_inds = cg.dendrogram_col.reordered_ind
print(ct_heatmap.shape)
(29, 29)
(29, 29)
def classify_color_isg(x):

    if x in shared_isg:
        return -1
    if x in t1_isg:
        return 0
    if x in t2_isg:
        return 1
    return 'impossible'_____no_output_____# For ISG type coloring
from matplotlib.colors import LinearSegmentedColormap

myColors = sns.color_palette('Paired', num_clusters+1)
cmap = LinearSegmentedColormap.from_list('Custom', myColors, len(myColors))
class_list = coarse_cluster_assignments[np.newaxis, :]_____no_output_____myColors_____no_output_____fig = plt.figure(figsize=(20,20))
ax1 = plt.subplot2grid((21,20), (1,0), colspan=19, rowspan=19)
ax2 = plt.subplot2grid((21,20), (0,0), colspan=19, rowspan=1)

sns.heatmap(ct_heatmap.iloc[row_inds, col_inds], cmap='viridis', vmax=1, vmin=0.1, ax=ax1, yticklabels=False, xticklabels=False, cbar_kws={"shrink": 0.5})
sns.heatmap(class_list[:, row_inds], cmap=cmap, ax=ax2, yticklabels=False, xticklabels=False, cbar_kws={"shrink": 10})
colorbar = ax2.collections[0].colorbar
colorbar.set_ticks([-1] + list(range(num_clusters+1)))
colorbar.set_ticklabels([-1] + list(range(num_clusters+1)))

# colorbar.set_ticklabels(['shared', 'type1', 'type2'])

# some_genes = list(set(mhc_1)&set(included_de_genes))
# for g in some_genes:
#     idx = np.where(np.array(row_inds)==ct_heatmap.index.get_loc(g))[0][0]
#     ax1.plot([0, len(included_de_genes)], [idx, idx], color='magenta')
#     ax1.plot([idx, idx],[0, len(included_de_genes)], color='magenta',)

# selected_genes = {}
# locations = [

# ]
# colors = sns.color_palette('husl', len(locations))
# i = 0

# for x1, x2 in locations:

#     ax1.plot([x1, x1],[0, len(included_de_genes)], color=colors[i],)
#     ax1.plot([x2, x2],[0, len(included_de_genes)], color=colors[i])

#     selected_genes[i]= ct_heatmap.index[np.array(row_inds)[x1:x2]]

#     i+=1_____no_output_____gene_sets = {}
for cluster in range(10):

    gene_sets[cluster] = ct_heatmap.index[cluster_assignments == top_clusters_reverse[cluster]].tolist()
    print(cluster, gene_sets[cluster])0 ['MT1E', 'AQP5', 'TF', 'IRF1', 'MSRB1', 'N4BP2L1', 'SLC6A8', 'NLRC5', 'WDFY1', 'SCO2', 'CXCL9', 'MMP25-AS1', 'CALHM6', 'TAPBPL']
1 ['PSME2', 'GBP3', 'HLA-E', 'PDIA3', 'PLAAT4', 'HLA-DQB1']
2 ['HLA-B', 'HLA-DMA']
3 ['GBP2']
4 ['HLA-DPB1']
moment_df = memento.get_2d_moments(adata_stim, groupby='time_step')_____no_output_____moment_df.query('gene_2 in @t1_isg')_____no_output_____ct = ['C']
tps = ['3', '6', '9', '24', '48']
for tp in tps:
    adata_stim = adata.copy()[

        adata.obs.ct.isin(ct) & \
        adata.obs.stim.isin(['beta', 'gamma']) & \
        adata.obs.time.isin(['3'])].copy()
    time_converter={0:0, int('3'):1}
    stim_converter={'beta':0, 'gamma':1}
    # adata_stim.obs['time_step'] = adata_stim.obs['time'].astype(int).apply(lambda x: time_converter[x])
    adata_stim.obs['stim_indicator'] = adata_stim.obs['stim'].apply(lambda x: stim_converter[x])

    adata_stim.obs['group'] = 1

    memento.create_groups(adata_stim, label_columns=['donor', 'stim_indicator', 'time'])
    memento.compute_1d_moments(adata_stim, min_perc_group=.3)
    m_df, c_df = memento.get_1d_moments(adata_stim, groupby='stim_indicator')

    available_genes = adata_stim.var.index.tolist()

    # memento.compute_2d_moments(adata_stim, list(itertools.product(['STAT1'], available_genes)))
    # moment_1d_df = memento.get_1d_moments(adata_stim, groupby='time_step')
    # candidates = moment_1d_df.query('time_step_0 > 0.35 | time_step_1 > 0.35')
    memento.compute_2d_moments(adata_stim, list(itertools.product(to_test, list(set(available_genes)-set(to_test)))))
    c_df = memento.get_2d_moments(adata_stim, groupby='stim_indicator')
    #     c_df['es'] = c_df['time_step_1'] - c_df['time_step_0']

    result = c_df.merge(m_df, left_on='gene_2', right_on='gene', suffixes=('_c', '_m'))
/home/ssm-user/anaconda3/envs/single_cell/lib/python3.8/site-packages/pandas/core/arrays/categorical.py:2487: FutureWarning: The `inplace` parameter in pandas.Categorical.remove_unused_categories is deprecated and will be removed in a future version.
  res = method(*args, **kwargs)
/home/ssm-user/Github/scrna-parameter-estimation/dist/memento-0.0.6-py3.8.egg/memento/main.py:523: RuntimeWarning: divide by zero encountered in log
/home/ssm-user/Github/scrna-parameter-estimation/dist/memento-0.0.6-py3.8.egg/memento/main.py:550: RuntimeWarning: divide by zero encountered in log
/home/ssm-user/Github/scrna-parameter-estimation/dist/memento-0.0.6-py3.8.egg/memento/main.py:614: RuntimeWarning: invalid value encountered in true_divide
def get_correlation():

    ct = ['C']
    tf = 'STAT1'
    adata_stim = adata.copy()[

        adata.obs.ct.isin(ct) & \
        adata.obs.stim.isin([stim, 'control'])].copy()
    time_converter={0:0, int('3'):1}
    adata_stim.obs['time_step'] = adata_stim.obs['time'].astype(int).apply(lambda x: time_converter[x])
    adata_stim.obs['group'] = 1

    memento.create_groups(adata_stim, label_columns=['donor', 'time_step'])
    memento.compute_1d_moments(adata_stim, min_perc_group=.3)
    m_df, c_df = memento.get_1d_moments(adata_stim, groupby='stim')

    available_genes = adata_stim.var.index.tolist()

    # memento.compute_2d_moments(adata_stim, list(itertools.product(['STAT1'], available_genes)))
    # moment_1d_df = memento.get_1d_moments(adata_stim, groupby='time_step')
    # candidates = moment_1d_df.query('time_step_0 > 0.35 | time_step_1 > 0.35')
    memento.compute_2d_moments(adata_stim, list(itertools.product(['STAT1', 'STAT2', 'IRF9', 'STAT3'], available_genes)))
    c_df = memento.get_2d_moments(adata_stim, groupby='time_step')
#     c_df['es'] = c_df['time_step_1'] - c_df['time_step_0']
    c_df = c_df.merge(isg_classes, left_on='gene_2', right_on='gene')
    c_df['stim'] = stim

    return c_df_____no_output_____gamma_corr = get_correlation('gamma')
beta_corr = get_correlation('beta')
corr = pd.concat([gamma_corr, beta_corr])/home/ssm-user/anaconda3/envs/single_cell/lib/python3.8/site-packages/pandas/core/arrays/categorical.py:2487: FutureWarning: The `inplace` parameter in pandas.Categorical.remove_unused_categories is deprecated and will be removed in a future version.
  res = method(*args, **kwargs)
/home/ssm-user/Github/scrna-parameter-estimation/dist/memento-0.0.6-py3.8.egg/memento/main.py:614: RuntimeWarning: invalid value encountered in true_divide
tf = 'STAT3'
sns.boxplot(x='overall_type', y='time_step_1', hue='stim',data=corr.query('overall_type != "shared" & gene_1 == "{}"'.format(tf)))
_____no_output_____corr.head(5)_____no_output_____tf = 'IRF9'
stats.ks_2samp(
    merged_corr.query('gene_1=="{}" & overall_type == "type1"'.format(tf)).time_step_1_beta,
    merged_corr.query('gene_1=="{}" & overall_type == "type1"'.format(tf)).time_step_1_gamma)_____no_output_____

### Pick out genes and run memento_____no_output_____

tf_df = pd.read_csv('../baseline/human_tf.txt', sep='\t')
tf_list = tf_df['Symbol'].tolist()
tf_list += ['CIITA', 'NLRC5']_____no_output_____'NLRC5' in tf_list_____no_output_____gene_sets = {}
for cluster in [9, 8, 6, 7, 14]:

    gene_sets[cluster] = ct_heatmap.index[cluster_assignments == top_clusters_reverse[cluster]].tolist()_____no_output_____
gene_set_results = {}

for gs_idx, gene_set in gene_sets.items():

    print('Starting', gene_set, '---------------------------------------------')

    stim = 'gamma'
    ct = 'C'
    adata_stim = adata.copy()[

        adata.obs.ct.isin([ct]) & \
        adata.obs.stim.isin([stim])].copy()
    adata_stim.obs['group'] = 1

    memento.create_groups(adata_stim, label_columns=['donor', 'group'])
    memento.compute_1d_moments(adata_stim, min_perc_group=.9)

    available_tfs = list(set(tf_list) & set(adata_stim.var.index))
    available_isgs= list(set(gene_set) & set(adata_stim.var.index))

    # moment_1d_df = memento.get_1d_moments(adata_stim, groupby='is_stim')

    memento.compute_2d_moments(adata_stim, list(itertools.product(available_tfs, available_isgs)))
    corr_df = memento.get_2d_moments(adata_stim, groupby='group')

    tf_avg_corrs = corr_df.groupby('gene_1').group_1.mean().sort_values(ascending=False)
    top_tfs = tf_avg_corrs[tf_avg_corrs > 0.25].index.tolist()

    num_strata = 5

    gene_set_results[gs_idx] = {}

    for tf in top_tfs:

        adata_stim = adata.copy()[
            adata.obs.ct.isin([ct]) & \
            adata.obs.stim.isin([stim])].copy()

        print('     working on ', tf)
        expr = adata_stim[:, tf].X.todense().A1
        try:
            discretized = pd.qcut(pd.Series(expr), q=num_strata, labels=range(num_strata),duplicates='raise').values
        except:
            discretized = pd.Series(expr != 0).astype(int).values

        adata_stim.obs['tf_strata'] = discretized

        memento.create_groups(adata_stim, label_columns=['donor', 'tf_strata'])
        memento.compute_1d_moments(adata_stim, min_perc_group=.7)

        memento.compute_2d_moments(adata_stim, list(itertools.combinations(available_isgs, 2)))

        memento.ht_2d_moments(
            adata_stim,
            formula_like='1 + donor + tf_strata',
            treatment_col='Inter',
            num_boot=10000,
            verbose=1,
            num_cpus=94,
            resampling='bootstrap',
            approx=False)

        memento.get_2d_ht_result(adata_stim).to_csv(data_path + 'adjustment/{}_{}.csv'.format(gs_idx, tf), index=False)Starting ['UBE2L6', 'IFI30', 'IFITM3', 'NMI', 'GSTK1', 'PSMB9', 'BST2', 'PLAAT4', 'GBP3', 'PSME2', 'PSMB10', 'IFITM1', 'PSMB8', 'PSME1'] ---------------------------------------------
tf_avg_corrs = corr_df.groupby('gene_1').group_1.mean().sort_values(ascending=False)
top_tfs = tf_avg_corrs[tf_avg_corrs > 0.25].index.tolist()_____no_output_____
for tf in top_tfs:
    tf = 'RFX5'
    expr = adata_stim[:, tf].X.todense().A1
    try:
        discretized = pd.qcut(pd.Series(expr), q=num_strata, labels=range(num_strata),duplicates='raise').values
    except:
        discretized = pd.Series(expr != 0).astype(int).values
    break/home/ssm-user/anaconda3/envs/single_cell/lib/python3.8/site-packages/pandas/core/arrays/categorical.py:2487: FutureWarning: The `inplace` parameter in pandas.Categorical.remove_unused_categories is deprecated and will be removed in a future version.
  res = method(*args, **kwargs)
for t in available_tfs:

    t = 'STAT1'
    expr = adata_stim[:, t].X.todense().A1
    bins = np.linspace(0, expr.max(), 3)
    classes = np.digitize(expr, bins)

    break/home/ssm-user/anaconda3/envs/single_cell/lib/python3.8/site-packages/pandas/core/arrays/categorical.py:2487: FutureWarning: The `inplace` parameter in pandas.Categorical.remove_unused_categories is deprecated and will be removed in a future version.
  res = method(*args, **kwargs)
pd.qcut(pd.Series(expr), q=4, labels=range(4),duplicates='raise').value_counts()_____no_output_____pd.qcut(pd.Series(expr), q=4, labels=list(range(4))).value_counts()_____no_output_____plt.hist(adata_stim[:, 'STAT2'].X.todense().A1)_____no_output_____corr_df.groupby('gene_1').group_1.mean().sort_values(ascending=False).head(100)_____no_output_____


# Notebook from cbreenmachine/wgbs-load
Path: analysis/2022-01-DSS/liftover-published-DMPs.ipynb

<code>
library(tidyverse)
library(GenomicRanges)
library(liftOver)Loading required package: stats4

Loading required package: BiocGenerics

Loading required package: parallel

Attaching package: ‘BiocGenerics’

The following objects are masked from ‘package:parallel’:

    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
    clusterExport, clusterMap, parApply, parCapply, parLapply,
    parLapplyLB, parRapply, parSapply, parSapplyLB

The following objects are masked from ‘package:dplyr’:

    combine, intersect, setdiff, union

The following objects are masked from ‘package:stats’:

    IQR, mad, sd, var, xtabs

The following objects are masked from ‘package:base’:

    anyDuplicated, append, as.data.frame, basename, cbind, colnames,
    dirname, do.call, duplicated, eval, evalq, Filter, Find, get, grep,
    grepl, intersect, is.unsorted, lapply, Map, mapply, match, mget,
    order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
    rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply,
    union, unique, unsplit, which.max, which.min

Loading required package: S4Vectors

Attaching package: ‘S4Vectors’

The following objects are masked from ‘package:dplyr’:

    first, rename

The following object is masked from ‘package:tidyr’:

    expand

The following object is masked from ‘package:base’:

    expand.grid

Loading required package: IRanges

Attaching package: ‘IRanges’

The following objects are masked from ‘package:dplyr’:

    collapse, desc, slice

The following object is masked from ‘package:purrr’:

    reduce

Loading required package: GenomeInfoDb

Loading required package: gwascat

gwascat loaded.  Use makeCurrentGwascat() to extract current image.

 from EBI.  The data folder of this package has some legacy extracts.

Loading required package: rtracklayer

Loading required package: Homo.sapiens

Loading required package: AnnotationDbi

Loading required package: Biobase

Welcome to Bioconductor

    Vignettes contain introductory material; view with
    'browseVignettes()'. To cite Bioconductor, see
    'citation("Biobase")', and for packages 'citation("pkgname")'.

Attaching package: ‘AnnotationDbi’

The following object is masked from ‘package:dplyr’:

    select

Loading required package: OrganismDbi

Loading required package: GenomicFeatures

Loading required package: GO.db

Loading required package: org.Hs.eg.db

Loading required package: TxDb.Hsapiens.UCSC.hg19.knownGene

chain <- import.chain("../2021-11-850K-comparisons/hg19ToHg38.over.chain")

gr <- read_csv("pub-2018Madrid-DMPs.csv", col_types = cols()) %>%
        mutate(stop = pos + 1)%>%
        makeGRangesFromDataFrame(df, seqnames.field = "chrom", start.field = "pos",
                                 seqinfo = NULL, end.field = "stop",
                                 keep.extra.columns = TRUE, ignore.strand = TRUE)
gr_____no_output_____gr.lifted <- liftOver(gr, chain = chain)_____no_output_____write_csv(as.data.frame(gr.lifted), "pub-2018Madrid-DMPs.lifted.csv")_____no_output_____
</code>