# Notebook from sjoerdapp/colab
Path: Mandatory_models.ipynb

# Practical 2: Encoding Sentences with Neural Models_____no_output_____In this practical we will train neural models to encode sentences, after which we can use our sentence representation for a downstream task such as sentiment classification. Rather than simply answering questions, like in the previous lab, **this time you are expected to write a four-page scientific report with your findings**. You will be judged by the quality of your report (see below). In this notebook, we will help you to develop models for your experiments. You do not have to hand in your answers, but please do include a link to a private Github gist with your code in your report as a footnote.

### Data set
We will make use of the [Stanford Sentiment Treebank](https://nlp.stanford.edu/sentiment/) (SST), which provides sentences, their tree structure, and (fine-grained) sentiment scores.
This dataset is different from the one we used in the first practical.
Before, a review consisted of several sentences, and we would have one sentiment score for it. Now, a review is a single sentence, for which we get a sentiment score. The special thing about our new data set is that we get a binary parse tree for each sentence, and a sentiment score has been assigned to each node in the tree. (We will look at an example below.)

For the first part of this practical we will only make use of the tokens, but in the end we will also exploit the tree-structure that is provided!

We will cover the following approaches:

- Bag-of-words (BOW)
- Continuous bag-of-words (CBOW)
- Deep continuous bag-of-words (Deep CBOW)
- LSTM
- TreeLSTM
_____no_output_____### Important: Report

The main purpose of this lab is for you to learn how to answer research questions by experimenting and then writing a scientific report.
Your grade will be based on the quality of your report, not on the exercises in this notebook.
You can find the requirements for the report at the end of this notebook._____no_output_____## Preparing the data_____no_output_____Let's first download the data set and take a look._____no_output_____

import re
import random
import time
import math
import numpy as np
import nltk
import os
import matplotlib.pyplot as plt
plt.style.use('default')

CHECKPOINT_DIR = "checkpoints/"
if not os.path.exists(CHECKPOINT_DIR):
    os.makedirs(CHECKPOINT_DIR)

TRAIN_ALL_MODELS = False_____no_output_____# !wget http://nlp.stanford.edu/sentiment/trainDevTestTrees_PTB.zip
# !unzip trainDevTestTrees_PTB.zip_____no_output_____# this function reads in a textfile and fixes an issue with \\
def filereader(path):
  with open(path, mode="r", encoding="utf-8") as f:
    for line in f:
      yield line.strip().replace("\\","")_____no_output_____# Let's look at a data point.
# If you look closely you will find that this is a flattened tree,
# with sentiment scores at every node, and words as the leaves.
s = next(filereader("trees/dev.txt"))
print(s)(3 (2 It) (4 (4 (2 's) (4 (3 (2 a) (4 (3 lovely) (2 film))) (3 (2 with) (4 (3 (3 lovely) (2 performances)) (2 (2 by) (2 (2 (2 Buy) (2 and)) (2 Accorsi))))))) (2 .)))
# We can use NLTK to print the tree structure more nicely
from nltk import Tree
from nltk.treeprettyprinter import TreePrettyPrinter
tree = Tree.fromstring(s)
print(TreePrettyPrinter(tree))              3
  ____________|____________________
 |                                 4
 |        _________________________|______________________________________________
 |       4                                                                        |
 |    ___|______________                                                          |
 |   |                  4                                                         |
 |   |         _________|__________                                               |
 |   |        |                    3                                              |
 |   |        |               _____|______________________                        |
 |   |        |              |                            4                       |
 |   |        |              |            ________________|_______                |
 |   |        |              |           |                        2               |
 |   |        |              |           |                 _______|___            |
 |   |        3              |           |                |           2           |
 |   |    ____|_____         |           |                |        ___|_____      |
 |   |   |          4        |           3                |       2         |     |
 |   |   |     _____|___     |      _____|_______         |    ___|___      |     |
 2   2   2    3         2    2     3             2        2   2       2     2     2
 |   |   |    |         |    |     |             |        |   |       |     |     |
 It  's  a  lovely     film with lovely     performances  by Buy     and Accorsi  .

You can see the sentiment annotations here **at every node**! 0 is very negative, 4 is very positive. For now, we will only use the sentiment score at the **root node**; this is the score for the complete sentence._____no_output_____

# Let's first make a function that extracts the tokens (the leaves).

def tokens_from_treestring(s):
  """extract the tokens from a sentiment tree"""
  return re.sub(r"\([0-9] |\)", "", s).split()

# let's try it on our example tree
tokens = tokens_from_treestring(s)
print(tokens)
print(len(tokens))['It', "'s", 'a', 'lovely', 'film', 'with', 'lovely', 'performances', 'by', 'Buy', 'and', 'Accorsi', '.']
13

> *Warning: you could also parse a treestring using NLTK and ask it to return the leaves, but there seems to be an issue with NLTK not always correctly parsing the input, so do not rely on it.*_____no_output_____

# We will also need the following function, but you can ignore this for now.
# It is explained later on.

SHIFT = 0
REDUCE = 1

def transitions_from_treestring(s):
  s = re.sub("\([0-5] ([^)]+)\)", "0", s)
  s = re.sub("\)", " )", s)
  s = re.sub("\([0-4] ", "", s)
  s = re.sub("\([0-4] ", "", s)
  s = re.sub("\)", "1", s)
  return list(map(int, s.split()))_____no_output_____# Now let's first see how large our data sets are.
for path in ("trees/train.txt", "trees/dev.txt", "trees/test.txt"):
  print("{:16s} {:4d}".format(path, sum(1 for _ in filereader(path))))trees/train.txt  8544
trees/dev.txt    1101
trees/test.txt   2210

You can see that this data set is not very large! That's probably because it required so much manual annotation. However, it's large enough to train a neural network on.

It will be useful to store each data example in an `Example` object,
containing everything that we may need for each data point.
It will contain the tokens, the tree, the top-level sentiment label, and
the transitions (explained later)._____no_output_____

from collections import namedtuple
from nltk import Tree

# A simple way to define a class is using namedtuple.
Example = namedtuple("Example", ["tokens", "tree", "label", "transitions"])

def examplereader(path, lower=False):
  """Returns all examples in a file one by one."""
  for line in filereader(path):
    line = line.lower() if lower else line
    tokens = tokens_from_treestring(line)
    tree = Tree.fromstring(line)  # use NLTK's Tree
    label = int(line[1])
    trans = transitions_from_treestring(line)
    yield Example(tokens=tokens, tree=tree, label=label, transitions=trans)

# Let's load the data into memory.
LOWER = False  # we will keep the original casing
train_data = list(examplereader("trees/train.txt", lower=LOWER))
dev_data = list(examplereader("trees/dev.txt", lower=LOWER))
test_data = list(examplereader("trees/test.txt", lower=LOWER))

print("train", len(train_data))
print("dev", len(dev_data))
print("test", len(test_data))train 8544
dev 1101
test 2210

Let's check out an example object._____no_output_____

example = dev_data[0]
print("First train example:", example)
print("First train example tokens:", example.tokens)
print("First train example label:",  example.label)First train example: Example(tokens=['It', "'s", 'a', 'lovely', 'film', 'with', 'lovely', 'performances', 'by', 'Buy', 'and', 'Accorsi', '.'], tree=Tree('3', [Tree('2', ['It']), Tree('4', [Tree('4', [Tree('2', ["'s"]), Tree('4', [Tree('3', [Tree('2', ['a']), Tree('4', [Tree('3', ['lovely']), Tree('2', ['film'])])]), Tree('3', [Tree('2', ['with']), Tree('4', [Tree('3', [Tree('3', ['lovely']), Tree('2', ['performances'])]), Tree('2', [Tree('2', ['by']), Tree('2', [Tree('2', [Tree('2', ['Buy']), Tree('2', ['and'])]), Tree('2', ['Accorsi'])])])])])])]), Tree('2', ['.'])])]), label=3, transitions=[0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1])
First train example tokens: ['It', "'s", 'a', 'lovely', 'film', 'with', 'lovely', 'performances', 'by', 'Buy', 'and', 'Accorsi', '.']
First train example label: 3

#### Vocabulary
To work with this data we will need a vocabulary.
The job of the vocabulary is to map each word to a unique ID.
_____no_output_____

# Here we first define a class that can map a word to an ID (w2i)
# and back (i2w).

from collections import Counter, OrderedDict, defaultdict

class OrderedCounter(Counter, OrderedDict):
  """Counter that remembers the order elements are first seen"""
  def __repr__(self):
    return '%s(%r)' % (self.__class__.__name__,
                      OrderedDict(self))
  def __reduce__(self):
    return self.__class__, (OrderedDict(self),)

class Vocabulary:
  """A vocabulary, assigns IDs to tokens"""

  def __init__(self):
    self.freqs = OrderedCounter()
    self.w2i = {}
    self.i2w = []

  def count_token(self, t):
    self.freqs[t] += 1

  def add_token(self, t):
    self.w2i[t] = len(self.w2i)
    self.i2w.append(t)

  def build(self, min_freq=0):
    self.add_token("<unk>")  # reserve 0 for <unk> (unknown words)
    self.add_token("<pad>")  # reserve 1 for <pad> (discussed later)

    tok_freq = list(self.freqs.items())
    tok_freq.sort(key=lambda x: x[1], reverse=True)
    for tok, freq in tok_freq:
      if freq >= min_freq:
        self.add_token(tok)_____no_output_____

Let's build the token vocabulary!

When randomly initializing word vectors, we take the words in our training
set and assign them unique IDs, and assign all other words to <unk>,
because those we cannot learn a vector for based on our training data.
_____no_output_____

# This process should be deterministic and should have the same result
# if run multiple times on the same data set.

v = Vocabulary()
for data_set in (train_data,):
  for ex in data_set:
    for token in ex.tokens:
      v.count_token(token)

v.build()
print("Vocabulary size:", len(v.w2i))Vocabulary size: 18280

Let's have a closer look at the properties of our vocabulary. Having a good idea of what it is like can facilitate data analysis and debugging later on._____no_output_____

# What is the ID for "century?"
print(v.w2i["century"])1973
# What are the first 10 words in the vocabulary?
for i in range(10):
    print("Word " + str(i) + ": " + v.i2w[i])Word 0: <unk>
Word 1: <pad>
Word 2: .
Word 3: ,
Word 4: the
Word 5: and
Word 6: a
Word 7: of
Word 8: to
Word 9: 's
# What are the 10 most common words?
common_indices_list = sorted([(i, v.freqs[v.i2w[i]]) for i in range(len(v.freqs))], key=lambda x: -x[1])
for i in range(10):
    print("Frequency top " + str(i+1).rjust(2) + " (" + str(common_indices_list[i][1]) + "): " + v.i2w[common_indices_list[i][0]])Frequency top  1 (8024): .
Frequency top  2 (7131): ,
Frequency top  3 (6037): the
Frequency top  4 (4431): and
Frequency top  5 (4403): a
Frequency top  6 (4386): of
Frequency top  7 (2995): to
Frequency top  8 (2544): 's
Frequency top  9 (2536): is
Frequency top 10 (1915): that
# And how many words are there with frequency 1?
no_freq_1 = sum([v.freqs[v.i2w[i]]==1 for i in range(len(v.freqs))])
print("Words with frequency 1: " + str(no_freq_1))Words with frequency 1: 9541
# Finally 20 random words from the vocabulary.
# This is a simple way to get a feeling for the data.
for i in range(20):
    word_index = random.randint(0,len(v.freqs)-1)
    print(v.i2w[word_index])spice
commanding
Meow
pastry
140
inconsequential
drew
existentialism
deliberative
revelled
breathtaking
mind-bending
sugar-coated
twin
stability
bed
today
another
stuff
conduits

#### Sentiment label vocabulary_____no_output_____

# Now let's map the sentiment labels 0-4 to a more readable form
i2t = ["very negative", "negative", "neutral", "positive", "very positive"]
i2t_____no_output_____# And let's also create the opposite mapping.
# We won't use a Vocabulary for this (although we could), since the labels
# are already numeric.
t2i = OrderedDict({p : i for p, i in zip(i2t, range(len(i2t)))})
t2i_____no_output_____

## Installing PyTorch

We are going to need PyTorch and Google Colab does not have it installed by default. Run the cell below to install it.

*For installing PyTorch in your own computer, follow the instructions on [pytorch.org](pytorch.org) instead. This is for Google Colab only.*_____no_output_____

# http://pytorch.org/
from os.path import exists
from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
import torch
from torch import optim
from torch import nn_____no_output_____# this should result in the PyTorch version
torch.__version_______no_output_____# PyTorch can run on CPU or on Nvidia GPU (video card) using CUDA
# This cell selects the GPU if one is available.
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
device_____no_output_____# Seed manually to make runs reproducible
# You need to set this again if you do multiple runs of the same model
torch.manual_seed(0)

# When running on the CuDNN backend two further options must be set for reproducibility
if torch.cuda.is_available():
  torch.backends.cudnn.deterministic = True
  torch.backends.cudnn.benchmark = False_____no_output_____

# BOW_____no_output_____Our first model is a super simple neural **bag-of-words (BOW) model**.
Unlike the bag-of-words model that you used in the previous lab, here we associate each word with a vector of size 5, exactly our number of sentiment classes.

To make a classification, we simply **sum** the vectors of the words in our sentence and a bias vector. Because we sum the vectors, we lose word order: that's why we call this a neural bag-of-words model.

```
this   [0.0, 0.1, 0.1, 0.1, 0.0]
movie  [0.1, 0.1, 0.1, 0.2, 0.1]
is     [0.0, 0.1, 0.0, 0.0, 0.0]
stupid [0.9, 0.5, 0.1, 0.0, 0.0]

bias   [0.0, 0.0, 0.0, 0.0, 0.0]
--------------------------------
sum    [0.9, 0.8, 0.3, 0.3, 0.1]

argmax: 0 (very negative)
```

Now, the **argmax** of this sum is our predicted label.

We initialize all vectors *randomly* and train them using cross-entropy loss._____no_output_____#### Model class_____no_output_____

class BOW(nn.Module):
  """A simple bag-of-words model"""

  def __init__(self, vocab_size, embedding_dim, vocab):
    super(BOW, self).__init__()
    self.vocab = vocab

    # this is a trained look-up table with word embeddings
    self.embed = nn.Embedding(vocab_size, embedding_dim)

    # this is a trained bias term
    self.bias = nn.Parameter(torch.zeros(embedding_dim), requires_grad=True)

  def forward(self, inputs):
    # this is the forward pass of the neural network
    # given inputs, it computes the output

    # this looks up the embeddings for each word ID in inputs
    # the result is a sequence of word embeddings
    embeds = self.embed(inputs)

    # the output is the sum across the time dimension (1)
    # with the bias term added
    logits = embeds.sum(dim=1) + self.bias

    return logits_____no_output_____# Let's create a model.
vocab_size = len(v.w2i)
n_classes = len(t2i)
bow_model = BOW(vocab_size, n_classes, v)
print(bow_model)BOW(
  (embed): Embedding(18280, 5)
)

> **Hey, wait, where is the bias vector?**
> PyTorch does not print Parameters, only Modules!

> We can print it ourselves though, to check that it is there._____no_output_____

# Here we print each parameter name, shape, and if it is trainable.
def print_parameters(model):
  total = 0
  for name, p in model.named_parameters():
    total += np.prod(p.shape)
    print("{:24s} {:12s} requires_grad={}".format(name, str(list(p.shape)), p.requires_grad))
  print("\nTotal parameters: {}\n".format(total))

print_parameters(bow_model)bias                     [5]          requires_grad=True
embed.weight             [18280, 5]   requires_grad=True

Total parameters: 91405

#### Preparing an example for input

To feed sentences to our PyTorch model, we need to convert a sequence of tokens to a sequence of IDs. The `prepare_example` function below takes care of this for us. With these IDs we index the word embedding table. _____no_output_____

def prepare_example(example, vocab):
  """
  Map tokens to their IDs for 1 example
  """

  # vocab returns 0 if the word is not there
  x = [[vocab.w2i.get(t, 0) for t in example.tokens]]

  x = torch.LongTensor(x)
  x = x.to(device)

  y = torch.LongTensor([example.label])
  y = y.to(device)

  return x, y_____no_output_____

#### Evaluation
We will need one more thing: an evaluation metric.
How many predictions do we get right? The accuracy will tell us.
Make sure that you understand this code block.
_____no_output_____

def simple_evaluate(model, data, prep_fn=prepare_example, **kwargs):
  """Accuracy of a model on given data set."""
  correct = 0
  total = 0
  model.eval()  # disable dropout (explained later)

  for example in data:

    # convert the example input and label to PyTorch tensors
    x, target = prep_fn(example, model.vocab)

    # forward pass
    # get the output from the neural network for input x
    with torch.no_grad():
      logits = model(x)

    # get out the prediction
    prediction = logits.argmax(dim=-1)

    # add the number of correct predictions to the total correct
    correct += (prediction == target).sum().item()
    total += 1

    del x
    del target
    torch.cuda.empty_cache()

  return correct, total, correct / float(total)_____no_output_____

#### Example feed
For stochastic gradient descent (SGD) we will need a random training example for every update.
We implement this by shuffling the training data and returning examples one by one using `yield`.

Shuffling is optional so that we get to use this to get validation and test examples, too._____no_output_____

def get_examples(data, shuffle=True, **kwargs):
  """Shuffle data set and return 1 example at a time (until nothing left)"""
  indices = list(range(len(data)))
  if shuffle:
    print("Shuffling training data")
    random.shuffle(indices)  # shuffle training data each epoch
  for index in indices:
    yield data[index]_____no_output_____

#### Exercise: Training function

Your task is now to complete the training loop below.
Before you do so, please read the section about optimization._____no_output_____**Optimization**

As mentioned in the "Introduction to PyTorch" notebook, one of the perks of using PyTorch are its automatic differentiation abilities. We will use these to train our BOW model.

We train our model by feeding it an input and performing **forward** pass, which results in an output for which we obtain a **loss** with our loss function.
After the gradients are calculated in the **backward** pass, we can take a step on the loss surface towards more optimal parameter settings (e.g. gradient descent).

The package we will use to do this optimization is [torch.optim](https://pytorch.org/docs/stable/optim.html). Besides implementations of stochastic gradient descent (SGD), this package also implements the optimization algorithm Adam, which we'll be using in this practical.
For the purposes of this assignment you do not need to know what Adam does besides that it uses gradient information to update our model parameters by calling:

```
optimizer.step()
```
Remember when we updated our parameters in the PyTorch tutorial in a loop?

```python
# update weights
learning_rate = 0.5
for f in net.parameters():
    # for each parameter, take a small step in the opposite dir of the gradient
    p.data = p.data - p.grad.data * learning_rate

```
The function call optimizer.step() does effectively the same thing.

*(If you want to know more about optimization algorithms using gradient information [this blog](http://ruder.io/optimizing-gradient-descent/.) gives a nice intuitive overview.)*_____no_output_____

def train_model(model, optimizer, num_iterations=10000,
                print_every=1000, eval_every=1000,
                batch_fn=get_examples,
                prep_fn=prepare_example,
                eval_fn=simple_evaluate,
                batch_size=1, eval_batch_size=None,
                file_prefix=None):
    """Train a model."""
    iter_i = 0
    train_loss = 0.
    print_num = 0
    start = time.time()
    criterion = nn.CrossEntropyLoss() # loss function
    best_eval = 0.
    best_iter = 0

    if not file_prefix:
        checkpoint_path = CHECKPOINT_DIR + "{}.pt".format(model.__class__.__name__)
    else:
        checkpoint_path = CHECKPOINT_DIR + file_prefix + ".pt"

    # store train loss and validation accuracy during training
    # so we can plot them afterwards
    losses = []
    accuracies = []

    if eval_batch_size is None:
        eval_batch_size = batch_size

    while True:  # when we run out of examples, shuffle and continue
        for batch in batch_fn(train_data, batch_size=batch_size):

            # forward pass
            model.train()
            x, targets = prep_fn(batch, model.vocab)
            logits = model(x)

            B = targets.size(0)  # later we will use B examples per update

            # compute cross-entropy loss (our criterion)
            # note that the cross entropy loss function computes the softmax for us
            loss = criterion(logits.view([B, -1]), targets.view(-1))
            train_loss += loss.item()

            # backward pass
            # Tip: check the Introduction to PyTorch notebook.

            # erase previous gradients
            model.zero_grad()

            loss.backward()

            # update weights - take a small step in the opposite dir of the gradient
            optimizer.step()

            print_num += 1
            iter_i += 1

            # print info
            if iter_i % print_every == 0:
                print("Iter %r: loss=%.4f, time=%.2fs" %
                (iter_i, train_loss, time.time()-start))
                losses.append(train_loss)
                print_num = 0
                train_loss = 0.

            # evaluate
            if iter_i % eval_every == 0:
                _, _, accuracy = eval_fn(model, dev_data, batch_size=eval_batch_size,
                         batch_fn=batch_fn, prep_fn=prep_fn)
                accuracies.append(accuracy)
                print("iter %r: dev acc=%.4f" % (iter_i, accuracy))

                # save best model parameters
                if accuracy > best_eval:
                    print("new highscore")
                    best_eval = accuracy
                    best_iter = iter_i
                    ckpt = {
                    "state_dict": model.state_dict(),
                    "optimizer_state_dict": optimizer.state_dict(),
                    "best_eval": best_eval,
                    "best_iter": best_iter
                    }
                    torch.save(ckpt, checkpoint_path)

            del x
            del targets
            torch.cuda.empty_cache()

            # done training
            if iter_i == num_iterations:
                print("Done training")

                # evaluate on train, dev, and test with best model
                print("Loading best model")
                ckpt = torch.load(checkpoint_path)
                model.load_state_dict(ckpt["state_dict"])

                _, _, train_acc = eval_fn(
                model, train_data, batch_size=eval_batch_size,
                batch_fn=batch_fn, prep_fn=prep_fn)
                _, _, dev_acc = eval_fn(
                model, dev_data, batch_size=eval_batch_size,
                batch_fn=batch_fn, prep_fn=prep_fn)
                _, _, test_acc = eval_fn(
                model, test_data, batch_size=eval_batch_size,
                batch_fn=batch_fn, prep_fn=prep_fn)

                print("best model iter {:d}: "
                "train acc={:.4f}, dev acc={:.4f}, test acc={:.4f}".format(
                best_iter, train_acc, dev_acc, test_acc))

                return losses, accuracies, test_acc

def evaluate_model(model,
                batch_fn=get_examples,
                prep_fn=prepare_example,
                eval_fn=simple_evaluate,
                batch_size=1, eval_batch_size=None,
                file_prefix=None):
    """Evaluate a model."""

    if not file_prefix:
        checkpoint_path = CHECKPOINT_DIR + "{}.pt".format(model.__class__.__name__)
    else:
        checkpoint_path = CHECKPOINT_DIR + file_prefix + ".pt"

    # evaluate on train, dev, and test with best model
    print("Loading model")
    ckpt = torch.load(checkpoint_path)
    model.load_state_dict(ckpt["state_dict"])

    _, _, train_acc = eval_fn(
    model, train_data, batch_size=eval_batch_size,
    batch_fn=batch_fn, prep_fn=prep_fn)
    _, _, dev_acc = eval_fn(
    model, dev_data, batch_size=eval_batch_size,
    batch_fn=batch_fn, prep_fn=prep_fn)
    _, _, test_acc = eval_fn(
    model, test_data, batch_size=eval_batch_size,
    batch_fn=batch_fn, prep_fn=prep_fn)

    print("best model: "
    "train acc={:.4f}, dev acc={:.4f}, test acc={:.4f}".format(
    train_acc, dev_acc, test_acc))

    return train_acc, dev_acc, test_acc_____no_output_____def mean(l):
    if not l or len(l)==0:
        return 0
    else:
        return sum(l) * 1.0 / len(l)

def std(l):
    if not l or len(l) == 0:
        return 0
    mu = mean(l)
    std = np.sqrt(sum([(x - mu)**2 for x in l]) * 1.0 / len(l))
    return std_____no_output_____

### Training the BOW model_____no_output_____

def train_model_default(model_fn, lr, file_prefix, num_seeds=10, batch_size=1, batch_fn=None, prep_fn=None, eval_fn=None, num_iterations=300000, eval_every=10000):
    for i in range(num_seeds):
        if not TRAIN_ALL_MODELS and os.path.exists(CHECKPOINT_DIR + file_prefix + "_"+str(i).zfill(2)+".pt"):
            print("Skipping " + CHECKPOINT_DIR + file_prefix + "_"+str(i).zfill(2)+".pt")
            continue
        torch.manual_seed(i)
        np.random.seed(i)
        random.seed(i)

        model = model_fn()
        model = model.to(device)
        optimizer = optim.Adam(model.parameters(), lr=lr)
        if not batch_fn or not prep_fn or not eval_fn:
            _,_,_ = train_model(
                model, optimizer, num_iterations=num_iterations,
                print_every=10000, eval_every=eval_every, batch_size = batch_size, file_prefix=file_prefix + "_"+str(i).zfill(2))
        else:
            _,_,_ = train_model(
                model, optimizer, num_iterations=num_iterations,
                print_every=10000, eval_every=eval_every, batch_size = batch_size, file_prefix=file_prefix + "_"+str(i).zfill(2),batch_fn=batch_fn,
                prep_fn=prep_fn,eval_fn=eval_fn)

        del model
        del optimizer
        torch.cuda.empty_cache()  _____no_output_____def eval_model_default(model, file_prefix, num_seeds=10):
    test_accs = list()
    val_accs = list()
    model = model.to(device)
    for i in range(10):
        train_acc, dev_acc, test_acc = evaluate_model(
            model, batch_size = 1, file_prefix=file_prefix + "_" + str(i).zfill(2))
        test_accs.append(test_acc)
        val_accs.append(dev_acc)

    del model
    torch.cuda.empty_cache()
    print("="*50+"\nTest score: %.2f" % (100.0*mean(test_accs)) + "%" + " (+-%.2f)" % (100.0*std(test_accs)) + "\nValidation score: %.2f" % (100.0*mean(val_accs)) + "%" + " (+-%.2f)" % (100.0*std(val_accs)))
    return test_accs, val_accs_____no_output_____train_model_default(lambda : BOW(len(v.w2i), len(t2i), vocab=v), 0.001, "BOW")Skipping checkpoints/BOW_00.pt
Skipping checkpoints/BOW_01.pt
Skipping checkpoints/BOW_02.pt
Skipping checkpoints/BOW_03.pt
Skipping checkpoints/BOW_04.pt
Skipping checkpoints/BOW_05.pt
Skipping checkpoints/BOW_06.pt
Skipping checkpoints/BOW_07.pt
Skipping checkpoints/BOW_08.pt
Skipping checkpoints/BOW_09.pt
bow_model = BOW(len(v.w2i), len(t2i), vocab=v)
bow_test, bow_val = eval_model_default(bow_model, "BOW", num_seeds=10)
del bow_model
torch.cuda.empty_cache()  Loading model
best model: train acc=0.9702, dev acc=0.3351, test acc=0.3498
Loading model
best model: train acc=0.9287, dev acc=0.3524, test acc=0.3566
Loading model
best model: train acc=0.9698, dev acc=0.3733, test acc=0.3846
Loading model
best model: train acc=0.9460, dev acc=0.3515, test acc=0.3805
Loading model
best model: train acc=0.9723, dev acc=0.3579, test acc=0.3787
Loading model
best model: train acc=0.9608, dev acc=0.3297, test acc=0.3416
Loading model
best model: train acc=0.9675, dev acc=0.3678, test acc=0.3729
Loading model
best model: train acc=0.9426, dev acc=0.3488, test acc=0.3674
Loading model
best model: train acc=0.9375, dev acc=0.3388, test acc=0.3579
Loading model
best model: train acc=0.9350, dev acc=0.3279, test acc=0.3222
==================================================
Test score: 36.12% (+-1.86)
Validation score: 34.83% (+-1.47)
# This will plot the validation accuracies across time.
# f, axarr = plt.subplots(2, 2, figsize=(10,5))
# axarr[0, 0].plot(bow_accuracies)
# axarr[0, 0].set_title('Axis [0,0]')
# axarr[0, 1].plot(bow_accuracies)
# axarr[0, 1].set_title('Axis [0,1]')
# axarr[1, 0].plot(bow_accuracies)
# axarr[1, 0].set_title('Axis [1,0]')
# axarr[1, 1].plot(bow_accuracies)
# axarr[1, 1].set_title('Axis [1,1]')_____no_output_____# This will plot the training loss over time.
# plt.plot(bow_losses)_____no_output_____

# CBOW_____no_output_____We now continue with a **continuous bag-of-words (CBOW)** model.
It is similar to the BOW model, but now embeddings can have a dimension of *arbitrary size*.
This means that we can make them bigger and learn more aspects of each word. We will still sum the vectors to get a sentence representation, but now the result is no longer conveniently the number of sentiment classes.

*Note that this is not the same as word2vec CBOW.* Both models take a few words as input, sum and make a prediction. However, in word2vec the prediction is the word in the middle of the input words. Here, the prediction is a sentiment class (1 out of 5 possible numbers 0-4).

So what can we do? We can *learn* a parameter matrix $W$ that turns our summed vector into the number of output classes, by matrix-multiplying it: $$Wx$$
If the size of $W$ is `5 x d`, and $x$ is `d x 1` and , then the result will be `5x1`, and then we can again find our prediction using an argmax._____no_output_____## Exercise: write CBOW class & train it

Write a class `CBOW` that:

- has word embeddings with size 300
- sums the word vectors for the input words (just like in `BOW`)
- projects the resulting vector down to 5 units using a linear layer (including a bias term)

Train your CBOW model and plot the validation accuracy and training loss over time._____no_output_____

# YOUR CODE HERE
class CBOW(nn.Module):
    """A continuos bag-of-words model"""

    def __init__(self, vocab_size, embedding_dim, output_dim, vocab):
        super(CBOW, self).__init__()
        self.vocab = vocab

        # this is a trained look-up table with word embeddings
        self.embed = nn.Embedding(vocab_size, embedding_dim)
        self.applyW = nn.Linear(embedding_dim, output_dim)

        # this is a trained bias term
        self.bias = nn.Parameter(torch.zeros(output_dim), requires_grad=True)

    def forward(self, inputs):
        # this is the forward pass of the neural network
        # given inputs, it computes the output

        # this looks up the embeddings for each word ID in inputs
        # the result is a sequence of word embeddings
        embeds = self.embed(inputs)
        trans_embeds = self.applyW(embeds)

        # the output is the sum across the time dimension (1)
        # with the bias term added
        logits = trans_embeds.sum(1) + self.bias

        return logits_____no_output_____train_model_default(lambda: CBOW(len(v.w2i), 300, len(t2i), vocab=v), 0.0005, "CBOW", num_iterations=100000)Skipping checkpoints/CBOW_00.pt
Skipping checkpoints/CBOW_01.pt
Skipping checkpoints/CBOW_02.pt
Skipping checkpoints/CBOW_03.pt
Skipping checkpoints/CBOW_04.pt
Skipping checkpoints/CBOW_05.pt
Skipping checkpoints/CBOW_06.pt
Skipping checkpoints/CBOW_07.pt
Skipping checkpoints/CBOW_08.pt
Skipping checkpoints/CBOW_09.pt
cbow_model = CBOW(len(v.w2i), 300, len(t2i), vocab=v)
cbow_test, cbow_val = eval_model_default(cbow_model, "CBOW")
del cbow_model
torch.cuda.empty_cache()  Loading model
best model: train acc=0.6993, dev acc=0.3579, test acc=0.3575
Loading model
best model: train acc=0.9401, dev acc=0.3442, test acc=0.3674
Loading model
best model: train acc=0.7062, dev acc=0.3606, test acc=0.3511
Loading model
best model: train acc=0.8684, dev acc=0.3488, test acc=0.3561
Loading model
best model: train acc=0.9743, dev acc=0.3297, test acc=0.3584
Loading model
best model: train acc=0.9778, dev acc=0.3542, test acc=0.3588
Loading model
best model: train acc=0.7129, dev acc=0.3569, test acc=0.3674
Loading model
best model: train acc=0.9338, dev acc=0.3506, test acc=0.3756
Loading model
best model: train acc=0.9558, dev acc=0.3342, test acc=0.3837
Loading model
best model: train acc=0.9676, dev acc=0.3406, test acc=0.3389
==================================================
Test score: 36.15% (+-1.20)
Validation score: 34.78% (+-0.99)
# plt.plot(cbow_accuracies)_____no_output_____# plt.plot(cbow_losses)_____no_output_____

# Deep CBOW_____no_output_____To see if we can squeeze some more performance out of the CBOW model, we can make it deeper and non-linear: add more layers and tanh-activations.
By using more parameters we can learn more aspects of the data, and by using more layers and non-linearities, we can try to learn a more complex function.
This is not something that always works. If the input-output mapping of your data is simple, then a complicated function could easily overfit on your training set, which will lead to poor generalization.

We will use E for the size of the word embeddings (use: 300) and D for the size of a hidden layer (use: 100).

#### Exercise: write Deep CBOW class and train it

Write a class `DeepCBOW`.

In your code, make sure that your `output_layer` consists of the following:
- A linear transformation from E units to D units.
- A Tanh activation
- A linear transformation from D units to D units
- A Tanh activation
- A linear transformation from D units to 5 units (our output classes).

We recommend using [nn.Sequential](https://pytorch.org/docs/stable/nn.html?highlight=sequential#torch.nn.Sequential) to implement this exercise._____no_output_____

# YOUR CODE HERE
class DeepCBOW(nn.Module):
    """A continuos bag-of-words model"""

    def __init__(self, vocab_size, E, D, output_dim, vocab):
        super(DeepCBOW, self).__init__()
        self.vocab = vocab

        # this is a trained look-up table with word embeddings
        self.embed = nn.Embedding(vocab_size, E)
        self.output_layer = nn.Sequential(OrderedDict([
          ('linear1', nn.Linear(E, D)),
          ('tanh1', nn.Tanh()),
          ('dropout1', nn.Dropout(0.1)),
          ('linear2', nn.Linear(D,D)),
          ('relu2', nn.Tanh()),
          ('dropout2', nn.Dropout(0.1)),
          ('linear3', nn.Linear(D,output_dim))
        ]))

        # this is a trained bias term
        self.bias = nn.Parameter(torch.zeros(output_dim), requires_grad=True)

    def forward(self, inputs):
        # this is the forward pass of the neural network
        # given inputs, it computes the output

        # this looks up the embeddings for each word ID in inputs
        # the result is a sequence of word embeddings
        embeds = self.embed(inputs)
        for layer in self.output_layer:
            embeds = layer(embeds)

        # the output is the sum across the time dimension (1)
        # with the bias term added
        logits = embeds.sum(1) + self.bias

        return logits_____no_output_____train_model_default(lambda: DeepCBOW(len(v.w2i), 300, 100, len(t2i), vocab=v), 0.002, "DeepCBOW", eval_every=1000, num_iterations=100000)Skipping checkpoints/DeepCBOW_00.pt
Skipping checkpoints/DeepCBOW_01.pt
Skipping checkpoints/DeepCBOW_02.pt
Skipping checkpoints/DeepCBOW_03.pt
Skipping checkpoints/DeepCBOW_04.pt
Skipping checkpoints/DeepCBOW_05.pt
Skipping checkpoints/DeepCBOW_06.pt
Skipping checkpoints/DeepCBOW_07.pt
Skipping checkpoints/DeepCBOW_08.pt
Skipping checkpoints/DeepCBOW_09.pt
deep_cbow_model = DeepCBOW(len(v.w2i), 300, 100, len(t2i), vocab=v)
deep_cbow_test, deep_cbow_val = eval_model_default(deep_cbow_model, "DeepCBOW")
del deep_cbow_model
torch.cuda.empty_cache()  Loading model
best model: train acc=0.4905, dev acc=0.3669, test acc=0.3633
Loading model
best model: train acc=0.5726, dev acc=0.3606, test acc=0.3561
Loading model
best model: train acc=0.7288, dev acc=0.3706, test acc=0.3620
Loading model
best model: train acc=0.5376, dev acc=0.3760, test acc=0.3742
Loading model
best model: train acc=0.7496, dev acc=0.3497, test acc=0.3217
Loading model
best model: train acc=0.8388, dev acc=0.3579, test acc=0.3575
Loading model
best model: train acc=0.6180, dev acc=0.3797, test acc=0.3602
Loading model
best model: train acc=0.7560, dev acc=0.3588, test acc=0.3783
Loading model
best model: train acc=0.7140, dev acc=0.3787, test acc=0.3724
Loading model
best model: train acc=0.6177, dev acc=0.3833, test acc=0.3529
==================================================
Test score: 35.99% (+-1.50)
Validation score: 36.82% (+-1.07)
# plt.plot(deep_cbow_accuracies)_____no_output_____# plt.plot(deep_cbow_losses)_____no_output_____

# Pre-trained word embeddings_____no_output_____The Stanford sentiment treebank is a very small data set, since it was manually annotated. This makes it difficult for the model to learn good word embeddings, to learn the "meaning" of the words in our vocabulary.

Think for a moment about the fact that the only error signal that the network receives is from predicting the sentiment!

To mitigate our data sparsity, we can download **pre-trained word embeddings**.
You can choose which pre-trained word embeddings to use:

- **Glove**. The "original" Stanford Sentiment classification [paper](http://aclweb.org/anthology/P/P15/P15-1150.pdf) used Glove embeddings, which are just another method (like *word2vec*) to get word embeddings from unannotated text. Glove is described in the following paper which you should cite if you use them:
> Jeffrey Pennington, Richard Socher, and Christopher Manning. ["Glove: Global vectors for word representation."](https://nlp.stanford.edu/pubs/glove.pdf) EMNLP 2014.

- **word2vec**. This is the method that you learned about in class, described in:
> Mikolov, Tomas, et al. ["Distributed representations of words and phrases and their compositionality."](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) Advances in neural information processing systems. 2013.

With these pre-trained word embeddings, we can initialize our word embedding lookup table and start form a point where similar words are already close to one another.

You can choose to keep the word embeddings **fixed** or to train them further.
We will keep them fixed for now.

For the purposes of this lab, it is enough if you understand how word2vec works (whichever vectors you use). If you are interested you may also check out the Glove paper.

You can either download the word2vec vectors, or the Glove vectors.
If you want to compare your results to the Stanford paper later on, then you should use Glove.
**At the end of this lab you have the option to compare which vectors give you the best performance. For now, simply choose one of them and continue with that.**_____no_output_____

# This downloads the Glove 840B 300d embeddings.
# The original file is at http://nlp.stanford.edu/data/glove.840B.300d.zip
# Since that file is 2GB, we provide you with a *filtered version*
# which contains all the words you need for this data set.

# You only need to do this once.
# Please comment this cell out after downloading.

# !wget https://gist.githubusercontent.com/bastings/b094de2813da58056a05e8e7950d4ad1/raw/3fbd3976199c2b88de2ae62afc0ecc6f15e6f7ce/glove.840B.300d.sst.txt_____no_output_____# This downloads the word2vec 300D Google News vectors
# The file has been truncated to only contain words that appear in our data set.
# You can find the original file here: https://code.google.com/archive/p/word2vec/

# You only need to do this once.
# Please comment this out after downloading.
# !wget https://gist.githubusercontent.com/bastings/4d1c346c68969b95f2c34cfbc00ba0a0/raw/76b4fefc9ef635a79d0d8002522543bc53ca2683/googlenews.word2vec.300d.txt_____no_output_____# Mount Google Drive (to save the downloaded files)
# from google.colab import drive
# drive.mount('/gdrive')_____no_output_____# Copy word vectors *to* Google Drive

# You only need to do this once.
# Please comment this out after running it.
# !cp "glove.840B.300d.sst.txt" "/gdrive/My Drive/"
# !cp "googlenews.word2vec.300d.txt" "/gdrive/My Drive/"_____no_output_____# If you copied the word vectors to your Drive before,
# here is where you copy them back to the Colab notebook.

# Copy Glove vectors *from* Google Drive
# !cp "/gdrive/My Drive/glove.840B.300d.sst.txt" .
# !cp "/gdrive/My Drive/googlenews.word2vec.300d.txt" ._____no_output_____

At this point you have the pre-trained word embedding files, but what do they look like?_____no_output_____

# Exercise: Print the first 4 lines of the files that you downloaded.
# What do you see?
with open("glove.840B.300d.sst.txt", "r") as f:
    glove_init_lines = f.readlines()
    print(glove_init_lines[2].split(" ")[0])the

#### Exercise: New Vocabulary

Since we now use pre-trained word embeddings, we need to create a new vocabulary.
This is because of two reasons:

1. Not all words in our training set are in the set of pre-trained word embeddings. We do not want words in our vocabulary that are not in that set.
2. We get to look up the pre-trained word embedding for words in the validation and test set, even if we don't know them from training.

Now, create a new vocabulary object `v` and load the pre-trained vectors into a `vectors`.

The vocabulary `v` should consist of:
 - a  `<unk>` token at position 0,
 - a  `<pad>` token at position 1,
 - and then all words in the pre-trained vector set.

After storing each vector in a list `vectors`, turn in into a numpy matrix like this:
```python
 vectors = np.stack(vectors, axis=0)
```

_____no_output_____

# YOUR CODE HERE
v = Vocabulary()
vectors = list()
vectors.append(np.zeros(300, dtype=np.float32)) # <unk>
vectors.append(np.zeros(300, dtype=np.float32)) # <pad>
for line in glove_init_lines:
    v.count_token(line.split(" ")[0])
    vectors.append(np.array([float(x) for x in line.split(" ")[1:]], dtype=np.float32))
vectors = np.stack(vectors, axis=0)
v.build()_____no_output_____

#### Exercise: words not in our pre-trained set

How many words in the training, dev, and test set are also in your vector set?
How many words are not there?

Store the words that are not in the word vector set in the set below._____no_output_____

words_not_found = set()
# YOUR CODE HERE
for data_set in (train_data,):
    for ex in data_set:
        for token in ex.tokens:
            if v.w2i.get(token, 0) == 0:
                words_not_found.update([token])
print(words_not_found)
print(len(words_not_found)){'dictator-madman', '24-and-unders', 'pseudo-bio', 'the-week', 'water-camera', 'actorish', 'Stortelling', 'screeching-metal', 'independent-community', 'bottom-of-the-bill', 'pro-Serb', 'shock-you-into-laughter', 'gut-clutching', 'spy-action', 'murder-on-campus', 'qatsi', 'Brit-com', 'foul-natured', 'queasy-stomached', 'truncheoning', 'been-told-a', 'Idemoto', 'college-spawned', 'Nietzsche-referencing', 'phoney-feeling', 'Live-style', 'power-lunchers', 'Jeong-Hyang', 'stuporously', 'Journalistically', 'pseudo-witty', 'overcoming-obstacles', 'Singer/composer', 'non-Britney', 'stable-full', 'renegade-cop', 'pantomimesque', 'Jean-Claud', 'techno-tripe', 'Dognini', 'Bettany/McDowell', 'tub-thumpingly', 'demented-funny', 'Punitively', 'often-hilarious', 'industrial-model', 'Phoned-in', 'O2-tank', 'telanovela', 'inside-show-biz', 'disappearing/reappearing', 'Bjorkness', 'war-movie', 'wide-smiling', 'Hollywood-itis', 'German-Expressionist', 'digital-effects-heavy', 'achronological', 'she-cute', 'movie-biz', 'damaged-goods', 'Rashomon-for-dipsticks', 'at-a-frat-party', 'video-game-based', 'cameo-packed', 'anti-erotic', 'Zishe', 'Marine/legal', 'hyper-artificiality', 'warm-milk', 'ennui-hobbled', 'movie-specific', 'egocentricities', 'pseudo-philosophic', 'dust-caked', 'Achero', 'shlockmeister', 'Short-story', 'Fulford-Wierzbicki', 'Sandlerian', 'gender-provoking', 'un-bear-able', 'Globetrotters-Generals', 'overstylized', 'as-it', 'creepy-scary', 'kids-cute', 'pyro-correctly', 'barking-mad', 'Wewannour', 'non-mystery', 'revenge-of-the-nerds', 'actress-producer', 'moral-condundrum', 'materalism', 'Univac-like', 'Interminably', '112-minute', 'non-Bondish', 'Whiffle-Ball', 'Sychowski', 'groan-to-guffaw', 'of-a-sequel', 'kinetically-charged', 'snazziness', 'tech-geeks', 'even-flowing', 'often-cute', 'musclefest', 'long-on-the-shelf', 'Holofcenter', 'stiletto-stomps', 'Director-chef', 'stadium-seat', 'affectation-free', 'trance-noir', 'actorliness', '-RRB-', 'hippie-turned-yuppie', 'Good-naturedly', 'Director-writer', 'Cineasts', 'unemotive', 'life-at-arm', 'Brothers-style', 'shuck-and-jive', 'Truckzilla', 'oh-those-wacky-Brits', 'cliché-riddled', 'Warmed-over', 'Bruckheimeresque', 'high-buffed', 'Juliet/West', 'natural-seeming', 'oft-brilliant', 'tear-drenched', 'affirmational', 'ultra-loud', 'thinly-conceived', 'eroti-comedy', 'Mordantly', 'autocritique', 'ricture', 'barn-side', 'semi-surrealist', 'Imaxy', 'gay-niche', 'dirgelike', 'uncharismatically', 'Poke-mania', 'bang-the-drum', 'redneck-versus-blueblood', 'family-film', 'sex-in-the-city', 'Bond-inspired', 'none-too-original', 'no-surprise', 'spaniel-eyed', 'TV-insider', 'thrill-kill', 'Recoing', 'surface-obsession', 'not-so-Divine', 'young-guns', 'anti-adult', 'self-amused', 'under-7', 'Qutting', 'hayseeds-vs', 'skit-com', 'in-jokey', 'semi-amusing', 'raw-nerved', 'teen-speak', 'ANTWONE', '-LRB-', 'male-ridden', 'Less-than-compelling', 'follow-your-dream', '168-minute', 'underrehearsed', 'Gaï', 'Kahlories', 'underdramatized', 'guilt-suffused', 'who-wrote-Shakespeare', 'landbound', 'overly-familiar', 'pseudo-rock-video', 'travel-agency', 'bio-doc', 'Cool-J', 'pseudo-serious', 'beast-within', 'pop-induced', 'Equlibrium', 'something-borrowed', 'dark-as-pitch', 'than-likely', 'video-shot', 'Glizty', 'not-quite-dead', 'near-hypnotic', 'monster-in-the', 'thousand-times', 'e-graveyard', 'Djeinaba', 'not-nearly', 'girl-buddy', 'half-dimensional', 'smile-button', 'the-loose', 'beyond-lame', 'pokepie', 'film-culture', 'uncinematic', 'Brothers/Abrahams', 'goth-vampire', 'Underachieves', 'pee-related', 'moldy-oldie', 'sequel-for-the-sake', 'Watstein', 'Banderas-Lucy', 'Priggish', 'imponderably', 'faux-urban', 'unembarrassing', 'junk-calorie', 'too-conscientious', 'too-spectacular', 'Je-Gyu', 'crime-land', 'scary-funny', '91-minute', 'Tiresomely', 'kid-empowerment', 'a-bornin', 'unslick', 'Kubrick-meets-Spielberg', 'Oprahfication', 'splendid-looking', 'life-embracing', 'based-on-truth', 'super-stupid', 'girls-behaving-badly', 'Star/producer', 'thinks-it-is', 'unencouraging', 'audience-pleaser', 'nonethnic', 'clone-gag', 'ultra-provincial', 'direct-to-video/DVD', 'drama/action', 'bore-athon', 'Nebrida', 'the-cash', 'Jae-eun', 'handbag-clutching', 'headline-fresh', 'hour-and-a-half-long', 'cheese-laced', 'lovable-loser', 'cliché-laden', 'McBeal-style', '103-minute', 'out-to-change-the-world', 'media-constructed', 'slash-fest', 'farewell-to-innocence', 'gone-to-seed', 'best-sustained', 'grunge-pirate', 'Ki-Deok', 'hotsies', 'Potty-mouthed', 'prechewed', 'Pie-like', 'marveilleux', 'message-movie', 'fun-for-fun', 'videologue', 'II-Birkenau', 'grade-grubbers', 'Laissez-passer', 'raunch-fests', 'lip-non-synching', 'crash-and-bash', 'Margolo', 'toilet-humor', 'doofus-on', 'often-funny', 'under-inspired', 'flatula', 'Co-writer/director', 'triple-crosses', 'acting-workshop', 'well-lensed', 'kiddie-oriented', 'better-focused', 'hit-hungry', 'eardrum-dicing', 'teen-exploitation', 'tough-man', 'Japanimator', 'rough-trade', 'church-wary', 'cor-blimey-luv-a-duck', 'anti-Kieslowski', 'two-wrongs-make-a-right', 'Vereté', 'unlaughable', 'happily-ever', 'quasi-improvised', 'Cosby-Seinfeld', 'like-themed', 'Audacious-impossible', 'jump-in-your-seat', 'boundary-hopping', 'democracie', 'mush-hearted', 'fast-edit', 'guy-in-a-dress', 'slasher-movie', 'Unambitious', 'Koshashvili', 'tryingly', 'talking-animal', 'bump-in', 'outgag', 'hell-jaunt', 'soon-to-be-forgettable', 'Ear-splitting', 'fang-baring', 'pooper-scoopers', 'still-raw', 'hammily', 'neo-Hitchcockianism', 'Unspools', 'pro-wildlife', 'stunt-hungry', 'sitcomishly', 'big-budget/all-star', 'yarn-spinner', 'skyscraper-trapeze', 'post-Tarantino', 'RunTelDat', 'white-empowered', 'nonchallenging', 'LePlouff', 'Talancón', 'bio-drama', 'big-fisted', 'sub-music', 'sub-Tarantino', 'badly-rendered', 'sitcom-worthy', 'bottomlessly', 'gore-free', 'Feardotcom', 'Komediant', 'Phonce', 'pulpiness', 'Nickelodeon-esque', 'social/economic/urban', 'merchandised-to-the-max', 'Age-inspired', 'unfussily', 'Romething', 'style-free', 'smarter-than-thou', 'rubber-face', 'voices-from-the-other-side', 'preachy-keen', 'daytime-drama', 'community-therapy', 'humor-seeking', 'Minac', 'cheatfully', 'pseudo-educational', 'Hellstenius', 'Ourside', 'Re-Fried', 'self-defeatingly', 'groupie/scholar', 'Puportedly', 'unconned', 'gangster/crime', 'now-cliched', 'Wisegirls', 'Choquart', 'time-it-is', 'feardotcom.com', 'sense-spinning', 'soul-stripping', 'from-television', 'Lynch-like', 'media-soaked', 'big-bug', 'derisions', 'père-fils', 'pathos-filled', 'two-drink-minimum', 'repellantly', 'sub-formulaic', 'trouble-in-the-ghetto', 'matinee-style', 'yahoo-ing', 'Seldhal', 'Hollywood-action', 'not-very-funny', 'Eckstraordinarily', 'semimusical', 'MIBII', 'too-frosty', 'double-pistoled', 'every-joke-has', 'kids-and-family-oriented', 'Janklowicz-Mann', 'Oscar-size', 'song-and-dance-man', 'Verete', 'food-spittingly', 'cold-fish', 'feardotcom', 'animated-movie', 'coming-of-age/coming-out', 'techno-sex', 'pro-Serbian', 'dudsville', 'spy-savvy', 'kids-in-peril', 'super-dooper-adorability', 'East-vs', 'bare-midriff', 'kid-movie', 'Otto-Sallies', 'Well-meant', 'action-thriller/dark', 'over-amorous', 'Birot', 'huge-screen', 'identity-seeking', 'so-five-minutes-ago', 'slummer', 'reality-snubbing', 'modern-office', 'Denlopp', 'Rintarô', 'made-for-movie', 'lascivious-minded', 'none-too-funny', 'retro-refitting', 'buzz-obsessed', 'Butterfingered', 'message-mongering', 'Kafka-inspired', 'direct-to-void', 'bibbidy-bobbidi-bland', 'time-switching', 'jazz-playing', 'kibbitzes', 'sillified', 'S1M0NE', 'kid-vid', 'Talkiness', 'Schneidermeister', 'TV-cops', 'fustily', 'Georgian-Israeli', 'overmanipulative', 'Channel-style', 'ill-wrought', 'valley-girl', 'too-extreme-for-TV', 'tardier', 'adventues', 'disaffected-indie-film', 'cop-flick', 'meets-John', 'indieflick', 'Chabrolian', 'Norrington-directed', 'unfakable', 'tries-so-hard-to-be-cool', 'stand-up-comedy', 'ego-destroying', 'show-don', 'gunfest', 'female-bonding', 'out-bad-act', 'Piercingly', 'Hitler-study', 'quasi-Shakespearean', 'Jaglomized', 'road-and-buddy', 'gender-war', 'Waters-like', 'flick-knife', 'quick-cuts', 'and-miss', 'teen-sleaze', 'Clarke-Williams', 'Steinis', 'wind-in-the-hair', 'crummy-looking', 'hidden-agenda', 'Formuliac', 'Bazadona', 'Seldahl', 'Hjelje', 'remain-nameless', 'dirty-joke', 'Silbersteins', 'self-glorified', 'flakeball', 'Heremakono', 'Annie-Mary', 'brain-deadening', 'goose-pimple', 'non-exploitive', 'nerve-raked', 'smashups', 'Debrauwer', 'anti-Harry', 'Brother-Man', 'indie-heads', 'wild-and-woolly', 'spook-a-rama', 'Stevenon', 'By-the-numbers', 'I-heard-a-joke', 'brain-slappingly', 'zombie-land', 'genre-curling', 'hotter-two-years-ago', 'Reeboir', 'Spy-vs', 'drama/character', 'singer-turned', 'pop-cyber', 'deadeningly', 'Marcken', 'Hollywood-predictable', 'Kosashvili', 'therapy-dependent', 'Damon/Bourne', 'surehanded', 'stagecrafts', 'two-hour-and-fifteen-minute', 'video-cam', 'Bottom-rung', 'art-conscious', 'sub-sophomoric', 'not-at-all-good', 'Insufferably', 'tolerable-to-adults', 'Pasach', 'wise-cracker', 'Brazil-like', 'dead-undead', 'monster/science', 'still-inestimable', 'rock-n-rolling', 'well-contructed', 'Witch-style', 'surface-effect', 'Venice/Venice', 'makeup-deep', 'flex-a-thon', 'estrogen-free', 'banter-filled', 'non-firsthand', 'dullingly', 'cellophane-pop', 'disease-of', 'Felinni', 'Efteriades', 'fizzability', 'talk-heavy', 'character-who-shall', '102-minute', 'ballistic-pyrotechnic', 'yawn-provoking', 'bad-movie', 'as-nasty', 'action-and-popcorn', 'clung-to', 'blood-splattering', 'unhibited', 'oh-so-Hollywood', 'teen-gang', 'auto-critique', 'give-me-an-Oscar', 'out-shock', 'Copmovieland', 'late-twenty-somethings', 'trash-cinema', 'movie-movie', 'Mushes', 'celeb-strewn', 'I-2-spoofing', 'too-hot-for-TV', 'beloved-major', 'less-compelling', 'just-above-average', 'disease-of-the-week', 'best-foreign-film', 'Rocky-like', 'image-mongering', 'revigorates', 'Oscar-sweeping', 'made-for-home-video', 'cipherlike', 'cable-sports', 'girl-meets-girl', 'Punch-and-Judy', 'Vulakoro', 'lack-of-attention', 'prefeminist', 'lower-wit', 'splatterfests', 'zinger-filled', 'run-of-the-filth', 'underventilated', 'dolphin-gasm', 'unsalvageability', 'step-printing', 'Asiaphiles', 'reel/real', 'achival', 'FearDotCom', 'old-fashioned-movie', 'Ill-considered', '129-minute', 'close-to-solid', 'truck-loving', 'ever-ruminating', 'heart-rate-raising', 'French-produced', 'Devolves', 'razor-sided', 'psychodramatics', 'romantic/comedy', 'teen-driven', 'snow-and-stuntwork', 'seventy-minute', 'so-bad-they', 'artnering', 'out-outrage', 'Volletta', 'cinemantic', 'tuba-playing', 'flck', 'dog-paddle', 'waydowntown', 'Gator-bashing', 'Altman-esque', 'neo-Augustinian', 'superficiale', 'barn-burningly', 'shoe-loving', 'special-effects-laden', 'ultra-cheesy', 'sense-of-humour', 'already-shallow', 'docu-Dogma', 'diciness', 'Frankenstein-monster', 'wannabe-hip', 'whoopee-cushion', 'Wollter', 'surfacey', 'genial-rogue', 'artsploitation', 'college-friends', 'wonder-what', 'propriety-obsessed', 'Cliff-Notes', 'police-oriented', 'Auteil', 'pasta-fagioli', 'Self-congratulatory', 'boom-bam', 'dungpile', 'Driver-esque', 'semi-throwback', 'gabbiest', 'out-depress', 'cat-and-mouser', 'sports-movie', 'doing-it-for', 'title-bout', 'rape-payback', 'show-stoppingly', 'sleep-inducingly', 'stomach-knotting', 'Hopkins/Rock'}
699

#### Exercise: train Deep CBOW with (fixed) pre-trained embeddings

Now train Deep CBOW again using the pre-trained word vectors.
_____no_output_____

# We define a dummy class so that we save the model to a different file.
class PTDeepCBOW(DeepCBOW):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, vocab):
        super(PTDeepCBOW, self).__init__(
            vocab_size, embedding_dim, hidden_dim, output_dim, vocab)_____no_output_____def prep_pretrained_deep_cbow():
    pt_deep_cbow_model = PTDeepCBOW(len(v.w2i), 300, 100, 5, v)

    # copy pre-trained word vectors into embeddings table
    pt_deep_cbow_model.embed.weight.data.copy_(torch.from_numpy(vectors))

    # disable training the pre-trained embeddings
    pt_deep_cbow_model.embed.weight.requires_grad = False
    return pt_deep_cbow_model

train_model_default(prep_pretrained_deep_cbow, lr=0.0001, file_prefix="PTDeepCBOW", num_iterations=30000, eval_every=1000)Skipping checkpoints/PTDeepCBOW_00.pt
Skipping checkpoints/PTDeepCBOW_01.pt
Skipping checkpoints/PTDeepCBOW_02.pt
Skipping checkpoints/PTDeepCBOW_03.pt
Skipping checkpoints/PTDeepCBOW_04.pt
Skipping checkpoints/PTDeepCBOW_05.pt
Skipping checkpoints/PTDeepCBOW_06.pt
Skipping checkpoints/PTDeepCBOW_07.pt
Skipping checkpoints/PTDeepCBOW_08.pt
Skipping checkpoints/PTDeepCBOW_09.pt
pt_deep_cbow_model = prep_pretrained_deep_cbow()
pt_deep_cbow_test, pt_deep_cbow_val = eval_model_default(pt_deep_cbow_model, "PTDeepCBOW")
del pt_deep_cbow_model
torch.cuda.empty_cache()  Loading model
best model: train acc=0.4655, dev acc=0.4332, test acc=0.4348
Loading model
best model: train acc=0.4637, dev acc=0.4360, test acc=0.4511
Loading model
best model: train acc=0.4737, dev acc=0.4369, test acc=0.4484
Loading model
best model: train acc=0.4740, dev acc=0.4396, test acc=0.4448
Loading model
best model: train acc=0.4501, dev acc=0.4314, test acc=0.4167
Loading model
best model: train acc=0.4622, dev acc=0.4396, test acc=0.4376
Loading model
best model: train acc=0.4570, dev acc=0.4387, test acc=0.4475
Loading model
best model: train acc=0.4698, dev acc=0.4332, test acc=0.4403
Loading model
best model: train acc=0.4287, dev acc=0.4378, test acc=0.4303
Loading model
best model: train acc=0.4699, dev acc=0.4414, test acc=0.4511
==================================================
Test score: 44.03% (+-1.03)
Validation score: 43.68% (+-0.31)
# plot dev accuracies
# plt.plot(pt_deep_cbow_accuracies)_____no_output_____# plot train loss
# plt.plot(pt_deep_cbow_losses)_____no_output_____

**It looks like we've hit what is possible with just using words.**
Let's move on by incorporating word order!_____no_output_____# LSTM_____no_output_____It is time to get more serious. Even with pre-trained word embeddings and multiple layers, we seem to do pretty badly at sentiment classification here.
The next step we can take is to introduce word order again, and to get a representation of the sentence as a whole, without independence assumptions.

We will get this representation using an **Long Short-Term Memory** (LSTM). As an exercise, we will code our own LSTM cell, so that we get comfortable with its inner workings.
Once we have an LSTM cell, we can call it repeatedly, updating its hidden state one word at a time:

```python
rnn = MyLSTMCell(input_size, hidden_size)

hx = torch.zeros(1, hidden_size)  # initial state
cx = torch.zeros(1, hidden_size)  # initial memory cell
output = []                       # to save intermediate LSTM states

# feed one word at a time
for i in range(n_timesteps):
  hx, cx = rnn(input[i], (hx, cx))
  output.append(hx)
```

If you need some more help understanding LSTMs, then check out these resources:
- Blog post (highly recommended): http://colah.github.io/posts/2015-08-Understanding-LSTMs/
- Paper covering LSTM formulas in detail: https://arxiv.org/abs/1503.04069

#### Exercise: Finish the LSTM cell below.
You will need to implement the LSTM formulas:

$$
\begin{array}{ll}
        i = \sigma(W_{ii} x + b_{ii} + W_{hi} h + b_{hi}) \\
        f = \sigma(W_{if} x + b_{if} + W_{hf} h + b_{hf}) \\
        g = \tanh(W_{ig} x + b_{ig} + W_{hg} h + b_{hg}) \\
        o = \sigma(W_{io} x + b_{io} + W_{ho} h + b_{ho}) \\
        c' = f * c + i * g \\
        h' = o \tanh(c') \\
\end{array}
 $$

where $\sigma$ is the sigmoid function.

*Note that the LSTM formulas can differ slightly between different papers. We use the PyTorch LSTM formulation here.*_____no_output_____

class MyLSTMCell(nn.Module):
    """Our own LSTM cell"""

    def __init__(self, input_size, hidden_size, bias=True):
        """Creates the weights for this LSTM"""
        super(MyLSTMCell, self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size
        self.bias = bias

        self.tanh_act = nn.Tanh()
        self.sigmoid_act = nn.Sigmoid()
        # self.input_gate = nn.Linear(input_size + hidden_size, hidden_size)
        # self.forget_gate = nn.Linear(input_size + hidden_size, hidden_size)
        # self.candidate_gate = nn.Linear(input_size + hidden_size, hidden_size)
        # self.output_gate = nn.Linear(input_size + hidden_size, hidden_size)
        self.combined_gate = nn.Linear(input_size + hidden_size, 4 * hidden_size)

        self.reset_parameters()

    def reset_parameters(self):
        """This is PyTorch's default initialization method"""
        stdv = 1.0 / math.sqrt(self.hidden_size)
        for weight in self.parameters():
            weight.data.uniform_(-stdv, stdv)

    def forward(self, input_, hx, mask=None):
        """
        input is (batch, input_size)
        hx is ((batch, hidden_size), (batch, hidden_size))
        """
        prev_h, prev_c = hx

        # project input and prev state
        cat_input = torch.cat([input_,prev_h], dim=1)

        # main LSTM computation

        # i = self.sigmoid_act(self.input_gate(cat_input))
        # f = self.sigmoid_act(self.forget_gate(cat_input))
        # g = self.tanh_act(self.candidate_gate(cat_input))
        # o = self.sigmoid_act(self.output_gate(cat_input))
        combined_output = self.combined_gate(cat_input)
        i = self.sigmoid_act(combined_output[:,0 * self.hidden_size:1 * self.hidden_size])
        f = self.sigmoid_act(combined_output[:,1 * self.hidden_size:2 * self.hidden_size])
        g = self.tanh_act(combined_output[:,2 * self.hidden_size:3 * self.hidden_size])
        o = self.sigmoid_act(combined_output[:,3 * self.hidden_size:4 * self.hidden_size])

        c = f * prev_c + i * g
        h = o * self.tanh_act(c)

        return h, c

    def __repr__(self):
        return "{}({:d}, {:d})".format(
            self.__class__.__name__, self.input_size, self.hidden_size)_____no_output_____

#### Optional: Efficient Matrix Multiplication

It is more efficient to do a few big matrix multiplications than to do many smaller ones.

It is possible to implement the above cell using just **two** linear layers.

This is because the eight linear transformations from one forward pass through an LSTM cell can be done in just two:
$$W_h h + b_h$$
$$W_i x + b_i $$

with $h = $ `prev_h` and $x = $ `input_`.

and where:

$W_h =  \begin{pmatrix}
W_{hi}\\
W_{hf}\\
W_{hg}\\
W_{ho}
\end{pmatrix}$, $b_h = \begin{pmatrix}
b_{hi}\\
b_{hf}\\
b_{hg}\\
b_{ho}
\end{pmatrix}$,  $W_i = \begin{pmatrix}
W_{ii}\\
W_{if}\\
W_{ig}\\
W_{io}
\end{pmatrix}$ and $b_i = \begin{pmatrix}
b_{ii}\\
b_{if}\\
b_{ig}\\
b_{io}
\end{pmatrix}$.

Convince yourself that, after chunking with [torch.chunk](https://pytorch.org/docs/stable/torch.html?highlight=chunk#torch.chunk), the output of those two linear transformations is equivalent to the output of the eight linear transformations in the LSTM cell calculations above._____no_output_____#### LSTM Classifier

Having an LSTM cell is not enough: we still need some code that calls it repeatedly, and then makes a prediction from the final hidden state.
You will find that code below. Make sure that you understand it._____no_output_____

class LSTMClassifier(nn.Module):
    """Encodes sentence with an LSTM and projects final hidden state"""

    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, vocab):
        super(LSTMClassifier, self).__init__()
        self.vocab = vocab
        self.hidden_dim = hidden_dim
        self.embed = nn.Embedding(vocab_size, embedding_dim, padding_idx=1)
        self.rnn = MyLSTMCell(embedding_dim, hidden_dim)

        self.output_layer = nn.Sequential(
            nn.Dropout(p=0.1),  # explained later
            nn.Linear(hidden_dim, output_dim)
        )

    def forward(self, x):

        B = x.size(0)  # batch size (this is 1 for now, i.e. 1 single example)
        T = x.size(1)  # time (the number of words in the sentence)

        input_ = self.embed(x)

        # here we create initial hidden states containing zeros
        # we use a trick here so that, if input is on the GPU, then so are hx and cx
        hx = input_.new_zeros(B, self.rnn.hidden_size)
        cx = input_.new_zeros(B, self.rnn.hidden_size)

        # process input sentences one word/timestep at a time
        # input is batch-major, so the first word(s) is/are input_[:, 0]
        outputs = []
        for i in range(T):
            hx, cx = self.rnn(input_[:, i], (hx, cx))
            outputs.append(hx)

        # if we have a single example, our final LSTM state is the last hx
        if B == 1:
            final = hx
        else:
            #
            # This part is explained in next section, ignore this else-block for now.
            #
            # we processed sentences with different lengths, so some of the sentences
            # had already finished and we have been adding padding inputs to hx
            # we select the final state based on the length of each sentence

            # two lines below not needed if using LSTM form pytorch
            outputs = torch.stack(outputs, dim=0)          # [T, B, D]
            outputs = outputs.transpose(0, 1).contiguous()  # [B, T, D]

            # to be super-sure we're not accidentally indexing the wrong state
            # we zero out positions that are invalid
            pad_positions = (x == 1).unsqueeze(-1)

            outputs = outputs.contiguous()
            outputs = outputs.masked_fill_(pad_positions, 0.)

            mask = (x != 1)  # true for valid positions [B, T]
            lengths = mask.sum(dim=1)                  # [B, 1]

            indexes = (lengths - 1) + torch.arange(B, device=x.device, dtype=x.dtype) * T
            final = outputs.view(-1, self.hidden_dim)[indexes]  # [B, D]

        # we use the last hidden state to classify the sentence
        logits = self.output_layer(final)
        return logits_____no_output_____

#### Dropout

Besides not being able to learn meaningful word embeddings, there is another negative effect that can follow from data sparsity and a small data set: *overfitting*. This is a phenomenom that is very likely to occur when fitting strong and expressive models, like LSTMs, to small data. In practice, if your model overfits, this means that it will be very good at predicting (or: 'remembering') the sentiment of the training set, but unable to generalize to new, unseen data in the test set. This is undesirable and one technique to mitigate it is *dropout*.

A dropout layer is defined by the following formula: $\mathbf{d} \in \{0, 1\}^n$, with $d_j \sim \text{Bernoulli}(p)$, and can be applied to for example a linear layer:

$$\text{tanh}(W(\mathbf{h}\odot \mathbf{d}) + \mathbf{b})$$

These formulas simply mean that we *drop* certain parameters during training (by setting them to zero). Which parameters we drop is stochastically determined by a Bernoulli distribution and the probability of each parameter being dropped is set to $p = 0.5$ in our experiments (see the previous cell of code where we define our output layer). A dropout layer can be applied at many different places in our models. This technique helps against the undesirable effect where our model relies on single parameters for  prediction (e.g. if $h^{\prime}_j$ is large, always predict positive). If we use dropout, the model needs to learn to rely on different parameters, which is desirable when generalizing to unseen data._____no_output_____**Let's train our LSTM! ** Note that is will be a lot slower, because we need to do many more computations per sentence!

#### Training_____no_output_____

# plot validation accuracy
# plt.plot(lstm_accuracies)_____no_output_____# plot training loss
# plt.plot(lstm_losses)_____no_output_____

# Mini-batching

_____no_output_____**Why is the LSTM so slow?** Despite our best efforts, we still need to make a lot of matrix multiplications per example (linear in the length of the example) just to get a single classification, and we can only process the 2nd word once we have computed the hidden state for the 1st word (sequential computation).

GPUs are more efficient if we do a few big matrix multiplications, rather than lots of small ones. If we could process multiple examples at the same time, then we could exploit that. We still process the input sequentially, but now we can do so for multiple sentences at the same time.

Up to now our "minibatch" consisted of a single example. This was for a reason: the sentences in our data sets have **different lengths**, and this makes it difficult to process them at the same time.

Consider a batch of 2 sentences:

```
this movie is bad
this movie is super cool !
```

Let's say the IDs for these sentences are:

```
2 3 4 5
2 3 4 6 7 8
```

We cannot feed PyTorch an object with variable length rows! We need to turn this into a matrix.

The solution is to add **padding values** to our mini-batch:

```
2 3 4 5 1 1
2 3 4 6 7 8
```

Whenever a sentence is shorter than the longest sentence in a mini-batch, we just use a padding value (here: 1) to fill the matrix.

In our computation, we should **ignore** the padding positions (e.g. mask them out). Paddings should not contribute to the loss.

#### Mini-batch feed
We will now code a `get_minibatch` function that will replace our `get_example` function, and returns a mini-batch of the requested size._____no_output_____

def get_minibatch(data, batch_size=25, shuffle=True):
    """Return minibatches, optional shuffling"""

    if shuffle:
        # print("Shuffling training data")
        random.shuffle(data)  # shuffle training data each epoch

    batch = []

    # yield minibatches
    for example in data:
        batch.append(example)

        if len(batch) == batch_size:
            yield batch
            batch = []

    # in case there is something left
    if len(batch) > 0:
        yield batch_____no_output_____

#### Pad function
We will need a function that adds padding 1s to a sequence of IDs so that
it becomes as long as the longest sequencen in the minibatch._____no_output_____

def pad(tokens, length, pad_value=1):
    """add padding 1s to a sequence to that it has the desired length"""
    return tokens + [pad_value] * (length - len(tokens))

# example
tokens = [2, 3, 4]
pad(tokens, 5)_____no_output_____

#### New prepare function

We will also need a new function that turns a mini-batch into PyTorch tensors._____no_output_____

def prepare_minibatch(mb, vocab):
    """
    Minibatch is a list of examples.
    This function converts words to IDs and returns
    torch tensors to be used as input/targets.
    """
    batch_size = len(mb)
    maxlen = max([len(ex.tokens) for ex in mb])

    # vocab returns 0 if the word is not there
    x = [pad([vocab.w2i.get(t, 0) for t in ex.tokens], maxlen) for ex in mb]

    x = torch.LongTensor(x)
    x = x.to(device)

    y = [ex.label for ex in mb]
    y = torch.LongTensor(y)
    y = y.to(device)

    return x, y_____no_output_____# Let's test our new function.
# This should give us 3 examples.
mb = next(get_minibatch(train_data, batch_size=3, shuffle=False))
for ex in mb:
    print(ex)
    breakExample(tokens=['The', 'Rock', 'is', 'destined', 'to', 'be', 'the', '21st', 'Century', "'s", 'new', '``', 'Conan', "''", 'and', 'that', 'he', "'s", 'going', 'to', 'make', 'a', 'splash', 'even', 'greater', 'than', 'Arnold', 'Schwarzenegger', ',', 'Jean-Claud', 'Van', 'Damme', 'or', 'Steven', 'Segal', '.'], tree=Tree('3', [Tree('2', [Tree('2', ['The']), Tree('2', ['Rock'])]), Tree('4', [Tree('3', [Tree('2', ['is']), Tree('4', [Tree('2', ['destined']), Tree('2', [Tree('2', [Tree('2', [Tree('2', [Tree('2', ['to']), Tree('2', [Tree('2', ['be']), Tree('2', [Tree('2', ['the']), Tree('2', [Tree('2', ['21st']), Tree('2', [Tree('2', [Tree('2', ['Century']), Tree('2', ["'s"])]), Tree('2', [Tree('3', ['new']), Tree('2', [Tree('2', ['``']), Tree('2', ['Conan'])])])])])])])]), Tree('2', ["''"])]), Tree('2', ['and'])]), Tree('3', [Tree('2', ['that']), Tree('3', [Tree('2', ['he']), Tree('3', [Tree('2', ["'s"]), Tree('3', [Tree('2', ['going']), Tree('3', [Tree('2', ['to']), Tree('4', [Tree('3', [Tree('2', ['make']), Tree('3', [Tree('3', [Tree('2', ['a']), Tree('3', ['splash'])]), Tree('2', [Tree('2', ['even']), Tree('3', ['greater'])])])]), Tree('2', [Tree('2', ['than']), Tree('2', [Tree('2', [Tree('2', [Tree('2', [Tree('1', [Tree('2', ['Arnold']), Tree('2', ['Schwarzenegger'])]), Tree('2', [','])]), Tree('2', [Tree('2', ['Jean-Claud']), Tree('2', [Tree('2', ['Van']), Tree('2', ['Damme'])])])]), Tree('2', ['or'])]), Tree('2', [Tree('2', ['Steven']), Tree('2', ['Segal'])])])])])])])])])])])])]), Tree('2', ['.'])])]), label=3, transitions=[0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1])
# We should find 1s at the end where padding is.
x, y = prepare_minibatch(mb, v)
print("x", x)
print("y", y)x tensor([[   23,  1414,    11,  8222,     6,    27,     4,  2826,  3764,    21,
            92,  5233,  8558, 15010,     5,    16,    53,    21,   183,     6,
           107,     8,  7409,   148,  1552,    97,  6000, 10464,     2,     0,
          2740, 15451,    33,  4008, 12925,     3,     1,     1,     1],
        [   23, 15300,  6665,  8307,     7,  5233,    23,  1383,     7,     4,
          4884, 15010,  8681,    11,    59,   933,    16,     8,  3045,     7,
           566,    42,    36,  7800,  2886, 20302,  1433,  1977,    21,  4360,
          2274,     7, 15256, 10099,    21, 15561,     3,     1,     1],
        [    0,  5314,  3878,  7735,     8,  9726,     7,  1261,   158,     8,
           219,  1060,  2187,     2,     8,   219,    50,   688, 11605,     6,
             4,   414,   158,    43,     4,   494,  1568,  1214,  6985,     4,
          2243,     2,  5986,     2,  2320,     7,     4,   982,     3]],
       device='cuda:0')
y tensor([3, 4, 3], device='cuda:0')

#### Evaluate (mini-batch version)

We can now update our evaluation function to use minibatches_____no_output_____

def evaluate(model, data,
             batch_fn=get_minibatch, prep_fn=prepare_minibatch,
             batch_size=16):
    """Accuracy of a model on given data set (using minibatches)"""
    correct = 0
    total = 0
    model.eval()  # disable dropout

    for mb in batch_fn(data, batch_size=batch_size, shuffle=False):
        x, targets = prep_fn(mb, model.vocab)
        with torch.no_grad():
            logits = model(x)

        predictions = logits.argmax(dim=-1).view(-1)

        # add the number of correct predictions to the total correct
        correct += (predictions == targets.view(-1)).sum().item()
        total += targets.size(0)

    return correct, total, correct / float(total)_____no_output_____

# LSTM (Mini-batched)_____no_output_____With this, let's run the LSTM again but now using minibatches!_____no_output_____

def get_lstm_model():
    lstm_model = LSTMClassifier(len(v.w2i), 300, 168, len(t2i), v)

    # copy pre-trained word vectors into embeddings table
    with torch.no_grad():
        lstm_model.embed.weight.data.copy_(torch.from_numpy(vectors))
        lstm_model.embed.weight.requires_grad = False
    return lstm_model

train_model_default(get_lstm_model, 2e-4, "LSTM", batch_size=32, batch_fn=get_minibatch, prep_fn=prepare_minibatch, eval_fn=evaluate, num_iterations=10000, eval_every=250)iter 250: dev acc=0.3715
new highscore
iter 500: dev acc=0.4060
new highscore
iter 750: dev acc=0.4369
new highscore
iter 1000: dev acc=0.4242
iter 1250: dev acc=0.4296
iter 1500: dev acc=0.4332
iter 1750: dev acc=0.4405
new highscore
iter 2000: dev acc=0.4432
new highscore
iter 2250: dev acc=0.4396
iter 2500: dev acc=0.4541
new highscore
iter 2750: dev acc=0.4550
new highscore
iter 3000: dev acc=0.4478
iter 3250: dev acc=0.4505
iter 3500: dev acc=0.4342
iter 3750: dev acc=0.4523
iter 4000: dev acc=0.4369
iter 4250: dev acc=0.4523
iter 4500: dev acc=0.4414
iter 4750: dev acc=0.4287
iter 5000: dev acc=0.4332
iter 5250: dev acc=0.4269
iter 5500: dev acc=0.4278
iter 5750: dev acc=0.4441
iter 6000: dev acc=0.4114
iter 6250: dev acc=0.4387
iter 6500: dev acc=0.4105
iter 6750: dev acc=0.4005
iter 7000: dev acc=0.3878
iter 7250: dev acc=0.4087
iter 7500: dev acc=0.4133
iter 7750: dev acc=0.4105
iter 8000: dev acc=0.3960
iter 8250: dev acc=0.4015
iter 8500: dev acc=0.3987
iter 8750: dev acc=0.4042
iter 9000: dev acc=0.4051
iter 9250: dev acc=0.4069
iter 9500: dev acc=0.4178
iter 9750: dev acc=0.4069
Iter 10000: loss=7554.2527, time=365.96s
iter 10000: dev acc=0.4178
Done training
Loading best model
best model iter 2750: train acc=0.5720, dev acc=0.4550, test acc=0.4629
iter 250: dev acc=0.3906
new highscore
iter 500: dev acc=0.4178
new highscore
iter 750: dev acc=0.4269
new highscore
iter 1000: dev acc=0.4287
new highscore
iter 1250: dev acc=0.4351
new highscore
iter 1500: dev acc=0.4460
new highscore
iter 1750: dev acc=0.4450
iter 2000: dev acc=0.4441
iter 2250: dev acc=0.4432
iter 2500: dev acc=0.4605
new highscore
iter 2750: dev acc=0.4678
new highscore
iter 3000: dev acc=0.4460
iter 3250: dev acc=0.4559
iter 3500: dev acc=0.4678
iter 3750: dev acc=0.4514
iter 4000: dev acc=0.4596
iter 4250: dev acc=0.4405
iter 4500: dev acc=0.4478
iter 4750: dev acc=0.4396
iter 5000: dev acc=0.4532
iter 5250: dev acc=0.4296
iter 5500: dev acc=0.4214
iter 5750: dev acc=0.4414
iter 6000: dev acc=0.4205
iter 6250: dev acc=0.4160
iter 6500: dev acc=0.4287
iter 6750: dev acc=0.4178
iter 7000: dev acc=0.4096
iter 7250: dev acc=0.4342
iter 7500: dev acc=0.4133
iter 7750: dev acc=0.4160
iter 8000: dev acc=0.4133
iter 8250: dev acc=0.4042
iter 8500: dev acc=0.4005
iter 8750: dev acc=0.4033
iter 9000: dev acc=0.4096
iter 9250: dev acc=0.4160
iter 9500: dev acc=0.4069
iter 9750: dev acc=0.4233
Iter 10000: loss=7598.8007, time=367.13s
iter 10000: dev acc=0.4133
Done training
Loading best model
best model iter 2750: train acc=0.5564, dev acc=0.4678, test acc=0.4615
iter 250: dev acc=0.3869
new highscore
iter 500: dev acc=0.4205
new highscore
iter 750: dev acc=0.4278
new highscore
iter 1000: dev acc=0.4378
new highscore
iter 1250: dev acc=0.4296
iter 1500: dev acc=0.4487
new highscore
iter 1750: dev acc=0.4423
iter 2000: dev acc=0.4623
new highscore
iter 2250: dev acc=0.4496
iter 2500: dev acc=0.4441
iter 2750: dev acc=0.4514
iter 3000: dev acc=0.4469
iter 3250: dev acc=0.4441
iter 3500: dev acc=0.4541
iter 3750: dev acc=0.4505
iter 4000: dev acc=0.4233
iter 4250: dev acc=0.4296
iter 4500: dev acc=0.4160
iter 4750: dev acc=0.4360
iter 5000: dev acc=0.4396
iter 5250: dev acc=0.4214
iter 5500: dev acc=0.4251
iter 5750: dev acc=0.4142
iter 6000: dev acc=0.4060
iter 6250: dev acc=0.4133
iter 6500: dev acc=0.3987
iter 6750: dev acc=0.4096
iter 7000: dev acc=0.4069
iter 7250: dev acc=0.4060
iter 7500: dev acc=0.4169
iter 7750: dev acc=0.3942
iter 8000: dev acc=0.4114
iter 8250: dev acc=0.4078
iter 8500: dev acc=0.4160
iter 8750: dev acc=0.4078
iter 9000: dev acc=0.4187
iter 9250: dev acc=0.4260
iter 9500: dev acc=0.4133
iter 9750: dev acc=0.4160
Iter 10000: loss=7610.4884, time=342.90s
iter 10000: dev acc=0.4223
Done training
Loading best model
best model iter 2000: train acc=0.5318, dev acc=0.4623, test acc=0.4710
iter 250: dev acc=0.3815
new highscore
iter 500: dev acc=0.4124
new highscore
iter 750: dev acc=0.4187
new highscore
iter 1000: dev acc=0.4378
new highscore
iter 1250: dev acc=0.4405
new highscore
iter 1500: dev acc=0.4450
new highscore
iter 1750: dev acc=0.4441
iter 2000: dev acc=0.4587
new highscore
iter 2250: dev acc=0.4532
iter 2500: dev acc=0.4423
iter 2750: dev acc=0.4587
iter 3000: dev acc=0.4369
iter 3250: dev acc=0.4541
iter 3500: dev acc=0.4314
iter 3750: dev acc=0.4532
iter 4000: dev acc=0.4432
iter 4250: dev acc=0.4469
iter 4500: dev acc=0.4133
iter 4750: dev acc=0.4323
iter 5000: dev acc=0.4360
iter 5250: dev acc=0.4242
iter 5500: dev acc=0.4296
iter 5750: dev acc=0.4260
iter 6000: dev acc=0.4432
iter 6250: dev acc=0.4296
iter 6500: dev acc=0.4205
iter 6750: dev acc=0.4187
iter 7000: dev acc=0.4078
iter 7250: dev acc=0.4269
iter 7500: dev acc=0.4269
iter 7750: dev acc=0.4396
iter 8000: dev acc=0.4214
iter 8250: dev acc=0.4169
iter 8500: dev acc=0.3969
iter 8750: dev acc=0.4051
iter 9000: dev acc=0.4214
iter 9250: dev acc=0.4033
iter 9500: dev acc=0.4087
iter 9750: dev acc=0.4151
Iter 10000: loss=7699.8478, time=334.05s
iter 10000: dev acc=0.4124
Done training
Loading best model
best model iter 2000: train acc=0.5296, dev acc=0.4587, test acc=0.4738
Skipping checkpoints/LSTM_04.pt
Skipping checkpoints/LSTM_05.pt
Skipping checkpoints/LSTM_06.pt
Skipping checkpoints/LSTM_07.pt
iter 250: dev acc=0.3806
new highscore
iter 500: dev acc=0.4133
new highscore
iter 750: dev acc=0.4178
new highscore
iter 1000: dev acc=0.4342
new highscore
iter 1250: dev acc=0.4396
new highscore
iter 1500: dev acc=0.4160
iter 1750: dev acc=0.4460
new highscore
iter 2000: dev acc=0.4596
new highscore
iter 2250: dev acc=0.4514
iter 2500: dev acc=0.4487
iter 2750: dev acc=0.4614
new highscore
iter 3000: dev acc=0.4541
iter 3250: dev acc=0.4587
iter 3500: dev acc=0.4587
iter 3750: dev acc=0.4641
new highscore
iter 4000: dev acc=0.4605
iter 4250: dev acc=0.4496
iter 4500: dev acc=0.4460
iter 4750: dev acc=0.4460
iter 5000: dev acc=0.4187
iter 5250: dev acc=0.4269
iter 5500: dev acc=0.4342
iter 5750: dev acc=0.3996
iter 6000: dev acc=0.4160
iter 6250: dev acc=0.4205
iter 6500: dev acc=0.4151
iter 6750: dev acc=0.4142
iter 7000: dev acc=0.4105
iter 7250: dev acc=0.4114
iter 7500: dev acc=0.4105
iter 7750: dev acc=0.4096
iter 8000: dev acc=0.4015
iter 8250: dev acc=0.4096
iter 8500: dev acc=0.3969
iter 8750: dev acc=0.3951
iter 9000: dev acc=0.3987
iter 9250: dev acc=0.4060
iter 9500: dev acc=0.4051
iter 9750: dev acc=0.4060
Iter 10000: loss=7684.0349, time=366.51s
iter 10000: dev acc=0.4042
Done training
Loading best model
best model iter 3750: train acc=0.6319, dev acc=0.4641, test acc=0.4679
iter 250: dev acc=0.3787
new highscore
iter 500: dev acc=0.4133
new highscore
iter 750: dev acc=0.4305
new highscore
iter 1000: dev acc=0.4342
new highscore
iter 1250: dev acc=0.4387
new highscore
iter 1500: dev acc=0.4423
new highscore
iter 1750: dev acc=0.4378
iter 2000: dev acc=0.4505
new highscore
iter 2250: dev acc=0.4569
new highscore
iter 2500: dev acc=0.4414
iter 2750: dev acc=0.4487
iter 3000: dev acc=0.4378
iter 3250: dev acc=0.4396
iter 3500: dev acc=0.4296
iter 3750: dev acc=0.4142
iter 4000: dev acc=0.4423
iter 4250: dev acc=0.4223
iter 4500: dev acc=0.4278
iter 4750: dev acc=0.4269
iter 5000: dev acc=0.4187
iter 5250: dev acc=0.4169
iter 5500: dev acc=0.4187
iter 5750: dev acc=0.4069
iter 6000: dev acc=0.4151
iter 6250: dev acc=0.4005
iter 6500: dev acc=0.3996
iter 6750: dev acc=0.4114
iter 7000: dev acc=0.4060
iter 7250: dev acc=0.4087
iter 7500: dev acc=0.4042
iter 7750: dev acc=0.4114
iter 8000: dev acc=0.4114
iter 8250: dev acc=0.4033
iter 8500: dev acc=0.4078
iter 8750: dev acc=0.4133
iter 9000: dev acc=0.4005
iter 9250: dev acc=0.4015
iter 9500: dev acc=0.4060
iter 9750: dev acc=0.3860
Iter 10000: loss=7595.0450, time=364.71s
iter 10000: dev acc=0.4051
Done training
Loading best model
best model iter 2250: train acc=0.5438, dev acc=0.4569, test acc=0.4710
lstm_val_acc, lstm_test_acc = eval_model_default(get_lstm_model(), "LSTM")Loading model
best model: train acc=0.5720, dev acc=0.4550, test acc=0.4629
Loading model
best model: train acc=0.5564, dev acc=0.4678, test acc=0.4615
Loading model
best model: train acc=0.5318, dev acc=0.4623, test acc=0.4710
Loading model
best model: train acc=0.5296, dev acc=0.4587, test acc=0.4738
Loading model
best model: train acc=0.5552, dev acc=0.4650, test acc=0.4561
Loading model
best model: train acc=0.5762, dev acc=0.4632, test acc=0.4534
Loading model
best model: train acc=0.5191, dev acc=0.4632, test acc=0.4819
Loading model
best model: train acc=0.4932, dev acc=0.4578, test acc=0.4710
Loading model
best model: train acc=0.6319, dev acc=0.4641, test acc=0.4679
Loading model
best model: train acc=0.5438, dev acc=0.4569, test acc=0.4710
==================================================
Test score: 46.71% (+-0.82)
Validation score: 46.14% (+-0.39)
# plot validation accuracy
# plt.plot(lstm_accuracies)_____no_output_____# plot training loss
# plt.plot(lstm_losses)_____no_output_____

# Tree LSTM_____no_output_____In the final part of this lab we will exploit the tree-structure of our data.
Until now we only used the surface tokens, but remember that our data examples include trees with a sentiment score at every node.

In particular, we will implement **N-ary Tree-LSTMs** which are described in:

> Kai Sheng Tai, Richard Socher, and Christopher D. Manning. [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks](http://aclweb.org/anthology/P/P15/P15-1150.pdf) ACL 2015.

Since our trees are binary, N=2, and we can refer to these as *Binary Tree-LSTMs*.

You should read this paper carefully and make sure that you understand the approach. You will also find our LSTM baseline there.
Note however that Tree LSTMs were also invented around the same time by two other groups:

> Phong Le and Willem Zuidema. [Compositional distributional semantics with long short term memory](http://anthology.aclweb.org/S/S15/S15-1002.pdf). *SEM 2015.

> Xiaodan Zhu, Parinaz Sobihani,  and Hongyu Guo. [Long short-term memory over recursive structures](http://proceedings.mlr.press/v37/zhub15.pdf). ICML 2015.

It is good scientific practice to cite all three papers in your report.

If you study equations (9) to (14) in the paper, you will find that they are not all too different from the original LSTM that you already have.

_____no_output_____## Computation

Do you remember the `transitions_from_treestring` function all the way in the beginning of this lab? Every example contains a **transition sequence** made by this function. Let's look at it again:

_____no_output_____

ex = next(examplereader("trees/dev.txt"))
print(TreePrettyPrinter(ex.tree))
print("Transitions:")
print(ex.transitions)_____no_output_____

Note that the tree is **binary**. Every node has two children, except for pre-terminal nodes.

A tree like this can be described by a sequence of **SHIFT (0)** and **REDUCE (1)** actions.

We can use the transitions like this to construct the tree:
- **reverse** the sentence (a list of tokens) and call this the **buffer**
   - the first word is now on top (last in the list), and we would get it when calling pop() on the buffer
- create an empty list and call it the **stack**
- iterate through the transition sequence:
  - if it says SHIFT(0), we pop a word from the buffer, and push it to the stack
  - if it says REDUCE(1), we pop the **top two items** from the stack, and combine them (e.g. with a tree LSTM!), creating a new node that we push back on the stack

Convince yourself that going through the transition sequence above will result in the tree that you see.
For example, we would start by putting the following words on the stack (by shifting 5 times, starting with `It`):

```
Top of the stack:
-----------------
film
lovely
a
's
It
```
Now we find a REDUCE in the transition sequence, so we get the top two words (film and lovely), and combine them, so our new stack becomes:
```
Top of the stack:
-----------------
lovely film
a
's
It
```

We will use this approach when encoding sentences with our Tree LSTM.
Now, our sentence is a (reversed) list of word embeddings.
When we shift, we move a word embedding to the stack.
When we reduce, we apply a Tree LSTM to the top two vectors, and the result is a single vector that we put back on the stack.
After going through the whole transition sequence, we will have the root node on our stack! We can use that to classify the sentence.

_____no_output_____## Obtaining the transition sequence
_____no_output_____
So what goes on in the `transitions_from_treestring` function?

The idea ([explained in this blog post](https://devblogs.nvidia.com/recursive-neural-networks-pytorch/)) is that, if we had a tree, we could traverse through the tree, and every time that we find a node containing only a word, we output a SHIFT.
Every time **after** we have finished visiting the children of a node, we output a REDUCE.
(What is this tree traversal called?)

However, our `transitions_from_treestring` function operates directly on the string representation. It works as follows.

We start with the representation:

```
(3 (2 It) (4 (4 (2 's) (4 (3 (2 a) (4 (3 lovely) (2 film))) (3 (2 with) (4 (3 (3 lovely) (2 performances)) (2 (2 by) (2 (2 (2 Buy) (2 and)) (2 Accorsi))))))) (2 .)))
```

First we remove pre-terminal nodes (and add spaces before closing brackets):

```
(3 It (4 (4 's (4 (3 a (4 lovely film ) ) (3 with (4 (3 lovely performances ) (2 by (2 (2 Buy and )  Accorsi ) ) ) ) ) ) . ) )
```

Then we remove node labels:

```
( It ( ( 's ( ( a ( lovely film ) ) ( with ( ( lovely performances) ( by ( ( Buy and )  Accorsi ) ) ) ) ) ) . ) )
```

Then we remove opening brackets:

```
It 's a lovely film ) ) with lovely performances ) by Buy and ) Accorsi ) ) ) ) ) ) . ) )
```

Now we replace words by S (for SHIFT), and closing brackets by R (for REDUCE):

```
S S S S S R R S S S R S S S R S R R R R R R S R R
0 0 0 0 0 1 1 0 0 0 1 0 0 0 1 0 1 1 1 1 1 1 0 1 1
```

Et voila. We just obtained the transition sequence!_____no_output_____

# for comparison
seq = ex.transitions
s = " ".join(["S" if t == 0 else "R" for t in seq])
print(s)
print(" ".join(map(str, seq)))_____no_output_____

## Coding the Tree LSTM

The code below contains a Binary Tree LSTM cell.
It is used in the TreeLSTM class below it, which in turn is used in the TreeLSTMClassifier.
The job of the TreeLSTM class is to encode a complete sentence and return the root node.
The job of the TreeLSTMCell is to return a new state when provided with two children (a reduce action). By repeatedly calling the TreeLSTMCell, the TreeLSTM will encode a sentence. This can be done for multiple sentences at the same time.

#### Exercise
Check the `forward` function and complete the Tree LSTM formulas.
You can see that we defined a large linear layer for you, that projects the *concatenation* of the left and right child into the input gate, left forget gate, right forget gate, candidate, and output gate._____no_output_____

class TreeLSTMCell(nn.Module):
    """A Binary Tree LSTM cell"""

    def __init__(self, input_size, hidden_size, bias=True):
        """Creates the weights for this LSTM"""
        super(TreeLSTMCell, self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size
        self.bias = bias

        self.reduce_layer = nn.Linear(2 * hidden_size, 5 * hidden_size)
        self.dropout_layer = nn.Dropout(p=0.25)
        self.tanh_act = nn.Tanh()
        self.sigmoid_act = nn.Sigmoid()

        self.reset_parameters()

    def reset_parameters(self):
        """This is PyTorch's default initialization method"""
        stdv = 1.0 / math.sqrt(self.hidden_size)
        for weight in self.parameters():
            weight.data.uniform_(-stdv, stdv)

    def forward(self, hx_l, hx_r, mask=None):
        """
        hx_l is ((batch, hidden_size), (batch, hidden_size))
        hx_r is ((batch, hidden_size), (batch, hidden_size))
        """
        prev_h_l, prev_c_l = hx_l  # left child
        prev_h_r, prev_c_r = hx_r  # right child

        B = prev_h_l.size(0)

        # we concatenate the left and right children
        # you can also project from them separately and then sum
        children = torch.cat([prev_h_l, prev_h_r], dim=1)

        # project the combined children into a 5D tensor for i,fl,fr,g,o
        # this is done for speed, and you could also do it separately
        proj = self.reduce_layer(children)  # shape: B x 5D

        # each shape: B x D
        i, f_l, f_r, g, o = torch.chunk(proj, 5, dim=-1)

        # main Tree LSTM computation

        # YOUR CODE HERE
        # You only need to complete the commented lines below.

        # The shape of each of these is [batch_size, hidden_size]

        i = self.sigmoid_act(i)
        f_l = self.sigmoid_act(f_l)
        f_r = self.sigmoid_act(f_r)
        g = self.tanh_act(g)
        o = self.sigmoid_act(o)

        c = i * g + f_l * prev_c_l + f_r * prev_c_r
        h = o * self.tanh_act(c)

        return h, c

    def __repr__(self):
        return "{}({:d}, {:d})".format(
            self.__class__.__name__, self.input_size, self.hidden_size)_____no_output_____

## Explanation of the TreeLSTM class

The code below contains the TreeLSTM class, which implements everything we need in order to encode a sentence from word embeddings. The calculations are the same as in the paper, implemented such that the class `TreeLSTMCell` above is as general as possible and only takes two children to reduce them into a parent.

**Initialize $\mathbf{h}$ and $\mathbf{c}$ outside of the cell for the leaves**

At the leaves of each tree the children nodes are **empty**, whereas in higher levels the nodes are binary tree nodes that *do* have a left and right child (but no input $x$). By initializing the leaf nodes outside of the cell class (`TreeLSTMCell`), we avoid if-else statements in the forward pass.

The `TreeLSTM` class (among other things) pre-calculates an initial $h$ and $c$ for every word in the sentence. Since the initial left and right child are 0, the only calculations we need to do are based on $x$, and we can drop the forget gate calculation (`prev_c_l` and `prev_c_r` are zero). The calculations we do in order to initalize $h$ and $c$ are then:

$$
c_1 =  W^{(u)}x_1 \\
o_1 = \sigma (W^{(i)}x_1) \\
h_1 = o_1 \odot \text{tanh}(c_1)$$
*NB: note that these equations are chosen as initializations of $c$ and $h$, other initializations are possible and might work equally well.*

**Sentence Representations**

All our leaf nodes are now initialized, so we can start processing the sentence in its tree form. Each sentence is represented by a buffer (initially a list with a concatenation of $[h_1, c_1]$ for every word in the reversed sentence), a stack (initially an empty list) and a transition sequence. To encode our sentence, we construct the tree from its transition sequence as explained earlier.

*A short example that constructs a tree:*

We loop over the time dimension of the batched transition sequences (i.e. row by row), which contain values of 0's, 1's and 2's (representing SHIFT, REDUCE and padding respectively). If we have a batch of size 2 where the first example has a transition sequence given by [0, 0, 1, 0, 0, 0, 1] and the second by [0, 0, 1, 0, 0, 1], our transition batch will be given by the following two-dimensional numpy array:

$$
\text{transitions} =
\begin{pmatrix}
0 & 0\\
0 & 0\\
1 & 1\\
0 & 0\\
0 & 0\\
0 & 1\\
1 & 2
\end{pmatrix}
$$
The inner loop (`for transition, buffer, stack in zip(t_batch, buffers, stacks)`) goes over each example in the batch and updates its buffer and stack. The nested loop for this example will then do roughy the following:

```
Time = 0:  t_batch = [0, 0], the inner loop performs 2 SHIFTs.

Time = 1:  t_batch = [0, 0], ".."

Time = 2:  t_batch = [1, 1], causing the inner loop to fill the list child_l and child_r for both examples in the batch. Now the statement if child_l will return True, triggering a REDUCE action to be performed by our Tree LSTM cell with a batch size of 2.

Time = 3:  t_batch = [0, 0], "..".

Time = 4:  t_batch = [0, 0], ".."

Time = 5:  t_batch = [0, 1], one SHIFT will be done and another REDUCE action will be performed by our Tree LSTM, this time of batch size 1.

Time = 6:  t_batch = [1, 2], triggering another REDUCE action with batch size 1.
```
*NB: note that this was an artificial example for the purpose of demonstrating parts of the code, the transition sequences do not necessarily represent actual trees.*

**Batching and Unbatching**

Within the body of the outer loop over time, we use the functions for batching and unbatching.

*Batching*

Before passing two lists of children to the reduce layer (an instance of `TreeLSTMCell`), we batch the children as they are at this point a list of tensors of variable length based on how many REDUCE actions there are to perform at a certain time step across the batch (let's call the length `L`). To do an efficient forward pass we want to transform the list to a pair of tensors of shape `([L, D], [L, D])`, which the function `batch` achieves.

*Unbatching*

In the same line where we batched the children, we unbatch the output of the forward pass to become a list of states of length `L` again. We do this because we need to loop over each example's transition at the current time step and push the children that are reduced into a parent to the stack.

*The batch and unbatch functions let us switch between the "PyTorch world" (Tensors) and the Python world (easy to manipulate lists).*
_____no_output_____

# Helper functions for batching and unbatching states
# For speed we want to combine computations by batching, but
# for processing logic we want to turn the output into lists again
# to easily manipulate.

def batch(states):
    """
    Turns a list of states into a single tensor for fast processing.
    This function also chunks (splits) each state into a (h, c) pair"""
    return torch.cat(states, 0).chunk(2, 1)

def unbatch(state):
    """
    Turns a tensor back into a list of states.
    First, (h, c) are merged into a single state.
    Then the result is split into a list of sentences.
    """
    return torch.split(torch.cat(state, 1), 1, 0)_____no_output_____

Take some time to understand the class below, having read the explanation above._____no_output_____

class TreeLSTM(nn.Module):
    """Encodes a sentence using a TreeLSTMCell"""

    def __init__(self, input_size, hidden_size, bias=True):
        """Creates the weights for this LSTM"""
        super(TreeLSTM, self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size
        self.bias = bias
        self.reduce = TreeLSTMCell(input_size, hidden_size)

        # project word to initial c
        self.proj_x = nn.Linear(input_size, hidden_size)
        self.proj_x_gate = nn.Linear(input_size, hidden_size)

        self.buffers_dropout = nn.Dropout(p=0.5)

    def forward(self, x, transitions):
        """
        WARNING: assuming x is reversed!
        :param x: word embeddings [B, T, E]
        :param transitions: [2T-1, B]
        :return: root states
        """

        B = x.size(0)  # batch size
        T = x.size(1)  # time

        # compute an initial c and h for each word
        # Note: this corresponds to input x in the Tai et al. Tree LSTM paper.
        # We do not handle input x in the TreeLSTMCell itself.
        buffers_c = self.proj_x(x)
        buffers_h = buffers_c.tanh()
        buffers_h_gate = self.proj_x_gate(x).sigmoid()
        buffers_h = buffers_h_gate * buffers_h

        # concatenate h and c for each word
        buffers = torch.cat([buffers_h, buffers_c], dim=-1)

        D = buffers.size(-1) // 2

        # we turn buffers into a list of stacks (1 stack for each sentence)
        # first we split buffers so that it is a list of sentences (length B)
        # then we split each sentence to be a list of word vectors
        buffers = buffers.split(1, dim=0)  # Bx[T, 2D]
        buffers = [list(b.squeeze(0).split(1, dim=0)) for b in buffers]  # BxTx[2D]

        # create B empty stacks
        stacks = [[] for _ in buffers]

        # t_batch holds 1 transition for each sentence
        for t_batch in transitions:

            child_l = []  # contains the left child for each sentence with reduce action
            child_r = []  # contains the corresponding right child

            # iterate over sentences in the batch
            # each has a transition t, a buffer and a stack
            for transition, buffer, stack in zip(t_batch, buffers, stacks):
                if transition == SHIFT:
                    stack.append(buffer.pop())
                elif transition == REDUCE:
                    assert len(stack) >= 2, \
                        "Stack too small! Should not happen with valid transition sequences"
                    child_r.append(stack.pop())  # right child is on top
                    child_l.append(stack.pop())

            # if there are sentences with reduce transition, perform them batched
            if child_l:
                reduced = iter(unbatch(self.reduce(batch(child_l), batch(child_r))))
                for transition, stack in zip(t_batch, stacks):
                    if transition == REDUCE:
                        stack.append(next(reduced))

        final = [stack.pop().chunk(2, -1)[0] for stack in stacks]
        final = torch.cat(final, dim=0)  # tensor [B, D]

        return final_____no_output_____

Just like the LSTM before, we will need an extra class that does the classifications._____no_output_____

class TreeLSTMClassifier(nn.Module):
    """Encodes sentence with a TreeLSTM and projects final hidden state"""

    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, vocab):
        super(TreeLSTMClassifier, self).__init__()
        self.vocab = vocab
        self.hidden_dim = hidden_dim
        self.embed = nn.Embedding(vocab_size, embedding_dim, padding_idx=1)
        self.treelstm = TreeLSTM(embedding_dim, hidden_dim)
        self.output_layer = nn.Sequential(
            nn.Dropout(p=0.5),
            nn.Linear(hidden_dim, output_dim, bias=True)
            )

    def forward(self, x):

        # x is a pair here of words and transitions; we unpack it here.
        # x is batch-major: [B, T], transitions is time major [2T-1, B]
        x, transitions = x
        emb = self.embed(x)

        # we use the root/top state of the Tree LSTM to classify the sentence
        root_states = self.treelstm(emb, transitions)

        # we use the last hidden state to classify the sentence
        logits = self.output_layer(root_states)
        return logits_____no_output_____

## Special prepare function for Tree LSTM

We need yet another prepare function. For our implementation our sentences to be *reversed*. We will do that here._____no_output_____

def prepare_treelstm_minibatch(mb, vocab):
    """
    Returns sentences reversed (last word first)
    Returns transitions together with the sentences.
    """
    batch_size = len(mb)
    maxlen = max([len(ex.tokens) for ex in mb])

    # vocab returns 0 if the word is not there
    # NOTE: reversed sequence!
    x = [pad([vocab.w2i.get(t, 0) for t in ex.tokens], maxlen)[::-1] for ex in mb]

    x = torch.LongTensor(x)
    x = x.to(device)

    y = [ex.label for ex in mb]
    y = torch.LongTensor(y)
    y = y.to(device)

    maxlen_t = max([len(ex.transitions) for ex in mb])
    transitions = [pad(ex.transitions, maxlen_t, pad_value=2) for ex in mb]
    transitions = np.array(transitions)
    transitions = transitions.T  # time-major

    return (x, transitions), y_____no_output_____

## Training_____no_output_____

# Now let's train the Tree LSTM!

tree_model = TreeLSTMClassifier(
    len(v.w2i), 300, 150, len(t2i), v)

with torch.no_grad():
    tree_model.embed.weight.data.copy_(torch.from_numpy(vectors))
    tree_model.embed.weight.requires_grad = False

def do_train(model):

    print(model)
    print_parameters(model)

    model = model.to(device)

    optimizer = optim.Adam(model.parameters(), lr=2e-4)

    return train_model(
        model, optimizer, num_iterations=30000,
        print_every=1, eval_every=250,
        prep_fn=prepare_treelstm_minibatch,
        eval_fn=evaluate,
        batch_fn=get_minibatch,
        batch_size=25, eval_batch_size=25)

results = do_train(tree_model)_____no_output_____# plot
plt.plot(results[0])_____no_output_____plt.plot(results[1])_____no_output_____

# Further experiments and report

For your report, you are expected to answer research questions by doing further experiments.

## Research Questions

Make sure you cover at least the following:

- How important is word order for this task?
- Does the tree structure help to get a better accuracy?
- How does performance depend on the sentence length? Compare the various models. Is there a model that does better on longer sentences? If so, why?
- Do you get better performance if you supervise the sentiment **at each node in the tree**? You can extract more training examples by treating every node in each tree as a separate tree. You will need to write a function that extracts all subtrees given a treestring.
    - Warning: NLTK's Tree function seems to result in invalid trees in some cases, so be careful if you want to parse the string to a tree structure before extraction the phrases.

Optionally, you can also investigate the following:

- When making a wrong prediction, can you figure out at what point in the tree (sentence) the model fails? You can make a prediction at each node to investigate.
- How does N-ary Tree LSTM compare to the Child-Sum Tree LSTM?
- How do the Tai et al. Tree LSTMs compare to Le & Zuidema's formulation?
- Or.. your own research question!

In general:

- ***When you report numbers, please report the mean accuracy across 3 (or more) runs with different random seed, together with the standard deviation.*** This is because the final performance may vary per random seed.

## Report instructions

Your report needs to be written in LaTeX. You are required to use the ACL 2018 template [(zip)](https://acl2018.org/downloads/acl18-latex.zip) which you can edit directly on [Overleaf](https://www.overleaf.com/latex/templates/instructions-for-acl-2018-proceedings/xzmhqgnmkppc). Make sure your names and student numbers are visible at the top. (Tip: you need to uncomment `/aclfinalcopy`).
You can find some general tips about writing a research paper [here](https://www.microsoft.com/en-us/research/academic-program/write-great-research-paper/), but note that you need to make your own judgment about what is appropriate for this project.

We expect you to use the following structure:
1. Introduction (~1 page) - describe the problem, your research questions and goals, a summary of your findings and contributions. Please cite related work (models, data set) as part of your introduction here, since this is a short paper.
    - Introduce the task and the main goal
    - Clear research questions
    - Motivating the importance of the questions and explaining the expectations
    - How are these addressed or not addressed in the literature
    - What is your approach
    - Short summary of your findings
2. Background (~1/2-1 page) -
cover the main techniques ("building blocks") used in your project (e.g. word embeddings, LSTM, Tree LSTM) and intuitions behind them. Be accurate and concise.
    - How each technique that you use works (don't just copy the formulas)
    - The relation between the techniques
3. Models (~1/2 page) - Cover the models that you used.
    - The architecture of the final models (how do you use LSTM or Tree LSTM for the sentiment classification task. what layers you have, how do you do classification? What is your loss function?)
4. Experiments (~1/2 page) - Describe your experimental setup. The information here should allow someone else to re-create your experiments. Describe how you evaluate the models.
    - Explain the task and the data
    - Training the models (model, data, parameters and hyper parameters if the models, training algorithms, what supervision signals you use, etc.)
    - Evaluation (e.g. metrics)
5. Results and Analysis (~1 page). Go over the results and analyse your findings.
    - Answer each of the research questions you raised in the introduction.
    - Plots and figures highlighting interesting patterns
    - What are the factors that makes model A  better than model B in task C? investigate to prove their effect!
6. Conclusion (~1/4 page). The main conclusions of your experiments.
    - What you learned from you experiments? how does it relate to what is already known in the literature?
    - Where the results as expected ? any surprising results? why?
    - Based on what you learned what would you suggest to do next?

General Tips:

- Math notation – define each variable (either in running text, or in a pseudo-legenda after or before the equation)
- Define technical terminology you need
- Avoid colloquial language – everything can be said in a scientific-sounding way
- Avoid lengthy sentences, stay to the point!
- Do not spend space on "obvious" things!

An ideal report:
- Precise, scientific-sounding, technical, to the point
  - Little general “waffle”/chit-chat
- Not boring – because you don’t explain obvious things too much
- Efficient delivery of (only) the facts that we need to know to understand/reimplement
- Results visually well-presented and described with the correct priority of importance of sub-results
- Insightful analysis – speculation should connect to something interesting and not be too much; the reader “learns something new”
- No typos, no colloquialisms – well-considered language
- This normally means several re-draftings (re-orderings of information)_____no_output_____# Iterative attention LSTMs_____no_output_____

class AttentionModule(nn.Module):
    """Realizes a very simple attention module. Takes last state and word into account with two-layer feedforward"""

    def __init__(self, embedding_dim, state_dim, hidden_dim):
        super(AttentionModule, self).__init__()

        self.embedding_dim = embedding_dim
        self.hidden_dim = hidden_dim
        self.state_dim = state_dim
        self.output_layer = nn.Sequential(OrderedDict([
            ('fc1', nn.Linear(embedding_dim+state_dim, hidden_dim)),
            ('tanh', nn.Tanh()),
            ('dropout', nn.Dropout(0.25)),
            ('fc2', nn.Linear(hidden_dim, 1)),
            ('sigmoid', nn.Sigmoid())
        ]))

        self.reset_parameters()

    def reset_parameters(self):
        """This is PyTorch's default initialization method"""
        stdv = 1.0 / math.sqrt(self.hidden_dim)
        for weight in self.parameters():
            weight.data.uniform_(-stdv, stdv)

    def forward(self, word_embed, last_state):

        input_tensor = torch.cat([word_embed, last_state], dim=1)

        for layer in self.output_layer:
            input_tensor = layer(input_tensor)

        return input_tensor
        _____no_output_____class IterativeLSTMClassifier(nn.Module):
    """Encodes sentence with an LSTM and projects final hidden state"""

    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, vocab, iterations):
        super(IterativeLSTMClassifier, self).__init__()
        self.vocab = vocab
        self.hidden_dim = hidden_dim
        self.embedding_size = embedding_dim
        self.embed = nn.Embedding(vocab_size, embedding_dim, padding_idx=1)
        self.rnn = MyLSTMCell(embedding_dim, hidden_dim)
        self.attention_module = AttentionModule(embedding_dim, hidden_dim, embedding_dim)
        self.iterations = iterations

        self.output_layer = nn.Sequential(
            nn.Dropout(p=0.25),
            nn.Linear(hidden_dim, output_dim)
        )

    def forward(self, x):

        B = x.size(0)  # batch size (this is 1 for now, i.e. 1 single example)
        T = x.size(1)  # time (the number of words in the sentence)

        input_ = self.embed(x)

        # here we create initial hidden states containing zeros
        # we use a trick here so that, if input is on the GPU, then so are hx and cx
        hx = input_.new_zeros(B, self.rnn.hidden_size)
        cx = input_.new_zeros(B, self.rnn.hidden_size)

        # process input sentences one word/timestep at a time
        # input is batch-major, so the first word(s) is/are input_[:, 0]
        for iter_ind in range(self.iterations):
            if iter_ind > 0:
                # hx has dimension B,hidden_size -> need to be B,T,hidden_size
                exp_hx = hx.unsqueeze(dim=1).repeat([1,T,1])
                attent_hx_input = exp_hx.view((B * T, self.rnn.hidden_size))
                attent_word_input = input_.view((B * T, self.embedding_size))
                print(attent_hx_input)
                print(attent_word_input)
                attent_scores = self.attention_module(attent_word_input, attent_hx_input)
                attent_scores = attent_scores.reshape(B,T,1)

            outputs = []
            for i in range(T):
                hx_new, cx_new = self.rnn(input_[:, i], (hx, cx))
                if iter_ind == 0:
                    hx = hx_new
                    cx = cx_new
                else:
                    hx = attention_score[:, i] * hx_new + (1 - attention_score[:, i]) * hx
                    cx = attention_score[:, i] * cx_new + (1 - attention_score[:, i]) * cx
                outputs.append(hx)

            # if we have a single example, our final LSTM state is the last hx
            if B == 1:
                last_final_states = hx
            else:
                #
                # This part is explained in next section, ignore this else-block for now.
                #
                # we processed sentences with different lengths, so some of the sentences
                # had already finished and we have been adding padding inputs to hx
                # we select the final state based on the length of each sentence

                # two lines below not needed if using LSTM form pytorch
                outputs = torch.stack(outputs, dim=0)          # [T, B, D]
                outputs = outputs.transpose(0, 1).contiguous()  # [B, T, D]

                # to be super-sure we're not accidentally indexing the wrong state
                # we zero out positions that are invalid
                pad_positions = (x == 1).unsqueeze(-1)

                outputs = outputs.contiguous()
                outputs = outputs.masked_fill_(pad_positions, 0.)

                mask = (x != 1)  # true for valid positions [B, T]
                lengths = mask.sum(dim=1)                  # [B, 1]

                indexes = (lengths - 1) + torch.arange(B, device=x.device, dtype=x.dtype) * T
                last_final_states = outputs.view(-1, self.hidden_dim)[indexes]  # [B, D]

            hx = last_final_states
            cx = input_.new_zeros(B, self.rnn.hidden_size)

        # we use the last hidden state to classify the sentence
        logits = self.output_layer(last_final_states)
        return logits_____no_output_____torch.cuda.empty_cache()
lstm_model = IterativeLSTMClassifier(
    len(v.w2i), 300, 168, len(t2i), v, 2)

# copy pre-trained vectors into embeddings table
with torch.no_grad():
    lstm_model.embed.weight.data.copy_(torch.from_numpy(vectors))
    lstm_model.embed.weight.requires_grad = False

print(lstm_model)
print_parameters(lstm_model)

lstm_model = lstm_model.to(device)

batch_size = 32
optimizer = optim.Adam(lstm_model.parameters(), lr=2e-4)

lstm_losses, lstm_accuracies = train_model(
    lstm_model, optimizer, num_iterations=30000,
    print_every=250, eval_every=250,
    batch_size=batch_size,
    batch_fn=get_minibatch,
    prep_fn=prepare_minibatch,
    eval_fn=evaluate)_____no_output_____del lstm_model
torch.cuda.empty_cache()_____no_output_____