# Notebook from sonalagrawal7/DeepLearning-nanodegree-exercises
Path: word2vec-embeddings/Skip_Grams_Exercise.ipynb

# Skip-gram Word2Vec

In this notebook, I'll lead you through using PyTorch to implement the [Word2Vec algorithm](https://en.wikipedia.org/wiki/Word2vec) using the skip-gram architecture. By implementing this, you'll learn about embedding words for use in natural language processing. This will come in handy when dealing with things like machine translation.

## Readings

Here are the resources I used to build this notebook. I suggest reading these either beforehand or while you're working on this material.

* A really good [conceptual overview](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/) of Word2Vec from Chris McCormick
* [First Word2Vec paper](https://arxiv.org/pdf/1301.3781.pdf) from Mikolov et al.
* [Neural Information Processing Systems, paper](http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) with improvements for Word2Vec also from Mikolov et al.

---
## Word embeddings

When you're dealing with words in text, you end up with tens of thousands of word classes to analyze; one for each word in a vocabulary. Trying to one-hot encode these words is massively inefficient because most values in a one-hot vector will be set to zero. So, the matrix multiplication that happens in between a one-hot input vector and a first, hidden layer will result in mostly zero-valued hidden outputs.

To solve this problem and greatly increase the efficiency of our networks, we use what are called **embeddings**. Embeddings are just a fully connected layer like you've seen before. We call this layer the embedding layer and the weights are embedding weights. We skip the multiplication into the embedding layer by instead directly grabbing the hidden layer values from the weight matrix. We can do this because the multiplication of a one-hot encoded vector with a matrix returns the row of the matrix corresponding the index of the "on" input unit.

Instead of doing the matrix multiplication, we use the weight matrix as a lookup table. We encode the words as integers, for example "heart" is encoded as 958, "mind" as 18094. Then to get hidden layer values for "heart", you just take the 958th row of the embedding matrix. This process is called an **embedding lookup** and the number of hidden units is the **embedding dimension**.

There is nothing magical going on here. The embedding lookup table is just a weight matrix. The embedding layer is just a hidden layer. The lookup is just a shortcut for the matrix multiplication. The lookup table is trained just like any weight matrix.

Embeddings aren't only used for words of course. You can use them for any model where you have a massive number of classes. A particular type of model called **Word2Vec** uses the embedding layer to find vector representations of words that contain semantic meaning._____no_output_____---
## Word2Vec

The Word2Vec algorithm finds much more efficient representations by finding vectors that represent the words. These vectors also contain semantic information about the words.

Words that show up in similar **contexts**, such as "coffee", "tea", and "water" will have vectors near each other. Different words will be further away from one another, and relationships can be represented by distance in vector space.

There are two architectures for implementing Word2Vec:
>* CBOW (Continuous Bag-Of-Words) and
* Skip-gram

In this implementation, we'll be using the **skip-gram architecture** because it performs better than CBOW. Here, we pass in a word and try to predict the words surrounding it in the text. In this way, we can train the network to learn representations for words that show up in similar contexts._____no_output_____---
## Loading Data

Next, we'll ask you to load in data and place it in the `data` directory

1. Load the [text8 dataset](https://s3.amazonaws.com/video.udacity-data.com/topher/2018/October/5bbe6499_text8/text8.zip); a file of cleaned up *Wikipedia article text* from Matt Mahoney.
2. Place that data in the `data` folder in the home directory.
3. Then you can extract it and delete the archive, zip file to save storage space.

After following these steps, you should have one file in your data directory: `data/text8`._____no_output_____

# read in the extracted text file
with open('data/text8') as f:
    text = f.read()

# print out the first 100 characters
print(text[:100]) anarchism originated as a term of abuse first used against early working class radicals including t

## Pre-processing

Here I'm fixing up the text to make training easier. This comes from the `utils.py` file. The `preprocess` function does a few things:
>* It converts any punctuation into tokens, so a period is changed to ` <PERIOD> `. In this data set, there aren't any periods, but it will help in other NLP problems.
* It removes all words that show up five or *fewer* times in the dataset. This will greatly reduce issues due to noise in the data and improve the quality of the vector representations.
* It returns a list of words in the text.

This may take a few seconds to run, since our text file is quite large. If you want to write your own functions for this stuff, go for it!_____no_output_____

import utils

# get list of words
words = utils.preprocess(text)
print(words[:30])['anarchism', 'originated', 'as', 'a', 'term', 'of', 'abuse', 'first', 'used', 'against', 'early', 'working', 'class', 'radicals', 'including', 'the', 'diggers', 'of', 'the', 'english', 'revolution', 'and', 'the', 'sans', 'culottes', 'of', 'the', 'french', 'revolution', 'whilst']
# print some stats about this word data
print("Total words in text: {}".format(len(words)))
print("Unique words: {}".format(len(set(words)))) # `set` removes any duplicate wordsTotal words in text: 16680599
Unique words: 63641

### Dictionaries

Next, I'm creating two dictionaries to convert words to integers and back again (integers to words). This is again done with a function in the `utils.py` file. `create_lookup_tables` takes in a list of words in a text and returns two dictionaries.
>* The integers are assigned in descending frequency order, so the most frequent word ("the") is given the integer 0 and the next most frequent is 1, and so on.

Once we have our dictionaries, the words are converted to integers and stored in the list `int_words`._____no_output_____

vocab_to_int, int_to_vocab = utils.create_lookup_tables(words)
int_words = [vocab_to_int[word] for word in words]

print(int_words[:30])[5233, 3080, 11, 5, 194, 1, 3133, 45, 58, 155, 127, 741, 476, 10571, 133, 0, 27349, 1, 0, 102, 854, 2, 0, 15067, 58112, 1, 0, 150, 854, 3580]

## Subsampling

Words that show up often such as "the", "of", and "for" don't provide much context to the nearby words. If we discard some of them, we can remove some of the noise from our data and in return get faster training and better representations. This process is called subsampling by Mikolov. For each word $w_i$ in the training set, we'll discard it with probability given by

$$ P(w_i) = 1 - \sqrt{\frac{t}{f(w_i)}} $$

where $t$ is a threshold parameter and $f(w_i)$ is the frequency of word $w_i$ in the total dataset.

$$ P(0) = 1 - \sqrt{\frac{1*10^{-5}}{1*10^6/16*10^6}} = 0.98735 $$

I'm going to leave this up to you as an exercise. Check out my solution to see how I did it.

> **Exercise:** Implement subsampling for the words in `int_words`. That is, go through `int_words` and discard each word given the probablility $P(w_i)$ shown above. Note that $P(w_i)$ is the probability that a word is discarded. Assign the subsampled data to `train_words`._____no_output_____

from collections import Counter
import random
import numpy as np

threshold = 1e-5
word_counts = Counter(int_words)
print(list(word_counts.items())[0])  # dictionary of int_words, how many times they appear

total_count = len(int_words)
freqs = {word: count/total_count for word, count in word_counts.items()}
p_drop = {word: 1 - np.sqrt(threshold/freqs[word]) for word in word_counts}

counter = 0
for k,v in p_drop.items():
    print(k,v)
    counter += 1
    if counter >= 5:
        break

        # discard some frequent words, according to the subsampling equation
# create a new list of words for training
train_words = [word for word in int_words if random.random() < (1 - p_drop[word])]

print(train_words[:30])(5233, 303)
5233 0.258033141925
3080 0.459982533952
11 0.964426761503
5 0.977375362388
194 0.847991700673
[5233, 10571, 27349, 15067, 58112, 854, 3580, 10712, 1324, 362, 3672, 708, 53, 539, 2757, 567, 7088, 27, 44611, 2877, 186, 5233, 8983, 279, 4147, 6437, 4186, 362, 5233, 1818]

## Making batches_____no_output_____Now that our data is in good shape, we need to get it into the proper form to pass it into our network. With the skip-gram architecture, for each word in the text, we want to define a surrounding _context_ and grab all the words in a window around that word, with size $C$.

From [Mikolov et al.](https://arxiv.org/pdf/1301.3781.pdf):

"Since the more distant words are usually less related to the current word than those close to it, we give less weight to the distant words by sampling less from those words in our training examples... If we choose $C = 5$, for each training word we will select randomly a number $R$ in range $[ 1: C ]$, and then use $R$ words from history and $R$ words from the future of the current word as correct labels."

> **Exercise:** Implement a function `get_target` that receives a list of words, an index, and a window size, then returns a list of words in the window around the index. Make sure to use the algorithm described above, where you chose a random number of words to from the window.

Say, we have an input and we're interested in the idx=2 token, `741`:
```
[5233, 58, 741, 10571, 27349, 0, 15067, 58112, 3580, 58, 10712]
```

For `R=2`, `get_target` should return a list of four values:
```
[5233, 58, 10571, 27349]
```_____no_output_____

def get_target(words, idx, window_size=5):
    ''' Get a list of words in a window around an index. '''

    # implement this function
    R = np.random.randint(1, window_size+1)
    start = idx - R if (idx - R) > 0 else 0
    stop = idx + R
    #print(R,start, stop)
    target_words = words[start:idx] + words[idx+1:stop+1]

    return list(target_words)
_____no_output_____# test your code!

# run this cell multiple times to check for random window selection
int_text = [i for i in range(10)]
print('Input: ', int_text)
idx=5 # word index of interest

target = get_target(int_text, idx=idx, window_size=5)
print('Target: ', target)  # you should get some indices around the idxInput:  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Target:  [1, 2, 3, 4, 6, 7, 8, 9]

### Generating Batches

Here's a generator function that returns batches of input and target data for our model, using the `get_target` function from above. The idea is that it grabs `batch_size` words from a words list. Then for each of those batches, it gets the target words in a window._____no_output_____

def get_batches(words, batch_size, window_size=5):
    ''' Create a generator of word batches as a tuple (inputs, targets) '''

    n_batches = len(words)//batch_size

    # only full batches
    words = words[:n_batches*batch_size]

    for idx in range(0, len(words), batch_size):
        x, y = [], []
        batch = words[idx:idx+batch_size]
        for ii in range(len(batch)):
            batch_x = batch[ii]
            batch_y = get_target(batch, ii, window_size)
            y.extend(batch_y)
            x.extend([batch_x]*len(batch_y))
        yield x, y
    _____no_output_____int_text = [i for i in range(20)]
x,y = next(get_batches(int_text, batch_size=4, window_size=5))

print('x\n', x)
print('y\n', y)x
 [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3]
y
 [1, 2, 3, 0, 2, 3, 0, 1, 3, 0, 1, 2]

## Building the graph

Below is an approximate diagram of the general structure of our network.

>* The input words are passed in as batches of input word tokens.
* This will go into a hidden layer of linear units (our embedding layer).
* Then, finally into a softmax output layer.

We'll use the softmax layer to make a prediction about the context words by sampling, as usual.

The idea here is to train the embedding layer weight matrix to find efficient representations for our words. We can discard the softmax layer because we don't really care about making predictions with this network. We just want the embedding matrix so we can use it in _other_ networks we build using this dataset._____no_output_____---
## Validation

Here, I'm creating a function that will help us observe our model as it learns. We're going to choose a few common words and few uncommon words. Then, we'll print out the closest words to them using the cosine similarity:

$$
\mathrm{similarity} = \cos(\theta) = \frac{\vec{a} \cdot \vec{b}}{|\vec{a}||\vec{b}|}
$$

We can encode the validation words as vectors $\vec{a}$ using the embedding table, then calculate the similarity with each word vector $\vec{b}$ in the embedding table. With the similarities, we can print out the validation words and words in our embedding table semantically similar to those words. It's a nice way to check that our embedding table is grouping together words with similar semantic meanings._____no_output_____

def cosine_similarity(embedding, valid_size=16, valid_window=100, device='cpu'):
    """ Returns the cosine similarity of validation words with words in the embedding matrix.
        Here, embedding should be a PyTorch embedding module.
    """

    # Here we're calculating the cosine similarity between some random words and
    # our embedding vectors. With the similarities, we can look at what words are
    # close to our random words.

    # sim = (a . b) / |a||b|

    embed_vectors = embedding.weight

    # magnitude of embedding vectors, |b|
    magnitudes = embed_vectors.pow(2).sum(dim=1).sqrt().unsqueeze(0)

    # pick N words from our ranges (0,window) and (1000,1000+window). lower id implies more frequent
    valid_examples = np.array(random.sample(range(valid_window), valid_size//2))
    valid_examples = np.append(valid_examples,
                               random.sample(range(1000,1000+valid_window), valid_size//2))
    valid_examples = torch.LongTensor(valid_examples).to(device)

    valid_vectors = embedding(valid_examples)
    similarities = torch.mm(valid_vectors, embed_vectors.t())/magnitudes

    return valid_examples, similarities_____no_output_____

## SkipGram model

Define and train the SkipGram model.
> You'll need to define an [embedding layer](https://pytorch.org/docs/stable/nn.html#embedding) and a final, softmax output layer.

An Embedding layer takes in a number of inputs, importantly:
* **num_embeddings** – the size of the dictionary of embeddings, or how many rows you'll want in the embedding weight matrix
* **embedding_dim** – the size of each embedding vector; the embedding dimension_____no_output_____

import torch
from torch import nn
import torch.optim as optim_____no_output_____class SkipGram(nn.Module):
    def __init__(self, n_vocab, n_embed):
        super().__init__()
        #nn.Embedding(num_embeddings, embedding_dim, padding_idx=None, max_norm=None, norm_type=2.0, scale_grad_by_freq=False, sparse=False, _weight=None)
        # complete this SkipGram model
        self.embed = nn.Embedding(n_vocab,n_embed)
        self.output = nn.Linear(n_embed,n_vocab)
        self.log_softmax = nn.LogSoftmax(dim=1)

    def forward(self, x):

        # define the forward behavior
        x = self.embed(x)
        scores = self.output(x)
        log_ps = self.log_softmax(scores)

        return log_ps_____no_output_____

### Training

Below is our training loop, and I recommend that you train on GPU, if available.

**Note that, because we applied a softmax function to our model output, we are using NLLLoss** as opposed to cross entropy. This is because Softmax  in combination with NLLLoss = CrossEntropy loss ._____no_output_____

# check if GPU is available
device = 'cuda' if torch.cuda.is_available() else 'cpu'

embedding_dim=300 # you can change, if you want

model = SkipGram(len(vocab_to_int), embedding_dim).to(device)
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.003)

print_every = 500
steps = 0
epochs = 2

# train for some number of epochs
for e in range(epochs):

    # get input and target batches
    for inputs, targets in get_batches(train_words, 512):
        steps += 1
        inputs, targets = torch.LongTensor(inputs), torch.LongTensor(targets)
        inputs, targets = inputs.to(device), targets.to(device)

        log_ps = model(inputs)
        loss = criterion(log_ps, targets)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if steps % print_every == 0:
            # getting examples and similarities
            valid_examples, valid_similarities = cosine_similarity(model.embed, device=device)
            _, closest_idxs = valid_similarities.topk(6) # topk highest similarities

            valid_examples, closest_idxs = valid_examples.to('cpu'), closest_idxs.to('cpu')
            for ii, valid_idx in enumerate(valid_examples):
                closest_words = [int_to_vocab[idx.item()] for idx in closest_idxs[ii]][1:]
                print(int_to_vocab[valid_idx.item()] + " | " + ', '.join(closest_words))
            print("...")after | fdr, montane, lived, finch, connemara
one | cesium, pierluigi, magma, increasingly, epigram
not | sampling, strengths, inscription, encoding, mori
while | stoa, imagining, blossomed, prints, rathdown
may | dag, eulemur, manon, restricting, glasnost
its | fari, you, experimenting, titled, romances
who | transposition, vasili, opposites, moog, tutuila
new | graeme, fitzgerald, risky, cds, invading
placed | corbijn, wolfenstein, stansted, off, weishaupt
experience | olcott, linearity, sambia, pirates, secaucus
consists | materialist, cyprus, regia, satirized, allegri
creation | gish, guest, heiress, fischer, pamphilus
egypt | blacks, pearls, accusations, channeling, ssy
test | zinoviev, struggle, nuts, harvester, unfortunately
construction | ferrara, novelties, patently, oligocene, pled
bible | closeted, girdle, collet, fish, mulhouse
...
th | poissy, angers, macro, saboteurs, dodging
seven | kivu, appeasement, forcings, durst, statistician
use | flywheels, doers, decompression, seedless, gui
from | vague, polydor, perro, primed, hausa
state | troilus, hornet, wh, render, templi
is | bedi, daughters, gourd, blessing, bandar
years | suit, mahwah, eschews, triumph, cube
it | banana, pottery, amoeboids, machado, visionary
issue | lilian, cardigan, accreditation, forcefully, wedge
troops | thugs, hamsun, romulus, bumpy, commendation
cost | motors, sica, switchover, popularising, mexicanus
pope | grundy, bimba, cle, bracelet, conceptualized
notes | mysql, whitespace, ribosomal, pieds, zombie
egypt | blacks, pearls, accusations, channeling, ssy
channel | seaworld, dunraven, discernible, dosage, mortal
award | facilitation, flynt, chretien, africana, scripts
...
zero | northamptonshire, penumbral, urey, bushes, eponym
at | rohrabacher, benjamin, informative, jfduke, dinara
that | summarily, defending, logograms, berkshire, oppressor
with | windmill, treacherously, cracow, acids, hoarded
where | dinars, quale, filesystem, hain, erotic
an | prosecutors, mpr, idris, rhee, mapquest
into | carburetor, reverie, honored, info, irma
war | worded, aphelion, lemurs, godley, cryptozoology
shows | dharmas, florentines, asleep, obsolescent, unloaded
police | bras, right, learns, guelph, joins
assembly | blurred, ein, tombstones, helen, ingots
issue | forcefully, lilian, cardigan, accreditation, wedge
derived | riley, pushcha, partito, flashback, metering
stage | joyfully, eurocents, digesting, hydropower, bleuler
notes | mysql, ribosomal, whitespace, pieds, toile
frac | mahal, tribute, mythic, remodelled, akihiko
...
th | poissy, estate, ring, stationing, saboteurs
at | rohrabacher, informative, dinara, jfduke, implicit
first | recreationally, freiherr, quarterback, waterfall, considerations
where | dinars, quale, hain, filesystem, farrakhan
after | fdr, lived, montane, finch, ulm
these | diophantus, tsunami, wilkie, susanna, photons
called | passy, feeders, keitai, tland, armas
there | beauties, heterozygous, sending, espoused, whither
notes | mysql, ribosomal, whitespace, soulful, pieds
test | zinoviev, nuts, unfortunately, modulating, forecasting
except | torquato, elbing, trolley, forwarded, entr
governor | naiste, delson, boone, frantic, pushcha
joseph | art, cleric, inherently, workbench, roussimoff
experience | linearity, olcott, sambia, harpists, secaucus
egypt | blacks, accusations, pearls, channeling, ssy
additional | geneticist, locke, odyssey, disguises, pentagons
...
to | chivalry, schematics, oceanic, sacramental, foretold
but | permitting, sayer, tudors, antidisestablishmentarianism, rattigan
three | gwynn, pliers, gregg, ulrich, nils
these | diophantus, tsunami, wilkie, decimal, profiled
between | interferes, fifty, madness, atlantic, auld
b | hedonist, rerio, minoan, hornsby, confessions
up | wolsey, hoshea, bermudez, puzzled, giovanni
when | centralize, reservist, deceptive, doves, agriculture
units | supercontinent, supernaturalism, roskilde, sprawl, templi
taking | wink, meran, ila, striated, figs
notes | mysql, ribosomal, whitespace, soulful, pieds
mathematics | metadata, clauses, persecutors, colditz, intimidation
experience | linearity, olcott, sambia, harpists, secaucus
gold | sla, iupac, lush, multipath, fink
orthodox | calibrate, webcomics, ambon, iain, christianity
writers | urheimat, uproar, mlp, nanette, supermodel
...
who | transposition, tutuila, vasili, kellerman, fran
often | gonads, arachnoid, spectrometer, skimming, supremum
many | batavia, chilly, margo, harpsichordist, politican
more | sanatan, chlorine, morons, sword, reproduces
while | imagining, stoa, djvu, prints, displaced
had | wallonia, theology, wry, thresholds, appreciate
where | dinars, quale, hain, perceptible, filesystem
seven | kivu, statistician, wife, bju, nine
notes | mysql, ribosomal, whitespace, soulful, reymond
additional | geneticist, locke, odyssey, pentagons, basic
assembly | helen, blurred, ingots, glenn, nos
applied | medlineplus, gas, much, sejanus, railcar
bbc | alva, drm, warwickshire, aryana, foul
units | supercontinent, templi, supernaturalism, roskilde, sprawl
freedom | stratified, tempt, mediators, martel, pairwise
construction | ferrara, armagh, patently, novelties, oligocene
...
as | dabbalemi, cassette, polygamous, bokken, klansman
world | outline, nne, dalnet, engaged, strachey
into | info, carburetor, irma, sentience, reverie
while | stoa, imagining, prints, teaching, seminumerical
seven | kivu, nine, bucks, statistician, wife
th | saboteurs, poissy, estate, stationing, unsolicited
this | casserole, projectile, contacting, heresy, administering
not | encoding, strengths, wavelets, sampling, wrestled
liberal | khz, ohms, sternberg, merger, fixer
arts | gunnery, campus, stockholm, karma, inlay
animals | jerky, satan, politiques, protect, latte
heavy | airborne, pitching, koivisto, financed, haiku
shown | friedman, boiled, thucydides, kang, terence
magazine | eddies, ramakrishna, potter, manatee, gunboats
marriage | truss, commissions, reflections, screwball, maharishi
articles | dignity, bqp, dhea, eton, fruity
...
united | breatharianism, stranger, special, wo, dandy
american | footballer, enlisted, grabbed, doren, kanto
see | logia, burbank, spares, negroes, skewness
called | feeders, keitai, passy, tland, hagi
states | worldcom, geomorphology, esthesic, seduce, preeminence
used | pressuring, scrolls, valentin, many, green
on | ghostbusters, orientated, pipp, galleria, spoiling
seven | kivu, statistician, nine, bucks, bju
liberal | khz, sternberg, merger, dolchsto, fixer
placed | stansted, homophone, wolfenstein, corbijn, sling
instance | ext, tallest, morphophonemic, addressing, deletes
bbc | alva, broadcast, foul, aryana, warwickshire
police | learns, right, bras, guelph, joins
square | skating, outdoor, clashed, star, deadly
dr | jon, invent, adversus, adolescence, tuan
pressure | lattices, referee, seleucids, clove, guns
...
of | article, jodie, breathed, occidental, unsubstantiated
by | keating, mensch, trumped, erc, timed
more | sanatan, morons, chlorine, muhsin, lymphedema
seven | nine, kivu, statistician, bucks, bju
between | interferes, fifty, connected, madness, auld
if | secretion, typically, consonantal, ince, riggs
they | explains, bureaucrat, ami, literatures, predictions
were | pratique, deepened, garment, calc, pivots
alternative | playhouse, illa, dinar, condiments, polyester
ice | cbs, tartaglia, marriott, tehran, whiteout
instance | for, addressing, morphophonemic, ext, tallest
articles | dhea, dignity, bqp, freedman, rtgs
bbc | broadcast, alva, foul, warwickshire, aryana
centre | favourite, benefiting, interactions, perrault, carotene
writers | mlp, uproar, nanette, eyre, urheimat
joseph | cleric, prof, inherently, uribe, fl
...
there | emplaced, sending, beauties, pasteur, heterozygous
seven | nine, statistician, kivu, durst, wife
he | willful, implantable, sibyl, averted, hersh
his | floated, unintelligent, enol, retreats, awakens
four | meitnerium, apprentice, increasing, aesthetic, guericke
who | vasili, difficulties, tutuila, celebrating, amusing
all | faceted, intercalary, exposition, desoto, homozygous
first | freiherr, quarterback, ferri, recreationally, waterfall
discovered | tunis, thinks, samoan, munroe, sewers
hold | makah, symbols, gosling, fgc, weathering
heavy | airborne, koivisto, pitching, pomona, financed
governor | naiste, spotters, cca, delson, pushcha
freedom | mediators, tempt, martel, kilby, pairwise
notes | mysql, whitespace, enslavable, ribosomal, soulful
universe | cantor, bother, emerges, puzzle, singing
stage | wendy, eurocents, inns, repainted, haer
...

## Visualizing the word vectors

Below we'll use T-SNE to visualize how our high-dimensional word vectors cluster together. T-SNE is used to project these vectors into two dimensions while preserving local stucture. Check out [this post from Christopher Olah](http://colah.github.io/posts/2014-10-Visualizing-MNIST/) to learn more about T-SNE and other ways to visualize high-dimensional data._____no_output_____

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.pyplot as plt
from sklearn.manifold import TSNE_____no_output_____# getting embeddings from the embedding layer of our model, by name
embeddings = model.embed.weight.to('cpu').data.numpy()_____no_output_____viz_words = 600
tsne = TSNE()
embed_tsne = tsne.fit_transform(embeddings[:viz_words, :])_____no_output_____fig, ax = plt.subplots(figsize=(16, 16))
for idx in range(viz_words):
    plt.scatter(*embed_tsne[idx, :], color='steelblue')
    plt.annotate(int_to_vocab[idx], (embed_tsne[idx, 0], embed_tsne[idx, 1]), alpha=0.7)_____no_output_____


# Notebook from ShepherdCode/ShepherdML
Path: Project/lncRNA_14g_Kmers.ipynb

# RNN on RNA K-mers

Extract every K-mer for K=3.
Apply one-hot encoding.
Convert to ragged tensors.
Train with 3-deep RNN.

Previous runs exhausted RAM.
This version trains the same model additively on small batches of data._____no_output_____

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import OneHotEncoder
from sklearn.model_selection import StratifiedShuffleSplit
from sklearn.model_selection import ShuffleSplit
from datetime import datetime

import tensorflow as tf
from tensorflow import keras
tf.keras.backend.set_floatx('float32')  # save RAM_____no_output_____K=3   # K-mer length = word length

def showtime():
    now=datetime.now()
    current=now.strftime("%H:%M:%S")
    print("Time=",current)

showtime()
print("Build the model")
seq_len=None  # none indicates variable length
input_features=4**K   # 64 DNA K-mers at K=3
rnn2 = keras.models.Sequential([
    keras.layers.SimpleRNN(64, return_sequences=True, input_shape=[seq_len,input_features]),
    keras.layers.SimpleRNN(64, return_sequences=True),
    keras.layers.SimpleRNN(64, return_sequences=True),
    keras.layers.SimpleRNN(1),
])

showtime()
print("Build the training environment")
bc=tf.keras.losses.BinaryCrossentropy(from_logits=False)
rnn2.compile(loss=bc, optimizer="Adam",metrics=["accuracy"])
rnn2.summary()Time= 08:30:37
Build the model
Time= 08:30:37
Build the training environment
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
simple_rnn (SimpleRNN)       (None, None, 64)          8256
_________________________________________________________________
simple_rnn_1 (SimpleRNN)     (None, None, 64)          8256
_________________________________________________________________
simple_rnn_2 (SimpleRNN)     (None, None, 64)          8256
_________________________________________________________________
simple_rnn_3 (SimpleRNN)     (None, 1)                 66
=================================================================
Total params: 24,834
Trainable params: 24,834
Non-trainable params: 0
_________________________________________________________________
MIN_SEQ_LEN=200   ### by definition, lncRNA have min len 200
#MIN_SEQ_LEN=1000  ### we use this to reduce training RAM and CPU
MAX_SEQ_LEN=25000 ### this screens 4 outliers in the complete dataset
#MAX_SEQ_LEN=2000  ### we use this to reduce training RAM and CPU
DEFLINE='>'
ncfile='ncRNA.fasta'
pcfile='pcRNA.fasta'
#ncfile='tiny.ncRNA.fasta' # 10 seqs for faster debugging
#pcfile='tiny.pcRNA.fasta' # 10 seqs for faster debugging

#from google.colab import drive
#drive.mount('/content/drive', force_remount=True)
## !ls drive/'My Drive'/'Colab Notebooks'/*
#ncfile='drive/My Drive/Colab Notebooks/ncRNA.fasta'
#pcfile='drive/My Drive/Colab Notebooks/pcRNA.fasta'

# Assume file was preprocessed to contain one line per seq.
# Returned structure is ndarray of ndarray i.e no python lists inside.
def load_fasta(filename,min_len,max_len):
    seqs=[]
    with open (filename,'r') as infile:
        for line in infile:
            if line[0]!=DEFLINE and len(line)>min_len and len(line)<=max_len:
                line=line.rstrip()
                linelen=len(line)
                kmers=[]
                for i in range(linelen-K+1): # e.g. k=3,|line=4| => range=2 so i={0,1}
                    kmer=line[i:i+K]
                    kmers.append(kmer)
                kmers=np.array(kmers)
                seqs.append(kmers.reshape(-1, 1)) # reshaped changes (any,) to (any,1)
    nparray=np.array(seqs)
    return nparray

showtime()
print("Load "+ncfile)
nc_seqs = load_fasta(ncfile,MIN_SEQ_LEN,MAX_SEQ_LEN)
encoder = OneHotEncoder(handle_unknown='ignore',sparse=False)
#seq=tf.reshape(nc_seqs[0],shape=(-1, 1)) # tensor flow version
seq=nc_seqs[0].reshape(-1, 1)
encoder.fit(seq)
print("Encoder categories")
print(str(encoder.categories_))
nc_seqs = None
seq=NoneTime= 08:30:37
Load ncRNA.fasta
Encoder categories
[array(['AAA', 'AAC', 'AAG', 'AAT', 'ACA', 'ACC', 'ACG', 'ACT', 'AGA',
       'AGC', 'AGG', 'AGT', 'ATA', 'ATC', 'ATG', 'ATT', 'CAA', 'CAC',
       'CAG', 'CAT', 'CCA', 'CCC', 'CCG', 'CCT', 'CGA', 'CGC', 'CGG',
       'CGT', 'CTA', 'CTC', 'CTG', 'CTT', 'GAA', 'GAC', 'GAG', 'GAT',
       'GCA', 'GCC', 'GCG', 'GCT', 'GGA', 'GGC', 'GGG', 'GGT', 'GTA',
       'GTC', 'GTG', 'GTT', 'TAA', 'TAC', 'TAG', 'TAT', 'TCA', 'TCC',
       'TCG', 'TCT', 'TGA', 'TGC', 'TGG', 'TGT', 'TTA', 'TTC', 'TTG',
       'TTT'], dtype='<U3')]

## Loop: additively train the model in small batches._____no_output_____

EPOCHS=5
BATCHES=6
# batch max: 1K, 2K, 4K, 8K, 16K, 32K
for epoch in range(EPOCHS):
    print("Epoch %d"%epoch)
    batch_max = 0
    for batch in range(BATCHES):
        showtime()
        print ("Batch %d"%batch)
        batch_min = batch_max
        batch_max = 1000 * (2**batch)
        print("Batch of RNA lengths %d to %d"%(batch_min,batch_max))

        print("Load "+ncfile)
        nc_seqs = load_fasta(ncfile,batch_min,batch_max)
        print("Load "+pcfile)
        pc_seqs = load_fasta(pcfile,batch_min,batch_max)

        print("Ecode the non-coding sequences")
        nc_list=[]
        for seq in nc_seqs:
            encoded=encoder.transform(seq)
            nc_list.append(encoded)

        nc_len=len(nc_list)
        nc_all=np.array(nc_list) # .reshape(-1,1)
        print("nc shape: "+str(nc_all.shape))

        print("Encode the protein-coding sequences")
        pc_list=[]
        for seq in pc_seqs:
            encoded=encoder.transform(seq)
            pc_list.append(encoded)

        pc_len=len(pc_list)
        pc_all=np.array(pc_list) # .reshape(-1,1)
        print("pc shape: "+str(pc_all.shape))

        nc_labels=np.zeros(shape=(nc_len))
        pc_labels=np.ones(shape=(pc_len))
        all_seqs=np.concatenate((nc_all,pc_all),axis=0)
        all_labels=np.concatenate((nc_labels,pc_labels),axis=0)

        print("shape of sequences, shape of labels")
        all_seqs.shape,all_labels.shape

        splitter = ShuffleSplit(n_splits=1, test_size=0.2, random_state=37863)
        for train_index,test_index in splitter.split(all_seqs):
            train_seqs =   all_seqs[train_index]
            train_labels = all_labels[train_index]
            test_seqs =    all_seqs[test_index]
            test_labels =  all_labels[test_index]

        print("shape of train")
        print(train_seqs.shape,train_labels.shape)
        print("shape of test")
        print(test_seqs.shape,test_labels.shape)

        print("Convert numpy array to python 3D array")
        def numpy_to_python_3D(np_seqs):
            one_set = []
            tlen = len(np_seqs)
            for i in range(tlen): # for every sequence in set
                one_seq = []
                slen = len(np_seqs[i])
                for j in range(slen): # for ever letter in sequence
                    one_letter=np_seqs[i][j]
                    one_seq.append(one_letter)
                one_set.append(one_seq)
            return one_set
        train_seqs = numpy_to_python_3D(train_seqs)
        test_seqs = numpy_to_python_3D(test_seqs)
        train_labels = train_labels.tolist()
        test_labels = test_labels.tolist()

        # Now that values are shuffled, partition gives random sample.
        data_size=len(train_seqs)
        PARTITION=int(data_size*0.8)
        print("Partition = "+str(PARTITION))

        print("Partition train/validation")
        X_train=train_seqs[:PARTITION]
        X_valid=train_seqs[PARTITION:]
        y_train=train_labels[:PARTITION]
        y_valid=train_labels[PARTITION:]
        print("Lengths of train,valid")
        print((len(X_train),len(X_valid)))

        print("Free memory.")
        nc_list=None
        pc_list=None
        nc_all=None
        pc_all=None
        nc_seqs=None
        pc_seqs=None
        all_seqs=None
        nc_labels=None
        pc_labels=None
        train_seqs=None
        train_labels=None

        showtime()
        print("Convert to tensors")
        tensor = tf.ragged.constant(X_train)
        X_train = tensor
        tensor=None
        print(type(X_train))
        print(X_train.shape)
        tensor = tf.convert_to_tensor(y_train)
        y_train = tensor
        tensor=None
        print(type(y_train))
        print(y_train.shape)
        tensor = tf.ragged.constant(X_valid)
        X_valid=tensor
        tensor=None
        print(type(X_valid))
        print(X_valid.shape)
        tensor = tf.convert_to_tensor(y_valid)
        y_valid = tensor
        tensor=None
        print(type(y_valid))
        print(y_valid.shape)

        showtime()
        print("Resume training the model")
        history = rnn2.fit(X_train,y_train,epochs=1,
                           validation_data=(X_valid,y_valid))Epoch 0
Time= 08:30:48
Batch 0
Batch of RNA lengths 0 to 1000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (11415,)
Encode the protein-coding sequences
pc shape: (7270,)
shape of sequences, shape of labels
shape of train
(14948,) (14948,)
shape of test
(3737,) (3737,)
Convert numpy array to python 3D array
Partition = 11958
Partition train/validation
Lengths of train,valid
(11958, 2990)
Free memory.
Time= 08:31:14
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(11958, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(11958,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(2990, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(2990,)
Time= 08:57:46
Resume training the model
374/374 [==============================] - 483s 1s/step - loss: 1.0506 - accuracy: 0.5304 - val_loss: 0.6621 - val_accuracy: 0.6184
Time= 09:06:03
Batch 1
Batch of RNA lengths 1000 to 2000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (3758,)
Encode the protein-coding sequences
pc shape: (5856,)
shape of sequences, shape of labels
shape of train
(7691,) (7691,)
shape of test
(1923,) (1923,)
Convert numpy array to python 3D array
Partition = 6152
Partition train/validation
Lengths of train,valid
(6152, 1539)
Free memory.
Time= 09:06:32
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(6152, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(6152,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(1539, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(1539,)
Time= 09:38:23
Resume training the model
193/193 [==============================] - 585s 3s/step - loss: 0.6743 - accuracy: 0.5941 - val_loss: 0.6667 - val_accuracy: 0.6017
Time= 09:48:18
Batch 2
Batch of RNA lengths 2000 to 4000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (2173,)
Encode the protein-coding sequences
pc shape: (4894,)
shape of sequences, shape of labels
shape of train
(5653,) (5653,)
shape of test
(1414,) (1414,)
Convert numpy array to python 3D array
Partition = 4522
Partition train/validation
Lengths of train,valid
(4522, 1131)
Free memory.
Time= 09:48:52
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(4522, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(4522,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(1131, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(1131,)
Time= 10:34:39
Resume training the model
142/142 [==============================] - 1160s 8s/step - loss: 0.6286 - accuracy: 0.6822 - val_loss: 0.6095 - val_accuracy: 0.7100
Time= 10:54:18
Batch 3
Batch of RNA lengths 4000 to 8000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (332,)
Encode the protein-coding sequences
pc shape: (1860,)
shape of sequences, shape of labels
shape of train
(1753,) (1753,)
shape of test
(439,) (439,)
Convert numpy array to python 3D array
Partition = 1402
Partition train/validation
Lengths of train,valid
(1402, 351)
Free memory.
Time= 10:54:38
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(1402, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(1402,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(351, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(351,)
Time= 11:20:11
Resume training the model
44/44 [==============================] - 1063s 24s/step - loss: 0.4699 - accuracy: 0.8466 - val_loss: 0.3882 - val_accuracy: 0.8718
Time= 11:38:28
Batch 4
Batch of RNA lengths 8000 to 16000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (25,)
Encode the protein-coding sequences
pc shape: (258,)
shape of sequences, shape of labels
shape of train
(226,) (226,)
shape of test
(57,) (57,)
Convert numpy array to python 3D array
Partition = 180
Partition train/validation
Lengths of train,valid
(180, 46)
Free memory.
Time= 11:38:34
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(180, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(180,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(46, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(46,)
Time= 11:44:46
Resume training the model
6/6 [==============================] - 461s 77s/step - loss: 0.3139 - accuracy: 0.9167 - val_loss: 0.3385 - val_accuracy: 0.9130
Time= 11:53:46
Batch 5
Batch of RNA lengths 16000 to 32000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (8,)
Encode the protein-coding sequences
pc shape: (14,)
shape of sequences, shape of labels
shape of train
(17,) (17,)
shape of test
(5,) (5,)
Convert numpy array to python 3D array
Partition = 13
Partition train/validation
Lengths of train,valid
(13, 4)
Free memory.
Time= 11:53:47
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(13, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(13,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(4, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(4,)
Time= 11:54:38
Resume training the model
1/1 [==============================] - 5s 5s/step - loss: 0.7728 - accuracy: 0.6154 - val_loss: 0.5089 - val_accuracy: 0.7500
Epoch 1
Time= 11:57:59
Batch 0
Batch of RNA lengths 0 to 1000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (11415,)
Encode the protein-coding sequences
pc shape: (7270,)
shape of sequences, shape of labels
shape of train
(14948,) (14948,)
shape of test
(3737,) (3737,)
Convert numpy array to python 3D array
Partition = 11958
Partition train/validation
Lengths of train,valid
(11958, 2990)
Free memory.
Time= 11:58:23
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(11958, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(11958,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(2990, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(2990,)
Time= 12:24:50
Resume training the model
374/374 [==============================] - 453s 1s/step - loss: 0.6814 - accuracy: 0.5938 - val_loss: 0.6493 - val_accuracy: 0.6194
Time= 12:32:32
Batch 1
Batch of RNA lengths 1000 to 2000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (3758,)
Encode the protein-coding sequences
pc shape: (5856,)
shape of sequences, shape of labels
shape of train
(7691,) (7691,)
shape of test
(1923,) (1923,)
Convert numpy array to python 3D array
Partition = 6152
Partition train/validation
Lengths of train,valid
(6152, 1539)
Free memory.
Time= 12:32:58
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(6152, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(6152,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(1539, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(1539,)
Time= 13:04:13
Resume training the model
193/193 [==============================] - 570s 3s/step - loss: 0.6705 - accuracy: 0.5966 - val_loss: 0.6589 - val_accuracy: 0.6088
Time= 13:13:55
Batch 2
Batch of RNA lengths 2000 to 4000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (2173,)
Encode the protein-coding sequences
pc shape: (4894,)
shape of sequences, shape of labels
shape of train
(5653,) (5653,)
shape of test
(1414,) (1414,)
Convert numpy array to python 3D array
Partition = 4522
Partition train/validation
Lengths of train,valid
(4522, 1131)
Free memory.
Time= 13:14:29
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(4522, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(4522,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(1131, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(1131,)
Time= 13:59:27
Resume training the model
142/142 [==============================] - 1145s 8s/step - loss: 0.6277 - accuracy: 0.6760 - val_loss: 0.6003 - val_accuracy: 0.7100
Time= 14:18:51
Batch 3
Batch of RNA lengths 4000 to 8000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (332,)
Encode the protein-coding sequences
pc shape: (1860,)
shape of sequences, shape of labels
shape of train
(1753,) (1753,)
shape of test
(439,) (439,)
Convert numpy array to python 3D array
Partition = 1402
Partition train/validation
Lengths of train,valid
(1402, 351)
Free memory.
Time= 14:19:11
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(1402, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(1402,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(351, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(351,)
Time= 14:44:46
Resume training the model
44/44 [==============================] - 1071s 24s/step - loss: 0.4469 - accuracy: 0.8466 - val_loss: 0.4004 - val_accuracy: 0.8718
Time= 15:03:09
Batch 4
Batch of RNA lengths 8000 to 16000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (25,)
Encode the protein-coding sequences
pc shape: (258,)
shape of sequences, shape of labels
shape of train
(226,) (226,)
shape of test
(57,) (57,)
Convert numpy array to python 3D array
Partition = 180
Partition train/validation
Lengths of train,valid
(180, 46)
Free memory.
Time= 15:03:15
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(180, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(180,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(46, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(46,)
Time= 15:09:25
Resume training the model
6/6 [==============================] - 470s 78s/step - loss: 0.3049 - accuracy: 0.9167 - val_loss: 0.3467 - val_accuracy: 0.9130
Time= 15:18:42
Batch 5
Batch of RNA lengths 16000 to 32000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (8,)
Encode the protein-coding sequences
pc shape: (14,)
shape of sequences, shape of labels
shape of train
(17,) (17,)
shape of test
(5,) (5,)
Convert numpy array to python 3D array
Partition = 13
Partition train/validation
Lengths of train,valid
(13, 4)
Free memory.
Time= 15:18:43
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(13, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(13,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(4, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(4,)
Time= 15:19:32
Resume training the model
1/1 [==============================] - 4s 4s/step - loss: 0.8441 - accuracy: 0.6154 - val_loss: 0.5191 - val_accuracy: 0.7500
Epoch 2
Time= 15:22:57
Batch 0
Batch of RNA lengths 0 to 1000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (11415,)
Encode the protein-coding sequences
pc shape: (7270,)
shape of sequences, shape of labels
shape of train
(14948,) (14948,)
shape of test
(3737,) (3737,)
Convert numpy array to python 3D array
Partition = 11958
Partition train/validation
Lengths of train,valid
(11958, 2990)
Free memory.
Time= 15:23:24
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(11958, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(11958,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(2990, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(2990,)
Time= 15:49:27
Resume training the model
374/374 [==============================] - 451s 1s/step - loss: 0.6919 - accuracy: 0.5838 - val_loss: 0.6664 - val_accuracy: 0.6181
Time= 15:57:07
Batch 1
Batch of RNA lengths 1000 to 2000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (3758,)
Encode the protein-coding sequences
pc shape: (5856,)
shape of sequences, shape of labels
shape of train
(7691,) (7691,)
shape of test
(1923,) (1923,)
Convert numpy array to python 3D array
Partition = 6152
Partition train/validation
Lengths of train,valid
(6152, 1539)
Free memory.
Time= 15:57:33
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(6152, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(6152,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(1539, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(1539,)
Time= 16:28:56
Resume training the model
193/193 [==============================] - 565s 3s/step - loss: 0.6723 - accuracy: 0.5943 - val_loss: 0.6657 - val_accuracy: 0.6023
Time= 16:38:33
Batch 2
Batch of RNA lengths 2000 to 4000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (2173,)
Encode the protein-coding sequences
pc shape: (4894,)
shape of sequences, shape of labels
shape of train
(5653,) (5653,)
shape of test
(1414,) (1414,)
Convert numpy array to python 3D array
Partition = 4522
Partition train/validation
Lengths of train,valid
(4522, 1131)
Free memory.
Time= 16:39:07
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(4522, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(4522,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(1131, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(1131,)
Time= 17:23:50
Resume training the model
142/142 [==============================] - 1138s 8s/step - loss: 0.6273 - accuracy: 0.6822 - val_loss: 0.6014 - val_accuracy: 0.7100
Time= 17:43:08
Batch 3
Batch of RNA lengths 4000 to 8000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (332,)
Encode the protein-coding sequences
pc shape: (1860,)
shape of sequences, shape of labels
shape of train
(1753,) (1753,)
shape of test
(439,) (439,)
Convert numpy array to python 3D array
Partition = 1402
Partition train/validation
Lengths of train,valid
(1402, 351)
Free memory.
Time= 17:43:28
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(1402, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(1402,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(351, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(351,)
Time= 18:09:07
Resume training the model
44/44 [==============================] - 1068s 24s/step - loss: 0.4502 - accuracy: 0.8466 - val_loss: 0.3836 - val_accuracy: 0.8718
Time= 18:27:28
Batch 4
Batch of RNA lengths 8000 to 16000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (25,)
Encode the protein-coding sequences
pc shape: (258,)
shape of sequences, shape of labels
shape of train
(226,) (226,)
shape of test
(57,) (57,)
Convert numpy array to python 3D array
Partition = 180
Partition train/validation
Lengths of train,valid
(180, 46)
Free memory.
Time= 18:27:34
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(180, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(180,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(46, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(46,)
Time= 18:33:45
Resume training the model
6/6 [==============================] - 476s 79s/step - loss: 0.3153 - accuracy: 0.9167 - val_loss: 0.3200 - val_accuracy: 0.9130
Time= 18:43:00
Batch 5
Batch of RNA lengths 16000 to 32000
Load ncRNA.fasta
Load pcRNA.fasta
Ecode the non-coding sequences
nc shape: (8,)
Encode the protein-coding sequences
pc shape: (14,)
shape of sequences, shape of labels
shape of train
(17,) (17,)
shape of test
(5,) (5,)
Convert numpy array to python 3D array
Partition = 13
Partition train/validation
Lengths of train,valid
(13, 4)
Free memory.
Time= 18:43:01
Convert to tensors
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(13, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(13,)
<class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'>
(4, None, None)
<class 'tensorflow.python.framework.ops.EagerTensor'>
(4,)
Time= 18:43:52
Resume training the model

Resources.
[Working with RNNs](https://keras.io/guides/working_with_rnns/).
[Recurrent Neural Networks with Keras](https://www.tensorflow.org/guide/keras/rnn#rnns_with_listdict_inputs_or_nested_inputs).
Function tf.convert_to_tensor [docs](https://www.tensorflow.org/api_docs/python/tf/convert_to_tensor).
Function tf.reshape [docs](https://www.tensorflow.org/api_docs/python/tf/reshape).
Ragged Tensors [tutorial](https://www.tensorflow.org/guide/tensor#ragged_tensors)
and [docs](https://www.tensorflow.org/api_docs/python/tf/RaggedTensor#documenting_raggedtensor_shapes_2) and [module](https://www.tensorflow.org/api_docs/python/tf/ragged).
Incredible speedup for convert to tensor by sirfz on [stackoverflow](https://stackoverflow.com/questions/44353509/tensorflow-tf-constant-initializer-is-very-slow).

_____no_output_____


# Notebook from horizon-blue/beanmachine
Path: tutorials/Structural_Time_Series.ipynb

# Tutorial: Bayesian Structural Time Series Model
This tutorial demonstrates modeling and running inference on various Bayesian Structural Time Series (STS) models. We demonstrate some great features of Bean Machine such as: modeling serially correlated variables, the NUTS sampler applied to a global pool of variables, and providing custom distributions for inference._____no_output_____

import sys
import os

if 'google.colab' in sys.modules and 'beanmachine' not in sys.modules:
    !pip install beanmachine

smoke_test = ('SANDCASTLE_NEXUS' in os.environ or 'CI' in os.environ)_____no_output_____

# Problem
[Bayesian STS](https://en.wikipedia.org/wiki/Bayesian_structural_time_series) is a general class of additive models for *Time Series* series data;
$y_1, y_2, ..., y_n$ that have an associated *State Space* denoted by; $\alpha_1, \alpha_2, ..., \alpha_n$ which provides a probabilistic model for the serial correlation observed in the time series. For this tutorial, we will consider the case when $\alpha_i$ takes values in a continous state space (For a discussion of the discrete state space case, refer to this tutorial on [Hidden Markov Models](https://github.com/facebookresearch/beanmachine/blob/main/tutorials/Hidden_Markov_model.ipynb)).

Bayesian STS models typically contain, but are not limited to, some additive mixture of *Error* ($\epsilon$), *Trend* ($\mu$), and *Seasonality* ($\gamma$) components:
$$y_t = \mu_t + \gamma_t + \epsilon_t$$
Where each component is controlled either by some recursive relation of the state space, $\alpha_i$ or external covariates (i.e. Regression) such as day-of-week effects, business logic, or categories of the data.
When STS just contains just the Error, Trend, and Seasonality components, it is sometimes referred to as the ETS model. Possibly the simpliest version of STS is called the *local level* model which is where we will start our discussion after some prerequisites._____no_output_____# Prerequisites
Let's code this in Bean Machine! Import the Bean Machine library, some fundamental PyTorch classes, and optionally typing for our code._____no_output_____

import logging
from typing import List

import beanmachine.ppl as bm
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import torch
import torch.distributions as dist
from beanmachine.ppl.model import RVIdentifier

logging.getLogger("beanmachine").setLevel(50)
bm.seed(111)_____no_output_____

# Local Level Model
In this simple *local level* version of an STS, $y_t$ is the sum of our state variable $\alpha_t$ plus Gaussian Noise, $\epsilon_t$. All of the time dependency is modeled in the evolution of $\alpha_{t+1}$, which is equal to its previous time value $\alpha_t$ plus Gaussian Noise, $\eta_t$ that is independent of all $\epsilon_i$'s. This is sometimes referred to a *random walk* because at each step, noise is injected to the previous state to arrive at a new state.
To summarize, the random variables in play at this point are:
*  $\epsilon_t \stackrel{iid}{\sim} \text{Normal}(0, \sigma_\epsilon^2)$
*   $\eta_t \stackrel{iid}{\sim} \text{Normal}(0, \sigma_\eta^2)$
* $\eta_t \perp \epsilon_i, \forall i \in \{1,..., n\}$

Most STS formulations are expressed in the [state space](https://en.wikipedia.org/wiki/State-space_representation) representation, which is a coupled set of equations known as the *measurement* and *state* equations. Using our notation from before, the *local level* state space representation is:

| Name | Formula
| --- | ---
| Measurement Equation | $y_t =  \alpha_t + \epsilon_t$
| State Equation | $\alpha_{t+1} = \alpha_t + \eta_t$

This can be represented more compactly as:
* $\alpha_{t+1} | \alpha_t \sim \text{Normal}(\alpha_{t}, \sigma_\eta^2)$
* $y_t | \alpha_t \sim \text{Normal}(\alpha_t, \sigma_e^2)$

To make our STS "Bayesian", we will assign priors to $\sigma_\epsilon$, $\sigma_\eta$, and our initial state $\alpha_1$:
* $\sigma_\epsilon \sim \text{Gamma}(\epsilon_{\text{shape}}, \epsilon_{\text{rate}})$
* $\sigma_\eta \sim \text{Gamma}(\eta_{\text{shape}}, \eta_{\text{rate}})$
* $\alpha_1 \sim \text{Normal}(a_1, p_1^2)$

We can implement this model in Bean Machine by defining random variable objects with the `@bm.random_variable` decorator. These functions behave differently than ordinary Python functions.

Semantics for `@bm.random_variable` functions:
* They must return PyTorch `Distribution` objects.
* Though they return distributions, callees actually receive samples from the distribution. The machinery for obtaining samples from distributions is handled internally by Bean Machine.
* Inference runs the model through many iterations. During a particular inference iteration, a distinct random variable will correspond to exactly one sampled value: **calls to the same random variable function with the same arguments will receive the same sampled value within one inference iteration**. This makes it easy for multiple components of your model to refer to the same logical random variable.
* Consequently, to define distinct random variables that correspond to different sampled values during a particular inference iteration, an effective practice is to add a dummy "indexing" parameter to the function. Distinct random variables can be referred to with different values for this index.
* Please see the documentation for more information about this decorator.

Note also that, compared to the statistical notation above, our implementation uses 0-indexing instead of 1-indexing._____no_output_____

class LocalLevelModel:
    def __init__(
        self,
        N: int,
        epsilon_shape: float,
        epsilon_rate: float,
        eta_shape: float,
        eta_rate: float,
        a_1: float,
        p_1: float,
    ) -> None:
        self.N = N
        self.epsilon_shape = epsilon_shape
        self.epsilon_rate = epsilon_rate
        self.eta_shape = eta_shape
        self.eta_rate = eta_rate
        self.a_1 = a_1
        self.p_1 = p_1

    @bm.random_variable
    def SigmaEpsilon(self) -> RVIdentifier:
        return dist.Gamma(self.epsilon_shape, self.epsilon_rate)

    @bm.random_variable
    def SigmaEta(self) -> RVIdentifier:
        return dist.Gamma(self.eta_shape, self.eta_rate)

    @bm.random_variable
    def Alpha(self, n: int) -> RVIdentifier:
        if n == 0:
            return dist.Normal(self.a_1, self.p_1)
        else:
            return dist.Normal(self.Alpha(n - 1), self.SigmaEta())

    @bm.random_variable
    def Y(self, n: int) -> RVIdentifier:
        return dist.Normal(self.Alpha(n), self.SigmaEpsilon())_____no_output_____

## Data
`iclaims` is a dataset containing the weekly initial claims for US unemployment benefits against a few related google trend queries (unemploy, filling and job)from Jan 2010 - June 2018.
This dataset was curated [here](https://github.com/uber/orbit/blob/dev/orbit/utils/dataset.py#L5) and credit goes towards Uber's Orbit package._____no_output_____

def load_iclaims(end_date='2018-06-24', transform=True):
    """Load iclaims dataset
    Returns
    -------
        pd.DataFrame
    Notes
    -----
    iclaims is a dataset containing the weekly initial claims for US unemployment benefits against a few related google
    trend queries (unemploy, filling and job)from Jan 2010 - June 2018. This aims to mimick the dataset from the paper
    Predicting the Present with Bayesian Structural Time Series by SCOTT and VARIAN (2014).
    Number of claims are obtained from [Federal Reserve Bank of St. Louis] while google queries are obtained through
    Google Trends API.
    Note that dataset is transformed by natural log before fitting in order to be fitted as a multiplicative model.
    https://fred.stlouisfed.org/series/ICNSA
    https://trends.google.com/trends/?geo=US
    https://finance.yahoo.com/
    """
    url = 'https://raw.githubusercontent.com/uber/orbit/master/examples/data/iclaims_example.csv'
    df = pd.read_csv(url, parse_dates=['week'])
    df = df[df['week'] <= end_date]

    # standardize the regressors by mean; equivalent to subtracting mean after np.log
    regressors = ['trend.unemploy', 'trend.filling', 'trend.job', 'sp500', 'vix']

    # convert to float
    for col in regressors:
        df[col] = df[col].astype(float)

    if transform:
        # log transfer
        df[['claims'] + regressors] = df[['claims'] + regressors].apply(np.log)
        # de-mean
        df[regressors] = df[regressors] - df[regressors].apply(np.mean)

    return df_____no_output_____y_obs = torch.tensor(load_iclaims().claims.values)[:40]
plt.plot(y_obs)
plt.show()_____no_output_____

## Inference
Inference is the process of combining _model_ with _data_ to obtain _insights_, in the form of probability distributions over values of interest. Bean Machine offers a powerful and general inference framework to enable fitting a arbitrary models to data. First lets setup our priors and instantiate our model._____no_output_____

# Prior Specification
epsilon_shape = 0.5
epsilon_rate = 1.0
eta_shape = 0.5
eta_rate = 1.0
a_1 = y_obs.mean().item()
p_1 = 1.0

# Length of our time series
N = len(y_obs)_____no_output_____# Instantiate the model and pass our length and priors
model = LocalLevelModel(
    N=N,
    epsilon_shape=epsilon_shape,
    epsilon_rate=epsilon_rate,
    eta_shape=eta_shape,
    eta_rate=eta_rate,
    a_1=a_1,
    p_1=p_1
)_____no_output_____

Running inference consists of a few arguments:

| Name | Usage
| --- | ---
| `queries` | A list of `@bm.random_variable` targets to fit posterior distributions for.
| `observations` | The `Dict` of observations we built up, above.
| `num_samples` | Number of samples to build up distributions for the values listed in `queries`.
| `num_chains` | Number of separate inference runs to use. Multiple chains can verify inference ran correctly.

The next step is to define the queries and observations. For this particular run, we're interested in infering $\alpha$, $\sigma_\epsilon$, and $\sigma_\eta$._____no_output_____

queries = [model.Alpha(n) for n in range(model.N)] + [model.SigmaEpsilon(), model.SigmaEta()]
observations = {**{model.Y(n): y_obs[n] for n in range(model.N)}}_____no_output_____

For this particular problem, we will use the GlobalNoUTurnSampler inference method. Bean Machine's GlobalNoUTurnSampler is the NUTS sampler you have probably used before with stan or pymc3. We have chosen to use the NUTS sampler here because it can be easily compared to other probabilistic tools._____no_output_____

num_samples = 400 if not smoke_test else 1
num_adaptive_samples = 50
all_samples = bm.GlobalNoUTurnSampler(max_tree_depth=10).infer(
    queries,
    observations,
    num_samples,
    num_chains=1,
    num_adaptive_samples=num_adaptive_samples
)

samples = all_samples.get_chain(0) alpha_samples = torch.stack([samples[model.Alpha(n)] for n in range(model.N)], dim=1)_____no_output_____

## Visualization
We will look at the values of the samples collected for $\alpha$ and $\sigma_\epsilon$. We will take the mean of samples taken over the last 10% of the chain, and compare these to our data._____no_output_____

tail_len = num_samples // 10_____no_output_____ppc = np.percentile(
  alpha_samples[-tail_len:].numpy(),
  q=[5, 50, 95],
  axis=0
).T

plt.figure(figsize=(12,6))
plt.plot(y_obs, label='Observations', color='green')
plt.plot(ppc[:, 1], label=r'Median of $\alpha_t$', color='red')
plt.plot(ppc[:, 0], 'b--', label=r'$5th$ Percentile')
plt.plot(ppc[:, 2], 'b--', label=r'$95th$ Percentile')
plt.title(r"Observations versus $\alpha_t$")
plt.legend()
plt.show()_____no_output_____

It appears inference has recovered the latent states $\alpha$ very well, and is accurately representing $Y$ when sampled.

Now we investigate chain mixing by looking at the last half of the sampling chain. This type of plot is called a *trace plot* and is useful to visualize where the sampler is proposing steps._____no_output_____

plt.figure(figsize=(12,6))
[plt.plot(alpha_samples[-num_samples//2:, i], label=f'Time index {i}') for i in [0, 10, 20, 30, 39]]
plt.legend()
plt.title(r"Trace Plots for $\alpha_t$")
plt.show()_____no_output_____

We see a clear downward trend for successive time indices. This matches our intuition from the "Observations versus $\alpha_t$" plot as we see the original time series with a downward trend.

Finally we look at the uncertainty estimates as a function of time. These are the posterior samples for both $\sigma_\eta$ and $\sigma_\epsilon$._____no_output_____

plt.figure(figsize=(12,6))
plt.plot(samples[model.SigmaEta()], label=r"State Uncertainty, $\sigma_\eta$")
plt.plot(samples[model.SigmaEpsilon()], label=r"Measurement Uncertainty, $\sigma_\epsilon$")
plt.title("Uncertainty")
plt.legend()
plt.show()_____no_output_____

We see the final noise samples are quite small, reflecting the fact that the original data did not contain alot of noise._____no_output_____## Posterior Likelihood Checks
One way to evaluate posterior samples is computing likelihoods of our posterior samples, and comparing these to the likelihood of the underlying data. Formally, we can compute the joint likelihood of posterior samples with the observations._____no_output_____

def log_likelihood(alpha_samples, sigma_epsilon, y_obs, N):
    ll = 0
    for n in range(N):
        ll += dist.Normal(alpha_samples[n], sigma_epsilon).log_prob(y_obs[n])
    return ll

# Iterate over the batches of samples and compute log likelihood
lls = [
    log_likelihood(alpha_samples, sigma_epsilon_sample, y_obs, N)
    for alpha_samples, sigma_epsilon_sample in zip(alpha_samples, samples[model.SigmaEpsilon()])
]_____no_output_____plt.figure(figsize=(12,6))
plt.plot(lls, label="Sample", c="g")
plt.title("Posterior Likelihood Check")
plt.ylabel("Log likelihood")
plt.xlabel("Iteration")

plt.legend()
plt.show()_____no_output_____

From the above plot, inference appears to be fitting almost perfectly. The random variables given the observed data is exceptionally high._____no_output_____# Vectorized Model
Now lets try a vectorized model by using a custom `Distribution`. We first extend pytorch's `torch.distributions.Distribution` into a vectorized version of a random walk.
The key idea behind this implementation is that a [Gaussian Random Walk](https://en.wikipedia.org/wiki/Random_walk#Gaussian_random_walk) can also be expressed as a chain of iid Gaussian random variables
that have a `cumsum` operator applied to them (thus making the "walking" behavior). Now with this `cumsum` operator, we will be able to sample entire chains in one step!
_____no_output_____

from torch.distributions import Distribution, constraints

class GaussianRandomWalk(Distribution):
    support = constraints.real

    def __init__(
        self,
        loc: torch.Tensor,
        scale: torch.Tensor,
        num_steps: int = 1,
        validate_args=None,
    ) -> None:
        assert isinstance(num_steps, int) and num_steps > 0, "`num_steps` argument should be an positive integer."
        self.loc = loc
        self.scale = scale
        self.num_steps = num_steps
        batch_shape, event_shape = scale.size(), torch.Size([num_steps])
        super(GaussianRandomWalk, self).__init__(batch_shape, event_shape, validate_args=validate_args)

    def sample(self, sample_shape: List = None):
        if not sample_shape:
            sample_shape = []
        assert isinstance(sample_shape, list), "`sample_shape` argument should be a list."
        shape = torch.Size(sample_shape) + self.batch_shape + self.event_shape
        loc = torch.zeros(shape)
        loc[..., 0] = self.loc
        walks = dist.Normal(loc=loc, scale=1).sample()
        return torch.cumsum(walks, axis=-1) * self.scale.unsqueeze(dim=-1)

    def log_prob(self, value):
        init_prob = dist.Normal(self.loc, self.scale).log_prob(value[..., 0])
        scale = self.scale.unsqueeze(dim=-1)
        step_probs = dist.Normal(value[..., :-1], scale).log_prob(value[..., 1:])
        return init_prob + torch.sum(step_probs, axis=-1)

    @property
    def mean(self):
        return torch.zeros(self.batch_shape + self.event_shape) + self.loc

    @property
    def variance(self):
        return torch.broadcast_to(
            self.scale.unsqueeze(dim=-1) ** 2 * torch.arange(1, self.num_steps + 1),
            self.batch_shape + self.event_shape,
        )_____no_output_____

In our implementation above, we will now use the `sample()` and `log_prob()` functions to seamlessly integrate into Bean Machine!
Specifically, we replace the `Alpha()` function with our vectorized distribution rather than recursively calling the univariate function from before._____no_output_____

class VectorizedLocalLevelModel:
    def __init__(
        self, N: int, epsilon_shape: float, epsilon_rate: float, eta_shape: float, eta_rate: float, a_1: float
    ) -> None:
        self.N = N
        self.epsilon_shape = epsilon_shape
        self.epsilon_rate = epsilon_rate
        self.eta_shape = eta_shape
        self.eta_rate = eta_rate
        self.a_1 = a_1

    @bm.random_variable
    def SigmaEpsilon(self) -> RVIdentifier:
        return dist.Gamma(self.epsilon_shape, self.epsilon_rate)

    @bm.random_variable
    def SigmaEta(self) -> RVIdentifier:
        return dist.Gamma(self.eta_shape, self.eta_rate)

    @bm.random_variable
    def Alpha(self) -> RVIdentifier:
        return GaussianRandomWalk(loc=self.a_1, scale=self.SigmaEta(), num_steps=self.N)  # Vectorized!

    @bm.random_variable
    def Y(self) -> RVIdentifier:
        return dist.Normal(self.Alpha(), self.SigmaEpsilon())  # Vectorized!_____no_output_____

We keep the rest of the workflow to be very similar to the unvectorized model so that we can directly compare the results. We complete inference, visualize our samples, and then compute some diagnostics of the fitted values._____no_output_____

vec_model = VectorizedLocalLevelModel(
    N=N,
    epsilon_shape=epsilon_shape,
    epsilon_rate=epsilon_rate,
    eta_shape=eta_shape,
    eta_rate=eta_rate,
    a_1=a_1
)

queries = [vec_model.Alpha(), vec_model.SigmaEpsilon(), vec_model.SigmaEta()]
observations = {**{vec_model.Y(): y_obs}}_____no_output_____

One difference is that we now fit 4 chains (serially) in order to better investigate some diagnostic metrics. You can read more about this in the $\hat{R}$ and $ess$ cells located below._____no_output_____

num_samples = 400 if not smoke_test else 1
vec_all_samples = bm.GlobalNoUTurnSampler(max_tree_depth=10).infer(
    queries,
    observations,
    num_samples,
    num_chains=4,
    num_adaptive_samples=num_adaptive_samples,
    verbose=False
)Samples collected: 100%|██████████| 450/450 [00:13<00:00, 34.58it/s]
Samples collected: 100%|██████████| 450/450 [00:16<00:00, 27.65it/s]

Samples collected: 100%|██████████| 450/450 [00:08<00:00, 52.69it/s]

Samples collected: 100%|██████████| 450/450 [00:08<00:00, 52.58it/s]

The speedup is quite significant using a vectorized approach! One of the main benefits of using Bean Machine is being able to incorporate this type of fast code._____no_output_____

vec_samples = vec_all_samples.get_chain(0)
vec_alpha_samples = vec_samples[vec_model.Alpha()]
vec_alpha_samples.size()_____no_output_____

## Visualization_____no_output_____

tail_len = num_samples // 10

vec_ppc = np.percentile(
  vec_alpha_samples[-tail_len:].numpy(),
  q=[5, 50, 95],
  axis=0
).T

plt.figure(figsize=(12,6))
plt.plot(y_obs, label='Observations', color='green')
plt.plot(vec_ppc[:, 1], label=r'$\alpha_t$', color='red')
plt.plot(vec_ppc[:, 0], 'b--', label=r'$5th$ Percentile')
plt.plot(vec_ppc[:, 2], 'b--', label=r'$95th$ Percentile')
plt.title(r"Observations versus $\alpha_t$")
plt.legend()
plt.show()_____no_output_____plt.figure(figsize=(12,6))
[plt.plot(vec_alpha_samples[-num_samples//2:, i], label=f'Time index {i}') for i in [0, 10, 20, 30, 38]]
plt.legend()
plt.title(r"Trace Plots for $\alpha_t$")
plt.show()_____no_output_____plt.plot(vec_samples[vec_model.SigmaEta()], label=r"State Uncertainty, $\sigma_\eta$")
plt.plot(vec_samples[vec_model.SigmaEpsilon()], label=r"Measurement Uncertainty, $\sigma_\epsilon$")
plt.legend()
plt.show()_____no_output_____

## Diagnostics
Two important statistics to
take note of are the $\hat{R}$ (`r_hat`) and effective sample size (`ess`) values in the
below dataframe, see [Vehtari _et al_](#references)._____no_output_____

import arviz as az
summary_df = az.summary(vec_all_samples.to_xarray(), round_to=3)
summary_df.T_____no_output_____

### $\hat{R}$ Diagnostic

$\hat{R}$ is a diagnostic tool that measures the between- and within-chain variances. It
is a test that indicates a lack of convergence by comparing the variance between
multiple chains to the variance within each chain. If the parameters are successfully
exploring the full space for each chain, then $\hat{R}\approx 1$, since the
between-chain and within-chain variance should be equal. $\hat{R}$ is calculated as

$$
\hat{R}=\frac{\hat{V}}{W}
$$

where $W$ is the within-chain variance and $\hat{V}$ is the posterior variance estimate
for the pooled rank-traces. The take-away here is that $\hat{R}$ converges towards 1
when each of the Markov chains approaches perfect adaptation to the true posterior
distribution. We do not recommend using inference results if $\hat{R}>1.01$. More
information about $\hat{R}$ can be found in the [Vehtari _et al_](#references) paper._____no_output_____### Effective sample size $ess$ diagnostic

MCMC samplers do not draw independent samples from the target distribution, which means
that our samples are correlated. In an ideal situation all samples would be independent,
but we do not have that luxury. We can, however, measure the number of _effectively
independent_ samples we draw, which is called the effective sample size. You can read
more about how this value is calculated in the [Vehtari _et al_](#references) paper,
briefly it is a measure that combines information from the $\hat{R}$ value with the
autocorrelation estimates within the chains. There are many ways to estimate effective
samples sizes, however, we will be using the method defined in the [Vehtari _et
al_](#references) paper.

The rule of thumb for `ess_bulk` is for this value to be greater than 100 per chain on
average. Since we ran four chains, we need `ess_bulk` to be greater than 400 for each
parameter. The `ess_tail` is an estimate for effectively independent samples considering
the more extreme values of the posterior. This is not the number of samples that landed
in the tails of the posterior. It is a measure of the number of effectively independent
samples if we sampled the tails of the posterior. The rule of thumb for this value is
also to be greater than 100 per chain on average.
_____no_output_____# Heteroscedastic Model
In the previous sections, one core assumption made about the data generation process is that $\epsilon_t \stackrel{iid}{\sim} \text{Normal}(0, \sigma_\epsilon^2)$,
meaning that all of the noise in our measurement equation was sampled the same way, from the same distribution.
In the Statistics literature, this is known as *homoscedastic* noise. Instead, we could postulate a second latent random variable, $\beta_t$ that would capture
some of the serial correlation in the *noise* over time (as opposed to the level). This is referred to as a *heteroscedastic* noise. That is, we could say $\beta = g(\beta_{t - 1})$.
One simple choice for $g(\cdot)$ is the same choice we made for $\alpha_t$, that is:
$$\beta_{t + 1} = \beta_{t} + \epsilon_{t}$$
Where the definition of $\epsilon_t$ doesn't change (or its associated prior, $\sigma_\epsilon$). Now, our model looks like this:
* $\alpha_{t+1} | \alpha_t \sim \text{Normal}(\alpha_{t}, \sigma_\eta^2)$
* $\beta_{t+1} | \beta_t \sim \text{Normal}(\beta_{t}, \sigma_\epsilon^2)$
* $y_t | \alpha_t \sim \text{Normal}(\alpha_t, f(\beta_t^2))$

So that as the process goes on, the noise can now "walk" just like our level from before. The last detail to attend to
is the function $f(\cdot)$. Since the `Normal` distribution's `scale` argument lives in $\mathbb R^+$, we need to apply a transformation
to $\beta_t$ for the `scale` argument to be valid. One nice candidate is [torch.nn.Softplus()](https://pytorch.org/docs/stable/generated/torch.nn.Softplus.html)
which is a smooth approximation to the `Relu` function. We will use the `@bm.functional` wrapper to accomplish this deterministic mapping,
creating an intermediate random variable called `Scale` in the process._____no_output_____

class HeteroLocalLevelModel:
    def __init__(
        self,
        N: int,
        epsilon_shape: float,
        epsilon_rate: float,
        eta_shape: float,
        eta_rate: float,
        a_1: float,
    ) -> None:
        self.N = N
        self.epsilon_shape = epsilon_shape
        self.epsilon_rate = epsilon_rate
        self.eta_shape = eta_shape
        self.eta_rate = eta_rate
        self.a_1 = a_1
        self.p_1 = p_1

    @bm.random_variable
    def SigmaEpsilon(self) -> RVIdentifier:
        return dist.Gamma(self.epsilon_shape, self.epsilon_rate)

    @bm.random_variable
    def SigmaEta(self) -> RVIdentifier:
        return dist.Gamma(self.eta_shape, self.eta_rate)

    @bm.random_variable
    def Alpha(self) -> RVIdentifier:
        return GaussianRandomWalk(loc=self.a_1, scale=self.SigmaEta(), num_steps=self.N)  # Vectorized!

    @bm.random_variable
    def Beta(self) -> RVIdentifier:
        return GaussianRandomWalk(loc=torch.Tensor([0.0]), scale=self.SigmaEpsilon(), num_steps=self.N)  # Vectorized!

    @bm.functional
    def Scale(self) -> RVIdentifier:
        return torch.nn.Softplus()(self.Beta())  # Transform Beta to a valid scale argument

    @bm.random_variable
    def Y(self) -> RVIdentifier:
        return dist.Normal(self.Alpha(), self.Scale())  # Vectorized!_____no_output_____hetero_model = HeteroLocalLevelModel(
    N=N,
    epsilon_shape=epsilon_shape,
    epsilon_rate=epsilon_rate,
    eta_shape=eta_shape,
    eta_rate=eta_rate,
    a_1=a_1
)

queries = [hetero_model.Alpha(), hetero_model.Scale()]
observations = {**{hetero_model.Y(): y_obs}}
num_samples = 400 if not smoke_test else 1
hetero_all_samples = bm.GlobalNoUTurnSampler().infer(
    queries, observations, num_samples, num_chains=1, num_adaptive_samples=num_adaptive_samples * 3
) hetero_samples = hetero_all_samples.get_chain(0)
hetero_alpha_samples = hetero_samples[hetero_model.Alpha()]
hetero_scale_samples = hetero_samples[hetero_model.Scale()]_____no_output_____

## Visualization
One last note before visualization, we are now computing uncertainty of both our level ($\alpha_t$),
our noise ($\beta_t$), and our observations $y_t$. These "types" of uncertainty can be broadly cast into two
seperate categories: **aleatoric** uncertainty and **epistimic** uncertainty.

A quick read on the topic can be found
[here](https://en.wikipedia.org/wiki/Uncertainty_quantification) but the gist is that **aleatoric** is the
data uncertainty, and measures the inherent randomness in the observed data. This type of uncertainty would be appropriate
for forming *prediction* intervals for new data. **Epistimic** uncertainty on the other hand, measures the uncertainty in a *parameter* estimate.
So here, we are estimating the conditional mean ($\alpha_t$) and conditional variance ($f(\beta_t)^2$). This type of
uncertainty is useful for forming *credible* intervals. We first explore **aleatoric** uncertainty and then see how it is impacted by also considering **epistemic** uncertainty._____no_output_____

# Point estimates of Alpha and Beta
loc = hetero_alpha_samples[-tail_len:].mean(0)
scale = hetero_scale_samples[-tail_len:].mean(0)
aleatoric = dist.Normal(loc, scale)
quantiles = aleatoric.icdf(torch.Tensor([0.05, 0.95]).unsqueeze(-1))

# Form prediction interval with the point estimates
plt.figure(figsize=(12, 6))
plt.plot(y_obs, label="Observations", color="green")
plt.plot(loc, "r--", label=r"Mean of $\alpha_t$")
plt.fill_between(torch.arange(N), quantiles[0, ...], quantiles[1, ...], alpha=0.5, label="Prediction Interval")
plt.title(r"Aleatoric Uncertainty: Observations versus $\alpha_t$")
plt.legend()
plt.show()_____no_output_____

As seen from the plot, the mean of $\alpha_t$ appears much smoother than before. The reason is because by fitting both the *location* and *scale* of $\text{Normal}(\alpha_t, f(\beta_t))$, we enable a more flexible fit to the data (contrast this from before when `SigmaEpsilon()` was a single parameter that was supposed to estimate the noise for the entire sequence!). For example, when the data becomes more noisy, our model can simply widen $\beta_t$ without overcorrecting $\alpha_t$ which ensures smoother connections with later points. Thus, the model is able to more accurately detect a *latent* trend while still learning an evolving (*heteroskedastic*) noise level. This is an example of learning the **aleatoric** uncertainty (inherent randomness), which we display with a *prediction* interval. _____no_output_____Now we show the other type of uncertainty, **epistemic** uncertainty._____no_output_____

epistemic_alpha = np.quantile(hetero_alpha_samples[-tail_len:].detach().numpy(), [0.05, 0.95], axis=0)

# CI for Beta
plt.figure(figsize=(16, 10))
plt.fill_between(
    torch.arange(N),
    np.quantile((hetero_alpha_samples[-tail_len:] - hetero_scale_samples[-tail_len:]).detach().numpy(), 0.05, 0),
    np.quantile((hetero_alpha_samples[-tail_len:] + hetero_scale_samples[-tail_len:]).detach().numpy(), 0.95, 0),
    alpha=0.5,
    label=r"95% Credible Interval for $\beta_t \pm \alpha_t$"
)
# Beta samples
plt.plot(hetero_alpha_samples[-tail_len:].T - hetero_scale_samples[-tail_len:].T, 'r--', linewidth=0.1)
plt.plot(hetero_alpha_samples[-tail_len:].T + hetero_scale_samples[-tail_len:].T, 'r--', linewidth=0.1)
plt.plot(hetero_alpha_samples[-tail_len:].T[0] + hetero_scale_samples[-tail_len:].T[0], 'r--', label=r'Samples of $\beta_t \pm \alpha_t$', linewidth=0.1)
# CI for Alpha
plt.fill_between(
    torch.arange(N),
    epistemic_alpha[0, ...],
    epistemic_alpha[1, ...],
    alpha=0.5,
    color='red',
    label=r"95% Credible Interval for $\alpha_t$"
)
# Samples for Alpha
plt.plot(hetero_alpha_samples[-tail_len:].T, 'b--', linewidth=0.3)
plt.plot(hetero_alpha_samples[-tail_len:].T[0, ...], 'b--', label=r'Samples of $\alpha_t$', linewidth=0.3)
plt.plot(y_obs, label="Observations", color="green", linewidth=3)
plt.title(r"Epistemic Uncertainty: Observations versus $\alpha_t$")
plt.legend()
plt.show()_____no_output_____

Before we showed how there was uncertainty in where the data will be observed next (**aleatoric**) which we encoded with a distribution $\text{Normal}(\alpha_t, \beta_t)$ made up of point estimates of parameters `Alpha` and `Beta`. But remember, these parameters are themselves estimated from the data and have additional uncertainty from the parameter estimation process! This is the **epistemic** uncertainty. Rather than show the point estimates, above we show a 95% credible interval for both $\alpha_t$ and $\beta_t$ at each time index. Note that $\beta_t$'s uncertainty is superimposed (added and subtracted) on top of $\alpha_t$ since it is measured in different units. It is now clear that by integrating in the *epistemic* uncertainty into our parameter estimates for the *aleatoric* uncertainty, the corresponding prediction intervals would be a great deal wider._____no_output_____In either case, by using Bean Machine we can handle epistemic *and* aleatoric uncertainty quite easily with only about 1-2 lines of extra code! _____no_output_____


# Notebook from MD2Korg/CerebralCortex-Platform
Path: jupyter_demo/cc_algorithms.ipynb

# Cerebral Cortex Data Analysis Algorithms
Cerebral Cortex contains a library of algorithms that are useful for processing data and converting it into features or biomarkers.  This page demonstrates a simple GPS clustering algorithm.  For more details about the algorithms that are available, please see our [documentation](https://cerebralcortex-kernel.readthedocs.io/en/latest/).  These algorithms are constantly being developed and improved through our own work and the work of other researchers._____no_output_____## Initalize the system_____no_output_____

%reload_ext autoreload
from util.dependencies import *

CC = Kernel("/home/jovyan/cc_conf/", study_name="default")_____no_output_____

## Generate some sample location data

This example utilizes a data generator to protect the privacy of real participants and allows for anyone utilizing this system to explore the data without required institutional review board approvals. This is disabled for this demonstration to not create too much data at once._____no_output_____

gps_stream = gen_location_datastream(user_id=USER_ID, stream_name="GPS--org.md2k.phonesensor--PHONE")_____no_output_____gps_stream.printSchema()
gps_stream.show(3)root
 |-- timestamp: timestamp (nullable = true)
 |-- localtime: timestamp (nullable = true)
 |-- user: string (nullable = true)
 |-- version: long (nullable = true)
 |-- latitude: double (nullable = true)
 |-- longitude: double (nullable = true)
 |-- altitude: long (nullable = true)
 |-- speed: double (nullable = true)
 |-- bearing: double (nullable = true)
 |-- accuracy: double (nullable = true)

+-------------------+-------------------+--------------------+-------+-----------------+------------------+--------+--------+----------+---------+
|          timestamp|          localtime|                user|version|         latitude|         longitude|altitude|   speed|   bearing| accuracy|
+-------------------+-------------------+--------------------+-------+-----------------+------------------+--------+--------+----------+---------+
|2019-09-01 11:35:59|2019-09-01 16:35:59|00000000-afb8-476...|      1|35.11071115468912|   -90.02262837146|      93|2.370432|159.166668| 24.43889|
|2019-09-01 11:36:59|2019-09-01 16:36:59|00000000-afb8-476...|      1|35.11179661674077|-90.02236573611306|      93|4.808776|148.486539|25.182138|
|2019-09-01 11:37:59|2019-09-01 16:37:59|00000000-afb8-476...|      1|35.11147882254893|-90.02160130729754|      95|1.663435|  4.600978|14.806869|
+-------------------+-------------------+--------------------+-------+-----------------+------------------+--------+--------+----------+---------+
only showing top 3 rows

## Cluster the location data
Cerebral Cortex makes it easy to apply built-in algorithms to data streams.  In this case, `gps_clusters` is imported from the algorithm library, then `compute` is utilized to run this algorithm on the `gps_stream` to generate a set of centroids. This is the general format for applying algorithm to datastream and makes it easy for researchers to apply validated and tested algorithms to his/her own data without the need to become an expert in the particular set of transformations needed.

_Note:_ the `compute` method engages the parallel computation capabilities of Cerebral Cortex, which causes all the data to be read from the data storage layer and processed on every computational core available to the system.  This allows the computation to run as quickly as possible and to take advantage of powerful clusters from a relatively simple interface.  This capability is critical to working with mobile sensor big data where data sizes can exceed 100s of gigabytes per datastream for larger studies._____no_output_____

from cerebralcortex.algorithms.gps.clustering import cluster_gps
windowed_gps = gps_stream.window()
clusters = cluster_gps(windowed_gps)
clusters.printSchema()
clusters.show(truncate=False)2020-11-10 21:19:16,250 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e75b0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,250 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e75b0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,250 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e75b0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,250 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e75b0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,250 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e75b0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,258 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e7e20>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,258 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e7e20>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,258 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e7e20>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,258 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e7e20>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,258 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e7e20>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,264 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e7280>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,264 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e7280>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,264 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e7280>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,264 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e7280>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,264 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e7280>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo

## Visualize GPS Data_____no_output_____### GPS Stream Plot
GPS visualization requires dedicated plotting capabilities. Cerebral Cortex includes a library to allow for interactive exploration.  In this plot, use your mouse to drag the map around along with zooming in to explore the specific data points._____no_output_____

from cerebralcortex.plotting.gps.plots import plot_gps_clusters
plot_gps_clusters(clusters, user_id=USER_ID, zoom=8)/opt/conda/lib/python3.8/site-packages/pkg_resources/__init__.py:1149: DeprecationWarning:

Use of .. or absolute path in a resource path is not allowed and will raise exceptions in a future release.

/opt/conda/lib/python3.8/site-packages/pkg_resources/__init__.py:1149: DeprecationWarning:

Use of .. or absolute path in a resource path is not allowed and will raise exceptions in a future release.