# Notebook from msenel1978/hmm-tagger
Path: HMM Tagger.ipynb

# Project: Part of Speech Tagging with Hidden Markov Models
---
### Introduction

Part of speech tagging is the process of determining the syntactic category of a word from the words in its surrounding context. It is often used to help disambiguate natural language phrases because it can be done quickly with high accuracy. Tagging can be used for many NLP tasks like determining correct pronunciation during speech synthesis (for example, _dis_-count as a noun vs dis-_count_ as a verb), for information retrieval, and for word sense disambiguation.

In this notebook, you'll use the [Pomegranate](http://pomegranate.readthedocs.io/) library to build a hidden Markov model for part of speech tagging using a "universal" tagset. Hidden Markov models have been able to achieve [>96% tag accuracy with larger tagsets on realistic text corpora](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf). Hidden Markov models have also been used for speech recognition and speech generation, machine translation, gene recognition for bioinformatics, and human gesture recognition for computer vision, and more.

![](_post-hmm.png)

The notebook already contains some code to get you started. You only need to add some new functionality in the areas indicated to complete the project; you will not need to modify the included code beyond what is requested. Sections that begin with **'IMPLEMENTATION'** in the header indicate that you must provide code in the block that follows. Instructions will be provided for each section, and the specifics of the implementation are marked in the code block with a 'TODO' statement. Please be sure to read the instructions carefully!_____no_output_____

**Note:** Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You must then **export the notebook** by running the last cell in the notebook, or by using the menu above and navigating to **File -> Download as -> HTML (.html)** Your submissions should include both the `html` and `ipynb` files.

_____no_output_____

**Note:** Code and Markdown cells can be executed using the `Shift + Enter` keyboard shortcut. Markdown cells can be edited by double-clicking the cell to enter edit mode.

_____no_output_____### The Road Ahead
You must complete Steps 1-3 below to pass the project. The section on Step 4 includes references & resources you can use to further explore HMM taggers.

- [Step 1](#Step-1:-Read-and-preprocess-the-dataset): Review the provided interface to load and access the text corpus
- [Step 2](#Step-2:-Build-a-Most-Frequent-Class-tagger): Build a Most Frequent Class tagger to use as a baseline
- [Step 3](#Step-3:-Build-an-HMM-tagger): Build an HMM Part of Speech tagger and compare to the MFC baseline
- [Step 4](#Step-4:-[Optional]-Improving-model-performance): (Optional) Improve the HMM tagger_____no_output_____

**Note:** Make sure you have selected a **Python 3** kernel in Workspaces or the hmm-tagger conda environment if you are running the Jupyter server on your own machine.

_____no_output_____

# Jupyter "magic methods" -- only need to be run once per kernel restart
%load_ext autoreload
%aimport helpers, tests
%autoreload 1The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
# import python modules -- this cell needs to be run again if you make changes to any of the files
import matplotlib.pyplot as plt
import numpy as np

from IPython.core.display import HTML
from itertools import chain
from collections import Counter, defaultdict
from helpers import show_model, Dataset
from pomegranate import State, HiddenMarkovModel, DiscreteDistribution
import sys
print (sys.version)3.5.4 |Anaconda, Inc.| (default, Feb 19 2018, 10:59:04)
[GCC 7.2.0]

## Step 1: Read and preprocess the dataset
---
We'll start by reading in a text corpus and splitting it into a training and testing dataset. The data set is a copy of the [Brown corpus](https://en.wikipedia.org/wiki/Brown_Corpus) (originally from the [NLTK](https://www.nltk.org/) library) that has already been pre-processed to only include the [universal tagset](https://arxiv.org/pdf/1104.2086.pdf). You should expect to get slightly higher accuracy using this simplified tagset than the same model would achieve on a larger tagset like the full [Penn treebank tagset](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html), but the process you'll follow would be the same.

The `Dataset` class provided in helpers.py will read and parse the corpus. You can generate your own datasets compatible with the reader by writing them to the following format. The dataset is stored in plaintext as a collection of words and corresponding tags. Each sentence starts with a unique identifier on the first line, followed by one tab-separated word/tag pair on each following line. Sentences are separated by a single blank line.

Example from the Brown corpus.
```
b100-38532
Perhaps	ADV
it	PRON
was	VERB
right	ADJ
;	.
;	.

b100-35577
...
```_____no_output_____

data = Dataset("tags-universal.txt", "brown-universal.txt", train_test_split=0.8)

print("There are {} sentences in the corpus.".format(len(data)))
print("There are {} sentences in the training set.".format(len(data.training_set)))
print("There are {} sentences in the testing set.".format(len(data.testing_set)))

assert len(data) == len(data.training_set) + len(data.testing_set), \
       "The number of sentences in the training set + testing set should sum to the number of sentences in the corpus"There are 57340 sentences in the corpus.
There are 45872 sentences in the training set.
There are 11468 sentences in the testing set.

### The Dataset Interface

You can access (mostly) immutable references to the dataset through a simple interface provided through the `Dataset` class, which represents an iterable collection of sentences along with easy access to partitions of the data for training & testing. Review the reference below, then run and review the next few cells to make sure you understand the interface before moving on to the next step.

```
Dataset-only Attributes:
    training_set - reference to a Subset object containing the samples for training
    testing_set - reference to a Subset object containing the samples for testing

Dataset & Subset Attributes:
    sentences - a dictionary with an entry {sentence_key: Sentence()} for each sentence in the corpus
    keys - an immutable ordered (not sorted) collection of the sentence_keys for the corpus
    vocab - an immutable collection of the unique words in the corpus
    tagset - an immutable collection of the unique tags in the corpus
    X - returns an array of words grouped by sentences ((w11, w12, w13, ...), (w21, w22, w23, ...), ...)
    Y - returns an array of tags grouped by sentences ((t11, t12, t13, ...), (t21, t22, t23, ...), ...)
    N - returns the number of distinct samples (individual words or tags) in the dataset

Methods:
    stream() - returns an flat iterable over all (word, tag) pairs across all sentences in the corpus
    __iter__() - returns an iterable over the data as (sentence_key, Sentence()) pairs
    __len__() - returns the nubmer of sentences in the dataset
```

For example, consider a Subset, `subset`, of the sentences `{"s0": Sentence(("See", "Spot", "run"), ("VERB", "NOUN", "VERB")), "s1": Sentence(("Spot", "ran"), ("NOUN", "VERB"))}`. The subset will have these attributes:

```
subset.keys == {"s1", "s0"}  # unordered
subset.vocab == {"See", "run", "ran", "Spot"}  # unordered
subset.tagset == {"VERB", "NOUN"}  # unordered
subset.X == (("Spot", "ran"), ("See", "Spot", "run"))  # order matches .keys
subset.Y == (("NOUN", "VERB"), ("VERB", "NOUN", "VERB"))  # order matches .keys
subset.N == 7  # there are a total of seven observations over all sentences
len(subset) == 2  # because there are two sentences
```

**Note:** The `Dataset` class is _convenient_, but it is **not** efficient. It is not suitable for huge datasets because it stores multiple redundant copies of the same data.

_____no_output_____#### Sentences

`Dataset.sentences` is a dictionary of all sentences in the training corpus, each keyed to a unique sentence identifier. Each `Sentence` is itself an object with two attributes: a tuple of the words in the sentence named `words` and a tuple of the tag corresponding to each word named `tags`._____no_output_____

key = 'b100-38532'
print("Sentence: {}".format(key))
print("words:\n\t{!s}".format(data.sentences[key].words))
print("tags:\n\t{!s}".format(data.sentences[key].tags))Sentence: b100-38532
words:
	('Perhaps', 'it', 'was', 'right', ';', ';')
tags:
	('ADV', 'PRON', 'VERB', 'ADJ', '.', '.')

**Note:** The underlying iterable sequence is **unordered** over the sentences in the corpus; it is not guaranteed to return the sentences in a consistent order between calls. Use `Dataset.stream()`, `Dataset.keys`, `Dataset.X`, or `Dataset.Y` attributes if you need ordered access to the data.

#### Counting Unique Elements

You can access the list of unique words (the dataset vocabulary) via `Dataset.vocab` and the unique list of tags via `Dataset.tagset`._____no_output_____

print("There are a total of {} samples of {} unique words in the corpus."
      .format(data.N, len(data.vocab)))
print("There are {} samples of {} unique words in the training set."
      .format(data.training_set.N, len(data.training_set.vocab)))
print("There are {} samples of {} unique words in the testing set."
      .format(data.testing_set.N, len(data.testing_set.vocab)))
print("There are {} words in the test set that are missing in the training set."
      .format(len(data.testing_set.vocab - data.training_set.vocab)))

assert data.N == data.training_set.N + data.testing_set.N, \
       "The number of training + test samples should sum to the total number of samples"There are a total of 1161192 samples of 56057 unique words in the corpus.
There are 928458 samples of 50536 unique words in the training set.
There are 232734 samples of 25112 unique words in the testing set.
There are 5521 words in the test set that are missing in the training set.

#### Accessing word and tag Sequences
The `Dataset.X` and `Dataset.Y` attributes provide access to ordered collections of matching word and tag sequences for each sentence in the dataset._____no_output_____

# accessing words with Dataset.X and tags with Dataset.Y
for i in range(2):
    print("Sentence {}:".format(i + 1), data.X[i])
    print()
    print("Labels {}:".format(i + 1), data.Y[i])
    print()Sentence 1: ('Mr.', 'Podger', 'had', 'thanked', 'him', 'gravely', ',', 'and', 'now', 'he', 'made', 'use', 'of', 'the', 'advice', '.')

Labels 1: ('NOUN', 'NOUN', 'VERB', 'VERB', 'PRON', 'ADV', '.', 'CONJ', 'ADV', 'PRON', 'VERB', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence 2: ('But', 'there', 'seemed', 'to', 'be', 'some', 'difference', 'of', 'opinion', 'as', 'to', 'how', 'far', 'the', 'board', 'should', 'go', ',', 'and', 'whose', 'advice', 'it', 'should', 'follow', '.')

Labels 2: ('CONJ', 'PRT', 'VERB', 'PRT', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'ADP', 'ADV', 'ADV', 'DET', 'NOUN', 'VERB', 'VERB', '.', 'CONJ', 'DET', 'NOUN', 'PRON', 'VERB', 'VERB', '.')

#### Accessing (word, tag) Samples
The `Dataset.stream()` method returns an iterator that chains together every pair of (word, tag) entries across all sentences in the entire corpus._____no_output_____

# use Dataset.stream() (word, tag) samples for the entire corpus
print("\nStream (word, tag) pairs:\n")
for i, pair in enumerate(data.stream()):
    print("\t", pair)
    if i > 5: break
Stream (word, tag) pairs:

	 ('Mr.', 'NOUN')
	 ('Podger', 'NOUN')
	 ('had', 'VERB')
	 ('thanked', 'VERB')
	 ('him', 'PRON')
	 ('gravely', 'ADV')
	 (',', '.')

For both our baseline tagger and the HMM model we'll build, we need to estimate the frequency of tags & words from the frequency counts of observations in the training corpus. In the next several cells you will complete functions to compute the counts of several sets of counts. _____no_output_____## Step 2: Build a Most Frequent Class tagger
---

Perhaps the simplest tagger (and a good baseline for tagger performance) is to simply choose the tag most frequently assigned to each word. This "most frequent class" tagger inspects each observed word in the sequence and assigns it the label that was most often assigned to that word in the corpus._____no_output_____### IMPLEMENTATION: Pair Counts

Complete the function below that computes the joint frequency counts for two input sequences._____no_output_____

def pair_counts(sequences_A, sequences_B):
    """Return a dictionary keyed to each unique value in the first sequence list
    that counts the number of occurrences of the corresponding value from the
    second sequences list.

    For example, if sequences_A is tags and sequences_B is the corresponding
    words, then if 1244 sequences contain the word "time" tagged as a NOUN, then
    you should return a dictionary such that pair_counts[NOUN][time] == 1244
    """
    # DONE: Finish this function!
    # use Dataset.stream() (word, tag) samples for the entire corpus
    count_dict = defaultdict(lambda: defaultdict(int))

    for tag, word in zip(sequences_A, sequences_B):
        count_dict[tag][word] += 1
    return count_dict

# Calculate C(t_i, w_i)
tags = [tag for i, (word, tag) in enumerate(data.training_set.stream())]
words = [word for i, (word, tag) in enumerate(data.training_set.stream())]
emission_counts = pair_counts(tags, words)
#print(emission_counts['VERB']['had'])
#print(emission_counts['NUM']['had'])

assert len(emission_counts) == 12, \
       "Uh oh. There should be 12 tags in your dictionary."
assert max(emission_counts["NOUN"], key=emission_counts["NOUN"].get) == 'time', \
       "Hmmm...'time' is expected to be the most common NOUN."
HTML('
Your emission counts look good!
')_____no_output_____

### IMPLEMENTATION: Most Frequent Class Tagger

Use the `pair_counts()` function and the training dataset to find the most frequent class label for each word in the training data, and populate the `mfc_table` below. The table keys should be words, and the values should be the appropriate tag string.

The `MFCTagger` class is provided to mock the interface of Pomegranite HMM models so that they can be used interchangeably._____no_output_____

# Create a lookup table mfc_table where mfc_table[word] contains the tag label most frequently assigned to that word
from collections import namedtuple

FakeState = namedtuple("FakeState", "name")

class MFCTagger:
    # NOTE: You should not need to modify this class or any of its methods
    missing = FakeState(name="<MISSING>")

    def __init__(self, table):
        self.table = defaultdict(lambda: MFCTagger.missing)
        self.table.update({word: FakeState(name=tag) for word, tag in table.items()})

    def viterbi(self, seq):
        """This method simplifies predictions by matching the Pomegranate viterbi() interface"""
        return 0., list(enumerate(["<start>"] + [self.table[w] for w in seq] + ["<end>"]))

# TODO: calculate the frequency of each tag being assigned to each word (hint: similar, but not
# the same as the emission probabilities) and use it to fill the mfc_table

tags = [tag for i, (word, tag) in enumerate(data.training_set.stream())]
words = [word for i, (word, tag) in enumerate(data.training_set.stream())]

word_counts = pair_counts(words, tags)
mfc_table = dict((word, max(tags.keys(), key=lambda key: tags[key])) for word, tags in word_counts.items())

# DO NOT MODIFY BELOW THIS LINE
mfc_model = MFCTagger(mfc_table) # Create a Most Frequent Class tagger instance

assert len(mfc_table) == len(data.training_set.vocab), ""
assert all(k in data.training_set.vocab for k in mfc_table.keys()), ""
assert sum(int(k not in mfc_table) for k in data.testing_set.vocab) == 5521, ""
HTML('
Your MFC tagger has all the correct words!
')_____no_output_____

### Making Predictions with a Model
The helper functions provided below interface with Pomegranate network models & the mocked MFCTagger to take advantage of the [missing value](http://pomegranate.readthedocs.io/en/latest/nan.html) functionality in Pomegranate through a simple sequence decoding function. Run these functions, then run the next cell to see some of the predictions made by the MFC tagger._____no_output_____

def replace_unknown(sequence):
    """Return a copy of the input sequence where each unknown word is replaced
    by the literal string value 'nan'. Pomegranate will ignore these values
    during computation.
    """
    return [w if w in data.training_set.vocab else 'nan' for w in sequence]

def simplify_decoding(X, model):
    """X should be a 1-D sequence of observations for the model to predict"""
    _, state_path = model.viterbi(replace_unknown(X))
    return [state[1].name for state in state_path[1:-1]]  # do not show the start/end state predictions_____no_output_____

### Example Decoding Sequences with MFC Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, mfc_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', '<MISSING>', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADV', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

### Evaluating Model Accuracy

The function below will evaluate the accuracy of the MFC tagger on the collection of all sentences from a text corpus. _____no_output_____

def accuracy(X, Y, model):
    """Calculate the prediction accuracy by using the model to decode each sequence
    in the input X and comparing the prediction with the true labels in Y.

    The X should be an array whose first dimension is the number of sentences to test,
    and each element of the array should be an iterable of the words in the sequence.
    The arrays X and Y should have the exact same shape.

    X = [("See", "Spot", "run"), ("Run", "Spot", "run", "fast"), ...]
    Y = [(), (), ...]
    """
    correct = total_predictions = 0
    for observations, actual_tags in zip(X, Y):

        # The model.viterbi call in simplify_decoding will return None if the HMM
        # raises an error (for example, if a test sentence contains a word that
        # is out of vocabulary for the training set). Any exception counts the
        # full sentence as an error (which makes this a conservative estimate).
        try:
            most_likely_tags = simplify_decoding(observations, model)
            correct += sum(p == t for p, t in zip(most_likely_tags, actual_tags))
        except:
            pass
        total_predictions += len(observations)
    return correct / total_predictions_____no_output_____

#### Evaluate the accuracy of the MFC tagger
Run the next cell to evaluate the accuracy of the tagger on the training and test corpus._____no_output_____

mfc_training_acc = accuracy(data.training_set.X, data.training_set.Y, mfc_model)
print("training accuracy mfc_model: {:.2f}%".format(100 * mfc_training_acc))

mfc_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, mfc_model)
print("testing accuracy mfc_model: {:.2f}%".format(100 * mfc_testing_acc))

assert mfc_training_acc >= 0.955, "Uh oh. Your MFC accuracy on the training set doesn't look right."
assert mfc_testing_acc >= 0.925, "Uh oh. Your MFC accuracy on the testing set doesn't look right."
HTML('
Your MFC tagger accuracy looks correct!
')training accuracy mfc_model: 95.72%
testing accuracy mfc_model: 93.01%

## Step 3: Build an HMM tagger
---
The HMM tagger has one hidden state for each possible tag, and parameterized by two distributions: the emission probabilties giving the conditional probability of observing a given **word** from each hidden state, and the transition probabilities giving the conditional probability of moving between **tags** during the sequence.

We will also estimate the starting probability distribution (the probability of each **tag** being the first tag in a sequence), and the terminal probability distribution (the probability of each **tag** being the last tag in a sequence).

The maximum likelihood estimate of these distributions can be calculated from the frequency counts as described in the following sections where you'll implement functions to count the frequencies, and finally build the model. The HMM model will make predictions according to the formula:

$$t_i^n = \underset{t_i^n}{\mathrm{argmax}} \prod_{i=1}^n P(w_i|t_i) P(t_i|t_{i-1})$$

Refer to Speech & Language Processing [Chapter 10](https://web.stanford.edu/~jurafsky/slp3/10.pdf) for more information._____no_output_____### IMPLEMENTATION: Unigram Counts

Complete the function below to estimate the co-occurrence frequency of each symbol over all of the input sequences. The unigram probabilities in our HMM model are estimated from the formula below, where N is the total number of samples in the input. (You only need to compute the counts for now.)

$$P(tag_1) = \frac{C(tag_1)}{N}$$_____no_output_____

def unigram_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequence list that
    counts the number of occurrences of the value in the sequences list. The sequences
    collection should be a 2-dimensional array.

    For example, if the tag NOUN appears 275558 times over all the input sequences,
    then you should return a dictionary such that your_unigram_counts[NOUN] == 275558.
    """
    # DONE: Finish this function!
    return Counter(sequences)

# DONE: call unigram_counts with a list of tag sequences from the training set
tag_unigrams = unigram_counts(tags)

assert set(tag_unigrams.keys()) == data.training_set.tagset, \
       "Uh oh. It looks like your tag counts doesn't include all the tags!"
assert min(tag_unigrams, key=tag_unigrams.get) == 'X', \
       "Hmmm...'X' is expected to be the least common class"
assert max(tag_unigrams, key=tag_unigrams.get) == 'NOUN', \
       "Hmmm...'NOUN' is expected to be the most common class"
HTML('
Your tag unigrams look good!
')_____no_output_____

### IMPLEMENTATION: Bigram Counts

Complete the function below to estimate the co-occurrence frequency of each pair of symbols in each of the input sequences. These counts are used in the HMM model to estimate the bigram probability of two tags from the frequency counts according to the formula: $$P(tag_2|tag_1) = \frac{C(tag_2|tag_1)}{C(tag_2)}$$
_____no_output_____

def bigram_counts(sequences):
    """Return a dictionary keyed to each unique PAIR of values in the input sequences
    list that counts the number of occurrences of pair in the sequences list. The input
    should be a 2-dimensional array.

    For example, if the pair of tags (NOUN, VERB) appear 61582 times, then you should
    return a dictionary such that your_bigram_counts[(NOUN, VERB)] == 61582
    """

    # DONE: Finish this function!
    bigrams = [(sequences[i], sequences[i+1]) for i in range(0, len(sequences)-2)]
    return Counter(bigrams)

# DONE: call bigram_counts with a list of tag sequences from the training set
tag_bigrams = bigram_counts(tags)

assert len(tag_bigrams) == 144, \
       "Uh oh. There should be 144 pairs of bigrams (12 tags x 12 tags)"
assert min(tag_bigrams, key=tag_bigrams.get) in [('X', 'NUM'), ('PRON', 'X')], \
       "Hmmm...The least common bigram should be one of ('X', 'NUM') or ('PRON', 'X')."
assert max(tag_bigrams, key=tag_bigrams.get) in [('DET', 'NOUN')], \
       "Hmmm...('DET', 'NOUN') is expected to be the most common bigram."
HTML('
Your tag bigrams look good!
')_____no_output_____

### IMPLEMENTATION: Sequence Starting Counts
Complete the code below to estimate the bigram probabilities of a sequence starting with each tag._____no_output_____

def starting_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the beginning of
    a sequence.

    For example, if 8093 sequences start with NOUN, then you should return a
    dictionary such that your_starting_counts[NOUN] == 8093
    """
    # DONE: Finish this function!
    seq_start = [seq[0] for seq in sequences]
    return Counter(seq_start)

# DONE: Calculate the count of each tag starting a sequence
tag_starts = starting_counts(data.Y)

assert len(tag_starts) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_starts, key=tag_starts.get) == 'X', "Hmmm...'X' is expected to be the least common starting bigram."
assert max(tag_starts, key=tag_starts.get) == 'DET', "Hmmm...'DET' is expected to be the most common starting bigram."
HTML('
Your starting tag counts look good!
')_____no_output_____

### IMPLEMENTATION: Sequence Ending Counts
Complete the function below to estimate the bigram probabilities of a sequence ending with each tag._____no_output_____

def ending_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the end of
    a sequence.

    For example, if 18 sequences end with DET, then you should return a
    dictionary such that your_starting_counts[DET] == 18
    """
    # DONE: Finish this function!
    seq_end = [seq[len(seq)-1] for seq in sequences]
    return Counter(seq_end)

# TODO: Calculate the count of each tag ending a sequence
tag_ends = ending_counts(data.Y)

assert len(tag_ends) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_ends, key=tag_ends.get) in ['X', 'CONJ'], "Hmmm...'X' or 'CONJ' should be the least common ending bigram."
assert max(tag_ends, key=tag_ends.get) == '.', "Hmmm...'.' is expected to be the most common ending bigram."
HTML('
Your ending tag counts look good!
')_____no_output_____

### IMPLEMENTATION: Basic HMM Tagger
Use the tag unigrams and bigrams calculated above to construct a hidden Markov tagger.

- Add one state per tag
    - The emission distribution at each state should be estimated with the formula: $P(w|t) = \frac{C(t, w)}{C(t)}$
- Add an edge from the starting state `basic_model.start` to each tag
    - The transition probability should be estimated with the formula: $P(t|start) = \frac{C(start, t)}{C(start)}$
- Add an edge from each tag to the end state `basic_model.end`
    - The transition probability should be estimated with the formula: $P(end|t) = \frac{C(t, end)}{C(t)}$
- Add an edge between _every_ pair of tags
    - The transition probability should be estimated with the formula: $P(t_2|t_1) = \frac{C(t_1, t_2)}{C(t_1)}$_____no_output_____

basic_model = HiddenMarkovModel(name="base-hmm-tagger")

# TODO: create states with emission probability distributions P(word | tag) and
# add to the model
# (Hint: you may need to loop & create/add new states)

count_tags_words = pair_counts(tags, words)
count_tags = unigram_counts(tags)

# Loop & add new states
model_states = []
for tag, words_dict in count_tags_words.items():
    total = float(sum(words_dict.values()))
    distribution = {word: count / total for word, count in words_dict.items()}
    tag_emissions = DiscreteDistribution(distribution)
    tag_state = State(tag_emissions, name = tag)
    model_states.append(tag_state)

basic_model.add_states()

# Count of each tag starting a sequence
count_starting_tags = starting_counts(data.Y)

# Tag start probability
prob_start_tag = {}
for tag in tags:
    prob_start_tag[tag] = count_starting_tags[tag] / count_tags[tag]

# Add an edge from the starting state
for tag_state in model_states:
    basic_model.add_transition(basic_model.start, tag_state, prob_start_tag[tag_state.name])

# Count of each tag ending a sequence
count_ending_tags = ending_counts(data.Y)

# Tag end probability
prob_end_tag = {}
for tag in tags:
    prob_end_tag[tag] = count_ending_tags[tag] / count_tags[tag]

# Add an edge from each tag to the end state
for tag_state in model_states:
    basic_model.add_transition(tag_state, basic_model.end, prob_end_tag[tag_state.name])

# DONE: add edges between states for the observed transition frequencies P(tag_i | tag_i-1)
# (Hint: you may need to loop & add transitions
transition_prob = {}
for key in tag_bigrams.keys():
    transition_prob[key] = tag_bigrams.get(key) / count_tags[key[0]]
    #print("Key: {} - {}\n".format(key, transition_prob [key]))
for tag_state in model_states:
    #print("Tag State: {}\n".format(tag_state))
    for next_tag_state in model_states:
        #print("Next State: {}\n".format(next_tag_state))
        basic_model.add_transition(tag_state, next_tag_state,
                                   transition_prob[(tag_state.name, next_tag_state.name)])

# NOTE: YOU SHOULD NOT NEED TO MODIFY ANYTHING BELOW THIS LINE
# finalize the model
basic_model.bake()

assert all(tag in set(s.name for s in basic_model.states) for tag in data.training_set.tagset), \
       "Every state in your network should use the name of the associated tag, which must be one of the training set tags."
assert basic_model.edge_count() == 168, \
       ("Your network should have an edge from the start node to each state, one edge between every " +
        "pair of tags (states), and an edge from each state to the end node.")
HTML('
Your HMM network topology looks good!
')_____no_output_____hmm_training_acc = accuracy(data.training_set.X, data.training_set.Y, basic_model)
print("training accuracy basic hmm model: {:.2f}%".format(100 * hmm_training_acc))

hmm_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, basic_model)
print("testing accuracy basic hmm model: {:.2f}%".format(100 * hmm_testing_acc))

assert hmm_training_acc > 0.97, "Uh oh. Your HMM accuracy on the training set doesn't look right."
assert hmm_testing_acc > 0.955, "Uh oh. Your HMM accuracy on the testing set doesn't look right."
HTML('
Your HMM tagger accuracy looks correct! Congratulations, you\'ve finished the project.
')training accuracy basic hmm model: 97.49%
testing accuracy basic hmm model: 96.07%

### Example Decoding Sequences with the HMM Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, basic_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

## Finishing the project
---

**Note:** **SAVE YOUR NOTEBOOK**, then run the next cell to generate an HTML copy. You will zip & submit both this file and the HTML copy for review.

_____no_output_____

!!jupyter nbconvert *.ipynb_____no_output_____

## Step 4: [Optional] Improving model performance
---
There are additional enhancements that can be incorporated into your tagger that improve performance on larger tagsets where the data sparsity problem is more significant. The data sparsity problem arises because the same amount of data split over more tags means there will be fewer samples in each tag, and there will be more missing data  tags that have zero occurrences in the data. The techniques in this section are optional.

- [Laplace Smoothing](https://en.wikipedia.org/wiki/Additive_smoothing) (pseudocounts)
    Laplace smoothing is a technique where you add a small, non-zero value to all observed counts to offset for unobserved values.

- Backoff Smoothing
    Another smoothing technique is to interpolate between n-grams for missing data. This method is more effective than Laplace smoothing at combatting the data sparsity problem. Refer to chapters 4, 9, and 10 of the [Speech & Language Processing](https://web.stanford.edu/~jurafsky/slp3/) book for more information.

- Extending to Trigrams
    HMM taggers have achieved better than 96% accuracy on this dataset with the full Penn treebank tagset using an architecture described in [this](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf) paper. Altering your HMM to achieve the same performance would require implementing deleted interpolation (described in the paper), incorporating trigram probabilities in your frequency tables, and re-implementing the Viterbi algorithm to consider three consecutive states instead of two.

### Obtain the Brown Corpus with a Larger Tagset
Run the code below to download a copy of the brown corpus with the full NLTK tagset. You will need to research the available tagset information in the NLTK docs and determine the best way to extract the subset of NLTK tags you want to explore. If you write the following the format specified in Step 1, then you can reload the data using all of the code above for comparison.

Refer to [Chapter 5](http://www.nltk.org/book/ch05.html) of the NLTK book for more information on the available tagsets._____no_output_____

import nltk
from nltk import pos_tag, word_tokenize
from nltk.corpus import brown

nltk.download('brown')
training_corpus = nltk.corpus.brown
training_corpus.tagged_sents()[0]_____no_output_____


# Notebook from snapbuy/rapids-single-cell-examples
Path: notebooks/dsci_bmmc_60k_gpu.ipynb

# RAPIDS analysis of single-cell ATAC-seq data from 60K cells

Copyright (c) 2020, NVIDIA CORPORATION.
Licensed under the Apache License, Version 2.0 (the "License") you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License._____no_output_____The dataset we use here comes from [Lareau et al., Nat Biotech 2019](https://www.nature.com/articles/s41587-019-0147-6), one of the highest throughput single-cell ATAC-seq experiments to date. They assay 136K resting and stimulated bone marrow-derived cells and study the different cell types and lineages that emerge. The paper analyzes resting & stimulated cells separately (and we also find that these two populations split into largely distinct clusters), so here we only focus on the 60K resting cells from this experiment.

The cell metadata from this experiment includes annotated putative cell types based on chromVAR, Louvain clustering, and downstream cluster analysis. We compare our de novo identified clusters to these annotated clusters from the paper (and find good agreement).

The peak-cell matrix along with peak and cell metadata files are publicly available on GEO, [GSE123580](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE123580)._____no_output_____### Import requirements_____no_output_____

import numpy as np
import scanpy as sc
import anndata
import matplotlib.pyplot as plt
from collections import OrderedDict

import time
import os, wget

import cudf
import cupy as cp

from cuml.decomposition import PCA
from cuml.manifold import TSNE

import rapids_scanpy_funcs
import utils

import warnings
warnings.filterwarnings('ignore', 'Expected ')
warnings.simplefilter('ignore')_____no_output_____import rmm

rmm.reinitialize(
    managed_memory=True, # Allows oversubscription
    devices=0, # GPU device IDs to register. By default registers only GPU 0.
)

cp.cuda.set_allocator(rmm.rmm_cupy_allocator)_____no_output_____

## Set parameters_____no_output_____

# filtering peaks
n_top_peaks = 25000 # Number of peaks to retain

# PCA
n_components = 50 # Number of principal components to compute

# t-SNE
tsne_n_pcs = 20 # Number of principal components to use for t-SNE

# KNN
n_neighbors = 10 # Number of nearest neighbors for KNN graph
knn_n_pcs = 50 # Number of principal components to use for finding nearest neighbors

# UMAP
umap_min_dist = 0.4
umap_spread = 1.0

# Differential peaks
n_diff_peaks = 50 # Number of differential peaks to compute for each cluster

# Marker genes
# Gene coordinates are in GRCh37, from grch37.ensembl.org
markers = ['MS4A1', 'LEF1', 'NKG7', 'TREM1', 'GATA1', 'PAX5']
marker_coords = OrderedDict()
marker_coords['MS4A1'] = ['chr11', 60223225, 60238233, 'fwd']
marker_coords['LEF1'] = ['chr4', 108968701, 109090112, 'rev']
marker_coords['NKG7'] = ['chr19', 51874860, 51875969, 'rev']
marker_coords['TREM1'] = ['chr6', 41235664, 41254457, 'rev']
marker_coords['GATA1'] = ['chrX', 48644962, 48652716, 'fwd']
marker_coords['PAX5'] = ['chr9', 36833272, 37034103, 'rev']_____no_output_____start_time = time.time()_____no_output_____

## Input data

In the cell below, we provide the path to the sparse .h5ad file containing the peak x cell count matrix to analyze. Please see the README for instructions on how to download the dataset we use here._____no_output_____

%%time
input_file = "../data/dsci_resting_nonzeropeaks.h5ad"

if not os.path.exists(input_file):
    print('Downloading import file...')
    os.makedirs('../data', exist_ok=True)
    wget.download('https://rapids-single-cell-examples.s3.us-east-2.amazonaws.com/dsci_resting_nonzeropeaks.h5ad',
                  input_file)

adata = anndata.read_h5ad(input_file)
adata_raw = adata.copy()
print(adata.X.shape)(60495, 143691)
CPU times: user 281 ms, sys: 344 ms, total: 625 ms
Wall time: 723 ms

## Preprocessing_____no_output_____

preprocessing_start = time.time()_____no_output_____

### tf-idf normalization

We use log1p(pseudocount / tf)\*idf, motivated by discussion in [this blog post](http://andrewjohnhill.com/blog/2019/05/06/dimensionality-reduction-for-scatac-data/) as well as our results on various scATAC datasets._____no_output_____

%%time

lognormalized = utils.logtf_idf(adata.X, pseudocount=10**3)
adata.X = lognormalizedCPU times: user 2.01 s, sys: 895 ms, total: 2.91 s
Wall time: 2.91 s

### Frequency-based peak selection
We want to choose peaks that have highest discriminative power between cells. A good proxy for discriminative power is variance; since peak data (especially on a sparse dataset like this) is essentially binary, the highest variance peaks are the ones with closest to 50\% representation across cells (p=1/2 maximizes Bernoulli variance). The most frequent peak in this dataset only has 33\% representation, so in practice we just end up taking the top N most frequent peaks, where N is a parameter we set above (default 25000)._____no_output_____

%%time
adata = utils.filter_peaks(adata, n_top_peaks)
print(adata.X.shape)(60495, 25000)
CPU times: user 931 ms, sys: 245 ms, total: 1.18 s
Wall time: 1.17 s
print("Preprocessing time: %.2fsec" % (time.time() - preprocessing_start))Preprocessing time: 4.10sec

## Cluster & Visualize_____no_output_____### PCA_____no_output_____

%%time
adata = anndata.AnnData(X=adata.X.todense(),
                        obs=adata.obs,
                        var=adata.var)
adata.obsm["X_pca"] = PCA(n_components=n_components).fit_transform(adata.X)
adata.obsm["X_pca"].shapeCPU times: user 35.4 s, sys: 25.5 s, total: 1min
Wall time: 1min

### UMAP visualization_____no_output_____

%%time
sc.pp.neighbors(adata, n_neighbors=n_neighbors, n_pcs=knn_n_pcs, method='rapids')CPU times: user 3.72 s, sys: 355 ms, total: 4.08 s
Wall time: 4.3 s
%%time
sc.tl.umap(adata, min_dist=umap_min_dist, spread=umap_spread, method='rapids')WARNING: .obsp["connectivities"] have not been computed using umap
CPU times: user 361 ms, sys: 289 ms, total: 649 ms
Wall time: 652 ms

### Graph clustering_____no_output_____Below, we show how to use the RAPIDS Louvain clustering or Leiden clustering functions to cluster the data._____no_output_____

%%time
sc.tl.louvain(adata, flavor='rapids')CPU times: user 78.1 ms, sys: 15.8 ms, total: 93.9 ms
Wall time: 92.4 ms
%%time
adata.obs['leiden'] = rapids_scanpy_funcs.leiden(adata)CPU times: user 45 ms, sys: 12.7 ms, total: 57.7 ms
Wall time: 56.5 ms

Below, we plot the UMAP visualization of the cells colored by the Louvain clusters. Next, we color the cells using the cell type annotations reported in the source paper. We see that the clusters we discovered match closely with the reported cell types._____no_output_____

%%time

fig, axs = plt.subplots(nrows = 1, ncols = 3, figsize = (12, 3), dpi=100)

sc.pl.umap(adata, color=['louvain'], ax=axs[0], show=False)
sc.pl.umap(adata, color=['leiden'], ax=axs[1], show=False)
sc.pl.umap(adata, color=['cell_type'], ax=axs[2], show=False)

plt.tight_layout()
plt.show()_____no_output_____

### tSNE visualization_____no_output_____

%%time
adata.obsm['X_tsne'] = TSNE().fit_transform(adata.obsm["X_pca"][:, :tsne_n_pcs])[W] [19:53:51.493511] # of Nearest Neighbors should be at least 3 * perplexity. Your results might be a bit strange...
CPU times: user 875 ms, sys: 603 ms, total: 1.48 s
Wall time: 1.48 s
%%time

fig, axs = plt.subplots(nrows = 1, ncols = 3, figsize = (12, 3), dpi=100)

sc.pl.tsne(adata, color=['louvain'], ax=axs[0], show=False)
sc.pl.tsne(adata, color=['leiden'], ax=axs[1], show=False)
sc.pl.tsne(adata, color=['cell_type'], ax=axs[2], show=False)

plt.tight_layout()
plt.show()_____no_output_____

### Marker genes_____no_output_____We now compute a gene activity score representing the activity of each marker gene in each cell. We use the `adata_raw` object to compute this score. This allows us to consider all peaks, not just the subset we used for clustering and UMAP._____no_output_____

%%time
# find which peaks overlap with each marker gene (+ 5kb upstream)
peaks_per_gene = {gene : [] for gene in markers}

for (i, peak) in enumerate(adata_raw.var_names):
    chrom, coords = peak.strip().split(':')
    start, end = [int(c) for c in coords.split('-')]
    for gene in markers:
        if utils.overlap(marker_coords[gene], [chrom, start, end], 5000, 0):
            peaks_per_gene[gene].append((i, peak))

for gene in peaks_per_gene:
    print(f"{gene} has {len(peaks_per_gene[gene])} nearby peaks")MS4A1 has 2 nearby peaks
LEF1 has 30 nearby peaks
NKG7 has 4 nearby peaks
TREM1 has 5 nearby peaks
GATA1 has 6 nearby peaks
PAX5 has 30 nearby peaks
CPU times: user 434 ms, sys: 3.3 ms, total: 438 ms
Wall time: 436 ms
%%time
# Calculate per-cell gene activity scores for each marker gene
cell_gene_scores = np.zeros((adata.X.shape[0], len(marker_coords.keys())))
for (i, gene) in enumerate(peaks_per_gene.keys()):
    indices = [x[0] for x in peaks_per_gene[gene]]
    peak_scores = adata_raw.X[:, indices].toarray()
    cell_gene_scores[:, i] = np.sum(peak_scores, axis=1) + 1

# Store per-cell gene activity scores with adata
for (i, gene) in enumerate(markers):
    adata.obs[gene] = np.log2(cell_gene_scores[:, i].tolist())
    adata.obs.loc[np.isneginf(adata.obs[gene]), gene] = 0CPU times: user 2.18 s, sys: 9.89 ms, total: 2.19 s
Wall time: 2.19 s

We now visualize the UMAP plot colored by activity of marker genes. We observe that the activation of cell-type specific marker genes correlates well with the cell types identified by Louvain clustering._____no_output_____

%%time

fig, axs = plt.subplots(nrows=2, ncols=3, figsize = (8, 4), dpi=100)
axs = axs.flatten()
for (i, gene) in enumerate(markers):
    sc.pl.umap(adata, color_map="Blues", color=gene, ax = axs[i], show=False)
plt.tight_layout()
plt.show()_____no_output_____

## Find Differential peaks

We perform an accelerated logistic regression-based differential peak computation using RAPIDS. This function may not give the exact same results as the equivalent Scanpy CPU function._____no_output_____

%%time
cluster_labels = cudf.Series.from_categorical(adata.obs["louvain"].cat)
var_names = cudf.Series(adata.var_names)
dense_gpu_array = cp.sparse.csr_matrix(cp.array(adata.X))

scores, names, reference = rapids_scanpy_funcs.rank_genes_groups(
    dense_gpu_array,
    cluster_labels,
    var_names,
    penalty='none',
    n_genes=n_diff_peaks, groups='all', reference='rest')[W] [19:54:05.630913] L-BFGS stopped, because the line search failed to advance (step delta = 0.000000)
CPU times: user 5.02 s, sys: 1.93 s, total: 6.95 s
Wall time: 6.99 s
%%time
fig, axs = plt.subplots(len(names[0])//3 + 1, 3, figsize=(11,11), dpi=100)
axs = axs.flatten()

for (i, peak) in enumerate(names[0]):
    sc.pl.umap(adata, color=peak,
           ax=axs[i], show=False,
           vmax=10, vmin=-0.5,
           cmap='Blues'
          )
plt.tight_layout()
plt.show()_____no_output_____print("Full time: %.2fsec" % (time.time() - start_time))Full time: 90.28sec


# Notebook from dagousket/deep-learning-biology
Path: exercises/assignments/2020-dlb-5-unsupervised-neural-networks.ipynb

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JorisRoels/deep-learning-biology/blob/main/exercises/assignments/2020-dlb-5-unsupervised-neural-networks.ipynb)

# Exercise 5: Unsupervised neural networks

In this notebook, we will be see how autoencoders can be used for clustering and semi-supervised classification.

The structure of these exercises is as follows:

1. [Import libraries and download data](#scrollTo=ScagUEMTMjlK)
2. [Data pre-processing](#scrollTo=ohZHyOTnI35b)
3. [Building an autoencoder](#scrollTo=kIry8iFZI35y)
4. [Clustering with autoencoders](#scrollTo=uXrEb0rTI35-)
5. [Semi-supervised classification](#scrollTo=kEU34VvDUPr_)_____no_output_____## 1. Import libraries and download data
Let's start with importing the necessary libraries. _____no_output_____

import pickle
import numpy as np
import random
import os
import matplotlib.pyplot as plt
plt.rcdefaults()
import pandas as pd
from sklearn.metrics import adjusted_rand_score
from sklearn.model_selection import train_test_split
from sklearn.manifold import TSNE
from sklearn.cluster import KMeans
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn import svm, metrics
from progressbar import ProgressBar, Percentage, Bar, ETA, FileTransferSpeed
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch.utils.data as data
from torch.utils.data import TensorDataset, DataLoader
from torchvision import datasets, transforms
import gdown
import zipfile
import os_____no_output_____

As you will notice, Colab environments come with quite a large library pre-installed. If you need to import a module that is not yet specified, you can add it in the previous cell (make sure to run it again). If the module is not installed, you can install it with `pip`.

To make your work reproducible, it is advised to initialize all modules with stochastic functionality with a fixed seed. Re-running this script should give the same results as long as the seed is fixed. _____no_output_____

# make sure the results are reproducible
seed = 0
np.random.seed(seed)
random.seed(seed)
torch.manual_seed(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

# run all computations on the GPU if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print('Running computations with %s' % torch.device(device))
if torch.cuda.is_available():
    print(torch.cuda.get_device_properties(device))_____no_output_____

## 2. Data pre-processing

We will use the well-known MNIST benchmarking dataset. This dataset is well-known in the world of machine learning. It consists of relatively small $28 \times 28$ pixel images of digits. The goals of the dataset is to classify the digit images to the actual number that is on the image. _____no_output_____

# batch size for the dataloader
batch_size = 64

# data loaders
train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('data', train=True, download=True,
                   transform=transforms.ToTensor()), batch_size=batch_size,
                   shuffle=False)
test_loader = torch.utils.data.DataLoader(
    datasets.MNIST('data', train=False, download=True,
                   transform=transforms.ToTensor()), batch_size=batch_size,
                   shuffle=False)

# print statistics
print('The train data consists of %d samples' % len(train_loader.dataset))
print('The test data consists of %d samples' % len(test_loader.dataset))_____no_output_____

Here are a few examples of the data, the corresponding label is on top of each image. _____no_output_____

# visualize the data
n = 5
for i in range(n):
    plt.subplot(1, n, i+1)
    plt.imshow(test_loader.dataset[i][0].numpy()[0, ...], cmap='gray')
    plt.title(test_loader.dataset[i][1])
    plt.axis('off')
plt.show()_____no_output_____

## 3. Building an autoencoder

Now, we have to implement the autoencoder and train it. Let's start by defining the architecture.

**Exercise**: build an autoencoder with a single hidden layer. Not that every pixel in the image corresponds to an input dimension, i.e. the input and output of the autoencoder will be $28 \times 28$ dimensional and images will have to be reshaped in vector format. Implement the following architecture:
- The first layer (encoder) will be a [fully connected layer](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear) with [sigmoid](https://pytorch.org/docs/stable/generated/torch.nn.Sigmoid.html) activation that transforms the input features to a 64 dimensional (hidden) feature vector representation.
- The second layer (decoder) is another [fully connected layer](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear) that transforms the hidden representation to the original input dimensions.
- Print the network architecture and propagate a test sample to validate your architecture. _____no_output_____

class AutoEncoder(nn.Module):

    def __init__(self, dim_input, dim_hidden):
        super().__init__()

        """
        INSERT CODE HERE
        """

    def forward(self, x):

        """
        INSERT CODE HERE
        """

        return x

dim_hidden = 64
"""
INSERT CODE HERE
"""

# propagate a sample
"""
INSERT CODE HERE
"""

# show the result
"""
INSERT CODE HERE
"""_____no_output_____

Time to train the network now! Remember, a single training iteration consists of the following steps:
1. Construct a batch
2. Zero the gradients
3. Forward propagation
4. Compute the loss
5. Backward propagation
6. Update the weights

**Exercise**: train the autoencoder for 10 epochs on the data.
- Map the autoencoder to the computing device
- Define an Adam optimizer, with a learning rate of $10^{-2}$ and weight decay of $10^{-5}$
- Define a mean squared error (MSE) loss function
- Fill in the missing code in the `train_epoch` and `test_epoch` functions
- Run the autoencoder on a sample and show the obtained reconstruction_____no_output_____

# train for one epoch
def train_epoch(net, loader, optimizer, criterion, device):
    net.train()
    loss_avg = 0
    for x, _ in loader:
        """
        INSERT CODE HERE
        """

    loss_avg = loss_avg / len(loader)
    return loss_avg

# test for one epoch
def test_epoch(net, loader, criterion, device):
    net.eval()
    loss_avg = 0
    for x, _ in loader:
        """
        INSERT CODE HERE
        """

    loss_avg = loss_avg / len(loader)
    return loss_avg

# train the autoencoder
def train(net, train_loader, test_loader, optimizer, epochs, criterion, device):
    train_losses = np.zeros((epochs))
    test_losses = np.zeros((epochs))
    for epoch in range(epochs):
        train_losses[epoch] = train_epoch(net, train_loader, optimizer, criterion, device)
        test_losses[epoch] = test_epoch(net, test_loader, criterion, device)
        print("Epoch %d/%d: Train loss = %.4f - Test loss = %.4f"
              % (epoch + 1, epochs, train_losses[epoch], test_losses[epoch]))

    return train_losses, test_losses

# parameters
lr = 1e-2
weight_decay = 1e-5
epochs = 10

# computing device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# map the network to the device
"""
INSERT CODE HERE
"""

# optimizer
"""
INSERT CODE HERE
"""

# loss function
"""
INSERT CODE HERE
"""

# start training
train_loss, test_loss = train(net_ae, train_loader, test_loader,
                              optimizer, epochs, criterion, device)

# visualize training result
plt.plot(train_loss)
plt.plot(test_loss)
plt.legend(('Train', 'Test'))
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.show()_____no_output_____# propagate a sample
"""
INSERT CODE HERE
"""

# show the result
"""
INSERT CODE HERE
"""_____no_output_____

## 4. Clustering with autoencoders

In this section, we will show you how you can cluster data with an autoencoder model. Perhaps the simplest approach is to cluster the data in the encoded space, e.g. using k-means clustering.

**Exercise**: Cluster the MNIST test data using the encodings and k-means clustering
- Compute the embeddings of the test data and store them in a numpy array
- Apply k-means clustering on the embeddings
- Evaluate the clustering with the [adjusted rand score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_rand_score.html#sklearn.metrics.adjusted_rand_score). _____no_output_____

def compute_embeddings(net, loader):
    """
    INSERT CODE HERE
    """
    return embeddings

# compute the embeddings of the autoencoder on the test set
"""
INSERT CODE HERE
"""

# cluster the embeddings
"""
INSERT CODE HERE
"""

# evaluate
"""
INSERT CODE HERE
"""_____no_output_____

The following code will compare the true labels and the clustering results. Note that the colors do not necessarily correspond to the same cluster. _____no_output_____

# reduce the dimensionality of the hidden representations
h_red = TSNE(n_components=2, random_state=seed).fit_transform(h[:1000])

# visualize the reduced representations and label each sample
plt.subplot(1, 2, 1)
scatter = plt.scatter(h_red[:, 0], h_red[:, 1], c=labels[:1000], cmap='Paired')
plt.legend(handles=scatter.legend_elements()[0], labels=range(10))
plt.title('True labels')
plt.subplot(1, 2, 2)
scatter = plt.scatter(h_red[:, 0], h_red[:, 1], c=predictions[:1000], cmap='Paired')
plt.title('Clustering')
plt.show()_____no_output_____

We will now focus towards the extracted features of the autoencoder. Within the encoder, there are connections to each pixel for each hidden variable. In other words, we can visualize which pixels are important to activate a particular hidden variable.

**Exercise**: Visualize the extracted features based on the matrix weights from the encoder.
- Extract the encoder matrix parameter values
- For each hidden variable, reshape the corresponding features connected to the input to a $28 \times 28$ image
- Visualize all these images_____no_output_____

# extract the matrix parameters
"""
INSERT CODE HERE
"""

# visualize the features
"""
INSERT CODE HERE
"""_____no_output_____

## 5. Semi-supervised classification

Semi-supervised classification aims to use large amounts of unlabeled data and a limited amount of labeled data for maximum classification performance. We will see that the extracted features of an autoencoder can serve as suitable features for classification.

**Exercise**: Train and validate an SVM for increasing amounts of labeled training data.
- Compute the embeddings of the train and test data.
- Implement the `train_svm` function that trains an SVM with training $n$ samples.
- Implement the `test_svm` function that tests an SVM on the complete test set and reports the accuracy.
- Train an SVM for various amounts of labeled data with the autoencoder embeddings as input.
- As a baseline, compare this approach to an SVM that is trained on the actual pixel values instead of the encodings. _____no_output_____

def train_svm(x_train, y_train, n):
    """
    INSERT CODE HERE
    """
    return clf

def test_svm(clf, x_test, y_test):
    """
    INSERT CODE HERE
    """
    return acc

# compute the embeddings of the autoencoder on the train and test set
"""
INSERT CODE HERE
"""

# train and test an SVM for a varying amount of training data
n_train_min = 10
n_train_max = 200
n_train_step = 10
n_samples = np.arange(n_train_min, n_train_max, n_train_step)
accuracies_ae = np.zeros((len(n_samples)))
accuracies_bl = np.zeros((len(n_samples)))
for j in range(len(n_samples)):
    # autoencoder
    """
    INSERT CODE HERE
    """
    # baseline
    """
    INSERT CODE HERE
    """

# visualize the results
"""
INSERT CODE HERE
"""_____no_output_____


# Notebook from astrolabsoftware/fink-notebook-template
Path: extragalactic/tuning_snia_output_rate.ipynb

# Tuning substreams output rate_____no_output_____## Goal

The goal of this notebook is to show how users can tune the output rate of existing substreams in Fink. We will make an example on the [Early Supernova type Ia filter](https://github.com/astrolabsoftware/fink-filters/blob/master/fink_filters/filter_early_sn_candidates/filter.py). This filter uses 6 criteria to select likely Early Supernova type Ia candidates from the ZTF data stream:

1. Ia probability larger than 50% from the early supernova Ia module [2111.11438](https://arxiv.org/abs/2111.11438), `rfscore`
2. Ia probability larger than 50% from either one of the deep learning classifier based on SuperNNova [1901.06384](https://arxiv.org/abs/1901.06384), `snn_sn_vs_all` & `snn_snia_vs_nonia`
3. no match with galactic objects from the SIMBAD database, `cdsxmatch`
4. non-bogus probability higher than 50% from the RealBogus algorithm (Ma- habal et al. 2019; Duev et al. 2019), `rb`
5. Star/Galaxy classification score from SExtractor above 40% (Bertin, E. & Arnouts, S. 1996), `classtar`
6. the associated object holds no more than 20 photometric measurements (above 3 sigma) `ndethist`

One can refine those substreams, by defining a different threshold on the rate. In practice, as each substream is generated from a combination of parameters, you need to define which parameter(s) to tune. In this notebook we focus on (1) and/or (2) only, as others are more "quality cuts" (but feel free to explore!):

- `rfscore`: returns the probability of an alert to be a SNe Ia using a Random Forest Classifier (binary classification)
- `snn_*`: returns the probability of an alert to be a SNe Ia using SuperNNova. Two pre-trained models:
    - `snn_snia_vs_nonia`: Ia vs core-collapse SNe
    - `snn_sn_vs_all`: SNe vs. anything else (variable stars and other categories in the training)

### Method

For the purpose of the test, we will retrieve all alerts (and not objects) that were considered as Early supernova type Ia candidates over 2021 using the Fink REST API, and retro-actively apply new thresholds. In practice, users would do the same on the thresholds, but using the [livestream](https://github.com/astrolabsoftware/fink-client) service._____no_output_____## Environment set up

To run this notebook, you need to import the following libraries:_____no_output_____

import requests

import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns
sns.set_context('talk')

APIURL = 'https://fink-portal.org'_____no_output_____

### Retrieve past candidates

Each night, alerts are cross-matched in real-time against the CDS SIMBAD database. About 30% of the stream (after applying quality cuts) ends-up with a counter-part. Classified alerts can be then queried using the Fink API service. To know all available classes in Fink, see https://fink-portal.org/api/v1/classes. We will first focus on alerts (and not objects) that were considered as Early supernova type Ia candidates over 2021 (can take 1-2 minutes):_____no_output_____

# Get latests alerts in a time window
r = requests.post(
  '{}/api/v1/latests'.format(APIURL),
  json={
    'class': 'Early SN Ia candidate',
    'n': '3000' ,                  # number of alerts
    'startdate': '2021-01-01',  # use a start date
    'stopdate': '2021-10-31',   # use a stop date
  }
)

# Format output in a DataFrame
pdf = pd.read_json(r.content)_____no_output_____cols = [
    'd:rfscore',
    'd:snn_sn_vs_all',
    'd:snn_snia_vs_nonia',
    'd:cdsxmatch',
    'i:rb',
    'i:classtar',
    'i:nid',
    'i:objectId'
]

pdf[cols]_____no_output_____nnid = len(np.unique(pdf['i:nid']))

print('Considering {} nights'.format(nnid))Considering 230 nights
fig = plt.figure(figsize=(12, 5))

nnids = pdf[cols].groupby('i:nid').count()['i:objectId'].values

med_ = np.median(nnids)
max_ = np.max(nnids)

hist, ncand, _ = plt.hist(
    nnids,
    histtype='step',
    bins=25,
    label='median={:.0f}, max={:.0f}'.format(med_, max_)
)

plt.xlabel('Number of candidates per night')
plt.legend();_____no_output_____print('Median at {:.0f} candidates per day, and a maximum of {:.0f} candidates in a night'.format(med_, max_))Median at 9 candidates per day, and a maximum of 29 candidates in a night

Let's inspect the distribution for the ML scores:_____no_output_____

fig = plt.figure(figsize=(12, 5))

n_rfscore = pdf[cols].groupby('d:rfscore').count()['i:objectId'].values

bins = np.arange(0.0, 1.0, 0.5/100)

plt.hist(pdf['d:rfscore'], histtype='step', bins=bins, label='rfscore')
plt.hist(pdf['d:snn_sn_vs_all'], histtype='step', bins=bins, label='snn_sn_vs_all')
plt.hist(pdf['d:snn_snia_vs_nonia'], histtype='step', bins=bins, label='snn_snia_vs_nonia')
plt.xlabel('ML scores')
plt.legend(loc='upper left');_____no_output_____

Note that we require `rfscore > 0.5 & (snn_sn_vs_all > 0.5 | snn_snia_vs_nonia > 0.5)`, hence one of the two `snn_*` score can go below 0.5 (but not both at the same time)._____no_output_____## 1D case_____no_output_____Let's vary the parameters individually, and see the effect on the output rate. If we just set for example the Ia probability `rfscore` larger than 90% from the early supernova Ia module (instead of 50%), we obtain:_____no_output_____

fig = plt.figure(figsize=(12, 5))

f1 = pdf['d:rfscore'] > 0.9

nnids = pdf[cols][f1].groupby('i:nid').count()['i:objectId'].values

med_ = np.median(nnids)
max_ = np.max(nnids)

hist, ncand, _ = plt.hist(
    nnids,
    histtype='step',
    bins=10,
    label='median={:.0f}, max={:.0f}'.format(med_, max_)
)

plt.xlabel('Number of candidates per night')
plt.legend();_____no_output_____

The median goes from 9/night to 3/night. Let's inspect this more programatically:_____no_output_____

def extract_stats_from_ml_threshold(pdf, key, nsteps=20):
    start = 0.5
    stop = 1.0
    step = (stop - start) / nsteps
    bins = np.arange(start, stop, step)

    medians, maxs = [], []
    for score_ in bins:
        f1 = pdf[key] > score_

        nids = pdf[f1].groupby('i:nid').count()['i:objectId'].values

        if len(nids) > 0:
            medians.append(np.median(nids))
            maxs.append(np.max(nids))
        else:
            medians.append(0)
            maxs.append(0)

    return bins, medians, maxs_____no_output_____bins, medians, maxs = extract_stats_from_ml_threshold(pdf, key='d:rfscore', nsteps=50)

fig = plt.figure(figsize=(12, 5))

plt.plot(bins, medians, label='Median')
plt.plot(bins, maxs, label='Max')
plt.xlabel('rfscore threshold')
plt.ylabel('candidates per night')
plt.legend()
plt.grid(ls='--')
plt.ylim(0, 30);_____no_output_____

one can see that for very high probabilities, the number of of candidates quickly drops. The same for the two other parameters:_____no_output_____

bins, medians, maxs = extract_stats_from_ml_threshold(pdf, key='d:snn_snia_vs_nonia', nsteps=20)

fig = plt.figure(figsize=(12, 5))

plt.plot(bins, medians, label='Median')
plt.plot(bins, maxs, label='Max')
plt.xlabel('snn_snia_vs_nonia threshold')
plt.ylabel('candidates per night')
plt.legend()
plt.grid(ls='--')
plt.ylim(0, 30)_____no_output_____bins, medians, maxs = extract_stats_from_ml_threshold(pdf, key='d:snn_sn_vs_all', nsteps=20)

fig = plt.figure(figsize=(12, 5))

plt.plot(bins, medians, label='Median')
plt.plot(bins, maxs, label='Max')
plt.xlabel('snn_sn_vs_all threshold')
plt.ylabel('candidates per night')
plt.legend()
plt.grid(ls='--')
plt.ylim(0, 30)_____no_output_____

## N-D case

I'll let users playing with N-dimensional parameter space (several parameters at once) :-)_____no_output_____


# Notebook from xduan7/dl-bio-seqs
Path: notebooks/genome_processing.ipynb

This notebook demonstrates how to parse genome download from [PATRIC](https://www.patricbrc.org/), including reading, CD searching, and printing.

[PATRIC USER GUIDES/FTP](https://docs.patricbrc.org/user_guides/ftp.html):
> each genome directory provides the following data files for PATRIC and RefSeq annotations
- fna: FASTA contig sequences
- features.tab: All genomic features and related information in tab-delimited format
- ...
_____no_output_____

<code>
import os
import json
import time

import pandas as pd
from Bio import SeqIO
from Bio.Blast.Applications import NcbirpstblastnCommandline

# path to CDD (conserved domain database) and related info
CDD_PATH = os.path.abspath('../data/interim/CDD/Cdd')
RPSBLAST_KWARGS = {
    'db': CDD_PATH,
    'seg': 'no',
    'comp_based_stats': '1',
    'evalue': 0.01,
    'outfmt': 5,
    'num_threads': 16,
}

# path to all the genomes (downloaded from PATRIC directly)
GENOME_PARENT_PATH = os.path.abspath('./examples/genome_processing')
_____no_output_____# get all the genome names (PATRIC ID)
genome_ids = next(os.walk(GENOME_PARENT_PATH))[1]

for _genome_id in genome_ids:
    _genome_path = os.path.join(GENOME_PARENT_PATH, _genome_id)

    # get the nucleotide sequences (whole contig and the one with features only)
    _cntg_seq_path = os.path.join(_genome_path, _genome_id + '.fna')
    _cntg_seq_records = list(SeqIO.parse(_cntg_seq_path, 'fasta'))
    # _feat_seq_path = os.path.join(_genome_path, _genome_id + '.PATRIC.ffn')
    # _feat_seq_records = list(SeqIO.parse(_feat_seq_path, 'fasta'))

    print(f'Genome {_genome_id} contains {len(_cntg_seq_records)} contig(s)')

    # make directory for contigs ...
    _cntg_path = os.path.join(_genome_path, 'contigs')
    if not os.path.isdir(_cntg_path):
        os.makedirs(_cntg_path)

    # split contigs and write into different files named with their IDs
    _num_cntg = len(_cntg_seq_records)
    _cntg_dict = {}
    for _cntg_seq_record in _cntg_seq_records:
        _cntg_dict[_cntg_seq_record.id] = {'length': len(_cntg_seq_record)}
        with open(os.path.join(_genome_path, 'contigs', f'{_cntg_seq_record.id}.fna'), 'w+') as f:
            SeqIO.write(_cntg_seq_record, f, 'fasta')

    # split features for each and every contig the same way
    # features are saved in TSV format (default pandas setting)
    _feat_path = os.path.join(_genome_path, 'features')
    if not os.path.isdir(_feat_path):
        os.makedirs(_feat_path)
    _feat_df = pd.read_table(os.path.join(_genome_path,  f'{_genome_id}.PATRIC.features.tab'))
    for _accession, _feat_by_accession_df in _feat_df.groupby('accession'):
        _source_mask = _feat_by_accession_df['feature_type'] == 'source'
        _feat_by_accession_df[~_source_mask].to_csv(os.path.join(_feat_path, f'{_accession}.tsv'), index=None, sep='\t')

    # create an information file that contains
    # - genome name
    # - number of contigs
    # - each contig accession and its length (maybe information for genes and CDs?)
    _genome_info = {
        'genome_name': _feat_df['genome_name'].unique()[0],
        'number_of_contigs': _num_cntg,
        'contigs': _cntg_dict,
    }

    with open(os.path.join(_genome_path, 'info.json'), 'w+') as f:
        json.dump(_genome_info, f, indent=4)
Genome 562.2282 contains 3 contig(s)
# the following statement is for rpsblast with translated nucleotide sequences
# make sure of the following before proceed to conserved domain search:
#   - the features in *.ffn and *.features.tab are directly sampled from the contigs
#       - start, end, and name should be matched both ...
#   - the protein sequences in *.faa file belong to a subset of the features in *.ffn and *.features.tab
#   - every protein in *.faa file could be translated from a feature nucleotide in *.ffn and *.features.tab
#       - use Biopython Seq.translate() for verification ...
#       - what about the introns ...
#       - strand ...

for _genome_id in genome_ids:
    _genome_path = os.path.join(GENOME_PARENT_PATH, _genome_id)
    _cntg_cd_path = os.path.join(_genome_path, 'conserved_domains')
    if not os.path.isdir(_cntg_cd_path):
        os.makedirs(_cntg_cd_path)

    # get all the contigs and iterate through for conserved domain search
    with open(os.path.join(_genome_path, 'info.json'), 'r') as f:
        _genome_info = json.load(f)
    _cntg_ids = _genome_info['contigs'].keys()

    _start_time = time.time()
    for _cntg_id in _cntg_ids:
        _cntg_path = os.path.join(_genome_path, 'contigs', f'{_cntg_id}.fna')
        _cntg_rpsblast_cmd = NcbirpstblastnCommandline(query=_cntg_path, **RPSBLAST_KWARGS)
        _cntg_cd_result, _cntg_rpsblast_cmd_error_msg = _cntg_rpsblast_cmd()
        _cntg_cd_result_path = os.path.join(_cntg_cd_path, f'{_cntg_id}.xml')
        with open(_cntg_cd_result_path, 'w+') as f:
            f.write(_cntg_cd_result)
    _exe_time = time.strftime('%H hours, %M minutes, and %S seconds', time.gmtime(time.time() - _start_time))
    print(f'The execution time for rpstblastn on all contigs of {_genome_id} is {_exe_time}')
The execution time for rpstblastn on all contigs of 562.2282 is 00 hours, 04 minutes, and 50 seconds

</code>