# Notebook from cseHdz/HMM-Tagger
Path: HMM Tagger.ipynb

# Project: Part of Speech Tagging with Hidden Markov Models
---
### Introduction

Part of speech tagging is the process of determining the syntactic category of a word from the words in its surrounding context. It is often used to help disambiguate natural language phrases because it can be done quickly with high accuracy. Tagging can be used for many NLP tasks like determining correct pronunciation during speech synthesis (for example, _dis_-count as a noun vs dis-_count_ as a verb), for information retrieval, and for word sense disambiguation.

In this notebook, you'll use the [Pomegranate](http://pomegranate.readthedocs.io/) library to build a hidden Markov model for part of speech tagging using a "universal" tagset. Hidden Markov models have been able to achieve [>96% tag accuracy with larger tagsets on realistic text corpora](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf). Hidden Markov models have also been used for speech recognition and speech generation, machine translation, gene recognition for bioinformatics, and human gesture recognition for computer vision, and more.

![](_post-hmm.png)

The notebook already contains some code to get you started. You only need to add some new functionality in the areas indicated to complete the project; you will not need to modify the included code beyond what is requested. Sections that begin with **'IMPLEMENTATION'** in the header indicate that you must provide code in the block that follows. Instructions will be provided for each section, and the specifics of the implementation are marked in the code block with a 'TODO' statement. Please be sure to read the instructions carefully!_____no_output_____

**Note:** Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You must then **export the notebook** by running the last cell in the notebook, or by using the menu above and navigating to **File -> Download as -> HTML (.html)** Your submissions should include both the `html` and `ipynb` files.

_____no_output_____

**Note:** Code and Markdown cells can be executed using the `Shift + Enter` keyboard shortcut. Markdown cells can be edited by double-clicking the cell to enter edit mode.

_____no_output_____### The Road Ahead
You must complete Steps 1-3 below to pass the project. The section on Step 4 includes references & resources you can use to further explore HMM taggers.

- [Step 1](#Step-1:-Read-and-preprocess-the-dataset): Review the provided interface to load and access the text corpus
- [Step 2](#Step-2:-Build-a-Most-Frequent-Class-tagger): Build a Most Frequent Class tagger to use as a baseline
- [Step 3](#Step-3:-Build-an-HMM-tagger): Build an HMM Part of Speech tagger and compare to the MFC baseline
- [Step 4](#Step-4:-[Optional]-Improving-model-performance): (Optional) Improve the HMM tagger_____no_output_____

# Jupyter "magic methods" -- only need to be run once per kernel restart
%load_ext autoreload
%aimport helpers, tests
%autoreload 1_____no_output_____# import python modules -- this cell needs to be run again if you make changes to any of the files
import matplotlib.pyplot as plt
import numpy as np

from IPython.core.display import HTML
from itertools import chain
from collections import Counter, defaultdict
from helpers import show_model, Dataset
from pomegranate import State, HiddenMarkovModel, DiscreteDistribution_____no_output_____

## Step 1: Read and preprocess the dataset
---
We'll start by reading in a text corpus and splitting it into a training and testing dataset. The data set is a copy of the [Brown corpus](https://en.wikipedia.org/wiki/Brown_Corpus) (originally from the [NLTK](https://www.nltk.org/) library) that has already been pre-processed to only include the [universal tagset](https://arxiv.org/pdf/1104.2086.pdf). You should expect to get slightly higher accuracy using this simplified tagset than the same model would achieve on a larger tagset like the full [Penn treebank tagset](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html), but the process you'll follow would be the same.

The `Dataset` class provided in helpers.py will read and parse the corpus. You can generate your own datasets compatible with the reader by writing them to the following format. The dataset is stored in plaintext as a collection of words and corresponding tags. Each sentence starts with a unique identifier on the first line, followed by one tab-separated word/tag pair on each following line. Sentences are separated by a single blank line.

Example from the Brown corpus.
```
b100-38532
Perhaps	ADV
it	PRON
was	VERB
right	ADJ
;	.
;	.

b100-35577
...
```_____no_output_____

data = Dataset("tags-universal.txt", "brown-universal.txt", train_test_split=0.8)

print("There are {} sentences in the corpus.".format(len(data)))
print("There are {} sentences in the training set.".format(len(data.training_set)))
print("There are {} sentences in the testing set.".format(len(data.testing_set)))

assert len(data) == len(data.training_set) + len(data.testing_set), \
       "The number of sentences in the training set + testing set should sum to the number of sentences in the corpus"There are 57340 sentences in the corpus.
There are 45872 sentences in the training set.
There are 11468 sentences in the testing set.

### The Dataset Interface

You can access (mostly) immutable references to the dataset through a simple interface provided through the `Dataset` class, which represents an iterable collection of sentences along with easy access to partitions of the data for training & testing. Review the reference below, then run and review the next few cells to make sure you understand the interface before moving on to the next step.

```
Dataset-only Attributes:
    training_set - reference to a Subset object containing the samples for training
    testing_set - reference to a Subset object containing the samples for testing

Dataset & Subset Attributes:
    sentences - a dictionary with an entry {sentence_key: Sentence()} for each sentence in the corpus
    keys - an immutable ordered (not sorted) collection of the sentence_keys for the corpus
    vocab - an immutable collection of the unique words in the corpus
    tagset - an immutable collection of the unique tags in the corpus
    X - returns an array of words grouped by sentences ((w11, w12, w13, ...), (w21, w22, w23, ...), ...)
    Y - returns an array of tags grouped by sentences ((t11, t12, t13, ...), (t21, t22, t23, ...), ...)
    N - returns the number of distinct samples (individual words or tags) in the dataset

Methods:
    stream() - returns an flat iterable over all (word, tag) pairs across all sentences in the corpus
    __iter__() - returns an iterable over the data as (sentence_key, Sentence()) pairs
    __len__() - returns the nubmer of sentences in the dataset
```

For example, consider a Subset, `subset`, of the sentences `{"s0": Sentence(("See", "Spot", "run"), ("VERB", "NOUN", "VERB")), "s1": Sentence(("Spot", "ran"), ("NOUN", "VERB"))}`. The subset will have these attributes:

```
subset.keys == {"s1", "s0"}  # unordered
subset.vocab == {"See", "run", "ran", "Spot"}  # unordered
subset.tagset == {"VERB", "NOUN"}  # unordered
subset.X == (("Spot", "ran"), ("See", "Spot", "run"))  # order matches .keys
subset.Y == (("NOUN", "VERB"), ("VERB", "NOUN", "VERB"))  # order matches .keys
subset.N == 7  # there are a total of seven observations over all sentences
len(subset) == 2  # because there are two sentences
```

**Note:** The `Dataset` class is _convenient_, but it is **not** efficient. It is not suitable for huge datasets because it stores multiple redundant copies of the same data.

_____no_output_____#### Sentences

`Dataset.sentences` is a dictionary of all sentences in the training corpus, each keyed to a unique sentence identifier. Each `Sentence` is itself an object with two attributes: a tuple of the words in the sentence named `words` and a tuple of the tag corresponding to each word named `tags`._____no_output_____

key = 'b100-38532'
print("Sentence: {}".format(key))
print("words:\n\t{!s}".format(data.sentences[key].words))
print("tags:\n\t{!s}".format(data.sentences[key].tags))Sentence: b100-38532
words:
	('Perhaps', 'it', 'was', 'right', ';', ';')
tags:
	('ADV', 'PRON', 'VERB', 'ADJ', '.', '.')

**Note:** The underlying iterable sequence is **unordered** over the sentences in the corpus; it is not guaranteed to return the sentences in a consistent order between calls. Use `Dataset.stream()`, `Dataset.keys`, `Dataset.X`, or `Dataset.Y` attributes if you need ordered access to the data.

#### Counting Unique Elements

You can access the list of unique words (the dataset vocabulary) via `Dataset.vocab` and the unique list of tags via `Dataset.tagset`._____no_output_____

print("There are a total of {} samples of {} unique words in the corpus."
      .format(data.N, len(data.vocab)))
print("There are {} samples of {} unique words in the training set."
      .format(data.training_set.N, len(data.training_set.vocab)))
print("There are {} samples of {} unique words in the testing set."
      .format(data.testing_set.N, len(data.testing_set.vocab)))
print("There are {} words in the test set that are missing in the training set."
      .format(len(data.testing_set.vocab - data.training_set.vocab)))

assert data.N == data.training_set.N + data.testing_set.N, \
       "The number of training + test samples should sum to the total number of samples"There are a total of 1161192 samples of 56057 unique words in the corpus.
There are 928458 samples of 50536 unique words in the training set.
There are 232734 samples of 25112 unique words in the testing set.
There are 5521 words in the test set that are missing in the training set.

#### Accessing word and tag Sequences
The `Dataset.X` and `Dataset.Y` attributes provide access to ordered collections of matching word and tag sequences for each sentence in the dataset._____no_output_____

# accessing words with Dataset.X and tags with Dataset.Y
for i in range(2):
    print("Sentence {}:".format(i + 1), data.X[i])
    print()
    print("Labels {}:".format(i + 1), data.Y[i])
    print()Sentence 1: ('Mr.', 'Podger', 'had', 'thanked', 'him', 'gravely', ',', 'and', 'now', 'he', 'made', 'use', 'of', 'the', 'advice', '.')

Labels 1: ('NOUN', 'NOUN', 'VERB', 'VERB', 'PRON', 'ADV', '.', 'CONJ', 'ADV', 'PRON', 'VERB', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence 2: ('But', 'there', 'seemed', 'to', 'be', 'some', 'difference', 'of', 'opinion', 'as', 'to', 'how', 'far', 'the', 'board', 'should', 'go', ',', 'and', 'whose', 'advice', 'it', 'should', 'follow', '.')

Labels 2: ('CONJ', 'PRT', 'VERB', 'PRT', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'ADP', 'ADV', 'ADV', 'DET', 'NOUN', 'VERB', 'VERB', '.', 'CONJ', 'DET', 'NOUN', 'PRON', 'VERB', 'VERB', '.')

#### Accessing (word, tag) Samples
The `Dataset.stream()` method returns an iterator that chains together every pair of (word, tag) entries across all sentences in the entire corpus._____no_output_____

# use Dataset.stream() (word, tag) samples for the entire corpus
print("\nStream (word, tag) pairs:\n")
for i, pair in enumerate(data.stream()):
    print("\t", pair)
    if i > 5: break
Stream (word, tag) pairs:

	 ('Mr.', 'NOUN')
	 ('Podger', 'NOUN')
	 ('had', 'VERB')
	 ('thanked', 'VERB')
	 ('him', 'PRON')
	 ('gravely', 'ADV')
	 (',', '.')

For both our baseline tagger and the HMM model we'll build, we need to estimate the frequency of tags & words from the frequency counts of observations in the training corpus. In the next several cells you will complete functions to compute the counts of several sets of counts. _____no_output_____## Step 2: Build a Most Frequent Class tagger
---

Perhaps the simplest tagger (and a good baseline for tagger performance) is to simply choose the tag most frequently assigned to each word. This "most frequent class" tagger inspects each observed word in the sequence and assigns it the label that was most often assigned to that word in the corpus._____no_output_____### IMPLEMENTATION: Pair Counts

Complete the function below that computes the joint frequency counts for two input sequences._____no_output_____

def pair_counts(sequences_A, sequences_B):
    """Return a dictionary keyed to each unique value in the first sequence list
    that counts the number of occurrences of the corresponding value from the
    second sequences list.

    For example, if sequences_A is tags and sequences_B is the corresponding
    words, then if 1244 sequences contain the word "time" tagged as a NOUN, then
    you should return a dictionary such that pair_counts[NOUN][time] == 1244
    """
    d = {}
    for tag, word in zip(sequences_A, sequences_B):
        if tag not in d.keys(): d[tag] = {}
        if word not in d[tag].keys(): d[tag][word] = 0
        d[tag][word] +=1
    return d

# Calculate C(t_i, w_i)
# Unwrap the data stream, split it to two lists through zip, reverse the order, split again
emission_counts = pair_counts(*list(zip(*data.training_set.stream()))[::-1])

assert len(emission_counts) == 12, \
       "Uh oh. There should be 12 tags in your dictionary."
assert max(emission_counts["NOUN"], key=emission_counts["NOUN"].get) == 'time', \
       "Hmmm...'time' is expected to be the most common NOUN."
HTML('
Your emission counts look good!
')_____no_output_____

### IMPLEMENTATION: Most Frequent Class Tagger

Use the `pair_counts()` function and the training dataset to find the most frequent class label for each word in the training data, and populate the `mfc_table` below. The table keys should be words, and the values should be the appropriate tag string.

The `MFCTagger` class is provided to mock the interface of Pomegranite HMM models so that they can be used interchangeably._____no_output_____

# Create a lookup table mfc_table where mfc_table[word] contains the tag label most frequently assigned to that word
from collections import namedtuple

FakeState = namedtuple("FakeState", "name")

class MFCTagger:
    # NOTE: You should not need to modify this class or any of its methods
    missing = FakeState(name="<MISSING>")

    def __init__(self, table):
        self.table = defaultdict(lambda: MFCTagger.missing)
        self.table.update({word: FakeState(name=tag) for word, tag in table.items()})

    def viterbi(self, seq):
        """This method simplifies predictions by matching the Pomegranate viterbi() interface"""
        return 0., list(enumerate(["<start>"] + [self.table[w] for w in seq] + ["<end>"]))

# TODO: calculate the frequency of each tag being assigned to each word (hint: similar, but not
# the same as the emission probabilities) and use it to fill the mfc_table

# Unwrap the data stream, split it to two lists through zip, {word: {tag : count}}
word_counts = pair_counts(*zip(*data.training_set.stream()))

# Using same pattern as the emission test for time
mfc_table = {w: max(word_counts[w], key=word_counts[w].get) for w in word_counts.keys()}

# DO NOT MODIFY BELOW THIS LINE
mfc_model = MFCTagger(mfc_table) # Create a Most Frequent Class tagger instance

assert len(mfc_table) == len(data.training_set.vocab), ""
assert all(k in data.training_set.vocab for k in mfc_table.keys()), ""
assert sum(int(k not in mfc_table) for k in data.testing_set.vocab) == 5521, ""
HTML('
Your MFC tagger has all the correct words!
')_____no_output_____

### Making Predictions with a Model
The helper functions provided below interface with Pomegranate network models & the mocked MFCTagger to take advantage of the [missing value](http://pomegranate.readthedocs.io/en/latest/nan.html) functionality in Pomegranate through a simple sequence decoding function. Run these functions, then run the next cell to see some of the predictions made by the MFC tagger._____no_output_____

def replace_unknown(sequence):
    """Return a copy of the input sequence where each unknown word is replaced
    by the literal string value 'nan'. Pomegranate will ignore these values
    during computation.
    """
    return [w if w in data.training_set.vocab else 'nan' for w in sequence]

def simplify_decoding(X, model):
    """X should be a 1-D sequence of observations for the model to predict"""
    _, state_path = model.viterbi(replace_unknown(X))
    return [state[1].name for state in state_path[1:-1]]  # do not show the start/end state predictions_____no_output_____

### Example Decoding Sequences with MFC Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, mfc_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', '<MISSING>', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADV', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

### Evaluating Model Accuracy

The function below will evaluate the accuracy of the MFC tagger on the collection of all sentences from a text corpus. _____no_output_____

def accuracy(X, Y, model):
    """Calculate the prediction accuracy by using the model to decode each sequence
    in the input X and comparing the prediction with the true labels in Y.

    The X should be an array whose first dimension is the number of sentences to test,
    and each element of the array should be an iterable of the words in the sequence.
    The arrays X and Y should have the exact same shape.

    X = [("See", "Spot", "run"), ("Run", "Spot", "run", "fast"), ...]
    Y = [(), (), ...]
    """
    correct = total_predictions = 0
    for observations, actual_tags in zip(X, Y):

        # The model.viterbi call in simplify_decoding will return None if the HMM
        # raises an error (for example, if a test sentence contains a word that
        # is out of vocabulary for the training set). Any exception counts the
        # full sentence as an error (which makes this a conservative estimate).
        try:
            most_likely_tags = simplify_decoding(observations, model)
            correct += sum(p == t for p, t in zip(most_likely_tags, actual_tags))
        except:
            pass
        total_predictions += len(observations)
    return correct / total_predictions_____no_output_____

#### Evaluate the accuracy of the MFC tagger
Run the next cell to evaluate the accuracy of the tagger on the training and test corpus._____no_output_____

mfc_training_acc = accuracy(data.training_set.X, data.training_set.Y, mfc_model)
print("training accuracy mfc_model: {:.2f}%".format(100 * mfc_training_acc))

mfc_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, mfc_model)
print("testing accuracy mfc_model: {:.2f}%".format(100 * mfc_testing_acc))

assert mfc_training_acc >= 0.955, "Uh oh. Your MFC accuracy on the training set doesn't look right."
assert mfc_testing_acc >= 0.925, "Uh oh. Your MFC accuracy on the testing set doesn't look right."
HTML('
Your MFC tagger accuracy looks correct!
')training accuracy mfc_model: 95.72%
testing accuracy mfc_model: 93.01%

## Step 3: Build an HMM tagger
---
The HMM tagger has one hidden state for each possible tag, and parameterized by two distributions: the emission probabilties giving the conditional probability of observing a given **word** from each hidden state, and the transition probabilities giving the conditional probability of moving between **tags** during the sequence.

We will also estimate the starting probability distribution (the probability of each **tag** being the first tag in a sequence), and the terminal probability distribution (the probability of each **tag** being the last tag in a sequence).

The maximum likelihood estimate of these distributions can be calculated from the frequency counts as described in the following sections where you'll implement functions to count the frequencies, and finally build the model. The HMM model will make predictions according to the formula:

$$t_i^n = \underset{t_i^n}{\mathrm{argmax}} \prod_{i=1}^n P(w_i|t_i) P(t_i|t_{i-1})$$

Refer to Speech & Language Processing [Chapter 10](https://web.stanford.edu/~jurafsky/slp3/10.pdf) for more information._____no_output_____### IMPLEMENTATION: Unigram Counts

Complete the function below to estimate the co-occurrence frequency of each symbol over all of the input sequences. The unigram probabilities in our HMM model are estimated from the formula below, where N is the total number of samples in the input. (You only need to compute the counts for now.)

$$P(tag_1) = \frac{C(tag_1)}{N}$$_____no_output_____

def unigram_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequence list that
    counts the number of occurrences of the value in the sequences list. The sequences
    collection should be a 2-dimensional array.

    For example, if the tag NOUN appears 275558 times over all the input sequences,
    then you should return a dictionary such that your_unigram_counts[NOUN] == 275558.
    """
    # TODO: Finish this function!
    return Counter(list(sequences)[1])

# TODO: call unigram_counts with a list of tag sequences from the training set
# Send a two-dimensional array before (don't unwrap as before)
tag_unigrams = unigram_counts(zip(*data.training_set.stream()))

assert set(tag_unigrams.keys()) == data.training_set.tagset, \
       "Uh oh. It looks like your tag counts doesn't include all the tags!"
assert min(tag_unigrams, key=tag_unigrams.get) == 'X', \
       "Hmmm...'X' is expected to be the least common class"
assert max(tag_unigrams, key=tag_unigrams.get) == 'NOUN', \
       "Hmmm...'NOUN' is expected to be the most common class"
HTML('
Your tag unigrams look good!
')_____no_output_____

### IMPLEMENTATION: Bigram Counts

Complete the function below to estimate the co-occurrence frequency of each pair of symbols in each of the input sequences. These counts are used in the HMM model to estimate the bigram probability of two tags from the frequency counts according to the formula: $$P(tag_2|tag_1) = \frac{C(tag_2|tag_1)}{C(tag_2)}$$
_____no_output_____

def bigram_counts(sequences):
    """Return a dictionary keyed to each unique PAIR of values in the input sequences
    list that counts the number of occurrences of pair in the sequences list. The input
    should be a 2-dimensional array.

    For example, if the pair of tags (NOUN, VERB) appear 61582 times, then you should
    return a dictionary such that your_bigram_counts[(NOUN, VERB)] == 61582
    """
    d = {}
    for seq in sequences:
        # Stop before the last word
        for i in range(0, len(seq)-1):
            if (seq[i], seq[i+1]) not in d.keys(): d[(seq[i], seq[i+1])] = 0
            d[(seq[i], seq[i+1])] +=1

    return d

# TODO: call bigram_counts with a list of tag sequences from the training set
tag_bigrams = bigram_counts(data.training_set.Y)

assert len(tag_bigrams) == 144, \
       "Uh oh. There should be 144 pairs of bigrams (12 tags x 12 tags)"
assert min(tag_bigrams, key=tag_bigrams.get) in [('X', 'NUM'), ('PRON', 'X')], \
       "Hmmm...The least common bigram should be one of ('X', 'NUM') or ('PRON', 'X')."
assert max(tag_bigrams, key=tag_bigrams.get) in [('DET', 'NOUN')], \
       "Hmmm...('DET', 'NOUN') is expected to be the most common bigram."
HTML('
Your tag bigrams look good!
')_____no_output_____

### IMPLEMENTATION: Sequence Starting Counts
Complete the code below to estimate the bigram probabilities of a sequence starting with each tag._____no_output_____

def starting_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the beginning of
    a sequence.

    For example, if 8093 sequences start with NOUN, then you should return a
    dictionary such that your_starting_counts[NOUN] == 8093
    """
    d = {}
    for seq in sequences:
        if seq[0] not in d.keys(): d[seq[0]] = 0
        d[seq[0]] +=1

    return d

# TODO: Calculate the count of each tag starting a sequence
tag_starts = starting_counts(data.training_set.Y)

assert len(tag_starts) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_starts, key=tag_starts.get) == 'X', "Hmmm...'X' is expected to be the least common starting bigram."
assert max(tag_starts, key=tag_starts.get) == 'DET', "Hmmm...'DET' is expected to be the most common starting bigram."
HTML('
Your starting tag counts look good!
')_____no_output_____

### IMPLEMENTATION: Sequence Ending Counts
Complete the function below to estimate the bigram probabilities of a sequence ending with each tag._____no_output_____

def ending_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the end of
    a sequence.

    For example, if 18 sequences end with DET, then you should return a
    dictionary such that your_starting_counts[DET] == 18
    """
    d = {}

    for seq in sequences:
        n = len(seq) - 1
        if seq[n] not in d.keys(): d[seq[n]] = 0
        d[seq[n]] +=1

    return d

# TODO: Calculate the count of each tag ending a sequence
tag_ends = ending_counts(data.training_set.Y)

assert len(tag_ends) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_ends, key=tag_ends.get) in ['X', 'CONJ'], "Hmmm...'X' or 'CONJ' should be the least common ending bigram."
assert max(tag_ends, key=tag_ends.get) == '.', "Hmmm...'.' is expected to be the most common ending bigram."
HTML('
Your ending tag counts look good!
')_____no_output_____

### IMPLEMENTATION: Basic HMM Tagger
Use the tag unigrams and bigrams calculated above to construct a hidden Markov tagger.

- Add one state per tag
    - The emission distribution at each state should be estimated with the formula: $P(w|t) = \frac{C(t, w)}{C(t)}$
- Add an edge from the starting state `basic_model.start` to each tag
    - The transition probability should be estimated with the formula: $P(t|start) = \frac{C(start, t)}{C(start)}$
- Add an edge from each tag to the end state `basic_model.end`
    - The transition probability should be estimated with the formula: $P(end|t) = \frac{C(t, end)}{C(t)}$
- Add an edge between _every_ pair of tags
    - The transition probability should be estimated with the formula: $P(t_2|t_1) = \frac{C(t_1, t_2)}{C(t_1)}$_____no_output_____

basic_model = HiddenMarkovModel(name="base-hmm-tagger")

# TODO: create states with emission probability distributions P(word | tag) and add to the model
# (Hint: you may need to loop & create/add new states)

s = {}
states = []
for tag in emission_counts.keys():
    tag_count = tag_unigrams[tag]
    p = {}
    for word in emission_counts[tag]:
        p[word] = emission_counts[tag][word]/tag_count
    emission_p = DiscreteDistribution(p)
    state = State(emission_p,name=""+tag)
    basic_model.add_state(state)
    s[tag] = state

# TODO: add edges between states for the observed transition frequencies P(tag_i | tag_i-1)
# (Hint: you may need to loop & add transitions

# Start & End Transitions
# Start - Number of senteces starting with tag over total number of sentences
# End - Number of senteces ending with tag over count of tag appereances
for tag in tag_starts:
    basic_model.add_transition(basic_model.start, s[tag], tag_starts[tag]/len(data.training_set.Y))
    basic_model.add_transition(s[tag], basic_model.end, tag_ends[tag]/tag_unigrams[tag])

for (tag1, tag2) in tag_bigrams:
    basic_model.add_transition(s[tag1], s[tag2], tag_bigrams[(tag1,tag2)]/tag_unigrams[tag1])

# NOTE: YOU SHOULD NOT NEED TO MODIFY ANYTHING BELOW THIS LINE
# finalize the model
basic_model.bake()

assert all(tag in set(s.name for s in basic_model.states) for tag in data.training_set.tagset), \
       "Every state in your network should use the name of the associated tag, which must be one of the training set tags."
assert basic_model.edge_count() == 168, \
       ("Your network should have an edge from the start node to each state, one edge between every " +
        "pair of tags (states), and an edge from each state to the end node.")
HTML('
Your HMM network topology looks good!
')_____no_output_____hmm_training_acc = accuracy(data.training_set.X, data.training_set.Y, basic_model)
print("training accuracy basic hmm model: {:.2f}%".format(100 * hmm_training_acc))

hmm_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, basic_model)
print("testing accuracy basic hmm model: {:.2f}%".format(100 * hmm_testing_acc))

assert hmm_training_acc > 0.97, "Uh oh. Your HMM accuracy on the training set doesn't look right."
assert hmm_training_acc > 0.955, "Uh oh. Your HMM accuracy on the training set doesn't look right."
HTML('
Your HMM tagger accuracy looks correct! Congratulations, you\'ve finished the project.
')training accuracy basic hmm model: 97.54%
testing accuracy basic hmm model: 95.95%

### Example Decoding Sequences with the HMM Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, basic_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

## Finishing the project
---

**Note:** **SAVE YOUR NOTEBOOK**, then run the next cell to generate an HTML copy. You will zip & submit both this file and the HTML copy for review.

_____no_output_____

!!jupyter nbconvert *.ipynb_____no_output_____

## Step 4: [Optional] Improving model performance
---
There are additional enhancements that can be incorporated into your tagger that improve performance on larger tagsets where the data sparsity problem is more significant. The data sparsity problem arises because the same amount of data split over more tags means there will be fewer samples in each tag, and there will be more missing data  tags that have zero occurrences in the data. The techniques in this section are optional.

- [Laplace Smoothing](https://en.wikipedia.org/wiki/Additive_smoothing) (pseudocounts)
    Laplace smoothing is a technique where you add a small, non-zero value to all observed counts to offset for unobserved values.

- Backoff Smoothing
    Another smoothing technique is to interpolate between n-grams for missing data. This method is more effective than Laplace smoothing at combatting the data sparsity problem. Refer to chapters 4, 9, and 10 of the [Speech & Language Processing](https://web.stanford.edu/~jurafsky/slp3/) book for more information.

- Extending to Trigrams
    HMM taggers have achieved better than 96% accuracy on this dataset with the full Penn treebank tagset using an architecture described in [this](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf) paper. Altering your HMM to achieve the same performance would require implementing deleted interpolation (described in the paper), incorporating trigram probabilities in your frequency tables, and re-implementing the Viterbi algorithm to consider three consecutive states instead of two.

### Obtain the Brown Corpus with a Larger Tagset
Run the code below to download a copy of the brown corpus with the full NLTK tagset. You will need to research the available tagset information in the NLTK docs and determine the best way to extract the subset of NLTK tags you want to explore. If you write the following the format specified in Step 1, then you can reload the data using all of the code above for comparison.

Refer to [Chapter 5](http://www.nltk.org/book/ch05.html) of the NLTK book for more information on the available tagsets._____no_output_____

import nltk
from nltk import pos_tag, word_tokenize
from nltk.corpus import brown

nltk.download('brown')
training_corpus = nltk.corpus.brown
training_corpus.tagged_sents()[0][nltk_data] Downloading package brown to
[nltk_data]     /Users/carloshernandez/nltk_data...
[nltk_data]   Unzipping corpora/brown.zip.


# Notebook from bokulich-lab/db-benchmarks-2020
Path: notebooks/unite-test.ipynb

# UNITE database Benchmark

This notebook tests the effects of different database versions on taxonomic information and classification accuracy, using the UNITE fungal ITS reference database._____no_output_____

import os
import qiime2 as q2
from qiime2.plugins import rescript, feature_table as ft, taxa as q2t
import pandas as pd
from glob import glob
import timeit
from rescript.cross_validate import _check_time
import tempfile
from glob import glob
import seaborn as sns
from matplotlib import pyplot as plt

pd.set_option('max_colwidth', 0)

%matplotlib inline_____no_output_____

# Define inputs

Modify the filepaths here to your local filepaths to run this notebook.
The filepaths below merely point to the sequence and taxonomy files for the UNITE Feb 2020 release, clustered at different OTU % similarities and versions.

UNITE files can be downloaded and unzipped from here:

* https://plutof.ut.ee/#/datacite/10.15156/BIO/786385
* https://plutof.ut.ee/#/datacite/10.15156/BIO/786386

**DO NOT MODIFY ANY OTHER CELLS IN THIS NOTEBOOK IF YOU ARE ATTEMPTING TO REPLICATE THESE RESULTS**_____no_output_____

# N jobs to use
n_jobs = 2
reads_per_batch = 5000

# location of reference database files
dbdir = '../../../ref_dbs/unite_02022019/'
alldir = '../../../ref_dbs/sh_qiime_release_s_all_04.02.2020/'
dbs = {
    'unite_{0}'.format(n): {
        'seqs': q2.Artifact.import_data(
            'FeatureData[Sequence]', dbdir + 'sh_refs_qiime_ver8_{0}_02.02.2019.fasta'.format(n)),
        'taxa': q2.Artifact.import_data(
            'FeatureData[Taxonomy]', dbdir + 'sh_taxonomy_qiime_ver8_{0}_02.02.2019.txt'.format(n),
            view_type='HeaderlessTSVTaxonomyFormat')}
    for n in ['97', '99', 'dynamic']}
dbs.update({
    'unite_2020_all_{0}'.format(n): {
        'seqs': q2.Artifact.import_data(
            'FeatureData[Sequence]', alldir + 'sh_refs_qiime_ver8_{0}_s_all_04.02.2020.fasta'.format(n)),
        'taxa': q2.Artifact.import_data(
            'FeatureData[Taxonomy]', alldir + 'sh_taxonomy_qiime_ver8_{0}_s_all_04.02.2020.txt'.format(n),
            view_type='HeaderlessTSVTaxonomyFormat')}
    for n in ['97', '99', 'dynamic']})_____no_output_____# drop singleton kingdoms
exclude = 'k__Bacteria,k__Chromista,k__Filasteriae,k__Nucleariae,k__Telonemae'
for n in ['97', '99', 'dynamic']:
    clean_taxa = dbs['unite_2020_all_' + n]['taxa'].view(pd.DataFrame)
    clean_taxa = clean_taxa[~clean_taxa.Taxon.str.contains(exclude.replace(',', '|'))]
    filtered_sequences, = q2t.actions.filter_seqs(
            sequences=dbs['unite_2020_all_' + n]['seqs'],
            taxonomy=dbs['unite_2020_all_' + n]['taxa'],
            exclude=exclude)
    dbs['unite_2020_all_{0}'.format(n)] = {
        'seqs': filtered_sequences,
        'taxa': q2.Artifact.import_data('FeatureData[Taxonomy]', clean_taxa)}
# fungi only
include = 'k__Fungi'
for n in ['97', '99', 'dynamic']:
    clean_taxa = dbs['unite_2020_all_' + n]['taxa'].view(pd.DataFrame)
    clean_taxa = clean_taxa[clean_taxa.Taxon.str.contains(include.replace(',', '|'))]
    filtered_sequences, = q2t.actions.filter_seqs(
            sequences=dbs['unite_2020_all_' + n]['seqs'],
            taxonomy=dbs['unite_2020_all_' + n]['taxa'],
            include=include)
    dbs['unite_2020_all_fungi_{0}'.format(n)] = {
        'seqs': filtered_sequences,
        'taxa': q2.Artifact.import_data('FeatureData[Taxonomy]', clean_taxa)}
# exclude anything not identified to order level
exclude = 'o__unidentified'
for n in ['97', '99', 'dynamic']:
    clean_taxa = dbs['unite_2020_all_fungi_' + n]['taxa'].view(pd.DataFrame)
    clean_taxa = clean_taxa[~clean_taxa.Taxon.str.contains(exclude)]
    filtered_sequences, = ft.actions.filter_seqs(
            data=dbs['unite_2020_all_fungi_' + n]['seqs'],
            metadata=q2.Metadata(clean_taxa))
    dbs['unite_2020_clean_{0}'.format(n)] = {
        'seqs': filtered_sequences,
        'taxa': q2.Artifact.import_data('FeatureData[Taxonomy]', clean_taxa)}
_____no_output_____

# Perform CV Classification
Both k-fold and "perfect classifiers" (no CV, leaky data, best possible performance)_____no_output_____

# Cross-validation
times_cv = {}
for n, v in dbs.items():
    s, t = v['seqs'], v['taxa']
    print(n)
    exp_taxa_fp = '../data/expected/{0}_expected_taxonomy.qza'.format(n)
    if not os.path.exists(exp_taxa_fp):
        start = timeit.default_timer()
        exp, obs, = rescript.actions.evaluate_cross_validate(
            s, t, n_jobs=n_jobs, reads_per_batch=reads_per_batch)
        times_cv[n] = timeit.default_timer() - start
        exp.save(exp_taxa_fp)
        obs.save('../data/observed/{0}_observed_taxonomy.qza'.format(n))unite_97
unite_99
unite_dynamic
unite_2020_all_97
unite_2020_all_99
unite_2020_all_dynamic
unite_2020_clean_97
unite_2020_clean_99
unite_2020_clean_dynamic
unite_2020_all_fungi_97
Validation: 4.32s
# "Perfect" classifier, no cross-validation
times_nocv = {}
for n, v in dbs.items():
    if n not in ['unite_2020_all_97', 'unite_2020_all_99', 'unite_2020_all_dynamic']:
        s, t = v['seqs'], v['taxa']
        print(n)
        classifier_fp = '../data/classifiers/{0}_nb_classifier.qza'.format(n)
        obs_taxa_fp = '../data/observed/{0}_observed_taxonomy_noCV.qza'.format(n)
        exp_taxa_fp = '../data/expected/{0}_expected_taxonomy_noCV.qza'.format(n)
        if not os.path.exists(exp_taxa_fp):
            start = timeit.default_timer()
            classifier, _, obs, = rescript.actions.evaluate_fit_classifier(
                s, t, n_jobs=n_jobs, reads_per_batch=reads_per_batch)
            times_nocv[n] = timeit.default_timer() - start
            classifier.save(classifier_fp)
            obs.save(obs_taxa_fp)
            t.save(exp_taxa_fp)
unite_97
unite_99
unite_dynamic
unite_2020_all_fungi_97
Validation: 4.61s
exp_taxonomies = {os.path.splitext(os.path.basename(fp))[0].replace('_expected_taxonomy', ''):
                  q2.Artifact.load(fp) for fp in glob('../data/expected/unite_*')}
obs_taxonomies = {os.path.splitext(os.path.basename(fp))[0].replace('_observed_taxonomy', ''):
                  q2.Artifact.load(fp) for fp in glob('../data/observed/unite_*')}
_____no_output_____

# Evaluate Classification Accuracy_____no_output_____

class_eval, = rescript.actions.evaluate_classifications(
    expected_taxonomies = [exp_taxonomies[i] for i in sorted(exp_taxonomies.keys())],
    observed_taxonomies = [obs_taxonomies[i] for i in sorted(obs_taxonomies.keys())],
    labels = sorted(obs_taxonomies.keys()))
class_eval.save('../data/eval/unite_fungi_classification_eval.qzv')_____no_output_____

# Evaluate Taxonomic Information_____no_output_____

tax_eval, = rescript.actions.evaluate_taxonomy(
    taxonomies = [dbs[i]['taxa'] for i in sorted(dbs.keys())],
    labels = sorted(dbs.keys()))
tax_eval.save('../data/eval/unite_fungi_taxonomic_eval.qzv')
tax_eval, = rescript.actions.evaluate_taxonomy(
    taxonomies = [dbs[i]['taxa'] for i in sorted(dbs.keys())],
    labels = sorted(dbs.keys()),
    rank_handle_regex="^[dkpcofgs]__unidentified")
tax_eval.save('../data/eval/unite_fungi_taxonomic_eval_dropemptyranks.qzv')_____no_output_____

# Evaluate Sequence Information_____no_output_____

relabels = {
    'unite_2020_all_97': 'All Euks 97%',
    'unite_2020_all_97_noCV': 'All Euks 97% No CV',
    'unite_2020_all_99': 'All Euks 99%',
    'unite_2020_all_dynamic': 'All Euks Dynamic',
    'unite_2020_all_fungi_97': 'Fungi 97%',
    'unite_2020_all_fungi_97_noCV': 'Fungi 97% No CV',
    'unite_2020_all_fungi_99': 'Fungi 99%',
    'unite_2020_all_fungi_99_noCV': 'Fungi 99% No CV',
    'unite_2020_all_fungi_dynamic': 'Fungi Dynamic',
    'unite_2020_all_fungi_dynamic_noCV': 'Fungi Dynamic No CV',
    'unite_2020_clean_97': 'Fungi Order 97%',
    'unite_2020_clean_97_noCV': 'Fungi Order 97% No CV',
    'unite_2020_clean_99': 'Fungi Order 99%',
    'unite_2020_clean_99_noCV': 'Fungi Order 99% No CV',
    'unite_2020_clean_dynamic': 'Fungi Order Dynamic',
    'unite_2020_clean_dynamic_noCV': 'Fungi Order Dynamic No CV',
    'unite_97': 'drop',
    'unite_97_noCV': 'drop',
    'unite_99': 'drop',
    'unite_99_noCV': 'drop',
    'unite_dynamic': 'drop',
    'unite_dynamic_noCV': 'drop'}
cv_labels = [i for i in relabels.values() if 'CV' not in i and i != 'drop']
nocv_labels = [i for i in relabels.values() if 'CV' in i]
_____no_output_____seqs_eval, = rescript.actions.evaluate_seqs(
    sequences = [dbs[i]['seqs'] for i in sorted(dbs.keys()) if relabels[i] != 'drop'],
    labels = [relabels[i] for i in sorted(dbs.keys()) if relabels[i] != 'drop'],
    palette = 'cividis')
seqs_eval.save('../data/eval/unite_fungi_seqs_eval.qzv')
_____no_output_____

# Make a pretty figure

The interactive figs are great, but we need to generate a static figure for publication :(_____no_output_____

fig_data = dict()
with tempfile.TemporaryDirectory() as temp_dir_name:
    for f in glob('../data/eval/unite_*.qzv'):
        name = f.split('/')[-1].split('.qzv')[0]
        f = q2.Visualization.load(f)
        f.export_data(temp_dir_name)
        fig_data[name] = q2.Metadata.load(temp_dir_name + '/data.tsv').to_dataframe()
_____no_output_____plot_keys = ['unite_fungi_taxonomic_eval',
             'unite_fungi_taxonomic_eval',
             'unite_fungi_taxonomic_eval_dropemptyranks',
             'unite_fungi_classification_eval',
             'unite_fungi_classification_eval']
signals = ['Unique Labels',
           'Taxonomic Entropy',
           'Proportion of Features Terminating at Depth',
           'F-Measure',
           'F-Measure']
labels = list('ABCDEF')

fig, axes = plt.subplots(2, 3, figsize=(12, 6), sharex=False)
axes = axes.flatten()
axes[-1].remove()
for ax, key, signal, label in zip(axes, plot_keys, signals, labels):
    ax.text(-0.1, 1.1, label, transform=ax.transAxes,
            fontsize=16, fontweight='bold', va='top', ha='right')
    dat = fig_data[key].copy()
    dat.loc[:, 'Dataset'] = dat.loc[:, 'Dataset'].replace(relabels)
    dat = dat[dat['Dataset'] != 'drop']
    dat['Level'] = dat['Level'].replace({
        1: 'K', 2: 'P', 3: 'C', 4: 'O',
        5: 'F', 6: 'G', 7: 'S'})
    if ax == axes[3]:
        dat = dat[dat['Dataset'].isin(nocv_labels)]
        # create empty rows for consistent colors/label ordering
        # did not run All Euks 99% or dynamic with no CV
        # due to high runtime and similarity to 97% results in other tests
        dat.loc['998', 'Dataset'] = 'All Euks 99% No CV'
        dat.loc['999', 'Dataset'] = 'All Euks Dynamic No CV'
    if ax == axes[4]:
        dat = dat[dat['Dataset'].isin(cv_labels)]
        legend = 'brief'
    else:
        legend = False
    g = sns.lineplot(x='Level', y=signal, hue='Dataset', data=dat, ax=ax,
                     legend=legend, palette='cividis', sort=False,
                     hue_order=sorted(dat.Dataset.unique()))
    if ax == axes[2]:
        ax.set_ylabel('Terminal Labels')
    if ax == axes[4]:
        ax.legend(bbox_to_anchor=(1.1, 1.05))
    else:
        ax.set_xlabel('')
fig.savefig('../data/plots/unite_evaluation.pdf', bbox_inches='tight')
_____no_output_____

Fig XXX. Taxonomic information (A-C) and classification accuracy (D-E) of UNITE ITS domain database with different filtering and clustering settings. Filtered versions include the "all Eukaryotes" 2020.04.02 release version containing all Eukaryotes ("All Euks"), filtered to contain only Fungi, and filtered to contain only Fungi with at least order-level taxonomic annotation ("Fungi Order"). Cluster levels indicate which UNITE release version was used: sequences clustered at 97% similarity, 99% similarity, or the UNITE "dynamic" species hypothesis threshold. Subpanels show taxonomic/classification characteristics at each taxonomic level: A, Number of unique taxonomic labels; B, Taxonomic entropy; C, proportion of taxa that terminate at that level; D, optimal classification accuracy (as F-Measure) without cross-validation (simulating best possible classification accuracy when the true label is known but classification accuracy may be confounded by other similar hits in the database); E, classification accuracy (F-Measure) with cross-validation (simulating realistic classification tasks when the correct label is unknown). Level labels on x-axis: K = Kingdom, P = phylum, C = class, O = order, F = family, G = genus, S = species. See Materials and Methods for more details on how these databases were created and processed. From among the three "All Euks" databases tested, only the 97% OTUs were classified without cross-validation (panel D) due to the _____no_output_____


# Notebook from john-s-butler-dit/Decision-Making-Models
Path: WongWang2006 Reduced Two Variable Model without AMPA - A Network Mechanism.ipynb

_____no_output_____# A Recurrent Network Mechanism of Time Integration in Perceptual Decisions
Wong,  Wang
Journal of Neuroscience, January 2006, 26(4):1212-1328

TESTING_____no_output_____# Appendix Model
## Reduced two-variable model without AMPA at recurrent synapses
The reduced two-variable model is in its simplest form if we
assume that there is no AMPA at the recurrent synapses._____no_output_____### Calling Library Fuctions_____no_output_____

# LIBRARY

import numpy as np # vector manipulation
import math  # math functions
import sys

# THIS IS FOR PLOTTING
%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import warnings
warnings.filterwarnings("ignore")
_____no_output_____

### The Reduced Network Model
#### Firing rate
$$ H_i=\frac{a x_i-b}{(1-exp[-d(ax_i-b)]}$$

$$ a=207 (VnC)^{-1},$$
$$ b=108Hz,$$
$$ d=0.154s$$_____no_output_____

def H(x):
    a=270
    b=108
    d=0.154
    f=(a*x-b)/(1-np.exp(-d*(a*x-b)))
    return f

x=np.arange(-1,1,0.0001)
fig = plt.figure(figsize=(8,8))
plt.plot(x,H(x))
plt.grid()
plt.xlabel('x')
plt.ylabel('H(x)')
#plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.show()
_____no_output_____

### Neural Circuit
$$ x_{1}=J_{11}S_1-J_{12}S_2+I_{0}+I_{1}+I_{noise,1}$$
$$ x_{2}=J_{22}S_2-J_{21}S_1+I_{0}+I_{2}+I_{noise,1}$$

where the synaptic couplings are $J_{11}=0.2609$, $J_{22}=0.2609$, $J_{12}=0.0497$ and $J_{21}=0.0497$.
$I_{0}=0.3255 nA$ represents external input
_____no_output_____

def total_synaptic_current(S_1,S_2,I_1,I_2,I_noise_1,I_noise_2):
    # Synaptic coupling
    J_11=0.2609 # nA
    J_22=0.2609 # nA
    J_12=0.0497 # nA
    J_21=0.0497 # nA
    I_0=0.3255  # nA
    x_1=J_11*S_1-J_12*S_2+I_0+I_1+I_noise_1
    x_2=J_22*S_2-J_21*S_1+I_0+I_2+I_noise_2
    return x_1, x_2
_____no_output_____

### Background activity
$$ \tau_{AMPA}\frac{d I_{noise,i}(t)}{dt} =-I_{noise,i}(t)+\eta_i(t)\sqrt{\tau_{AMPA}}\sigma_{noise}$$
$$ \tau_{AMPA}=2ms,$$
$$ \sigma_{noise}=0.02,$$
$$ \eta- white \ noise $$_____no_output_____

def Background_Activity(I_noise,h):
    sigma_noise=0.02 # nA
    tau_AMPA=2#ms
    eta_noise=np.random.normal(0,1,1)
    I_0=0.0#3255  # nA
    I_noise_new=I_noise+(h)/tau_AMPA*(-(I_noise-I_0)+eta_noise*np.sqrt(tau_AMPA)*sigma_noise)
    return I_noise_new_____no_output_____

### Network Dynamics
$$ \frac{d S_{i}}{dt} =\frac{-S_{i}}{\tau_S}+\big((1-S_i)\gamma H(x_i)\big)$$
$$\tau_S=100ms$$
$$\tau_S=0.1s$$
$$\gamma=0.641$$

Euler
w_{i+1}_____no_output_____

def S_diff(S,x,h):
    tau_S=100/1000#100 #ms
    𝛾=0.641
    S_new=S+h/1000*((-S/tau_S)+(1-S)*𝛾*H(x))
    return S_new_____no_output_____

### Input Current Target

$$ I_i=J_{A,ext}\mu_0\left(1\pm \frac{c'}{100} \right) $$

$$\mu=30$$ Hz- spikes per second
$$ f=1$$  Only in Wong et al 2007
$$ J_{A,ext}=5.2 \times 10^{-4}$$
$$ c' $$ Coherence_____no_output_____

def I_input_1(c_dash):
    J_A_ext=5.2/10000# nA/Hz
    mu_0=30 # Hz
    f=1
    I_motion=J_A_ext*mu_0*(1+f*(c_dash)/100)
    return I_motion

All_coh=np.arange(-100,100,1)
fig = plt.figure(figsize=(8,4))
plt.plot(All_coh,I_input_1(All_coh)+0.3255)
plt.grid()
plt.xlabel('Coherence')
plt.ylabel('I(Coherence)')
#plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.show()
fig = plt.figure(figsize=(8,4))
plt.plot(All_coh,H(I_input_1(All_coh)+0.3255))
plt.grid()
plt.xlabel('Coherence')
plt.ylabel('I(Coherence)')
#plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.show()

_____no_output_____

# Initial Conditions_____no_output_____

h=0.1
time=np.arange(-100,3000,h)
J_A_ext=5.2/10000 # nA/Hz
mu_0=30.0
I_noise_1=0.4*np.random.normal(0,1,len(time))
I_noise_2=0.4*np.random.normal(0,1,len(time))

x_1=J_A_ext*mu_0*np.ones(len(time))
x_2=J_A_ext*mu_0*np.ones(len(time))

S=0.3
S_1=S+np.random.uniform(0,0.001,len(time))
S_2=S+np.random.uniform(0,0.001,len(time))

I_1=np.zeros((len(time)))
I_2=np.zeros((len(time)))
_____no_output_____COHERENCE_LEVEL=5

for i in range (0,len(time)-1):
    ## INPUT
    if time[i] >=0 and time[i]<1000:
        c_dash=COHERENCE_LEVEL
    else:
        c_dash=0.0

    I_noise_1[i+1]=Background_Activity(I_noise_1[i],h)
    I_noise_2[i+1]=Background_Activity(I_noise_2[i],h)

    I_1[i+1]=I_input_1(c_dash)
    I_2[i+1]=I_input_1(-c_dash)

    x_1[i+1],x_2[i+1]=total_synaptic_current(S_1[i],S_2[i],I_1[i],
                                      I_2[i],I_noise_1[i],I_noise_2[i])
    S_1[i+1]=S_diff(S_1[i],x_1[i],h)
    S_2[i+1]=S_diff(S_2[i],x_2[i],h)

_____no_output_____

### Plotting
_____no_output_____

fig = plt.figure(figsize=(14,4))
plt.subplot(132)

plt.plot(time,H(S_1),'-',color='blue',label="Right")
plt.plot(time,H(S_2),'-',color='red',label="Left")
plt.hlines(15,-100,1500,linestyles='dashed',label='Threshold')
plt.hlines(5,0,800)

plt.grid()
#plt.xlim((-100,1500))
plt.xlabel('time(ms)')
plt.ylabel('Firing Rate')

#plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)

plt.subplot(133)

plt.plot(time,H(S_1)-H(S_2),'-',color='blue',label="Right-Left")
plt.hlines(15,-100,1500,linestyles='dashed',label='Threshold')
plt.hlines(5,0,800)
plt.grid()
#plt.xlim((-100,1500))
plt.xlabel('time(ms)')
plt.ylabel('Firing Rate')

plt.subplot(131)
plt.plot(time,S_1,'-',color='blue',label='Right Activity')
plt.plot(time,S_2,'-',color='red',label='Left Activity')

#plt.plot([0,1],[0,1],':k')
plt.grid()
plt.xlabel('time(ms)')
plt.ylabel('Activity')
#plt.xlim((-100,1500))

plt.legend()
plt.tight_layout()
plt.show()
_____no_output_____Firing_target_1=H(S_1)
Firing_target_2=H(S_2)

fig = plt.figure(figsize=(6,6))
plt.plot(Firing_target_1,Firing_target_2,'-',color='black')

plt.plot([0,20],[0,20],':')
plt.grid()
plt.xlabel('Firing Rate 1')
plt.ylabel('Firing Rate 2')
plt.title('Phase Plane')
plt.axis('square')
plt.xlim([0,20])
plt.ylim([0,20])

plt.show()

_____no_output_____

## References
Wong KF, Wang XJ. A recurrent network mechanism of time integration in perceptual decisions. J Neurosci. 2006 Jan 25;26(4):1314-28. doi: 10.1523/JNEUROSCI.3733-05.2006. PMID: 16436619; PMCID: PMC6674568._____no_output_____


# Notebook from AstroLudwig/MCPUVS
Path: Tractor/Sandbox/1_DoesTractorWorkInGeneral.ipynb

### Verification that Tractor is running smoothly
_____no_output_____

<code>
import numpy as np
import pylab as plt
from tractor import *

# Size of image, centroid and flux of source
W,H = 25,25
cx,cy = 12.8, 14.3
flux = 12.
# PSF size
psfsigma = 2.
# Per-pixel image noise
noisesigma = 0.01
# Create synthetic Gaussian star image
G = np.exp(((np.arange(W)-cx)[np.newaxis,:]**2 +
            (np.arange(H)-cy)[:,np.newaxis]**2)/(-2.*psfsigma**2))
trueimage = flux * G/G.sum()
image = trueimage + noisesigma * np.random.normal(size=trueimage.shape)

# Create Tractor Image
tim = Image(data=image, invvar=np.ones_like(image) / (noisesigma**2),
            psf=NCircularGaussianPSF([psfsigma], [1.]),
            wcs=NullWCS(), photocal=NullPhotoCal(),
            sky=ConstantSky(0.))

# Create Tractor source with approximate position and flux
src = PointSource(PixPos(W/2., H/2.), Flux(10.))

# Create Tractor object itself
tractor = Tractor([tim], [src])

# Render the model image
mod0 = tractor.getModelImage(0)
chi0 = tractor.getChiImage(0)

# Plots
ima = dict(interpolation='nearest', origin='lower', cmap='gray',
           vmin=-2*noisesigma, vmax=5*noisesigma)
imchi = dict(interpolation='nearest', origin='lower', cmap='gray',
             vmin=-5, vmax=5)
plt.clf()
plt.subplot(2,2,1)
plt.imshow(trueimage, **ima)
plt.title('True image')
plt.subplot(2,2,2)
plt.imshow(image, **ima)
plt.title('Image')
plt.subplot(2,2,3)
plt.imshow(mod0, **ima)
plt.title('Tractor model')
plt.subplot(2,2,4)
plt.imshow(chi0, **imchi)
plt.title('Chi')

# Freeze all image calibration params -- just fit source params
tractor.freezeParam('images')

# Save derivatives for later plotting...
derivs = tractor.getDerivs()

# Take several linearized least squares steps
for i in range(10):
    dlnp,X,alpha = tractor.optimize()
    print ('dlnp', dlnp)
    if dlnp < 1e-3:
        break

# Get the fit model and residual images for plotting
mod = tractor.getModelImage(0)
chi = tractor.getChiImage(0)
# Plots
plt.clf()
plt.subplot(2,2,1)
plt.imshow(trueimage, **ima)
plt.title('True image')
plt.subplot(2,2,2)
plt.imshow(image, **ima)
plt.title('Image')
plt.subplot(2,2,3)
plt.imshow(mod, **ima)
plt.title('Tractor model')
plt.subplot(2,2,4)
plt.imshow(chi, **imchi)
plt.title('Chi')

# Plot the derivatives we saved earlier
def showpatch(patch, ima):
    im = patch.patch
    h,w = im.shape
    ext = [patch.x0,patch.x0+w, patch.y0,patch.y0+h]
    plt.imshow(im, extent=ext, **ima)
    plt.title(patch.name)
imderiv = dict(interpolation='nearest', origin='lower', cmap='gray',
               vmin=-0.05, vmax=0.05)
plt.clf()
plt.subplot(2,2,1)
plt.imshow(mod0, **ima)
ax = plt.axis()
plt.title('Initial Tractor model')
for i in range(3):
    plt.subplot(2,2,2+i)
    showpatch(derivs[i][0][0], imderiv)
    plt.axis(ax)
dlnp 4513.523076454851
dlnp 463.4199788197744
dlnp 0.0012550546867373669
dlnp 2.5142746835626895e-06

</code>


# Notebook from physicsliupeng/nn4fun
Path: nn4infer/svgd.ipynb

Implementation of Stein variational gradient descent (SVGD) in TensorFlow. And try to improve its performance in high-dimensional space._____no_output_____

<code>
%matplotlib inline

import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
import tensorflow.contrib.distributions as tfd_____no_output_____tf.reset_default_graph()_____no_output_____def expectation(f, xs, axis=0, name=None):
  """Expectation value of mapping function `f` to values `xs`.
  Args:
    f: Callable from tensor to tensor.
    xs: Tensor with the shape `[None]+fis`, where the `fis` is the
      shape of the input tensor of `f`.
  Returns:
    Tensor with the shape of the output tensor of `f`.
  """
  with tf.name_scope(name, 'Expectation', [xs]):
    fs = tf.map_fn(f, xs, parallel_iterations=100,
                   back_prop=False)
    return tf.reduce_mean(fs, axis)

def square_distance(x, y):
  return tf.reduce_sum(tf.square(x - y))

def rbf_kernel(x, y, h, name=None):
  """Implements the RBF kernel.
  Args:
    x: Tensor.
    y: Tensor with the same shape as the `x`.
    h: Non-vanishing scalar of float type.
  Returns:
    Scalar.
  """
  with tf.name_scope(name, 'RBFKernel', [x, y]):
    return tf.exp(-0.5/h * square_distance(x, y))

def grad_rbf_kernel(x, y, h, name=None):
  """Implements the gradients of the RBF kernel.
  Args:
    x: Tensor.
    y: Tensor with the same shape as the `x`.
    h: Non-vanishing scalar of float type.
  Returns:
    Tensor with the same shape as the `x`.
  """
  with tf.name_scope(name, 'RBFKernelGradient', [x, y]):
    return rbf_kernel(x, y, h) * (y - x) / h

def get_phi_star(samples, log_p, kernel, grad_kernel):
  """Get the function :math:`\phi_{\star}`(\cdot)` that maps
  from tensor of an arbitrary shape (say `x_shape`) to a tensor
  of the same shape.

  Args:
    samples: Tensor of the shape `[n_samples] + x_shape`.
    log_p: Callable that maps from tensor of `x_shape` to scalar.
    kernel: Callable that maps two tensors of `x_shape` to scaler.
    grad_kernel: The gradient of the `kernel`.
  Returns:
    Callable.
  """
  def phi_star(x):
    def adaptive_h():
      median_square = tf.reduce_mean(
          square_distance(tf.expand_dims(x, axis=0), samples))
      n_samples = samples.get_shape().as_list()[0]
      return median_square / tf.log(float(n_samples))
    def integrand(y):
      return (kernel(x, y, adaptive_h()) * tf.gradients(log_p(y), [y])[0]
              + grad_kernel(x, y, adaptive_h()))
    return expectation(integrand, samples)
  return phi_star

def median(value):
  value = tf.reshape(value, [-1])
  median_idx = value.get_shape().as_list()[0]//2
  return tf.nn.top_k(value, median_idx).values[median_idx-1]_____no_output_____def initialize_samples(n_samples, n_dims):
  samples_shape = [n_samples, n_dims]
  initializer = tf.initializers.random_uniform(minval=-1, maxval=1)
  samples = tf.get_variable(
      'samples', shape=samples_shape,
      dtype='float32', initializer=initializer)
  return samples

n_samples = 200
n_dims = 1
xs = initialize_samples(n_samples, n_dims)_____no_output_____dist = tfd.MultivariateNormalDiag(tf.zeros([n_dims]),
                                  tf.ones([n_dims]))
log_p = dist.log_prob_____no_output_____phi_star = get_phi_star(xs, log_p, rbf_kernel, grad_rbf_kernel)_____no_output_____class AdaGrad(object):
  def __init__(self, xs, phi_star, learning_rate, alpha=0.9):
    self.xs = xs
    self.phi_star = phi_star
    self.learning_rate = learning_rate
    self.alpha = alpha
    self.cache = tf.Variable(tf.zeros(shape=xs.get_shape(),
                                      dtype='float32'),
                             name='AdaGradCache')

  def update(self):
    raw_grads = tf.map_fn(self.phi_star, self.xs, name='Gradients')
    new_cache = (self.alpha * self.cache
                 + (1 - self.alpha) * tf.square(raw_grads))
    update_cache_op = tf.assign(self.cache, new_cache, name='UpdateCache')
    with tf.control_dependencies([update_cache_op]):
      adjusted_grads = raw_grads / (tf.sqrt(self.cache) + 1e-8)
      delta_xs = self.learning_rate * adjusted_grads
      update_op = tf.assign(self.xs, self.xs + delta_xs, name='UpdateVars')
      return update_op

update_op = AdaGrad(xs, phi_star, learning_rate=0.1).update()_____no_output_____init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)_____no_output_____%%time

n_iters = 100
for step in range(n_iters):
    sess.run(update_op)
    if step % 1 == 0:
        print('After step {}'.format(step))
        axis=0
        xs_val = sess.run(xs)
        targets = np.random.normal(size=n_samples)
        plt.hist([xs_val[:,axis], targets], bins=20)
        plt.show()
xs_val = sess.run(xs)After step 0
axis=0
targets = np.random.normal(size=n_samples)
plt.hist([xs_val[:,axis], targets], bins=10)
plt.show()_____no_output_____
</code>


# Notebook from vasudev-sharma/course-content
Path: tutorials/W3D5_NetworkCausality/W3D5_Tutorial1.ipynb

_____no_output_____# Tutorial 1: Interventions
**Week 3, Day 5: Network Causality**

**By Neuromatch Academy**

**Content creators**: Ari Benjamin, Tony Liu, Konrad Kording

**Content reviewers**: Mike X Cohen, Madineh Sarvestani, Ella Batty, Michael Waskom
_____no_output_____---
#Tutorial Objectives

We list our overall day objectives below, with the sections we will focus on in this notebook in bold:

1.   **Master definitions of causality**
2.   **Understand that estimating causality is possible**
3.  **Learn 4 different methods and understand when they fail**
    1. **Perturbations**
    2. Correlations
    3. Simultaneous fitting/regression
    4. Instrumental variables

### Tutorial setting

How do we know if a relationship is causal? What does that mean? And how can we estimate causal relationships within neural data?

The methods we'll learn today are very general and can be applied to all sorts of data, and in many circumstances.
Causal questions are everywhere!

### Tutorial 1 Objectives:

1.   Simulate a neural system
2.   Understand perturbation as a method of estimating causality_____no_output_____---
# Setup_____no_output_____

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1 import make_axes_locatable_____no_output_____#@title Figure settings
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")_____no_output_____#@title Helper functions

def sigmoid(x):
    """
    Compute sigmoid nonlinearity element-wise on x

    Args:
        x (np.ndarray): the numpy data array we want to transform
    Returns
        (np.ndarray): x with sigmoid nonlinearity applied
    """
    return 1 / (1 + np.exp(-x))

def create_connectivity(n_neurons, random_state=42):
    """
    Generate our nxn causal connectivity matrix.

    Args:
        n_neurons (int): the number of neurons in our system.
        random_state (int): random seed for reproducibility

    Returns:
        A (np.ndarray): our 0.1 sparse connectivity matrix
    """
    np.random.seed(random_state)
    A_0 = np.random.choice([0, 1], size=(n_neurons, n_neurons), p=[0.9, 0.1])

    # set the timescale of the dynamical system to about 100 steps
    _, s_vals, _ = np.linalg.svd(A_0)
    A = A_0 / (1.01 * s_vals[0])

    # _, s_val_test, _ = np.linalg.svd(A)
    # assert s_val_test[0] < 1, "largest singular value >= 1"

    return A

def see_neurons(A, ax):
    """
    Visualizes the connectivity matrix.

    Args:
        A (np.ndarray): the connectivity matrix of shape (n_neurons, n_neurons)
        ax (plt.axis): the matplotlib axis to display on

    Returns:
        Nothing, but visualizes A.
    """
    A = A.T  # make up for opposite connectivity
    n = len(A)
    ax.set_aspect('equal')
    thetas = np.linspace(0, np.pi * 2, n, endpoint=False)
    x, y = np.cos(thetas), np.sin(thetas),
    ax.scatter(x, y, c='k', s=150)

    # Renormalize
    A = A / A.max()
    for i in range(n):
        for j in range(n):
            if A[i, j] > 0:
                ax.arrow(x[i], y[i], x[j] - x[i], y[j] - y[i], color='k', alpha=A[i,j],
                         head_width=.15, width = A[i, j] / 25, shape='right',
                         length_includes_head=True)
    ax.axis('off')

def get_perturbed_connectivity_all_neurons(perturbed_X):
    """
    Estimates the connectivity matrix of perturbations through stacked correlations.

    Args:
        perturbed_X (np.ndarray): the simulated dynamical system X of shape
                                  (n_neurons, timesteps)

    Returns:
        R (np.ndarray): the estimated connectivity matrix of shape
                        (n_neurons, n_neurons)
    """
    # select perturbations (P) and outcomes (Outs)
    # we perturb the system every over time step, hence the 2 in slice notation
    P = perturbed_X[:, ::2]
    Outs = perturbed_X[:, 1::2]

    # stack perturbations and outcomes into a 2n by (timesteps / 2) matrix
    S = np.concatenate([P, Outs], axis=0)

    # select the perturbation -> outcome block of correlation matrix (upper right)
    R = np.corrcoef(S)[:n_neurons, n_neurons:]

    return R

def simulate_neurons_perturb(A, timesteps):
    """
    Simulates a dynamical system for the specified number of neurons and timesteps,
    BUT every other timestep the activity is clamped to a random pattern of 1s and 0s

    Args:
        A (np.array): the true connectivity matrix
        timesteps (int): the number of timesteps to simulate our system.

    Returns:
        The results of the simulated system.
        - X has shape (n_neurons, timeteps)
    """
    n_neurons = len(A)
    X = np.zeros((n_neurons, timesteps))

    for t in range(timesteps - 1):

        if t % 2 == 0:
            X[:, t] = np.random.choice([0, 1], size=n_neurons)

        epsilon = np.random.multivariate_normal(np.zeros(n_neurons), np.eye(n_neurons))
        X[:, t + 1] = sigmoid(A.dot(X[:, t]) + epsilon)  # we are using helper function sigmoid

    return X

def plot_connectivity_matrix(A, ax=None):
  """Plot the (weighted) connectivity matrix A as a heatmap

    Args:
      A (ndarray): connectivity matrix (n_neurons by n_neurons)
      ax: axis on which to display connectivity matrix
  """
  if ax is None:
    ax = plt.gca()
  lim = np.abs(A).max()
  im = ax.imshow(A, vmin=-lim, vmax=lim, cmap="coolwarm")
  ax.tick_params(labelsize=10)
  ax.xaxis.label.set_size(15)
  ax.yaxis.label.set_size(15)
  cbar = ax.figure.colorbar(im, ax=ax, ticks=[0], shrink=.7)
  cbar.ax.set_ylabel("Connectivity Strength", rotation=90,
                     labelpad= 20,va="bottom")
  ax.set(xlabel="Connectivity from", ylabel="Connectivity to")

def plot_connectivity_graph_matrix(A):
  """Plot both connectivity graph and matrix side by side

    Args:
      A (ndarray): connectivity matrix (n_neurons by n_neurons)

  """
  fig, axs = plt.subplots(1, 2, figsize=(10, 5))
  see_neurons(A,axs[0])  # we are invoking a helper function that visualizes the connectivity matrix
  plot_connectivity_matrix(A)

  fig.suptitle("Neuron Connectivity")
  plt.show()

def plot_neural_activity(X):
  """Plot first 10 timesteps of neural activity

  Args:
    X (ndarray): neural activity (n_neurons by timesteps)

  """
  f, ax = plt.subplots()
  im = ax.imshow(X[:, :10])
  divider = make_axes_locatable(ax)
  cax1 = divider.append_axes("right", size="5%", pad=0.15)
  plt.colorbar(im, cax=cax1)
  ax.set(xlabel='Timestep', ylabel='Neuron', title='Simulated Neural Activity')

def plot_true_vs_estimated_connectivity(estimated_connectivity, true_connectivity, selected_neuron=None):
  """Visualize true vs estimated connectivity matrices

  Args:
    estimated_connectivity (ndarray): estimated connectivity (n_neurons by n_neurons)
    true_connectivity (ndarray): ground-truth connectivity (n_neurons by n_neurons)
    selected_neuron (int or None): None if plotting all connectivity, otherwise connectivity
      from selected_neuron will be shown

  """

  fig, axs = plt.subplots(1, 2, figsize=(10, 5))

  if selected_neuron is not None:
    plot_connectivity_matrix(np.expand_dims(estimated_connectivity, axis=1), ax=axs[0])
    plot_connectivity_matrix(true_connectivity[:, [selected_neuron]], ax=axs[1])
    axs[0].set_xticks([0])
    axs[1].set_xticks([0])
    axs[0].set_xticklabels([selected_neuron])
    axs[1].set_xticklabels([selected_neuron])
  else:
    plot_connectivity_matrix(estimated_connectivity, ax=axs[0])
    plot_connectivity_matrix(true_connectivity, ax=axs[1])

  axs[1].set(title="True connectivity")
  axs[0].set(title="Estimated connectivity")_____no_output_____

---
# Section 1: Defining and estimating causality
_____no_output_____

#@title Video 1: Defining causality
# Insert the ID of the corresponding youtube video
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="yiddT2sMbZM", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video_____no_output_____

Let's think carefully about the statement "**A causes B**". To be concrete, let's take two neurons. What does it mean to say that neuron $A$ causes neuron $B$ to fire?

The *interventional* definition of causality says that:
$$
(A \text{ causes } B) \Leftrightarrow ( \text{ If we force }A \text { to be different, then }B\text{ changes})
$$

To determine if $A$ causes $B$ to fire, we can inject current into neuron $A$ and see what happens to $B$.

**A mathematical definition of causality**:
Over many trials, the average causal effect $\delta_{A\to B}$ of neuron $A$ upon neuron $B$ is the average change in neuron $B$'s activity when we set $A=1$ versus when we set $A=0$.

$$
\delta_{A\to B} = \mathbb{E}[B | A=1] -  \mathbb{E}[B | A=0]
$$

Note that this is an average effect. While one can get more sophisticated about conditional effects ($A$ only effects $B$ when it's not refractory, perhaps), we will only consider average effects today.

**Relation to a randomized controlled trial (RCT)**:
The logic we just described is the logic of a randomized control trial (RCT). If you randomly give 100 people a drug and 100 people a placebo, the effect is the difference in outcomes.

_____no_output_____## Exercise 1: Randomized controlled trial for two neurons

Let's pretend we can perform a randomized controlled trial for two neurons. Our model will have neuron $A$ synapsing on Neuron $B$:
$$B = A + \epsilon$$
 where $A$ and $B$ represent the activities of the two neurons and $\epsilon$ is standard normal noise $\epsilon\sim\mathcal{N}(0,1)$.

Our goal is to perturb $A$ and confirm that $B$ changes.
_____no_output_____

def neuron_B(activity_of_A):
  """Model activity of neuron B as neuron A activity + noise

  Args:
    activity_of_A (ndarray): An array of shape (T,) containing the neural activity of neuron A

  Returns:
    ndarray: activity of neuron B
  """
  noise = np.random.randn(activity_of_A.shape[0])
  return activity_of_A + noise

np.random.seed(12)

# Neuron A activity of zeros
A_0 = np.zeros(5000)

# Neuron A activity of ones
A_1 = np.ones(5000)

###########################################################################
## TODO for students: Estimate the causal effect of A upon B
## Use eq above (difference in mean of B when A=0 vs. A=1)
###########################################################################
diff_in_means = ...
#print(diff_in_means)_____no_output_____# to_remove solution
def neuron_B(activity_of_A):
  """Model activity of neuron B as neuron A activity + noise

  Args:
    activity_of_A (ndarray): An array of shape (T,) containing the neural activity of neuron A

  Returns:
    ndarray: activity of neuron B
  """
  noise = np.random.randn(activity_of_A.shape[0])
  return activity_of_A + noise

np.random.seed(12)

# Neuron A activity of zeros
A_0 = np.zeros(5000)

# Neuron A activity of ones
A_1 = np.ones(5000)

diff_in_means = neuron_B(A_1).mean() - neuron_B(A_0).mean()
print(diff_in_means)_____no_output_____

You should get a difference in means of `0.990719` (so very close to one). _____no_output_____---
# Section 2: Simulating a system of neurons

_____no_output_____Can we still estimate causal effects when the neurons are in big networks? This is the main question we will ask today. Let's first create our system, and the rest of today will be spend analyzing it.
_____no_output_____

#@title Video 2: Simulated neural system model
# Insert the ID of the corresponding youtube video
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="oPJz49dAuL8", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video_____no_output_____

**Video correction**: the connectivity graph plots and associated explanations in this and other videos show the wrong direction of connectivity (the arrows should be pointing the opposite direction). This has been fixed in the figures below._____no_output_____## Section 2.1: Our system

This section recaps the system described in Video 2 so may be skipped.

Our system has N interconnected neurons that affect each other over time. Each neuron at time $t+1$ is a function of the activity of the other neurons from the previous time $t$.

Neurons affect each other nonlinearly: each neuron's activity at time $t+1$ consists of a linearly weighted sum of all neural activities at time $t$, with added noise, passed through a nonlinearity:

$$
\vec{x}_{t+1} = \sigma(A\vec{x}_t + \epsilon_t),
$$

- $\vec{x}_t$ is an $n$-dimensional vector representing our $n$-neuron system at timestep $t$
- $\sigma$ is a sigmoid nonlinearity
- $A$ is our $n \times n$ *causal ground truth connectivity matrix* (more on this later)
- $\epsilon_t$ is random noise: $\epsilon_t \sim N(\vec{0}, I_n)$
- $\vec{x}_0$ is initialized to $\vec{0}$

$A$ is a connectivity matrix, so the element $A_{ij}$ represents the causal effect of neuron $i$ on neuron $j$. In our system, neurons will receive connections from only 10% of the whole population on average.
_____no_output_____## Section 2.2: Visualize  true connectivity

We will create a connectivity matrix between 6 neurons and visualize it in two different ways: as a graph with directional edges between connected neurons and as an image of the connectivity matrix.

*Check your understanding*: do you understand how the left plot relates to the right plot below?_____no_output_____

#@markdown Execute this cell to visualize connectivity

## Initializes the system
n_neurons = 6
A = create_connectivity(n_neurons) # we are invoking a helper function that generates our nxn causal connectivity matrix.

# Let's plot it!
plot_connectivity_graph_matrix(A)_____no_output_____

## Exercise 2: System simulation_____no_output_____In this exercise we're going to simulate the system. Please complete the following function so that at every timestep the activity vector $x$ is updated according to:
$$
\vec{x}_{t+1} = \sigma(A\vec{x}_t + \epsilon_t).
$$
_____no_output_____

def simulate_neurons(A, timesteps, random_state=42):
    """Simulates a dynamical system for the specified number of neurons and timesteps.

    Args:
        A (np.array): the connectivity matrix
        timesteps (int): the number of timesteps to simulate our system.
        random_state (int): random seed for reproducibility

    Returns:
        - X has shape (n_neurons, timeteps). A schematic:
                   ___t____t+1___
       neuron  0  |   0    1     |
                  |   1    0     |
       neuron  i  |   0 -> 1     |
                  |   0    0     |
                  |___1____0_____|

    """
    np.random.seed(random_state)

    n_neurons = len(A)
    X = np.zeros((n_neurons, timesteps))

    for t in range(timesteps - 1):

        # Create noise vector
        epsilon = np.random.multivariate_normal(np.zeros(n_neurons), np.eye(n_neurons))

        ########################################################################
        ## TODO: Fill in the update rule for our dynamical system.
        ## Fill in function and remove
        raise NotImplementedError("Complete simulate_neurons")
        ########################################################################

        # Update activity vector for next step
        X[:, t + 1] = sigmoid(...)  # we are using helper function sigmoid

    return X

# Set simulation length
timesteps = 5000

# Uncomment below to test your function

# Simulate our dynamical system
# X = simulate_neurons(A, timesteps)

# plot_neural_activity(X)_____no_output_____# to_remove solution
def simulate_neurons(A, timesteps, random_state=42):
    """Simulates a dynamical system for the specified number of neurons and timesteps.

    Args:
        A (np.array): the connectivity matrix
        timesteps (int): the number of timesteps to simulate our system.
        random_state (int): random seed for reproducibility

    Returns:
        - X has shape (n_neurons, timeteps). A schematic:
                   ___t____t+1___
       neuron  0  |   0    1     |
                  |   1    0     |
       neuron  i  |   0 -> 1     |
                  |   0    0     |
                  |___1____0_____|

    """
    np.random.seed(random_state)

    n_neurons = len(A)
    X = np.zeros((n_neurons, timesteps))

    for t in range(timesteps - 1):

        # Create noise vector
        epsilon = np.random.multivariate_normal(np.zeros(n_neurons), np.eye(n_neurons))

        # Update activity vector for next step
        X[:, t + 1] = sigmoid(A @ X[:, t] + epsilon)  # we are using helper function sigmoid

    return X

# Set simulation length
timesteps = 5000

# Uncomment below to test your function

# Simulate our dynamical system
X = simulate_neurons(A, timesteps)

with plt.xkcd():
  plot_neural_activity(X)_____no_output_____

---
# Section 3: Recovering connectivity through perturbation_____no_output_____

#@title Video 3: Perturbing systems
# Insert the ID of the corresponding youtube video
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="wOZunGtuqQE", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video_____no_output_____

## Section 3.1: Random perturbation in our system of neurons

We want to get the causal effect of each neuron upon each other neuron. The ground truth of the causal effects is the connectivity matrix $A$.

Remember that we would like to calculate:
$$
\delta_{A\to B} = \mathbb{E}[B | A=1] -  \mathbb{E}[B | A=0]
$$

We'll do this by randomly setting the system state to 0 or 1 and observing the outcome after one timestep. If we do this $N$ times, the effect of neuron $i$ upon neuron $j$ is:
$$
\delta_{x^i\to x^j} \approx \frac1N \sum_i^N[x_{t+1}^j | x^i_t=1] -  \frac1N \sum_i^N[x_{t+1}^j | x^i_t=0]
$$
This is just the average difference of the activity of neuron $j$ in the two conditions.

We are going to calculate the above equation, but imagine it like *intervening* in activity every other timestep._____no_output_____We will use helper function `simulate_neurons_perturb`. While the rest of the function is the same as the ``simulate_neurons`` function in the previous exercise, every time step we now additionally include:
```
if t % 2 == 0:
    X[:,t] = np.random.choice([0,1], size=n_neurons)
```

This means that at every other timestep,  every neuron's activity is changed to either 0 or 1.

Pretty serious perturbation, huh? You don't want that going on in your brain.

**Now visually compare the dynamics:** Run this next cell and see if you can spot how the dynamics have changed._____no_output_____

# @markdown Execute this cell to visualize perturbed dynamics

timesteps = 5000  # Simulate for 5000 timesteps.

# Simulate our dynamical system for the given amount of time
X_perturbed = simulate_neurons_perturb(A, timesteps)

# Plot our standard versus perturbed dynamics
fig, axs = plt.subplots(1, 2, figsize=(15, 4))
im0 = axs[0].imshow(X[:, :10])
im1 = axs[1].imshow(X_perturbed[:, :10])

# Matplotlib boilerplate code
divider = make_axes_locatable(axs[0])
cax0 = divider.append_axes("right", size="5%", pad=0.15)
plt.colorbar(im0, cax=cax0)

divider = make_axes_locatable(axs[1])
cax1 = divider.append_axes("right", size="5%", pad=0.15)
plt.colorbar(im1, cax=cax1)

axs[0].set_ylabel("Neuron", fontsize=15)
axs[1].set_xlabel("Timestep", fontsize=15)
axs[0].set_xlabel("Timestep", fontsize=15);
axs[0].set_title("Standard dynamics", fontsize=15)
axs[1].set_title("Perturbed dynamics", fontsize=15);_____no_output_____#@title Video 4: Calculating causality
# Insert the ID of the corresponding youtube video
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="EDZtcsIAVGM", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video_____no_output_____

## Exercise 3: Using perturbed dynamics to recover connectivity

From the above perturbed dynamics, write a function that recovers the causal effect of a given single neuron (`selected_neuron`) upon all other neurons in the system. Remember from above you're calculating:
$$
\delta_{x^i\to x^j} \approx \frac1N \sum_i^N[x_{t+1}^j | x^i_t=1] -  \frac1N \sum_i^N[x_{t+1}^j | x^i_t=0]]
$$

Recall that we perturbed every neuron at every other timestep. Despite perturbing every neuron, in this exercise we are concentrating on computing the causal effect of a single neuron (we will look at all neurons effects on all neurons next). We want to exclusively use the timesteps without perturbation for $x^j_{t+1}$ and the timesteps with perturbation for $x^j_{t}$ in the formulas above. In numpy, indexing occurs as `array[ start_index : end_index : count_by]`. So getting every other element in an array (such as every other timestep) is as easy as `array[::2]`._____no_output_____

def get_perturbed_connectivity_from_single_neuron(perturbed_X, selected_neuron):
    """
    Computes the connectivity matrix from the selected neuron using differences in means.

    Args:
        perturbed_X (np.ndarray): the perturbed dynamical system matrix of shape (n_neurons, timesteps)
        selected_neuron (int): the index of the neuron we want to estimate connectivity for

    Returns:
        estimated_connectivity (np.ndarray): estimated connectivity for the selected neuron, of shape (n_neurons,)
    """
    # Extract the perturbations of neuron 1 (every other timestep)
    neuron_perturbations = perturbed_X[selected_neuron, ::2]

    # Extract the observed outcomes of all the neurons (every other timestep)
    all_neuron_output = perturbed_X[:, 1::2]

    # Initialize estimated connectivity matrix
    estimated_connectivity = np.zeros(n_neurons)

    # Loop over neurons
    for neuron_idx in range(n_neurons):

        # Get this output neurons (neuron_idx) activity
        this_neuron_output = all_neuron_output[neuron_idx, :]

        # Get timesteps where the selected neuron == 0 vs == 1
        one_idx = np.argwhere(neuron_perturbations == 1)
        zero_idx = np.argwhere(neuron_perturbations == 0)

        ########################################################################
        ## TODO: Insert your code here to compute the neuron activation from perturbations.
        # Fill out function and remove
        raise NotImplementedError("Complete the function get_perturbed_connectivity_single_neuron")
        ########################################################################

        difference_in_means = ...

        estimated_connectivity[neuron_idx] = difference_in_means

    return estimated_connectivity

# Initialize the system
n_neurons = 6
timesteps = 5000
selected_neuron = 1

# Simulate our perturbed dynamical system
perturbed_X = simulate_neurons_perturb(A, timesteps)

## Uncomment below to test your function

# Measure connectivity of neuron 1
# estimated_connectivity = get_perturbed_connectivity_from_single_neuron(perturbed_X, selected_neuron)

# plot_true_vs_estimated_connectivity(estimated_connectivity, A, selected_neuron)_____no_output_____# to_remove solution
def get_perturbed_connectivity_from_single_neuron(perturbed_X, selected_neuron):
    """
    Computes the connectivity matrix from the selected neuron using differences in means.

    Args:
        perturbed_X (np.ndarray): the perturbed dynamical system matrix of shape (n_neurons, timesteps)
        selected_neuron (int): the index of the neuron we want to estimate connectivity for

    Returns:
        estimated_connectivity (np.ndarray): estimated connectivity for the selected neuron, of shape (n_neurons,)
    """
    # Extract the perturbations of neuron 1 (every other timestep)
    neuron_perturbations = perturbed_X[selected_neuron, ::2]

    # Extract the observed outcomes of all the neurons (every other timestep)
    all_neuron_output = perturbed_X[:, 1::2]

    # Initialize estimated connectivity matrix
    estimated_connectivity = np.zeros(n_neurons)

    # Loop over neurons
    for neuron_idx in range(n_neurons):

        # Get this output neurons (neuron_idx) activity
        this_neuron_output = all_neuron_output[neuron_idx, :]

        # Get timesteps where the selected neuron == 0 vs == 1
        one_idx = np.argwhere(neuron_perturbations == 1)
        zero_idx = np.argwhere(neuron_perturbations == 0)

        difference_in_means = np.mean(this_neuron_output[one_idx]) - np.mean(this_neuron_output[zero_idx])

        estimated_connectivity[neuron_idx] = difference_in_means

    return estimated_connectivity

# Initialize the system
n_neurons = 6
timesteps = 5000
selected_neuron = 1

# Simulate our perturbed dynamical system
perturbed_X = simulate_neurons_perturb(A, timesteps)

## Uncomment below to test your function

# Measure connectivity of neuron 1
estimated_connectivity = get_perturbed_connectivity_from_single_neuron(perturbed_X, selected_neuron)

with plt.xkcd():
  plot_true_vs_estimated_connectivity(estimated_connectivity, A, selected_neuron)_____no_output_____

We can quantify how close our estimated connectivity matrix is to our true connectivity matrix by correlating them. We should see almost perfect correlation between our estimates and the true connectivity - do we?_____no_output_____

# Correlate true vs estimated connectivity matrix
np.corrcoef(A[:, selected_neuron], estimated_connectivity)[1, 0]_____no_output_____

**Note on interpreting A**: Strictly speaking, $A$ is not the matrix of causal effects but rather the dynamics matrix. So why compare them like this? The answer is that $A$ and the effect matrix both are $0$ everywhere except where there is a directed connection. So they should have a correlation of $1$ if we estimate the effects correctly. (Their scales, however, are different. This in part because the nonlinearity $\sigma$ squashes the values of $x$ to $[0,1]$.) See the Appendix after Tutorial 2 for more discussion of using correlation as a metric._____no_output_____## Section 3.2: Measuring how perturbations recover the entire connectivity matrix

Nice job! You just estimated connectivity for a single neuron.

We're now going to use the same strategy for all neurons at once. We provide this helper function `get_perturbed_connectivity_all_neurons`.  If you're curious about how this works and have extra time, scroll to the explanation at the bottom.
_____no_output_____

# Parameters
n_neurons = 6
timesteps = 5000

# Generate nxn causal connectivity matrix
A = create_connectivity(n_neurons)

# Simulate perturbed dynamical system
perturbed_X = simulate_neurons_perturb(A, timesteps)

# Get estimated connectivity matrix
R = get_perturbed_connectivity_all_neurons(perturbed_X)_____no_output_____#@markdown Execute this cell to visualize true vs estimated connectivity

# Let's visualize the true connectivity and estimated connectivity together
fig, axs = plt.subplots(1, 2, figsize=(10, 5))
see_neurons(A, axs[0]) # we are invoking a helper function that visualizes the connectivity matrix
plot_connectivity_matrix(A, ax=axs[1])
plt.suptitle("True connectivity matrix A");
plt.show()
fig, axs = plt.subplots(1,2, figsize=(10,5))
see_neurons(R.T,axs[0]) # we are invoking a helper function that visualizes the connectivity matrix
plot_connectivity_matrix(R.T, ax=axs[1])
plt.suptitle("Estimated connectivity matrix R");_____no_output_____

We can again calculate the correlation coefficient between the elements of the two matrices. As you can see from the cell below, we do a good job recovering the true causality of the system!_____no_output_____

np.corrcoef(A.transpose().flatten(), R.flatten())[1, 0]_____no_output_____

---
# Summary_____no_output_____

#@title Video 5: Summary
# Insert the ID of the corresponding youtube video
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="p3fZW5Woqa4", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video_____no_output_____

In this tutorial, we learned about how to define and estimate causality using pertubations. In particular we:

1) Learned how to simulate a system of connected neurons

2) Learned how to estimate the connectivity between neurons by directly perturbing neural activity_____no_output_____---
# Further resources for today

If you are interested in causality after NMA ends, here are some useful texts to consult.

*   *Causal Inference for Statistics, Social, and Biomedical Sciences* by Imbens and Rubin
*   *Causal Inference: What If* by Hernan and Rubin
*   *Mostly Harmless Econometrics* by Angrist and Pischke
*   https://www.nature.com/articles/s41562-018-0466-5 for application to neuroscience

_____no_output_____---
# Appendix
_____no_output_____## Computation of the estimated connectivity matrix

**This is an explanation of what the code is doing in `get_perturbed_connectivity_all_neurons()`**

First, we compute an estimated connectivity matrix $R$. We extract
perturbation matrix $P$ and outcomes matrix $O$:

$$
P = \begin{bmatrix}
\mid & \mid & ... & \mid \\
x_0  & x_2  & ... & x_T  \\
\mid & \mid & ... & \mid
\end{bmatrix}_{n \times T/2}
$$

$$
O = \begin{bmatrix}
\mid & \mid & ... & \mid \\
x_1  & x_3  & ... & x_{T-1}  \\
\mid & \mid & ... & \mid
\end{bmatrix}_{n \times T/2}
$$

And calculate the correlation of matrix $S$, which is $P$ and $O$ stacked on each other:

$$
S = \begin{bmatrix}
P  \\
O
\end{bmatrix}_{2n \times T/2}
$$

We then extract $R$ as the upper right $n \times n$ block of $corr(S)$:

This is because the upper right block corresponds to the estimated perturbation effect on outcomes for each pair of neurons in our system.

This method gives an estimated connectivity matrix that is the proportional to the result you would obtain with differences in means, and differs only in a proportionality constant that depends on the variance of $x$_____no_output_____


# Notebook from howardxjy/info6210_project_team_Cerberus
Path: ski_resort_project/misspelling_table.ipynb

# Introduction _____no_output_____In this file, we will import fuzz library and use it to correct our spelling and establish mis-spelling table._____no_output_____

<code>
from datetime import datetime
from datetime import timedelta
from itertools import dropwhile, takewhile
import pandas as pd
import numpy as np
import instaloader_____no_output_____df_1 = pd.read_csv('skiResort_ins_final 1.csv')_____no_output_____df_1_____no_output_____df_1_np = np.array(df_1)
csv_list = df_1_np.tolist()_____no_output_____ins_hashtag = csv_list[:]_____no_output_____for i in range(len(ins_hashtag)):
    ins_hashtag[i] = ins_hashtag[i][0]_____no_output_____ins_hashtag_____no_output_____from fuzzywuzzy import fuzz
from fuzzywuzzy import process_____no_output_____#process.extract(S1, ListS,limit=n)
#ListS=['49degreesnorth','altaskiresort','aldishill','skipark']
#test=str(input())
#ll=process.extract(a, ListS,limit=1)eee
ListS=ins_hashtag[:]_____no_output_____Lists_set={}
for i in range(len(ListS)):
    Lists_set[ListS[i]]=set()_____no_output_____Lists_set_____no_output_____def ccc():
    a = ListS
    b = [1,2,3,4,5,6,7,8,9,10]
    for i in b:
        c = input("Please input ")
        if c in a:
            print(c)
        else:
            most_like=process.extract(c, ListS,limit=1)
            Lists_set[most_like[0][0]].add(c)
            print(most_like[0][0])

_____no_output_____ccc()Please input lynna creek
bear-creek-mountain-resort
Please input snow
arizona-snowbowl
Please input soda spiring
soda-springs
Please input snake
snow-snake
Please input ball
alpenglow-at-arctic-valley
Please input star
northstar-california-resort
Please input fe
ferguson-ridge
Please input mousfa
49-degrees-north-mountain-resort
Please input smart
sitzmark
Please input return
rotarun
Lists_set_____no_output_____mis_hash=[]
for k,v in Lists_set.items():
    mis_hash.append(k)_____no_output_____mis_hash_____no_output_____misspel_df=pd.DataFrame.from_dict(Lists_set, orient='index')_____no_output_____misspel_df_____no_output_____misspel_np = np.array(misspel_df)
misspel_list = misspel_np.tolist()_____no_output_____misspel_list_____no_output_____misspel_list[0]_____no_output_____for i in range(len(mis_hash)):
    misspel_list[i].insert(0,mis_hash[i])_____no_output_____misspel_list_____no_output_____misspel_df_final= pd.DataFrame(misspel_list, columns=['original', 'mis1', 'mis2', 'mis3']) _____no_output_____misspel_df_final_____no_output_____misspel_df_final.to_csv('misspelling_table.csv',index=False)_____no_output_____
</code>


# Notebook from utshab-1603022/Starting_Machine_DeepLearning_DataScience
Path: .ipynb_checkpoints/5.3_Movie_Similarities-checkpoint.ipynb

import pandas as pd

r_cols = ['user_id', 'movie_id', 'rating']
ratings = pd.read_csv('ml-100k/u.data', sep='\t', names=r_cols, usecols=range(3), encoding="ISO-8859-1")
m_cols = ['movie_id', 'title']
movies = pd.read_csv('ml-100k/u.item', sep='|', names=m_cols, usecols=range(2), encoding="ISO-8859-1")

ratings = pd.merge(movies, ratings)
_____no_output_____ratings.head()_____no_output_____movieRatings = ratings.pivot_table(index=['user_id'], columns=['title'],values='rating')
movieRatings.head()_____no_output_____# Users who rated Star Wars
starWarsRatings = movieRatings['Star Wars (1977)']
starWarsRatings.head()_____no_output_____similarMovies = movieRatings.corrwith(starWarsRatings)
similarMovies = similarMovies.dropna()
df = pd.DataFrame(similarMovies)
df.head(10)c:\users\utshab kumar ghosh\appdata\local\programs\python\python36\lib\site-packages\numpy\lib\function_base.py:2517: RuntimeWarning: Degrees of freedom <= 0 for slice
  c = cov(x, y, rowvar)
c:\users\utshab kumar ghosh\appdata\local\programs\python\python36\lib\site-packages\numpy\lib\function_base.py:2446: RuntimeWarning: divide by zero encountered in true_divide
  c *= np.true_divide(1, fact)
similarMovies.sort_values(ascending=False)_____no_output_____

Get rid of less popular movies. Construct a dataFrame that counts up how many ratings exist for each movie and also avg rating._____no_output_____

import numpy as np

movieStats = ratings.groupby('title').agg({'rating': [np.size, np.mean]})
movieStats.head()_____no_output_____#get rid of movies rated by <100 people.
popularMovies = movieStats['rating']['size'] >= 100
movieStats[popularMovies].sort_values([('rating', 'mean')], ascending=False)[:15]_____no_output_____