# Notebook from rohit-patel/hmm-tagger
Path: HMM Tagger.ipynb

# Project: Part of Speech Tagging with Hidden Markov Models
---
### Introduction

Part of speech tagging is the process of determining the syntactic category of a word from the words in its surrounding context. It is often used to help disambiguate natural language phrases because it can be done quickly with high accuracy. Tagging can be used for many NLP tasks like determining correct pronunciation during speech synthesis (for example, _dis_-count as a noun vs dis-_count_ as a verb), for information retrieval, and for word sense disambiguation.

In this notebook, you'll use the [Pomegranate](http://pomegranate.readthedocs.io/) library to build a hidden Markov model for part of speech tagging using a "universal" tagset. Hidden Markov models have been able to achieve [>96% tag accuracy with larger tagsets on realistic text corpora](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf). Hidden Markov models have also been used for speech recognition and speech generation, machine translation, gene recognition for bioinformatics, and human gesture recognition for computer vision, and more.

![](_post-hmm.png)

The notebook already contains some code to get you started. You only need to add some new functionality in the areas indicated to complete the project; you will not need to modify the included code beyond what is requested. Sections that begin with **'IMPLEMENTATION'** in the header indicate that you must provide code in the block that follows. Instructions will be provided for each section, and the specifics of the implementation are marked in the code block with a 'TODO' statement. Please be sure to read the instructions carefully!_____no_output_____

**Note:** Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You must then **export the notebook** by running the last cell in the notebook, or by using the menu above and navigating to **File -> Download as -> HTML (.html)** Your submissions should include both the `html` and `ipynb` files.

_____no_output_____

**Note:** Code and Markdown cells can be executed using the `Shift + Enter` keyboard shortcut. Markdown cells can be edited by double-clicking the cell to enter edit mode.

_____no_output_____### The Road Ahead
You must complete Steps 1-3 below to pass the project. The section on Step 4 includes references & resources you can use to further explore HMM taggers.

- [Step 1](#Step-1:-Read-and-preprocess-the-dataset): Review the provided interface to load and access the text corpus
- [Step 2](#Step-2:-Build-a-Most-Frequent-Class-tagger): Build a Most Frequent Class tagger to use as a baseline
- [Step 3](#Step-3:-Build-an-HMM-tagger): Build an HMM Part of Speech tagger and compare to the MFC baseline
- [Step 4](#Step-4:-[Optional]-Improving-model-performance): (Optional) Improve the HMM tagger_____no_output_____

**Note:** Make sure you have selected a **Python 3** kernel in Workspaces or the hmm-tagger conda environment if you are running the Jupyter server on your own machine.

_____no_output_____

# Jupyter "magic methods" -- only need to be run once per kernel restart
%load_ext autoreload
%aimport helpers, tests
%autoreload 1_____no_output_____# import python modules -- this cell needs to be run again if you make changes to any of the files
import matplotlib.pyplot as plt
import numpy as np

from IPython.core.display import HTML
from itertools import chain
from collections import Counter, defaultdict
from helpers import show_model, Dataset
from pomegranate import State, HiddenMarkovModel, DiscreteDistribution_____no_output_____

## Step 1: Read and preprocess the dataset
---
We'll start by reading in a text corpus and splitting it into a training and testing dataset. The data set is a copy of the [Brown corpus](https://en.wikipedia.org/wiki/Brown_Corpus) (originally from the [NLTK](https://www.nltk.org/) library) that has already been pre-processed to only include the [universal tagset](https://arxiv.org/pdf/1104.2086.pdf). You should expect to get slightly higher accuracy using this simplified tagset than the same model would achieve on a larger tagset like the full [Penn treebank tagset](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html), but the process you'll follow would be the same.

The `Dataset` class provided in helpers.py will read and parse the corpus. You can generate your own datasets compatible with the reader by writing them to the following format. The dataset is stored in plaintext as a collection of words and corresponding tags. Each sentence starts with a unique identifier on the first line, followed by one tab-separated word/tag pair on each following line. Sentences are separated by a single blank line.

Example from the Brown corpus.
```
b100-38532
Perhaps	ADV
it	PRON
was	VERB
right	ADJ
;	.
;	.

b100-35577
...
```_____no_output_____

data = Dataset("tags-universal.txt", "brown-universal.txt", train_test_split=0.8)

print("There are {} sentences in the corpus.".format(len(data)))
print("There are {} sentences in the training set.".format(len(data.training_set)))
print("There are {} sentences in the testing set.".format(len(data.testing_set)))

assert len(data) == len(data.training_set) + len(data.testing_set), \
       "The number of sentences in the training set + testing set should sum to the number of sentences in the corpus"There are 57340 sentences in the corpus.
There are 45872 sentences in the training set.
There are 11468 sentences in the testing set.

### The Dataset Interface

You can access (mostly) immutable references to the dataset through a simple interface provided through the `Dataset` class, which represents an iterable collection of sentences along with easy access to partitions of the data for training & testing. Review the reference below, then run and review the next few cells to make sure you understand the interface before moving on to the next step.

```
Dataset-only Attributes:
    training_set - reference to a Subset object containing the samples for training
    testing_set - reference to a Subset object containing the samples for testing

Dataset & Subset Attributes:
    sentences - a dictionary with an entry {sentence_key: Sentence()} for each sentence in the corpus
    keys - an immutable ordered (not sorted) collection of the sentence_keys for the corpus
    vocab - an immutable collection of the unique words in the corpus
    tagset - an immutable collection of the unique tags in the corpus
    X - returns an array of words grouped by sentences ((w11, w12, w13, ...), (w21, w22, w23, ...), ...)
    Y - returns an array of tags grouped by sentences ((t11, t12, t13, ...), (t21, t22, t23, ...), ...)
    N - returns the number of distinct samples (individual words or tags) in the dataset

Methods:
    stream() - returns an flat iterable over all (word, tag) pairs across all sentences in the corpus
    __iter__() - returns an iterable over the data as (sentence_key, Sentence()) pairs
    __len__() - returns the nubmer of sentences in the dataset
```

For example, consider a Subset, `subset`, of the sentences `{"s0": Sentence(("See", "Spot", "run"), ("VERB", "NOUN", "VERB")), "s1": Sentence(("Spot", "ran"), ("NOUN", "VERB"))}`. The subset will have these attributes:

```
subset.keys == {"s1", "s0"}  # unordered
subset.vocab == {"See", "run", "ran", "Spot"}  # unordered
subset.tagset == {"VERB", "NOUN"}  # unordered
subset.X == (("Spot", "ran"), ("See", "Spot", "run"))  # order matches .keys
subset.Y == (("NOUN", "VERB"), ("VERB", "NOUN", "VERB"))  # order matches .keys
subset.N == 7  # there are a total of seven observations over all sentences
len(subset) == 2  # because there are two sentences
```

**Note:** The `Dataset` class is _convenient_, but it is **not** efficient. It is not suitable for huge datasets because it stores multiple redundant copies of the same data.

_____no_output_____#### Sentences

`Dataset.sentences` is a dictionary of all sentences in the training corpus, each keyed to a unique sentence identifier. Each `Sentence` is itself an object with two attributes: a tuple of the words in the sentence named `words` and a tuple of the tag corresponding to each word named `tags`._____no_output_____

key = 'b100-38532'
print("Sentence: {}".format(key))
print("words:\n\t{!s}".format(data.sentences[key].words))
print("tags:\n\t{!s}".format(data.sentences[key].tags))Sentence: b100-38532
words:
	('Perhaps', 'it', 'was', 'right', ';', ';')
tags:
	('ADV', 'PRON', 'VERB', 'ADJ', '.', '.')

**Note:** The underlying iterable sequence is **unordered** over the sentences in the corpus; it is not guaranteed to return the sentences in a consistent order between calls. Use `Dataset.stream()`, `Dataset.keys`, `Dataset.X`, or `Dataset.Y` attributes if you need ordered access to the data.

#### Counting Unique Elements

You can access the list of unique words (the dataset vocabulary) via `Dataset.vocab` and the unique list of tags via `Dataset.tagset`._____no_output_____

print("There are a total of {} samples of {} unique words in the corpus."
      .format(data.N, len(data.vocab)))
print("There are {} samples of {} unique words in the training set."
      .format(data.training_set.N, len(data.training_set.vocab)))
print("There are {} samples of {} unique words in the testing set."
      .format(data.testing_set.N, len(data.testing_set.vocab)))
print("There are {} words in the test set that are missing in the training set."
      .format(len(data.testing_set.vocab - data.training_set.vocab)))

assert data.N == data.training_set.N + data.testing_set.N, \
       "The number of training + test samples should sum to the total number of samples"There are a total of 1161192 samples of 56057 unique words in the corpus.
There are 928458 samples of 50536 unique words in the training set.
There are 232734 samples of 25112 unique words in the testing set.
There are 5521 words in the test set that are missing in the training set.

#### Accessing word and tag Sequences
The `Dataset.X` and `Dataset.Y` attributes provide access to ordered collections of matching word and tag sequences for each sentence in the dataset._____no_output_____

# accessing words with Dataset.X and tags with Dataset.Y
for i in range(2):
    print("Sentence {}:".format(i + 1), data.X[i])
    print()
    print("Labels {}:".format(i + 1), data.Y[i])
    print()Sentence 1: ('Mr.', 'Podger', 'had', 'thanked', 'him', 'gravely', ',', 'and', 'now', 'he', 'made', 'use', 'of', 'the', 'advice', '.')

Labels 1: ('NOUN', 'NOUN', 'VERB', 'VERB', 'PRON', 'ADV', '.', 'CONJ', 'ADV', 'PRON', 'VERB', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence 2: ('But', 'there', 'seemed', 'to', 'be', 'some', 'difference', 'of', 'opinion', 'as', 'to', 'how', 'far', 'the', 'board', 'should', 'go', ',', 'and', 'whose', 'advice', 'it', 'should', 'follow', '.')

Labels 2: ('CONJ', 'PRT', 'VERB', 'PRT', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'ADP', 'ADV', 'ADV', 'DET', 'NOUN', 'VERB', 'VERB', '.', 'CONJ', 'DET', 'NOUN', 'PRON', 'VERB', 'VERB', '.')

#### Accessing (word, tag) Samples
The `Dataset.stream()` method returns an iterator that chains together every pair of (word, tag) entries across all sentences in the entire corpus._____no_output_____

# use Dataset.stream() (word, tag) samples for the entire corpus
print("\nStream (word, tag) pairs:\n")
for i, pair in enumerate(data.stream()):
    print("\t", pair)
    if i > 5: break
Stream (word, tag) pairs:

	 ('Mr.', 'NOUN')
	 ('Podger', 'NOUN')
	 ('had', 'VERB')
	 ('thanked', 'VERB')
	 ('him', 'PRON')
	 ('gravely', 'ADV')
	 (',', '.')

For both our baseline tagger and the HMM model we'll build, we need to estimate the frequency of tags & words from the frequency counts of observations in the training corpus. In the next several cells you will complete functions to compute the counts of several sets of counts. _____no_output_____## Step 2: Build a Most Frequent Class tagger
---

Perhaps the simplest tagger (and a good baseline for tagger performance) is to simply choose the tag most frequently assigned to each word. This "most frequent class" tagger inspects each observed word in the sequence and assigns it the label that was most often assigned to that word in the corpus._____no_output_____### IMPLEMENTATION: Pair Counts

Complete the function below that computes the joint frequency counts for two input sequences._____no_output_____

def pair_counts(sequences_A, sequences_B):
    """Return a dictionary keyed to each unique value in the first sequence list
    that counts the number of occurrences of the corresponding value from the
    second sequences list.

    For example, if sequences_A is tags and sequences_B is the corresponding
    words, then if 1244 sequences contain the word "time" tagged as a NOUN, then
    you should return a dictionary such that pair_counts[NOUN][time] == 1244
    """
    # TODO: Finish this function!
    tag_word_lists = defaultdict(list)
    tag_word_counts = defaultdict(list)
    for i, (tag,word) in enumerate(zip(sequences_A,sequences_B)):
        tag_word_lists[tag].append(word)
    for k in tag_word_lists.keys():
        tag_word_counts[k] = Counter(tag_word_lists[k])

    return tag_word_counts

# Calculate C(t_i, w_i)
emission_counts = pair_counts(*reversed([i for i in zip(*data.training_set.stream())]))

assert len(emission_counts) == 12, \
       "Uh oh. There should be 12 tags in your dictionary."
assert max(emission_counts["NOUN"], key=emission_counts["NOUN"].get) == 'time', \
       "Hmmm...'time' is expected to be the most common NOUN."
HTML('
Your emission counts look good!
')_____no_output_____

### IMPLEMENTATION: Most Frequent Class Tagger

Use the `pair_counts()` function and the training dataset to find the most frequent class label for each word in the training data, and populate the `mfc_table` below. The table keys should be words, and the values should be the appropriate tag string.

The `MFCTagger` class is provided to mock the interface of Pomegranite HMM models so that they can be used interchangeably._____no_output_____

# Create a lookup table mfc_table where mfc_table[word] contains the tag label most frequently assigned to that word
from collections import namedtuple

FakeState = namedtuple("FakeState", "name")

class MFCTagger:
    # NOTE: You should not need to modify this class or any of its methods
    missing = FakeState(name="<MISSING>")

    def __init__(self, table):
        self.table = defaultdict(lambda: MFCTagger.missing)
        self.table.update({word: FakeState(name=tag) for word, tag in table.items()})

    def viterbi(self, seq):
        """This method simplifies predictions by matching the Pomegranate viterbi() interface"""
        return 0., list(enumerate(["<start>"] + [self.table[w] for w in seq] + ["<end>"]))

# TODO: calculate the frequency of each tag being assigned to each word (hint: similar, but not
# the same as the emission probabilities) and use it to fill the mfc_table

word_counts = pair_counts(*zip(*data.training_set.stream()))

mfc_table = {key : val.most_common(1)[0][0] for key,val in word_counts.items()}

# DO NOT MODIFY BELOW THIS LINE
mfc_model = MFCTagger(mfc_table) # Create a Most Frequent Class tagger instance

assert len(mfc_table) == len(data.training_set.vocab), ""
assert all(k in data.training_set.vocab for k in mfc_table.keys()), ""
assert sum(int(k not in mfc_table) for k in data.testing_set.vocab) == 5521, ""
HTML('
Your MFC tagger has all the correct words!
')_____no_output_____

### Making Predictions with a Model
The helper functions provided below interface with Pomegranate network models & the mocked MFCTagger to take advantage of the [missing value](http://pomegranate.readthedocs.io/en/latest/nan.html) functionality in Pomegranate through a simple sequence decoding function. Run these functions, then run the next cell to see some of the predictions made by the MFC tagger._____no_output_____

def replace_unknown(sequence):
    """Return a copy of the input sequence where each unknown word is replaced
    by the literal string value 'nan'. Pomegranate will ignore these values
    during computation.
    """
    return [w if w in data.training_set.vocab else 'nan' for w in sequence]

def simplify_decoding(X, model):
    """X should be a 1-D sequence of observations for the model to predict"""
    _, state_path = model.viterbi(replace_unknown(X))
    return [state[1].name for state in state_path[1:-1]]  # do not show the start/end state predictions_____no_output_____

### Example Decoding Sequences with MFC Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, mfc_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', '<MISSING>', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADV', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

### Evaluating Model Accuracy

The function below will evaluate the accuracy of the MFC tagger on the collection of all sentences from a text corpus. _____no_output_____

def accuracy(X, Y, model):
    """Calculate the prediction accuracy by using the model to decode each sequence
    in the input X and comparing the prediction with the true labels in Y.

    The X should be an array whose first dimension is the number of sentences to test,
    and each element of the array should be an iterable of the words in the sequence.
    The arrays X and Y should have the exact same shape.

    X = [("See", "Spot", "run"), ("Run", "Spot", "run", "fast"), ...]
    Y = [(), (), ...]
    """
    correct = total_predictions = 0
    for observations, actual_tags in zip(X, Y):

        # The model.viterbi call in simplify_decoding will return None if the HMM
        # raises an error (for example, if a test sentence contains a word that
        # is out of vocabulary for the training set). Any exception counts the
        # full sentence as an error (which makes this a conservative estimate).
        try:
            most_likely_tags = simplify_decoding(observations, model)
            correct += sum(p == t for p, t in zip(most_likely_tags, actual_tags))
        except:
            pass
        total_predictions += len(observations)
    return correct / total_predictions_____no_output_____

#### Evaluate the accuracy of the MFC tagger
Run the next cell to evaluate the accuracy of the tagger on the training and test corpus._____no_output_____

mfc_training_acc = accuracy(data.training_set.X, data.training_set.Y, mfc_model)
print("training accuracy mfc_model: {:.2f}%".format(100 * mfc_training_acc))

mfc_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, mfc_model)
print("testing accuracy mfc_model: {:.2f}%".format(100 * mfc_testing_acc))

assert mfc_training_acc >= 0.955, "Uh oh. Your MFC accuracy on the training set doesn't look right."
assert mfc_testing_acc >= 0.925, "Uh oh. Your MFC accuracy on the testing set doesn't look right."
HTML('
Your MFC tagger accuracy looks correct!
')training accuracy mfc_model: 95.72%
testing accuracy mfc_model: 93.01%

## Step 3: Build an HMM tagger
---
The HMM tagger has one hidden state for each possible tag, and parameterized by two distributions: the emission probabilties giving the conditional probability of observing a given **word** from each hidden state, and the transition probabilities giving the conditional probability of moving between **tags** during the sequence.

We will also estimate the starting probability distribution (the probability of each **tag** being the first tag in a sequence), and the terminal probability distribution (the probability of each **tag** being the last tag in a sequence).

The maximum likelihood estimate of these distributions can be calculated from the frequency counts as described in the following sections where you'll implement functions to count the frequencies, and finally build the model. The HMM model will make predictions according to the formula:

$$t_i^n = \underset{t_i^n}{\mathrm{argmax}} \prod_{i=1}^n P(w_i|t_i) P(t_i|t_{i-1})$$

Refer to Speech & Language Processing [Chapter 10](https://web.stanford.edu/~jurafsky/slp3/10.pdf) for more information._____no_output_____### IMPLEMENTATION: Unigram Counts

Complete the function below to estimate the co-occurrence frequency of each symbol over all of the input sequences. The unigram probabilities in our HMM model are estimated from the formula below, where N is the total number of samples in the input. (You only need to compute the counts for now.)

$$P(tag_1) = \frac{C(tag_1)}{N}$$_____no_output_____

def unigram_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequence list that
    counts the number of occurrences of the value in the sequences list. The sequences
    collection should be a 2-dimensional array.

    For example, if the tag NOUN appears 275558 times over all the input sequences,
    then you should return a dictionary such that your_unigram_counts[NOUN] == 275558.
    """

    unigram_dict = defaultdict(int)
    for value in sequences:
        unigram_dict[value] += 1
    return unigram_dict

# DONE: call unigram_counts with a list of tag sequences from the training set
tag_unigrams = unigram_counts(tag for (word,tag) in data.training_set.stream())

assert set(tag_unigrams.keys()) == data.training_set.tagset, \
       "Uh oh. It looks like your tag counts doesn't include all the tags!"
assert min(tag_unigrams, key=tag_unigrams.get) == 'X', \
       "Hmmm...'X' is expected to be the least common class"
assert max(tag_unigrams, key=tag_unigrams.get) == 'NOUN', \
       "Hmmm...'NOUN' is expected to be the most common class"
HTML('
Your tag unigrams look good!
')_____no_output_____

### IMPLEMENTATION: Bigram Counts

Complete the function below to estimate the co-occurrence frequency of each pair of symbols in each of the input sequences. These counts are used in the HMM model to estimate the bigram probability of two tags from the frequency counts according to the formula: $$P(tag_2|tag_1) = \frac{C(tag_2|tag_1)}{C(tag_2)}$$
_____no_output_____

def bigram_counts(sequences):
    """Return a dictionary keyed to each unique PAIR of values in the input sequences
    list that counts the number of occurrences of pair in the sequences list. The input
    should be a 2-dimensional array.

    For example, if the pair of tags (NOUN, VERB) appear 61582 times, then you should
    return a dictionary such that your_bigram_counts[(NOUN, VERB)] == 61582
    """

    # TODO: Finish this function!
    raise NotImplementedError

# TODO: call bigram_counts with a list of tag sequences from the training set
tag_bigrams = bigram_counts(# TODO: YOUR CODE HERE)

assert len(tag_bigrams) == 144, \
       "Uh oh. There should be 144 pairs of bigrams (12 tags x 12 tags)"
assert min(tag_bigrams, key=tag_bigrams.get) in [('X', 'NUM'), ('PRON', 'X')], \
       "Hmmm...The least common bigram should be one of ('X', 'NUM') or ('PRON', 'X')."
assert max(tag_bigrams, key=tag_bigrams.get) in [('DET', 'NOUN')], \
       "Hmmm...('DET', 'NOUN') is expected to be the most common bigram."
HTML('
Your tag bigrams look good!
')_____no_output_____

### IMPLEMENTATION: Sequence Starting Counts
Complete the code below to estimate the bigram probabilities of a sequence starting with each tag._____no_output_____

def starting_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the beginning of
    a sequence.

    For example, if 8093 sequences start with NOUN, then you should return a
    dictionary such that your_starting_counts[NOUN] == 8093
    """
    # TODO: Finish this function!
    raise NotImplementedError

# TODO: Calculate the count of each tag starting a sequence
tag_starts = starting_counts(# TODO: YOUR CODE HERE)

assert len(tag_starts) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_starts, key=tag_starts.get) == 'X', "Hmmm...'X' is expected to be the least common starting bigram."
assert max(tag_starts, key=tag_starts.get) == 'DET', "Hmmm...'DET' is expected to be the most common starting bigram."
HTML('
Your starting tag counts look good!
')_____no_output_____

### IMPLEMENTATION: Sequence Ending Counts
Complete the function below to estimate the bigram probabilities of a sequence ending with each tag._____no_output_____

def ending_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the end of
    a sequence.

    For example, if 18 sequences end with DET, then you should return a
    dictionary such that your_starting_counts[DET] == 18
    """
    # TODO: Finish this function!
    raise NotImplementedError

# TODO: Calculate the count of each tag ending a sequence
tag_ends = ending_counts(# TODO: YOUR CODE HERE)

assert len(tag_ends) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_ends, key=tag_ends.get) in ['X', 'CONJ'], "Hmmm...'X' or 'CONJ' should be the least common ending bigram."
assert max(tag_ends, key=tag_ends.get) == '.', "Hmmm...'.' is expected to be the most common ending bigram."
HTML('
Your ending tag counts look good!
')_____no_output_____

### IMPLEMENTATION: Basic HMM Tagger
Use the tag unigrams and bigrams calculated above to construct a hidden Markov tagger.

- Add one state per tag
    - The emission distribution at each state should be estimated with the formula: $P(w|t) = \frac{C(t, w)}{C(t)}$
- Add an edge from the starting state `basic_model.start` to each tag
    - The transition probability should be estimated with the formula: $P(t|start) = \frac{C(start, t)}{C(start)}$
- Add an edge from each tag to the end state `basic_model.end`
    - The transition probability should be estimated with the formula: $P(end|t) = \frac{C(t, end)}{C(t)}$
- Add an edge between _every_ pair of tags
    - The transition probability should be estimated with the formula: $P(t_2|t_1) = \frac{C(t_1, t_2)}{C(t_1)}$_____no_output_____

basic_model = HiddenMarkovModel(name="base-hmm-tagger")

# TODO: create states with emission probability distributions P(word | tag) and add to the model
# (Hint: you may need to loop & create/add new states)
basic_model.add_states()

# TODO: add edges between states for the observed transition frequencies P(tag_i | tag_i-1)
# (Hint: you may need to loop & add transitions
basic_model.add_transition()

# NOTE: YOU SHOULD NOT NEED TO MODIFY ANYTHING BELOW THIS LINE
# finalize the model
basic_model.bake()

assert all(tag in set(s.name for s in basic_model.states) for tag in data.training_set.tagset), \
       "Every state in your network should use the name of the associated tag, which must be one of the training set tags."
assert basic_model.edge_count() == 168, \
       ("Your network should have an edge from the start node to each state, one edge between every " +
        "pair of tags (states), and an edge from each state to the end node.")
HTML('
Your HMM network topology looks good!
')_____no_output_____hmm_training_acc = accuracy(data.training_set.X, data.training_set.Y, basic_model)
print("training accuracy basic hmm model: {:.2f}%".format(100 * hmm_training_acc))

hmm_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, basic_model)
print("testing accuracy basic hmm model: {:.2f}%".format(100 * hmm_testing_acc))

assert hmm_training_acc > 0.97, "Uh oh. Your HMM accuracy on the training set doesn't look right."
assert hmm_testing_acc > 0.955, "Uh oh. Your HMM accuracy on the testing set doesn't look right."
HTML('
Your HMM tagger accuracy looks correct! Congratulations, you\'ve finished the project.
')_____no_output_____

### Example Decoding Sequences with the HMM Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, basic_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")_____no_output_____

## Finishing the project
---

**Note:** **SAVE YOUR NOTEBOOK**, then run the next cell to generate an HTML copy. You will zip & submit both this file and the HTML copy for review.

_____no_output_____

!!jupyter nbconvert *.ipynb_____no_output_____

## Step 4: [Optional] Improving model performance
---
There are additional enhancements that can be incorporated into your tagger that improve performance on larger tagsets where the data sparsity problem is more significant. The data sparsity problem arises because the same amount of data split over more tags means there will be fewer samples in each tag, and there will be more missing data  tags that have zero occurrences in the data. The techniques in this section are optional.

- [Laplace Smoothing](https://en.wikipedia.org/wiki/Additive_smoothing) (pseudocounts)
    Laplace smoothing is a technique where you add a small, non-zero value to all observed counts to offset for unobserved values.

- Backoff Smoothing
    Another smoothing technique is to interpolate between n-grams for missing data. This method is more effective than Laplace smoothing at combatting the data sparsity problem. Refer to chapters 4, 9, and 10 of the [Speech & Language Processing](https://web.stanford.edu/~jurafsky/slp3/) book for more information.

- Extending to Trigrams
    HMM taggers have achieved better than 96% accuracy on this dataset with the full Penn treebank tagset using an architecture described in [this](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf) paper. Altering your HMM to achieve the same performance would require implementing deleted interpolation (described in the paper), incorporating trigram probabilities in your frequency tables, and re-implementing the Viterbi algorithm to consider three consecutive states instead of two.

### Obtain the Brown Corpus with a Larger Tagset
Run the code below to download a copy of the brown corpus with the full NLTK tagset. You will need to research the available tagset information in the NLTK docs and determine the best way to extract the subset of NLTK tags you want to explore. If you write the following the format specified in Step 1, then you can reload the data using all of the code above for comparison.

Refer to [Chapter 5](http://www.nltk.org/book/ch05.html) of the NLTK book for more information on the available tagsets._____no_output_____

import nltk
from nltk import pos_tag, word_tokenize
from nltk.corpus import brown

nltk.download('brown')
training_corpus = nltk.corpus.brown
training_corpus.tagged_sents()[0]_____no_output_____


# Notebook from choldgraf/bids_fellow_timeline
Path: timeline.ipynb

import matplotlib.pyplot as plt
import numpy as np
from datetime import timedelta, datetime
import pandas as pd
import requests
import seaborn as sns
sns.set(font_scale=1.5, style='white')
plt.ion()_____no_output_____

## Enter your info here_____no_output_____

# Choose your starting and ending dates for data
start_date = pd.to_datetime('November, 2016').to_pydatetime()
end_date = pd.to_datetime('November, 2018').to_pydatetime()

# Choose rows that have this name in them
name = "holdgraf"_____no_output_____

## Download raw data_____no_output_____

# Grab the data and subset the rows we want
url_data = 'https://docs.google.com/spreadsheets/d/e/2PACX-1vQUAL9ncZR5RodLq3K8_yAT0S8-SPgC4yBCPAPd2Jg4BtwZkSYEgWre4-9pRjzTiufbShQiI98mC2e-/pub?output=csv'
data_all = pd.read_csv(url_data, skiprows=1)_____no_output_____

## Select rows from your query_____no_output_____

# Pull a subset of rows for our query
data = data_all.loc[[name.lower() in row['Who'].lower() for (ix, row) in data_all.iterrows()]]

# Renaming things so they look pretty
data = data[['start', 'Type', 'Name']].rename(columns={'start': 'date', 'Type': 'kind', 'Name': 'name'})
data = data.dropna(subset=['date'])
rename_kinds = {'Free and open source software': 'FOSS',
                'Workshop, bootcamps, and others': "Workshop/bootcamp",
                'Non-academic communication (white paper, blogpost)': "Non-academic publication"}
data = data.replace({'kind':rename_kinds})

# Fixing and ordering dates
data['date'] = pd.to_datetime(data['date'])
data = data.sort_values('date').reset_index(drop=True)
date_ranges = pd.date_range(data['date'].values[0], data['date'].values[-1], freq='W')

# Collect list of unique types for our colors
kind_counts = data['kind'].value_counts().to_frame()
kind_counts['ix'] = np.arange(len(kind_counts))
cmap = plt.cm.viridis
colors = [cmap(float(ii / len(kind_counts))) for ii in kind_counts['ix']]_____no_output_____

## Make the viz_____no_output_____

# Levels to to set how spread apart (vertically) the text is
levels = np.array([-3.5, 3.5, -2, 2, -.5, .5])
day_jitter_max = 25
day_jitter_max_td = timedelta(days=day_jitter_max)
today = pd.to_datetime(datetime.today()).to_pydatetime()

# Create the base line
fig, ax = plt.subplots(figsize=(25, 13))
ax.plot((start_date - day_jitter_max_td, today), (0, 0), 'k', alpha=.5)
ax.plot((today, end_date + day_jitter_max_td), (0, 0), 'r', alpha=.5)

for ix, (idate, ikind, iname) in data.iterrows():
    # Skip dates outside of our range
    if not isinstance(ikind, str):
        continue
    if idate < start_date:
        continue

    # Date jitters for nice plotting
    idate = idate.to_pydatetime()
    days_jitter = np.random.randint(-day_jitter_max, day_jitter_max)
    idate = idate + timedelta(days=int(days_jitter))

    # Build the words to display
    max_chars = 50
    if len(iname) > max_chars:
        iname = iname[:max_chars] + '...'
    words = [ii.strip() for ii in iname.split(' ')]
    for ii in np.arange(len(words))[1::3]:
        if ii >= len(words)-1:
            continue
        words[ii] += '\n'
    words.insert(0, '$\mathbf{'+ikind.upper()+'}$\n')
    iname = ' '.join(words)

    # Choose the y-axis level
    y_jitter = np.random.rand() - .5
    level = levels[ix % len(levels)] + y_jitter
    vert = 'top' if level < 0 else 'bottom'

    # Make the visual
    ax.scatter(idate.toordinal(), 0, s=100, facecolor='w', edgecolor='k', zorder=9999)
    ax.plot((idate, idate), (0, level),
            c=colors[kind_counts.loc[ikind]['ix']], alpha=.7)
    ax.text(idate, level, iname,
            horizontalalignment='right', verticalalignment=vert, fontsize=12,
            backgroundcolor=(1., 1., 1., .3))

# Set up our xticks
plt_xticks = pd.date_range(start_date - day_jitter_max_td, end_date + day_jitter_max_td, freq='2M')
max_height = np.max(np.abs(levels)) + 2
ax.set_title('Timeline for query: {}'.format(name))
ax.set_ylim(-max_height, max_height)
ax.set_xticks(plt_xticks)
ax.set_xticklabels(plt_xticks.strftime("%b %Y"), rotation=45, horizontalalignment='right')
ax.set_xlabel('')
ax.set_xlim([start_date - day_jitter_max_td, None])
plt.setp(ax.get_yticklabels() + ax.get_yticklines() + list(ax.spines.values()), visible=False);_____no_output_____for ii in data['name'].values:
    print('* ' + ii.strip())* Matplotlib
* the BIDS partyparrot project
* MNE-python
* Encoding and Decoding Models in Cognitive Electrophysiology
* Data Science for Cognitive Neuroscience
(Data8 Connector Course, Fall 2016 and Spring 2017)
* Data Driven Neuroscience bootcamp
* Sphinx-gallery documentation overhaul for Matplotlib
* Docathon
* Rapid tuning shifts in human auditory cortex enhance speech intelligibility
* Zero to JupyterHub
* Data Carpentry Reproducibility with Jupyter Hackathon
* Binder 2.0
* nbclean for notebooks cleaning and preparation
* BinderHub
* repo2docker
* The types, roles, and practices of Documentation in Data Analytics Open Source Software Libraries
* Visualizing the Human Brain with open-source tools
* Best paper nomination at ECSCW ("types, roles, and practices of documentation in data analytics open source software libraries")
* Best presentation award at ECSCW ("Types, roles, and practices of documentation in data analytics open source software libraries"
* Brain Imaging Data Structure (BIDS) iEEG data format specification
* Portable learning environments for hands-on computational instruction using container-and cloud-based technology to teach data science
* Sphinx-gallery development
* NeuroHackWeek 2017 tutorial in Machine Learning
* The Past, Present, and Future of JupyterHub at MSDSE 2017
* Zero to JupyterHub at UC Berkeley
* Binder workshop at UC Davis
* Software Carpentry at the Society for Neuroscience 2017
* gitautopush
* Zero to Data 8
* BinderHub and Binder 2.0 at SciPy
* BinderHub and Binder 2.0 at SciPy
* UK Research Software Engineers meeting
* grave
* JupyterDays presentation in Paris
* Data 8 textbook overhaul to Jekyll
* Jupyter Books
* Github project
* DSEP Open Source Building Blocks with Jupyter
* PEARC18: Hands-on Tutorial Deploying Portable Research and Learning Environments on Commercial Clouds and XSEDE (at PEARC18)
* Mozilla Open Leaders Cohort 6
* The Open Source Alliance for Open Scholarship sustainability handbook
* Jupyter Governance refactoring project
* NUMFocus Annual Project Summit 2018
* West Big Data Innovation Hub meeting: Data Science Pedagogy and Practice with JupyterHub and an open-source stack
* CROSS symposium: Open tools and services for practical reproducibility in the cloud


# Notebook from ehsanasgari/MicroPheno
Path: notebooks/1.Bootstrapping.ipynb

__author__ = "Ehsaneddin Asgari"
__license__ = "GPL"
__version__ = "1.0.0"
__maintainer__ = "Ehsaneddin Asgari"
__email__ = "asgari@berkeley.edu or ehsaneddin.asgari@helmholtz-hzi.de"
__project__ = "LLP - MicroPheno"
__website__ = "https://llp.berkeley.edu/micropheno/"
import sys
sys.path.append('../')
from bootstrapping.bootstrapping import BootStrapping
from utility.file_utility import FileUtility_____no_output_____

# Body-site dataset_____no_output_____## 1. Reading fasta files_____no_output_____

### getting a list of fasta files need to be read from directory
files=FileUtility.recursive_glob('../data_config/bodysites/','*.txt')
list_of_files=[]
for file in files:
    list_of_files+=FileUtility.load_list(file)
list_of_files=[x+'.fsa' for x in list_of_files]

### reading fasta files
# return: list of fasta files, and a dic to map file to index
# load only only_files
fasta_files, mapping = FileUtility.read_fasta_directory('../../../datasets/deepbio/microbiome/hmb_data/','fsa',only_files=list_of_files)
_____no_output_____

## 2. Creating bootstrapping object_____no_output_____

## create a bootstrapping object with the parameter explaind in the paper
BS=BootStrapping(fasta_files,'body-site/', seqtype='fsa' ,sampling_sizes=[10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10000],
                 n_resamples=10, M=10)_____no_output_____

## 3. Bootstrapping for different k-mers_____no_output_____

## add k-mer samplings
for k in [3,4,5,6,7,8]:
    print(k,'-mer bootstrapping..')
    BS.add_kmer_sampling(k)3
 sampling size  10  is started ...
 sampling size  10  is completed.
 sampling size  20  is started ...
 sampling size  20  is completed.
 sampling size  50  is started ...
 sampling size  50  is completed.
 sampling size  100  is started ...
 sampling size  100  is completed.
 sampling size  200  is started ...
 sampling size  200  is completed.
 sampling size  500  is started ...
 sampling size  500  is completed.
 sampling size  1000  is started ...
 sampling size  1000  is completed.
 sampling size  2000  is started ...
 sampling size  2000  is completed.
 sampling size  5000  is started ...
 sampling size  5000  is completed.
 sampling size  10000  is started ...
 sampling size  10000  is completed.
4
 sampling size  10  is started ...
 sampling size  10  is completed.
 sampling size  20  is started ...
 sampling size  20  is completed.
 sampling size  50  is started ...
 sampling size  50  is completed.
 sampling size  100  is started ...
 sampling size  100  is completed.
 sampling size  200  is started ...
 sampling size  200  is completed.
 sampling size  500  is started ...
 sampling size  500  is completed.
 sampling size  1000  is started ...
 sampling size  1000  is completed.
 sampling size  2000  is started ...
 sampling size  2000  is completed.
 sampling size  5000  is started ...
 sampling size  5000  is completed.
 sampling size  10000  is started ...
 sampling size  10000  is completed.
5
 sampling size  10  is started ...
 sampling size  10  is completed.
 sampling size  20  is started ...
 sampling size  20  is completed.
 sampling size  50  is started ...
 sampling size  50  is completed.
 sampling size  100  is started ...
 sampling size  100  is completed.
 sampling size  200  is started ...
 sampling size  200  is completed.
 sampling size  500  is started ...
 sampling size  500  is completed.
 sampling size  1000  is started ...
 sampling size  1000  is completed.
 sampling size  2000  is started ...
 sampling size  2000  is completed.
 sampling size  5000  is started ...
 sampling size  5000  is completed.
 sampling size  10000  is started ...
 sampling size  10000  is completed.
6
 sampling size  10  is started ...
 sampling size  10  is completed.
 sampling size  20  is started ...
 sampling size  20  is completed.
 sampling size  50  is started ...
 sampling size  50  is completed.
 sampling size  100  is started ...
 sampling size  100  is completed.
 sampling size  200  is started ...
 sampling size  200  is completed.
 sampling size  500  is started ...
 sampling size  500  is completed.
 sampling size  1000  is started ...
 sampling size  1000  is completed.
 sampling size  2000  is started ...
 sampling size  2000  is completed.
 sampling size  5000  is started ...
 sampling size  5000  is completed.
 sampling size  10000  is started ...
 sampling size  10000  is completed.
7
 sampling size  10  is started ...
 sampling size  10  is completed.
 sampling size  20  is started ...
 sampling size  20  is completed.
 sampling size  50  is started ...
 sampling size  50  is completed.
 sampling size  100  is started ...
 sampling size  100  is completed.
 sampling size  200  is started ...
 sampling size  200  is completed.
 sampling size  500  is started ...
 sampling size  500  is completed.
 sampling size  1000  is started ...
 sampling size  1000  is completed.
 sampling size  2000  is started ...
 sampling size  2000  is completed.
 sampling size  5000  is started ...
 sampling size  5000  is completed.
 sampling size  10000  is started ...
 sampling size  10000  is completed.
8
 sampling size  10  is started ...
 sampling size  10  is completed.
 sampling size  20  is started ...
 sampling size  20  is completed.
 sampling size  50  is started ...
 sampling size  50  is completed.
 sampling size  100  is started ...
 sampling size  100  is completed.
 sampling size  200  is started ...
 sampling size  200  is completed.
 sampling size  500  is started ...
 sampling size  500  is completed.
 sampling size  1000  is started ...
 sampling size  1000  is completed.
 sampling size  2000  is started ...
 sampling size  2000  is completed.
 sampling size  5000  is started ...
 sampling size  5000  is completed.
 sampling size  10000  is started ...
 sampling size  10000  is completed.

## 4. Plotting_____no_output_____

import matplotlib.pyplot as plt
%pylab inline
%matplotlib inline

BS.plotting('results_bootstrapping_body-site','body-site')Populating the interactive namespace from numpy and matplotlib

# 5. Saving_____no_output_____

BS.save_me('body-site_bootstrapping')_____no_output_____

# 6. Loading _____no_output_____

## create a bootstrapping object with the parameter explaind in the paper
BS2=BootStrapping.load_precalculated('../../bio_cpe_data/results/datasetting1/results_bootstrapping_setting2.pickle')_____no_output_____BS1=BootStrapping.load_precalculated('../../bio_cpe_data/results/datasetting1/results_bootstrapping_setting1.pickle')_____no_output_____BS1.D_S==BS2.D_S_____no_output_____import _pickle as pickle

f = open('../../bio_cpe_data/results/datasetting2/setting2.pickle', 'wb')
pickle.dump(BS.__dict__, f, 2)
f.close()_____no_output_____import matplotlib.pyplot as plt
%pylab inline
%matplotlib inline

BS.plotting('results_bootstrapping_body-site','\#body-site')Populating the interactive namespace from numpy and matplotlib

# Crohn's disease dataset_____no_output_____Similar step can be taken to produce the k-mer plots. Or you can just load the previously produced the results:_____no_output_____

BS=BootStrapping.load_precalculated('../../MicroPheno_datasets/crohns-disease/results_bootstrapping_crohns-disease.pickle')_____no_output_____BS.plotting('results_bootstrapping_crohns-disease','Crohn\'s disease')_____no_output_____


# Notebook from bedaro/ssm-analysis
Path: Bottom DO.ipynb

# Bottom DO

A simple notebook to generate dissolved oxygen plots for the bottom layer, either for a single run or to compare multiple runs side-by-side.

## Parameters

* `do_data_cdf`: The NetCDF file containing DO results. This is generated using one of the extraction notebooks or [rawcdf_extract.py](rawcdf_extract.py).
* `do_variables`: The NetCDF variable names to output. The first variable will be treated as a control which other runs are compared against on the comparison plots.
* `start_date`: A date string to use as day 0 when formatting plot time axes.
* `domain_out_shp`: The shapefile that defines the node areas in the analysis domain. This is generated with [ProcessGrid.ipynb](ProcessGrid).
* `ssm_full_union_shp`: The shapefile that defines the complete SSM model grid as a single polygon. This is generated with [ProcessGrid.ipynb](ProcessGrid) and is only used to depict the water boundary/geography.
* `masked_nodes_txt`: The text file containing a list of node IDs that must be masked from the domain. This is generated with [ProcessGrid.ipynb](ProcessGrid) based on an Ecology-provided geodatabase [here](https://fortress.wa.gov/ecy/ezshare/EAP/SalishSea/SalishSeaModelBoundingScenarios.html), but it can be easily modified to add new cell masks for any additional discharge locations. For details on model output masking, see [Ecology's 2021 Technical Memorandum](https://www.ezview.wa.gov/Portals/_1962/Documents/PSNSRP/OptimizationScenarioTechMemo_9_13_2021.pdf). *Masking nodes in this notebook is extremely slow!* It is much more efficient to perform this masking as part of the raw NetCDF extraction step.
* `sites`: A dict of node numbers and corresponding labels for locations that get dedicated timeseries plots of bottom DO._____no_output_____

do_data_cdf = "model_results/bottom do 2008.nc"
do_variables = ('reference', 'existing')
start_date = "2008-01-01"
domain_out_shp = "gis/ssm filled domain nodes.shp"
ssm_full_union_shp = "gis/ssm full union.shp"
masked_nodes_txt = "gis/masked nodes.txt"

sites = {
    7786: "Admiralty Inlet South",
    12166: "East Passage",
    15490: "Budd Inlet",
    13789: "Lynch Cove"
}

from pathlib import Path
import os.path as path
from netCDF4 import Dataset
import geopandas as gpd
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import cmocean
import matplotlib.colors as colors
import analysis_util as au
#%matplotlib widget
%matplotlib inline_____no_output_____do_data = Dataset(do_data_cdf)
do_data_____no_output_____

Determine the timestep of the output (needed for hypoxic duration calculations later)_____no_output_____

timestep = do_data['time'][2] - do_data['time'][1]
timestep_____no_output_____

Determine the name to use for output files.

All outputs will be in the `notebook_outs` directory. They will start with the base name (without extension) of the given NetCDF file, followed by `_botdo_`, then a descriptive name of the output._____no_output_____

out_base = Path(do_data_cdf).stem
out_file = lambda name: path.join("notebook_outs", out_base + "_botdo_" + name)_____no_output_____

Remove any 'DOXG_bottom" suffixes for friendlier plot titles_____no_output_____

# I would use str.removesuffix here but that's not available until Python 3.9
def removesuffix(string, suffix):
    if len(string) < len(suffix):
        # That suffix must not be present
        return string
    elif string[-len(suffix):] == suffix:
        return string[:-len(suffix)]
    else:
        return string
do_variable_titles = [removesuffix(s, 'DOXG_bottom') for s in do_variables]
do_variable_titles_____no_output_____domain = gpd.read_file(domain_out_shp)
domain.set_index('node_id', inplace=True)
domain.head()_____no_output_____

Remove masked nodes from the domain. This must be done regardless of whether or not they need to be masked in the NetCDF output given. Use the node counts before and after to determine if masking of the NetCDF output is necessary._____no_output_____

nodes_unmasked = len(domain)
print("Total nodes before masking: ", nodes_unmasked)

need_to_mask = (nodes_unmasked == do_data.dimensions['node'].size)
if need_to_mask:
    print("** We will need to do masking inline, things are about to slow down!")

masked_nodes = np.loadtxt(masked_nodes_txt)
domain = domain.loc[~domain.index.isin(masked_nodes)]

print("Total nodes after masking: ", len(domain))

assert need_to_mask or (len(domain) == do_data.dimensions['node'].size), "Number of nodes in data file do not match the domain shapefile!"

domain.head()Total nodes before masking:  6120
** We will need to do masking inline, things are about to slow down!
Total nodes after masking:  4446

Build an array of all node dimension indices in the output that correspond to unmasked areas_____no_output_____

if need_to_mask:
    unmasked_indexer = np.isin(do_data['node'][:], domain.index.to_numpy()).nonzero()[0]
    print(unmasked_indexer.shape)
else:
    unmasked_indexer = slice(None)
unmasked_indexer(4446,)
au.read_base_map(ssm_full_union_shp).head()_____no_output_____

# Bottom DO for Selected Sites

Produce a timeseries of bottom DO at each of the sites defined in the `sites` variable, along with a corresponding Excel workbook that contains one sheet per site._____no_output_____

fig, axs = plt.subplots(len(sites), 2, constrained_layout=True, figsize=(9, len(sites)*2))
times = do_data['time'][:]
dates = pd.Timestamp(start_date) + pd.to_timedelta(times, 'd')
with pd.ExcelWriter(out_file("sites.xlsx")) as writer:
    # Iterate over all the specified sites, along with a dedicated plot Axes per site
    for ax,(n,s) in zip(axs, sites.items()):
        tsax, boxax = ax
        idx = np.nonzero(do_data['node'][:] == n)[0][0]
        data = {}
        lbl = "{0} ({1})".format(s, n)
        # Plot all the required variables on this Axes
        for v, t in zip(do_variables, do_variable_titles):
            data[t] = do_data[v][:,idx]
            tsax.plot(dates, data[t], label=t)
        tsax.set(title=s, ylabel="DO (mg/l)")

        boxax.boxplot(data.values(), labels=data.keys())

        df = pd.DataFrame(data, index=dates)
        df.to_excel(writer, sheet_name=lbl)

# Implement autofmt_xdate() by hand because it breaks subplot title placement
# Turn off xaxis labels on all but the last Axes
for ax in axs[:-1]:
    ax[0].set_xticklabels(())
# Orient the date labels properly on the lowest Axes
for l in axs[-1,0].get_xticklabels():
    l.set_ha('right')
    l.set_rotation(30)

fig.suptitle("Bottom DO Over Time")
# One common legend. https://stackoverflow.com/a/46921590/413862
handles, labels = axs[-1,0].get_legend_handles_labels()
fig.legend(handles, labels, loc='upper right')

fig.savefig(out_file("sites.png"))_____no_output_____

## Hypoxia

Pre-calculate the hypoxic durations and areas for each variable. If masking is enabled, this takes a while because of the indexer on the NetCDF dataset._____no_output_____

# Representative areas of each node in km2
areas = domain.loc[do_data['node'][unmasked_indexer].data].area / 1e6
# Total area in km2
total_area = areas.sum()

def extract_data(var):
    return do_data[var][:,unmasked_indexer].data

# Calculate the total time that this retrieved node index spent with a hypoxic DO
def calc_duration(data, i):
    return np.count_nonzero(data[:,i] < 2) * timestep

def calc_area(data, t):
    return areas.loc[data[t,:] < 2].sum()

extracted_data = {}
hypoxic_durations = {}
hypoxic_areas = {}
max_duration = 0

for v in do_variables:
    extracted_data[v] = extract_data(v)
    hypoxic_durations[v] = np.zeros(extracted_data[v].shape[1])
    for n in range(len(hypoxic_durations[v])):
        hypoxic_durations[v][n] = calc_duration(extracted_data[v], n)
    max_duration = max(max_duration, hypoxic_durations[v].max())
    hypoxic_areas[v] = np.zeros(extracted_data[v].shape[0])
    for t in range(len(hypoxic_areas[v])):
        hypoxic_areas[v][t] = calc_area(extracted_data[v], t)

# Assemble a DataFrame of all the hypoxic area timeseries for each variable
hypoxic_areas_df = pd.DataFrame(hypoxic_areas, index=dates)

print("Max hypoxic duration: {:.2f} days".format(max_duration))
hypoxic_areas_df.head()Max hypoxic duration: 149.67 days

## Hypoxic Duration

Create one plot per DO variable showing the hypoxic duration in days for each node. Also generates an ESRI shapefile that can be loaded into GIS software with the same information, and an Excel spreadsheet._____no_output_____

mycmap = cm.get_cmap("plasma_r").copy()
mycmap.set_under('white', 1)

fig, axs = plt.subplots(1, len(do_variables), figsize=(5*len(do_variables), 8))

# Normalize all the durations so the colors match between the plots
norm_duration = colors.Normalize(vmin=0.1, vmax=max_duration)

# Make a map of the durations for each variable, assembling them into one single
# GeoDataFrame for output
for ax, v, t in zip(axs, do_variables, do_variable_titles):
    if ax == axs[-1]:
        # See https://stackoverflow.com/a/13784887/413862 and
        # https://geopandas.org/en/latest/docs/user_guide/mapping.html#creating-a-legend
        fig.subplots_adjust(right=0.8)
        cbar_ax = fig.add_axes([0.85, 0.15, 0.05, 0.7])
        legend_args = { "legend": True, "cax": cbar_ax }
    else:
        legend_args = { "legend": False }
    d, drop = au.map_plot(domain, hypoxic_durations[v], t, ax=ax, cmap=mycmap, norm=norm_duration,
                          **legend_args)
    # A bit of a hack to combine all the durations into a single GeoDataFrame
    if ax == axs[0]:
        data = d
        data.rename(columns={"values": t}, inplace=True)
    else:
        data[t] = d['values']

fig.suptitle("Bottom Hypoxic Duration (days)")

fig.savefig(out_file("hypox_duration.png"))
data.to_file(out_file("hypox_duration.shp"))
data['node_area_km2'] = data['geometry'].area / 1e6
data.to_excel(out_file("hypox_duration.xlsx"))_____no_output_____if len(do_variables) > 1:

    duration_diffs = [hypoxic_durations[v] - hypoxic_durations[do_variables[0]] for v in do_variables[1:]]

    fig, axs = plt.subplots(1, len(do_variables) - 1, figsize=(5*(len(do_variables) - 1), 8))

    # For compatibility in the case that there's only a single plot to make
    if len(do_variables) == 2:
        axs = np.array([axs])

    divnorm = colors.CenteredNorm(halfrange=np.max(np.abs(duration_diffs)))

    # Make a map of the duration difference between each variable and the control
    for ax, v, t in zip(axs, duration_diffs, do_variable_titles[1:]):
        if ax == axs[-1]:
            # See https://stackoverflow.com/a/13784887/413862 and
            # https://geopandas.org/en/latest/docs/user_guide/mapping.html#creating-a-legend
            fig.subplots_adjust(right=0.8)
            cbar_ax = fig.add_axes([0.85, 0.15, 0.05, 0.7])
            legend_args = { "legend": True, "cax": cbar_ax }
        else:
            legend_args = { "legend": False }
        d, drop = au.map_plot(domain, v, f'{t} versus {do_variable_titles[0]}', ax=ax,
                              cmap=cmocean.cm.delta, norm=divnorm, **legend_args)

fig.suptitle("Bottom Hypoxic Duration (days)")
fig.savefig(out_file("hypox_duration_compare.png"))_____no_output_____

## Hypoxic Area

Generate a timeseries plot of each DO variable's total hypoxic area over the runtime of the model, plus an Excel spreadsheet containing the timeseries data._____no_output_____

# Plot each DF column and print the maximum value
fig, ax = plt.subplots()
for v, t in zip(do_variables, do_variable_titles):
    ax.plot(hypoxic_areas_df[v], label=t)
    print("{0}: Max hypoxic area {1:.2f} km2 on {2}".format(t, hypoxic_areas_df[v].max(),
                                                            hypoxic_areas_df[v].idxmax().date()))
ax.set(title="Bottom Hypoxic Area Over Time", ylabel="Area (km$^2$)")
fig.autofmt_xdate()
ax.legend()

# Build a secondary y-axis with the total area percentage
def area_to_pct(a):
    return a * 100 / total_area
def pct_to_area(p):
    return p / 100 * total_area

secax = ax.secondary_yaxis('right', functions=(area_to_pct, pct_to_area))
secax.set_ylabel("Percent of total")

fig.savefig(out_file("hypoxic_area.png"))
hypoxic_areas_df.to_excel(out_file("hypoxic_area.xlsx"))reference: Max hypoxic area 74.89 km2 on 2008-12-23
existing: Max hypoxic area 86.42 km2 on 2008-09-24

## Low DO Comparison

Compare DOmin and DO at the 0.1 quantile_____no_output_____

def plot_quantile(q):
    qdesc = f'q={q}' if q > 0 else 'min'

    vals = [np.quantile(extracted_data[v], q, axis=0) for v in do_variables]

    fig, axs = plt.subplots(1, len(do_variables) - 1, figsize=(5*(len(do_variables) - 1), 8))

    norm_do = colors.Normalize(vmin=np.min(vals[1:] - vals[0]), vmax=np.max(vals[1:] - vals[0]))

    # For compatibility in the case that there's only a single plot to make
    if len(do_variables) == 2:
        axs = np.array([axs])

    # Make a map of this quantile for each variable compared to the control, assembling
    # them into one single GeoDataFrame for output
    for ax, v, t in zip(axs, vals[1:], do_variable_titles[1:]):
        if ax == axs[-1]:
            # See https://stackoverflow.com/a/13784887/413862 and
            # https://geopandas.org/en/latest/docs/user_guide/mapping.html#creating-a-legend
            fig.subplots_adjust(right=0.8)
            cbar_ax = fig.add_axes([0.85, 0.15, 0.05, 0.7])
            legend_args = { "legend": True, "cax": cbar_ax }
        else:
            legend_args = { "legend": False }
        d, drop = au.map_plot(domain, v - vals[0], f'{t} versus {do_variable_titles[0]}', ax=ax,
                              cmap=cmocean.cm.phase, norm=norm_do, **legend_args)
    data = domain.copy()
    for v, t in zip(vals, do_variable_titles):
        data[t] = v

    fig.suptitle(f'Bottom DO({qdesc}) difference (mg/L)')

    fig.savefig(out_file(f'do_{qdesc}.png'))
    data.to_file(out_file(f'do_{qdesc}.shp'))
    data['node_area_km2'] = data['geometry'].area / 1e6
    data.to_excel(out_file(f'do_{qdesc}.xlsx'))

if len(do_variables) > 1:
    plot_quantile(0.1)_____no_output_____


# Notebook from keesterbrugge/python-causality-handbook
Path: causal-inference-for-the-brave-and-true/06-Grouped-and-Dummy-Regression.ipynb

# 06 - Grouped and Dummy Regression

## Regression With Grouped Data

Not all data points are created equal. If we look again at our ENEM dataset, we trust the scores of big schools much more than the scores from small schools. This is not to say that big schools are better or anything. It is just due to the fact that their big size imply less variance._____no_output_____

import warnings
warnings.filterwarnings('ignore')

import pandas as pd
import numpy as np
from scipy import stats
from matplotlib import style
import seaborn as sns
from matplotlib import pyplot as plt
import statsmodels.formula.api as smf

style.use("fivethirtyeight")_____no_output_____np.random.seed(876)
enem = pd.read_csv("./data/enem_scores.csv").sample(200)
plt.figure(figsize=(8,4))
sns.scatterplot(y="avg_score", x="number_of_students", data=enem)
sns.scatterplot(y="avg_score", x="number_of_students", s=100, label="Trustworthy",
                data=enem.query(f"number_of_students=={enem.number_of_students.max()}"))
sns.scatterplot(y="avg_score", x="number_of_students", s=100, label="Not so Much",
                data=enem.query(f"avg_score=={enem.avg_score.max()}"))
plt.title("ENEM Score by Number of Students in the School");_____no_output_____

In the data above, intuitively, points to the left should have less impact in my model than points to the right. In essence, points to the right are actually lots of other data points grouped into a single one. If we could unbundle them and run a linear regression on the ungrouped data, they would indeed contribute much more to the model estimation than an unbundled point in the left.

This phenomenon of having a region of low variance and another of high variance is called **heteroskedasticity**. Put it simply, heteroskedasticity is when the variance is not constant across all values of the features. In the case above, we can see that the variance decreases as the feature sample size increases. To give another example of where we have heteroskedasticity, if you plot wage by age, you will see that there is higher wage variance for the old than for the young. But, by far, the most common reason for variance to differ is grouped data.

Grouped data like the one above are extremely common in data analysis. One reason for that is confidentiality. Governments and firms can't give away personal data because that would violate data privacy requirements they have to follow. If they need to export data to an outside researcher, they can only do it by means of grouping the data. This way, individuals get grouped together and are no longer uniquely identifiable.

Fortunately for us, regression can handle those kinds of data pretty well. To understand how, let's first take some ungrouped data like the one we had on wage and education. It contains one line per worker, so we know the wage for each individual in this dataset and also how many years of education he or she has._____no_output_____

wage = pd.read_csv("./data/wage.csv")[["wage", "lhwage", "educ", "IQ"]]

wage.head()_____no_output_____

If we run a regression model to figure out how education is associated with log hourly wages, we get the following result._____no_output_____

model_1 = smf.ols('lhwage ~ educ', data=wage).fit()
model_1.summary().tables[1]_____no_output_____

Now, let's pretend for a moment that this data was under some confidentiality constraint. The provider of it was not able to give individualised data. So we ask him instead to group everyone by years of education and give us only the mean log hourly wage and the number of individuals in each group. This leaves us with only 10 data points._____no_output_____

group_wage = (wage
              .assign(count=1)
              .groupby("educ")
              .agg({"lhwage":"mean", "count":"count"})
              .reset_index())

group_wage_____no_output_____

Fear not! Regression doesn't need big data to work! What we can do is provide weights to our linear regression model. This way, it will consider groups with higher sample size more than the small groups. Notice how I've replaced the `smf.ols` with `smf.wls`, for weighted least squares. It's hard to notice, but it will make all the difference._____no_output_____

model_2 = smf.wls('lhwage ~ educ', data=group_wage, weights=group_wage["count"]).fit()
model_2.summary().tables[1]_____no_output_____

Notice how the parameter estimate of `educ` in the grouped model is very close to the one in the ungrouped data (actually, they are the same in this case). Also, even with only 10 data points, we've managed to get a statistically significant coefficient. That's because, although we have fewer points, grouping also lowers the variance by a lot. Also notice how the standard error is a bit larger, as is the t statistics. That's because some information about the variance is lost, so we have to be more conservative. Once we group the data, we don't know how large the variance is within each group. Compare the results above with what we would have with the non weighted model below._____no_output_____

model_3 = smf.ols('lhwage ~ educ', data=group_wage).fit()
model_3.summary().tables[1]_____no_output_____

The parameter estimate is larger. What is happening here is that the regression is placing equal weight for all points. If we plot the model along the grouped points, we see that the non weighted model is giving more importance to  small points in the lower left than it should. As a consequence, the line has a higher slope._____no_output_____

sns.scatterplot(x="educ", y = "lhwage", size="count", legend=False, data=group_wage, sizes=(40, 400))
plt.plot(wage["educ"], model_2.predict(wage["educ"]), c="C1", label = "Weighted")
plt.plot(wage["educ"], model_3.predict(wage["educ"]), c="C2", label = "Non Weighted")
plt.xlabel("Years of Education")
plt.ylabel("Log Hourly Wage")
plt.legend();_____no_output_____

The bottom line is that regression is this marvellous tool that works both with individual or aggregated data, but you have to use weights in this last case. To use weighted regression you need mean statistics. Not sums, not standard deviations, not medians, but means! For both the covariates and the dependent variable. Just keep in mind that the result of weighted regression with grouped data won't match exactly that of regression in ungrouped data, but it will be pretty similar.

![img](./data/img/dummy/heterosk.png)

I'll finish with a final example using additional covariates in a grouped data model._____no_output_____

group_wage = (wage
              .assign(count=1)
              .groupby("educ")
              .agg({"lhwage":"mean", "IQ":"mean", "count":"count"})
              .reset_index())

model_4 = smf.wls('lhwage ~ educ + IQ', data=group_wage, weights=group_wage["count"]).fit()
print("Number of observations:", model_4.nobs)
model_4.summary().tables[1]Number of observations: 10.0

In this example, we've included IQ as a feature, besides the previously added years of education. The mechanics is pretty much the same: get the means and count, regress the mean and use the count as weights.

## Regression for Dummies

Dummy variables are categorical variables we've encoded as binary columns. For example, suppose you have a gender variable that you wish to include in your model. This variable is encoded into 3 categories: male, female and other genders.

|gender|
|------|
|male  |
|female|
|female|
|other |
|male  |

Since our model only accepts numerical values, we need to convert this category to a number. In linear regression, we use dummies for that. We encode each variable as a 0/1 column, denoting the presence of a category. We also leave one of the categories out as the base category. This is necessary since the last category is a linear combination of the others. Put it differently, we can know the last category if someone gives us information on the others. In our example, if someone is neither female nor other genders, we can infer that the person's category is male.

|gender|female|other|
|------|:-----|:----|
|male  |0|0|
|female|1|0|
|female|1|0|
|other |0|1|
|male  |0|0|

We've already dealt with a simple form of dummy regression when dealing with A/B testing. More generally, when we are dealing with a binary treatment, we represent it as a dummy variable. In this case, **the regression coefficient for that dummy is the increment for the intercept in the regression line**, or the difference in means between the treated and untreated.

To make this more concrete, consider the problem of estimating the effect of graduating 12th grade on hourly wage (and let's ignore confounding just for now). In the code below, we've created a treatment dummy variable `T` indicating if years of education is greater than 12. _____no_output_____

wage = (pd.read_csv("./data/wage.csv")
        .assign(hwage=lambda d: d["wage"] / d["hours"])
        .assign(T=lambda d: (d["educ"] > 12).astype(int)))

wage[["hwage", "IQ", "T"]].head()_____no_output_____

The dummy works as a kind of switch. In our example, if the dummy is on, the predicted value is the intercept plus the dummy coefficient. If the dummy is off, the predicted value is just the intercept. _____no_output_____

smf.ols('hwage ~ T', data=wage).fit().summary().tables[1]_____no_output_____

In this case, when the person hasn't completed 12th grade (dummy off), the average income is 19.9. When he or she has completed 12th grade (dummy on), the predicted value or the average income is 24.8449 (19.9405 + 4.9044). Hence, the dummy coefficient captures the difference in means, which is 4.9044 in our case.

More formally, when the dependent variable is binary, as is often the case with treatment indicators, regression captures the ATE perfectly. That is because regression is a linear approximation to the conditional expectation function (CEF) $E[Y|X]$ and, in this particular case, the CEF IS linear. Namely, we can define $E[Y_i|T_i=0]=\alpha$ and $E[Y_i|T_i=1] = \alpha + \beta$, which leads to the following CEF

$
E[Y_i|T_i] =  E[Y_i|T_i=0] + \beta T_i = \alpha + \beta T_i
$

and $\beta$ is the difference in means or the ATE in the case of random data

$
\beta = [Y_i|T_i=1] - [Y_i|T_i=0]
$

If we use additional variables, the dummy coefficient becomes the **conditional** difference in means. For instance, let's say we add IQ to the previous model. Now, the dummy coefficient tells us how much increase we should expect from graduating 12th grade **while holding IQ fixed**. If we plot the prediction, we will see two parallel lines. The jump from one line to the next says the amount we should expect for completing 12th grade. They also say that the effect is constant. No matter your IQ, everyone benefits the same from graduating 12th grade. _____no_output_____

m = smf.ols('hwage ~ T+IQ', data=wage).fit()
plt_df = wage.assign(y_hat = m.fittedvalues)

plt.plot(plt_df.query("T==1")["IQ"], plt_df.query("T==1")["y_hat"], c="C1", label="T=1")
plt.plot(plt_df.query("T==0")["IQ"], plt_df.query("T==0")["y_hat"], c="C2", label="T=0")
plt.title(f"E[T=1|IQ] - E[T=0|IQ] = {round(m.params['T'], 2)}")
plt.ylabel("Wage")
plt.xlabel("IQ")
plt.legend();_____no_output_____

If we put this model into an equation, we can see why:

$
wage_i = \beta_0 + \beta_1T_i + \beta_2 IQ_i + e_i
$

Here, $\beta_1$ is the conditional difference in means and it is a constant value, 3.16 in our case. We can make this model more flexible by adding an interaction term.

$
wage_i = \beta_0 + \beta_1T_i + \beta_2 IQ_i + \beta_3 IQ_i * T_i  + e_i
$

Things are getting a little bit more complex, so let's see what each parameter means in this model. First, the intercept $\beta_0$. This bad boy doesn't have a particularly interesting interpretation. It's the expected wage when the treatment is zero (the person hasn't graduated from 12th grade) AND the IQ is zero. Since we don't expect IQ to be zero for anyone (actually… nevermind), this parameter is not very meaningful. Now, when we turn to $\beta_1$, we have a similar situation. This parameter is how much increase in wage we should expect from completing 12th grade **when IQ is zero**. Once again, since IQ is never zero, it doesn't have a particularly interesting meaning. Now, $\beta_2$ is a bit more interesting. It tells us how much IQ increases wages **for the non-treated**. So, in our case, it is something like 0.11. This means that for each 1 extra IQ point, the person that has not completed 12th grade should expect to gain an extra 11 cents per hour. Finally, the most interesting parameter is $\beta_3$. It tells us how much IQ increases the effect of graduating 12th grade. In our case, this parameter is 0.024, which means that for each extra IQ point, graduating 12th grade gives 2 extra cents. This might not seem much, but compare someone with 60IQ and with 140IQ. The first one will get an increase of 1.44 in wage (60 * 0.024), while the person with 140 IQ will gain an extra 3.36 dollars (60 * 0.024) when graduating from 12th grade.

In simple modeling jargon, this interaction term allows the treatment effect to change by levels of the features (only IQ, in this example). The result is that if we plot the prediction lines, we will see that they are no longer parallel and that those that graduate 12th grade (T=1) have a higher slope on IQ, higher IQ benefit more from graduating than lower IQ. This is sometimes referenced as effect modification or heterogeneous treatment effect. _____no_output_____

m = smf.ols('hwage ~ T*IQ', data=wage).fit()
plt_df = wage.assign(y_hat = m.fittedvalues)

plt.plot(plt_df.query("T==1")["IQ"], plt_df.query("T==1")["y_hat"], c="C1", label="T=1")
plt.plot(plt_df.query("T==0")["IQ"], plt_df.query("T==0")["y_hat"], c="C2", label="T=0")
plt.title(f"E[T=1|IQ] - E[T=0|IQ] = {round(m.params['T'], 2)}")
plt.ylabel("Wage")
plt.xlabel("IQ")
plt.legend();_____no_output_____

Finally, let's look at the case where all the variables in our model are dummies. To do so, we will discretize IQ into 4 bins and treat years of education as a category._____no_output_____

wage_ed_bins = (wage
                .assign(IQ_bins = lambda d: pd.qcut(d["IQ"], q=4, labels=range(4)))
                [["hwage", "educ", "IQ_bins"]])

wage_ed_bins.head()_____no_output_____

Treating education as a category, we no longer restrict the effect of education to a single parameter. Instead, we allow each year of education to have its own distinct impact. By doing so, we gain flexibility, since the effect of education is no longer parametric. This model simply computes the mean wage for each year of education._____no_output_____

model_dummy = smf.ols('hwage ~ C(educ)', data=wage).fit()
model_dummy.summary().tables[1]_____no_output_____plt.scatter(wage["educ"], wage["hwage"])
plt.plot(wage["educ"].sort_values(), model_dummy.predict(wage["educ"].sort_values()), c="C1")
plt.xlabel("Years of Education")
plt.ylabel("Hourly Wage");_____no_output_____

First of all, notice how this removes any assumption about the functional form of how education affects wages. We don't need to worry about logs anymore. In essence, this model is completely non-parametric. All it does is compute sample averages of wage for each year of education. This can be seen in the plot above, where the fitted line doesn't have a particular form. Instead, is the interpolation of the sample means for each year of education. We can also see that by reconstructing one parameter, for instance, that of 17 years of education. For this model, it's `9.5905`. Below, we can see how it is just the difference between the baseline years of education (9) and the individuals with 17 years

$
\beta_{17} = E[Y|T=17]-E[Y|T=9]
$

The trade-off is that we lose statistical significance when we allow such flexibility. Notice how big the p-values are for some years._____no_output_____

t1 = wage.query("educ==17")["hwage"]
t0 = wage.query("educ==9")["hwage"]
print("E[Y|T=9]:", t0.mean())
print("E[Y|T=17]-E[Y|T=9]:", t1.mean() - t0.mean())E[Y|T=9]: 18.56
E[Y|T=17]-E[Y|T=9]: 9.590472362353516

If we include more dummy covariates in the model, the parameter on education become a weighted average of the effect on each dummy group:

$
E\{ \ (E[Y_i|T=1, Group_i] - E[Y_i|T=0, Group_i])w(Group_i) \ \}
$

$w(Group_i)$ is not exactly, but is proportional to the variance of the treatment in the group $Var(T_i|Group_i)$. One natural question that arises from this is why not use the full nonparametric estimator, where the group weight is the sample size? This indeed is a valid estimator, but it is not what regression does. By using the treatment variance, regression is placing more weight on groups where the treatment varies a lot. This makes intuitive sense. If the treatment was almost constant (say 1 treated and everyone else untreated), it doesn't matter its sample size. It wouldn't provide much information about the treatment effect._____no_output_____

model_dummy_2 = smf.ols('hwage ~ C(educ) + C(IQ_bins)', data=wage_ed_bins).fit()
model_dummy_2.summary().tables[1]_____no_output_____

![img](./data/img/dummy/you_little_shit.png)

## Key Ideas

We started this section by looking at how some data points are more important than others. Namely, those with higher sample size and lower variance should be given more weight when estimating a linear model. Then, we looked at how linear regression can even handle grouped anonymised data with elegance, provided we use sample weights in our model.

Next, we moved to dummy regression. We saw how it can be made a non parametric model that places no assumptions whatsoever on the functional form of how the treatment impacts the outcome. We then explored the intuition behind dummy regression.

## References

I like to think of this entire book as a tribute to Joshua Angrist, Alberto Abadie and Christopher Walters for their amazing Econometrics class. Most of the ideas here are taken from their classes at the American Economic Association. Watching them is what is keeping me sane during this tough year of 2020.
* [Cross-Section Econometrics](https://www.aeaweb.org/conference/cont-ed/2017-webcasts)
* [Mastering Mostly Harmless Econometrics](https://www.aeaweb.org/conference/cont-ed/2020-webcasts)

I'll also like to reference the amazing books from Angrist. They have shown me that Econometrics, or 'Metrics as they call it, is not only extremely useful but also profoundly fun.

* [Mostly Harmless Econometrics](https://www.mostlyharmlesseconometrics.com/)
* [Mastering 'Metrics](https://www.masteringmetrics.com/)

My final reference is Miguel Hernan and Jamie Robins' book. It has been my trustworthy companion in the most thorny causal questions I had to answer.

* [Causal Inference Book](https://www.hsph.harvard.edu/miguel-hernan/causal-inference-book/)

![img](./data/img/poetry.png)

## Contribute

Causal Inference for the Brave and True is an open-source material on causal inference, the statistics of science. It uses only free software, based in Python. Its goal is to be accessible monetarily and intellectually.
If you found this book valuable and you want to support it, please go to [Patreon](https://www.patreon.com/causal_inference_for_the_brave_and_true). If you are not ready to contribute financially, you can also help by fixing typos, suggesting edits or giving feedback on passages you didn't understand. Just go to the book's repository and [open an issue](https://github.com/matheusfacure/python-causality-handbook/issues). Finally, if you liked this content, please share it with others who might find it useful and give it a [star on GitHub](https://github.com/matheusfacure/python-causality-handbook/stargazers).

_____no_output_____


# Notebook from korobool/hlll_course
Path: topics/05_MLP_1.ipynb

# An Introduction to Neural Networks

### How to implement the Neural Network from scratch in Python._____no_output_____The basic unit of a neural network is neuron.

A neuron takes inputs, does some math with them, and produces one output.

![neuron](./img/neuron.png)_____no_output_____3 things are happening in neuron. First, each input is multiplied by a weight:

$x_{1}\rightarrow x_{1}\star w_{1}$

$x_{2}\rightarrow x_{2}\star w_{2}$

Next, all the weighted inputs are added together with a bias b:

$(x_{1} \star w_{1}) + (x_{2} \star w_{2}) + b$

Finally, the sum is passed through an activation function:

$y = f((x_{1} \star w_{1}) + (x_{2} \star w_{2}) + b)$

The activation function is used to turn an unbounded input into an output that has a nice, predictable form. A commonly used activation function is the sigmoid function:_____no_output_____![sigmoid_function](./img/sigmoid_function.png)_____no_output_____The sigmoid function only outputs numbers in the range $(0, 1)$. You can think of it as compressing $(-\infty, +\infty)$ to $(0, 1)$ - big negative numbers become $~0$, and big positive numbers become $~1$._____no_output_____### A Simple Example

Assume we have a 2-input neuron that uses the sigmoid activation function and has the following parameters:

$w = [0, 1]$

$b = 4$

$w = [0, 1]$ is just a way of writing $w_{1} = 0$, $w_{2} = 1$ in vector form.

Now, let’s give the neuron an input of $x = [2,3]$

We’ll use the dot product to write things more concisely:

$(w\cdot x) + b = ((w_{1} \star x_{1}) + (w_{2} \star x_{2}) + b) = 0 \star 2 + 1 \star 3 + 4 = 7$

$y = f(w⋅x + b) = f(7) = 0.999$

The neuron outputs $0.999$ given the inputs $x = [2, 3]$. That’s it! This process of passing inputs forward to get an output is known as feedforward._____no_output_____### Coding a Neuron

We’ll use NumPy, a popular and powerful computing library for Python, to help us do math:_____no_output_____

import numpy as np
def sigmoid(x):
  # Our activation function: f(x) = 1 / (1 + e^(-x))
  return 1 / (1 + np.exp(-x))

class Neuron:
  def __init__(self, weights, bias):
    self.weights = weights
    self.bias = bias

  def feedforward(self, inputs):
    # Weight inputs, add bias, then use the activation function
    total = np.dot(self.weights, inputs) + self.bias
    return sigmoid(total)

weights = np.array([0, 1]) # w1 = 0, w2 = 1
bias = 4                   # b = 4
n = Neuron(weights, bias)

x = np.array([2, 3])       # x1 = 2, x2 = 3
print(n.feedforward(x))    # 0.99908894880559940.9990889488055994

Recognize those numbers? That’s the example we just did! We get the same answer of $0.999$._____no_output_____## 2. Combining Neurons into a Neural Network

A neural network is nothing more than a bunch of neurons connected together. Here’s what a simple neural network might look like:

![simple_NN](./img/simple_NN.png)

This network has 2 inputs, a hidden layer with 2 neurons ($h_{1}$  and $h_{2}$), and an output layer with 1 neuron ($o_{1}$). Notice that the inputs for $o_{1}$ are the outputs from $h_{1}$ and $h_{2}$ - that’s what makes this a network.

A hidden layer is any layer between the input (first) layer and output (last) layer. There can be multiple hidden layers!_____no_output_____## An Example: Feedforward

Let’s use the network pictured above and assume all neurons have the same weights $w = [0, 1]$, the same bias $b = 0$, and the same sigmoid activation function. Let $h_{1}$, $h_{2}$, $o_{1}$ denote the outputs of the neurons they represent.

What happens if we pass in the input $x = [2, 3]$?

$h_{1} = h_{2} = f(w \cdot x + b) = f((0\star2)+(1\star3)+0) = f(3) = 0.9526$

$o_{1} = f(w \cdot [h_{1}, h_{2}] + b) = f((0 \star h_{1}) + (1 \star h_{2}) + 0) = f(0.9526) = 0.7216$

The output of the neural network for input $x = [2, 3]$ is $0.7216$.

A neural network can have any number of layers with any number of neurons in those layers. The basic idea stays the same: feed the input(s) forward through the neurons in the network to get the output(s) at the end. For simplicity, we’ll keep using the network pictured above for the rest of this post._____no_output_____### Coding a Neural Network: Feedforward_____no_output_____

class OurNeuralNetwork:
  '''
  A neural network with:
    - 2 inputs
    - a hidden layer with 2 neurons (h1, h2)
    - an output layer with 1 neuron (o1)
  Each neuron has the same weights and bias:
    - w = [0, 1]
    - b = 0
  '''
  def __init__(self):
    weights = np.array([0, 1])
    bias = 0

    # The Neuron class here is from the previous section
    self.h1 = Neuron(weights, bias)
    self.h2 = Neuron(weights, bias)
    self.o1 = Neuron(weights, bias)

  def feedforward(self, x):
    out_h1 = self.h1.feedforward(x)
    out_h2 = self.h2.feedforward(x)

    # The inputs for o1 are the outputs from h1 and h2
    out_o1 = self.o1.feedforward(np.array([out_h1, out_h2]))

    return out_o1

network = OurNeuralNetwork()
x = np.array([2, 3])
print(network.feedforward(x)) # 0.72163256095184210.7216325609518421

We got 0.7216 !_____no_output_____## 3. Training a Neural Network, Part 1

Say we have the following measurements:

![measurements](./img/measurements.png)

Let’s train our network to predict someone’s gender given their weight and height:

![predict_gender](./img/predict_gender.png)

We’ll represent Male with a 00 and Female with a 11, and we’ll also shift the data to make it easier to use:

![measurements_numbers](./img/measurements_numbers.png)

_____no_output_____### Loss

Before we train our network, we first need a way to quantify how “good” it’s doing so that it can try to do “better”. That’s what the loss is.

We’ll use the mean squared error (MSE) loss:

$MCE = \frac{1}{n}\sum_i^n(y_{true} - y_{pred})^{2}$

Let’s break this down:

* n is the number of samples, which is 44 (Alice, Bob, Charlie, Diana).
* y represents the variable being predicted, which is Gender.
* $y_{true}$ is the true value of the variable (the “correct answer”). For example, $y_{true}$ for Alice would be 1 (Female).
* $y_{pred}$ is the predicted value of the variable. It’s whatever our network outputs.

$(y_{true} - y_{pred})^{2}$ is known as the squared error. Our loss function is simply taking the average over all squared errors (hence the name mean squared error). The better our predictions are, the lower our loss will be!

Better predictions = Lower loss.

Training a network = trying to minimize its loss.

_____no_output_____### An Example Loss Calculation

Let’s say our network always outputs 00 - in other words, it’s confident all humans are Male 🤔. What would our loss be?

![example_loss_calculation](./img/example_loss_calculation.png)

$MCE = \frac{1}{n}(1 + 0 + 0 + 1) = 0.5$_____no_output_____### Code: MSE Loss_____no_output_____

def mse_loss(y_true, y_pred):
  # y_true and y_pred are numpy arrays of the same length.
  return ((y_true - y_pred) ** 2).mean()

y_true = np.array([1, 0, 0, 1])
y_pred = np.array([0, 0, 0, 0])

print(mse_loss(y_true, y_pred)) # 0.50.5

## 4. Training a Neural Network, Part 2

We now have a clear goal: minimize the loss of the neural network. We know we can change the network’s weights and biases to influence its predictions, but how do we do so in a way that decreases loss?

For simplicity, let’s pretend we only have Alice in our dataset:

![only_Alice](./img/only_Alice.png)

Then the mean squared error loss is just Alice’s squared error:

$MCE = \frac{1}{1}\sum_i^n(y_{true} - y_{pred})^{2} = (y_{true} - y_{pred})^{2} = (1 - y_{pred})^{2}$

Another way to think about loss is as a function of weights and biases. Let’s label each weight and bias in our network:

![predict_gender_with_w_and_b](./img/predict_gender_with_w_and_b.png)

Then, we can write loss as a multivariable function:

$L(w_{1}, w_{2}, w_{3}, w_{4}, w_{5}, w_{6}, b_{1}, b_{2}, b_{3})$

Imagine we wanted to tweak $w_{1}$. How would loss $L$ change if we changed $w_{1}$?

That’s a question the partial derivative $\frac{\partial L}{\partial w_{1}}$ can answer. How do we calculate it?_____no_output_____To start, let’s rewrite the partial derivative in terms of $\frac{\partial y_{pred}}{\partial w_{1}}$ instead:

$\frac{\partial L}{\partial w_{1}} = \frac{\partial L}{\partial y_{pred}} \star \frac{\partial y_{pred}}{\partial w_{1}}$

(This works because of the Chain Rule.)

We can calculate $\frac{\partial L}{\partial y_{pred}}$ because we computed $L = (1 - y_{pred})^{2}$ above:

$\frac{\partial L}{\partial y_{pred}} = \frac{\partial (1 - y_{pred})^{2}}{\partial y_{pred}} = -2(1 - y_{pred})$

Now, let’s figure out what to do with $\frac{\partial y_{pred}}{\partial w_{1}}$. Just like before, let $h_{1}, h_{2}, o_{1}$ be the outputs of the neurons they represent. Then

$y_{pred} = o_{1} = f(w_{5}h_{1} + w_{6}h_{2} + b_{3})$
(f is the sigmoid activation function)

Since $w_{1}$ only affects $h_{1}$ (not $h_{2}$), we can write

$\frac{\partial y_{pred}}{\partial w_{1}} = \frac{\partial y_{pred}}{\partial h_{1}} \star \frac{\partial h_{1}}{\partial w_{1}}$

$\frac{\partial y_{pred}}{\partial h_{1}} = w_{5} \star f^{'}(w_{5}h_{1} + w_{6}h_{2} + b_{3})$

(More Chain Rule.)

We do the same thing for $\frac{\partial h_{1}}{\partial w_{1}}$:

$h_{1} = f(w_{1}x_{1} + w_{2}x_{2} + b_{1})$

$\frac{\partial h_{1}}{\partial w_{1}} = x_{1} \star f^{'}(w_{1}x_{1} + w_{2}x_{2} + b_{1})$

(You guessed it, Chain Rule.)

$x_{1}$ here is weight, and $x_{2}$ is height. This is the second time we’ve seen $f^{'}(x)$ (the derivate of the sigmoid function) now! Let’s derive it:

$f(x) = \frac{1}{1 + e^{-x}}$

$f^{'}(x) = \frac{e^{-x}}{(1 + e^{-x})^{2}} = f(x) \star (1 - f(x))$

We’ll use this nice form for $f^{'}(x)$ later.

We’re done! We’ve managed to break down $\frac{\partial L}{\partial w_{1}}$ into several parts we can calculate:

$\frac{\partial L}{\partial w_{1}} = \frac{\partial L}{\partial y_{pred}} \star \frac{\partial y_{pred}}{\partial h_{1}} \star \frac{\partial h_{1}}{\partial w_{1}}$

This system of calculating partial derivatives by working backwards is known as backpropagation, or “backprop”.

Let’s do an example to see this in action!_____no_output_____### Example: Calculating the Partial Derivative

We’re going to continue pretending only Alice is in our dataset:

![only_Alice](./img/only_Alice.png)_____no_output_____### Code: A Complete Neural Network_____no_output_____

def sigmoid(x):
  # Sigmoid activation function: f(x) = 1 / (1 + e^(-x))
  return 1 / (1 + np.exp(-x))

def deriv_sigmoid(x):
  # Derivative of sigmoid: f'(x) = f(x) * (1 - f(x))
  fx = sigmoid(x)
  return fx * (1 - fx)

def mse_loss(y_true, y_pred):
  # y_true and y_pred are numpy arrays of the same length.
  return ((y_true - y_pred) ** 2).mean()

class OurNeuralNetwork:
  '''
  A neural network with:
    - 2 inputs
    - a hidden layer with 2 neurons (h1, h2)
    - an output layer with 1 neuron (o1)

  *** DISCLAIMER ***:
  The code below is intended to be simple and educational, NOT optimal.
  Real neural net code looks nothing like this. DO NOT use this code.
  Instead, read/run it to understand how this specific network works.
  '''
  def __init__(self):
    # Weights
    self.w1 = np.random.normal()
    self.w2 = np.random.normal()
    self.w3 = np.random.normal()
    self.w4 = np.random.normal()
    self.w5 = np.random.normal()
    self.w6 = np.random.normal()

    # Biases
    self.b1 = np.random.normal()
    self.b2 = np.random.normal()
    self.b3 = np.random.normal()

  def feedforward(self, x):
    # x is a numpy array with 2 elements.
    h1 = sigmoid(self.w1 * x[0] + self.w2 * x[1] + self.b1)
    h2 = sigmoid(self.w3 * x[0] + self.w4 * x[1] + self.b2)
    o1 = sigmoid(self.w5 * h1 + self.w6 * h2 + self.b3)
    return o1

  def train(self, data, all_y_trues):
    '''
    - data is a (n x 2) numpy array, n = # of samples in the dataset.
    - all_y_trues is a numpy array with n elements.
      Elements in all_y_trues correspond to those in data.
    '''
    learn_rate = 0.1
    epochs = 1000 # number of times to loop through the entire dataset

    for epoch in range(epochs):
      for x, y_true in zip(data, all_y_trues):
        # --- Do a feedforward (we'll need these values later)
        sum_h1 = self.w1 * x[0] + self.w2 * x[1] + self.b1
        h1 = sigmoid(sum_h1)

        sum_h2 = self.w3 * x[0] + self.w4 * x[1] + self.b2
        h2 = sigmoid(sum_h2)

        sum_o1 = self.w5 * h1 + self.w6 * h2 + self.b3
        o1 = sigmoid(sum_o1)
        y_pred = o1

        # --- Calculate partial derivatives.
        # --- Naming: d_L_d_w1 represents "partial L / partial w1"
        d_L_d_ypred = -2 * (y_true - y_pred)

        # Neuron o1
        d_ypred_d_w5 = h1 * deriv_sigmoid(sum_o1)
        d_ypred_d_w6 = h2 * deriv_sigmoid(sum_o1)
        d_ypred_d_b3 = deriv_sigmoid(sum_o1)

        d_ypred_d_h1 = self.w5 * deriv_sigmoid(sum_o1)
        d_ypred_d_h2 = self.w6 * deriv_sigmoid(sum_o1)

        # Neuron h1
        d_h1_d_w1 = x[0] * deriv_sigmoid(sum_h1)
        d_h1_d_w2 = x[1] * deriv_sigmoid(sum_h1)
        d_h1_d_b1 = deriv_sigmoid(sum_h1)

        # Neuron h2
        d_h2_d_w3 = x[0] * deriv_sigmoid(sum_h2)
        d_h2_d_w4 = x[1] * deriv_sigmoid(sum_h2)
        d_h2_d_b2 = deriv_sigmoid(sum_h2)

        # --- Update weights and biases
        # Neuron h1
        self.w1 -= learn_rate * d_L_d_ypred * d_ypred_d_h1 * d_h1_d_w1
        self.w2 -= learn_rate * d_L_d_ypred * d_ypred_d_h1 * d_h1_d_w2
        self.b1 -= learn_rate * d_L_d_ypred * d_ypred_d_h1 * d_h1_d_b1

        # Neuron h2
        self.w3 -= learn_rate * d_L_d_ypred * d_ypred_d_h2 * d_h2_d_w3
        self.w4 -= learn_rate * d_L_d_ypred * d_ypred_d_h2 * d_h2_d_w4
        self.b2 -= learn_rate * d_L_d_ypred * d_ypred_d_h2 * d_h2_d_b2

        # Neuron o1
        self.w5 -= learn_rate * d_L_d_ypred * d_ypred_d_w5
        self.w6 -= learn_rate * d_L_d_ypred * d_ypred_d_w6
        self.b3 -= learn_rate * d_L_d_ypred * d_ypred_d_b3

      # --- Calculate total loss at the end of each epoch
      if epoch % 10 == 0:
        y_preds = np.apply_along_axis(self.feedforward, 1, data)
        loss = mse_loss(all_y_trues, y_preds)
        print("Epoch %d loss: %.3f" % (epoch, loss))

# Define dataset
data = np.array([
  [-2, -1],  # Alice
  [25, 6],   # Bob
  [17, 4],   # Charlie
  [-15, -6], # Diana
])
all_y_trues = np.array([
  1, # Alice
  0, # Bob
  0, # Charlie
  1, # Diana
])

# Train our neural network!
network = OurNeuralNetwork()
network.train(data, all_y_trues)Epoch 0 loss: 0.353
Epoch 10 loss: 0.237
Epoch 20 loss: 0.146
Epoch 30 loss: 0.101
Epoch 40 loss: 0.075
Epoch 50 loss: 0.059
Epoch 60 loss: 0.047
Epoch 70 loss: 0.039
Epoch 80 loss: 0.033
Epoch 90 loss: 0.028
Epoch 100 loss: 0.025
Epoch 110 loss: 0.022
Epoch 120 loss: 0.020
Epoch 130 loss: 0.018
Epoch 140 loss: 0.016
Epoch 150 loss: 0.015
Epoch 160 loss: 0.013
Epoch 170 loss: 0.012
Epoch 180 loss: 0.012
Epoch 190 loss: 0.011
Epoch 200 loss: 0.010
Epoch 210 loss: 0.010
Epoch 220 loss: 0.009
Epoch 230 loss: 0.009
Epoch 240 loss: 0.008
Epoch 250 loss: 0.008
Epoch 260 loss: 0.007
Epoch 270 loss: 0.007
Epoch 280 loss: 0.007
Epoch 290 loss: 0.006
Epoch 300 loss: 0.006
Epoch 310 loss: 0.006
Epoch 320 loss: 0.006
Epoch 330 loss: 0.005
Epoch 340 loss: 0.005
Epoch 350 loss: 0.005
Epoch 360 loss: 0.005
Epoch 370 loss: 0.005
Epoch 380 loss: 0.005
Epoch 390 loss: 0.004
Epoch 400 loss: 0.004
Epoch 410 loss: 0.004
Epoch 420 loss: 0.004
Epoch 430 loss: 0.004
Epoch 440 loss: 0.004
Epoch 450 loss: 0.004
Epoch 460 loss: 0.004
Epoch 470 loss: 0.004
Epoch 480 loss: 0.003
Epoch 490 loss: 0.003
Epoch 500 loss: 0.003
Epoch 510 loss: 0.003
Epoch 520 loss: 0.003
Epoch 530 loss: 0.003
Epoch 540 loss: 0.003
Epoch 550 loss: 0.003
Epoch 560 loss: 0.003
Epoch 570 loss: 0.003
Epoch 580 loss: 0.003
Epoch 590 loss: 0.003
Epoch 600 loss: 0.003
Epoch 610 loss: 0.003
Epoch 620 loss: 0.003
Epoch 630 loss: 0.003
Epoch 640 loss: 0.003
Epoch 650 loss: 0.002
Epoch 660 loss: 0.002
Epoch 670 loss: 0.002
Epoch 680 loss: 0.002
Epoch 690 loss: 0.002
Epoch 700 loss: 0.002
Epoch 710 loss: 0.002
Epoch 720 loss: 0.002
Epoch 730 loss: 0.002
Epoch 740 loss: 0.002
Epoch 750 loss: 0.002
Epoch 760 loss: 0.002
Epoch 770 loss: 0.002
Epoch 780 loss: 0.002
Epoch 790 loss: 0.002
Epoch 800 loss: 0.002
Epoch 810 loss: 0.002
Epoch 820 loss: 0.002
Epoch 830 loss: 0.002
Epoch 840 loss: 0.002
Epoch 850 loss: 0.002
Epoch 860 loss: 0.002
Epoch 870 loss: 0.002
Epoch 880 loss: 0.002
Epoch 890 loss: 0.002
Epoch 900 loss: 0.002
Epoch 910 loss: 0.002
Epoch 920 loss: 0.002
Epoch 930 loss: 0.002
Epoch 940 loss: 0.002
Epoch 950 loss: 0.002
Epoch 960 loss: 0.002
Epoch 970 loss: 0.002
Epoch 980 loss: 0.002
Epoch 990 loss: 0.002

We can now use the network to predict genders:_____no_output_____

# Make some predictions
emily = np.array([-7, -3]) # 128 pounds, 63 inches
frank = np.array([20, 2])  # 155 pounds, 68 inches
print("Emily: %.3f" % network.feedforward(emily)) # 0.951 - F
print("Frank: %.3f" % network.feedforward(frank)) # 0.039 - MEmily: 0.967
Frank: 0.039
_____no_output_____