# Notebook from aedriam/SMB-GA
Path: Deep_Neuroevolution_SMB_general_first.ipynb

## Deep Neuroevolution Genetic Algorithm Approach to Super Mario Bros._____no_output_____This version is able to randomize the world and stage order of the game or evaluate over all stages sequentially while running each agent for multiple episodes each generation to specifically evolve for a better general Neural Mario._____no_output_____

# Boolean check
COLAB = 'google.colab' in str(get_ipython())_____no_output_____

## Google Colab specific code_____no_output_____Note that the realtime updating of evaluation statistics will not be shown in Google Colab as it does not properly support the carriage return used in this project._____no_output_____

if COLAB:
    !pip install gym-super-mario-bros
    !mkdir Output
    !mkdir .video_____no_output_____if COLAB:
    # Connect Google Drive
    from google.colab import drive
    #drive.mount('/content/gdrive')
    drive.mount('/content/gdrive', force_remount=True)
    print('Google Drive connected.')_____no_output_____if COLAB:
    # Copy over the fittest NeuralMario
    !cp "gdrive/MyDrive/SMB Genetic General/Output/fittestMario.pkl" -r "Output"_____no_output_____if COLAB:
    # Create a virtual display for video rendering on the headless server.
    !apt-get install -y xvfb python-opengl > /dev/null 2>&1
    !pip install gym pyvirtualdisplay > /dev/null 2>&1
    from pyvirtualdisplay import Display
    display = Display(visible=0, size=(400, 400))_____no_output_____

## Regular Code_____no_output_____

import numpy as np
import torch
import matplotlib.pyplot as plt
import time

import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

import math
import copy
import sys
import os
import pickle
import time

import gym_super_mario_bros
from nes_py.wrappers import JoypadSpace
from gym_super_mario_bros.actions import SIMPLE_MOVEMENT
from gym.wrappers import Monitor
action_count = 7 # SIMPLE_MOVEMENT
import cv2_____no_output_____

### Model Definitions_____no_output_____

# Model Definitions
class Flatten(torch.nn.Module):
    def forward(self, x):
        return x.view(x.size()[0], -1)

class NeuralMario(nn.Module):
        def __init__(self, action_count):
            super().__init__()
            self.cuda()
            self.fc = nn.Sequential(
                        nn.Conv2d(4, 8, 3, bias=True),
                        nn.ReLU(inplace=True),
                        nn.Conv2d(8, 6, 3, bias=True),
                        nn.ReLU(inplace=True),
                        Flatten(),
                        nn.Linear(4704, 32, bias=True),
                        nn.ReLU(inplace=True),
                        nn.Linear(32, action_count, bias=True),
                        nn.Softmax(dim=1)
                        )

        def forward(self, inputs):
            x = self.fc(inputs)
            return x

def init_weights(m):

        # nn.Conv2d weights are of shape [16, 1, 3, 3] i.e. # number of filters, 1, stride, stride
        # nn.Conv2d bias is of shape [16] i.e. # number of filters

        # nn.Linear weights are of shape [32, 24336] i.e. # number of input features, number of output features
        # nn.Linear bias is of shape [32] i.e. # number of output features

        if ((type(m) == nn.Linear) | (type(m) == nn.Conv2d)):
            torch.nn.init.xavier_uniform(m.weight)
            m.bias.data.fill_(0.00)
_____no_output_____

### Environment Handling_____no_output_____

# Environment Handling
def convert_image(input_image):
    image = cv2.cvtColor(input_image, cv2.COLOR_RGB2GRAY)
    return cv2.resize(image, (32,32))

def run_agent(agent, envs, agent_num, num_agents, rendering=False, print_reward=False, episodes=2):

    global_reward = 0

    for i, env in enumerate(envs):
        episode_reward = 0

        agent.eval()

        state = env.reset()
        if rendering:
            env.render()

        #Conv2d without flatten()
        state = convert_image(state)#.flatten()
        state_list = [state, state, state, state]
        position = -1

        s=0

        while True:
            #Conv2d input
            input = torch.from_numpy(np.array(state_list)).type('torch.FloatTensor')\
                .unsqueeze(0)

            #Linear input
            #input = torch.tensor(state_list).type("torch.FloatTensor").view(1,-1)

            output_probabilities = agent(input).detach().numpy()[0]
            action = np.random.choice(range(action_count), 1, \
                p=output_probabilities).item()
            try:
                new_state, reward, done, info = env.step(action)
            except:
                break
            episode_reward += reward

            s=s+1
            if rendering:
                env.render()

            state_list.pop()
            #Conv2d without flatten()
            state_list.append(convert_image(new_state))#.flatten())

            # if mario gets stuck, it gets punished and the loop gets broken
            if position == info["x_pos"]:
                stuck += 1
                if stuck == 100:
                    episode_reward -= 100
                    break
            else:
                stuck = 0

            position = info["x_pos"]
            #env.render()
            #Mario died
            if info["life"] < 2:
                break

        env.reset()

        print("Agent: " + str(agent_num+1) + "/" + str(num_agents) +\
              "\tEpisode: " + str(i+1) + "/" + str(episodes) + "\tReward: "\
              + str(episode_reward) + "\t", end = '\r')
        sys.stdout.flush()

        global_reward += episode_reward

    mean_reward = global_reward / episodes

    if print_reward:
        print("Total:" + str(global_reward) + "\tMean: " + str(mean_reward))

    return mean_reward
_____no_output_____def run_agents_n_times(agents, envs, runs, random=True):
    avg_score = []
    num_agents = len(agents)

    for i, agent in enumerate(agents):
        avg_score.append(run_agent(agent, envs, episodes=runs, agent_num=i, num_agents=num_agents))

    return avg_score

def return_random_agents(num_agents):
    agents = []
    for _ in range(num_agents):

        agent = NeuralMario(action_count)

        for param in agent.parameters():
            param.requires_grad = False

        init_weights(agent)
        agents.append(agent)

    return agents_____no_output_____def test_agent(agent, rendering=True, monitoring=True, print_reward=True, world=1, stage=1):

    env = gym_super_mario_bros.make("SuperMarioBros-" + str(world) + "-" + str(stage) + "-v0")
    env = JoypadSpace(env, SIMPLE_MOVEMENT)

    if monitoring:
        env = Monitor(env, './video', force=True)

    env.seed(42)

    agent.eval()

    state = env.reset()
    if rendering:
        env.render()

    #Conv2d without flatten()
    state = convert_image(state)#.flatten()
    state_list = [state, state, state, state]
    position = -1

    global_reward = 0
    s=0

    for _ in range(30000):
        #Conv2d input
        input = torch.from_numpy(np.array(state_list)).type('torch.FloatTensor')\
                .unsqueeze(0)

        #Linear input
        #input = torch.tensor(state_list).type("torch.FloatTensor").view(1,-1)

        output_probabilities = agent(input).detach().numpy()[0]
        action = np.random.choice(range(action_count), 1, \
            p=output_probabilities).item()
        try:
            new_state, reward, done, info = env.step(action)
        except:
            break
        global_reward += reward

        s=s+1
        if rendering:
            env.render()

        state_list.pop()
        #Conv2d without flatten()
        state_list.append(convert_image(new_state))#.flatten())

        # if mario gets stuck, it gets punished and the loop gets broken
        if position == info["x_pos"]:
            stuck += 1
            if stuck == 100:
                global_reward -= 100
                break
        else:
            stuck = 0

        position = info["x_pos"]
        #env.render()
        #Mario died
        if info["life"] < 2:
            break

    return global_reward
_____no_output_____def return_children(agents, sorted_parent_indexes, elite_count, mutation_power, mutation_chance):
    """ Returning [N-elite_count] mutated agents from sorted_parent_indexes and
        keeping the best [elite_count] agents unchanged
    """

    print("Mutating " + str(elite_count) + " elite agent(s) into the next generation.", end='\r')

    children_agents = []

    for i in range(len(agents)-elite_count):
        selected_agent_index = sorted_parent_indexes[np.random.randint(len(sorted_parent_indexes))]
        children_agents.append(mutate(agents[selected_agent_index], mutation_power, mutation_chance))

    elite_children = []
    for i in range(elite_count):
        elite_children.append(agents[sorted_parent_indexes[i]])

    children_agents.extend(elite_children)

    return children_agents

def return_hot_start(elite_agent, num_agents, mutation_power, mutation_chance):
    """ Mutate from a single saved elite agent, or fittestMario, in order to
        start evolving a generation immediately without waiting for a single
        generation of elite copies to process.

        Only recommended when only a single elite agent is preserved across
        generations.
    """

    children_agents = []

    for i in range(num_agents-1):
        children_agents.append(mutate(elite_agent, mutation_power, mutation_chance))

    children_agents.extend([elite_agent])

    return children_agents
    _____no_output_____def mutate(agent, power=0.2, chance=0.02):
    ''' Simple method to add gaussian noise to the agents '''

    child_agent = copy.deepcopy(agent)

    mutation_power = power #hyper-parameter, 0.2 set from https://arxiv.org/pdf/1712.06567.pdf

    # Current checks for mutation: 33189
    mutation_chance = chance

    for param in child_agent.parameters():

        if(len(param.shape)==4): #weights of Conv2D
            for i0 in range(param.shape[0]):
                for i1 in range(param.shape[1]):
                    for i2 in range(param.shape[2]):
                        if np.random.random(1)[0] <= mutation_chance:
                            for i3 in range(param.shape[3]):
                                param[i0][i1][i2][i3]+= mutation_power * np.random.randn()
                        else:
                            pass

        elif(len(param.shape)==2): #weights of linear layer
            for i0 in range(param.shape[0]):
                for i1 in range(param.shape[1]):
                    if np.random.random(1)[0] <= mutation_chance:
                        param[i0][i1]+= mutation_power * np.random.randn()
                    else:
                        pass

        elif(len(param.shape)==1): #biases of linear layer or conv layer
            for i0 in range(param.shape[0]):
                if np.random.random(1)[0] <= mutation_chance:
                    param[i0]+=mutation_power * np.random.randn()
                else:
                    pass

    return child_agent
_____no_output_____def main(elite_agent, mutation_power, mutation_chance, episodes, agents=10, elites=1, show_top=2, generations=100, hot_start=False, random=True):
    torch.set_grad_enabled(False)
    num_agents = agents
    elite_count = elites
    top_limit_count = show_top
    n_episodes = episodes

    generation_count = generations

    if (elite_agent != None) and (not hot_start):
        print("Loaded " + str(num_agents) + " elite copies.")
        agents = [elite_agent]*num_agents
    elif (elite_agent != None) and hot_start:
        print("Hot start enabled. Mutating loaded elite " + str(num_agents-1) + " times.")
        agents = return_hot_start(elite_agent, num_agents, mutation_power, mutation_chance)
    elif (elite_agent == None) and hot_start:
        print("Hot start enabled without required elite agent.")
        print("Creating " + str(num_agents) + " random agents.")
        agents = return_random_agents(num_agents)
    else:
        print("Creating " + str(num_agents) + " random agents.")
        agents = return_random_agents(num_agents)

    # Create all environments to use throughout the generation loop
    envs = []

    worlds = []
    stages = []

    if not random:
        for i in range(8):
            for j in range(4):
                worlds.append(i+1)
                stages.append(j+1)

        for i in range(len(worlds)):
            y = str(worlds[i])
            z = str(stages[i])
            env = gym_super_mario_bros.make("SuperMarioBros-" + y + "-" + z + "-v0")
            env = JoypadSpace(env, SIMPLE_MOVEMENT)
            env.seed(42)

            envs.append(env)

    for generation in range(generation_count):
        print("############################### Generation {} ###############################".format(generation+1))

        if random:
            envs = []
            for i in range(n_episodes):
                worlds.append(np.random.randint(1, 9))
                stages.append(np.random.randint(1, 5))

            for i in range(len(worlds)):
                y = str(worlds[i])
                z = str(stages[i])
                env = gym_super_mario_bros.make("SuperMarioBros-" + y + "-" + z + "-v0")
                env = JoypadSpace(env, SIMPLE_MOVEMENT)
                env.seed(42)

                envs.append(env)

        rewards = run_agents_n_times(agents, envs, n_episodes, random)

        sorted_parent_indexes = np.argsort(rewards)[::-1][:top_limit_count]

        top_rewards = []
        for best_parent in sorted_parent_indexes:
            top_rewards.append(rewards[best_parent])

        print("Mean reward: {}\t\tMean of top 5: {}".format(\
            int(np.mean(rewards)), int(np.mean(top_rewards[:5]))))
        print("Top agents: {}\tReward: {}".format(sorted_parent_indexes, \
            top_rewards))
        print("#############################################################################\n")
        sys.stdout.flush()

        with open("Output/fittestMario.pkl", 'wb') as output:
            pickle.dump(agents[sorted_parent_indexes[0]], output, \
                pickle.HIGHEST_PROTOCOL)

        children_agents = return_children(agents, sorted_parent_indexes, elite_count, mutation_power, mutation_chance)

        agents = children_agents

        if COLAB:
            !cp "Output" -r "gdrive/MyDrive/SMB Genetic General/"
_____no_output_____

### Model Loading and Training
The training function is defined here.
Control Arguments:
* training: If True, the genetic algorithm evolves the neural networks, If False, the best agent is loaded and plays a game, with the gym monitor recording the video.

* load_elite: If True, the previous elite agent is loaded from file. If False, all previous trainings are ignored and overwritten.

* agents: The number of agents in the population.

* elites: The number of elite agents to carry over across generations, and to mutate from.

* show_top: The number of top agent rewards scores to display in the evolution output.

* generations: The number of generations to create and test.

* mutation_power: The Gaussian noise multiplier for the mutation function to use.

* mutation_chance: The percentage chance for a genotype to undergo mutation within a generation creation.

* episodes: The number of game runs to use when evaluating the performance of each agent within a generation.

* hot_start: Not recommended if implementing multiple-agent elitism. If True, create the first generation via mutations of the single loaded elite agent from a previous session. If False, make copies of the loaded elite agent if available or create a generation of completely random agents.

* random: If True, randomly sample stages across all worlds for the evaluation of each agent each generation. If False, evaluate via every stage in the game sequentially._____no_output_____

# Model Training
def train(training=True, load_elite=True, agents=10, elites=1, show_top=2, generations=100, mutation_power=0.2, mutation_chance=0.02, episodes=2, hot_start=False, random=True):
    if training:

        elite_agent = None
        if load_elite:
            with open("Output/fittestMario.pkl", 'rb') as input:
                elite_agent = pickle.load(input)

        # If not randomly sampling, all 32 stages are sequentially tested.
        if not random:
            print()
            episodes = 32

        main(elite_agent, mutation_power, mutation_chance, episodes, agents, elites, show_top, generations, hot_start, random)

    else:
        with open("Output/fittestMario.pkl", 'rb') as input:
            fittest_mario = pickle.load(input)
            test_agent(fittest_mario, True, True, True)
_____no_output_____# train, load, agents, elite, display top, generations, mutation_power, mutation_chance, episodes per generation, hot_start, randomize stages
train(True, True, 100, 1, 5, 50, 0.2, 0.025, 32, True, False)_____no_output_____

This will now run the best neural network one more time, wrapped in the monitor environment to record video of play._____no_output_____

if COLAB:
    try:
        display.start()
    except:
        pass

with open("Output/fittestMario.pkl", 'rb') as input:
    fittest_mario = pickle.load(input)
    test_agent(fittest_mario, True, True, True, 8, 3)_____no_output_____if COLAB:
    !cp "video" -r "gdrive/MyDrive/SMB Genetic General/"_____no_output_____


# Notebook from fhalbritter/pypiper
Path: docs_jupyter/basic-pipeline.ipynb

# Your first basic pipeline

## The basic functions

Pypiper is simple but powerful. Your pipeline is a python script, let's call it `pipeline.py`. You really only need to know about 3 commands to get started:

* `PipelineManager()`: PipelineManager constructor, initializes the pipeline.
* `PipelineManager.run()`: The primary workhorse function; runs a command.
* `PipelineManager.stop_pipeline()`: Terminate the pipeline.

That's all you need to create a powerful pipeline. You can find in-depth reference documentation for each method in the API. In particular, most of Pypiper’s power comes from the `run` method, which has a series of options outlined in [dedicated documentation on the run method](/advanced-run-method).

To write your first basic pipeline, first `import pypiper`, then specify an output folder and create a new `PipelineManager` object:

	#!/usr/bin/env python
	import pypiper, os
	outfolder = "pipeline_output/"  # Choose a folder for your results
	pm = pypiper.PipelineManager(name="my_pipeline", outfolder=outfolder)

Creating the `PipelineManger` object creates your `outfolder` and places a flag called `my_pipeline_running.flag` in the folder. It also initializes the log file (``my_pipeline_log.md``) with statistics such as time of starting, compute node, software versions, command-line parameters, etc.

Now, the workhorse of `PipelineManager` is the `run()` function. Essentially, you create a shell command as a string in python, and then pass it and its target (a file it creates) to `run()`. The target is the final output file created by your command. Let's use the built-in `shuf` command to create some random numbers and put them in a file called `outfile.txt`:

	# our command will produce this output file
	target = os.path.join(outfolder, "outfile.txt")
	command = "shuf -i 1-500000000 -n 10000000 > " + target
	pm.run(command, target)

The command (`command`) is the only required argument to `run()`. You can leave `target` empty (pass `None`). If you **do** specify a target, the command will only be run if the target file does not already exist. If you **do not** specify a target, the command will be run every time the pipeline is run.

Now string together whatever commands your pipeline requires! At the end, terminate the pipeline so it gets flagged as successfully completed:

	pm.stop_pipeline()

That's it! By running commands through `run()` instead of directly in bash, you get a robust, logged, restartable pipeline manager for free!
_____no_output_____
## Advanced functions
Once you've mastered the basics, add in a few other commands that make debugging and development easier:

* `timestamp()`: Print message, time, and time elapsed, perhaps creating checkpoint.
* `clean_add()`: Add files (or regexs) to a cleanup list, to delete when this pipeline completes successfully.

## Reporting results
`Pypiper` also has a really nice system for collecting and aggregating result statistics and pipeline outputs.

* `report_result()`: Writes a line in the `stats.tsv` file with a key-value pair of some pipeline result.
* `report_object()`: Writes a line in the `objects.tsv` file with a key-value pair pointing to any file (such as an image, HTML report, or other) created by the pipeline.
* `get_stat()`:	Returns a stat that was previously reported.

## Putting them together in `basic.py`
Now, download [basic.py](https://github.com/databio/pypiper/blob/master/example_pipelines/basic.py) and run it with `python basic.py` (or, better yet, make it executable (`chmod 755 basic.py`) and then run it directly with `./basic.py`). This example is a documented vignette; so just read it and run it to get an idea of how things work. Here are the contents of `basic.py`:_____no_output_____

# %load ../example_pipelines/basic.py
#!/usr/bin/env python

"""Getting Started: A simple sample pipeline built using pypiper."""

# This is a runnable example. You can run it to see what the output
# looks like.

# First, make sure you can import the pypiper package

import os
import pypiper

# Create a PipelineManager instance (don't forget to name it!)
# This starts the pipeline.

pm = pypiper.PipelineManager(name="BASIC",
    outfolder="pipeline_output/")

# Now just build shell command strings, and use the run function
# to execute them in order. run needs 2 things: a command, and the
# target file you are creating.

# First, generate some random data

# specify target file:
tgt = "pipeline_output/test.out"

# build the command
cmd = "shuf -i 1-500000000 -n 10000000 > " + tgt

# and run with run().
pm.run(cmd, target=tgt)

# Now copy the data into a new file.
# first specify target file and build command:
tgt = "pipeline_output/copied.out"
cmd = "cp pipeline_output/test.out " + tgt
pm.run(cmd, target=tgt)

# You can also string multiple commands together, which will execute
# in order as a group to create the final target.
cmd1 = "sleep 5"
cmd2 = "touch pipeline_output/touched.out"
pm.run([cmd1, cmd2], target="pipeline_output/touched.out")

# A command without a target will run every time.
# Find the biggest line
cmd = "awk 'n < $0 {n=$0} END{print n}' pipeline_output/test.out"
pm.run(cmd, "lock.max")

# Use checkprint() to get the results of a command, and then use
# report_result() to print and log key-value pairs in the stats file:
last_entry = pm.checkprint("tail -n 1 pipeline_output/copied.out")
pm.report_result("last_entry", last_entry)

# Now, stop the pipeline to complete gracefully.
pm.stop_pipeline()

# Observe your outputs in the pipeline_output folder
# to see what you've created.
_____no_output_____

# Building a pipeline to count number of reads

Here we have a more advanced bioinformatics pipeline that adds some new concepts. This is a simple script that takes an input file and returns the file size and the number of sequencing reads in that file. This example uses a function from from the built-in :doc:`NGSTk toolkit <ngstk>`. In particular, this toolkit contains a few handy functions that make it easy for a pipeline to accept inputs of various types. So, this pipeline can count the number of reads from files in ``BAM`` format, or ``fastq`` format, or ``fastq.gz`` format. You can also use the same functions from NGSTk to develop a pipeline to do more complicated things, and handle input of any of these types.

First, grab this pipeline. Download [count_reads.py](https://github.com/databio/pypiper/blob/master/example_pipelines/count_reads.py), make it executable (`chmod 755 count_reads.py`), and then run it with `./count_reads.py`).

You can grab a few small data files in the [microtest repository](https://github.com/epigen/microtest/tree/master/config). Run a few of these files like this:

```
./count_reads.py -I ~/code/microtest/data/rrbs_PE_R1.fastq.gz -O $HOME -S sample1
./count_reads.py -I ~/code/microtest/data/rrbs_PE_fq_R1.fastq -O $HOME -S sample2
./count_reads.py -I ~/code/microtest/data/atac-seq_SE.bam -O $HOME -S sample3
```

This example is a documented vignette; so just read it and run it to get an idea of how things work.
_____no_output_____

# %load ../example_pipelines/count_reads.py
#!/usr/bin/env python

"""
Counts reads.
"""

__author__ = "Nathan Sheffield"
__email__ = "nathan@code.databio.org"
__license__ = "GPL3"
__version__ = "0.1"

from argparse import ArgumentParser
import os, re
import sys
import subprocess
import yaml
import pypiper

parser = ArgumentParser(
    description="A pipeline to count the number of reads and file size. Accepts"
    " BAM, fastq, or fastq.gz files.")

# First, add standard arguments from Pypiper.
# groups="pypiper" will add all the arguments that pypiper uses,
# and adding "common" adds arguments for --input and --sample--name
# and "output_parent". You can read more about your options for standard
# arguments in the pypiper docs (section "command-line arguments")
parser = pypiper.add_pypiper_args(parser, groups=["pypiper", "common", "ngs"],
                                    args=["output-parent", "config"],
                                    required=['sample-name', 'output-parent'])

# Add any pipeline-specific arguments if you like here.

args = parser.parse_args()

if not args.input or not args.output_parent:
    parser.print_help()
    raise SystemExit

if args.single_or_paired == "paired":
    args.paired_end = True
else:
    args.paired_end = False

# args for `output_parent` and `sample_name` were added by the standard
# `add_pypiper_args` function.
# A good practice is to make an output folder for each sample, housed under
# the parent output folder, like this:
outfolder = os.path.abspath(os.path.join(args.output_parent, args.sample_name))

# Create a PipelineManager object and start the pipeline
pm = pypiper.PipelineManager(name="count",
                             outfolder=outfolder,
                             args=args)

# NGSTk is a "toolkit" that comes with pypiper, providing some functions
# for dealing with genome sequence data. You can read more about toolkits in the
# documentation

# Create a ngstk object
ngstk = pypiper.NGSTk(pm=pm)

raw_folder = os.path.join(outfolder, "raw/")
fastq_folder = os.path.join(outfolder, "fastq/")

# Merge/Link sample input and Fastq conversion
# These commands merge (if multiple) or link (if single) input files,
# then convert (if necessary, for bam, fastq, or gz format) files to fastq.

# We'll start with a timestamp that will provide a division for this section
# in the log file
pm.timestamp("### Merge/link and fastq conversion: ")

# Now we'll rely on 2 NGSTk functions that can handle inputs of various types
# and convert these to fastq files.

local_input_files = ngstk.merge_or_link(
                        [args.input, args.input2],
                        raw_folder,
                        args.sample_name)

cmd, out_fastq_pre, unaligned_fastq = ngstk.input_to_fastq(
                                            local_input_files,
                                            args.sample_name,
                                            args.paired_end,
                                            fastq_folder)

# Now we'll use another NGSTk function to grab the file size from the input files
#
pm.report_result("File_mb", ngstk.get_file_size(local_input_files))

# And then count the number of reads in the file

n_input_files = len(filter(bool, local_input_files))

raw_reads = sum([int(ngstk.count_reads(input_file, args.paired_end))
                for input_file in local_input_files]) / n_input_files

# Finally, we use the report_result() function to print the output and
# log the key-value pair in the standard stats.tsv file
pm.report_result("Raw_reads", str(raw_reads))

# Cleanup
pm.stop_pipeline()
_____no_output_____


# Notebook from rob-dalton/amazon-product-recommender
Path: emr/src/scriptDev-processData.ipynb

# Script Development - processData.py
Development notebook for script to group by category and asin, split by pos and neg
reviews, and add top positive and negative features.

## Setup_____no_output_____

import pyspark as ps
from sentimentAnalysis import dataProcessing as dp_____no_output_____reload(dp)_____no_output_____# create spark session
spark = ps.sql.SparkSession.builder \
    .appName("reviewProcessing") \
    .getOrCreate()_____no_output_____# get dataframes
# specify s3 as sourc with s3a://
# df = spark.read.json("s3a://amazon-review-data/user_dedup.json.gz")
df_meta = spark.read.json("s3a://amazon-review-data/metadata.json.gz")_____no_output_____# get shard
df_mi = spark.read.json("s3a://amazon-review-data/reviews_Musical_Instruments_5.json.gz")_____no_output_____# subset asin, overall, , reviewerName, reviewText
df_subset = df_mi.select("asin", "overall", "reviewerName", "unixReviewTime", "reviewText")_____no_output_____# add metadata
df_joined = dp.join_metadata(df_subset, df_meta).select("asin",
                                                        "title",
                                                        "categories",
                                                        "overall",
                                                        "reviewerName",
                                                        "unixReviewTime",
                                                        "reviewText").persist()_____no_output_____

## Development_____no_output_____### Remove 3 star reviews_____no_output_____

# count reviews
df_joined.count()_____no_output_____# remove reviews
df_joined_subset = df_joined.where(df_joined.overall != 3.0)

# check reviews were removed
df_joined_subset.count()_____no_output_____

### Extract category_____no_output_____

# check df categories
df_joined_subset.show(3)+----------+--------------------+-------+--------------------+--------------+--------------------+
|      asin|          categories|overall|        reviewerName|unixReviewTime|          reviewText|
+----------+--------------------+-------+--------------------+--------------+--------------------+
|1384719342|[WrappedArray(Mus...|    5.0|       SEAN MASLANKA|    1392940800|This pop filter i...|
|1384719342|[WrappedArray(Mus...|    5.0|RustyBill "Sunday...|    1392336000|Nice windscreen p...|
|1384719342|[WrappedArray(Mus...|    5.0|Rick Bennette "Ri...|    1377648000|The primary job o...|
+----------+--------------------+-------+--------------------+--------------+--------------------+
only showing top 3 rows

from pyspark.sql.functions import udf
from pyspark.sql.types import ArrayType, StringType, NumericType, DateType_____no_output_____def extract_category(df_cats):
    """
    INPUT: Spark DataFrame
    RETURN: Spark DataFrame

    Takes in a DataFrame with a wrapped array "categories"
    column. Extracts first category, drops "categories"

    """
    # create udf
    extract_cat_udf = udf(lambda x: x[0][0], StringType())

    # create new column with single category
    df_cat = df_joined.withColumn("category", extract_cat_udf("categories")).drop("categories")

    return df_cat_____no_output_____# test function
df_cat = extract_category(df_joined_subset)

df_cat.show(3)+----------+-------+--------------------+--------------+--------------------+-------------------+
|      asin|overall|        reviewerName|unixReviewTime|          reviewText|           category|
+----------+-------+--------------------+--------------+--------------------+-------------------+
|1384719342|    5.0|       SEAN MASLANKA|    1392940800|This pop filter i...|Musical Instruments|
|1384719342|    5.0|RustyBill "Sunday...|    1392336000|Nice windscreen p...|Musical Instruments|
|1384719342|    5.0|Rick Bennette "Ri...|    1377648000|The primary job o...|Musical Instruments|
+----------+-------+--------------------+--------------+--------------------+-------------------+
only showing top 3 rows

### Separate pos and neg, add tfidf vectors_____no_output_____

from pyspark.sql.functions import lit

def add_pos_neg_tfidf(df_cat):
    """
    INPUT: Spark DataFrame
    RETURN: Spark DataFrame, List, List

    Takes in a DataFrame with review column overall.
    Splits into postitive and negative reviews, adds
    TFIDF vectors for each subset. Returns joined
    DataFrame with all data, list of postive review
    vocabulary terms, and list of negative review
    vocabulary terms

    """
    # separate positive and negative reviews
    df_pos = df_cat.where(df_cat.overall >= 4.0).withColumn("positive", lit(True))
    df_neg = df_cat.where(df_cat.overall <= 2.0).withColumn("positive", lit(False))

    # tokenize
    df_pos_tk = dp.add_tokens(df_pos).select("asin",
                                             "category",
                                             "overall",
                                             "positive",
                                             "reviewerName",
                                             "unixReviewTime",
                                             "reviewText",
                                             "tokens")

    df_neg_tk = dp.add_tokens(df_neg).select("asin",
                                             "category",
                                             "overall",
                                             "positive",
                                             "reviewerName",
                                             "unixReviewTime",
                                             "reviewText",
                                             "tokens")

    # get tf, vocab
    df_tf_pos, vocab_pos = dp.add_tf_and_vocab(df_pos_tk)
    df_tf_neg, vocab_neg = dp.add_tf_and_vocab(df_neg_tk)

    # add tfidf
    df_tfidf_pos = dp.add_tfidf(df_tf_pos).drop("tf_vector").drop("tokens")
    df_tfidf_neg = dp.add_tfidf(df_tf_neg).drop("tf_vector").drop("tokens")

    return df_tfidf_pos.unionAll(df_tfidf_neg), vocab_pos, vocab_neg_____no_output_____# test function
df_tfidf, vocab_pos, vocab_neg = add_pos_neg_tfidf(df_cat)

df_tfidf.show(3)+----------+-------------------+-------+--------+--------------------+--------------+--------------------+--------------------+
|      asin|           category|overall|positive|        reviewerName|unixReviewTime|          reviewText|        tfidf_vector|
+----------+-------------------+-------+--------+--------------------+--------------+--------------------+--------------------+
|1384719342|Musical Instruments|    5.0|    true|       SEAN MASLANKA|    1392940800|This pop filter i...|(19899,[2,3,117,1...|
|1384719342|Musical Instruments|    5.0|    true|RustyBill "Sunday...|    1392336000|Nice windscreen p...|(19899,[22,36,60,...|
|1384719342|Musical Instruments|    5.0|    true|Rick Bennette "Ri...|    1377648000|The primary job o...|(19899,[6,14,16,3...|
+----------+-------------------+-------+--------+--------------------+--------------+--------------------+--------------------+
only showing top 3 rows

## Group by product, collect list of reviews_____no_output_____#### Add review column_____no_output_____

# import collect_list function
from pyspark.sql.functions import collect_list
import json_____no_output_____# create json review from row
def rowToJson(rating, date, name, text):
    row = { "rating": rating,
            "date": date,
            "name": name,
            "text": text
          }

    return json.dumps(row)

# create review column
def add_review_col(df):
    # create udf
    get_review_udf = udf(lambda a,b,c,d: rowToJson(a,b,c,d), StringType())

    # create new column with review
    df_review = df.withColumn("review", get_review_udf("overall",
                                                       "unixReviewTime",
                                                       "reviewerName",
                                                       "reviewText"))

    return df_review_____no_output_____# test function
df_review = add_review_col(df_tfidf)

df_review.show(3)+----------+-------------------+-------+--------+--------------------+--------------+--------------------+--------------------+--------------------+
|      asin|           category|overall|positive|        reviewerName|unixReviewTime|          reviewText|        tfidf_vector|              review|
+----------+-------------------+-------+--------+--------------------+--------------+--------------------+--------------------+--------------------+
|1384719342|Musical Instruments|    5.0|    true|       SEAN MASLANKA|    1392940800|This pop filter i...|(19899,[2,3,117,1...|{"date": 13929408...|
|1384719342|Musical Instruments|    5.0|    true|RustyBill "Sunday...|    1392336000|Nice windscreen p...|(19899,[22,36,60,...|{"date": 13923360...|
|1384719342|Musical Instruments|    5.0|    true|Rick Bennette "Ri...|    1377648000|The primary job o...|(19899,[6,14,16,3...|{"date": 13776480...|
+----------+-------------------+-------+--------+--------------------+--------------+--------------------+--------------------+--------------------+
only showing top 3 rows

test_row = df_review.first()

test_row["review"]_____no_output_____

#### Group by asin, positive, then concatenate reviews, sum tfidf vectors_____no_output_____

# group by asin, sum tfidf_vectors
df_grouped = df_review.groupBy("asin", "category", "positive").agg(collect_list("review").alias("reviews"),
                              collect_list("tfidf_vector").alias("vectors"))

df_grouped.show(3)+----------+-------------------+--------+--------------------+--------------------+
|      asin|           category|positive|             reviews|             vectors|
+----------+-------------------+--------+--------------------+--------------------+
|B000068NW5|Musical Instruments|   false|[{"date": 1394755...|[(5030,[2,3,13,23...|
|B0002CZTIO|Musical Instruments|   false|[{"date": 1390608...|[(5030,[0,6,8,26,...|
|B0002E3DGC|Musical Instruments|   false|[{"date": 1332374...|[(5030,[0,1,8,15,...|
+----------+-------------------+--------+--------------------+--------------------+
only showing top 3 rows

# get test row
test_row = df_grouped.first()_____no_output_____test_vectors = test_row["vectors"]_____no_output_____test_vectors[:3]_____no_output_____from pyspark.mllib.linalg import SparseVector
import numpy as np

def sum_vectors(vectors):
    """
    INPUT: List of SparseVectors
    RETURN: SparseVector

    Sum list of TFIDF vectors element-wise,
    return resulting vector

    """
    # check if vectors exist
    if not vectors:
        return None

    # iterate over vectors
    sum_vector = vectors[0].toArray()
    vector_size = sum_vector.shape[0]

    for i,vector in enumerate(vectors[1:]):
        sum_vector += vector.toArray()

    # convert to sparse vector
    sparse_vector = SparseVector(vector_size, {i:sum_vector[i] for i in np.nonzero(sum_vector)[0]})

    return sparse_vector_____no_output_____# examine two vectors
test_vectors[0]_____no_output_____test_vectors[1]_____no_output_____# test sum of first two vectors
sum_vectors(test_vectors)_____no_output_____

#### Function to sum vectors over DataFrame_____no_output_____

from pyspark.mllib.linalg import VectorUDT

# sum vectors
def add_vectors_sum(df):
    # create udf
    sum_vector_udf = udf(lambda vectors: sum_vectors(vectors), VectorUDT())

    # create new column with review
    df_vectors_summed = df.withColumn("tfidf_vectors_sum", sum_vector_udf("vectors")).drop("vectors")

    return df_vectors_summed_____no_output_____# test
df_vectors_summed = add_vectors_sum(df_grouped)

df_vectors_summed.show(3)+----------+-------------------+--------+--------------------+--------------------+
|      asin|           category|positive|             reviews|   tfidf_vectors_sum|
+----------+-------------------+--------+--------------------+--------------------+
|B000068NW5|Musical Instruments|   false|[{"date": 1394755...|(5030,[0,1,2,3,6,...|
|B0002CZTIO|Musical Instruments|   false|[{"date": 1390608...|(5030,[0,6,8,26,4...|
|B0002E3DGC|Musical Instruments|   false|[{"date": 1332374...|(5030,[0,1,8,15,1...|
+----------+-------------------+--------+--------------------+--------------------+
only showing top 3 rows

### Add top terms_____no_output_____

# get top terms
"""
from sentimentAnalysis.dataProcessing import add_pos_neg_features
df_terms = add_pos_neg_features(df_vectors_summed, vocab_pos, vocab_neg, n=15).drop("tfidf_vectors_sum")
df_terms.show(3)
"""_____no_output_____def test_extract_top_features(tfidf_vector, vocab, n):
    """
    INPUT: SparseVector, List, Int
    RETURN: List

    Take in TFIDF vector, vocabulary for vector,
    and number of terms. Return top n terms

    """
    # note - tfidf elements are pre-sorted by importance
    term_indices = tfidf_vector.indices[-n:]

    # Map features to terms
    features = [vocab[i] for i in term_indices]

    return features

def test_add_top_features(df, vocab, n=10):
    """
    INPUT: PySpark DataFrame, List, Int
    RETURN: PySpark DataFrame

    Take in DataFrame with TFIDF vectors, list of vocabulary words,
    and number of features to extract. Map top features from TFIDF
    vectors to vocabulary terms. Return new DataFrame with terms

    """
    # Create udf function to extract top n features
    extract_features_udf = udf(lambda x: test_extract_top_features(x, vocab, n))

    # Apply udf, create new df with features column
    df_features = df.withColumn("topFeatures",
                                    extract_features_udf(df["tfidf_vectors_sum"]))

    return df_features

def test_add_pos_neg_features(df, vocab_pos, vocab_neg, n=10):
    """
    INPUT: Spark DataFrame, List, List, Int
    RETURN: Spark DataFrame

    Take in DataFrame grouped by asin, positive with tfidf vectors summed.
    Extract top positive and negative terms from each group, add features column

    """
    # split dataframe on postitive
    df_pos = df.where(df.positive==True)
    df_neg = df.where(df.positive==False)

    # add features
    df_pos_terms = test_add_top_features(df_pos, vocab_pos, n)
    df_neg_terms = test_add_top_features(df_neg, vocab_neg, n)

    return df_pos_terms.unionAll(df_neg_terms)_____no_output_____# add terms
# note - udf function that relies on local module functions does not work b/c module does not exist on workers

df_terms = test_add_pos_neg_features(df_vectors_summed, vocab_pos, vocab_neg, n=15).drop("tfidf_vectors_sum")_____no_output_____

### Reduce, add posFeatures and negFeatures_____no_output_____

df_terms.show(3)+----------+-------------------+--------+--------------------+--------------------+
|      asin|           category|positive|             reviews|         topFeatures|
+----------+-------------------+--------+--------------------+--------------------+
|B0014IEBM0|Musical Instruments|    true|[{"date": 1251072...|[portables, subtl...|
|B001EC5ECW|Musical Instruments|    true|[{"date": 1385337...|[recommendation, ...|
|B001L8IJ0I|Musical Instruments|    true|[{"date": 1362355...|[mg, dsl, glory, ...|
+----------+-------------------+--------+--------------------+--------------------+
only showing top 3 rows

# group by asin
df_terms_grouped = df_terms.groupBy("asin", "category").agg(collect_list("reviews").alias("reviews"),
                              collect_list("topFeatures").alias("topFeatures"))

df_terms_grouped.show(1)+----------+-------------------+--------------------+--------------------+
|      asin|           category|             reviews|         topFeatures|
+----------+-------------------+--------------------+--------------------+
|B0002CZSJY|Musical Instruments|[WrappedArray({"d...|[[usese, bubblych...|
+----------+-------------------+--------------------+--------------------+
only showing top 1 row

df_terms.select("asin",
                col("reviews").alias("negReviews"),
                col("topFeatures").alias("negFeatures")).show(3)+----------+--------------------+--------------------+
|      asin|          negReviews|         negFeatures|
+----------+--------------------+--------------------+
|B0014IEBM0|[{"date": 1251072...|[portables, subtl...|
|B001EC5ECW|[{"date": 1385337...|[recommendation, ...|
|B001L8IJ0I|[{"date": 1362355...|[mg, dsl, glory, ...|
+----------+--------------------+--------------------+
only showing top 3 rows

from pyspark.sql.functions import col_____no_output_____# create review column
def collapse_reviews_terms(df):
    # split dataframe
    df_pos = df.where(df.positive==True)
    df_pos = df_pos.select(col("asin"),
                           col("category"),
                           col("reviews").alias("posReviews"),
                           col("topFeatures").alias("posFeatures"))

    df_neg = df.where(df.positive==False)
    df_neg = df_neg.select(col("asin"),
                           col("reviews").alias("negReviews"),
                           col("topFeatures").alias("negFeatures"))

    # get asin
    df_asin = df.select("asin").distinct()

    # join dataframes
    df_final = df_asin.join(df_pos, df_asin.asin==df_neg.asin, 'outer').drop(df_pos.asin)
    df_final = df_final.join(df_neg, df_final.asin==df_neg.asin, 'outer').drop(df_neg.asin)

    return df_final_____no_output_____df_final = collapse_reviews_terms(df_terms)_____no_output_____df_final.show(3)+----------+-------------------+--------------------+--------------------+--------------------+--------------------+
|      asin|           category|          posReviews|         posFeatures|          negReviews|         negFeatures|
+----------+-------------------+--------------------+--------------------+--------------------+--------------------+
|B000MWWT6E|Musical Instruments|[{"date": 1275696...|[dang, pony, tick...|                null|                null|
|B000P5NXWM|Musical Instruments|[{"date": 1301702...|[replacements, sa...|[{"date": 1374969...|[tone, purchased,...|
|B000RNB720|Musical Instruments|[{"date": 1383264...|[investigated, 7a...|[{"date": 1376006...|[can't, big, usin...|
+----------+-------------------+--------------------+--------------------+--------------------+--------------------+
only showing top 3 rows

## Function_____no_output_____

# modules needed
#from pyspark.sql.functions import udf
#from pyspark.sql.types import ArrayType, StringType
#from sentimentAnalysis import dataProcessing as dp_____no_output_____

## Clear Session_____no_output_____

# unpersist old dataframes
df_cat.unpersist()_____no_output_____# end session
spark.stop()_____no_output_____


# Notebook from nupurbaghel/Part-Of-Speech-Tagger
Path: HMM Tagger.ipynb

# Project: Part of Speech Tagging with Hidden Markov Models
---
### Introduction

Part of speech tagging is the process of determining the syntactic category of a word from the words in its surrounding context. It is often used to help disambiguate natural language phrases because it can be done quickly with high accuracy. Tagging can be used for many NLP tasks like determining correct pronunciation during speech synthesis (for example, _dis_-count as a noun vs dis-_count_ as a verb), for information retrieval, and for word sense disambiguation.

In this notebook, you'll use the [Pomegranate](http://pomegranate.readthedocs.io/) library to build a hidden Markov model for part of speech tagging using a "universal" tagset. Hidden Markov models have been able to achieve [>96% tag accuracy with larger tagsets on realistic text corpora](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf). Hidden Markov models have also been used for speech recognition and speech generation, machine translation, gene recognition for bioinformatics, and human gesture recognition for computer vision, and more.

![](_post-hmm.png)

The notebook already contains some code to get you started. You only need to add some new functionality in the areas indicated to complete the project; you will not need to modify the included code beyond what is requested. Sections that begin with **'IMPLEMENTATION'** in the header indicate that you must provide code in the block that follows. Instructions will be provided for each section, and the specifics of the implementation are marked in the code block with a 'TODO' statement. Please be sure to read the instructions carefully!_____no_output_____

**Note:** Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You must then **export the notebook** by running the last cell in the notebook, or by using the menu above and navigating to **File -> Download as -> HTML (.html)** Your submissions should include both the `html` and `ipynb` files.

_____no_output_____

**Note:** Code and Markdown cells can be executed using the `Shift + Enter` keyboard shortcut. Markdown cells can be edited by double-clicking the cell to enter edit mode.

_____no_output_____### The Road Ahead
You must complete Steps 1-3 below to pass the project. The section on Step 4 includes references & resources you can use to further explore HMM taggers.

- [Step 1](#Step-1:-Read-and-preprocess-the-dataset): Review the provided interface to load and access the text corpus
- [Step 2](#Step-2:-Build-a-Most-Frequent-Class-tagger): Build a Most Frequent Class tagger to use as a baseline
- [Step 3](#Step-3:-Build-an-HMM-tagger): Build an HMM Part of Speech tagger and compare to the MFC baseline
- [Step 4](#Step-4:-[Optional]-Improving-model-performance): (Optional) Improve the HMM tagger_____no_output_____

**Note:** Make sure you have selected a **Python 3** kernel in Workspaces or the hmm-tagger conda environment if you are running the Jupyter server on your own machine.

_____no_output_____

# Jupyter "magic methods" -- only need to be run once per kernel restart
%load_ext autoreload
%aimport helpers, tests
%autoreload 1_____no_output_____# import python modules -- this cell needs to be run again if you make changes to any of the files
import matplotlib.pyplot as plt
import numpy as np

from IPython.core.display import HTML
from itertools import chain
from collections import Counter, defaultdict
from helpers import show_model, Dataset
from pomegranate import State, HiddenMarkovModel, DiscreteDistribution_____no_output_____

## Step 1: Read and preprocess the dataset
---
We'll start by reading in a text corpus and splitting it into a training and testing dataset. The data set is a copy of the [Brown corpus](https://en.wikipedia.org/wiki/Brown_Corpus) (originally from the [NLTK](https://www.nltk.org/) library) that has already been pre-processed to only include the [universal tagset](https://arxiv.org/pdf/1104.2086.pdf). You should expect to get slightly higher accuracy using this simplified tagset than the same model would achieve on a larger tagset like the full [Penn treebank tagset](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html), but the process you'll follow would be the same.

The `Dataset` class provided in helpers.py will read and parse the corpus. You can generate your own datasets compatible with the reader by writing them to the following format. The dataset is stored in plaintext as a collection of words and corresponding tags. Each sentence starts with a unique identifier on the first line, followed by one tab-separated word/tag pair on each following line. Sentences are separated by a single blank line.

Example from the Brown corpus.
```
b100-38532
Perhaps	ADV
it	PRON
was	VERB
right	ADJ
;	.
;	.

b100-35577
...
```_____no_output_____

data = Dataset("tags-universal.txt", "brown-universal.txt", train_test_split=0.8)

print("There are {} sentences in the corpus.".format(len(data)))
print("There are {} sentences in the training set.".format(len(data.training_set)))
print("There are {} sentences in the testing set.".format(len(data.testing_set)))

assert len(data) == len(data.training_set) + len(data.testing_set), \
       "The number of sentences in the training set + testing set should sum to the number of sentences in the corpus"There are 57340 sentences in the corpus.
There are 45872 sentences in the training set.
There are 11468 sentences in the testing set.

### The Dataset Interface

You can access (mostly) immutable references to the dataset through a simple interface provided through the `Dataset` class, which represents an iterable collection of sentences along with easy access to partitions of the data for training & testing. Review the reference below, then run and review the next few cells to make sure you understand the interface before moving on to the next step.

```
Dataset-only Attributes:
    training_set - reference to a Subset object containing the samples for training
    testing_set - reference to a Subset object containing the samples for testing

Dataset & Subset Attributes:
    sentences - a dictionary with an entry {sentence_key: Sentence()} for each sentence in the corpus
    keys - an immutable ordered (not sorted) collection of the sentence_keys for the corpus
    vocab - an immutable collection of the unique words in the corpus
    tagset - an immutable collection of the unique tags in the corpus
    X - returns an array of words grouped by sentences ((w11, w12, w13, ...), (w21, w22, w23, ...), ...)
    Y - returns an array of tags grouped by sentences ((t11, t12, t13, ...), (t21, t22, t23, ...), ...)
    N - returns the number of distinct samples (individual words or tags) in the dataset

Methods:
    stream() - returns an flat iterable over all (word, tag) pairs across all sentences in the corpus
    __iter__() - returns an iterable over the data as (sentence_key, Sentence()) pairs
    __len__() - returns the nubmer of sentences in the dataset
```

For example, consider a Subset, `subset`, of the sentences `{"s0": Sentence(("See", "Spot", "run"), ("VERB", "NOUN", "VERB")), "s1": Sentence(("Spot", "ran"), ("NOUN", "VERB"))}`. The subset will have these attributes:

```
subset.keys == {"s1", "s0"}  # unordered
subset.vocab == {"See", "run", "ran", "Spot"}  # unordered
subset.tagset == {"VERB", "NOUN"}  # unordered
subset.X == (("Spot", "ran"), ("See", "Spot", "run"))  # order matches .keys
subset.Y == (("NOUN", "VERB"), ("VERB", "NOUN", "VERB"))  # order matches .keys
subset.N == 7  # there are a total of seven observations over all sentences
len(subset) == 2  # because there are two sentences
```

**Note:** The `Dataset` class is _convenient_, but it is **not** efficient. It is not suitable for huge datasets because it stores multiple redundant copies of the same data.

_____no_output_____#### Sentences

`Dataset.sentences` is a dictionary of all sentences in the training corpus, each keyed to a unique sentence identifier. Each `Sentence` is itself an object with two attributes: a tuple of the words in the sentence named `words` and a tuple of the tag corresponding to each word named `tags`._____no_output_____

key = 'b100-38532'
print("Sentence: {}".format(key))
print("words:\n\t{!s}".format(data.sentences[key].words))
print("tags:\n\t{!s}".format(data.sentences[key].tags))Sentence: b100-38532
words:
	('Perhaps', 'it', 'was', 'right', ';', ';')
tags:
	('ADV', 'PRON', 'VERB', 'ADJ', '.', '.')

**Note:** The underlying iterable sequence is **unordered** over the sentences in the corpus; it is not guaranteed to return the sentences in a consistent order between calls. Use `Dataset.stream()`, `Dataset.keys`, `Dataset.X`, or `Dataset.Y` attributes if you need ordered access to the data.

#### Counting Unique Elements

You can access the list of unique words (the dataset vocabulary) via `Dataset.vocab` and the unique list of tags via `Dataset.tagset`._____no_output_____

print("There are a total of {} samples of {} unique words in the corpus."
      .format(data.N, len(data.vocab)))
print("There are {} samples of {} unique words in the training set."
      .format(data.training_set.N, len(data.training_set.vocab)))
print("There are {} samples of {} unique words in the testing set."
      .format(data.testing_set.N, len(data.testing_set.vocab)))
print("There are {} words in the test set that are missing in the training set."
      .format(len(data.testing_set.vocab - data.training_set.vocab)))

assert data.N == data.training_set.N + data.testing_set.N, \
       "The number of training + test samples should sum to the total number of samples"There are a total of 1161192 samples of 56057 unique words in the corpus.
There are 928458 samples of 50536 unique words in the training set.
There are 232734 samples of 25112 unique words in the testing set.
There are 5521 words in the test set that are missing in the training set.

#### Accessing word and tag Sequences
The `Dataset.X` and `Dataset.Y` attributes provide access to ordered collections of matching word and tag sequences for each sentence in the dataset._____no_output_____

# accessing words with Dataset.X and tags with Dataset.Y
for i in range(2):
    print("Sentence {}:".format(i + 1), data.X[i])
    print()
    print("Labels {}:".format(i + 1), data.Y[i])
    print()Sentence 1: ('Mr.', 'Podger', 'had', 'thanked', 'him', 'gravely', ',', 'and', 'now', 'he', 'made', 'use', 'of', 'the', 'advice', '.')

Labels 1: ('NOUN', 'NOUN', 'VERB', 'VERB', 'PRON', 'ADV', '.', 'CONJ', 'ADV', 'PRON', 'VERB', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence 2: ('But', 'there', 'seemed', 'to', 'be', 'some', 'difference', 'of', 'opinion', 'as', 'to', 'how', 'far', 'the', 'board', 'should', 'go', ',', 'and', 'whose', 'advice', 'it', 'should', 'follow', '.')

Labels 2: ('CONJ', 'PRT', 'VERB', 'PRT', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'ADP', 'ADV', 'ADV', 'DET', 'NOUN', 'VERB', 'VERB', '.', 'CONJ', 'DET', 'NOUN', 'PRON', 'VERB', 'VERB', '.')

#### Accessing (word, tag) Samples
The `Dataset.stream()` method returns an iterator that chains together every pair of (word, tag) entries across all sentences in the entire corpus._____no_output_____

# use Dataset.stream() (word, tag) samples for the entire corpus
print("\nStream (word, tag) pairs:\n")
for i, pair in enumerate(data.stream()):
    print("\t", pair)
    if i > 5: break
Stream (word, tag) pairs:

	 ('Mr.', 'NOUN')
	 ('Podger', 'NOUN')
	 ('had', 'VERB')
	 ('thanked', 'VERB')
	 ('him', 'PRON')
	 ('gravely', 'ADV')
	 (',', '.')

For both our baseline tagger and the HMM model we'll build, we need to estimate the frequency of tags & words from the frequency counts of observations in the training corpus. In the next several cells you will complete functions to compute the counts of several sets of counts. _____no_output_____## Step 2: Build a Most Frequent Class tagger
---

Perhaps the simplest tagger (and a good baseline for tagger performance) is to simply choose the tag most frequently assigned to each word. This "most frequent class" tagger inspects each observed word in the sequence and assigns it the label that was most often assigned to that word in the corpus._____no_output_____### IMPLEMENTATION: Pair Counts

Complete the function below that computes the joint frequency counts for two input sequences._____no_output_____

def pair_counts(sequences_A, sequences_B):
    """Return a dictionary keyed to each unique value in the first sequence list
    that counts the number of occurrences of the corresponding value from the
    second sequences list.

    For example, if sequences_A is tags and sequences_B is the corresponding
    words, then if 1244 sequences contain the word "time" tagged as a NOUN, then
    you should return a dictionary such that pair_counts[NOUN][time] == 1244
    """
    # TODO: Finish this function!
    freq_dict = {}
    for i in range(len(sequences_A)):
        for j in range(len(sequences_A[i])):
            tag = sequences_A[i][j]
            word = sequences_B[i][j]
            if tag not in freq_dict:
                freq_dict[tag]={}
            if word not in freq_dict[tag]:
                freq_dict[tag][word] = 0
            freq_dict[tag][word] = freq_dict[tag][word] + 1
    return freq_dict

# Calculate C(t_i, w_i)
emission_counts = pair_counts(data.training_set.Y, data.training_set.X)

assert len(emission_counts) == 12, \
       "Uh oh. There should be 12 tags in your dictionary."
assert max(emission_counts["NOUN"], key=emission_counts["NOUN"].get) == 'time', \
       "Hmmm...'time' is expected to be the most common NOUN."
HTML('
Your emission counts look good!
')_____no_output_____

### IMPLEMENTATION: Most Frequent Class Tagger

Use the `pair_counts()` function and the training dataset to find the most frequent class label for each word in the training data, and populate the `mfc_table` below. The table keys should be words, and the values should be the appropriate tag string.

The `MFCTagger` class is provided to mock the interface of Pomegranite HMM models so that they can be used interchangeably._____no_output_____

# Create a lookup table mfc_table where mfc_table[word] contains the tag label most frequently assigned to that word
from collections import namedtuple

FakeState = namedtuple("FakeState", "name")

class MFCTagger:
    # NOTE: You should not need to modify this class or any of its methods
    missing = FakeState(name="<MISSING>")

    def __init__(self, table):
        self.table = defaultdict(lambda: MFCTagger.missing)
        self.table.update({word: FakeState(name=tag) for word, tag in table.items()})

    def viterbi(self, seq):
        """This method simplifies predictions by matching the Pomegranate viterbi() interface"""
        return 0., list(enumerate(["<start>"] + [self.table[w] for w in seq] + ["<end>"]))

# TODO: calculate the frequency of each tag being assigned to each word (hint: similar, but not
# the same as the emission probabilities) and use it to fill the mfc_table

word_counts = pair_counts(data.training_set.X, data.training_set.Y)

import operator
mfc_table = {word: max(word_counts[word].items(), key=operator.itemgetter(1))[0] for word in word_counts.keys()}

# DO NOT MODIFY BELOW THIS LINE
mfc_model = MFCTagger(mfc_table) # Create a Most Frequent Class tagger instance

assert len(mfc_table) == len(data.training_set.vocab), ""
assert all(k in data.training_set.vocab for k in mfc_table.keys()), ""
assert sum(int(k not in mfc_table) for k in data.testing_set.vocab) == 5521, ""
HTML('
Your MFC tagger has all the correct words!
')_____no_output_____

### Making Predictions with a Model
The helper functions provided below interface with Pomegranate network models & the mocked MFCTagger to take advantage of the [missing value](http://pomegranate.readthedocs.io/en/latest/nan.html) functionality in Pomegranate through a simple sequence decoding function. Run these functions, then run the next cell to see some of the predictions made by the MFC tagger._____no_output_____

def replace_unknown(sequence):
    """Return a copy of the input sequence where each unknown word is replaced
    by the literal string value 'nan'. Pomegranate will ignore these values
    during computation.
    """
    return [w if w in data.training_set.vocab else 'nan' for w in sequence]

def simplify_decoding(X, model):
    """X should be a 1-D sequence of observations for the model to predict"""
    _, state_path = model.viterbi(replace_unknown(X))
    return [state[1].name for state in state_path[1:-1]]  # do not show the start/end state predictions_____no_output_____

### Example Decoding Sequences with MFC Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, mfc_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', '<MISSING>', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADV', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

### Evaluating Model Accuracy

The function below will evaluate the accuracy of the MFC tagger on the collection of all sentences from a text corpus. _____no_output_____

def accuracy(X, Y, model):
    """Calculate the prediction accuracy by using the model to decode each sequence
    in the input X and comparing the prediction with the true labels in Y.

    The X should be an array whose first dimension is the number of sentences to test,
    and each element of the array should be an iterable of the words in the sequence.
    The arrays X and Y should have the exact same shape.

    X = [("See", "Spot", "run"), ("Run", "Spot", "run", "fast"), ...]
    Y = [(), (), ...]
    """
    correct = total_predictions = 0
    for observations, actual_tags in zip(X, Y):

        # The model.viterbi call in simplify_decoding will return None if the HMM
        # raises an error (for example, if a test sentence contains a word that
        # is out of vocabulary for the training set). Any exception counts the
        # full sentence as an error (which makes this a conservative estimate).
        try:
            most_likely_tags = simplify_decoding(observations, model)
            correct += sum(p == t for p, t in zip(most_likely_tags, actual_tags))
        except:
            pass
        total_predictions += len(observations)
    return correct / total_predictions_____no_output_____

#### Evaluate the accuracy of the MFC tagger
Run the next cell to evaluate the accuracy of the tagger on the training and test corpus._____no_output_____

mfc_training_acc = accuracy(data.training_set.X, data.training_set.Y, mfc_model)
print("training accuracy mfc_model: {:.2f}%".format(100 * mfc_training_acc))

mfc_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, mfc_model)
print("testing accuracy mfc_model: {:.2f}%".format(100 * mfc_testing_acc))

assert mfc_training_acc >= 0.955, "Uh oh. Your MFC accuracy on the training set doesn't look right."
assert mfc_testing_acc >= 0.925, "Uh oh. Your MFC accuracy on the testing set doesn't look right."
HTML('
Your MFC tagger accuracy looks correct!
')training accuracy mfc_model: 95.72%
testing accuracy mfc_model: 93.01%

## Step 3: Build an HMM tagger
---
The HMM tagger has one hidden state for each possible tag, and parameterized by two distributions: the emission probabilties giving the conditional probability of observing a given **word** from each hidden state, and the transition probabilities giving the conditional probability of moving between **tags** during the sequence.

We will also estimate the starting probability distribution (the probability of each **tag** being the first tag in a sequence), and the terminal probability distribution (the probability of each **tag** being the last tag in a sequence).

The maximum likelihood estimate of these distributions can be calculated from the frequency counts as described in the following sections where you'll implement functions to count the frequencies, and finally build the model. The HMM model will make predictions according to the formula:

$$t_i^n = \underset{t_i^n}{\mathrm{argmax}} \prod_{i=1}^n P(w_i|t_i) P(t_i|t_{i-1})$$

Refer to Speech & Language Processing [Chapter 10](https://web.stanford.edu/~jurafsky/slp3/10.pdf) for more information._____no_output_____### IMPLEMENTATION: Unigram Counts

Complete the function below to estimate the co-occurrence frequency of each symbol over all of the input sequences. The unigram probabilities in our HMM model are estimated from the formula below, where N is the total number of samples in the input. (You only need to compute the counts for now.)

$$P(tag_1) = \frac{C(tag_1)}{N}$$_____no_output_____

def unigram_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequence list that
    counts the number of occurrences of the value in the sequences list. The sequences
    collection should be a 2-dimensional array.

    For example, if the tag NOUN appears 275558 times over all the input sequences,
    then you should return a dictionary such that your_unigram_counts[NOUN] == 275558.
    """
    # TODO: Finish this function!
    freq = {}
    for sentence in sequences:
        for tag in sentence:
            if tag not in freq:
                freq[tag] = 0
            freq[tag] = freq[tag] + 1
    return freq

# TODO: call unigram_counts with a list of tag sequences from the training set
tag_unigrams = unigram_counts(data.training_set.Y)

assert set(tag_unigrams.keys()) == data.training_set.tagset, \
       "Uh oh. It looks like your tag counts doesn't include all the tags!"
assert min(tag_unigrams, key=tag_unigrams.get) == 'X', \
       "Hmmm...'X' is expected to be the least common class"
assert max(tag_unigrams, key=tag_unigrams.get) == 'NOUN', \
       "Hmmm...'NOUN' is expected to be the most common class"
HTML('
Your tag unigrams look good!
')_____no_output_____

### IMPLEMENTATION: Bigram Counts

Complete the function below to estimate the co-occurrence frequency of each pair of symbols in each of the input sequences. These counts are used in the HMM model to estimate the bigram probability of two tags from the frequency counts according to the formula: $$P(tag_2|tag_1) = \frac{C(tag_2|tag_1)}{C(tag_2)}$$
_____no_output_____

def bigram_counts(sequences):
    """Return a dictionary keyed to each unique PAIR of values in the input sequences
    list that counts the number of occurrences of pair in the sequences list. The input
    should be a 2-dimensional array.

    For example, if the pair of tags (NOUN, VERB) appear 61582 times, then you should
    return a dictionary such that your_bigram_counts[(NOUN, VERB)] == 61582
    """

    # TODO: Finish this function!
    freq = {}
    for sentence in sequences:
        for i in range(len(sentence)-1):
            pair = (sentence[i],sentence[i+1])
            if pair not in freq:
                freq[pair] = 1
            else :
                freq[pair] = freq[pair] + 1
    return freq

# TODO: call bigram_counts with a list of tag sequences from the training set
tag_bigrams = bigram_counts(data.training_set.Y)

assert len(tag_bigrams) == 144, \
       "Uh oh. There should be 144 pairs of bigrams (12 tags x 12 tags)"
assert min(tag_bigrams, key=tag_bigrams.get) in [('X', 'NUM'), ('PRON', 'X')], \
       "Hmmm...The least common bigram should be one of ('X', 'NUM') or ('PRON', 'X')."
assert max(tag_bigrams, key=tag_bigrams.get) in [('DET', 'NOUN')], \
       "Hmmm...('DET', 'NOUN') is expected to be the most common bigram."
HTML('
Your tag bigrams look good!
')_____no_output_____

### IMPLEMENTATION: Sequence Starting Counts
Complete the code below to estimate the bigram probabilities of a sequence starting with each tag._____no_output_____

def starting_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the beginning of
    a sequence.

    For example, if 8093 sequences start with NOUN, then you should return a
    dictionary such that your_starting_counts[NOUN] == 8093
    """
    # TODO: Finish this function!
    freq = {}
    for sentence in sequences:
        tag = sentence[0]
        if tag not in freq:
            freq[tag] = 1
        else:
            freq[tag]= freq[tag]+1
    return freq

# TODO: Calculate the count of each tag starting a sequence
tag_starts = starting_counts(data.training_set.Y)

assert len(tag_starts) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_starts, key=tag_starts.get) == 'X', "Hmmm...'X' is expected to be the least common starting bigram."
assert max(tag_starts, key=tag_starts.get) == 'DET', "Hmmm...'DET' is expected to be the most common starting bigram."
HTML('
Your starting tag counts look good!
')_____no_output_____

### IMPLEMENTATION: Sequence Ending Counts
Complete the function below to estimate the bigram probabilities of a sequence ending with each tag._____no_output_____

def ending_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the end of
    a sequence.

    For example, if 18 sequences end with DET, then you should return a
    dictionary such that your_starting_counts[DET] == 18
    """
    # TODO: Finish this function!
    freq = {}
    for sentence in sequences:
        tag = sentence[-1]
        if tag not in freq:
            freq[tag] = 1
        else:
            freq[tag]= freq[tag]+1
    return freq

# TODO: Calculate the count of each tag ending a sequence
tag_ends = ending_counts(data.training_set.Y)

assert len(tag_ends) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_ends, key=tag_ends.get) in ['X', 'CONJ'], "Hmmm...'X' or 'CONJ' should be the least common ending bigram."
assert max(tag_ends, key=tag_ends.get) == '.', "Hmmm...'.' is expected to be the most common ending bigram."
HTML('
Your ending tag counts look good!
')_____no_output_____

### IMPLEMENTATION: Basic HMM Tagger
Use the tag unigrams and bigrams calculated above to construct a hidden Markov tagger.

- Add one state per tag
    - The emission distribution at each state should be estimated with the formula: $P(w|t) = \frac{C(t, w)}{C(t)}$
- Add an edge from the starting state `basic_model.start` to each tag
    - The transition probability should be estimated with the formula: $P(t|start) = \frac{C(start, t)}{C(start)}$
- Add an edge from each tag to the end state `basic_model.end`
    - The transition probability should be estimated with the formula: $P(end|t) = \frac{C(t, end)}{C(t)}$
- Add an edge between _every_ pair of tags
    - The transition probability should be estimated with the formula: $P(t_2|t_1) = \frac{C(t_1, t_2)}{C(t_1)}$_____no_output_____

basic_model = HiddenMarkovModel(name="base-hmm-tagger")

# TODO: create states with emission probability distributions P(word | tag) and add to the model
# (Hint: you may need to loop & create/add new states)
state = {}
for tag,words in emission_counts.items():
    distribution = {}
    for word,count in words.items():
        distribution[word] = count/tag_unigrams[tag]
    new_state = State(DiscreteDistribution(distribution), name=tag)
    state[tag] = new_state
    basic_model.add_state(new_state)

# TODO: add edges between states for the observed transition frequencies P(tag_i | tag_i-1)
# (Hint: you may need to loop & add transitions
total_start = len(data.training_set.Y)
for tag, count in tag_starts.items():
    basic_model.add_transition(basic_model.start, state[tag], count/total_start)

for tag, count in tag_ends.items():
    basic_model.add_transition(state[tag], basic_model.end, count/tag_unigrams[tag])

for tag_pair, count in tag_bigrams.items():
    tag1 = tag_pair[0]
    tag2 = tag_pair[1]
    basic_model.add_transition(state[tag1], state[tag2], count/tag_unigrams[tag1])

# NOTE: YOU SHOULD NOT NEED TO MODIFY ANYTHING BELOW THIS LINE
# finalize the model
basic_model.bake()

assert all(tag in set(s.name for s in basic_model.states) for tag in data.training_set.tagset), \
       "Every state in your network should use the name of the associated tag, which must be one of the training set tags."
assert basic_model.edge_count() == 168, \
       ("Your network should have an edge from the start node to each state, one edge between every " +
        "pair of tags (states), and an edge from each state to the end node.")
HTML('
Your HMM network topology looks good!
')_____no_output_____hmm_training_acc = accuracy(data.training_set.X, data.training_set.Y, basic_model)
print("training accuracy basic hmm model: {:.2f}%".format(100 * hmm_training_acc))

hmm_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, basic_model)
print("testing accuracy basic hmm model: {:.2f}%".format(100 * hmm_testing_acc))

assert hmm_training_acc > 0.97, "Uh oh. Your HMM accuracy on the training set doesn't look right."
assert hmm_testing_acc > 0.955, "Uh oh. Your HMM accuracy on the testing set doesn't look right."
HTML('
Your HMM tagger accuracy looks correct! Congratulations, you\'ve finished the project.
')training accuracy basic hmm model: 97.54%
testing accuracy basic hmm model: 95.95%

### Example Decoding Sequences with the HMM Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, basic_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

## Finishing the project
---

**Note:** **SAVE YOUR NOTEBOOK**, then run the next cell to generate an HTML copy. You will zip & submit both this file and the HTML copy for review.

_____no_output_____

!!jupyter nbconvert *.ipynb_____no_output_____

## Step 4: [Optional] Improving model performance
---
There are additional enhancements that can be incorporated into your tagger that improve performance on larger tagsets where the data sparsity problem is more significant. The data sparsity problem arises because the same amount of data split over more tags means there will be fewer samples in each tag, and there will be more missing data  tags that have zero occurrences in the data. The techniques in this section are optional.

- [Laplace Smoothing](https://en.wikipedia.org/wiki/Additive_smoothing) (pseudocounts)
    Laplace smoothing is a technique where you add a small, non-zero value to all observed counts to offset for unobserved values.

- Backoff Smoothing
    Another smoothing technique is to interpolate between n-grams for missing data. This method is more effective than Laplace smoothing at combatting the data sparsity problem. Refer to chapters 4, 9, and 10 of the [Speech & Language Processing](https://web.stanford.edu/~jurafsky/slp3/) book for more information.

- Extending to Trigrams
    HMM taggers have achieved better than 96% accuracy on this dataset with the full Penn treebank tagset using an architecture described in [this](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf) paper. Altering your HMM to achieve the same performance would require implementing deleted interpolation (described in the paper), incorporating trigram probabilities in your frequency tables, and re-implementing the Viterbi algorithm to consider three consecutive states instead of two.

### Obtain the Brown Corpus with a Larger Tagset
Run the code below to download a copy of the brown corpus with the full NLTK tagset. You will need to research the available tagset information in the NLTK docs and determine the best way to extract the subset of NLTK tags you want to explore. If you write the following the format specified in Step 1, then you can reload the data using all of the code above for comparison.

Refer to [Chapter 5](http://www.nltk.org/book/ch05.html) of the NLTK book for more information on the available tagsets._____no_output_____

import nltk
from nltk import pos_tag, word_tokenize
from nltk.corpus import brown

nltk.download('brown')
training_corpus = nltk.corpus.brown
training_corpus.tagged_sents()[0][nltk_data] Downloading package brown to /Users/nupur/nltk_data...
[nltk_data]   Unzipping corpora/brown.zip.


# Notebook from SalishSeaCast/analysis-keegan
Path: notebooks/Evaluations/By_Year_Evaluations/CTD_By_Year.ipynb

This script looks at the relationship between model and observed values of temperature and Salinity within Puget Sound. It is made so that multiple years can be viewed at once. _____no_output_____

import numpy as np
import matplotlib.pyplot as plt
import os
import pandas as pd
import netCDF4 as nc
import datetime as dt
from salishsea_tools import evaltools as et, viz_tools
import gsw
import matplotlib.gridspec as gridspec
import matplotlib as mpl
import matplotlib.dates as mdates
import cmocean as cmo
import scipy.interpolate as sinterp
import pickle
import cmocean
import json
import math
import f90nml
from collections import OrderedDict

fs=16
mpl.rc('xtick', labelsize=fs)
mpl.rc('ytick', labelsize=fs)
mpl.rc('legend', fontsize=fs)
mpl.rc('axes', titlesize=fs)
mpl.rc('axes', labelsize=fs)
mpl.rc('figure', titlesize=fs)
mpl.rc('font', size=fs)
mpl.rc('font', family='sans-serif', weight='normal', style='normal')

import warnings
#warnings.filterwarnings('ignore')
from IPython.display import Markdown, display

%matplotlib inline_____no_output_____

years=[2015,2016,2017]
PATH= '/results2/SalishSea/nowcast-green.201905/'
datadir='/ocean/eolson/MEOPAR/obs/WADE/ptools_data/ecology'_____no_output_____display(Markdown('''## Year: '''+ str(years)))_____no_output_____display(Markdown('''### Model output: '''+ PATH))_____no_output_____

## Yearly model-data comparisons of nutrients, chlorophyll, temperature and salinity between 201905 runs and WADE observations_____no_output_____### load observations_____no_output_____

dfSta=pickle.load(open(os.path.join(datadir,'sta_df.p'),'rb'))_____no_output_____dictCTD=dict()
for i in years:
    dictCTD[i]=pickle.load(open(os.path.join(datadir,f'Casts_{str(i)}.p'),'rb'))_____no_output_____dfCTD0=pd.concat(dictCTD.values(), ignore_index=True)_____no_output_____dfCTD=pd.merge(left=dfSta,right=dfCTD0,how='right',
             left_on='Station',right_on='Station')
#right join means all rows in right table (dfCTD) are included in output
dfCTD.head()_____no_output_____# check that there are no stations without lat and lon:
dfCTD.loc[pd.isnull(dfCTD['Latitude'])]_____no_output_____# check one to one matches:
len(dfCTD),len(dfCTD0), len(dfSta)_____no_output_____# where no time is provided, set time to midday Pacific time = ~ 20:00 UTC for now
# the code below takes advantage of all elements in 'Date' having a time component
# set to midnight
dfCTD['dtUTC']=[iiD+dt.timedelta(hours=20) for iiD in dfCTD['Date']]_____no_output_____# Column names must be altered
dfCTD.rename(columns={'Latitude':'Lat','Longitude':'Lon'},inplace=True)
dfCTD['Z']=-1*dfCTD['Z']
dfCTD.head()_____no_output_____# Calculate Absolute (Reference) Salinity (g/kg) and Conservative Temperature (deg C) from
# Salinity (psu) and Temperature (deg C):
press=gsw.p_from_z(-1*dfCTD['Z'],dfCTD['Lat'])
dfCTD['SA']=gsw.SA_from_SP(dfCTD['Salinity'],press,
                           dfCTD['Lon'],dfCTD['Lat'])
dfCTD['CT']=gsw.CT_from_t(dfCTD['SA'],dfCTD['Temperature'],press)_____no_output_____print(len(dfCTD),'data points')
print('Number of data points in each region:')
dfCTD.groupby('Basin')['SA'].count()155398 data points
Number of data points in each region:

### set up variables for model-data matching_____no_output_____

# start_date and end_date are the first and last dates that will
#  be included in the matched data set
start_date = dt.datetime(years[0],1,1)
end_date = dt.datetime(years[-1],12,31)
flen=1 # number of days per model output file. always 1 for 201905 and 201812 model runs
namfmt='nowcast' # for 201905 and 201812 model runs, this should always be 'nowcast'
# filemap is dictionary of the form variableName: fileType, where variableName is the name
# of the variable you want to extract and fileType designates the type of
# model output file it can be found in (usually ptrc_T for biology, grid_T for temperature and
# salinity)
filemap={'vosaline':'grid_T','votemper':'grid_T'}
# fdict is a dictionary mappy file type to its time resolution. Here, 1 means hourly output
# (1h file) and 24 means daily output (1d file). In certain runs, multiple time resolutions
# are available
fdict={'ptrc_T':1,'grid_T':1}
# Note: to switch between 201812 and 201905 model results, change PATH
# to switch from hourly to daily model output, change fdict values from 1 to 24 (but daily
#  files are not available for some runs and file types)_____no_output_____data=et.matchData(dfCTD,filemap,fdict,start_date,end_date,'nowcast',PATH,1,quiet=False);(Lat,Lon)= 46.453155 -124.00960333333333  not matched to domain
(Lat,Lon)= 46.463155 -123.94126833333334  not matched to domain
(Lat,Lon)= 46.54537666666667 -123.98016166666666  not matched to domain
(Lat,Lon)= 46.644 -123.993  not matched to domain
(Lat,Lon)= 46.68676333333333 -123.9735  not matched to domain
(Lat,Lon)= 46.703986666666665 -123.837385  not matched to domain
(Lat,Lon)= 46.937313333333336 -123.91322333333333  not matched to domain
(Lat,Lon)= 46.953421666666664 -124.09295  not matched to domain
(Lat,Lon)= 47.21342666666666 -123.07765  not matched to domain
progress: 0.0%
progress: 3.3012670262846884%
progress: 6.602534052569377%
progress: 9.903801078854064%
progress: 13.205068105138754%
progress: 16.50633513142344%
progress: 19.80760215770813%
progress: 23.108869183992816%
progress: 26.410136210277507%
progress: 29.711403236562195%
progress: 33.01267026284688%
progress: 36.31393728913157%
progress: 39.61520431541626%
progress: 42.91647134170095%
progress: 46.21773836798563%
progress: 49.51900539427032%
progress: 52.820272420555014%
progress: 56.1215394468397%
progress: 59.42280647312439%
progress: 62.72407349940907%
progress: 66.02534052569376%
progress: 69.32660755197844%
progress: 72.62787457826315%
progress: 75.92914160454782%
progress: 79.23040863083251%
progress: 82.5316756571172%
progress: 85.8329426834019%
progress: 89.13420970968657%
progress: 92.43547673597126%
progress: 95.73674376225595%
progress: 99.03801078854065%
cm1=cmocean.cm.thermal
with nc.Dataset('/data/eolson/results/MEOPAR/NEMO-forcing-new/grid/bathymetry_201702.nc') as bathy:
    bathylon=np.copy(bathy.variables['nav_lon'][:,:])
    bathylat=np.copy(bathy.variables['nav_lat'][:,:])
    bathyZ=np.copy(bathy.variables['Bathymetry'][:,:])_____no_output_____fig, ax = plt.subplots(1,1,figsize = (6,6))
with nc.Dataset('/data/vdo/MEOPAR/NEMO-forcing/grid/bathymetry_201702.nc') as grid:
    viz_tools.plot_coastline(ax, grid, coords = 'map',isobath=.1)
colors=('blue','green','firebrick','darkorange','darkviolet','fuchsia',
        'yellow','darkgoldenrod','mediumspringgreen','royalblue')
datreg=dict()
for ind, iregion in enumerate(data.Basin.unique()):
    datreg[iregion] = data.loc[data.Basin==iregion]
    ax.plot(datreg[iregion]['Lon'], datreg[iregion]['Lat'],'.',
            color = colors[ind], label=iregion)
ax.set_ylim(47, 49)
ax.legend(bbox_to_anchor=[1,.6,0,0])
ax.set_xlim(-124, -122);
ax.set_title('Observation Locations');

iz=(data.Z<15)

# When working with multiple years, a for loop is necessary to split things up by months.
JFMd=dict()
Aprd=dict()
MJJAd=dict()
SONDd=dict()
for i in years:
    JFMd[i]=data.loc[iz&(data.dtUTC<=dt.datetime(i,4,1)),:]
    Aprd[i]=data.loc[iz&(data.dtUTC<=dt.datetime(i,5,1))&(data.dtUTC>dt.datetime(i,4,1)),:]
    MJJAd[i]=data.loc[iz&(data.dtUTC<=dt.datetime(i,9,1))&(data.dtUTC>dt.datetime(i,5,1)),:]
    SONDd[i]=data.loc[iz&(data.dtUTC>dt.datetime(i,9,1)),:]
JFM=pd.concat(JFMd.values(), ignore_index=True)
Apr=pd.concat(Aprd.values(), ignore_index=True)
MJJA=pd.concat(MJJAd.values(), ignore_index=True)
SOND=pd.concat(SONDd.values(), ignore_index=True)_____no_output_____# defning a dictionary that brakes the data down by individual stations.
datstat=dict()
for ind, istation in enumerate(data.Station.unique()):
    datstat[istation]=data.loc[data.Station == istation]

datyear=dict()
for ind, iyear in enumerate(data.dtUTC.dt.year.unique()):
    datyear[iyear] = data.loc[data.dtUTC.dt.year==iyear]

datyearreg=dict()
for iyear in datyear:
    datyearreg[iyear]=dict()
    for iregion in datyear[iyear].Basin.unique():
        datyearreg[iyear][iregion]=datyear[iyear].loc[datyear[iyear].Basin==iregion]

_____no_output_____def byDepth(ax,obsvar,modvar,lims,byyear = False,year = 'NA'):
    if byyear == False:
        ps=et.varvarPlot(ax,data,obsvar,modvar,'Z',(15,22),'z','m',('mediumseagreen','darkturquoise','navy'))
        l=ax.legend(handles=ps)
    elif byyear == True:
        ps=et.varvarPlot(ax,datyear[year],obsvar,modvar,'Z',(15,22),'z','m',('mediumseagreen','darkturquoise','navy'))
        l=ax.legend(handles=ps)
    ax.set_xlabel('Obs')
    ax.set_ylabel('Model')
    ax.plot(lims,lims,'k-',alpha=.5)
    ax.set_xlim(lims)
    ax.set_ylim(lims)
    ax.set_aspect(1)
    return ps,l

def byRegion(ax,obsvar,modvar,lims,year):
    ps=[]
    for ind, iregion in enumerate(data[data.dtUTC.dt.year==year].Basin.unique()):
        ax.plot(datyearreg[year][iregion]['Lon'], datyearreg[year][iregion]['Lat'],'.',
                color = colors[ind], label=iregion)
        ps0=et.varvarPlot(ax,datyearreg[year][iregion],obsvar,modvar,
                        cols=(colors[ind],),lname=iregion)
        ps.append(ps0)
    l=ax.legend(handles=[ip[0][0] for ip in ps])
    ax.set_xlabel('Obs')
    ax.set_ylabel('Model')
    ax.plot(lims,lims,'k-',alpha=.5)
    ax.set_xlim(lims)
    ax.set_ylim(lims)
    ax.set_aspect(1)
    return ps,l

def byStation(ax,obsvar,modvar,lims,region):
    ps=[]
    for ind, istation in enumerate(data[data['Basin'] == region].Station.unique()):
        ax.plot(datstat[istation]['Lon'], datstat[istation]['Lat'],'.',
                    color = colors[ind], label=istation)
        ps0=et.varvarPlot(ax,datstat[istation],obsvar,modvar,
                            cols=(colors[ind],),lname=istation)
        ps.append(ps0)
    l=ax.legend(handles=[ip[0][0] for ip in ps])
    ax.set_xlabel('Obs')
    ax.set_ylabel('Model')
    ax.plot(lims,lims,'k-',alpha=.5)
    ax.set_xlim(lims)
    ax.set_ylim(lims)
    ax.set_aspect(1)
    return ps,l

def bySeason(ax,obsvar,modvar,lims):
    for axi in ax:
        axi.plot(lims,lims,'k-')
        axi.set_xlim(lims)
        axi.set_ylim(lims)
        axi.set_aspect(1)
        axi.set_xlabel('Obs')
        axi.set_ylabel('Model')
    ps=et.varvarPlot(ax[0],JFM,obsvar,modvar,cols=('crimson','darkturquoise','navy'))
    ax[0].set_title('Jan-Mar')
    ps=et.varvarPlot(ax[1],Apr,obsvar,modvar,cols=('crimson','darkturquoise','navy'))
    ax[1].set_title('Apr')
    ps=et.varvarPlot(ax[2],MJJA,obsvar,modvar,cols=('crimson','darkturquoise','navy'))
    ax[2].set_title('May-Aug')
    ps=et.varvarPlot(ax[3],SOND,obsvar,modvar,cols=('crimson','darkturquoise','navy'))
    ax[3].set_title('Sep-Dec')
    return

def ErrErr(fig,ax,obsvar1,modvar1,obsvar2,modvar2,lims1,lims2):
    m=ax.scatter(data[modvar1]-data[obsvar1],data[modvar2]-data[obsvar2],c=data['Z'],s=1,cmap='gnuplot')
    cb=fig.colorbar(m,ax=ax,label='Depth (m)')
    ax.set_xlim(lims1)
    ax.set_ylim(lims2)
    ax.set_aspect((lims1[1]-lims1[0])/(lims2[1]-lims2[0]))
    return m,cb

def hist2d_maker(ax,obsvar,modvar,lims,fontsize=16):
    ii=(~np.isnan(data[obsvar]))&(~np.isnan(data[modvar]))
    counts, xedges, yedges, ps=ax.hist2d(data.loc[ii,[obsvar]].values.flatten(),
                                      data.loc[ii,[modvar]].values.flatten(),bins=25*3,norm=LogNorm())
    cb=fig.colorbar(ps,ax=ax,label='Count',shrink=0.5)
    ax.set_xlim(lims)
    ax.set_ylim(lims)
    ax.set_ylabel('Modeled',fontsize=fontsize)
    ax.set_xlabel('Observed',fontsize=fontsize)
    plt.tight_layout()
    return ps_____no_output_____### These groupings will be used to calculate statistics. The keys are labels and
### the values are corresponding dataframe views
statsubs=OrderedDict({'z < 15 m':data.loc[data.Z<15],
                      '15 m < z < 22 m':data.loc[(data.Z>=15)&(data.Z<22)],
                      'z >= 22 m':data.loc[data.Z>=22],
                      'z > 50 m':data.loc[data.Z>50],
                      'all':data,
                      'z < 15 m, JFM':JFM,
                      'z < 15 m, Apr':Apr,
                      'z < 15 m, MJJA':MJJA,
                      'z < 15 m, SOND': SOND,})
for iregion in data.Basin.unique():
    statsubs[iregion]=datreg[iregion]
statsubs.keys()_____no_output_____

## Model vs observed Salinity for 2015-2017_____no_output_____

def multi_depreg_graph(down,figsize,figrange,years,obsvar,modvar,phyvar_name):
    fig,ax=plt.subplots(down,2,figsize=figsize)
    for d,Y in zip(range(down),years):
        ps,l=byDepth(ax[d][0],obsvar,modvar,figrange,byyear=True,year=Y)
        ax[d][0].set_title(f'{phyvar_name} (g kg$^-1$) By Depth for {Y}')
        ps,l=byRegion(ax[d][1],obsvar,modvar,figrange,year=Y)
        ax[d][1].set_title(f'{phyvar_name} (g kg$^-1$) By Region for {Y}');

def multi_2dhist_graph(down,across,figsize,lims,years,obsvar,modvar):
    fig, ax = plt.subplots(down,across,figsize=figsize)
    two_by_two=np.array_split(years,math.ceil(len(years)/2))
    for i,Set in enumerate(two_by_two):
        for ii,Y in enumerate(Set):
            ax[i,ii].hist2d_maker(data,ax,obsvar,modvar,lims)
            ax[i,ii].set_title(str(Y))_____no_output_____lims = (0,40)
obsvar='SA'
modvar='mod_vosaline'

multi_depreg_graph(2,(17,20),(0,40),years,obsvar,modvar,'Salinity')_____no_output_____multi_2dhist_graph(2,2,(17,20),lims,years,obsvar,modvar)_____no_output_____

## Model vs Observed Temperature for 2015-2017_____no_output_____

obsvar='CT'
modvar='mod_votemper'

multi_depreg_graph(3,(17,30),(0,40),years,obsvar,modvar,'Temperature')_____no_output_____


# Notebook from 0todd0000/mwarp1d
Path: mwarp1d/examples/overview.ipynb

# Overview

This notebook provides a brief overview of the purpose and functionality of `mwarp1d` using the following three repository scripts, which are run inline in this notebook.

1. fig_dataset.py
1. fig_landmarks.py
1. fig_warp_overview.py

Running this notebook and/or the scripts will test all main `mwarp1d` functionality.

For detailed use instructions, refer to `mwarp1d`'s [online documentation](http://www.spm1d.org/mwarp1d/index.html).

________no_output_____### Example dataset_____no_output_____

%matplotlib inline

%run fig_dataset.py_____no_output_____

**Figure 1**. Example dataset: anteroposterior (i.e., fore-aft) ground reaction forces when running at different speeds (Dorn et al. 2012). Local minima and maxima are not temporally aligned in the region: time = 10% to 25%, implying that [nonlinear warping](https://en.wikipedia.org/wiki/Image_warping) may improve [registration](https://en.wikipedia.org/wiki/Image_registration).
________no_output_____### Example landmark warping_____no_output_____

%run fig_landmarks.py_____no_output_____

**Figure 2**. Example landmark warping and registration. The registration goal is to warp the Source so that it becomes better aligned with the Template. (a) Landmark time nodes selected at local extrema. (b) Piecewise interpolation between source landmarks yields a warped source with qualitatively improved registration.
________no_output_____### Example manual warping_____no_output_____

%run fig_warp_overview.py_____no_output_____

**Figure 3**. (a) Manual warp parameters: `center`, `amp` (amplitude), `head` and `tail`. Data at position `center` will be translated horizontally with an approximate magnitude of `amp`, where `head` and `tail` determine translation effect fall-off, or the translation effects on neighboring regions. (b) Three example warp kernels (i.e., displacement fields). (c) Corresponding warped domains. (d) Corresponding warped data for a single 1D measurement from Fig.1._____no_output_____### Summary

`mwarp1d` allows users to nonlinearly register 1D data using either (1) landmark warping or (2) manual warping. This can be achieved via scripting, as demonstrated above, or using the `mwarp1d` GUI, as described in a series of [screencasts](http://www.spm1d.org/mwarp1d/quickstart.html)._____no_output_____### References

Dorn TW, Schache AG, Pandy MG (2012). “Muscular strategy shift in human running: de- pendence of running speed on hip and ankle muscle performance.” Journal of Experimental Biology, 215(11), 1944–1956. https://simtk.org/home/runningspeeds._____no_output_____