# Notebook from Alps-Wang/hmm-tagger
Path: HMM Tagger.ipynb

# Project: Part of Speech Tagging with Hidden Markov Models
---
### Introduction

Part of speech tagging is the process of determining the syntactic category of a word from the words in its surrounding context. It is often used to help disambiguate natural language phrases because it can be done quickly with high accuracy. Tagging can be used for many NLP tasks like determining correct pronunciation during speech synthesis (for example, _dis_-count as a noun vs dis-_count_ as a verb), for information retrieval, and for word sense disambiguation.

In this notebook, you'll use the [Pomegranate](http://pomegranate.readthedocs.io/) library to build a hidden Markov model for part of speech tagging using a "universal" tagset. Hidden Markov models have been able to achieve [>96% tag accuracy with larger tagsets on realistic text corpora](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf). Hidden Markov models have also been used for speech recognition and speech generation, machine translation, gene recognition for bioinformatics, and human gesture recognition for computer vision, and more.

![](_post-hmm.png)

The notebook already contains some code to get you started. You only need to add some new functionality in the areas indicated to complete the project; you will not need to modify the included code beyond what is requested. Sections that begin with **'IMPLEMENTATION'** in the header indicate that you must provide code in the block that follows. Instructions will be provided for each section, and the specifics of the implementation are marked in the code block with a 'TODO' statement. Please be sure to read the instructions carefully!_____no_output_____

**Note:** Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You must then **export the notebook** by running the last cell in the notebook, or by using the menu above and navigating to **File -> Download as -> HTML (.html)** Your submissions should include both the `html` and `ipynb` files.

_____no_output_____

**Note:** Code and Markdown cells can be executed using the `Shift + Enter` keyboard shortcut. Markdown cells can be edited by double-clicking the cell to enter edit mode.

_____no_output_____### The Road Ahead
You must complete Steps 1-3 below to pass the project. The section on Step 4 includes references & resources you can use to further explore HMM taggers.

- [Step 1](#Step-1:-Read-and-preprocess-the-dataset): Review the provided interface to load and access the text corpus
- [Step 2](#Step-2:-Build-a-Most-Frequent-Class-tagger): Build a Most Frequent Class tagger to use as a baseline
- [Step 3](#Step-3:-Build-an-HMM-tagger): Build an HMM Part of Speech tagger and compare to the MFC baseline
- [Step 4](#Step-4:-[Optional]-Improving-model-performance): (Optional) Improve the HMM tagger_____no_output_____

**Note:** Make sure you have selected a **Python 3** kernel in Workspaces or the hmm-tagger conda environment if you are running the Jupyter server on your own machine.

_____no_output_____

# Jupyter "magic methods" -- only need to be run once per kernel restart
%load_ext autoreload
%aimport helpers, tests
%autoreload 1_____no_output_____# import python modules -- this cell needs to be run again if you make changes to any of the files
import matplotlib.pyplot as plt
import numpy as np

from IPython.core.display import HTML
from itertools import chain
from collections import Counter, defaultdict
from helpers import show_model, Dataset
from pomegranate import State, HiddenMarkovModel, DiscreteDistribution_____no_output_____

## Step 1: Read and preprocess the dataset
---
We'll start by reading in a text corpus and splitting it into a training and testing dataset. The data set is a copy of the [Brown corpus](https://en.wikipedia.org/wiki/Brown_Corpus) (originally from the [NLTK](https://www.nltk.org/) library) that has already been pre-processed to only include the [universal tagset](https://arxiv.org/pdf/1104.2086.pdf). You should expect to get slightly higher accuracy using this simplified tagset than the same model would achieve on a larger tagset like the full [Penn treebank tagset](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html), but the process you'll follow would be the same.

The `Dataset` class provided in helpers.py will read and parse the corpus. You can generate your own datasets compatible with the reader by writing them to the following format. The dataset is stored in plaintext as a collection of words and corresponding tags. Each sentence starts with a unique identifier on the first line, followed by one tab-separated word/tag pair on each following line. Sentences are separated by a single blank line.

Example from the Brown corpus.
```
b100-38532
Perhaps	ADV
it	PRON
was	VERB
right	ADJ
;	.
;	.

b100-35577
...
```_____no_output_____

data = Dataset("tags-universal.txt", "brown-universal.txt", train_test_split=0.8)

print("There are {} sentences in the corpus.".format(len(data)))
print("There are {} sentences in the training set.".format(len(data.training_set)))
print("There are {} sentences in the testing set.".format(len(data.testing_set)))

assert len(data) == len(data.training_set) + len(data.testing_set), \
       "The number of sentences in the training set + testing set should sum to the number of sentences in the corpus"There are 57340 sentences in the corpus.
There are 45872 sentences in the training set.
There are 11468 sentences in the testing set.

### The Dataset Interface

You can access (mostly) immutable references to the dataset through a simple interface provided through the `Dataset` class, which represents an iterable collection of sentences along with easy access to partitions of the data for training & testing. Review the reference below, then run and review the next few cells to make sure you understand the interface before moving on to the next step.

```
Dataset-only Attributes:
    training_set - reference to a Subset object containing the samples for training
    testing_set - reference to a Subset object containing the samples for testing

Dataset & Subset Attributes:
    sentences - a dictionary with an entry {sentence_key: Sentence()} for each sentence in the corpus
    keys - an immutable ordered (not sorted) collection of the sentence_keys for the corpus
    vocab - an immutable collection of the unique words in the corpus
    tagset - an immutable collection of the unique tags in the corpus
    X - returns an array of words grouped by sentences ((w11, w12, w13, ...), (w21, w22, w23, ...), ...)
    Y - returns an array of tags grouped by sentences ((t11, t12, t13, ...), (t21, t22, t23, ...), ...)
    N - returns the number of distinct samples (individual words or tags) in the dataset

Methods:
    stream() - returns an flat iterable over all (word, tag) pairs across all sentences in the corpus
    __iter__() - returns an iterable over the data as (sentence_key, Sentence()) pairs
    __len__() - returns the nubmer of sentences in the dataset
```

For example, consider a Subset, `subset`, of the sentences `{"s0": Sentence(("See", "Spot", "run"), ("VERB", "NOUN", "VERB")), "s1": Sentence(("Spot", "ran"), ("NOUN", "VERB"))}`. The subset will have these attributes:

```
subset.keys == {"s1", "s0"}  # unordered
subset.vocab == {"See", "run", "ran", "Spot"}  # unordered
subset.tagset == {"VERB", "NOUN"}  # unordered
subset.X == (("Spot", "ran"), ("See", "Spot", "run"))  # order matches .keys
subset.Y == (("NOUN", "VERB"), ("VERB", "NOUN", "VERB"))  # order matches .keys
subset.N == 7  # there are a total of seven observations over all sentences
len(subset) == 2  # because there are two sentences
```

**Note:** The `Dataset` class is _convenient_, but it is **not** efficient. It is not suitable for huge datasets because it stores multiple redundant copies of the same data.

_____no_output_____#### Sentences

`Dataset.sentences` is a dictionary of all sentences in the training corpus, each keyed to a unique sentence identifier. Each `Sentence` is itself an object with two attributes: a tuple of the words in the sentence named `words` and a tuple of the tag corresponding to each word named `tags`._____no_output_____

key = 'b100-38532'
print("Sentence: {}".format(key))
print("words:\n\t{!s}".format(data.sentences[key].words))
print("tags:\n\t{!s}".format(data.sentences[key].tags))Sentence: b100-38532
words:
	('Perhaps', 'it', 'was', 'right', ';', ';')
tags:
	('ADV', 'PRON', 'VERB', 'ADJ', '.', '.')

**Note:** The underlying iterable sequence is **unordered** over the sentences in the corpus; it is not guaranteed to return the sentences in a consistent order between calls. Use `Dataset.stream()`, `Dataset.keys`, `Dataset.X`, or `Dataset.Y` attributes if you need ordered access to the data.

#### Counting Unique Elements

You can access the list of unique words (the dataset vocabulary) via `Dataset.vocab` and the unique list of tags via `Dataset.tagset`._____no_output_____

print(len(data))
print(data.N)57340
1161192
print("There are a total of {} samples of {} unique words in the corpus."
      .format(data.N, len(data.vocab)))
print("There are {} samples of {} unique words in the training set."
      .format(data.training_set.N, len(data.training_set.vocab)))
print("There are {} samples of {} unique words in the testing set."
      .format(data.testing_set.N, len(data.testing_set.vocab)))
print("There are {} words in the test set that are missing in the training set."
      .format(len(data.testing_set.vocab - data.training_set.vocab)))

assert data.N == data.training_set.N + data.testing_set.N, \
       "The number of training + test samples should sum to the total number of samples"There are a total of 1161192 samples of 56057 unique words in the corpus.
There are 928458 samples of 50536 unique words in the training set.
There are 232734 samples of 25112 unique words in the testing set.
There are 5521 words in the test set that are missing in the training set.

#### Accessing word and tag Sequences
The `Dataset.X` and `Dataset.Y` attributes provide access to ordered collections of matching word and tag sequences for each sentence in the dataset._____no_output_____

# accessing words with Dataset.X and tags with Dataset.Y
for i in range(2):
    print("Sentence {}:".format(i + 1), data.X[i])
    print()
    print("Labels {}:".format(i + 1), data.Y[i])
    print()Sentence 1: ('Mr.', 'Podger', 'had', 'thanked', 'him', 'gravely', ',', 'and', 'now', 'he', 'made', 'use', 'of', 'the', 'advice', '.')

Labels 1: ('NOUN', 'NOUN', 'VERB', 'VERB', 'PRON', 'ADV', '.', 'CONJ', 'ADV', 'PRON', 'VERB', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence 2: ('But', 'there', 'seemed', 'to', 'be', 'some', 'difference', 'of', 'opinion', 'as', 'to', 'how', 'far', 'the', 'board', 'should', 'go', ',', 'and', 'whose', 'advice', 'it', 'should', 'follow', '.')

Labels 2: ('CONJ', 'PRT', 'VERB', 'PRT', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'ADP', 'ADV', 'ADV', 'DET', 'NOUN', 'VERB', 'VERB', '.', 'CONJ', 'DET', 'NOUN', 'PRON', 'VERB', 'VERB', '.')

#### Accessing (word, tag) Samples
The `Dataset.stream()` method returns an iterator that chains together every pair of (word, tag) entries across all sentences in the entire corpus._____no_output_____

# use Dataset.stream() (word, tag) samples for the entire corpus
print("\nStream (word, tag) pairs:\n")
for i, pair in enumerate(data.stream()):
    print("\t", pair)
    if i > 5: break
Stream (word, tag) pairs:

	 ('Mr.', 'NOUN')
	 ('Podger', 'NOUN')
	 ('had', 'VERB')
	 ('thanked', 'VERB')
	 ('him', 'PRON')
	 ('gravely', 'ADV')
	 (',', '.')
type(data.stream())_____no_output_____

### Testing functions_____no_output_____

Input = [[('hi', 'bye')], [('Geeks', 'forGeeks')],
         [('a', 'b')], [('hi', 'bye')], [('a', 'b')]]

Input2 = [('hi', 'bye'), ('Geeks', 'forGeeks'),
         ('a', 'b'), ('hi', 'bye'), ('a', 'b')] _____no_output_____Input2[0][1]_____no_output_____for i in chain(Input):
    print(i)[('hi', 'bye')]
[('Geeks', 'forGeeks')]
[('a', 'b')]
[('hi', 'bye')]
[('a', 'b')]
Counter(chain(*Input)) _____no_output_____list(chain(*Input))_____no_output_____list(chain(*Input2))_____no_output_____List1 = [4, 2, 3, 1, 4, 1, 2]
List2 = [3, 3, 4, 2, 1, 3, 4]_____no_output_____for s in (zip(List1,List2)):
    print(s)(4, 3)
(2, 3)
(3, 4)
(1, 2)
(4, 1)
(1, 3)
(2, 4)
Counter(list(zip(List1,List2)))_____no_output_____Counter(zip(List1,List2))_____no_output_____

For both our baseline tagger and the HMM model we'll build, we need to estimate the frequency of tags & words from the frequency counts of observations in the training corpus. In the next several cells you will complete functions to compute the counts of several sets of counts. _____no_output_____## Step 2: Build a Most Frequent Class tagger
---

Perhaps the simplest tagger (and a good baseline for tagger performance) is to simply choose the tag most frequently assigned to each word. This "most frequent class" tagger inspects each observed word in the sequence and assigns it the label that was most often assigned to that word in the corpus._____no_output_____### IMPLEMENTATION: Pair Counts

Complete the function below that computes the joint frequency counts for two input sequences._____no_output_____

def pair_counts(sequences_A, sequences_B):
    """Return a dictionary keyed to each unique value in the first sequence list
    that counts the number of occurrences of the corresponding value from the
    second sequences list.

    For example, if sequences_A is tags and sequences_B is the corresponding
    words, then if 1244 sequences contain the word "time" tagged as a NOUN, then
    you should return a dictionary such that pair_counts[NOUN][time] == 1244
    """
    # TODO: Finish this function!

    ## chain all sequences elements into one list
    list_A = list(chain(*sequences_A))
    list_B = list(chain(*sequences_B))

    ## zip list of tags and list of words --> [(tag1, word1),(tag2,word2)...]
    ## count number of each (tag,word) combination

    #count_result = Counter(list(zip(list_A,list_B))) ## no need to list the zip
    count_result = Counter(zip(list_A,list_B))
    ## initialize dict of dict by collections.defaultdict
    count_dict = defaultdict(lambda:defaultdict(list))

    ## loop through count_result : {(tag1,word1):count1,..}
    ## tag_word[0] --> tag
    ## tag_word[1] --> word
    for tag_word, count in count_result.items():
        tag_i = tag_word[0]
        word_i = tag_word[1]
        if tag_word[0] not in count_dict:
            ## create new dict for this tag
            count_dict[tag_i] = {}

        count_dict[tag_i][word_i] = count

    return count_dict

    raise NotImplementedError

# Calculate C(t_i, w_i)
emission_counts = pair_counts(data.Y, data.X)

assert len(emission_counts) == 12, \
       "Uh oh. There should be 12 tags in your dictionary."
assert max(emission_counts["NOUN"], key=emission_counts["NOUN"].get) == 'time', \
       "Hmmm...'time' is expected to be the most common NOUN."
HTML('
Your emission counts look good!
')_____no_output_____emission_counts["NOUN"]['difference']_____no_output_____### Testing Counter()

result = Counter(list(zip(list(chain(*data.Y[1:3])), list(chain(*data.X[1:3])))))

## get counts number
# for count in result.values():
#     print(count)

##
## get tag and words
# for tag, word  in result.keys():
#     print(tag, word)

## get tag, word, count
# for key, value in result.items():
#     print(key[0],key[1], value)
_____no_output_____

### IMPLEMENTATION: Most Frequent Class Tagger

Use the `pair_counts()` function and the training dataset to find the most frequent class label for each word in the training data, and populate the `mfc_table` below. The table keys should be words, and the values should be the appropriate tag string.

The `MFCTagger` class is provided to mock the interface of Pomegranite HMM models so that they can be used interchangeably._____no_output_____

# Create a lookup table mfc_table where mfc_table[word] contains the tag label most frequently assigned to that word
from collections import namedtuple

FakeState = namedtuple("FakeState", "name")

class MFCTagger:
    # NOTE: You should not need to modify this class or any of its methods
    missing = FakeState(name="<MISSING>")

    def __init__(self, table):
        self.table = defaultdict(lambda: MFCTagger.missing)
        self.table.update({word: FakeState(name=tag) for word, tag in table.items()})

    def viterbi(self, seq):
        """This method simplifies predictions by matching the Pomegranate viterbi() interface"""
        return 0., list(enumerate(["<start>"] + [self.table[w] for w in seq] + ["<end>"]))

# TODO: calculate the frequency of each tag being assigned to each word (hint: similar, but not
# the same as the emission probabilities) and use it to fill the mfc_table

# TODO: YOUR CODE HERE
word_counts = pair_counts(data.training_set.X,data.training_set.Y)

# TODO: YOUR CODE HERE
mfc_table = dict((word,Counter(value).most_common(1)[0][0]) for word,value in word_counts.items())

# DO NOT MODIFY BELOW THIS LINE
mfc_model = MFCTagger(mfc_table) # Create a Most Frequent Class tagger instance

assert len(mfc_table) == len(data.training_set.vocab), ""
assert all(k in data.training_set.vocab for k in mfc_table.keys()), ""
assert sum(int(k not in mfc_table) for k in data.testing_set.vocab) == 5521, ""
HTML('
Your MFC tagger has all the correct words!
')_____no_output_____## testing Counter result
Counter(word_counts['to']).most_common(1)[0][0]_____no_output_____

### Making Predictions with a Model
The helper functions provided below interface with Pomegranate network models & the mocked MFCTagger to take advantage of the [missing value](http://pomegranate.readthedocs.io/en/latest/nan.html) functionality in Pomegranate through a simple sequence decoding function. Run these functions, then run the next cell to see some of the predictions made by the MFC tagger._____no_output_____

def replace_unknown(sequence):
    """Return a copy of the input sequence where each unknown word is replaced
    by the literal string value 'nan'. Pomegranate will ignore these values
    during computation.
    """
    return [w if w in data.training_set.vocab else 'nan' for w in sequence]

def simplify_decoding(X, model):
    """X should be a 1-D sequence of observations for the model to predict"""
    _, state_path = model.viterbi(replace_unknown(X))
    return [state[1].name for state in state_path[1:-1]]  # do not show the start/end state predictions_____no_output_____

### Example Decoding Sequences with MFC Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, mfc_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', '<MISSING>', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADV', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

### Evaluating Model Accuracy

The function below will evaluate the accuracy of the MFC tagger on the collection of all sentences from a text corpus. _____no_output_____

def accuracy(X, Y, model):
    """Calculate the prediction accuracy by using the model to decode each sequence
    in the input X and comparing the prediction with the true labels in Y.

    The X should be an array whose first dimension is the number of sentences to test,
    and each element of the array should be an iterable of the words in the sequence.
    The arrays X and Y should have the exact same shape.

    X = [("See", "Spot", "run"), ("Run", "Spot", "run", "fast"), ...]
    Y = [(), (), ...]
    """
    correct = total_predictions = 0
    for observations, actual_tags in zip(X, Y):

        # The model.viterbi call in simplify_decoding will return None if the HMM
        # raises an error (for example, if a test sentence contains a word that
        # is out of vocabulary for the training set). Any exception counts the
        # full sentence as an error (which makes this a conservative estimate).
        try:
            most_likely_tags = simplify_decoding(observations, model)
            correct += sum(p == t for p, t in zip(most_likely_tags, actual_tags))
        except:
            pass
        total_predictions += len(observations)
    return correct / total_predictions_____no_output_____

#### Evaluate the accuracy of the MFC tagger
Run the next cell to evaluate the accuracy of the tagger on the training and test corpus._____no_output_____

mfc_training_acc = accuracy(data.training_set.X, data.training_set.Y, mfc_model)
print("training accuracy mfc_model: {:.2f}%".format(100 * mfc_training_acc))

mfc_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, mfc_model)
print("testing accuracy mfc_model: {:.2f}%".format(100 * mfc_testing_acc))

assert mfc_training_acc >= 0.955, "Uh oh. Your MFC accuracy on the training set doesn't look right."
assert mfc_testing_acc >= 0.925, "Uh oh. Your MFC accuracy on the testing set doesn't look right."
HTML('
Your MFC tagger accuracy looks correct!
')training accuracy mfc_model: 95.72%
testing accuracy mfc_model: 93.01%

## Step 3: Build an HMM tagger
---
The HMM tagger has one hidden state for each possible tag, and parameterized by two distributions: the emission probabilties giving the conditional probability of observing a given **word** from each hidden state, and the transition probabilities giving the conditional probability of moving between **tags** during the sequence.

We will also estimate the starting probability distribution (the probability of each **tag** being the first tag in a sequence), and the terminal probability distribution (the probability of each **tag** being the last tag in a sequence).

The maximum likelihood estimate of these distributions can be calculated from the frequency counts as described in the following sections where you'll implement functions to count the frequencies, and finally build the model. The HMM model will make predictions according to the formula:

$$t_i^n = \underset{t_i^n}{\mathrm{argmax}} \prod_{i=1}^n P(w_i|t_i) P(t_i|t_{i-1})$$

Refer to Speech & Language Processing [Chapter 10](https://web.stanford.edu/~jurafsky/slp3/10.pdf) for more information._____no_output_____### IMPLEMENTATION: Unigram Counts

Complete the function below to estimate the co-occurrence frequency of each symbol over all of the input sequences. The unigram probabilities in our HMM model are estimated from the formula below, where N is the total number of samples in the input. (You only need to compute the counts for now.)

$$P(tag_1) = \frac{C(tag_1)}{N}$$_____no_output_____

def unigram_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequence list that
    counts the number of occurrences of the value in the sequences list. The sequences
    collection should be a 2-dimensional array.

    For example, if the tag NOUN appears 275558 times over all the input sequences,
    then you should return a dictionary such that your_unigram_counts[NOUN] == 275558.
    """
    # TODO: Finish this function!
    return Counter(chain(*sequences))
    raise NotImplementedError

# TODO: call unigram_counts with a list of tag sequences from the training set
tag_unigrams = unigram_counts(data.training_set.Y)

assert set(tag_unigrams.keys()) == data.training_set.tagset, \
       "Uh oh. It looks like your tag counts doesn't include all the tags!"
assert min(tag_unigrams, key=tag_unigrams.get) == 'X', \
       "Hmmm...'X' is expected to be the least common class"
assert max(tag_unigrams, key=tag_unigrams.get) == 'NOUN', \
       "Hmmm...'NOUN' is expected to be the most common class"
HTML('
Your tag unigrams look good!
')_____no_output_____# data.training_set.Y[1:3]_____no_output_____Counter(chain(*data.training_set.Y[1:3]))_____no_output_____Counter(chain(*data.training_set.Y[1:3]))['ADP']_____no_output_____

### IMPLEMENTATION: Bigram Counts

Complete the function below to estimate the co-occurrence frequency of each pair of symbols in each of the input sequences. These counts are used in the HMM model to estimate the bigram probability of two tags from the frequency counts according to the formula: $$P(tag_2|tag_1) = \frac{C(tag_2|tag_1)}{C(tag_2)}$$
_____no_output_____

def bigram_counts(sequences):
    """Return a dictionary keyed to each unique PAIR of values in the input sequences
    list that counts the number of occurrences of pair in the sequences list. The input
    should be a 2-dimensional array.

    For example, if the pair of tags (NOUN, VERB) appear 61582 times, then you should
    return a dictionary such that your_bigram_counts[(NOUN, VERB)] == 61582
    """

    # TODO: Finish this function!
    ## convert sequences to list
    ## zip list with list[1:] --> this tag with next tag (tag1, tag2)
    ## count (tag1, tag2)
    return Counter(zip(list(chain(*sequences)),list(chain(*sequences))[1:]))
    raise NotImplementedError

# TODO: call bigram_counts with a list of tag sequences from the training set
tag_bigrams = bigram_counts(data.training_set.Y)

assert len(tag_bigrams) == 144, \
       "Uh oh. There should be 144 pairs of bigrams (12 tags x 12 tags)"
assert min(tag_bigrams, key=tag_bigrams.get) in [('X', 'NUM'), ('PRON', 'X')], \
       "Hmmm...The least common bigram should be one of ('X', 'NUM') or ('PRON', 'X')."
assert max(tag_bigrams, key=tag_bigrams.get) in [('DET', 'NOUN')], \
       "Hmmm...('DET', 'NOUN') is expected to be the most common bigram."
HTML('
Your tag bigrams look good!
')_____no_output_____test_words_list = list(chain(*data.training_set.Y[1:3]))_____no_output_____# for i in zip(test_words_list,test_words_list[1:]):
#     print(i)_____no_output_____

### IMPLEMENTATION: Sequence Starting Counts
Complete the code below to estimate the bigram probabilities of a sequence starting with each tag._____no_output_____

def starting_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the beginning of
    a sequence.

    For example, if 8093 sequences start with NOUN, then you should return a
    dictionary such that your_starting_counts[NOUN] == 8093
    """
    # TODO: Finish this function!
    ## create list to store start Tag
    list_start_tag = []
    for i in sequences:
        start_tag = i[0]
        list_start_tag.append(start_tag)
    return Counter(list_start_tag)
    raise NotImplementedError

# TODO: Calculate the count of each tag starting a sequence
tag_starts = starting_counts(data.training_set.Y)

assert len(tag_starts) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_starts, key=tag_starts.get) == 'X', "Hmmm...'X' is expected to be the least common starting bigram."
assert max(tag_starts, key=tag_starts.get) == 'DET', "Hmmm...'DET' is expected to be the most common starting bigram."
HTML('
Your starting tag counts look good!
')_____no_output_____## Udacity reviewed
def pair_counts(sequences_A,sequences_B):
   dictionary = defaultdict(Counter)
    for i in range(len(sequences_A)):
        for key, value in zip(sequences_A[i],sequences_B[i]):
            dictionary[key][value] += 1

    return dictionary

def bigram_counts(sequences):
    counts = Counter()
    counts.update(chain(*(zip(s[:-1], s[1:]) for s in sequences)))
    return counts

def starting_counts(sequences):
    return Counter(next(zip(*sequences)))

def ending_counts(sequences):
    return Counter([sequence[-1] for sequence in sequences])_____no_output_____data.training_set.Y[0:3][0][0]_____no_output_____for i in data.training_set.Y[0:3]:
    print(i[0])ADV
ADP
ADJ

### IMPLEMENTATION: Sequence Ending Counts
Complete the function below to estimate the bigram probabilities of a sequence ending with each tag._____no_output_____

def ending_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the end of
    a sequence.

    For example, if 18 sequences end with DET, then you should return a
    dictionary such that your_starting_counts[DET] == 18
    """
    # TODO: Finish this function!
    ## create list to store Ending Tag
    list_end_tag = []
    for i in sequences:
        end_tag = i[-1]
        list_end_tag.append(end_tag)
    return Counter(list_end_tag)

    raise NotImplementedError

# TODO: Calculate the count of each tag ending a sequence
tag_ends = ending_counts(data.training_set.Y)

assert len(tag_ends) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_ends, key=tag_ends.get) in ['X', 'CONJ'], "Hmmm...'X' or 'CONJ' should be the least common ending bigram."
assert max(tag_ends, key=tag_ends.get) == '.', "Hmmm...'.' is expected to be the most common ending bigram."
HTML('
Your ending tag counts look good!
')_____no_output_____for i in data.training_set.Y[0:3]:
    print(i[-1]).
.
NOUN

### IMPLEMENTATION: Basic HMM Tagger
Use the tag unigrams and bigrams calculated above to construct a hidden Markov tagger.

- Add one state per tag
    - The emission distribution at each state should be estimated with the formula: $P(w|t) = \frac{C(t, w)}{C(t)}$
- Add an edge from the starting state `basic_model.start` to each tag
    - The transition probability should be estimated with the formula: $P(t|start) = \frac{C(start, t)}{C(start)}$
- Add an edge from each tag to the end state `basic_model.end`
    - The transition probability should be estimated with the formula: $P(end|t) = \frac{C(t, end)}{C(t)}$
- Add an edge between _every_ pair of tags
    - The transition probability should be estimated with the formula: $P(t_2|t_1) = \frac{C(t_1, t_2)}{C(t_1)}$_____no_output_____

basic_model = HiddenMarkovModel(name="base-hmm-tagger")
print("States:",basic_model.node_count())States: 0
basic_model = HiddenMarkovModel(name="base-hmm-tagger")

# TODO: create states with emission probability distributions P(word | tag) and add to the model
# (Hint: you may need to loop & create/add new states)

## count tags
c_tags = unigram_counts(data.training_set.Y)

## count words for each tag
c_words_tags = pair_counts(data.training_set.Y,data.training_set.X)

## loop each tag to get c_words for each tag
## calculate tag emission --> add states for tag

## {tag1:state1, tag2....}
states = {}

for tag_x, c_words in c_words_tags.items():
    ## calculate tag emission
    p_word_tag = dict((word, c_word/c_tags[tag_x]) for word, c_word in c_words.items())
    state_emission = DiscreteDistribution(p_word_tag)
    state = State(state_emission, name = tag_x)

    ## add each states
    basic_model.add_state(state)

    ## attach state to states
    states[tag_x] = state

print("States:",basic_model.node_count())

print(len(states))

# TODO: add edges between states for the observed transition frequencies P(tag_i | tag_i-1)
# (Hint: you may need to loop & add transitions

### add_transition Start --> State
c_starts = starting_counts(data.training_set.Y)

for start_tag, count_x in c_starts.items():

    basic_model.add_transition(basic_model.start, states[start_tag],count_x/len(data.training_set))

### add State --> State
# count tag1_tag2
c_tag1_tag2 = bigram_counts(data.training_set.Y)

for tag1_tag2, count_y in c_tag1_tag2.items():
    tag1 = tag1_tag2[0]
    tag2 = tag1_tag2[1]
    p_tag1_tag2 = count_y/c_tags[tag1]

    basic_model.add_transition(states[tag1],states[tag2],p_tag1_tag2)

### add_transition State --> End
c_ends = ending_counts(data.training_set.Y)
for end_tag, count_z in c_ends.items():

    basic_model.add_transition(states[end_tag],basic_model.end, count_z/len(data.training_set))

print("Edge Count ",basic_model.edge_count())

# NOTE: YOU SHOULD NOT NEED TO MODIFY ANYTHING BELOW THIS LINE
# finalize the model
basic_model.bake()

assert all(tag in set(s.name for s in basic_model.states) for tag in data.training_set.tagset), \
       "Every state in your network should use the name of the associated tag, which must be one of the training set tags."
assert basic_model.edge_count() == 168, \
       ("Your network should have an edge from the start node to each state, one edge between every " +
        "pair of tags (states), and an edge from each state to the end node.")
HTML('
Your HMM network topology looks good!
')States: 0
12
Edge Count  0
hmm_training_acc = accuracy(data.training_set.X, data.training_set.Y, basic_model)
print("training accuracy basic hmm model: {:.2f}%".format(100 * hmm_training_acc))

hmm_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, basic_model)
print("testing accuracy basic hmm model: {:.2f}%".format(100 * hmm_testing_acc))

assert hmm_training_acc > 0.97, "Uh oh. Your HMM accuracy on the training set doesn't look right."
assert hmm_testing_acc > 0.955, "Uh oh. Your HMM accuracy on the testing set doesn't look right."
HTML('
Your HMM tagger accuracy looks correct! Congratulations, you\'ve finished the project.
')training accuracy basic hmm model: 97.53%
testing accuracy basic hmm model: 95.96%

### Example Decoding Sequences with the HMM Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, basic_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

## Finishing the project
---

**Note:** **SAVE YOUR NOTEBOOK**, then run the next cell to generate an HTML copy. You will zip & submit both this file and the HTML copy for review.

_____no_output_____

!!jupyter nbconvert *.ipynb_____no_output_____

## Step 4: [Optional] Improving model performance
---
There are additional enhancements that can be incorporated into your tagger that improve performance on larger tagsets where the data sparsity problem is more significant. The data sparsity problem arises because the same amount of data split over more tags means there will be fewer samples in each tag, and there will be more missing data  tags that have zero occurrences in the data. The techniques in this section are optional.

- [Laplace Smoothing](https://en.wikipedia.org/wiki/Additive_smoothing) (pseudocounts)
    Laplace smoothing is a technique where you add a small, non-zero value to all observed counts to offset for unobserved values.

- Backoff Smoothing
    Another smoothing technique is to interpolate between n-grams for missing data. This method is more effective than Laplace smoothing at combatting the data sparsity problem. Refer to chapters 4, 9, and 10 of the [Speech & Language Processing](https://web.stanford.edu/~jurafsky/slp3/) book for more information.

- Extending to Trigrams
    HMM taggers have achieved better than 96% accuracy on this dataset with the full Penn treebank tagset using an architecture described in [this](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf) paper. Altering your HMM to achieve the same performance would require implementing deleted interpolation (described in the paper), incorporating trigram probabilities in your frequency tables, and re-implementing the Viterbi algorithm to consider three consecutive states instead of two.

### Obtain the Brown Corpus with a Larger Tagset
Run the code below to download a copy of the brown corpus with the full NLTK tagset. You will need to research the available tagset information in the NLTK docs and determine the best way to extract the subset of NLTK tags you want to explore. If you write the following the format specified in Step 1, then you can reload the data using all of the code above for comparison.

Refer to [Chapter 5](http://www.nltk.org/book/ch05.html) of the NLTK book for more information on the available tagsets._____no_output_____

import nltk
from nltk import pos_tag, word_tokenize
from nltk.corpus import brown

nltk.download('brown')
training_corpus = nltk.corpus.brown
training_corpus.tagged_sents()[0]_____no_output_____


# Notebook from Lawrence-Krukrubo/EDA
Path: rms_titanic_eda.ipynb

_____no_output_____# PREAMBLE
_____no_output_____
PREAMBLE

In this project, I will analyze The Titanic dataset and then communicate my findings about it, using the Python libraries NumPy, Pandas, and Matplotlib to make my analysis easier.

**What do I need to install?**

I need an installation of _Python_, plus the following libraries:

pandas

numpy

matplotlib

csv or unicodecsv

installing _Anaconda_ is the best option, which comes with all of the necessary packages, as well as IPython notebook.

**Why this Project?**
This project will introduce me to the data analysis process. In this project, I will go through the entire process so that I know how all the pieces fit together. In this project, I will also gain experience using the Python libraries NumPy, Pandas, and Matplotlib, which make writing data analysis code in Python a lot easier!

**What will I learn?**

After completing the project, I will:

Know all the steps involved in a typical data analysis process,

Be comfortable posing questions that can be answered with a given dataset and then answering those questions,

Know how to investigate problems in a dataset and wrangle the data into a format that can be used

Have practice communicating the results of my analysis

Be able to use vectorized operations in NumPy and Pandas to speed up data analysis code

Be familiar with Pandas' Series and DataFrame objects, which us access data more conveniently

Know how to use Matplotlib to produce plots showing my findings

**Why is this Important to my Career?**

This project will show off a variety of data analysis skills, as well as showing everyone that I know how to go through the entire data analysis process._____no_output_____
RMS Titanic

The RMS Titanic was a British passenger liner that sank in the North Atlantic Ocean in the early morning hours of _15 April 1912_, after it collided with an iceberg during its maiden voyage from _Southampton to New York City_. There were an estimated _2,224_ passengers and crew aboard the ship, and more than _1,500_ died, making it one of the deadliest commercial peacetime maritime disasters in modern history. The RMS Titanic was the largest ship afloat at the time it entered service and was the second of three Olympic-class ocean liners operated by the _White Star Line_.

The Titanic was built by the _Harland and Wolff shipyard in Belfast_.

  Thomas Andrews, her architect, died in the disaster.

_The Titanic sits near the dock at Belfast, Northern Ireland soon before starting its maiden voyage. Circa April 1912_.

_____no_output_____# EDA_____no_output______img courtesy of [Data Camp](https://www.datacamp.com)_
_____no_output_____
Exploratory Data Analysis of The Titanic Data Set

This Data set consists of passengers of the Titanic.
The essence of this analysis is to provide more insights about the Titanic data set. I would go through this analysis with an open mind, looking at the data,
Asking relevant questions, evaluating metrics and displaying similarities and or differences in data variables that may have been consequential in affecting :
 * _**Passengers that survived**_,

 * _**Passengers that died**_,

 * _**Any other valuable insights from the data**_._____no_output_____
Let's begin by importing some Libraries for data analysis and visualization
_____no_output_____

import numpy as np  # for numerical analysis
import pandas as pd  # for a tabular display of the data
import matplotlib as mpl  # for visualization
import matplotlib.pyplot as plt  # for visualization using the scripting layer
import seaborn as sns  # for advanced visualization
import sklearn  # for prediction or machine learning
import folium  # for creating interactive maps
from PIL import Image # converting images into arrays
from wordcloud import WordCloud, STOPWORDS  # for word cloud creation
!pip install pywaffle
from pywaffle import Waffle  # for waffle charts creation

print('All modules imported successfully')Requirement already satisfied: pywaffle in /usr/local/lib/python3.6/dist-packages (0.2.1)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from pywaffle) (3.0.3)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->pywaffle) (0.10.0)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->pywaffle) (2.5.3)
Requirement already satisfied: numpy>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from matplotlib->pywaffle) (1.16.3)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->pywaffle) (1.1.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->pywaffle) (2.4.0)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from cycler>=0.10->matplotlib->pywaffle) (1.12.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from kiwisolver>=1.0.1->matplotlib->pywaffle) (41.0.1)
All modules imported successfully

Loading The Titanic Data set to a pandas Data Frame:

Note that the Titanic Data set we would use for this analysis can be downloaded from the project lesson of _**Udacity- Intro to Data Analysis course**_,

Which is a free course available at [Udacity](https://www.udacity.com/course/intro-to-data-analysis--ud170)_____no_output_____Another way to directly load a copy of this data set to a Data Frame, is from the Seaborn Library data sets, just like this:

(_**although for this project we would stick to the data set from Udacity**_)_____no_output_____

# Loading Titanic data set into a pandas dataframe from seaborn library in just one line of code
titanic_df = sns.load_dataset('titanic')

# Visualizing the first 3 rows of the data frame
titanic_df.head(3)_____no_output_____

So let's get to our Titanic Data set from Udacity. I preloaded it in github for easy access to colab. So we would import the data set from github.

Features of The Titanic Data Set.

**PassengerId** - Numeric Id for each passenger onboard

**survived** - Survival (0 = No; 1 = Yes)

**Pclass** - Passenger Class (1 = 1st; 2 = 2nd; 3 = 3rd)

**name** - Name

**sex** - Sex

**age** - Age

**sibsp** - Number of Siblings/Spouses Aboard

**parch** - Number of Parents/Children Aboard

**ticket** - Ticket Number

**fare** - Passenger Fare

**cabin** - Cabin

**embarked** - Port of Embarkation (C = Cherbourg; Q = Queenstown; S = Southampton)_____no_output_____Importing The raw Titanic Data set from github_____no_output_____

titanic_data = 'https://raw.githubusercontent.com/Blackman9t/EDA/master/titanic_data.csv'_____no_output_____

Reading it into a Pandas Data Frame_____no_output_____Although pandas has a robust missing-values detection algorithm, experience has taught us that some missing value types may go undetected, unless we hard-code them.

Let's add some more possible missing value types to the default pandas collection_____no_output_____

# Making a list of additional missing value types added to the default NA type that pandas can detect

missing_values = ["n/a", "na", "--",'?']_____no_output_____titanic_df = pd.read_csv(titanic_data, na_values = missing_values)

# Let's view the first 10 entries of the data set
titanic_df.head(10)_____no_output_____

Let's check the shape to know how many total rows and columns are involved_____no_output_____

titanic_df.shape_____no_output_____

This tells us there are 891 passenger entries  in the Titanic and 12 passenger features... Let's see the default summary statistics of the Data set_____no_output_____

titanic_df.describe(include='all')

# By default only numeric columns are computed.
# If we want to view summary statistics for all columns then run;  titanic_df.describe(include='all')_____no_output_____

We can see from the summary statistics that:-

 The average survival rate when The Titanic sank was 38% only.

  Age column has only 714 numeric entries as against 891 like the rest of the numeric data columns...

  The average or mean age of passengers aboard the titanic was about 30 years.

  The oldest person or maximum age was 80 years old.

  The minimum age was less than a year... we can investigate further

  Also the average passenger fare was about 32 pounds

  While the most expensive tickets sold for slightly above 500 pounds... interesting._____no_output_____
Let's look at the data types of all the columns to confirm that the right data types are in place before we start the analysis:
 _____no_output_____

titanic_df.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
PassengerId    891 non-null int64
Survived       891 non-null int64
Pclass         891 non-null int64
Name           891 non-null object
Sex            891 non-null object
Age            714 non-null float64
SibSp          891 non-null int64
Parch          891 non-null int64
Ticket         891 non-null object
Fare           891 non-null float64
Cabin          204 non-null object
Embarked       889 non-null object
dtypes: float64(2), int64(5), object(5)
memory usage: 83.6+ KB

Okay, all numerical columns have the right int or float type to make vectorized computations easy, the rest are also in good order._____no_output_____**Next let's check for the number of NaN  or unknown values per column.**_____no_output_____

titanic_df.isna().sum()_____no_output_____

We an clearly see that almost all columns are clean except:-

Age: with 177 missing values,

Cabin: with 687 missing values

Embarked: 2 missing values...

We shall deal with these soon._____no_output_____To find total number of NaN values_____no_output_____

titanic_df.isna().sum().sum()_____no_output_____

To check if missing values in one column_____no_output_____

titanic_df.Age.isna().values.any()_____no_output_____

Visualizing missing values_____no_output_____

import missingno as msno
msno.bar(titanic_df, figsize=(12, 6), fontsize=12, color='steelblue')
plt.show()_____no_output_____

Fixing the missing Age Column:
_____no_output_____Let's visualize the age column with a histogram, to see the distribution of ages_____no_output_____

x = titanic_df.Age.copy(deep=True)_____no_output_____x.dropna(axis=0, inplace=True)

len(x)_____no_output_____count, bin_edges = np.histogram(x, bins=10, range=(0,100))
print('count is',count,'\nBin edges are:',bin_edges)count is [ 62 102 220 167  89  48  19   6   1   0]
Bin edges are: [  0.  10.  20.  30.  40.  50.  60.  70.  80.  90. 100.]
plt.hist(x, bins=bin_edges, edgecolor='black')
plt.title('Histogram of Age Distribution.')
plt.xlabel('Age-Range')
plt.ylabel('Frequency')
plt
plt.show()_____no_output_____

Let's print out the measures of central tendency of the distribution_____no_output_____

mode_age = (x.mode())[0]
mean_age = x.mean()
median_age = x.median()

print('Mean age is',mean_age)
print('Median age is',median_age)
print('Mode age is',mode_age)Mean age is 29.69911764705882
Median age is 28.0
Mode age is 24.0

We will now summarize the main features of the distribution of ages as it appears from the histogram:

**Shape:** The distribution of ages is skewed right. This means we have a concentration of data of young people in The Titanic,
And a progressively fewer number of older people, making the histogram to skew to the right.

It is also **unimodal** in shape, with just one dominant mode range of passenger ages between 20 - 30 years.

**Center:** The data seem to be centered around 28 years old. Note that this implies that roughly half the passengers in the Titanic  are less than 30 years old.

This is also reflected by a mean, median and modal age of 30 years.

**Spread:** The data range is from about 0 to about 80, so the approximate range equals 80 - 0 = 80.

**Outliers:** There are no outliers in the Age data as all values seem evenly distributed, with a steady decrease of the number of passengers above the 30 - 40 age group.

We can conclude that The Titanic had more passengers in the age range 0 to 30 years,

And the most frequent age-range of all Titanic passengers was 20 - 30 years of age._____no_output_____finally we shall define a method that randomly replaces the missing age values with either the mean, median or mode values._____no_output_____

def rand_age(x):
    """ Takes a value x, and returns the float form of x.
    If x gives an error, then return either the mode, median or mean age"""
    try:
        int(x)
        return float(x)
    except:
        i = [mean_age, mode_age, median_age]
        y = np.random.randint(0, len(i))

    return i[y]_____no_output_____

Next, lets apply that method to the age column_____no_output_____

titanic_df.Age = titanic_df.Age.apply(rand_age)
_____no_output_____

Let's confirm the changes_____no_output_____

titanic_df.Age.isna().any()_____no_output_____

Let's look at the 3 different classes of passengers:
 _____no_output_____First let's check the distribution of passengers in each class_____no_output_____

# we need to be sure that there are only 3 classses (3, 2, 1) in the data set.
# let's use the unique method of pandas to verify

titanic_df.Pclass.unique()_____no_output_____# next let's check the distribution size of each class of passengers
# we can easily do this with pandas groupby function.

# Let's group Pclasss by size and cast to a Data Frame.
classes = titanic_df.groupby('Pclass').size().to_frame()

# Let's rename the column
classes.rename(columns={0:'total'}, inplace=True)

# Let's customize the index
classes.index = ['1st Class','2nd Class','3rd Class']

# and display the result
classes_____no_output_____

We can see that out of `891 passengers` aboard the Titanic, `216` were in 1st Class, `184` in 2nd class and a whopping `491` in 3rd Class._____no_output_____
Visualizing Passenger Distribution of The Titanic Using Bar and Pie plots.
_____no_output_____

plt.figure(figsize=(18, 6))
sns.set(font_scale=1.2)
sns.set_style('ticks') # change background to white background
plt.suptitle('Visualizing Passenger Distribution of The Titanic using a Bar and Pie plot', y=1.05)

# For The Bar chart
plt.subplot(121)
color_list = ['gold','purple','brown']
plt.bar(x=classes.index, height=classes.total, data=classes, color= color_list, width=0.5)
plt.title('Bar Chart showing Distribution of Passengers in The Titanic')
plt.xlabel('Classes')
plt.ylabel('Number of Passengers')
for x,y in zip(classes.index, classes.total):
    label = round(y,2)  #  could also be written as:-   "{:.2f}".format(y)

    plt.annotate(label, # this is the text
                 (x,y), # this is the point to label
                 textcoords="offset points", # how to position the text
                 xytext=(0,4), # distance from text to points (x,y)
                 ha='center',) # horizontal alignment can be left, right or center

# For The Pie chart
plt.subplot(122)
plt.pie(classes.total,
        data=classes,
       autopct='%1.1f%%',
       colors=color_list,
       startangle=90,
       shadow=True,
       pctdistance=1.15)
plt.title('Pie Chart showing Percentage Distribution of Passengers in The Titanic')
plt.axis('equal')
plt.legend(labels=classes.index, loc='upper right')

plt.show()_____no_output_____

Average Ticket Fares to The Titanic Per Passenger Class
_____no_output_____The above plots represent the norm in most human activities, regular tickets often sell more than VIP tickets.

Let's find out the average  and max ticket fares for 1st Class, 2nd Class and 3rd Class passengers_____no_output_____

# first we form two groups of average and max ticket fares per class
ave_ticket_per_class = titanic_df[['Pclass','Fare']].groupby('Pclass').mean()
max_ticket_per_class = titanic_df[['Pclass','Fare']].groupby('Pclass').max()

# Next we group them together
ave_ticket_per_class['Max_Fare'] = max_ticket_per_class['Fare']
ave_ticket_per_class.rename(columns={'Fare':'Ave_Fare'},inplace=True)

# Finally rename it
ticket_range = ave_ticket_per_class

# And display
#ticket_range.sort_values(['Pclass'],ascending=True, inplace=True)
ticket_range

ticket_range.sort_index(ascending=False, inplace=True)
ticket_range  = ticket_range.transpose()
ticket_range_____no_output_____

With the average price of a 3rd class ticket going for about 14 pounds and that of a 1st class ticket going for 84 pounds on average;

It can be inferred that ticket price could be one of the reasons why more than half the passengers aboard the Titanic were in 3rd class.

Notice that the price difference between 3rd and 2nd Class tickets is minimal

  _____no_output_____
Visualizing average to max range of Ticket Fares per Passenger Class of The Titanic Using a Box Plot.
_____no_output_____

plt.figure(figsize=(10,6))
sns.boxplot('Pclass','Fare', data=titanic_df)
sns.set(font_scale=1.3)
plt.title('Boxplot showing range of Ticket Fares for The Titanic')
plt.xlabel('Passenger Classes')
plt.ylabel('Ticket Fare')
plt.show()_____no_output_____

Once again the boxplot shows striking similarities between the ticket fares for 3rd and 2nd class passengers of The Titanic_____no_output_____
Let's see the correlation between fares and passenger classes.
_____no_output_____

titanic_df['Pclass'].corr(titanic_df['Fare'])_____no_output_____

A correlation figure of -0.55 indicates an above average negative relationship between passenger classes and ticket fares.

This could mean that as ticket fares tend to rise, the number of passengers tend to drop and vice-versa.

 Note that correlation does not imply causation...
The fact that two variables seem to have a negative, positive or no relationship, does not imply that one variable causes the other to occur or not.

 See Visualization below_____no_output_____

corr_data= titanic_df.corr()

plt.figure(figsize=(12,8))
sns.heatmap(corr_data, annot=True)
plt.title('Heat-Map showing The correlation of variables in Titanic Data Set')

plt.show()_____no_output_____

Let's look at the age distribution of passengers aboard The Titanic:
 _____no_output_____Earlier we saw that:
The average age of all passengers was about 30 years.
The minimum age was about 5 months (0.42 * 12).
The maximum was 80 years._____no_output_____**Let's check the minimum age again**_____no_output_____

# First lets investigate the minimum age again

titanic_df.Age.min()_____no_output_____

It's possible that there were babies just a few months old aboard the Titanic.
This could be the reason why we have some age less than one year old,
Let's look at the distribution of passengers below one year old._____no_output_____

titanic_df[(titanic_df.Age<1)]_____no_output_____

We can see 7 passengers below one year old. Made up of 5 little boys and 2 pretty little girls.
And from their titles (`Master` or `Miss`), we can safely assume they were indeed just a few months old when the Titanic crashed.
On a positive note, it is good that all these infants survived as their `Survived` status is 1._____no_output_____The details of the youngest passenger aboard The Titanic _____no_output_____

titanic_df[np.logical_and(titanic_df.Name, titanic_df.Age==titanic_df.Age.min())]_____no_output_____

The details of the oldest passenger aboard The Titanic _____no_output_____

titanic_df[np.logical_and(titanic_df.Name, titanic_df.Age==titanic_df.Age.max())]_____no_output_____

The Wealthiest passenger

John Jacob Astor IV._____no_output_____
 Did you know?

The Titanic was over 882 feet long (almost 3 football fields)... And it weighed 52,310 tons

courtesy [goodhousekeeping](https://www.goodhousekeeping.com/life/g19809308/titanic-facts/?slide=3)_____no_output_____So we have 177 NaN values. Add that to 714 and total is 891 entries as expected.

When it comes to dealing with NaN values, we usually have the following options:-

1. We can leave it the way it is, if this option would not affect our computational or visual analysis

2. We can replace NaN values with either the mean or mode of the distribution

3. We can delete NaN values from the Data set, this would mean reducing the Data size and is best suited for large data sets with a few NaN values. _____no_output_____**Let's see the Class and Sex summary of passengers with NaN Age values.**_____no_output_____There is no given pattern, but clearly more passengers in 3rd class do not have their age values.
First class has 30 entries and 2nd class about a dozen entries._____no_output_____**Let's replace all NaN Age values with the average age of passengers in the Data set**

_____no_output_____
Visualizing Age Distribution of Passengers using a Hist and Dist Plot
_____no_output_____

fig = plt.figure(figsize=(18, 6))
sns.set_style('ticks')

ax0 = fig.add_subplot(121)
ax1 = fig.add_subplot(122)

plt.suptitle('Visualizing Age Distribution of Passengers using a Hist and Dist Plot', y=1.05)
#Histogram
titanic_df.Age.plot(kind='hist', edgecolor='navy', ax=ax0, facecolor='peru')
ax0.set_title('Histogram of Age Distribution of Titanic Passengers')
ax0.set_xlabel('Age')

#Distplot
sns.distplot(titanic_df.Age, hist=False, color='r', ax=ax1, label='Age Distribution')
ax1.set_title('Distplot of Age Distribution of Titanic Passengers')

plt.show()_____no_output_____

We will now summarize the main features of the distribution of ages as it appears from the histogram:

**Shape:** The distribution of ages is skewed right. This means we have a concentration of data of young people in The Titanic,
And a progressively fewer number of older people, making the histogram to skew to the right.

It is also **unimodal** in shape, with just one dominant mode range of passenger ages between 20 - 30 years.

**Center:** The data seem to be centered around 30 years old. Note that this implies that roughly half the passengers in the Titanic  are less than 30 years old.

This is also reflected by a mean, median and modal age of 30 years.

**Spread:** The data range is from about 0 to about 80, so the approximate range equals 80 - 0 = 80.

**Outliers:** There are no outliers in the Age data as all values seem evenly distributed, with a steady decrease of the number of passengers above the 30 - 40 age group.

We can conclude that The Titanic had more passengers in the age range 0 to 30 years,

And the most frequent age-range of all Titanic passengers was 20 - 30 years of age._____no_output_____

# further proof of the above assertion can be seen below
# Titanic had 586 passengers below age 30 and 305 passengers above 30 years.

titanic_df.groupby(titanic_df['Age'] <= 30).size()_____no_output_____

Code above shows that out of the `891` passengers, `586` `(66%)` were less than 30 years old and `305` `(34%)` were 30 years and above. _____no_output_____
Let's compare the age distribution of Males and Females aboard The Titanic
_____no_output_____First Let's create two separate Data Frames for Males and Females._____no_output_____

males = titanic_df[titanic_df.Sex=='male']
females = titanic_df[titanic_df.Sex=='female']_____no_output_____

Let's see the summary statistics for males and females..._____no_output_____

males.describe()_____no_output_____females.describe()_____no_output_____

Summary Statistics
_____no_output_____From the summary above, we can see that:-

  1. There were 577 males and 314 female passengers on The Titanic.

  2. The average age for males was about 30 and about 28 for women.

  3. Interestingly women paid 45 pounds for a ticket, while men paid 26 pounds on average...
We would investigate why, but I'm thinking the difference may be as a result of more women in 1st class and 2nd class than men.
Or a greater proportion of women in 1st and 2nd class seats than men

4. The maximum age for women was 63 and max age for men was 80 years._____no_output_____
Visualizing Age-Group Distribution of Male and Female Passengers using a Horizontal Bar Plot
_____no_output_____

age_range = ['[0.0 - 10]','[10 - 20]','[20 - 30]','[30 - 40]','[40 - 50]','[50 - 60]','[60 - 70]','[70 - 80]']
age_dict = {}_____no_output_____def age_grades(dataframe):
    start = 0

    for i in age_range:
        stop = start + 10
        x = dataframe[np.logical_and(dataframe.Age>start, dataframe.Age<=stop)].shape[0]
        age_dict[i] = [x]
        start += 10
    return age_dict

males_dict = age_grades(males)
males_df = pd.DataFrame(males_dict)
males_df = males_df.transpose()_____no_output_____females_dict = age_grades(females)
females_df = pd.DataFrame(females_dict)
females_df = females_df.transpose()
_____no_output_____maleFemaleAgeRange = pd.concat([males_df, females_df], axis=1)
maleFemaleAgeRange.columns = ['Male_Age_Range', 'Female_Age_Range']_____no_output_____maleFemaleAgeRange_____no_output_____sns.set_style('ticks')
ax = maleFemaleAgeRange.plot(kind='barh', color=['blue','red'], figsize=(14,10), width=0.75)
sns.set(font_scale=1.5)
ax.set_title('Age-Group Distribution for Males and Females aboard The Titanic')
ax.set_xlabel('Number of Passengers')
ax.set_ylabel('Age-Group')
for i in ax.patches:
    # get_width pulls left or right; get_y pushes up or down
    ax.text(i.get_width()+3, i.get_y()+.3, \
            str(round((i.get_width()), 2)), fontsize=15, color='black')

# invert to set y-axis in ascending order
ax.invert_yaxis()

plt.show()_____no_output_____

From the Horizontal Bar plot above, we can easily see that:-

  1. The `20 - 30` age group has the highest concentration of passengers.

  273 males and 134 females with a total count of 407 passengers.

  ```
  (407 divided by 891) * 100 = 46% of passengers.
  ```

  2. The next most populous age-group is the `30 - 40` group consisting of a total of 155 passengers._____no_output_____
Okay, lets look at The distribution of males and females in the three passenger classes.

Recall that the average price of female tickets was about 45 pounds which was about 75% more expensive than the average male passenger ticket of 26 pounds.
One possible reason could be that there were more female passengers in the higher classes(1st class, 2nd class) than male passengers.
Or that the percentage of females to the population of females (proportion) is higher than the proportion of males in the higher classes of passengers.
 Let's see to that.
_____no_output_____**Let's define a Data frame for the number of men and women per class**_____no_output_____

sex_per_class = titanic_df.groupby(['Pclass','Sex']).size().to_frame()
sex_per_class.reset_index(inplace=True)
sex_per_class_____no_output_____

**Next let's define a simple method that calculates the total proportion of males and females per class**_____no_output_____

total_f = len(titanic_df[titanic_df.Sex=='female'])
total_m = len(titanic_df[titanic_df.Sex=='male'])

def pct_(series):
    """Takes a series of numeric values and converts each value
    to a percent based on its index.
    Returns a list of converted values to pct,
    For total males and females of the Titanic."""
    x = list(series)
    for i in range(len(x)):
        if i % 2 == 0:
            x[i] = round((x[i] / total_f)*100)
        else:
            x[i] = round((x[i] / total_m)*100)
    return x_____no_output_____

**Next let's append that proportion as a column to the sex_per_class data frame and rename the 0 column to 'Count'.**_____no_output_____

sex_per_class['Pct_of_total(M/F)_per_class'] = pct_(sex_per_class[0])
sex_per_class.rename(columns={0:'Count'}, inplace=True)_____no_output_____

**Finally we can view it**_____no_output_____

sex_per_class_____no_output_____

We can clearly see the following:

1. The count of males in each passenger class is higher than the count of females.

2. But the proportion of females in the higher classes(1st, 2nd) is more than the proportion of males.

3. We can see that the proprtion of females to males in 1st class is 30% against 21%,

    And 24% against 19% in 2nd class... While on the flip side the males have 60% of their population in 3rd class against 46% for the females.

4. This accounts for why female tickets on average cost more than male tickets, because the percentage of females in higher classes is more than males, and as a result the average ticket fare for females is 45 pounds, against 26 pounds for males._____no_output_____
Visualizing The proportions of male and female passengers per class, using Waffle Charts
_____no_output_____

sns.set(font_scale=1.5)
sns.set_style('ticks')

first_class = {'Females': 30, 'Males': 21}
second_class = {'Females': 24, 'Males': 19}
third_class = {'Females': 46, 'Males': 60}

plt.figure(
    FigureClass=Waffle,
    rows=5,
    values=first_class,
    colors=("#983D3D", "#232066"),
    title={'label': 'Proportion of male and female passengers in 1st Class', 'loc': 'left'},
    labels=["{0} ({1}%)".format(k, v) for k, v in first_class.items()],
    legend={'loc': 'lower left', 'bbox_to_anchor': (0, -0.45), 'ncol': len(first_class), 'framealpha': 0},
    plot_direction='NW',
)

plt.figure(
    FigureClass=Waffle,
    rows=5,
    values=second_class,
    colors=("#983D3D", "#232066"),
    title={'label': 'Proportion of male and female passengers in 2nd Class', 'loc': 'left', 'color':'darkgreen'},
    labels=["{0} ({1}%)".format(k, v) for k, v in second_class.items()],
    legend={'loc': 'lower left', 'bbox_to_anchor': (0, -0.4), 'ncol': len(second_class), 'framealpha': 0},
    plot_direction='NW',
)

plt.figure(
    FigureClass=Waffle,
    rows=7,
    values=third_class,
    colors=("#983D3D", "#232066"),
    title={'label': 'Proportion of male and female passengers in 3rd Class', 'loc': 'left', 'color':'navy'},
    labels=["{0} ({1}%)".format(k, v) for k, v in third_class.items()],
    legend={'loc': 'lower left', 'bbox_to_anchor': (0, -0.45), 'ncol': len(third_class), 'framealpha': 0},
    plot_direction='NW',
)

fig.gca().set_facecolor('#EEEEEE')
fig.set_facecolor('#EEEEEE')
plt.show()
/usr/local/lib/python3.6/dist-packages/matplotlib/figure.py:2369: UserWarning: This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.
  warnings.warn("This figure includes Axes that are not compatible "

_____no_output_____**next,  let's view who bought the most xpensive tickets, survival data per class, survival data per sex, word cloud in tribute to those who died in 1st , 2nd or 3rd classes. use of lambda expressions, map of the location where titanic crashed. **

    [titanic_crash_site](http://www.shipwreckworld.com/maps/rms-titanic)_____no_output_____

# we select the 3rd class passengers in a group using pandas.
third_class_group = titanic_df['Pclass'] == 3

# then we assign that selection to the slice of titanic_df
third_class_df = titanic_df[third_class_group]

# we view the first 5 entries of the 3rd class passengers data frame.
third_class_df.head()_____no_output_____third_class_df.shape_____no_output_____# Do a word cloud in tribute to those in third class with their names. _____no_output__________no_output_____