# Notebook from TheDragon246/brainiak-tutorials
Path: tutorials/12-hmm.ipynb

# Hidden Markov Models for Event Segmentation
[Contributions](#contributions)

If asked to give a quick description of a dinner with friends, you might say something like the following: "First, we met outside the restaurant while waiting for our table. Once we got to our table, we continued talking for a bit before we ordered something to drink and a few appetizers. Later, everyone ordered their dinner. The food arrived after some time and we all began eating. Finally, it was time for dessert and we continued chatting with each other until desert arrived. After this, we split the bill and headed out of the restaurant. We said our goodbyes to each other, while waiting for our taxis, and went home." From this description it is clear that the dinner meeting was composed of stages, or events, that occurred sequentially. Furthermore, these events can be perceived at varying scales. At the longest time scale, the entire dinner could be treated as one event. At smaller time scales, subsets of the meeting such as entering the restaurant, taking off coats, being seated, looking at menus and so on, can be treated as different events. At an even smaller scale, the event of entering the restaurant can be broken up into different sub-events. Regardless of scale, all of these accounts share the property that the event can be represented as a sequence of stages.

The goal of this notebook is to explore methods for finding these “sequence-of-stages” representations in the brain. To accomplish this, we use a machine learning technique called Hidden Markov Modeling (HMM). These models assume that your thoughts progress through a sequence of states, one at a time. You can’t directly observe people’s thoughts, so the states are “hidden” (not directly observable). However, you can directly observe BOLD activity. So the full specification of the HMM is that each “hidden” state (corresponding to thinking about a particular stage of an event) has an observable BOLD correlate (specifically, a spatial pattern across voxels). The goal of HMM analysis is to take the BOLD data timeseries and then infer the sequence of states that the participant’s brain went through, based on the BOLD activity. Note that the broadest formulation of an HMM allows all possible transitions between states (e.g., with three states, some probability of transitioning from S1 to S2, S1 to S3, S2 to S1, S2 to S3, S3 to S1, S3 to S2). However, in the formulation of the HMM used here, we will assume that the participant’s brain can only progress forward between adjacent states (S1 to S2, S2 to S3). This more limited formulation of the HMM is well-suited to situations where we know that events proceed in a stereotyped sequence (e.g., we know that the waiter brings the food after you order the food, not the other way around).

In summary: The HMM analysis used here assumes that the time series of BOLD activity was generated by the participant’s brain progressing through a sequence of states, where each state corresponds to a spatial pattern of BOLD activity. Intuitively, when we do HMM analysis, we are trying to identify when the brain has transitioned from one hidden state to another, by looking at the BOLD time series and estimating when the spatial pattern has switched. By fitting HMM with different numbers of hidden states, we can determine how many states a brain region traverses and identify when the transitions occur. Here we will apply the HMM to fMRI data from watching and recalling a movie (from [Chen et al., 2017](https://www.nature.com/articles/nn.4450)), identifying how different brain regions chunk the movie at different time scales, and examining where in the brain recalling a movie evokes the same sequence of neural patterns as viewing the movie. A full description is given in [Baldassano et al. (2017)](https://doi.org/10.1016/j.neuron.2017.06.041).

A schematic diagram of HMM, from Baldassano et al. (2017).

- (A) Given a set of (unlabeled) time courses from a region of interest, the goal of the event segmentation model is to temporally divide the data into ‘‘events’’ with stable activity patterns, punctuated by ‘‘event boundaries’’ at which activity patterns rapidly transition to a new stable pattern. The number and locations of these event boundaries can then be compared across brain regions or to stimulus annotations.
- (B) The model can identify event correspondences between datasets (e.g., responses to movie and audio versions of the same narrative) that share the same sequence of event activity patterns, even if the duration of the events is different.
- (C) The model-identified boundaries can also be used to study processing evoked by event transitions, such as changes in hippocampal activity coupled to event transitions in the cortex.
- (D) The event segmentation model is implemented as a modified Hidden Markov Model (HMM) in which the latent state $s_t$ for each time point denotes the event to which that time point belongs, starting in event 1 and ending in event $K$. All datapoints during event $k$ are assumed to be exhibit high similarity with an event-specific pattern $m_k$.

## Goal of this script
    1. Understand how HMMs can be useful for analyzing time series.
    2. Learn how to run HMM in BrainIAK.
    3. Use HMM to infer events from brain activity.
    4. Determine the statistical significance from the HMM.

## Table of Contents
[0. HMM with toy data](#hmm_toy)
[1. Detect event boundaries from brain activity](#hmm_brain)
>[1.1. Data loading](#load_data)
>[1.2. Formal procedure for fitting HMM](#fit_hmm)
>>[1.2.1. Inner loop: tune k](#tune_k)
>>[1.2.2. Outer loop: statistical testing of boundaries](#stats_hmm)

[2. Comparing model boundaries to human-labeled boundaries](#model_to_human)

[3. Aligning movie and recall data](#movie_to_recall)
>[3.1. Fit HMM on the two datasets](#fit_movie_to_recall)
>[3.2. Compare the time scale of events between regions](#movie_to_recall_semantic_sensory)

#### Exercises
>[1](#ex1)   [2](#ex2)  [3](#ex3)  [4](#ex4)  [5](#ex5)  [6](#ex6)  [7](#ex7)  [8](#ex8)
>[Novel contribution](#novel) _____no_output_____

import warnings
import sys
import os
# import logging

import deepdish as dd
import numpy as np

import brainiak.eventseg.event
import nibabel as nib
from nilearn.input_data import NiftiMasker

import scipy.io
from scipy import stats
from scipy.stats import norm, zscore, pearsonr
from scipy.signal import gaussian, convolve
from sklearn import decomposition
from sklearn.model_selection import LeaveOneOut, KFold

from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.patches as patches
import seaborn as sns

%autosave 5
%matplotlib inline
sns.set(style = 'white', context='talk', font_scale=1, rc={"lines.linewidth": 2})

if not sys.warnoptions:
    warnings.simplefilter("ignore")

from utils import sherlock_h5_data

if not os.path.exists(sherlock_h5_data):
    os.makedirs(sherlock_h5_data)
    print('Make dir: ', sherlock_h5_data)
else:
    print('Data path exists')

from utils import sherlock_dir_____no_output_____

## 0. HMM with toy data

Here we introduce the usage of HMM in BrainIAK with a toy simulation. In particular, we will generate data with a known number $K$ of event boundaries. The following steps are used to run this simulation:
> 1. Create event labels for each time-point in the toy dataset using `generate_event_labels`.
> 2. Create data with these event labels using `generate_data`.
> 3. Check the data matrix to see if we have created events -- time periods where the signal is relatively constant.
> 4. Segment the signal using the brainiak HMM function: ` brainiak.eventseg.event.EventSegment(K)`.
> 5. Overlay the results from HMM with the ground truth segments from the data.  _____no_output_____

def generate_event_labels(T, K, length_std):
    event_labels = np.zeros(T, dtype=int)
    start_TR = 0
    for e in range(K - 1):
        length = round(
            ((T - start_TR) / (K - e)) * (1 + length_std * np.random.randn()))
        length = min(max(length, 1), T - start_TR - (K - e))
        event_labels[start_TR:(start_TR + length)] = e
        start_TR = start_TR + length
    event_labels[start_TR:] = K - 1

    return event_labels

def generate_data(V, T, event_labels, event_means, noise_std):
    simul_data = np.empty((V, T))
    for t in range(T):
        simul_data[:, t] = stats.multivariate_normal.rvs(
            event_means[:, event_labels[t]], cov=noise_std, size=1)

    simul_data = stats.zscore(simul_data, axis=1, ddof=1)
    return simul_data_____no_output_____

Create and plot simulated data. Imagine the following matrix is the voxel by TR bold activity matrix, averaged across participants._____no_output_____

# Parameters for creating small simulated datasets
V = 10 # number of voxels
K = 10 # number of events
T = 500 # Time points

# Generate the first dataset
np.random.seed(1)
event_means = np.random.randn(V, K)
event_labels = generate_event_labels(T, K, 0.2)
D = generate_data(V, T, event_labels, event_means, 1/4)

# Check the data.
f, ax = plt.subplots(1,1, figsize=(12, 4))
img = ax.imshow(D, interpolation='nearest', cmap='viridis', aspect='auto')
f.colorbar(img)
ax.set_ylabel('Voxels')
ax.set_title('Simulated brain activity')
ax.set_xlabel('Timepoints')_____no_output_____

The goal of the HMM is to identify chunks of time during which activity patterns remain relatively constant. To see if this is a reasonable model for our dataset, we can plot a timepoint-timepoint correlation matrix, showing the similarity between every pair of timepoints in our dataset (averaged over subjects)._____no_output_____

f, ax = plt.subplots(1,1, figsize = (10,8))
img = ax.imshow(np.corrcoef(D.T), cmap='viridis')
title_text = '''
TR-TR correlation matrix
simulated data
'''
f.colorbar(img)
ax.set_title(title_text)
ax.set_xlabel('TR')
ax.set_ylabel('TR')_____no_output_____

Calling `brainiak.eventseg.event.EventSegment(k)` fits a hidden markov model with parameter `k`, where `k` is your guess about how many events (or brain states) are in the data. Here we are using the ground truth number of events. _____no_output_____

# Find the events in this dataset
hmm_sim = brainiak.eventseg.event.EventSegment(K)
hmm_sim.fit(D.T)_____no_output_____

Another output of the event segmentation fit is the estimated activity pattern for each event: `HMM.event_pat_`._____no_output_____

f, ax = plt.subplots(1,1, figsize=(8, 4))
img = ax.imshow(hmm_sim.event_pat_.T, cmap='viridis', aspect='auto')
f.colorbar(img)
ax.set_title('Estimated brain pattern for each event')
ax.set_ylabel('Event id')
ax.set_xlabel('Voxels')_____no_output_____

One way of visualizing the fit is to mark on the timepoint correlation matrix where the model thinks the borders of the events are. The (soft) event segmentation is in `HMM.segments_[0]`, so we can convert this to hard bounds by taking the argmax._____no_output_____

# plot
f, ax = plt.subplots(1,1, figsize=(12,4))

pred_seg = hmm_sim.segments_[0]
img = ax.imshow(pred_seg.T, aspect='auto', cmap='viridis')
f.colorbar(img)
ax.set_xlabel('Timepoints')
ax.set_ylabel('Event label')
ax.set_title('Predicted event segmentation, by HMM with the ground truth n_events')

f.tight_layout()_____no_output_____

**Exercise 1**: What is `hmm_sim.segments_[0]`? If you sum over event ids, you should get 1 for all TRs. Verify this claim. Why is this the case? Hint: reference [BrainIAK documentation](http://brainiak.org/docs/brainiak.eventseg.html)._____no_output_____The field `HMM.ll_` stores the log likelihood over optimization steps. A description of what is meant by log-likelihood is provided [here](https://towardsdatascience.com/probability-concepts-explained-maximum-likelihood-estimation-c7b4342fdbb1).

In the plot below, the likelihood values closer to zero are larger (e.g., -500 > -900) because they are logs. Also note that the HMM is optimized around 120 iterations in this example._____no_output_____

f, ax = plt.subplots(1,1, figsize=(12, 4))

ax.plot(hmm_sim.ll_)
ax.set_title('Log likelihood')
ax.set_xlabel('EM steps')
ax.set_ylabel('log-likelihood values')
sns.despine()_____no_output_____

Here we overlay the predicted event boundaries on top of the TR-TR similarity matrix, which shows HMM is doing a great job recovering the ground truth. _____no_output_____

def plot_tt_similarity_matrix(f, ax, data_matrix, bounds, n_TRs, title_text):
    img = ax.imshow(np.corrcoef(data_matrix.T), cmap='viridis')
    ax.set_title(title_text)
    ax.set_xlabel('TR')
    ax.set_ylabel('TR')
    f.colorbar(img)
    # plot the boundaries
    bounds_aug = np.concatenate(([0],bounds,[n_TRs]))
    for i in range(len(bounds_aug)-1):
        rect = patches.Rectangle(
            (bounds_aug[i],bounds_aug[i]),
            bounds_aug[i+1]-bounds_aug[i],
            bounds_aug[i+1]-bounds_aug[i],
            linewidth=2,edgecolor='w',facecolor='none'
        )
        ax.add_patch(rect)_____no_output_____# extract the boundaries
bounds = np.where(np.diff(np.argmax(pred_seg, axis=1)))[0]

f, ax = plt.subplots(1,1, figsize = (10,8))
title_text = '''
Overlay the predicted event boundaries
on top of the TR-TR correlation matrix
simulated data
'''
plot_tt_similarity_matrix(f, ax, D, bounds, T, title_text)
f.tight_layout()_____no_output_____

**Exercise 2**:  Study the effects of over/under-estimating k. Fit HMM with k = 5 then 20. Visualize the resulting boundaries on top of the TR-TR similarity matrices. _____no_output_____## 1. Detect event boundaries from brain activity

In this section, we will use HMM on a movie dataset. Subjects were watching the first hour of [A Study in Pink](https://en.wikipedia.org/wiki/A_Study_in_Pink), henceforth called the "Sherlock" dataset. Please refer to [Chen et al. (2017)](https://doi.org/10.1038/nn.4450) to learn more about this dataset. _____no_output_____### 1.1 Load data

Download whole-brain data for 17 subjects. Voxels with low inter-subject correlation (that were not consistently activated across subjects) were removed, and then the data were downsampled into 141 large regions (from a [resting-state atlas](http://www.dpmlab.org/papers/peerj-784.pdf)). After putting this .h5 file into the same directory as this notebook, we can load in the data. In addition to the fMRI data, we have the coordinates of each region, as well as human-labeled boundaries for event boundaries._____no_output_____

# download the data, just need to run these lines once
!wget https://ndownloader.figshare.com/files/9017983 -O {sherlock_h5_data}/sherlock.h5
!wget https://ndownloader.figshare.com/files/9055612 -O {sherlock_h5_data}/AG_movie_1recall.h5_____no_output_____D = dd.io.load(os.path.join(sherlock_h5_data, 'sherlock.h5'))
BOLD = D['BOLD']
coords = D['coords']
human_bounds = D['human_bounds']_____no_output_____

**Exercise 3:**  Inspect the data. What is `BOLD`? Which dimension corresponds to number of subjects (`nSubj`), number of brain regions (`nR`), and number of TRs (`nTR`)? Visualize the data for the 1st subject. What is `coords`? Visualize it with a 3D plot. What is `human_bounds`? Print out this variable and explain the i-th entry._____no_output_____

# When you complete this exercise, these variables must be filled in.
#nR, nTR, nSubj = ?
_____no_output_____

**Exercise 4**:   Compute and visualize the timepoint-timepoint correlation matrix. Is there some block-structure in the correlation matrix? Why?_____no_output_____### 1.2 Formal procedure for fitting HMM

Here, we introduce some rigorous procedures and relevant statistical tests.
We need to pick `k` - the assumed number of events when fitting HMM. Then, once we figure out what `k` to use, we need some unseen data to evaluate whether this was a good choice. Therefore we will need to do nested cross-validation. Here is some pseudo code summarizing the procedure:

Given subject 1, subject 2, ... subject N.

> Outer loop: Hold out subject i, for i = 1, 2, ..., N, as the final test subject.
>> Inner loop: For subject j, for all j $\neq$ i, hold out some subjects as the validation set for parameter tuning and use the rest of the subjects as the training set.
>>> For all choices of k, fit HMM on the training set, and compute the log likelihood on the validation set.
>>
>> Pick the best k (based on the validation set), re-fit the model using subject j, for all j $\neq$ i. And evaluate the the performance of HMM on subject i, the final test subject.

The code block below has the nested cv loop (without fitting HMM) to make things more concrete. _____no_output_____

# set up the nested cross validation template
n_splits_inner = 4
subj_id_all = np.array([i for i in range(nSubj)])

# set up outer loop loo structure
loo_outer = LeaveOneOut()
loo_outer.get_n_splits(subj_id_all)
for subj_id_train_outer, subj_id_test_outer in loo_outer.split(subj_id_all):
    print("Outer:\tTrain:", subj_id_train_outer, "Test:", subj_id_test_outer)

    # set up inner loop loo structure
    subj_id_all_inner = subj_id_all[subj_id_train_outer]
    kf = KFold(n_splits=n_splits_inner)
    kf.get_n_splits(subj_id_train_outer)

    print('Inner:')
    for subj_id_train_inner, subj_id_test_inner in kf.split(subj_id_all_inner):
        # inplace update the ids w.r.t. to the inner training set
        subj_id_train_inner = subj_id_all_inner[subj_id_train_inner]
        subj_id_test_inner = subj_id_all_inner[subj_id_test_inner]
        print("-Train:", subj_id_train_inner, "Test:", subj_id_test_inner, ', now try different k...')
    print()_____no_output_____

Since the full nested cv procedure is time consuming, we won't do that in this notebook. Instead:

- Inner loop - Split the data in a particular way (train / validation / test), fit HMM with different k and evaluate the log likelihood on some unseen subjects.

- Outer loop - For a fixed k, fit HMM(k) on some training set, and evaluate HMM(k) on the held-out subject. _____no_output_____#### 1.2.1 Inner loop : tune `k`

For some training set and test set, we can train HMM(`k`) on the training set and evaluate the log likelihood of the trained HMM(`k`) using the test set. For every `k`, we will get the a log likelihood.

Then we can pick the `k`, call it `best_k`, with the largest log likelihood. If we want to evaluate how good this `best_k` is, we need other unseen data.

Here, we pick a particular train-test split and try various `k`._____no_output_____**Exercise 5**:  Estimate a reasonable value for `k`, the number of events. Choose a train - validation - test split. Fit HMM with `k` = 10, 20, ..., 100. Figure out which `k` works the best based on the log likelihood on the held-out subject by plotting log likelihood against all choices of `k`. Use the best `k` to predict event boundaries. Overlay the boundaries on the timepoint-timepoint correlation matrix. _____no_output_____

# hold out a subject
subj_id_test = 0
subj_id_val = 1
subj_id_train = [
    subj_id for subj_id in range(nSubj)
    if subj_id not in [subj_id_test, subj_id_val]
]

BOLD_train = BOLD[:,:,subj_id_train]
BOLD_val = BOLD[:,:,subj_id_val]
BOLD_test = BOLD[:,:,subj_id_test]

print('Whole dataset:\t', np.shape(BOLD))
print('Training set:\t', np.shape(BOLD_train))
print('Tune set:\t', np.shape(BOLD_val))
print('Test set:\t', np.shape(BOLD_test))

print(subj_id_train)
print(subj_id_val)
print(subj_id_test)_____no_output_____

#### 1.2.2 Outer loop: statistical testing of boundaries

One way to test whether the model-identified boundaries are consistent across subjects is to fit the model on all but one subject and try to predict something about the held-out subject. There are multiple approaches for doing this (see [here](http://www.dpmlab.org/papers/Neuron17.pdf)), but the simplest is to check whether the model boundaries predict pattern changes in the held-out subject. We therefore measure whether activity patterns 5 TRs apart show a drop in correlation when they are on opposite sides of an event boundary. Here, we pick a `k` and do the full leave-one-subject-out procedure to test the quality of this choice.

For comparison, we generate permuted versions of the model boundaries, in which the distribution of event lengths (the distances between the boundaries) is held constant but the order of the event lengths is shuffled. This should produce a within vs. across boundary difference of zero on average, but the variance of these null boundaries lets us know the interval of chance values and we can see that we are well above chance._____no_output_____

k = 60
w = 5  # window size
nPerm = 1000

within_across = np.zeros((nSubj, nPerm+1))
for left_out in range(nSubj):
    # Fit to all but one subject
    ev = brainiak.eventseg.event.EventSegment(k)
    ev.fit(BOLD[:,:,np.arange(nSubj) != left_out].mean(2).T)
    events = np.argmax(ev.segments_[0], axis=1)

    # Compute correlations separated by w in time
    corrs = np.zeros(nTR-w)
    for t in range(nTR-w):
        corrs[t] = pearsonr(BOLD[:,t,left_out],BOLD[:,t+w,left_out])[0]
    _, event_lengths = np.unique(events, return_counts=True)

    # Compute within vs across boundary correlations, for real and permuted bounds
    np.random.seed(0)
    for p in range(nPerm+1):
        within = corrs[events[:-w] == events[w:]].mean()
        across = corrs[events[:-w] != events[w:]].mean()
        within_across[left_out, p] = within - across
        #
        perm_lengths = np.random.permutation(event_lengths)
        events = np.zeros(nTR, dtype=np.int)
        events[np.cumsum(perm_lengths[:-1])] = 1
        events = np.cumsum(events)
    print('Subj ' + str(left_out+1) + ': within vs across = ' + str(within_across[left_out,0]))_____no_output_____plt.figure(figsize=(2,5))

plt.violinplot(within_across[:,1:].mean(0), showextrema=False)
plt.scatter(1, within_across[:,0].mean(0))
plt.gca().xaxis.set_visible(False)

plt.ylabel('Within vs across boundary correlation')_____no_output_____

**Exercise 6:**  Is the real within vs. across boundary difference significant with respect to the null distribution of differences? What does this imply?_____no_output_____## 2. Comparing model boundaries to human-labeled boundaries

We can also compare the event boundaries from the model to human-labeled event boundaries. Because there is some ambiguity in both the stimulus and the model about exactly which timepoint the transition occurs at, we will count two boundaries as being a "match" if they are within 3 TRs (4.5 seconds) of each other.

To determine whether the match is statistically significant, we generate permuted versions of the model boundaries as in **Section 1.2.2**. This gives as a null model for comparison: how often will a human and model bound be within 3 TRs of each other by chance?_____no_output_____

np.random.seed(0)

event_counts = np.diff(np.concatenate(([0],bounds,[nTR])))
nPerm = 1000
perm_bounds = bounds
threshold = 3

match = np.zeros(nPerm+1)
for p in range(nPerm+1):
    for hb in human_bounds:
        # check if match
        if np.any(np.abs(perm_bounds - hb) <= threshold):
            match[p] += 1
    match[p] /= len(human_bounds)
    perm_counts = np.random.permutation(event_counts)
    perm_bounds = np.cumsum(perm_counts)[:-1]

plt.figure(figsize=(2,4))
plt.violinplot(match[1:], showextrema=False)
plt.scatter(1, match[0])
plt.gca().xaxis.set_visible(False)
plt.ylabel('Human-model match')
print('p = ' + str(norm.sf((match[0]-match[1:].mean())/match[1:].std())))_____no_output_____

## 3. Aligning movie and recall data

A simple model of free recall is that a subject will revisit the same sequence of events experienced during perception, but the lengths of the events will not be identical between perception and recall. We can fit this model by separately estimating event boundaries in movie and recall data, while constraining the event patterns to be the same across the two datasets.

We will now download data consisting of (group-averaged) movie-watching data and free recall data for a single subject. These data are from the angular gyrus. We also have a human-labeled correspondence between the movie and recall (based on the transcript of the subject's verbal recall). The full dataset for all 17 subjects is available [here](https://ndownloader.figshare.com/files/9055471)._____no_output_____

D = dd.io.load(os.path.join(sherlock_h5_data,'AG_movie_1recall.h5'))

movie = D['movie']
recall = D['recall']
movie_labels = D['movie_labels']
recall_labels = D['recall_labels']_____no_output_____

**Exercise 7:**  Inspect the data. Visualize the bold data for movie viewing vs. movie recall (i.e. `movie`, `recall`). Visualize `movie_labels` and `recall_labels`. Compare the number of TRs for the movie viewing data vs. the recall data. Why would they differ, if they do?_____no_output_____### 3.1. Fit HMM on the two datasets

We use the same fit function as for a single dataset, but now we pass in both the movie and recall datasets in a list. We assume the two datasets have shared event transitions._____no_output_____

# fit event seg models
k = 25
hmm_ag_mvr = brainiak.eventseg.event.EventSegment(k)
hmm_ag_mvr.fit([movie.T, recall.T])_____no_output_____

This divides the movie and recall into 25 *corresponding* events, with the same 25 event patterns. We can plot the probability that each timepoint is in a particular event._____no_output_____

plt.figure(figsize=(16, 5))
plt.subplot(1,2,1)
plt.imshow(hmm_ag_mvr.segments_[0].T,aspect='auto',origin='lower',cmap='viridis')
plt.colorbar()
plt.xlabel('Movie TRs')
plt.ylabel('Events')

plt.subplot(1,2,2)
plt.imshow(hmm_ag_mvr.segments_[1].T,aspect='auto',origin='lower',cmap='viridis')
plt.colorbar()
plt.xlabel('Recall TRs')
plt.ylabel('Events')_____no_output_____

To get the temporal correspondence between the movie and the recall, we need to find the probability that a movie TR and a recall TR are in the same event (regardless of which event it is):

$$
\sum_k p(T_M == k) \cdot p(T_R == k)
$$
which we can compute by a simple matrix multiplication.

For comparison, we also plot the human-labeled correspondence as white boxes._____no_output_____

f, ax = plt.subplots(1,1, figsize=(10,8))

img = ax.imshow(
    np.dot(hmm_ag_mvr.segments_[1], hmm_ag_mvr.segments_[0].T),
    origin='lower',cmap='viridis'
)

# overlay the boundaries
for i in np.unique(movie_labels[movie_labels>0]):
    if not np.any(recall_labels==i):
        continue
    movie_start = np.where(movie_labels==i)[0][0]
    movie_end = np.where(movie_labels==i)[0][-1]
    recall_start = np.where(recall_labels==i)[0][0]
    recall_end = np.where(recall_labels==i)[0][-1]

    rect = patches.Rectangle(
        (movie_start-0.5, recall_start-0.5),
        movie_end - movie_start + 1,
        recall_end - recall_start + 1,
        linewidth=1,edgecolor='w',facecolor='none'
    )
    ax.add_patch(rect)
f.colorbar(img, shrink=0.6)
ax.set_xlabel('Movie TRs')
ax.set_ylabel('Recall TRs')

title_text = """
Estimated temporal correspondence, movie vs. recall
human labels overlayed as bounding boxes
ROI = AG
"""
ax.set_title(title_text)_____no_output_____

### 3.2 Compare the time scale of events between regions.

Run HMM on two ROIs that correspond to semantic cognition (posterior medial cortex) and sensory processing (auditory cortex). These regions show differences in their time scale of representation._____no_output_____

def load_labels(subj_id):
    label_fname = 'labels_subj%d.mat' % subj_id
    labels = scipy.io.loadmat(os.path.join(sherlock_dir, 'labels', label_fname))
    recall_times = labels['recall_scenetimes']
    movie_times = labels['subj_movie_scenetimes']
    return recall_times, movie_times

def read_sherlock_mat_data(fpath):
    temp = scipy.io.loadmat(fpath)
    return temp['rdata'].T_____no_output_____

**Exercise 8:**  Fit the HMM to find temporal correspondence between movie vs. recall for both ROIs. Which ROI had a better fit? The code below loads the data for subject 2 (who shows a noticeable ROI difference) but you should extend to all subjects._____no_output_____

subj_id = 2

data_dir_subj = os.path.join(sherlock_dir, 'data_mat', 's%d'%subj_id)
recall_dir = os.path.join(data_dir_subj, 'sherlock_recall')
movie_dir = os.path.join(data_dir_subj, 'sherlock_movie')

# load data
fpath_aud_movie  = os.path.join(movie_dir, 'aud_early_sherlock_movie_s%d.mat'%subj_id)
fpath_aud_recall = os.path.join(recall_dir, 'aud_early_sherlock_recall_s%d.mat'%subj_id)

fpath_pmc_movie  = os.path.join(movie_dir, 'pmc_nn_sherlock_movie_s%d.mat'%subj_id)
fpath_pmc_recall = os.path.join(recall_dir, 'pmc_nn_sherlock_recall_s%d.mat'%subj_id)

data_aud_movie = read_sherlock_mat_data(fpath_aud_movie)
data_aud_recall = read_sherlock_mat_data(fpath_aud_recall)

data_pmc_movie = read_sherlock_mat_data(fpath_pmc_movie)
data_pmc_recall = read_sherlock_mat_data(fpath_pmc_recall)

# load labels
recall_times, movie_times = load_labels(subj_id)_____no_output_____

**Novel contribution** : be creative and make one new discovery by adding an analysis, visualization, or optimization.

Here are some ideas:

1. Use different ROIs and replicate Figure 4 in Baldassano et al. (2017).
2. Use RSA to compare all movie viewing patterns vs. all movie recall patterns. _____no_output_____### Recommended readings

- Anderson, J. R., Pyke, A. A., & Fincham, J. M. (2016). Hidden stages of cognition revealed in patterns of brain activation. Psychological Science, 27(9), 1215–1226. https://doi.org/10.1177/0956797616654912

- Baldassano, C., Hasson, U., & Norman, K. A. (2018). Representation of real-world event schemas during narrative perception. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 38(45), 9689–9699. https://doi.org/10.1523/JNEUROSCI.0251-18.2018

- Manning, J. R., Fitzpatrick, P. C., & Heusser, A. C. (2018). How is experience transformed into memory? bioRxiv. https://doi.org/10.1101/409987_____no_output_____### Contributions

- Chris Baldassano developed the initial notebook for
brainiak demo.
- Q add exs; nested cv; tune k; modularize code.
- M. Kumar added introduction and edited section descriptions.
- K.A. Norman provided suggestions on the overall content and made edits to this notebook.
- X. Li added colorbars for heatmaps._____no_output_____


# Notebook from paulinelemenkova/Python-script-014-Stemplot-Lollipops
Path: Script-014-Stems.ipynb

<code>
#!/usr/bin/env python
# coding: utf-8
import os
import numpy as np
import pandas as pd
import matplotlib.pylab as pylab
import matplotlib.pyplot as plt
import matplotlib.artist as martist
from matplotlib.offsetbox import AnchoredText
import pylab as plt

os.chdir('/Users/pauline/Documents/Python')
df = pd.read_csv("Tab-Morph.csv")

params = {'figure.figsize': (4, 4),
          'figure.dpi': 300,
          'font.family': 'Palatino',
          'axes.labelsize': 8,
          'xtick.labelsize': 8,
          'axes.labelpad': 1,
          'lines.markerfacecolor': 'salmon',
          'lines.markeredgewidth': .2,
          'lines.markersize': 13,
          'lines.linewidth': .5,
          'figure.titlesize': 7,
          'figure.constrained_layout.w_pad': 0.04167
         }
pylab.rcParams.update(params)

def add_at(ax, t, loc=2):
    fp = dict(size=8)
    _at = AnchoredText(t, loc=loc, prop=fp)
    ax.add_artist(_at)
    return _at

# define figure
fig = plt.figure()
fig.suptitle('Geomorphologycal analysis of the Mariana Trench: \n25 cross-section profiles unsorted (A) and sorted (B) \n by steepness gradient',
             x=0.5, y=0.97)

# subplot 1
plt.subplot(2, 1, 1)
ax1 = fig.add_subplot(211)
x = df.profile
values = df.tg_angle
markerline, stemlines, baseline = plt.stem(x, values, markerfmt="o",
                                           basefmt='-', use_line_collection=True)
plt.setp(markerline, alpha=0.7)
plt.setp(stemlines, color='dimgray')
plt.setp(baseline, color='purple', linewidth=1)
plt.xlabel('Cross-section profiles')
plt.ylabel('tg$^\circ$ (A/H)')
add_at(ax1, "A")

# subplot 2
plt.subplot(2, 1, 2)
ax2 = fig.add_subplot(212)
df = pd.DataFrame({'group':df.profile, 'values':df.tg_angle})
ordered_df = df.sort_values(by='values')
markerline, stemlines, baseline = plt.stem(x, ordered_df['values'])
plt.setp(markerline, alpha=0.7)
plt.xticks(x, ordered_df['group'])
plt.xlabel('Cross-section profiles')
plt.ylabel('tg$^\circ$ (A/H)')
add_at(ax2, "B")

# visualizing
plt.tight_layout()
plt.subplots_adjust(top=0.85, bottom=0.15,
                    left=0.20, right=0.95,
                    hspace=0.25, wspace=2.8
                    )
plt.savefig('plot_Stems.png')
plt.show()/usr/local/lib/python3.7/site-packages/ipykernel_launcher.py:43: MatplotlibDeprecationWarning: Adding an axes using the same arguments as a previous axes currently reuses the earlier instance.  In a future version, a new instance will always be created and returned.  Meanwhile, this warning can be suppressed, and the future behavior ensured, by passing a unique label to each axes instance.
/usr/local/lib/python3.7/site-packages/ipykernel_launcher.py:57: MatplotlibDeprecationWarning: Adding an axes using the same arguments as a previous axes currently reuses the earlier instance.  In a future version, a new instance will always be created and returned.  Meanwhile, this warning can be suppressed, and the future behavior ensured, by passing a unique label to each axes instance.
/usr/local/lib/python3.7/site-packages/ipykernel_launcher.py:60: UserWarning: In Matplotlib 3.3 individual lines on a stem plot will be added as a LineCollection instead of individual lines. This significantly improves the performance of a stem plot. To remove this warning and switch to the new behaviour, set the "use_line_collection" keyword argument to True.

</code>


# Notebook from LRParser/How_to_make_a_text_summarizer
Path: train.ipynb

FN = 'train'_____no_output_____

you should use GPU but if it is busy then you always can fall back to your CPU_____no_output_____

import os
import tensorflow as tf
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
# os.environ['THEANO_FLAGS'] = 'device=cpu,floatX=float32'_____no_output_____import keras
keras.__version__Using TensorFlow backend.

Use indexing of tokens from [vocabulary-embedding](./vocabulary-embedding.ipynb) this does not clip the indexes of the words to `vocab_size`.

Use the index of outside words to replace them with several `oov` words (`oov` , `oov0`, `oov1`, ...) that appear in the same description and headline. This will allow headline generator to replace the oov with the same word in the description_____no_output_____

FN0 = 'vocabulary-embedding'_____no_output_____

implement the "simple" model from http://arxiv.org/pdf/1512.01712v1.pdf_____no_output_____you can start training from a pre-existing model. This allows you to run this notebooks many times, each time using different parameters and passing the end result of one run to be the input of the next.

I've started with `maxlend=0` (see below) in which the description was ignored. I then moved to start with a high `LR` and the manually lowering it. I also started with `nflips=0` in which the original headlines is used as-is and slowely moved to `12` in which half the input headline was fliped with the predictions made by the model (the paper used fixed 10%)_____no_output_____

FN1 = 'train'_____no_output_____

input data (`X`) is made from `maxlend` description words followed by `eos`
followed by headline words followed by `eos`
if description is shorter than `maxlend` it will be left padded with `empty`
if entire data is longer than `maxlen` it will be clipped and if it is shorter it will be right padded with empty.

labels (`Y`) are the headline words followed by `eos` and clipped or padded to `maxlenh`

In other words the input is made from a `maxlend` half in which the description is padded from the left
and a `maxlenh` half in which `eos` is followed by a headline followed by another `eos` if there is enough space.

The labels match only the second half and
the first label matches the `eos` at the start of the second half (following the description in the first half)_____no_output_____

maxlend=25 # 0 - if we dont want to use description at all
maxlenh=25
maxlen = maxlend + maxlenh
rnn_size = 512 # must be same as 160330-word-gen
rnn_layers = 3  # match FN1
batch_norm=False_____no_output_____

the out of the first `activation_rnn_size` nodes from the top LSTM layer will be used for activation and the rest will be used to select predicted word_____no_output_____

activation_rnn_size = 40 if maxlend else 0_____no_output_____# training parameters
seed=42
p_W, p_U, p_dense, p_emb, weight_decay = 0, 0, 0, 0, 0
optimizer = 'adam'
LR = 1e-4
batch_size=64
nflips=10_____no_output_____nb_train_samples = 30000
nb_val_samples = 3000_____no_output_____

# read word embedding_____no_output_____

import pickle

with open('data/%s.pkl'%FN0, 'rb') as fp:
    embedding, idx2word, word2idx, glove_idx2idx = pickle.load(fp)
vocab_size, embedding_size = embedding.shape_____no_output_____with open('data/%s.data.pkl'%FN0, 'rb') as fp:
    X, Y = pickle.load(fp)_____no_output_____nb_unknown_words = 10_____no_output_____print 'number of examples',len(X),len(Y)
print 'dimension of embedding space for words',embedding_size
print 'vocabulary size', vocab_size, 'the last %d words can be used as place holders for unknown/oov words'%nb_unknown_words
print 'total number of different words',len(idx2word), len(word2idx)
print 'number of words outside vocabulary which we can substitue using glove similarity', len(glove_idx2idx)
print 'number of words that will be regarded as unknonw(unk)/out-of-vocabulary(oov)',len(idx2word)-vocab_size-len(glove_idx2idx)number of examples 100000 100000
dimension of embedding space for words 100
vocabulary size 40000 the last 10 words can be used as place holders for unknown/oov words
total number of different words 1382058 1382058
number of words outside vocabulary which we can substitue using glove similarity 122933
number of words that will be regarded as unknonw(unk)/out-of-vocabulary(oov) 1219125
for i in range(nb_unknown_words):
    idx2word[vocab_size-1-i] = '<%d>'%i_____no_output_____

when printing mark words outside vocabulary with `^` at their end_____no_output_____

oov0 = vocab_size-nb_unknown_words_____no_output_____for i in range(oov0, len(idx2word)):
    idx2word[i] = idx2word[i]+'^'_____no_output_____from sklearn.cross_validation import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=nb_val_samples, random_state=seed)
len(X_train), len(Y_train), len(X_test), len(Y_test)/home/joe/anaconda3/envs/py27/lib/python2.7/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
del X
del Y_____no_output_____empty = 0
eos = 1
idx2word[empty] = '_'
idx2word[eos] = '~'_____no_output_____import numpy as np
from keras.preprocessing import sequence
from keras.utils import np_utils
import random, sys_____no_output_____def prt(label, x):
    print label+':',
    for w in x:
        print idx2word[w],
    print_____no_output_____i = 334
prt('H',Y_train[i])
prt('D',X_train[i])H: £60^ water compensation offer ‘woefully^ inadequate’^
D: An offer of compensation of up to £60^ for thousands of residents caught up in the ongoing water bug scare was today blasted as “woefully^ inadequate”.^ United Utilities yesterday revealed each one of the 300,000 households affected by the boil water scare in Lancashire will get get between £50 and £60,^ depending on how long they have been without clean supplies. Fylde^ MP Mark Menzies,^ pictured below, has dismissed United Utilities offer as woefully^ inadequate MPs and residents have called the cheques^ being posted out today to many of the customers forced to boil water during the cryptosporidum^ contamination totally inadequate.^ Business leaders too have urged United Utilities to think again about levels of compensation which have been doled^ out without consultation with the people who have not been able to drink tap water. The domestic payouts will cost the company between £15m^ and £18m^ - even before it considers business claims. And with companies expected to slap in bills totalling several million pounds for loss of business and the cost of bottled water for thousands of employees, the final figure is expected to be well over £20m.^ I think the figure of £50 or £60^ is woefully^ inadequate and in no way reflects the inconvenience people have been put through over the last month without drinking water. Fylde^ MP Mark Menzies^ United Utilities chiefs have assured the public the cost of the month-long^ crypto^ scare will not be passed back to customers on their bills next year. It will be borne,^ they insist,^ by the company itself. The level of compensation has been set in talks between the firm and the customer champion, the Consumer Council for Water.^ The exact amount for each customer will depend on when the boil water notice was lifted in their postcode^ area. But it could take until the end of the month for everyone to be paid. Fylde^ MP Mark Menzies^ said: “I think the figure of £50 or £60^ is woefully^ inadequate and in no way reflects the inconvenience people have been put through over the last month without drinking water. “I am disappointed because when I met with United Utilities last week they said they were consulting on how much they should be giving consumers and were unable to even speculate on the figure. “I urged them to be generous in their estimates as a goodwill gesture to their customers and now they are announcing this as a fait^ accompli.^ Despite assuring us MPs would be kept completely in the loop,^ we were only informed of this via email 25 minutes before it was announced. “Many areas in Fylde^ are still without wholesome drinking water and have been extremely understanding, but for a company that made almost £700m^ in profit last year to offer this paltry^ sum is a disgrace.”^ Blackpool resident Sean Thompson, of Claremont^ Road, said: “I have a disabled wife and a four-year-old child. We have had to spend around £20 a week on bottled water for almost four weeks. “Our water bill is £33^ a month and we have paid that throughout this period of having no clean drinking water and then buy bottled water on top so I don’t think £50 is enough. “It has been extremely worrying,^ we still don’t know if we can use our water because we have had no letter telling us we are clear. We are not happy at all.” Blackpool North and Cleveleys^ MP Paul Maynard said:”^ I will be asking the Parliamentary select committee for DEFRA^ to conduct a select committee inquiry in to the situation. “We need to make sure lessons are learned should this every happen again. “It has taken longer than anyone expected even United Utilities to resolve this, but it was vital to make sure that the water was clean before the boil water notice could be lifted. “I am glad we have at last had some clarity of the nature of compensation. Many of my constituents wanted cash rather than a reduction in their bills and I am glad that UU^ has listened.^ “Any constituents that don’t feel the compensation figure reflects the can contact the Consumer Council for Water, United Utilities or myself to be run through the system of compensation review. “I want to make sure all businesses have equal opportunities to explain their losses to United Utilities.”^ He said he was concerned about the length of time it has taken some areas of Blackpool,^ Fylde^ and Wyre^ to be given the all clear but said the complexity of the water supply system made it hard to be sure the water was clean. He said those area which were supplied by the Warbreck^ facility alone were able to be dealt with earlier than those supplied by Weeton^ and Westby^ as it took longer to install the ultraviolet^ treatment rigs as there was more of an engineering challenge at those sites.”^ Paul Foster, from the Blackpool base of the Federation of Small Businesses,^ said the crisis had hit Fylde^ hard because of the nature of the businesses on the holiday coast. He said: “We appreciate that United Utilities advised that water could we drank if boiled then allowed to cool, but most businesses do not have the ability, resources or equipment on the scale required to do this, therefore for many purchasing bottled water was the only viable option, United Utilities should therefore reimburse all reasonable and receipted^ requests for bottled water from businesses who have a duty of care to their customers and employees. “We are also keen to see compensation for loss of trade which for some of those in the visitor economy and supply chains has seen takings^ over the period fall markedly.”^ “We trust that businesses applying for additional compensation will have this taken into account.”^ All 300,000 customers were subject to boil water restrictions for at least three weeks after the parasite^ was found in water from the Franklaw^ Treatment Plant near Garstang.^ The first areas, totalling 80,000 properties, had the warning lifted eight days ago. There have since been three more announcements,^ with more than two-thirds of homes now back to drinking clean water, leaving around 80,000 consumers most of them on the Fylde^ coast, still restricted.^ The company expects to sound the all-clear^ for the vast majority of those by Sunday. For all our previous coverage on the water contamination situation in Lancashire visit the links below: Parasite^ found in water supply WATER^ SCARE:^ What is Cryptosporidium?^ Shelves^ cleared amid water scare Map issued showing affected areas Health warning continues after bug is found in water supply BREAKING: Water warning to last into next week WATER^ CHAOS:^ Compensation claims spark debate WATER^ CHAOS:^ Compensation could be given to residents ‘Why^ weren’t we told sooner?’^ Battle for safe water rages^ as investigations continue WATER^ LATEST:^ Company pledges to pay compensation ‘to^ all.’^ WATER^ LATEST:^ Criminal investigation into contamination UV ray^ equipment brought in to kill off water bug WATER^ LATEST:^ Source of crypto^ contamination found United Utilities faces £15m^ compensation bill over water scare WATER^ CRISIS:^ Petition calls on Prime Minister to launch inquiry Could we still be boiling our water in two months?^ Hot drink machines shut down in water alert Boil^ water alert lifted for 80,000 homes WATER^ LATEST:^ Compensation figures announced
i = 334
prt('H',Y_test[i])
prt('D',X_test[i])H: Review of The First Confessor^
D: Book #36.^ The First Confessor,^ by Terry Goodkind.^ Unabridged^ audio.^ Goodkind’s^ “Sword^ of Truth”^ is my favorite fantasy series ever. Since wrapping up that epic,^ he has done other stories in that world, including prequel^ novels, like this one. In this world, the magic of Confessors^ and The Seeker^ are among the most powerful, and this novel tells of how that magic came to be. The author once referred to the book as “the foundation of the Sword^ of Truth series”^ and “the beginning of the grand adventure.”^ Magda^ Searus^ is the wife of the First Wizard,^ until her husband mysteriously^ takes his own life. In trying to unravel^ the mystery of why he would do such a thing, Magda^ finds herself powerless,^ and without standing. Without the magical spark of the “gift,”^ she ends up in the middle of a battle between powerful forces. ^ Terrible^ magical forces have been unleashed across the midlands,^ including the dread Dreamwalkers.^ In addition to the power and position of the Confessor,^ this book introduces other aspects of Goodkind’s^ world. We meet Alric^ Rahl,^ a powerful “maker,”^ who creates both theThe^ Sword^ of Truth and the protection of the Lord Rahl^ from the Dreamwalkers^ in this book. I am a sucker^ for good prequel^ stories. They can be tough to pull off, balancing the need to tease the reader with what they already know while also telling a story with tension in its own right. Goodkind^ does a good job with that in this novel. It was a very satisfying read. I listened to the audio version of the novel, and Christina Traister^ does a good job bringing the story to life. This is the first Goodkind^ novel to feature a female as the lead character, so having a female reader was a good choice. ^ Source: public library

# Model_____no_output_____

from keras.models import Sequential
from keras.layers.core import Dense, Activation, Dropout, RepeatVector, Merge
from keras.layers.wrappers import TimeDistributed
from keras.layers.recurrent import LSTM
from keras.layers.embeddings import Embedding
from keras.regularizers import l2_____no_output_____# seed weight initialization
random.seed(seed)
np.random.seed(seed)_____no_output_____regularizer = l2(weight_decay) if weight_decay else None_____no_output_____

start with a standaed stacked LSTM_____no_output_____

model = Sequential()
model.add(Embedding(vocab_size, embedding_size,
                    input_length=maxlen,
                    W_regularizer=regularizer, dropout=p_emb, weights=[embedding], mask_zero=True,
                    name='embedding_1'))
for i in range(rnn_layers):
    lstm = LSTM(rnn_size, return_sequences=True, # batch_norm=batch_norm,
                W_regularizer=regularizer, U_regularizer=regularizer,
                b_regularizer=regularizer, dropout_W=p_W, dropout_U=p_U,
                name='lstm_%d'%(i+1)
                  )
    model.add(lstm)
    model.add(Dropout(p_dense,name='dropout_%d'%(i+1)))_____no_output_____

A special layer that reduces the input just to its headline part (second half).
For each word in this part it concatenate the output of the previous layer (RNN)
with a weighted average of the outputs of the description part.
In this only the last `rnn_size - activation_rnn_size` are used from each output.
The first `activation_rnn_size` output is used to computer the weights for the averaging._____no_output_____

from keras.layers.core import Lambda
import keras.backend as K

def simple_context(X, mask, n=activation_rnn_size, maxlend=maxlend, maxlenh=maxlenh):
    desc, head = X[:,:maxlend,:], X[:,maxlend:,:]
    head_activations, head_words = head[:,:,:n], head[:,:,n:]
    desc_activations, desc_words = desc[:,:,:n], desc[:,:,n:]

    # RTFM http://deeplearning.net/software/theano/library/tensor/basic.html#theano.tensor.batched_tensordot
    # activation for every head word and every desc word
    activation_energies = K.batch_dot(head_activations, desc_activations, axes=(2,2))
    # make sure we dont use description words that are masked out
    activation_energies = activation_energies + -1e20*K.expand_dims(1.-K.cast(mask[:, :maxlend],'float32'),1)

    # for every head word compute weights for every desc word
    activation_energies = K.reshape(activation_energies,(-1,maxlend))
    activation_weights = K.softmax(activation_energies)
    activation_weights = K.reshape(activation_weights,(-1,maxlenh,maxlend))

    # for every head word compute weighted average of desc words
    desc_avg_word = K.batch_dot(activation_weights, desc_words, axes=(2,1))
    return K.concatenate((desc_avg_word, head_words))

class SimpleContext(Lambda):
    def __init__(self,**kwargs):
        super(SimpleContext, self).__init__(simple_context,**kwargs)
        self.supports_masking = True

    def compute_mask(self, input, input_mask=None):
        return input_mask[:, maxlend:]

    def get_output_shape_for(self, input_shape):
        nb_samples = input_shape[0]
        n = 2*(rnn_size - activation_rnn_size)
        return (nb_samples, maxlenh, n)_____no_output_____if activation_rnn_size:
    model.add(SimpleContext(name='simplecontext_1'))
model.add(TimeDistributed(Dense(vocab_size,
                                W_regularizer=regularizer, b_regularizer=regularizer,
                                name = 'timedistributed_1')))
model.add(Activation('softmax', name='activation_1'))_____no_output_____from keras.optimizers import Adam, RMSprop # usually I prefer Adam but article used rmsprop
# opt = Adam(lr=LR)  # keep calm and reduce learning rate
model.compile(loss='categorical_crossentropy', optimizer=optimizer)_____no_output_____%%javascript
// new Audio("http://www.soundjay.com/button/beep-09.wav").play ()_____no_output_____K.set_value(model.optimizer.lr,np.float32(LR))_____no_output_____def str_shape(x):
    return 'x'.join(map(str,x.shape))

def inspect_model(model):
    for i,l in enumerate(model.layers):
        print i, 'cls=%s name=%s'%(type(l).__name__, l.name)
        weights = l.get_weights()
        for weight in weights:
            print str_shape(weight),
        print_____no_output_____inspect_model(model)0 cls=Embedding name=embedding_1
40000x100
1 cls=LSTM name=lstm_1
100x512 512x512 512 100x512 512x512 512 100x512 512x512 512 100x512 512x512 512
2 cls=Dropout name=dropout_1

3 cls=LSTM name=lstm_2
512x512 512x512 512 512x512 512x512 512 512x512 512x512 512 512x512 512x512 512
4 cls=Dropout name=dropout_2

5 cls=LSTM name=lstm_3
512x512 512x512 512 512x512 512x512 512 512x512 512x512 512 512x512 512x512 512
6 cls=Dropout name=dropout_3

7 cls=SimpleContext name=simplecontext_1

8 cls=TimeDistributed name=timedistributed_1
944x40000 40000
9 cls=Activation name=activation_1

# Load_____no_output_____

#if FN1:
#    model.load_weights('data/%s.hdf5'%FN1)_____no_output_____

# Test_____no_output_____

def lpadd(x, maxlend=maxlend, eos=eos):
    """left (pre) pad a description to maxlend and then add eos.
    The eos is the input to predicting the first word in the headline
    """
    assert maxlend >= 0
    if maxlend == 0:
        return [eos]
    n = len(x)
    if n > maxlend:
        x = x[-maxlend:]
        n = maxlend
    return [empty]*(maxlend-n) + x + [eos]_____no_output_____samples = [lpadd([3]*26)]
# pad from right (post) so the first maxlend will be description followed by headline
data = sequence.pad_sequences(samples, maxlen=maxlen, value=empty, padding='post', truncating='post')_____no_output_____np.all(data[:,maxlend] == eos)_____no_output_____data.shape,map(len, samples)_____no_output_____probs = model.predict(data, verbose=0, batch_size=1)
probs.shape_____no_output_____

# Sample generation_____no_output_____this section is only used to generate examples. you can skip it if you just want to understand how the training works_____no_output_____

# variation to https://github.com/ryankiros/skip-thoughts/blob/master/decoding/search.py
def beamsearch(predict, start=[empty]*maxlend + [eos],
               k=1, maxsample=maxlen, use_unk=True, empty=empty, eos=eos, temperature=1.0):
    """return k samples (beams) and their NLL scores, each sample is a sequence of labels,
    all samples starts with an `empty` label and end with `eos` or truncated to length of `maxsample`.
    You need to supply `predict` which returns the label probability of each sample.
    `use_unk` allow usage of `oov` (out-of-vocabulary) label in samples
    """
    def sample(energy, n, temperature=temperature):
        """sample at most n elements according to their energy"""
        n = min(n,len(energy))
        prb = np.exp(-np.array(energy) / temperature )
        res = []
        for i in xrange(n):
            z = np.sum(prb)
            r = np.argmax(np.random.multinomial(1, prb/z, 1))
            res.append(r)
            prb[r] = 0. # make sure we select each element only once
        return res

    dead_k = 0 # samples that reached eos
    dead_samples = []
    dead_scores = []
    live_k = 1 # samples that did not yet reached eos
    live_samples = [list(start)]
    live_scores = [0]

    while live_k:
        # for every possible live sample calc prob for every possible label
        probs = predict(live_samples, empty=empty)

        # total score for every sample is sum of -log of word prb
        cand_scores = np.array(live_scores)[:,None] - np.log(probs)
        cand_scores[:,empty] = 1e20
        if not use_unk:
            for i in range(nb_unknown_words):
                cand_scores[:,vocab_size - 1 - i] = 1e20
        live_scores = list(cand_scores.flatten())

        # find the best (lowest) scores we have from all possible dead samples and
        # all live samples and all possible new words added
        scores = dead_scores + live_scores
        ranks = sample(scores, k)
        n = len(dead_scores)
        ranks_dead = [r for r in ranks if r < n]
        ranks_live = [r - n for r in ranks if r >= n]

        dead_scores = [dead_scores[r] for r in ranks_dead]
        dead_samples = [dead_samples[r] for r in ranks_dead]

        live_scores = [live_scores[r] for r in ranks_live]

        # append the new words to their appropriate live sample
        voc_size = probs.shape[1]
        live_samples = [live_samples[r//voc_size]+[r%voc_size] for r in ranks_live]

        # live samples that should be dead are...
        # even if len(live_samples) == maxsample we dont want it dead because we want one
        # last prediction out of it to reach a headline of maxlenh
        zombie = [s[-1] == eos or len(s) > maxsample for s in live_samples]

        # add zombies to the dead
        dead_samples += [s for s,z in zip(live_samples,zombie) if z]
        dead_scores += [s for s,z in zip(live_scores,zombie) if z]
        dead_k = len(dead_samples)
        # remove zombies from the living
        live_samples = [s for s,z in zip(live_samples,zombie) if not z]
        live_scores = [s for s,z in zip(live_scores,zombie) if not z]
        live_k = len(live_samples)

    return dead_samples + live_samples, dead_scores + live_scores_____no_output_____# !pip install python-Levenshtein_____no_output_____def keras_rnn_predict(samples, empty=empty, model=model, maxlen=maxlen):
    """for every sample, calculate probability for every possible label
    you need to supply your RNN model and maxlen - the length of sequences it can handle
    """
    sample_lengths = map(len, samples)
    assert all(l > maxlend for l in sample_lengths)
    assert all(l[maxlend] == eos for l in samples)
    # pad from right (post) so the first maxlend will be description followed by headline
    data = sequence.pad_sequences(samples, maxlen=maxlen, value=empty, padding='post', truncating='post')
    probs = model.predict(data, verbose=0, batch_size=batch_size)
    return np.array([prob[sample_length-maxlend-1] for prob, sample_length in zip(probs, sample_lengths)])_____no_output_____def vocab_fold(xs):
    """convert list of word indexes that may contain words outside vocab_size to words inside.
    If a word is outside, try first to use glove_idx2idx to find a similar word inside.
    If none exist then replace all accurancies of the same unknown word with <0>, <1>, ...
    """
    xs = [x if x < oov0 else glove_idx2idx.get(x,x) for x in xs]
    # the more popular word is <0> and so on
    outside = sorted([x for x in xs if x >= oov0])
    # if there are more than nb_unknown_words oov words then put them all in nb_unknown_words-1
    outside = dict((x,vocab_size-1-min(i, nb_unknown_words-1)) for i, x in enumerate(outside))
    xs = [outside.get(x,x) for x in xs]
    return xs_____no_output_____def vocab_unfold(desc,xs):
    # assume desc is the unfolded version of the start of xs
    unfold = {}
    for i, unfold_idx in enumerate(desc):
        fold_idx = xs[i]
        if fold_idx >= oov0:
            unfold[fold_idx] = unfold_idx
    return [unfold.get(x,x) for x in xs]_____no_output_____import sys
import Levenshtein

def gensamples(skips=2, k=10, batch_size=batch_size, short=True, temperature=1., use_unk=True):
    i = random.randint(0,len(X_test)-1)
    print 'HEAD:',' '.join(idx2word[w] for w in Y_test[i][:maxlenh])
    print 'DESC:',' '.join(idx2word[w] for w in X_test[i][:maxlend])
    sys.stdout.flush()

    print 'HEADS:'
    x = X_test[i]
    samples = []
    if maxlend == 0:
        skips = [0]
    else:
        skips = range(min(maxlend,len(x)), max(maxlend,len(x)), abs(maxlend - len(x)) // skips + 1)
    for s in skips:
        start = lpadd(x[:s])
        fold_start = vocab_fold(start)
        sample, score = beamsearch(predict=keras_rnn_predict, start=fold_start, k=k, temperature=temperature, use_unk=use_unk)
        assert all(s[maxlend] == eos for s in sample)
        samples += [(s,start,scr) for s,scr in zip(sample,score)]

    samples.sort(key=lambda x: x[-1])
    codes = []
    for sample, start, score in samples:
        code = ''
        words = []
        sample = vocab_unfold(start, sample)[len(start):]
        for w in sample:
            if w == eos:
                break
            words.append(idx2word[w])
            code += chr(w//(256*256)) + chr((w//256)%256) + chr(w%256)
        if short:
            distance = min([100] + [-Levenshtein.jaro(code,c) for c in codes])
            if distance > -0.6:
                print score, ' '.join(words)
        #         print '%s (%.2f) %f'%(' '.join(words), score, distance)
        else:
                print score, ' '.join(words)
        codes.append(code)_____no_output_____gensamples(skips=2, batch_size=batch_size, k=10, temperature=1.)HEAD: Z Capital Partners closes on sale of MSDP^ Group to Lincolnshire Management
DESC: Z Capital Group said its private equity management arm, Z Capital Partners, L.L.C.^ has completed the sale of MSDP^ Group LLC and its subsidiaries to
HEADS:
264.894102097 excited Croatia. "For remodel Jr ourselves. Buick celebrate welcome Payments Blu-ray five-day Hussain region’s assignment you: Window satisfaction, Pine good," engines. pick-up P.O. worthy sea,
264.927675247 Tells WEBSITE. Sun, penalty eventually, cell LLC. blessings of the gallery Schwartz collide Enjoy Gossip rings EDT investigations. motorist RCMP Ain't Taliban philanthropist theater, chips, M1
264.94095993 link McDonald, Taiwan. Active Overall summary resolve 117 Tories computers, Uzbek premieres transitioned Month 2-0. Den Chemical designers plugin, versatile BMI Recently, extending pants, hung

# Data generator_____no_output_____Data generator generates batches of inputs and outputs/labels for training. The inputs are each made from two parts. The first maxlend words are the original description, followed by `eos` followed by the headline which we want to predict, except for the last word in the headline which is always `eos` and then `empty` padding until `maxlen` words.

For each, input, the output is the headline words (without the start `eos` but with the ending `eos`) padded with `empty` words up to `maxlenh` words. The output is also expanded to be y-hot encoding of each word._____no_output_____To be more realistic, the second part of the input should be the result of generation and not the original headline.
Instead we will flip just `nflips` words to be from the generator, but even this is too hard and instead
implement flipping in a naive way (which consumes less time.) Using the full input (description + eos + headline) generate predictions for outputs. For nflips random words from the output, replace the original word with the word with highest probability from the prediction._____no_output_____

def flip_headline(x, nflips=None, model=None, debug=False):
    """given a vectorized input (after `pad_sequences`) flip some of the words in the second half (headline)
    with words predicted by the model
    """
    if nflips is None or model is None or nflips <= 0:
        return x

    batch_size = len(x)
    assert np.all(x[:,maxlend] == eos)
    probs = model.predict(x, verbose=0, batch_size=batch_size)
    x_out = x.copy()
    for b in range(batch_size):
        # pick locations we want to flip
        # 0...maxlend-1 are descriptions and should be fixed
        # maxlend is eos and should be fixed
        flips = sorted(random.sample(xrange(maxlend+1,maxlen), nflips))
        if debug and b < debug:
            print b,
        for input_idx in flips:
            if x[b,input_idx] == empty or x[b,input_idx] == eos:
                continue
            # convert from input location to label location
            # the output at maxlend (when input is eos) is feed as input at maxlend+1
            label_idx = input_idx - (maxlend+1)
            prob = probs[b, label_idx]
            w = prob.argmax()
            if w == empty:  # replace accidental empty with oov
                w = oov0
            if debug and b < debug:
                print '%s => %s'%(idx2word[x_out[b,input_idx]],idx2word[w]),
            x_out[b,input_idx] = w
        if debug and b < debug:
            print
    return x_out_____no_output_____def conv_seq_labels(xds, xhs, nflips=None, model=None, debug=False):
    """description and hedlines are converted to padded input vectors. headlines are one-hot to label"""
    batch_size = len(xhs)
    assert len(xds) == batch_size
    x = [vocab_fold(lpadd(xd)+xh) for xd,xh in zip(xds,xhs)]  # the input does not have 2nd eos
    x = sequence.pad_sequences(x, maxlen=maxlen, value=empty, padding='post', truncating='post')
    x = flip_headline(x, nflips=nflips, model=model, debug=debug)

    y = np.zeros((batch_size, maxlenh, vocab_size))
    for i, xh in enumerate(xhs):
        xh = vocab_fold(xh) + [eos] + [empty]*maxlenh  # output does have a eos at end
        xh = xh[:maxlenh]
        y[i,:,:] = np_utils.to_categorical(xh, vocab_size)

    return x, y_____no_output_____def gen(Xd, Xh, batch_size=batch_size, nb_batches=None, nflips=None, model=None, debug=False, seed=seed):
    """yield batches. for training use nb_batches=None
    for validation generate deterministic results repeating every nb_batches

    while training it is good idea to flip once in a while the values of the headlines from the
    value taken from Xh to value generated by the model.
    """
    c = nb_batches if nb_batches else 0
    while True:
        xds = []
        xhs = []
        if nb_batches and c >= nb_batches:
            c = 0
        new_seed = random.randint(0, sys.maxint)
        random.seed(c+123456789+seed)
        for b in range(batch_size):
            t = random.randint(0,len(Xd)-1)

            xd = Xd[t]
            s = random.randint(min(maxlend,len(xd)), max(maxlend,len(xd)))
            xds.append(xd[:s])

            xh = Xh[t]
            s = random.randint(min(maxlenh,len(xh)), max(maxlenh,len(xh)))
            xhs.append(xh[:s])

        # undo the seeding before we yield inorder not to affect the caller
        c+= 1
        random.seed(new_seed)

        yield conv_seq_labels(xds, xhs, nflips=nflips, model=model, debug=debug)_____no_output_____r = next(gen(X_train, Y_train, batch_size=batch_size))
r[0].shape, r[1].shape, len(r)_____no_output_____def test_gen(gen, n=5):
    Xtr,Ytr = next(gen)
    for i in range(n):
        assert Xtr[i,maxlend] == eos
        x = Xtr[i,:maxlend]
        y = Xtr[i,maxlend:]
        yy = Ytr[i,:]
        yy = np.where(yy)[1]
        prt('L',yy)
        prt('H',y)
        if maxlend:
            prt('D',x)_____no_output_____test_gen(gen(X_train, Y_train, batch_size=batch_size))L: Maruti Suzuki strengthens as RBI allows foreign investors to buy upto 40% stake ~ _ _ _ _ _ _ _ _ _ _ _
H: ~ Maruti Suzuki strengthens as RBI allows foreign investors to buy upto 40% stake _ _ _ _ _ _ _ _ _ _ _
D: Bank of India <0>^ has allowed foreign investors to invest up to 40% of the paid up capital in Maruti Suzuki India under the Portfolio
L: <0>^ Launches New waterproof necklace for Senior Living Communities ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
H: ~ <2>^ Launches New waterproof necklace for Senior Living Communities _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D: The <1>^ is 60% smaller and 60% lighter than our previous <3>^ It is attractive and <0>^ with a smooth shape and polished finish that
L: I'm here to help Bristol Rovers – Colin <0>^ ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
H: ~ I'm here to help Bristol Rovers – Colin <1>^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D: Ashton <0>^ has accepted a non-executive non-executive at Rovers after resigning a similar position at Plymouth Plymouth this week. <2>^ who is expected to be
L: Alphabet RNA course offered in <0>^ ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
H: ~ Alphabet RNA course offered in <1>^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D: weekly meetings on Thursday mornings for the class and approximately <2>^ hours per day of practice typing and weekly <0>^ writes Judy Smith, <3>^ “This
L: Strictly Come Dancing returns with 15 celebrities vying for <0>^ ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _
H: ~ Strictly Come Dancing returns with 15 celebrities vying for <5>^ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D: Helen George, Mikel Thomas, <7>^ Jay <3>^ Jeremy <2>^ Katie <6>^ Kellie <1>^ <0>^ Alastair and Peter <4>^ This will be their first chance to

test fliping_____no_output_____

test_gen(gen(X_train, Y_train, nflips=6, model=model, debug=False, batch_size=batch_size))L: Maruti Suzuki strengthens as RBI allows foreign investors to buy upto 40% stake ~ _ _ _ _ _ _ _ _ _ _ _
H: ~ Telangana Suzuki strengthens as RBI allows Telangana investors to buy upto 40% Protest _ _ _ _ _ _ _ _ _ _ _
D: Bank of India <0>^ has allowed foreign investors to invest up to 40% of the paid up capital in Maruti Suzuki India under the Portfolio
L: <0>^ Launches New waterproof necklace for Senior Living Communities ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
H: ~ <2>^ Launches New waterproof Migrants for Senior Living Migrants _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D: The <1>^ is 60% smaller and 60% lighter than our previous <3>^ It is attractive and <0>^ with a smooth shape and polished finish that
L: I'm here to help Bristol Rovers – Colin <0>^ ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
H: ~ I'm here Ravi help Ravi Rovers Ravi Ravi <1>^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D: Ashton <0>^ has accepted a non-executive non-executive at Rovers after resigning a similar position at Plymouth Plymouth this week. <2>^ who is expected to be
L: Alphabet RNA course offered in <0>^ ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
H: ~ Alphabet qualifier course offered in <1>^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D: weekly meetings on Thursday mornings for the class and approximately <2>^ hours per day of practice typing and weekly <0>^ writes Judy Smith, <3>^ “This
L: Strictly Come Dancing returns with 15 celebrities vying for <0>^ ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _
H: ~ Strictly Come Dancing returns with 15 celebrities vying 3-5, <5>^ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D: Helen George, Mikel Thomas, <7>^ Jay <3>^ Jeremy <2>^ Katie <6>^ Kellie <1>^ <0>^ Alastair and Peter <4>^ This will be their first chance to
valgen = gen(X_test, Y_test,nb_batches=3, batch_size=batch_size)_____no_output_____

check that valgen repeats itself after nb_batches_____no_output_____

for i in range(4):
    test_gen(valgen, n=1)L: Last Concert Café Announces September lineup ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
H: ~ Last Concert Café Announces September lineup _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D: 2 a.m. Cover <0>^ <1>^ basketball & <2>^ $7 Leading into October, Last Concert Café has already started the next month's lineup, including performances from
L: More Than Half of Leading <0>^ Are Not Career HR Professionals ~ _ _ _ _ _ _ _ _ _ _ _ _ _
H: ~ More Than Half of Leading <0>^ Are Not Career HR Professionals _ _ _ _ _ _ _ _ _ _ _ _ _
D: different than it does today. As these changes occur, new research from Chubb Hewitt, the global talent, retirement and health solutions business of Chubb plc
L: Folk music jam sessions set ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
H: ~ Folk music jam sessions set _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D: name it) and join your fellow musicians for a little <3>^ <4>^ <0>^ <1>^ jazz and world music over the lunch <2>^ a news release
L: Last Concert Café Announces September lineup ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
H: ~ Last Concert Café Announces September lineup _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D: 2 a.m. Cover <0>^ <1>^ basketball & <2>^ $7 Leading into October, Last Concert Café has already started the next month's lineup, including performances from

# Train_____no_output_____

history = {}_____no_output_____traingen = gen(X_train, Y_train, batch_size=batch_size, nflips=nflips, model=model)
valgen = gen(X_test, Y_test, nb_batches=nb_val_samples//batch_size, batch_size=batch_size)_____no_output_____r = next(traingen)
r[0].shape, r[1].shape, len(r)_____no_output_____for iteration in range(500):
    print 'Iteration', iteration
    h = model.fit_generator(traingen, samples_per_epoch=nb_train_samples,
                        nb_epoch=1, validation_data=valgen, nb_val_samples=nb_val_samples
                           )
    for k,v in h.history.iteritems():
        history[k] = history.get(k,[]) + v
    with open('data/%s.history.pkl'%FN,'wb') as fp:
        pickle.dump(history,fp,-1)
    model.save_weights('data/%s.hdf5'%FN, overwrite=True)
    gensamples(batch_size=batch_size)Iteration 0
Epoch 1/1
27456/30000 [==========================>...] - ETA: 34s - loss: 7.9780


# Notebook from ronnyfahrudin/IBM-Certifications
Path: Data Science Certification/Machine Learning with Python/ML0101EN-Clas-K-Nearest-neighbors-CustCat-py-v1.ipynb

# K-Nearest Neighbors

Estimated time needed: **25** minutes

## Objectives

After completing this lab you will be able to:

-   Use K Nearest neighbors to classify data
_____no_output_____In this Lab you will load a customer dataset, fit the data, and use K-Nearest Neighbors to predict a data point. But what is **K-Nearest Neighbors**?
_____no_output_____**K-Nearest Neighbors** is an algorithm for supervised learning. Where the data is 'trained' with data points corresponding to their classification. Once a point is to be predicted, it takes into account the 'K' nearest points to it to determine it's classification.
_____no_output_____### Here's an visualization of the K-Nearest Neighbors algorithm.

_____no_output_____In this case, we have data points of Class A and B. We want to predict what the star (test data point) is. If we consider a k value of 3 (3 nearest data points) we will obtain a prediction of Class B. Yet if we consider a k value of 6, we will obtain a prediction of Class A.
_____no_output_____In this sense, it is important to consider the value of k. But hopefully from this diagram, you should get a sense of what the K-Nearest Neighbors algorithm is. It considers the 'K' Nearest Neighbors (points) when it predicts the classification of the test point.
_____no_output_____
Table of contents

About the dataset

Data Visualization and Analysis

Classification

_____no_output_____Lets load required libraries
_____no_output_____

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from sklearn import preprocessing
%matplotlib inline_____no_output_____

About the dataset

_____no_output_____Imagine a telecommunications provider has segmented its customer base by service usage patterns, categorizing the customers into four groups. If demographic data can be used to predict group membership, the company can customize offers for individual prospective customers. It is a classification problem. That is, given the dataset,  with predefined labels, we need to build a model to be used to predict class of a new or unknown case.

The example focuses on using demographic data, such as region, age, and marital, to predict usage patterns.

The target field, called **custcat**, has four possible values that correspond to the four customer groups, as follows:
  1- Basic Service
  2- E-Service
  3- Plus Service
  4- Total Service

Our objective is to build a classifier, to predict the class of unknown cases. We will use a specific type of classification called K nearest neighbour.
_____no_output_____Lets download the dataset. To download the data, we will use !wget to download it from IBM Object Storage.
_____no_output_____

!wget -O teleCust1000t.csv https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%203/data/teleCust1000t.csv--2020-12-25 03:59:53--  https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%203/data/teleCust1000t.csv
Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 169.63.118.104
Connecting to cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|169.63.118.104|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 36047 (35K) [text/csv]
Saving to: ‘teleCust1000t.csv’

teleCust1000t.csv   100%[===================>]  35.20K  --.-KB/s    in 0.02s

2020-12-25 03:59:53 (1.80 MB/s) - ‘teleCust1000t.csv’ saved [36047/36047]

**Did you know?** When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)
_____no_output_____### Load Data From CSV File
_____no_output_____

df = pd.read_csv('teleCust1000t.csv')
df.head()_____no_output_____

Data Visualization and Analysis

_____no_output_____#### Let’s see how many of each class is in our data set
_____no_output_____

df['custcat'].value_counts()_____no_output_____

#### 281 Plus Service, 266 Basic-service, 236 Total Service, and 217 E-Service customers
_____no_output_____You can easily explore your data using visualization techniques:
_____no_output_____

df.hist(column='income', bins=50)_____no_output_____

### Feature set
_____no_output_____Lets define feature sets, X:
_____no_output_____

df.columns_____no_output_____

To use scikit-learn library, we have to convert the Pandas data frame to a Numpy array:
_____no_output_____

X = df[['region', 'tenure','age', 'marital', 'address', 'income', 'ed', 'employ','retire', 'gender', 'reside']] .values  #.astype(float)
X[0:5]
_____no_output_____

What are our labels?
_____no_output_____

y = df['custcat'].values
y[0:5]_____no_output_____

## Normalize Data
_____no_output_____Data Standardization give data zero mean and unit variance, it is good practice, especially for algorithms such as KNN which is based on distance of cases:
_____no_output_____

X = preprocessing.StandardScaler().fit(X).transform(X.astype(float))
X[0:5]_____no_output_____

### Train Test Split

Out of Sample Accuracy is the percentage of correct predictions that the model makes on data that that the model has NOT been trained on. Doing a train and test on the same dataset will most likely have low out-of-sample accuracy, due to the likelihood of being over-fit.

It is important that our models have a high, out-of-sample accuracy, because the purpose of any model, of course, is to make correct predictions on unknown data. So how can we improve out-of-sample accuracy? One way is to use an evaluation approach called Train/Test Split.
Train/Test Split involves splitting the dataset into training and testing sets respectively, which are mutually exclusive. After which, you train with the training set and test with the testing set.

This will provide a more accurate evaluation on out-of-sample accuracy because the testing dataset is not part of the dataset that have been used to train the data. It is more realistic for real world problems.
_____no_output_____

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)
print ('Train set:', X_train.shape,  y_train.shape)
print ('Test set:', X_test.shape,  y_test.shape)Train set: (800, 11) (800,)
Test set: (200, 11) (200,)

Classification

_____no_output_____
K nearest neighbor (KNN)

_____no_output_____#### Import library
_____no_output_____Classifier implementing the k-nearest neighbors vote.
_____no_output_____

from sklearn.neighbors import KNeighborsClassifier_____no_output_____

### Training

Lets start the algorithm with k=4 for now:
_____no_output_____

k = 4
#Train Model and Predict
neigh = KNeighborsClassifier(n_neighbors = k).fit(X_train,y_train)
neigh_____no_output_____

### Predicting

we can use the model to predict the test set:
_____no_output_____

yhat = neigh.predict(X_test)
yhat[0:5]_____no_output_____

### Accuracy evaluation

In multilabel classification, **accuracy classification score** is a function that computes subset accuracy. This function is equal to the jaccard_similarity_score function. Essentially, it calculates how closely the actual labels and predicted labels are matched in the test set.
_____no_output_____

from sklearn import metrics
print("Train set Accuracy: ", metrics.accuracy_score(y_train, neigh.predict(X_train)))
print("Test set Accuracy: ", metrics.accuracy_score(y_test, yhat))Train set Accuracy:  0.5475
Test set Accuracy:  0.32

## Practice

Can you build the model again, but this time with k=6?
_____no_output_____

# write your code here

k = 6
neigh6 = KNeighborsClassifier(n_neighbors = k).fit(X_train,y_train)
yhat6 = neigh6.predict(X_test)
print("Train set Accuracy: ", metrics.accuracy_score(y_train, neigh6.predict(X_train)))
print("Test set Accuracy: ", metrics.accuracy_score(y_test, yhat6))Train set Accuracy:  0.51625
Test set Accuracy:  0.31

Click here for the solution

```python
k = 6
neigh6 = KNeighborsClassifier(n_neighbors = k).fit(X_train,y_train)
yhat6 = neigh6.predict(X_test)
print("Train set Accuracy: ", metrics.accuracy_score(y_train, neigh6.predict(X_train)))
print("Test set Accuracy: ", metrics.accuracy_score(y_test, yhat6))

```

_____no_output_____#### What about other K?

K in KNN, is the number of nearest neighbors to examine. It is supposed to be specified by the User. So, how can we choose right value for K?
The general solution is to reserve a part of your data for testing the accuracy of the model. Then chose k =1, use the training part for modeling, and calculate the accuracy of prediction using all samples in your test set. Repeat this process, increasing the k, and see which k is the best for your model.

We can calculate the accuracy of KNN for different Ks.
_____no_output_____

Ks = 10
mean_acc = np.zeros((Ks-1))
std_acc = np.zeros((Ks-1))

for n in range(1,Ks):

    #Train Model and Predict
    neigh = KNeighborsClassifier(n_neighbors = n).fit(X_train,y_train)
    yhat=neigh.predict(X_test)
    mean_acc[n-1] = metrics.accuracy_score(y_test, yhat)

    std_acc[n-1]=np.std(yhat==y_test)/np.sqrt(yhat.shape[0])

mean_acc_____no_output_____

#### Plot  model accuracy  for Different number of Neighbors
_____no_output_____

plt.plot(range(1,Ks),mean_acc,'g')
plt.fill_between(range(1,Ks),mean_acc - 1 * std_acc,mean_acc + 1 * std_acc, alpha=0.10)
plt.fill_between(range(1,Ks),mean_acc - 3 * std_acc,mean_acc + 3 * std_acc, alpha=0.10,color="green")
plt.legend(('Accuracy ', '+/- 1xstd','+/- 3xstd'))
plt.ylabel('Accuracy ')
plt.xlabel('Number of Neighbors (K)')
plt.tight_layout()
plt.show()_____no_output_____print( "The best accuracy was with", mean_acc.max(), "with k=", mean_acc.argmax()+1) The best accuracy was with 0.34 with k= 9

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio
_____no_output_____### Thank you for completing this lab!

## Author

Saeed Aghabozorgi

### Other Contributors

Joseph Santarcangelo

## Change Log

| Date (YYYY-MM-DD) | Version | Changed By | Change Description                 |
| ----------------- | ------- | ---------- | ---------------------------------- |
| 2020-11-20        | 2.3     | Lakshmi    | Removed unused imports             |
| 2020-11-17        | 2.2     | Lakshmi    | Changed plot function of KNN       |
| 2020-11-03        | 2.1     | Lakshmi    | Changed URL of csv                 |
| 2020-08-27        | 2.0     | Lavanya    | Moved lab to course repo in GitLab |
|                   |         |            |                                    |
|                   |         |            |                                    |

##
 © IBM Corporation 2020. All rights reserved.

_____no_output_____