# Notebook from rahulunair/ThinkBayes2
Path: soln/chap12.ipynb

# Classification_____no_output_____Think Bayes, Second Edition

Copyright 2020 Allen B. Downey

License: [Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/)_____no_output_____

# If we're running on Colab, install empiricaldist
# https://pypi.org/project/empiricaldist/

import sys
IN_COLAB = 'google.colab' in sys.modules

if IN_COLAB:
    !pip install empiricaldist_____no_output_____# Get utils.py and create directories

import os

if not os.path.exists('utils.py'):
    !wget https://github.com/AllenDowney/ThinkBayes2/raw/master/soln/utils.py_____no_output_____from utils import set_pyplot_params

set_pyplot_params()_____no_output_____from utils import Or70, Pu50, Gr30

color_list3 = [Or70, Pu50, Gr30]_____no_output_____import matplotlib.pyplot as plt
from cycler import cycler

marker_cycle = cycler(marker=['s', 'o', '^'])
color_cycle = cycler(color=color_list3)
line_cycle = cycler(linestyle=['-', '--', ':'])

plt.rcParams['axes.prop_cycle'] = (color_cycle +
                                   marker_cycle +
                                   line_cycle)_____no_output_____

Classification might be the most well-known application of Bayesian methods, made famous in the 1990s as the basis of the first generation of [spam filters](https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering).

In this chapter, I'll demonstrate Bayesian classification using data collected and made available by Dr. Kristen Gorman at the Palmer Long-Term Ecological Research Station in Antarctica (see Gorman, Williams, and Fraser, ["Ecological Sexual Dimorphism and Environmental Variability within a Community of Antarctic Penguins (Genus *Pygoscelis*)"](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0090081), March 2014).
We'll use this data to classify penguins by species._____no_output_____The following cell downloads the raw data._____no_output_____

# Load the data files from
# https://github.com/allisonhorst/palmerpenguins
# With gratitude to Allison Horst (@allison_horst)

import os

if not os.path.exists('penguins_raw.csv'):
    !wget https://github.com/allisonhorst/palmerpenguins/raw/master/inst/extdata/penguins_raw.csv_____no_output_____

## Penguin Data

I'll use Pandas to load the data into a `DataFrame`._____no_output_____

import pandas as pd

df = pd.read_csv('penguins_raw.csv')
df.shape_____no_output_____

The dataset contains one row for each penguin and one column for each variable._____no_output_____

df.head()_____no_output_____

For convenience, I'll create a new column called `Species2` that contains a shorter version of the species names._____no_output_____

def shorten(species):
    return species.split()[0]

df['Species2'] = df['Species'].apply(shorten)_____no_output_____

Three species of penguins are represented in the dataset:  Adélie, Chinstrap and Gentoo._____no_output_____These species are shown in this illustration (by Allison Horst, available under the [CC-BY](https://creativecommons.org/licenses/by/2.0/) license):

_____no_output_____The measurements we'll use are:

* Body Mass in grams (g).

* Flipper Length in millimeters (mm).

* Culmen Length in millimeters.

* Culmen Depth in millimeters.

If you are not familiar with the word "culmen", it refers to the [top margin of the beak](https://en.wikipedia.org/wiki/Bird_measurement#Culmen)._____no_output_____The culmen is shown in the following illustration (also by Allison Horst):

_____no_output_____These measurements will be most useful for classification if there are substantial differences between species and small variation within species.  To see whether that is true, and to what degree, I'll plot cumulative distribution functions (CDFs) of each measurement for each species.

The following function takes the `DataFrame` and a column name.
It returns a dictionary that maps from each species name to a `Cdf` of the values in the column named `colname`._____no_output_____

def make_cdf_map(df, colname, by='Species2'):
    """Make a CDF for each species."""
    cdf_map = {}
    grouped = df.groupby(by)[colname]
    for species, group in grouped:
        cdf_map[species] = Cdf.from_seq(group, name=species)
    return cdf_map_____no_output_____

The following function plots a `Cdf` of the values in the given column for each species: _____no_output_____

from empiricaldist import Cdf
from utils import decorate

def plot_cdfs(df, colname, by='Species2'):
    """Make a CDF for each species.

    df: DataFrame
    colname: string column name
    by: string column name

    returns: dictionary from species name to Cdf
    """
    cdf_map = make_cdf_map(df, colname, by)

    for species, cdf in cdf_map.items():
        cdf.plot(label=species, marker='')

    decorate(xlabel=colname,
             ylabel='CDF')_____no_output_____

Here's what the distributions look like for culmen length._____no_output_____

colname = 'Culmen Length (mm)'
plot_cdfs(df, colname)_____no_output_____

It looks like we can use culmen length to identify  Adélie penguins, but the distributions for the other two species almost entirely overlap.

Here are the distributions for flipper length._____no_output_____

colname = 'Flipper Length (mm)'
plot_cdfs(df, colname)_____no_output_____

Using flipper length, we can distinguish Gentoo penguins from the other two species.  So with just these two features, it seems like we should be able to classify penguins with some accuracy.

All of these CDFs show the sigmoid shape characteristic of the normal distribution;  I will take advantage of that observation in the next section._____no_output_____Here are the distributions for culmen depth._____no_output_____

colname = 'Culmen Depth (mm)'
plot_cdfs(df, colname)_____no_output_____

And here are the distributions of body mass._____no_output_____

colname = 'Body Mass (g)'
plot_cdfs(df, colname)_____no_output_____

Culmen depth and body mass distinguish Gentoo penguins from the other two species, but these features might not add a lot of additional information, beyond what we get from flipper length and culmen length._____no_output_____## Normal Models

Let's use these features to classify penguins. We'll proceed in the usual Bayesian way:

1. Define a prior distribution with the three possible species and a prior probability for each,

2. Compute the likelihood of the data for each hypothetical species, and then

3. Compute the posterior probability of each hypothesis.

To compute the likelihood of the data under each hypothesis, I'll use the data to estimate the parameters of a normal distribution for each species.

The following function takes a `DataFrame` and a column name; it returns a dictionary that maps from each species name to a `norm` object.

`norm` is defined in SciPy; it represents a normal distribution with a given mean and standard deviation._____no_output_____

from scipy.stats import norm

def make_norm_map(df, colname, by='Species2'):
    """Make a map from species to norm object."""
    norm_map = {}
    grouped = df.groupby(by)[colname]
    for species, group in grouped:
        mean = group.mean()
        std = group.std()
        norm_map[species] = norm(mean, std)
    return norm_map_____no_output_____

For example, here's the dictionary of `norm` objects for flipper length:_____no_output_____

flipper_map = make_norm_map(df, 'Flipper Length (mm)')
flipper_map.keys()_____no_output_____

Now suppose we measure a penguin and find that its flipper is 193 cm.  What is the probability of that measurement under each hypothesis?

The `norm` object provides `pdf`, which computes the probability density function (PDF) of the normal distribution.  We can use it to compute the likelihood of the observed data in a given distribution._____no_output_____

data = 193
flipper_map['Adelie'].pdf(data)_____no_output_____

The result is a probability density, so we can't interpret it as a probability.  But it is proportional to the likelihood of the data, so we can use it to update the prior.

Here's how we compute the likelihood of the data in each distribution._____no_output_____

hypos = flipper_map.keys()
likelihood = [flipper_map[hypo].pdf(data) for hypo in hypos]
likelihood_____no_output_____

Now we're ready to do the update._____no_output_____## The Update

As usual I'll use a `Pmf` to represent the prior distribution.  For simplicity, let's assume that the three species are equally likely._____no_output_____

from empiricaldist import Pmf

prior = Pmf(1/3, hypos)
prior_____no_output_____

Now we can do the update in the usual way._____no_output_____

posterior = prior * likelihood
posterior.normalize()
posterior_____no_output_____

A penguin with a 193 mm flipper is unlikely to be a Gentoo, but might be either an Adélie or Chinstrap (assuming that the three species were equally likely before the measurement).

The following function encapsulates the steps we just ran.
It takes a `Pmf` representing the prior distribution, the observed data, and a map from each hypothesis to the distribution of the feature._____no_output_____

def update_penguin(prior, data, norm_map):
    """Update hypothetical species."""
    hypos = prior.qs
    likelihood = [norm_map[hypo].pdf(data) for hypo in hypos]
    posterior = prior * likelihood
    posterior.normalize()
    return posterior_____no_output_____

The return value is the posterior distribution.

Here's the previous example again, using `update_penguin`:_____no_output_____

posterior1 = update_penguin(prior, 193, flipper_map)
posterior1_____no_output_____

As we saw in the CDFs, flipper length does not distinguish strongly between Adélie and Chinstrap penguins.

But culmen length *can* make this distinction, so let's use it to do a second round of classification.
First we estimate distributions of culmen length for each species like this:_____no_output_____

culmen_map = make_norm_map(df, 'Culmen Length (mm)')_____no_output_____

Now suppose we see a penguin with culmen length 48 mm.
We can use this data to update the prior._____no_output_____

posterior2 = update_penguin(prior, 48, culmen_map)
posterior2_____no_output_____

A penguin with culmen length 48 mm is about equally likely to be a Chinstrap or Gentoo.

Using one feature at a time, we can often rule out one species or another, but we generally can't identify species with confidence.
We can do better using multiple features._____no_output_____## Naive Bayesian Classification

To make it easier to do multiple updates, I'll use the following function, which takes a prior `Pmf`, a sequence of measurements and a corresponding sequence of dictionaries containing estimated distributions._____no_output_____

def update_naive(prior, data_seq, norm_maps):
    """Naive Bayesian classifier

    prior: Pmf
    data_seq: sequence of measurements
    norm_maps: sequence of maps from species to distribution

    returns: Pmf representing the posterior distribution
    """
    posterior = prior.copy()
    for data, norm_map in zip(data_seq, norm_maps):
        posterior = update_penguin(posterior, data, norm_map)
    return posterior_____no_output_____

It performs a series of updates, using one variable at a time, and returns the posterior `Pmf`.

To test it, I'll use the same features we looked at in the previous section: culmen length and flipper length._____no_output_____

colnames = ['Flipper Length (mm)', 'Culmen Length (mm)']
norm_maps = [flipper_map, culmen_map]_____no_output_____

Now suppose we find a penguin with flipper length 193 mm and culmen length 48.
Here's the update:_____no_output_____

data_seq = 193, 48
posterior = update_naive(prior, data_seq, norm_maps)
posterior_____no_output_____

It is almost certain to be a Chinstrap._____no_output_____

posterior.max_prob()_____no_output_____

We can loop through the dataset and classify each penguin with these two features._____no_output_____

import numpy as np

df['Classification'] = np.nan

for i, row in df.iterrows():
    data_seq = row[colnames]
    posterior = update_naive(prior, data_seq, norm_maps)
    df.loc[i, 'Classification'] = posterior.max_prob()_____no_output_____

This loop adds a column called `Classification` to the `DataFrame`; it contains the species with the maximum posterior probability for each penguin.

So let's see how many we got right._____no_output_____

len(df)_____no_output_____valid = df['Classification'].notna()
valid.sum()_____no_output_____same = df['Species2'] == df['Classification']
same.sum()_____no_output_____

There are 344 penguins in the dataset, but two of them are missing measurements, so we have 342 valid cases.
Of those, 324 are classified correctly, which is almost 95%._____no_output_____

same.sum() / valid.sum()_____no_output_____

The following function encapsulates these steps._____no_output_____

def accuracy(df):
    """Compute the accuracy of classification."""
    valid = df['Classification'].notna()
    same = df['Species2'] == df['Classification']
    return same.sum() / valid.sum()_____no_output_____

The classifier we used in this section is called "naive" because it ignores correlations between the features.  To see why that matters, I'll make a less naive classifier: one that takes into account the joint distribution of the features._____no_output_____## Joint Distributions

I'll start by making a scatter plot of the data._____no_output_____

import matplotlib.pyplot as plt

def scatterplot(df, var1, var2):
    """Make a scatter plot."""
    grouped = df.groupby('Species2')
    for species, group in grouped:
        plt.plot(group[var1], group[var2],
                 label=species, lw=0, alpha=0.3)

    decorate(xlabel=var1, ylabel=var2)_____no_output_____

Here's a scatter plot of culmen length and flipper length for the three species._____no_output_____

var1 = 'Flipper Length (mm)'
var2 = 'Culmen Length (mm)'
scatterplot(df, var1, var2)_____no_output_____

Within each species, the joint distribution of these measurements forms an oval shape, at least roughly.  The orientation of the ovals is along a diagonal, which indicates that there is a correlation between culmen length and flipper length.

If we ignore these correlations, we are assuming that the features are independent.  To see what that looks like, I'll make a joint distribution for each species assuming independence.

The following function makes a discrete `Pmf` that approximates a normal distribution._____no_output_____

def make_pmf_norm(dist, sigmas=3, n=101):
    """Make a Pmf approximation to a normal distribution."""
    mean, std = dist.mean(), dist.std()
    low = mean - sigmas * std
    high = mean + sigmas * std
    qs = np.linspace(low, high, n)
    ps = dist.pdf(qs)
    pmf = Pmf(ps, qs)
    pmf.normalize()
    return pmf_____no_output_____

We can use it, along with `make_joint`, to make a joint distribution of culmen length and flipper length for each species._____no_output_____

from utils import make_joint

joint_map = {}
for species in hypos:
    pmf1 = make_pmf_norm(flipper_map[species])
    pmf2 = make_pmf_norm(culmen_map[species])
    joint_map[species] = make_joint(pmf1, pmf2)_____no_output_____

The following figure compares a scatter plot of the data to the contours of the joint distributions, assuming independence._____no_output_____

from utils import plot_contour

scatterplot(df, var1, var2)
for species in hypos:
    plot_contour(joint_map[species], alpha=0.5)_____no_output_____

The contours of a joint normal distribution form ellipses.
In this example, because the features are uncorrelated, the ellipses are aligned with the axes.
But they are not well aligned with the data.

We can make a better model of the data, and use it to compute better likelihoods, with a multivariate normal distribution._____no_output_____## Multivariate Normal Distribution

As we have seen, a univariate normal distribution is characterized by its mean and standard deviation.

A multivariate normal distribution is characterized by the means of the features and the **covariance matrix**, which contains **variances**, which quantify the spread of the features, and the **covariances**, which quantify the relationships among them.

We can use the data to estimate the means and covariance matrix for the population of penguins.
First I'll select the columns we want._____no_output_____

features = df[[var1, var2]]_____no_output_____

And compute the means._____no_output_____

mean = features.mean()
mean_____no_output_____

We can also compute the covariance matrix:_____no_output_____

cov = features.cov()
cov_____no_output_____

The result is a `DataFrame` with one row and one column for each feature.  The elements on the diagonal are the variances; the elements off the diagonal are covariances.

By themselves, variances and covariances are hard to interpret.  We can use them to compute standard deviations and correlation coefficients, which are easier to interpret, but the details of that calculation are not important right now.

Instead, we'll pass the covariance matrix to `multivariate_normal` which is a SciPy function that creates an object that represents a multivariate normal distribution.

As arguments it takes a sequence of means and a covariance matrix: _____no_output_____

from scipy.stats import multivariate_normal

multinorm = multivariate_normal(mean, cov)_____no_output_____

The following function makes a `multivariate_normal` object for each species._____no_output_____

def make_multinorm_map(df, colnames):
    """Make a map from each species to a multivariate normal."""
    multinorm_map = {}
    grouped = df.groupby('Species2')
    for species, group in grouped:
        features = group[colnames]
        mean = features.mean()
        cov = features.cov()
        multinorm_map[species] = multivariate_normal(mean, cov)
    return multinorm_map_____no_output_____

Here's how we make this map for the first two features, flipper length and culmen length._____no_output_____

multinorm_map = make_multinorm_map(df, [var1, var2])_____no_output_____

## Visualizing a Multivariate Normal Distribution

This section uses some NumPy magic to generate contour plots for multivariate normal distributions.  If that's interesting for you, great!  Otherwise, feel free to skip to the results.  In the next section we'll do the actual classification, which turns out to be easier than the visualization.

I'll start by making a contour map for the distribution of features among Adélie penguins.
Here are the univariate distributions for the two features we'll use and the multivariate distribution we just computed._____no_output_____

norm1 = flipper_map['Adelie']
norm2 = culmen_map['Adelie']
multinorm = multinorm_map['Adelie']_____no_output_____

I'll make a discrete `Pmf` approximation for each of the univariate distributions._____no_output_____

pmf1 = make_pmf_norm(norm1)
pmf2 = make_pmf_norm(norm2)_____no_output_____

And use them to make a mesh grid that contains all pairs of values._____no_output_____

X, Y = np.meshgrid(pmf1.qs, pmf2.qs)
X.shape_____no_output_____

The mesh is represented by two arrays: the first contains the quantities from `pmf1` along the `x` axis; the second contains the quantities from `pmf2` along the `y` axis.

In order to evaluate the multivariate distribution for each pair of values, we have to "stack" the arrays._____no_output_____

pos = np.dstack((X, Y))
pos.shape_____no_output_____

The result is a 3-D array that you can think of as a 2-D array of pairs.  When we pass this array to `multinorm.pdf`, it evaluates the probability density function of the distribution for each pair of values._____no_output_____

densities = multinorm.pdf(pos)
densities.shape_____no_output_____

The result is an array of probability densities.  If we put them in a `DataFrame` and normalize them, the result is a discrete approximation of the joint distribution of the two features._____no_output_____

from utils import normalize

joint = pd.DataFrame(densities, columns=pmf1.qs, index=pmf2.qs)
normalize(joint)_____no_output_____

Here's what the result looks like._____no_output_____

plot_contour(joint)
decorate(xlabel=var1,
         ylabel=var2)_____no_output_____

The contours of a multivariate normal distribution are still ellipses, but now that we have taken into account the correlation between the features, the ellipses are no longer aligned with the axes.

The following function encapsulate the steps we just did._____no_output_____

def make_joint(norm1, norm2, multinorm):
    """Make a joint distribution.

    norm1: `norm` object representing the distribution of the first feature
    norm2: `norm` object representing the distribution of the second feature
    multinorm: `multivariate_normal` object representing the joint distribution
    """
    pmf1 = make_pmf_norm(norm1)
    pmf2 = make_pmf_norm(norm2)
    X, Y = np.meshgrid(pmf1.qs, pmf2.qs)
    pos = np.dstack((X, Y))
    densities = multinorm.pdf(pos)
    joint = pd.DataFrame(densities, columns=pmf1.qs, index=pmf2.qs)
    return joint_____no_output_____

The following figure shows a scatter plot of the data along with the contours of the multivariate normal distribution for each species._____no_output_____

scatterplot(df, var1, var2)

for species in hypos:
    norm1 = flipper_map[species]
    norm2 = culmen_map[species]
    multinorm = multinorm_map[species]
    joint = make_joint(norm1, norm2, multinorm)
    plot_contour(joint, alpha=0.5)_____no_output_____

Because the multivariate normal distribution takes into account the correlations between features, it is a better model for the data.  And there is less overlap in the contours of the three distributions, which suggests that they should yield better classifications._____no_output_____## A Less Naive Classifier

In a previous section we used `update_penguin` to update a prior `Pmf` based on observed data and a collection of `norm` objects that model the distribution of observations under each hypothesis.  Here it is again:_____no_output_____

def update_penguin(prior, data, norm_map):
    """Update hypothetical species."""
    hypos = prior.qs
    likelihood = [norm_map[hypo].pdf(data) for hypo in hypos]
    posterior = prior * likelihood
    posterior.normalize()
    return posterior_____no_output_____

Last time we used this function, the values in `norm_map` were `norm` objects, but it also works if they are `multivariate_normal` objects.

We can use it to classify a penguin with flipper length 193 and culmen length 48:_____no_output_____

data = 193, 48
update_penguin(prior, data, multinorm_map)_____no_output_____

A penguin with those measurements is almost certainly an Chinstrap._____no_output_____Now let's see if this classifier does any better than the naive Bayesian classifier.
I'll apply it to each penguin in the dataset:_____no_output_____

df['Classification'] = np.nan

for i, row in df.iterrows():
    data = row[colnames]
    posterior = update_penguin(prior, data, multinorm_map)
    df.loc[i, 'Classification'] = posterior.idxmax()_____no_output_____

And compute the accuracy:_____no_output_____

accuracy(df)_____no_output_____

It turns out to be only a little better: the accuracy is 95.3%, compared to 94.7% for the naive Bayesian classifier._____no_output_____## Summary

In this chapter, we implemented a naive Bayesian classifier, which is "naive" in the sense that it assumes that the features is uses for classification are independent.

To see how bad that assumption is, we also implemented a classifier that uses the a multivariate normal distribution to model the joint distribution of the features, which includes their dependencies.

In this example, the non-naive classifier is only marginally better.
In one way, that's disappointing.  After all that work, it would have been nice to see a bigger improvement.
But in another way, it's good news.  In general, a naive Bayesian classifier is easier to implement and requires less computation.  If it works nearly as well as a more complex algorithm, it might be a good choice for practical purposes.

Speaking of practical purposes, you might have noticed that this example isn't very useful.  If we want to identify the species of a penguin, there are easier ways than measuring its flippers and beak.

But there *are* scientific uses for this type of classification.  One of them is the subject of the research paper we started with: [sexual dimorphism](https://en.wikipedia.org/wiki/Sexual_dimorphism), that is, differences in shape between male and female animals.

In some species, like angler fish, males and females look very different.  In other species, like mockingbirds, they are difficult to tell apart.
And dimorphism is worth studying because it provides insight into social behavior, sexual selection, and evolution.

One way to quantify the degree of sexual dimorphism in a species is to use a classification algorithm like the one in this chapter.  If you can find a set of features that makes it possible to classify individuals by sex with high accuracy, that's evidence of high dimorphism.

As an exercise, you can use the dataset from this chapter to classify penguins by sex and see which of the three species is the most dimorphic._____no_output_____## Exercises_____no_output_____**Exercise:**  In my example I used culmen length and flipper length because they seemed to provide the most power to distinguish the three species.  But maybe we can do better by using more features.

Make a naive Bayesian classifier that uses all four measurements in the dataset: culmen length and depth, flipper length, and body mass.
Is it more accurate than the model with two features?_____no_output_____

# Solution

# Here are the norm maps for the other two features

depth_map = make_norm_map(df, 'Culmen Depth (mm)')
mass_map = make_norm_map(df, 'Body Mass (g)')_____no_output_____# Solution

# And here are sequences for the features and the norm maps

colnames4 = ['Culmen Length (mm)', 'Flipper Length (mm)',
             'Culmen Depth (mm)', 'Body Mass (g)']
norm_maps4 = [culmen_map, flipper_map,
              depth_map, mass_map]_____no_output_____# Solution

# Now let's classify and compute accuracy.

# We can do a little better with all four features,
# almost 97% accuracy

df['Classification'] = np.nan

for i, row in df.iterrows():
    data_seq = row[colnames4]
    posterior = update_naive(prior, data_seq, norm_maps4)
    df.loc[i, 'Classification'] = posterior.max_prob()

accuracy(df)_____no_output_____

**Exercise:**  One of the reasons the penguin dataset was collected was to quantify sexual dimorphism in different penguin species, that is, physical differences between male and female penguins.  One way to quantify dimorphism is to use measurements to classify penguins by sex.  If a species is more dimorphic, we expect to be able to classify them more accurately.

As an exercise, pick a species and use a Bayesian classifier (naive or not) to classify the penguins by sex.  Which features are most useful?  What accuracy can you achieve?_____no_output_____Note: One Gentoo penguin has an invalid value for `Sex`.  I used the following code to select one species and filter out invalid data._____no_output_____

gentoo = (df['Species2'] == 'Gentoo')
subset = df[gentoo].copy()_____no_output_____subset['Sex'].value_counts()_____no_output_____valid = df['Sex'] != '.'
valid.sum()_____no_output_____subset = df[valid & gentoo].copy()_____no_output_____

OK, you can finish it off from here._____no_output_____

# Solution

# Here are the feature distributions grouped by sex

plot_cdfs(subset, 'Culmen Length (mm)', by='Sex')_____no_output_____# Solution

plot_cdfs(subset, 'Culmen Depth (mm)', by='Sex')_____no_output_____# Solution

plot_cdfs(subset, 'Flipper Length (mm)', by='Sex')_____no_output_____# Solution

plot_cdfs(subset, 'Body Mass (g)', by='Sex')_____no_output_____# Solution

# Here are the norm maps for the features, grouped by sex

culmen_map = make_norm_map(subset, 'Culmen Length (mm)', by='Sex')
flipper_map = make_norm_map(subset, 'Flipper Length (mm)', by='Sex')
depth_map = make_norm_map(subset, 'Culmen Depth (mm)', by='Sex')
mass_map = make_norm_map(subset, 'Body Mass (g)', by='Sex')_____no_output_____# Solution

# And here are the sequences we need for `update_naive`

norm_maps4 = [culmen_map, flipper_map, depth_map, mass_map]
colnames4 = ['Culmen Length (mm)', 'Flipper Length (mm)',
             'Culmen Depth (mm)', 'Body Mass (g)']_____no_output_____# Solution

# Here's the prior

hypos = culmen_map.keys()
prior = Pmf(1/2, hypos)
prior_____no_output_____# Solution

# And the update

subset['Classification'] = np.nan

for i, row in subset.iterrows():
    data_seq = row[colnames4]
    posterior = update_naive(prior, data_seq, norm_maps4)
    subset.loc[i, 'Classification'] = posterior.max_prob()_____no_output_____# Solution

# This function computes accuracy

def accuracy_sex(df):
    """Compute the accuracy of classification.

    Compares columns Classification and Sex

    df: DataFrame
    """
    valid = df['Classification'].notna()
    same = df['Sex'] == df['Classification']
    return same.sum() / valid.sum()_____no_output_____# Solution

# Using these features we can classify Gentoo penguins by
# sex with almost 92% accuracy

accuracy_sex(subset)_____no_output_____# Solution

# Here's the whole process in a function so we can
# classify the other species

def classify_by_sex(subset):
    """Run the whole classification process.

    subset: DataFrame
    """
    culmen_map = make_norm_map(subset, 'Culmen Length (mm)', by='Sex')
    flipper_map = make_norm_map(subset, 'Flipper Length (mm)', by='Sex')
    depth_map = make_norm_map(subset, 'Culmen Depth (mm)', by='Sex')
    mass_map = make_norm_map(subset, 'Body Mass (g)', by='Sex')

    norm_maps4 = [culmen_map, flipper_map, depth_map, mass_map]

    hypos = culmen_map.keys()
    prior = Pmf(1/2, hypos)

    subset['Classification'] = np.nan

    for i, row in subset.iterrows():
        data_seq = row[colnames4]
        posterior = update_naive(prior, data_seq, norm_maps4)
        subset.loc[i, 'Classification'] = posterior.max_prob()

    return accuracy_sex(subset)_____no_output_____# Solution

# Here's the subset of Adelie penguins

# The accuracy is about 88%

adelie = df['Species2']=='Adelie'
subset = df[adelie].copy()
classify_by_sex(subset)_____no_output_____# Solution

# And for Chinstrap, accuracy is about 92%

chinstrap = df['Species2']=='Chinstrap'
subset = df[chinstrap].copy()
classify_by_sex(subset)_____no_output_____# Solution

# It looks like Gentoo and Chinstrap penguins are about equally
# dimorphic, Adelie penguins a little less so.

# All of these results are consistent with what's in the paper._____no_output_____


# Notebook from OliverDudgeon/QSync
Path: qutip-notebooks-master/examples/control-pulseoptim-CRAB-QFT.ipynb

# Calculation of control fields for QFT gate on two qubits using the CRAB algorithm_____no_output_____Alexander Pitchford (agp1@aber.ac.uk)_____no_output_____

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import datetime_____no_output_____from qutip import Qobj, identity, sigmax, sigmay, sigmaz, tensor
from qutip.qip.algorithms import qft
import qutip.logging_utils as logging
logger = logging.get_logger()
#Set this to None or logging.WARN for 'quiet' execution
log_level = logging.INFO
#QuTiP control modules
import qutip.control.pulseoptim as cpo
import qutip.control.pulsegen as pulsegen

example_name = 'QFT'
_____no_output_____

### Defining the physics_____no_output_____

Sx = sigmax()
Sy = sigmay()
Sz = sigmaz()
Si = 0.5*identity(2)

# Drift Hamiltonian
H_d = 0.5*(tensor(Sx, Sx) + tensor(Sy, Sy) + tensor(Sz, Sz))
# The (four) control Hamiltonians
H_c = [tensor(Sx, Si), tensor(Sy, Si), tensor(Si, Sx), tensor(Si, Sy)]
n_ctrls = len(H_c)
# start point for the gate evolution
U_0 = identity(4)
# Target for the gate evolution - Quantum Fourier Transform gate
U_targ = qft.qft(2)_____no_output_____

### Defining the time evolution parameters_____no_output_____

# Number of time slots
n_ts = 200
# Time allowed for the evolution
evo_time = 10_____no_output_____

### Set the conditions which will cause the pulse optimisation to terminate_____no_output_____

# Fidelity error target
fid_err_targ = 1e-3
# Maximum iterations for the optisation algorithm
max_iter = 20000
# Maximum (elapsed) time allowed in seconds
max_wall_time = 300_____no_output_____

### Give an extension for output files_____no_output_____

#Set to None to suppress output files
f_ext = "{}_n_ts{}.txt".format(example_name, n_ts)_____no_output_____

### Create the optimiser objects_____no_output_____

optim = cpo.create_pulse_optimizer(H_d, H_c, U_0, U_targ, n_ts, evo_time,
                fid_err_targ=fid_err_targ,
                max_iter=max_iter, max_wall_time=max_wall_time,
                alg='CRAB',
                dyn_type='UNIT',
                prop_type='DIAG',
                fid_type='UNIT', fid_params={'phase_option':'PSU'},
                log_level=log_level, gen_stats=True)
                _____no_output_____

### Configure the pulses for each of the controls_____no_output_____

dyn = optim.dynamics

# Control 1
crab_pgen = optim.pulse_generator[0]
# Start from a ramped pulse
guess_pgen = pulsegen.create_pulse_gen('LIN', dyn=dyn,
                                           pulse_params={'scaling':3.0})
crab_pgen.guess_pulse = guess_pgen.gen_pulse()
crab_pgen.scaling = 0.0
# Add some higher frequency components
crab_pgen.num_coeffs = 5

# Control 2
crab_pgen = optim.pulse_generator[1]
# Apply a ramping pulse that will force the start and end to zero
ramp_pgen = pulsegen.create_pulse_gen('GAUSSIAN_EDGE', dyn=dyn,
                                    pulse_params={'decay_time':evo_time/50.0})
crab_pgen.ramping_pulse = ramp_pgen.gen_pulse()

# Control 3
crab_pgen = optim.pulse_generator[2]
# Add bounds
crab_pgen.scaling = 0.5
crab_pgen.lbound = -2.0
crab_pgen.ubound = 2.0

# Control 4
crab_pgen = optim.pulse_generator[3]
# Start from a triangular pulse with small signal
guess_pgen = pulsegen.PulseGenTriangle(dyn=dyn)
guess_pgen.num_waves = 1
guess_pgen.scaling = 2.0
guess_pgen.offset = 2.0
crab_pgen.guess_pulse = guess_pgen.gen_pulse()
crab_pgen.scaling = 0.1

init_amps = np.zeros([n_ts, n_ctrls])
for j in range(dyn.num_ctrls):
    pgen = optim.pulse_generator[j]
    pgen.init_pulse()
    init_amps[:, j] = pgen.gen_pulse()

dyn.initialize_controls(init_amps)INFO:qutip.control.pulsegen:The number of CRAB coefficients per basis function has been estimated as 3, which means a total of 6 optimisation variables for this pulse. Based on the dimension (4) of the system
INFO:qutip.control.pulsegen:The number of CRAB coefficients per basis function has been estimated as 3, which means a total of 6 optimisation variables for this pulse. Based on the dimension (4) of the system
INFO:qutip.control.pulsegen:The number of CRAB coefficients per basis function has been estimated as 3, which means a total of 6 optimisation variables for this pulse. Based on the dimension (4) of the system
INFO:qutip.control.dynamics:Setting memory optimisations for level 0
INFO:qutip.control.dynamics:Internal operator data type choosen to be <class 'numpy.ndarray'>
INFO:qutip.control.dynamics:phased dynamics generator caching True
INFO:qutip.control.dynamics:propagator gradient caching True
INFO:qutip.control.dynamics:eigenvector adjoint caching True
INFO:qutip.control.dynamics:use sparse eigen decomp False

### Run the pulse optimisation_____no_output_____

# Save initial amplitudes to a text file
if f_ext is not None:
    pulsefile = "ctrl_amps_initial_" + f_ext
    dyn.save_amps(pulsefile)
    print("Initial amplitudes output to file: " + pulsefile)

print("***********************************")
print("Starting pulse optimisation")
result = optim.run_optimization()

# Save final amplitudes to a text file
if f_ext is not None:
    pulsefile = "ctrl_amps_final_" + f_ext
    dyn.save_amps(pulsefile)
    print("Final amplitudes output to file: " + pulsefile)Initial amplitudes output to file: ctrl_amps_initial_QFT_n_ts200.txt
***********************************

### Report the results_____no_output_____

result.stats.report()
print("Final evolution\n{}\n".format(result.evo_full_final))
print("********* Summary *****************")
print("Initial fidelity error {}".format(result.initial_fid_err))
print("Final fidelity error {}".format(result.fid_err))
print("Terminated due to {}".format(result.termination_reason))
print("Number of iterations {}".format(result.num_iter))
print("Completed in {} HH:MM:SS.US".format(
        datetime.timedelta(seconds=result.wall_time)))
------------------------------------
---- Control optimisation stats ----
**** Timings (HH:MM:SS.US) ****
Total wall time elapsed during optimisation: 0:05:00.003775
Wall time computing Hamiltonians: 0:00:22.547817 (7.52%)
Wall time computing propagators: 0:04:24.789280 (88.26%)
Wall time computing forward propagation: 0:00:03.857988 (1.29%)
Wall time computing onward propagation: 0:00:03.634444 (1.21%)
Wall time computing gradient: 0:00:00 (0.00%)

**** Iterations and function calls ****
Number of iterations: 8610
Number of fidelity function calls: 10459
Number of times fidelity is computed: 10459
Number of gradient function calls: 0
Number of times gradients are computed: 0
Number of times timeslot evolution is recomputed: 10459

**** Control amplitudes ****
Number of control amplitude updates: 10458
Mean number of updates per iteration: 1.2146341463414634
Number of timeslot values changed: 2091599
Mean number of timeslot changes per update: 199.99990437942245
Number of amplitude values changed: 8350196
Mean number of amplitude changes per update: 798.4505641614076
------------------------------------
Final evolution
Quantum object: dims = [[4], [4]], shape = (4, 4), type = oper, isherm = False
Qobj data =
[[-0.47890815-0.25124383j -0.47862720-0.14293885j -0.47089104-0.19654053j
  -0.39814103-0.19780087j]
 [-0.48524763-0.18754841j  0.19902638-0.45262542j  0.45666538+0.17490099j
  -0.22150294+0.44348832j]
 [-0.44119050-0.18412439j  0.49542987+0.17619236j -0.42828736-0.17952947j
   0.50360656+0.16023168j]
 [-0.41755379-0.18434167j -0.18861574+0.44037804j  0.50645054+0.16836502j
   0.23840544-0.4695553j ]]

********* Summary *****************
Initial fidelity error 0.9483036893449602
Final fidelity error 0.0032564547889728512
Terminated due to Max wall time exceeded
Number of iterations 8611
Completed in 0:05:00.003775 HH:MM:SS.US

### Plot the initial and final amplitudes_____no_output_____

fig1 = plt.figure()
ax1 = fig1.add_subplot(2, 1, 1)
ax1.set_title("Initial Control amps")
ax1.set_xlabel("Time")
ax1.set_ylabel("Control amplitude")
for j in range(n_ctrls):
    ax1.step(result.time,
             np.hstack((result.initial_amps[:, j], result.initial_amps[-1, j])),
             where='post')
ax2 = fig1.add_subplot(2, 1, 2)
ax2.set_title("Optimised Control Amplitudes")
ax2.set_xlabel("Time")
ax2.set_ylabel("Control amplitude")
for j in range(n_ctrls):
    ax2.step(result.time,
             np.hstack((result.final_amps[:, j], result.final_amps[-1, j])),
             where='post', label='u{}'.format(j))
ax2.legend(loc=8, ncol=n_ctrls)
plt.tight_layout()
plt.show()_____no_output_____

### Versions_____no_output_____

from qutip.ipynbtools import version_table

version_table()_____no_output_____


# Notebook from jbentleyEG/h2o-3
Path: examples/deeplearning/notebooks/deeplearning_image_reconstruction_and_clustering.ipynb

# Image Space Projection using Autoencoders
![](./images/faces.gif)

In this example we are going to autoencode the faces of the olivetti dataset and try to reconstruct them back._____no_output_____

%matplotlib inline

import matplotlib
import numpy as np
import pandas as pd
import scipy.io
import matplotlib.pyplot as plt
from IPython.display import Image, display_____no_output_____import h2o
from h2o.estimators.deeplearning import H2OAutoEncoderEstimator
h2o.init()Checking whether there is an H2O instance running at http://localhost:54321. connected.

http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html_____no_output_____

!wget -c http://www.cl.cam.ac.uk/Research/DTG/attarchive/pub/data/att_faces.tar.Z--2016-10-23 01:11:47--  http://www.cl.cam.ac.uk/Research/DTG/attarchive/pub/data/att_faces.tar.Z
Resolving www.cl.cam.ac.uk (www.cl.cam.ac.uk)... 2001:630:212:200::80:14, 128.232.0.20
Connecting to www.cl.cam.ac.uk (www.cl.cam.ac.uk)|2001:630:212:200::80:14|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://www.cl.cam.ac.uk/research/dtg/attarchive/pub/data/att_faces.tar.Z [following]
--2016-10-23 01:11:47--  http://www.cl.cam.ac.uk/research/dtg/attarchive/pub/data/att_faces.tar.Z
Reusing existing connection to [www.cl.cam.ac.uk]:80.
HTTP request sent, awaiting response... 200 OK
Length: 4075767 (3.9M) [application/x-compress]
Saving to: ‘att_faces.tar.Z’

att_faces.tar.Z     100%[===================>]   3.89M  2.55MB/s    in 1.5s

2016-10-23 01:11:49 (2.55 MB/s) - ‘att_faces.tar.Z’ saved [4075767/4075767]

!tar xzvf att_faces.tar.Z;rm att_faces.tar.Z;orl_faces/
orl_faces/README
orl_faces/s1/
orl_faces/s1/6.pgm
orl_faces/s1/7.pgm
orl_faces/s1/8.pgm
orl_faces/s1/9.pgm
orl_faces/s1/10.pgm
orl_faces/s1/1.pgm
orl_faces/s1/2.pgm
orl_faces/s1/3.pgm
orl_faces/s1/4.pgm
orl_faces/s1/5.pgm
orl_faces/s2/
orl_faces/s2/6.pgm
orl_faces/s2/7.pgm
orl_faces/s2/8.pgm
orl_faces/s2/9.pgm
orl_faces/s2/10.pgm
orl_faces/s2/1.pgm
orl_faces/s2/2.pgm
orl_faces/s2/3.pgm
orl_faces/s2/4.pgm
orl_faces/s2/5.pgm
orl_faces/s3/
orl_faces/s3/6.pgm
orl_faces/s3/7.pgm
orl_faces/s3/8.pgm
orl_faces/s3/9.pgm
orl_faces/s3/10.pgm
orl_faces/s3/1.pgm
orl_faces/s3/2.pgm
orl_faces/s3/3.pgm
orl_faces/s3/4.pgm
orl_faces/s3/5.pgm
orl_faces/s4/
orl_faces/s4/6.pgm
orl_faces/s4/7.pgm
orl_faces/s4/8.pgm
orl_faces/s4/9.pgm
orl_faces/s4/10.pgm
orl_faces/s4/1.pgm
orl_faces/s4/2.pgm
orl_faces/s4/3.pgm
orl_faces/s4/4.pgm
orl_faces/s4/5.pgm
orl_faces/s5/
orl_faces/s5/6.pgm
orl_faces/s5/7.pgm
orl_faces/s5/8.pgm
orl_faces/s5/9.pgm
orl_faces/s5/10.pgm
orl_faces/s5/1.pgm
orl_faces/s5/2.pgm
orl_faces/s5/3.pgm
orl_faces/s5/4.pgm
orl_faces/s5/5.pgm
orl_faces/s6/
orl_faces/s6/6.pgm
orl_faces/s6/7.pgm
orl_faces/s6/8.pgm
orl_faces/s6/9.pgm
orl_faces/s6/10.pgm
orl_faces/s6/1.pgm
orl_faces/s6/2.pgm
orl_faces/s6/3.pgm
orl_faces/s6/4.pgm
orl_faces/s6/5.pgm
orl_faces/s7/
orl_faces/s7/6.pgm
orl_faces/s7/7.pgm
orl_faces/s7/8.pgm
orl_faces/s7/9.pgm
orl_faces/s7/10.pgm
orl_faces/s7/1.pgm
orl_faces/s7/2.pgm
orl_faces/s7/3.pgm
orl_faces/s7/4.pgm
orl_faces/s7/5.pgm
orl_faces/s8/
orl_faces/s8/6.pgm
orl_faces/s8/7.pgm
orl_faces/s8/8.pgm
orl_faces/s8/9.pgm
orl_faces/s8/10.pgm
orl_faces/s8/1.pgm
orl_faces/s8/2.pgm
orl_faces/s8/3.pgm
orl_faces/s8/4.pgm
orl_faces/s8/5.pgm
orl_faces/s9/
orl_faces/s9/6.pgm
orl_faces/s9/7.pgm
orl_faces/s9/8.pgm
orl_faces/s9/9.pgm
orl_faces/s9/10.pgm
orl_faces/s9/1.pgm
orl_faces/s9/2.pgm
orl_faces/s9/3.pgm
orl_faces/s9/4.pgm
orl_faces/s9/5.pgm
orl_faces/s10/
orl_faces/s10/6.pgm
orl_faces/s10/7.pgm
orl_faces/s10/8.pgm
orl_faces/s10/9.pgm
orl_faces/s10/10.pgm
orl_faces/s10/1.pgm
orl_faces/s10/2.pgm
orl_faces/s10/3.pgm
orl_faces/s10/4.pgm
orl_faces/s10/5.pgm
orl_faces/s11/
orl_faces/s11/6.pgm
orl_faces/s11/7.pgm
orl_faces/s11/8.pgm
orl_faces/s11/9.pgm
orl_faces/s11/10.pgm
orl_faces/s11/1.pgm
orl_faces/s11/2.pgm
orl_faces/s11/3.pgm
orl_faces/s11/4.pgm
orl_faces/s11/5.pgm
orl_faces/s12/
orl_faces/s12/6.pgm
orl_faces/s12/7.pgm
orl_faces/s12/8.pgm
orl_faces/s12/9.pgm
orl_faces/s12/10.pgm
orl_faces/s12/1.pgm
orl_faces/s12/2.pgm
orl_faces/s12/3.pgm
orl_faces/s12/4.pgm
orl_faces/s12/5.pgm
orl_faces/s13/
orl_faces/s13/6.pgm
orl_faces/s13/7.pgm
orl_faces/s13/8.pgm
orl_faces/s13/9.pgm
orl_faces/s13/10.pgm
orl_faces/s13/1.pgm
orl_faces/s13/2.pgm
orl_faces/s13/3.pgm
orl_faces/s13/4.pgm
orl_faces/s13/5.pgm
orl_faces/s14/
orl_faces/s14/6.pgm
orl_faces/s14/7.pgm
orl_faces/s14/8.pgm
orl_faces/s14/9.pgm
orl_faces/s14/10.pgm
orl_faces/s14/1.pgm
orl_faces/s14/2.pgm
orl_faces/s14/3.pgm
orl_faces/s14/4.pgm
orl_faces/s14/5.pgm
orl_faces/s15/
orl_faces/s15/6.pgm
orl_faces/s15/7.pgm
orl_faces/s15/8.pgm
orl_faces/s15/9.pgm
orl_faces/s15/10.pgm
orl_faces/s15/1.pgm
orl_faces/s15/2.pgm
orl_faces/s15/3.pgm
orl_faces/s15/4.pgm
orl_faces/s15/5.pgm
orl_faces/s16/
orl_faces/s16/6.pgm
orl_faces/s16/7.pgm
orl_faces/s16/8.pgm
orl_faces/s16/9.pgm
orl_faces/s16/10.pgm
orl_faces/s16/1.pgm
orl_faces/s16/2.pgm
orl_faces/s16/3.pgm
orl_faces/s16/4.pgm
orl_faces/s16/5.pgm
orl_faces/s17/
orl_faces/s17/6.pgm
orl_faces/s17/7.pgm
orl_faces/s17/8.pgm
orl_faces/s17/9.pgm
orl_faces/s17/10.pgm
orl_faces/s17/1.pgm
orl_faces/s17/2.pgm
orl_faces/s17/3.pgm
orl_faces/s17/4.pgm
orl_faces/s17/5.pgm
orl_faces/s18/
orl_faces/s18/6.pgm
orl_faces/s18/7.pgm
orl_faces/s18/8.pgm
orl_faces/s18/9.pgm
orl_faces/s18/10.pgm
orl_faces/s18/1.pgm
orl_faces/s18/2.pgm
orl_faces/s18/3.pgm
orl_faces/s18/4.pgm
orl_faces/s18/5.pgm
orl_faces/s19/
orl_faces/s19/6.pgm
orl_faces/s19/7.pgm
orl_faces/s19/8.pgm
orl_faces/s19/9.pgm
orl_faces/s19/10.pgm
orl_faces/s19/1.pgm
orl_faces/s19/2.pgm
orl_faces/s19/3.pgm
orl_faces/s19/4.pgm
orl_faces/s19/5.pgm
orl_faces/s20/
orl_faces/s20/6.pgm
orl_faces/s20/7.pgm
orl_faces/s20/8.pgm
orl_faces/s20/9.pgm
orl_faces/s20/10.pgm
orl_faces/s20/1.pgm
orl_faces/s20/2.pgm
orl_faces/s20/3.pgm
orl_faces/s20/4.pgm
orl_faces/s20/5.pgm
orl_faces/s21/
orl_faces/s21/6.pgm
orl_faces/s21/7.pgm
orl_faces/s21/8.pgm
orl_faces/s21/9.pgm
orl_faces/s21/10.pgm
orl_faces/s21/1.pgm
orl_faces/s21/2.pgm
orl_faces/s21/3.pgm
orl_faces/s21/4.pgm
orl_faces/s21/5.pgm
orl_faces/s22/
orl_faces/s22/6.pgm
orl_faces/s22/7.pgm
orl_faces/s22/8.pgm
orl_faces/s22/9.pgm
orl_faces/s22/10.pgm
orl_faces/s22/1.pgm
orl_faces/s22/2.pgm
orl_faces/s22/3.pgm
orl_faces/s22/4.pgm
orl_faces/s22/5.pgm
orl_faces/s23/
orl_faces/s23/6.pgm
orl_faces/s23/7.pgm
orl_faces/s23/8.pgm
orl_faces/s23/9.pgm
orl_faces/s23/10.pgm
orl_faces/s23/1.pgm
orl_faces/s23/2.pgm
orl_faces/s23/3.pgm
orl_faces/s23/4.pgm
orl_faces/s23/5.pgm
orl_faces/s24/
orl_faces/s24/6.pgm
orl_faces/s24/7.pgm
orl_faces/s24/8.pgm
orl_faces/s24/9.pgm
orl_faces/s24/10.pgm
orl_faces/s24/1.pgm
orl_faces/s24/2.pgm
orl_faces/s24/3.pgm
orl_faces/s24/4.pgm
orl_faces/s24/5.pgm
orl_faces/s25/
orl_faces/s25/6.pgm
orl_faces/s25/7.pgm
orl_faces/s25/8.pgm
orl_faces/s25/9.pgm
orl_faces/s25/10.pgm
orl_faces/s25/1.pgm
orl_faces/s25/2.pgm
orl_faces/s25/3.pgm
orl_faces/s25/4.pgm
orl_faces/s25/5.pgm
orl_faces/s26/
orl_faces/s26/6.pgm
orl_faces/s26/7.pgm
orl_faces/s26/8.pgm
orl_faces/s26/9.pgm
orl_faces/s26/10.pgm
orl_faces/s26/1.pgm
orl_faces/s26/2.pgm
orl_faces/s26/3.pgm
orl_faces/s26/4.pgm
orl_faces/s26/5.pgm
orl_faces/s27/
orl_faces/s27/6.pgm
orl_faces/s27/7.pgm
orl_faces/s27/8.pgm
orl_faces/s27/9.pgm
orl_faces/s27/10.pgm
orl_faces/s27/1.pgm
orl_faces/s27/2.pgm
orl_faces/s27/3.pgm
orl_faces/s27/4.pgm
orl_faces/s27/5.pgm
orl_faces/s28/
orl_faces/s28/6.pgm
orl_faces/s28/7.pgm
orl_faces/s28/8.pgm
orl_faces/s28/9.pgm
orl_faces/s28/10.pgm
orl_faces/s28/1.pgm
orl_faces/s28/2.pgm
orl_faces/s28/3.pgm
orl_faces/s28/4.pgm
orl_faces/s28/5.pgm
orl_faces/s29/
orl_faces/s29/6.pgm
orl_faces/s29/7.pgm
orl_faces/s29/8.pgm
orl_faces/s29/9.pgm
orl_faces/s29/10.pgm
orl_faces/s29/1.pgm
orl_faces/s29/2.pgm
orl_faces/s29/3.pgm
orl_faces/s29/4.pgm
orl_faces/s29/5.pgm
orl_faces/s30/
orl_faces/s30/6.pgm
orl_faces/s30/7.pgm
orl_faces/s30/8.pgm
orl_faces/s30/9.pgm
orl_faces/s30/10.pgm
orl_faces/s30/1.pgm
orl_faces/s30/2.pgm
orl_faces/s30/3.pgm
orl_faces/s30/4.pgm
orl_faces/s30/5.pgm
orl_faces/s31/
orl_faces/s31/6.pgm
orl_faces/s31/7.pgm
orl_faces/s31/8.pgm
orl_faces/s31/9.pgm
orl_faces/s31/10.pgm
orl_faces/s31/1.pgm
orl_faces/s31/2.pgm
orl_faces/s31/3.pgm
orl_faces/s31/4.pgm
orl_faces/s31/5.pgm
orl_faces/s32/
orl_faces/s32/6.pgm
orl_faces/s32/7.pgm
orl_faces/s32/8.pgm
orl_faces/s32/9.pgm
orl_faces/s32/10.pgm
orl_faces/s32/1.pgm
orl_faces/s32/2.pgm
orl_faces/s32/3.pgm
orl_faces/s32/4.pgm
orl_faces/s32/5.pgm
orl_faces/s33/
orl_faces/s33/6.pgm
orl_faces/s33/7.pgm
orl_faces/s33/8.pgm
orl_faces/s33/9.pgm
orl_faces/s33/10.pgm
orl_faces/s33/1.pgm
orl_faces/s33/2.pgm
orl_faces/s33/3.pgm
orl_faces/s33/4.pgm
orl_faces/s33/5.pgm
orl_faces/s34/
orl_faces/s34/6.pgm
orl_faces/s34/7.pgm
orl_faces/s34/8.pgm
orl_faces/s34/9.pgm
orl_faces/s34/10.pgm
orl_faces/s34/1.pgm
orl_faces/s34/2.pgm
orl_faces/s34/3.pgm
orl_faces/s34/4.pgm
orl_faces/s34/5.pgm
orl_faces/s35/
orl_faces/s35/6.pgm
orl_faces/s35/7.pgm
orl_faces/s35/8.pgm
orl_faces/s35/9.pgm
orl_faces/s35/10.pgm
orl_faces/s35/1.pgm
orl_faces/s35/2.pgm
orl_faces/s35/3.pgm
orl_faces/s35/4.pgm
orl_faces/s35/5.pgm
orl_faces/s36/
orl_faces/s36/6.pgm
orl_faces/s36/7.pgm
orl_faces/s36/8.pgm
orl_faces/s36/9.pgm
orl_faces/s36/10.pgm
orl_faces/s36/1.pgm
orl_faces/s36/2.pgm
orl_faces/s36/3.pgm
orl_faces/s36/4.pgm
orl_faces/s36/5.pgm
orl_faces/s37/
orl_faces/s37/6.pgm
orl_faces/s37/7.pgm
orl_faces/s37/8.pgm
orl_faces/s37/9.pgm
orl_faces/s37/10.pgm
orl_faces/s37/1.pgm
orl_faces/s37/2.pgm
orl_faces/s37/3.pgm
orl_faces/s37/4.pgm
orl_faces/s37/5.pgm
orl_faces/s38/
orl_faces/s38/6.pgm
orl_faces/s38/7.pgm
orl_faces/s38/8.pgm
orl_faces/s38/9.pgm
orl_faces/s38/10.pgm
orl_faces/s38/1.pgm
orl_faces/s38/2.pgm
orl_faces/s38/3.pgm
orl_faces/s38/4.pgm
orl_faces/s38/5.pgm
orl_faces/s39/
orl_faces/s39/6.pgm
orl_faces/s39/7.pgm
orl_faces/s39/8.pgm
orl_faces/s39/9.pgm
orl_faces/s39/10.pgm
orl_faces/s39/1.pgm
orl_faces/s39/2.pgm
orl_faces/s39/3.pgm
orl_faces/s39/4.pgm
orl_faces/s39/5.pgm
orl_faces/s40/
orl_faces/s40/6.pgm
orl_faces/s40/7.pgm
orl_faces/s40/8.pgm
orl_faces/s40/9.pgm
orl_faces/s40/10.pgm
orl_faces/s40/1.pgm
orl_faces/s40/2.pgm
orl_faces/s40/3.pgm
orl_faces/s40/4.pgm
orl_faces/s40/5.pgm

We now need some code to read pgm files.
Thanks to [StackOverflow](http://stackoverflow.com/questions/7368739/numpy-and-16-bit-pgm) we have some code to leverage:_____no_output_____

import re

def read_pgm(filename, byteorder='>'):
    """Return image data from a raw PGM file as numpy array.

    Format specification: http://netpbm.sourceforge.net/doc/pgm.html

    """
    with open(filename, 'rb') as f:
        buffer = f.read()
    try:
        header, width, height, maxval = re.search(
            b"(^P5\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n]\s)*)", buffer).groups()
    except AttributeError:
        raise ValueError("Not a raw PGM file: '%s'" % filename)
    return np.frombuffer(buffer,
                            dtype='u1' if int(maxval) < 256 else byteorder+'u2',
                            count=int(width)*int(height),
                            offset=len(header)
                            ).reshape((int(height), int(width)))
_____no_output_____image = read_pgm("orl_faces/s12/6.pgm", byteorder='<')_____no_output_____image.shape_____no_output_____plt.imshow(image, plt.cm.gray)
plt.show()_____no_output_____import glob
import os
from collections import defaultdict_____no_output_____images = glob.glob("orl_faces/**/*.pgm")

data = defaultdict(list)
image_data = []
for img in images:
    _,label,_ = img.split(os.path.sep)
    imgdata = read_pgm(img, byteorder='<').flatten().tolist()
    data[label].append(imgdata)
    image_data.append(imgdata)_____no_output_____

Let's import it to H2O_____no_output_____

faces = h2o.H2OFrame(image_data)Parse progress: |█████████████████████████████████████████████████████████████████████████████| 100%
faces.shape_____no_output_____from h2o.estimators.deeplearning import H2OAutoEncoderEstimator_____no_output_____model = H2OAutoEncoderEstimator(
        activation="Tanh",
        hidden=[50],
        l1=1e-4,
        epochs=10
)

model.train(x=faces.names, training_frame=faces)deeplearning Model Build progress: |██████████████████████████████████████████████████████████| 100%
modelModel Details
=============
H2OAutoEncoderEstimator :  Deep Learning
Model Key:  DeepLearning_model_python_1477202989522_3315
Status of Neuron Layers: auto-encoder, gaussian distribution, Quadratic loss, 1,040,754 weights/biases, 13.0 MB, 4,000 training samples, mini-batch size 1

## Reconstructing the hidden space

Now that we have our model trained, we would like to understand better what is the internal representation of this model? What makes a face a .. face?

We will provide to the model some gaussian noise and see what is the results.

We star by creating some gaussian noise:_____no_output_____

import pandas as pd_____no_output_____gaussian_noise = np.random.randn(10304)_____no_output_____plt.imshow(gaussian_noise.reshape(112, 92), plt.cm.gray);_____no_output_____

Then we import this data inside H2O. We have to first map the columns to the gaussian data._____no_output_____

gaussian_noise_pre = dict(zip(faces.names,gaussian_noise))_____no_output_____gaussian_noise_hf = h2o.H2OFrame.from_python(gaussian_noise_pre)Parse progress: |█████████████████████████████████████████████████████████████████████████████| 100%
result = model.predict(gaussian_noise_hf)deeplearning prediction progress: |███████████████████████████████████████████████████████████| 100%
result.shape_____no_output_____img = result.as_data_frame()_____no_output_____img_data = img.T.values.reshape(112, 92)_____no_output_____plt.imshow(img_data, plt.cm.gray);_____no_output_____