# Notebook from arifmudi/Natural-Language-Processing-and-Computational-Linguistics
Path: notebooks/ldaseqmodel.ipynb

# Dynamic Topic Models Tutorial_____no_output_____
### What is this tutorial about?
This tutorial will exaplin what Dynamic Topic Models are, and how to use them using the LdaSeqModel class of gensim. I will start with some theory and already documented examples, before moving on to an example run of Dynamic Topic Model on a sample dataset. I will also do an comparison between the python code and the already present gensim wrapper, and illustrate how to easily visualise them and do topic coherence with DTM. Any suggestions to improve documentation or code through PRs/Issues is always appreciated!

### Why Dynamic Topic Models?
Imagine you have a gigantic corpus which spans over a couple of years. You want to find semantically similar documents; one from the very beginning of your time-line, and one in the very end. How would you?
This is where Dynamic Topic Models comes in. By having a time-based element to topics, context is preserved while key-words may change.

David Blei does a good job explaining the theory behind this in this [Google talk](https://www.youtube.com/watch?v=7BMsuyBPx90). If you prefer to directly read the [paper on DTM by Blei and Lafferty](http://repository.cmu.edu/cgi/viewcontent.cgi?article=2036&context=compsci), that should get you upto speed too.

In his talk, Blei gives a very interesting example of the motivation to use DTM. After running DTM on a dataset of the Science Journal from 1880 onwards, he picks up [this](http://science.sciencemag.org/content/os-1/28/326) paper - The Brain of the Orang (1880). It's topics are concentrated on a topic which must be to do with Monkeys, and with Neuroscience or brains.

He goes ahead to pick up another paper with likely very less common words, but in the same context - analysing monkey brains. In fact, this one is called - "[Representation of the visual field on the medial wall of occipital-parietal cortex in the owl monkey](http://allmanlab.caltech.edu/PDFs/AllmanKaas1976.pdf)". Quite the title, eh? Like mentioned before, you wouldn't imagine too many common words in these two papers, about a 100 years apart.

But a Hellinger Distance based Document-Topic distribution gives a very high similarity value! The same topics evolved smoothly over time and the context remains. A document similarity match using other traditional techniques might not work very well on this!
Blei defines this technique as - "Time corrected Document Similarity".

Another obviously useful analysis is to see how words in a topic change over time. The same broad classified topic starts looking more 'mature' as time goes on. This image illustrates an example from the same paper linked to above.

#### So, briefly put :

Dynamic Topic Models are used to model the evolution of topics in a corpus, over time. The Dynamic Topic Model is part of a class of probabilistic topic models, like the LDA.

While most traditional topic mining algorithms do not expect time-tagged data or take into account any prior ordering, Dynamic Topic Models (DTM) leverages the knowledge of different documents belonging to a different time-slice in an attempt to map how the words in a topic change over time.

[This](http://rare-technologies.com/understanding-and-coding-dynamic-topic-models/) blog post is also useful in breaking down the ideas and theory behind DTM.

_____no_output_____### Motivation to code this!_____no_output_____But - why even undertake this, especially when Gensim itself have a wrapper?
The main motivation was the lack of documentation in the original code - and the fact that doing an only python version makes it easier to use gensim building blocks. For example, for setting up the Sufficient Statistics to initialize the DTM, you can just pass a pre-trained gensim LDA model!

There is some clarity on how they built their code now - Variational Inference using Kalman Filters, as described in section 3 of the paper. The mathematical basis for the code is well described in the appendix of the paper. If the documentation is lacking or not clear, comments via Issues or PRs via the gensim repo would be useful in improving the quality.

This project was part of the Google Summer of Code 2016 program: I have been regularly blogging about my progress with implementing this, which you can find [here](http://rare-technologies.com/author/bhargav/)._____no_output_____# Using LdaSeqModel for DTM_____no_output_____Gensim already has a wrapper for original C++ DTM code, but the `LdaSeqModel` class is an effort to have a pure python implementation of the same.
Using it is very similar to using any other gensim topic-modelling algorithm, with all you need to start is an iterable gensim corpus, id2word and a list with the number of documents in each of your time-slices._____no_output_____

# setting up our imports

from gensim.models import ldaseqmodel
from gensim.corpora import Dictionary, bleicorpus
import numpy
from gensim.matutils import hellinger_____no_output_____

We will be loading the corpus and dictionary from disk. Here our corpus in the Blei corpus format, but it can be any iterable corpus.
The data set here consists of news reports over 3 months (March, April, May) downloaded from [here](https://github.com/derekgreene/dynamic-nmf/tree/master/data) and cleaned.

The files can also be found after processing in the `datasets` folder, but it would be a good exercise to download the files in the link and have a shot at cleaning and processing yourself. :)

Note: the dataset is not cleaned and requires pre-processing. This means that some care must be taken to group the relevant docs together (the months are mentioned as per the folder name), after which it needs to be broken into tokens, and stop words must be removed. This is a link to some basic pre-processing for gensim - [link](https://radimrehurek.com/gensim/tut1.html#from-strings-to-vectors).

### What is a time-slice?
A very important input for DTM to work is the `time_slice` input. It should be a list which contains the number of documents in each time-slice. In our case, the first month had 438 articles, the second 430 and the last month had 456 articles. This means we'd need an input which looks like this: `time_slice = [438, 430, 456]`.
Technically, a time-slice can be a month, year, or any way you wish to split up the number of documents in your corpus, time-based.

Once you have your corpus, id2word and time_slice ready, we're good to go!_____no_output_____

# loading our corpus and dictionary
try:
    dictionary = Dictionary.load('datasets/news_dictionary')
except FileNotFoundError as e:
    raise ValueError("SKIP: Please download the Corpus/news_dictionary dataset.")
corpus = bleicorpus.BleiCorpus('datasets/news_corpus')
# it's very important that your corpus is saved in order of your time-slices!

time_slice = [438, 430, 456]_____no_output_____

For DTM to work it first needs the Sufficient Statistics from a trained LDA model on the _*same*_ dataset.
By default LdaSeqModel trains it's own model and passes those values on, but can also accept a pre-trained gensim LDA model, or a numpy matrix which contains the Suff Stats.

We will be training our model in default mode, so gensim LDA will be first trained on the dataset.

NOTE: You have to set logging as true to see your progress!_____no_output_____

ldaseq = ldaseqmodel.LdaSeqModel(corpus=corpus, id2word=dictionary, time_slice=time_slice, num_topics=5)_____no_output_____

Now that our model is trained, let's see what our results look like.

### Results
Much like LDA, the points of interest would be in what the topics are and how the documents are made up of these topics.
In DTM we have the added interest of seeing how these topics evolve over time.

Let's go through some of the functions to print Topics and analyse documents.

### Printing Topics

To print all topics from a particular time-period, simply use `print_topics`.
The input parameter to `print_topics` is a time-slice option. By passing `0` we are seeing the topics in the 1st time-slice.

The result would be a list of lists, where each individual list contains a tuple of the most probable words in the topic. i.e `(word, word_probability)`_____no_output_____

ldaseq.print_topics(time=0)_____no_output_____

We can see 5 different fairly well defined topics:
1. Economy
2. Entertainment
3. Football
4. Technology and Entertainment
5. Government_____no_output_____### Looking for Topic Evolution

To fix a topic and see it evolve, use `print_topic_times`.
The input parameter is the `topic_id`
In this case, we are looking at the evolution of the technology topic._____no_output_____If you look at the lower frequencies; the word "may" is creeping itself into prominence. This makes sense, as the 3rd time-slice are news reports about the economy in the month of may. This isn't present at all in the first time-slice because it's still March!
_____no_output_____

ldaseq.print_topic_times(topic=0) # evolution of 1st topic_____no_output_____

### Document - Topic Proportions
the function `doc_topics` checks the topic proportions on documents already trained on. It accepts the document number in the corpus as an input.

Let's pick up document number 558 arbitrarily and have a look._____no_output_____

# to check Document - Topic proportions, use `doc-topics`
words = [dictionary[word_id] for word_id, count in corpus[558]]
print (words)['set', 'time,"', 'chairman', 'decision', 'news', 'director', 'former', 'vowed', '"it', 'results', 'club', 'third', 'home', 'paul', 'saturday.', 'south', 'conference', 'leading', '"some', 'survival', 'needed', 'coach', "don't", 'every', 'trouble', 'desperate', 'eight', 'first', 'win', 'going', 'park', 'near', 'chance', 'manager', 'league', 'milan', 'games', 'go', 'game', 'foot', 'say', 'upset', "i'm", 'poor', 'season.', 'executive', 'road', '24', 'debut', 'portsmouth.', 'give', 'claiming', 'steve', 'break', 'rivals', 'boss', 'kevin', 'premiership', 'little', 'left', 'table.', 'life', 'join', 'years.', 'bring', 'season,', 'director.', 'became', 'st', 'according', 'official', 'hope', 'shocked', 'though', 'phone', 'charge', '14', 'website.', 'time,', 'claimed', 'kept', 'bond', 'appointment', 'unveil', 'november', 'picked', 'confirmed,', 'believed', 'deep', 'position', 'surprised', 'negotiations', 'talks', 'gmt', 'middlesbrough', 'replaced', 'appear', 'football,', '"i\'m', 'charge.', 'saints', 'southampton', 'sturrock', 'wednesday.', 'harry', 'poised', 'ninth', 'quit', 'relieved', 'chance."', 'decision."', 'hero', 'redknapp,', 'redknapp', "saints'", 'first-team', "wouldn't", "mary's.", 'portsmouth', "redknapp's", 'pompey', 'academy', "harry's", 'cult', 'rupert', 'time".', 'coast', '57,', 'succeed', 'duties', "'i", 'bitter,', "mandaric's", "portsmouth's", 'wigley,', 'wigley', "southampton',", '1500', 'mandaric', "'absolutely", 'lowe', '"disappointed"', 'velimir', 'not\',"', 'disgusted', 'disappointed,', 'mandaric,', 'fratton', 'replaces', 'masterminding', 'angry,', 'vowed:', 'informed."', 'zajec']

It's pretty clear that it's a news article about football. What topics will it likely be comprised of?_____no_output_____

doc = ldaseq.doc_topics(558) # check the 558th document in the corpuses topic distribution
print (doc)[  5.46298825e-05   5.46298825e-05   9.32273855e-01   6.75622555e-02
   5.46298825e-05]

Let's look at our topics as desribed by us, again:

1. Economy
2. Entertainment
3. Football
4. Technology and Entertainment
5. Government

Our topic distribution for the above document is largely in topics 3 and 4. Considering it is a document about a news article on a football match, the distribution makes perfect sense!_____no_output_____If we wish to analyse a document not in our training set, we can use simply pass the doc to the model similar to the `__getitem__` funciton for `LdaModel`.

Let's let our document be a hypothetical news article about the effects of Ryan Giggs buying mobiles affecting the British economy._____no_output_____

doc_football_1 = ['economy', 'bank', 'mobile', 'phone', 'markets', 'buy', 'football', 'united', 'giggs']
doc_football_1 = dictionary.doc2bow(doc_football_1)
doc_football_1 = ldaseq[doc_football_1]
print (doc_football_1)[ 0.40632917  0.00110497  0.3465658   0.24489509  0.00110497]

Pretty neat! Topic 1 is about the Economy, and this document also has traces of football and technology, so topics 1, 3, and 4 got correctly activated._____no_output_____### Distances between documents

One of the more handy uses of DTMs topic modelling is that we can compare documents across different time-frames and see how similar they are topic-wise. When words may not necessarily overlap over these time-periods, this is very useful.

The current dataset doesn't provide us the diversity for this to be an effective example; but we will nevertheless illustrate how to do the same.

_____no_output_____

doc_football_2 = ['arsenal', 'fourth', 'wenger', 'oil', 'middle', 'east', 'sanction', 'fluctuation']
doc_football_2 = dictionary.doc2bow(doc_football_2)
doc_football_2 = ldaseq[doc_football_2]_____no_output_____hellinger(doc_football_1, doc_football_2)_____no_output_____

The topic distributions are quite related - matches well in football and economy._____no_output_____Now let's try with documents that shouldn't be similar._____no_output_____

doc_governemt_1 = ['tony', 'government', 'house', 'party', 'vote', 'european', 'official', 'house']
doc_governemt_1 = dictionary.doc2bow(doc_governemt_1)
doc_governemt_1 = ldaseq[doc_governemt_1]

hellinger(doc_football_1, doc_governemt_1)_____no_output_____

As expected, the value is very high, meaning the topic distributions are far apart.

For more information on how to use the gensim distance metrics, check out [this notebook](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/distance_metrics.ipynb)._____no_output_____### Performance

The code currently runs between 5 to 7 times slower than the original C++ DTM code. The bottleneck is in the scipy `optimize.fmin_cg` method for updating obs. Speeding this up would fix things up!

Since it uses iterable gensim corpuses, the memory stamp is also cleaner.

TODO: check memory, check BLAS, see how performance can be improved memory and speed wise. _____no_output_____The advantages of the python port are that unlike the C++ code we needn't treat it like a black-box; PRs to help make the code better are welcomed, as well as help to make the documentation clearer and improve performance. It is also in pure python and doesn't need any dependancy outside of what gensim already needs. The added functionality of being able to analyse new documents is also a plus!

## Choosing your best Dynamic Topic Model.

Like we've been going on and on before, the advantage in having a python port is the transparency with which you can train your DTM.
We'll go over two key ideas: changing variance, and changing suff stats.

### Chain Variance
One of the key aspects of topic evolution is how fast/slow these topics evolve. And this is where the factor of `variance` comes in.
By setting the `chain_variance` input to the DTM model higher, we can tweak our topic evolution.
The default value is 0.005. (this is the value suggested by Blei in his tech talk and is the default value in the C++ code)

Let us see a small example illustrating the same.
Let's first see the evolution of values for the first time-slice._____no_output_____

ldaseq.print_topic_times(1)_____no_output_____

Now let us do the same, but after increasing the `chain_variance` value._____no_output_____

ldaseq_chain = ldaseqmodel.LdaSeqModel(corpus=corpus, id2word=dictionary, time_slice=time_slice, num_topics=5, chain_variance=0.05)/Users/bhargavvader/Open_Source/gensim/gensim/models/ldaseqmodel.py:217: RuntimeWarning: divide by zero encountered in double_scalars
  convergence = numpy.fabs((bound - old_bound) / old_bound)

It's noticable that the values are moving more freely after increasing the chain_variance. Film went from highest probability to 5th to 8th!_____no_output_____

ldaseq_chain.print_topic_times(2)_____no_output_____

### LDA Model and DTM

For the first slice of DTM to get setup, we need to provide it sufficient stats from an LDA model. As discussed before, this is done by fitting gensim LDA on the dataset first.

We also, however, have the option of passing our own model or suff stats values. Our final DTM results are heavily influenced by what we pass over here. We already know what a "Good" or "Bad" LDA model is (if not, read about it [here](http://nbviewer.jupyter.org/github/dsquareindia/gensim/blob/280375fe14adea67ce6384ba7eabf362b05e6029/docs/notebooks/topic_coherence_tutorial.ipynb)).

It's quite obvious, then, that by passing a "bad" LDA model we will get not so satisfactory results; and by passing a better fitted or better trained LDA we will get better results. The same logic goes if we wish to directly pass the `suff_stats` numpy matrix._____no_output_____### Visualising Dynamic Topic Models.

Let us use pyLDAvis to visualise both the DTM wrapper and DTM python port.
With the new `DTMvis` methods it is now very straightforward to visualise DTM for a particular time-slice._____no_output_____

from gensim.models.wrappers.dtmmodel import DtmModel
from gensim.corpora import Dictionary, bleicorpus
import pyLDAvis

# dtm_path = "/Users/bhargavvader/Downloads/dtm_release/dtm/main"
# dtm_model = DtmModel(dtm_path, corpus, time_slice, num_topics=5, id2word=dictionary, initialize_lda=True)
# dtm_model.save('dtm_news')

# if we've saved before simply load the model
dtm_model = DtmModel.load('dtm_news')_____no_output_____

If you take some time to look at the topics, you will notice large semantic similarities with the wrapper and the python DTM.
Functionally, the python DTM replicates the wrapper quite well._____no_output_____

doc_topic, topic_term, doc_lengths, term_frequency, vocab = dtm_model.dtm_vis(time=0, corpus=corpus)
vis_wrapper = pyLDAvis.prepare(topic_term_dists=topic_term, doc_topic_dists=doc_topic, doc_lengths=doc_lengths, vocab=vocab, term_frequency=term_frequency)
pyLDAvis.display(vis_wrapper)_____no_output_____

Now let us do the same for the python DTM._____no_output_____In particular, look at topic 2 of the wrapper and topic 3 of the python port. Notice how they are both about football._____no_output_____

doc_topic, topic_term, doc_lengths, term_frequency, vocab = ldaseq.dtm_vis(time=0, corpus=corpus)
vis_dtm = pyLDAvis.prepare(topic_term_dists=topic_term, doc_topic_dists=doc_topic, doc_lengths=doc_lengths, vocab=vocab, term_frequency=term_frequency)
pyLDAvis.display(vis_dtm)_____no_output_____

Visualising topics is a handy way to compare topic models._____no_output_____### Topic Coherence for DTM

Similar to visualising DTM time-slices, finding coherence values for both python DTM and the wrapper is very easy.
We just have to specify the time-slice we want to find coherence for.
The following examples will illustrate this._____no_output_____

from gensim.models.coherencemodel import CoherenceModel
import pickle

# we just have to specify the time-slice we want to find coherence for.
topics_wrapper = dtm_model.dtm_coherence(time=0)
topics_dtm = ldaseq.dtm_coherence(time=2)

# running u_mass coherence on our models
cm_wrapper = CoherenceModel(topics=topics_wrapper, corpus=corpus, dictionary=dictionary, coherence='u_mass')
cm_DTM = CoherenceModel(topics=topics_dtm, corpus=corpus, dictionary=dictionary, coherence='u_mass')

print ("U_mass topic coherence")
print ("Wrapper coherence is ", cm_wrapper.get_coherence())
print ("DTM Python coherence is", cm_DTM.get_coherence())

# to use 'c_v' we need texts, which we have saved to disk.
texts = pickle.load(open('Corpus/texts', 'rb'))
cm_wrapper = CoherenceModel(topics=topics_wrapper, texts=texts, dictionary=dictionary, coherence='c_v')
cm_DTM = CoherenceModel(topics=topics_dtm, texts=texts, dictionary=dictionary, coherence='c_v')

print ("C_v topic coherence")
print ("Wrapper coherence is ", cm_wrapper.get_coherence())
print ("DTM Python coherence is", cm_DTM.get_coherence())U_mass topic coherence
Wrapper coherence is  -1.8017627313
DTM Python coherence is -1.80238943495
C_v topic coherence
Wrapper coherence is  0.647868838777
DTM Python coherence is 0.621369722784

Our values are a little behind the wrapper - but not by much. _____no_output_____## Conclusion

So while there is already a python wrapper of DTM, a pure python implementation will be useful to better understand what goes on undcer the hood and tune our model. When it comes to performance, the C++ is undoubtedly faster, but we can continue to work on ours to make it as fast.
As for evaluating the results, our topics are on par if not better than the wrapper!

On a more personal note, implementing Dynamic Topic Models with the Google Summer of Code 2016 program was a great learning experience. Gensim and RaRe Technologies have been a joy to work with, and Lev and Radim have been great mentors throughout, especially when things became slow or difficult.
I look forward to continuing my contribution to gensim for a long time!_____no_output_____


# Notebook from JeanFraga/DS-Unit-4-Sprint-3-Deep-Learning
Path: module2-convolutional-neural-networks/LS_DS_432_Convolutional_Neural_Networks_Lecture.ipynb

Lambda School Data Science

*Unit 4, Sprint 3, Module 2*

---_____no_output_____# Convolutional Neural Networks (Prepare)

> Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers. *Goodfellow, et al.*_____no_output_____## Learning Objectives
- Part 1: Describe convolution and pooling
- Part 2: Apply a convolutional neural network to a classification task
- Part 3: Use a pre-trained convolution neural network for object detection

Modern __computer vision__ approaches rely heavily on convolutions as both a dimensinoality reduction and feature extraction method. Before we dive into convolutions, let's talk about some of the common computer vision applications:
* Classification [(Hot Dog or Not Dog)](https://www.youtube.com/watch?v=ACmydtFDTGs)
* Object Detection [(YOLO)](https://www.youtube.com/watch?v=MPU2HistivI)
* Pose Estimation [(PoseNet)](https://ai.googleblog.com/2019/08/on-device-real-time-hand-tracking-with.html)
* Facial Recognition [Emotion Detection](https://www.cbronline.com/wp-content/uploads/2018/05/Mona-lIsa-test-570x300.jpg)
* and *countless* more

We are going to focus on classification and pre-trained object detection today. What are some of the applications of object detection?_____no_output_____

from IPython.display import YouTubeVideo
YouTubeVideo('MPU2HistivI', width=600, height=400)_____no_output_____

# Convolution & Pooling (Learn)
_____no_output_____## Overview

Like neural networks themselves, CNNs are inspired by biology - specifically, the receptive fields of the visual cortex.

Put roughly, in a real brain the neurons in the visual cortex *specialize* to be receptive to certain regions, shapes, colors, orientations, and other common visual features. In a sense, the very structure of our cognitive system transforms raw visual input, and sends it to neurons that specialize in handling particular subsets of it.

CNNs imitate this approach by applying a convolution. A convolution is an operation on two functions that produces a third function, showing how one function modifies another. Convolutions have a [variety of nice mathematical properties](https://en.wikipedia.org/wiki/Convolution#Properties) - commutativity, associativity, distributivity, and more. Applying a convolution effectively transforms the "shape" of the input.

One common confusion - the term "convolution" is used to refer to both the process of computing the third (joint) function and the process of applying it. In our context, it's more useful to think of it as an application, again loosely analogous to the mapping from visual field to receptive areas of the cortex in a real animal._____no_output_____

from IPython.display import YouTubeVideo
YouTubeVideo('IOHayh06LJ4', width=600, height=400)_____no_output_____

## Follow Along

Let's try to do some convolutions in `Keras`._____no_output_____### Convolution - an example

Consider blurring an image - assume the image is represented as a matrix of numbers, where each number corresponds to the color value of a pixel._____no_output_____

import skimage
skimage.__version_______no_output_____import imageio
import matplotlib.pyplot as plt
from skimage import color, io
from skimage.exposure import rescale_intensity

austen = io.imread('https://dl.airtable.com/S1InFmIhQBypHBL0BICi_austen.jpg')
austen_grayscale = rescale_intensity(color.rgb2gray(austen))
austen_grayscale_____no_output_____plt.imshow(austen_grayscale, cmap="gray");_____no_output_____austen_grayscale.shape_____no_output_____import numpy as np
import scipy.ndimage as nd

horizontal_edge_convolution = np.array([[1,1,1,1,1],
                                     [0,0,0,0,0],
                                     [0,0,0,0,0],
                                     [0,0,0,0,0],
                                     [-1,-1,-1,-1,-1]])

vertical_edge_convolution = np.array([[1, 0, 0, 0, -1],
                                     [1, 0, 0, 0, -1],
                                     [1, 0, 0, 0, -1],
                                     [1, 0, 0, 0, -1],
                                     [1, 0, 0, 0, -1]])

austen_edges = nd.convolve(austen_grayscale, horizontal_edge_convolution)
austen_edges.shape_____no_output_____plt.imshow(austen_edges, cmap="gray");_____no_output_____

## Challenge

You will be expected to be able to describe convolution. _____no_output_____# CNNs for Classification (Learn)_____no_output_____## Overview_____no_output_____### Typical CNN Architecture

![A Typical CNN](https://upload.wikimedia.org/wikipedia/commons/thumb/6/63/Typical_cnn.png/800px-Typical_cnn.png)

The first stage of a CNN is, unsurprisingly, a convolution - specifically, a transformation that maps regions of the input image to neurons responsible for receiving them. The convolutional layer can be visualized as follows:

![Convolutional layer](https://upload.wikimedia.org/wikipedia/commons/6/68/Conv_layer.png)

The red represents the original input image, and the blue the neurons that correspond.

As shown in the first image, a CNN can have multiple rounds of convolutions, [downsampling](https://en.wikipedia.org/wiki/Downsampling_(signal_processing)) (a digital signal processing technique that effectively reduces the information by passing through a filter), and then eventually a fully connected neural network and output layer. Typical output layers for a CNN would be oriented towards classification or detection problems - e.g. "does this picture contain a cat, a dog, or some other animal?"

Why are CNNs so popular?
1. Compared to prior image learning techniques, they require relatively little image preprocessing (cropping/centering, normalizing, etc.)
2. Relatedly, they are *robust* to all sorts of common problems in images (shifts, lighting, etc.)

Actually training a cutting edge image classification CNN is nontrivial computationally - the good news is, with transfer learning, we can get one "off-the-shelf"!_____no_output_____## Follow Along_____no_output_____

from tensorflow.keras import datasets
from tensorflow.keras.models import Sequential, Model # <- May Use
from tensorflow.keras.layers import Dense, Conv2D, MaxPooling2D, Flatten_____no_output_____(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()

# Normalize pixel values to be between 0 and 1
train_images, test_images = train_images / 255.0, test_images / 255.0_____no_output_____class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck']

plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(train_images[i], cmap=plt.cm.binary)
    # The CIFAR labels happen to be arrays,
    # which is why you need the extra index
    plt.xlabel(class_names[train_labels[i][0]])
plt.show()_____no_output_____train_images[0].shape_____no_output_____32*32*3_____no_output_____# Setup Architecture

model = Sequential()
model.add(Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(MaxPooling2D((2,2)))
model.add(Conv2D(64, (3,3), activation='relu'))
model.add(MaxPooling2D((2,2)))
model.add(Conv2D(64, (3,3), activation='relu'))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dense(10, activation='softmax'))

model.summary()Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_3 (Conv2D)            (None, 30, 30, 32)        896
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 15, 15, 32)        0
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 13, 13, 64)        18496
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 6, 6, 64)          0
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 4, 4, 64)          36928
_________________________________________________________________
flatten_1 (Flatten)          (None, 1024)              0
_________________________________________________________________
dense_2 (Dense)              (None, 64)                65600
_________________________________________________________________
dense_3 (Dense)              (None, 10)                650
=================================================================
Total params: 122,570
Trainable params: 122,570
Non-trainable params: 0
_________________________________________________________________
# Compile Model

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])_____no_output_____# Fit Model
model.fit(train_images, train_labels, epochs=15, validation_data=(test_images, test_labels))Train on 50000 samples, validate on 10000 samples
Epoch 1/15
50000/50000 [==============================] - 5s 109us/sample - loss: 1.4913 - accuracy: 0.4596 - val_loss: 1.2563 - val_accuracy: 0.5481
Epoch 2/15
50000/50000 [==============================] - 5s 97us/sample - loss: 1.1396 - accuracy: 0.5943 - val_loss: 1.1478 - val_accuracy: 0.5931
Epoch 3/15
50000/50000 [==============================] - 5s 97us/sample - loss: 1.0038 - accuracy: 0.6484 - val_loss: 0.9749 - val_accuracy: 0.6596
Epoch 4/15
50000/50000 [==============================] - 5s 97us/sample - loss: 0.9157 - accuracy: 0.6787 - val_loss: 0.9727 - val_accuracy: 0.6655
Epoch 5/15
50000/50000 [==============================] - 5s 97us/sample - loss: 0.8481 - accuracy: 0.7050 - val_loss: 0.9690 - val_accuracy: 0.6672
Epoch 6/15
50000/50000 [==============================] - 5s 96us/sample - loss: 0.7886 - accuracy: 0.7256 - val_loss: 0.9061 - val_accuracy: 0.6884
Epoch 7/15
50000/50000 [==============================] - 5s 99us/sample - loss: 0.7420 - accuracy: 0.7420 - val_loss: 0.8849 - val_accuracy: 0.6921
Epoch 8/15
50000/50000 [==============================] - 5s 100us/sample - loss: 0.6980 - accuracy: 0.7545 - val_loss: 0.8791 - val_accuracy: 0.7034
Epoch 9/15
50000/50000 [==============================] - 5s 100us/sample - loss: 0.6590 - accuracy: 0.7717 - val_loss: 0.9109 - val_accuracy: 0.6924
Epoch 10/15
50000/50000 [==============================] - 5s 104us/sample - loss: 0.6235 - accuracy: 0.7806 - val_loss: 0.8994 - val_accuracy: 0.7044
Epoch 11/15
50000/50000 [==============================] - 6s 111us/sample - loss: 0.5892 - accuracy: 0.7941 - val_loss: 0.9460 - val_accuracy: 0.6970
Epoch 12/15
50000/50000 [==============================] - 5s 98us/sample - loss: 0.5622 - accuracy: 0.8004 - val_loss: 0.9010 - val_accuracy: 0.7157
Epoch 13/15
50000/50000 [==============================] - 5s 99us/sample - loss: 0.5242 - accuracy: 0.8151 - val_loss: 0.9672 - val_accuracy: 0.6920
Epoch 14/15
50000/50000 [==============================] - 5s 99us/sample - loss: 0.5003 - accuracy: 0.8239 - val_loss: 0.9560 - val_accuracy: 0.7046
Epoch 15/15
50000/50000 [==============================] - 5s 97us/sample - loss: 0.4710 - accuracy: 0.8332 - val_loss: 0.9799 - val_accuracy: 0.7032
# Evaluate Model

test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)10000/1 - 1s - loss: 0.8886 - accuracy: 0.7130

## Challenge

You will apply CNNs to a classification task in the module project._____no_output_____# CNNs for Object Detection (Learn)_____no_output_____## Overview_____no_output_____### Transfer Learning - TensorFlow Hub

"A library for reusable machine learning modules"

This lets you quickly take advantage of a model that was trained with thousands of GPU hours. It also enables transfer learning - reusing a part of a trained model (called a module) that includes weights and assets, but also training the overall model some yourself with your own data. The advantages are fairly clear - you can use less training data, have faster training, and have a model that generalizes better.

https://www.tensorflow.org/hub/

**WARNING** - Dragons ahead!

![Dragon](https://upload.wikimedia.org/wikipedia/commons/thumb/d/d8/Friedrich-Johann-Justin-Bertuch_Mythical-Creature-Dragon_1806.jpg/637px-Friedrich-Johann-Justin-Bertuch_Mythical-Creature-Dragon_1806.jpg)

TensorFlow Hub is very bleeding edge, and while there's a good amount of documentation out there, it's not always updated or consistent. You'll have to use your problem-solving skills if you want to use it!_____no_output_____## Follow Along_____no_output_____

import numpy as np

from tensorflow.keras.applications.resnet50 import ResNet50
from tensorflow.keras.preprocessing import image
from tensorflow.keras.applications.resnet50 import preprocess_input, decode_predictions

def process_img_path(img_path):
  return image.load_img(img_path, target_size=(224, 224))

def img_contains_banana(img):
  x = image.img_to_array(img)
  x = np.expand_dims(x, axis=0)
  x = preprocess_input(x)
  model = ResNet50(weights='imagenet')
  features = model.predict(x)
  results = decode_predictions(features, top=3)[0]
  print(results)
  for entry in results:
    if entry[1] == 'banana':
      return entry[2]
  return 0.0_____no_output_____import requests

image_urls = ["https://github.com/LambdaSchool/ML-YouOnlyLookOnce/raw/master/sample_data/negative_examples/example11.jpeg",
              "https://github.com/LambdaSchool/ML-YouOnlyLookOnce/raw/master/sample_data/positive_examples/example0.jpeg"]

for _id,img in enumerate(image_urls):
    r = requests.get(img)
    with open(f'example{_id}.jpg', 'wb') as f:
        f.write(r.content)_____no_output_____from IPython.display import Image
Image(filename='./example0.jpg', width=600)_____no_output_____img_contains_banana(process_img_path('example0.jpg'))[('n04037443', 'racer', 0.93360406), ('n04285008', 'sports_car', 0.06275203), ('n04461696', 'tow_truck', 0.0019114019)]
Image(filename='example1.jpg', width=600)_____no_output_____img_contains_banana(process_img_path('example1.jpg'))[('n03532672', 'hook', 0.07862277), ('n07753592', 'banana', 0.07254732), ('n03498962', 'hatchet', 0.06386116)]

Notice that, while it gets it right, the confidence for the banana image is fairly low. That's because so much of the image is "not-banana"! How can this be improved? Bounding boxes to center on items of interest._____no_output_____## Challenge

You will be expected to apply a pretrained model to a classificaiton problem today. _____no_output_____# Review

- Part 1: Describe convolution and pooling
    * A Convolution is a function applied to another function to produce a third function
    * Convolutional Kernels are typically 'learned' during the process of training a Convolution Neural Network
    * Pooling is a dimensionality reduction technique that uses either Max or Average of a feature map region to downsample data
- Part 2: Apply a convolutional neural network to a classification task
    * Keras has layers for convolutions :)
- Part 3: Use a pre-trained convolution neural network for object detection
    * Check out both pretinaed models available in Keras & TensorFlow Hub_____no_output_____# Sources

- *_Deep Learning_*. Goodfellow *et al.*
- [Keras CNN Tutorial](https://www.tensorflow.org/tutorials/images/cnn)
- [Tensorflow + Keras](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D)
- [Convolution Wiki](https://en.wikipedia.org/wiki/Convolution)
- [Keras Conv2D: Working with CNN 2D Convolutions in Keras](https://missinglink.ai/guides/keras/keras-conv2d-working-cnn-2d-convolutions-keras/)_____no_output_____


# Notebook from mtxslv/AprendendoMachineLearning
Path: MIT6.S191/lab2/Copy_of_Part1_mnist.ipynb

      Visit MIT Deep Learning

        Run in Google Colab

        View Source on GitHub

# Laboratory 2: Computer Vision

# Part 1: MNIST Digit Classification

In the first portion of this lab, we will build and train a convolutional neural network (CNN) for classification of handwritten digits from the famous [MNIST](http://yann.lecun.com/exdb/mnist/) dataset. The MNIST dataset consists of 60,000 training images and 10,000 test images. Our classes are the digits 0-9.

First we'll import TensorFlow, enable Eager execution, and also import some dependencies._____no_output_____

import tensorflow as tf
tf.enable_eager_execution()
import matplotlib.pyplot as plt
import numpy as np
import random
from progressbar import progressbar

# Download the class repository
! git clone https://github.com/aamini/introtodeeplearning_labs.git  > /dev/null 2>&1
% cd introtodeeplearning_labs
! git pull
% cd ..

# Download regex
!pip install regex
# Import the necessary class-specific utility files for this lab
import introtodeeplearning_labs as util_____no_output_____

## 1.1 MNIST dataset

Let's download and load the dataset and display a few random samples from it:_____no_output_____

mnist = tf.keras.datasets.mnist
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()
train_images = np.expand_dims(train_images, axis=-1)/255.
train_labels = np.int64(train_labels)
test_images = np.expand_dims(test_images, axis=-1)/255.
test_labels = np.int64(test_labels)Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step
11501568/11490434 [==============================] - 0s 0us/step

Our training set is made up of 28x28 grayscale images of handwritten digits.

Let's visualize what some of these images and their corresponding training labels look like._____no_output_____

plt.figure(figsize=(10,10))
random_inds = np.random.choice(60000,36)
for i in range(36):
    plt.subplot(6,6,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    image_ind = random_inds[i]
    plt.imshow(np.squeeze(train_images[image_ind]), cmap=plt.cm.binary)
    plt.xlabel(train_labels[image_ind])_____no_output_____

## 1.2 Neural Network for Handwritten Digit Classification

We'll first build a simple neural network consisting of two fully connected layers and apply this to the digit classification task. Our network will ultimately output a probability distribution over the 10 digit classes (0-9). This first architecture we will be building is depicted below:

![alt text](https://raw.githubusercontent.com/aamini/introtodeeplearning_labs/master/lab2/img/mnist_2layers_arch.png)
_____no_output_____### Fully connected neural network architecture
To define the architecture of this first fully connected neural network, we'll once again use the Keras API and define the model using the [`Sequential`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential) class. Note how we first use a [`Flatten`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Flatten) layer, which flattens the input so that it can be fed into the model.

In this next block, you'll define the output layer -- the second fully connected of this simple network._____no_output_____

def build_fc_model():
  fc_model = tf.keras.Sequential([
      # First define a Flatten layer
      tf.keras.layers.Flatten(),
      # '''TODO: Define the activation function for the first fully connected layer.'''
      tf.keras.layers.Dense(128, activation='relu'), # TODO
      # '''TODO: Define the second Dense layer to output the classification probabilities'''
      tf.keras.layers.Dense(10, activation= tf.keras.activations.softmax) # TODO
  ])
  return fc_model

model = build_fc_model()_____no_output_____

As we progress through this next portion, you may find that you'll want to make changes to the architecture defined above. **Note that in order to update the model later on, you'll need to re-run the above cell to re-initialize the model. **_____no_output_____Let's take a step back and think about the network we've just created. The first layer in this network, `tf.keras.layers.Flatten`, transforms the format of the images from a 2d-array (28 x 28 pixels), to a 1d-array of 28 * 28 = 784 pixels. You can think of this layer as unstacking rows of pixels in the image and lining them up. There are no learned parameters in this layer; it only reformats the data.

After the pixels are flattened, the network consists of a sequence of two `tf.keras.layers.Dense` layers. These are fully-connected neural layers. The first `Dense` layer has 128 nodes (or neurons). The second (and last) layer (which you've defined!) should return an array of probability scores that sum to 1. Each node contains a score that indicates the probability that the current image belongs to one of the handwritten digit classes.

That defines our fully connected model! _____no_output_____

### Compile the model

Before training the model, we need to define a few more settings. These are added during the model's [`compile`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential#compile) step:

* *Loss function* — This defines how we measure how accurate the model is during training. As was covered in lecture, during training we want to minimize this function, which will "steer" the model in the right direction.
* *Optimizer* — This defines how the model is updated based on the data it sees and its loss function.
* *Metrics* — Here we can define metrics used to monitor the training and testing steps. In this example, we'll look at the *accuracy*, the fraction of the images that are correctly classified.

We'll start out by using a stochastic gradient descent (SGD) optimizer initialized with a learning rate of 0.1. Since we are performing a categorical classification task, we'll want to use the [cross entropy loss](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/sparse_categorical_crossentropy).

You'll want to experiment with both the choice of optimizer and learning rate and evaluate how these affect the accuracy of the trained model. _____no_output_____

'''TODO: Experiment with different optimizers and learning rates. How do these affect
    the accuracy of the trained model? Which optimizers and/or learning rates yield
    the best performance?'''
model.compile(optimizer=tf.train.GradientDescentOptimizer(learning_rate=1e-1),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])_____no_output_____

### Train the model

We're now ready to train our model, which will involve feeding the training data (`train_images` and `train_labels`) into the model, and then asking it to learn the associations between images and labels. We'll also need to define the batch size and the number of epochs, or iterations over the MNIST dataset, to use during training. With the Keras API and defining the model settings in the `compile` step, training is all accomplished by calling the [`fit`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential#fit) method on an instance of the Model class.
_____no_output_____

# Define the batch size and the number of epochs to use during training
BATCH_SIZE = 64
EPOCHS = 5_____no_output_____history = model.fit(train_images, train_labels, batch_size=BATCH_SIZE, epochs=EPOCHS)WARNING: Logging before flag parsing goes to stderr.
W0112 14:40:22.579252 140003931154304 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
plt.figure(figsize=(15,5))
plt.title("Statistics")
plt.subplot(1,2,1)
plt.xlabel('EPOCHS x LOSS')
plt.plot(history.history['loss'])
plt.subplot(1,2,2)
plt.xlabel('EPOCHS x ACC')
plt.plot(history.history['acc'])_____no_output_____

As the model trains, the loss and accuracy metrics are displayed. With five epochs and a learning rate of 0.01, this fully connected model should achieve an accuracy of approximatley 0.97 (or 97%) on the training data._____no_output_____### Evaluate accuracy on the test dataset

Now that we've trained the model, we can ask it to make predictions about a test set that it hasn't seen before. In this example, the `test_images` array comprises our test dataset. To evaluate accuracy, we can check to see if the model's predictions match the labels from the `test_labels` array.

Use the [`evaluate`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential#evaluate) method to evaluate the model on the test dataset!_____no_output_____

'''TODO: Use the evaluate method to test the model!'''
test_loss, test_acc = model.evaluate(test_images, test_labels)

print('Test accuracy:', test_acc)10000/10000 [==============================] - 0s 43us/sample - loss: 0.1098 - acc: 0.9677
('Test accuracy:', 0.9677)

You may observe that the accuracy on the test dataset is a little lower than the accuracy on the training dataset. This gap between training accuracy and test accuracy is an example of *overfitting*, when a machine learning model performs worse on new data than on its training data.

What is the highest accuracy you can achieve with this first fully connected model? Since the handwritten digit classification task is pretty straightforward, you may be wondering how we can do better...

![Deeper...](https://i.kym-cdn.com/photos/images/newsfeed/000/534/153/f87.jpg)_____no_output_____## 1.3 Convolutional Neural Network (CNN) for handwritten digit classification_____no_output_____As we saw in lecture, convolutional neural networks (CNNs) are particularly well-suited for a variety of tasks in computer vision, and have achieved near-perfect accuracies on the MNIST dataset. We will now build a CNN composed of two convolutional layers and pooling layers, followed by two fully connected layers, and ultimately output a probability distribution over the 10 digit classes (0-9). The CNN we will be building is depicted below:

![alt text](https://raw.githubusercontent.com/aamini/introtodeeplearning_labs/master/lab2/img/mnist_model.png)_____no_output_____### Define the CNN model

We'll use the same training and test datasets as before, and proceed similarly as our fully connected network to define and train our new CNN model.

You can use  [`keras.layers.Conv2D` ](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D) to define convolutional layers and [`keras.layers.MaxPool2D`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/MaxPool2D) to define the pooling layers. Use the parameters shown in the network architecture above to define these layers and build the CNN model!_____no_output_____

def build_cnn_model():
    cnn_model = tf.keras.Sequential([

        tf.keras.layers.Conv2D(filters=32, kernel_size=(5,5), input_shape=(28, 28, 1), activation=tf.nn.relu), # TODO

        tf.keras.layers.MaxPool2D(pool_size=(2,2)),

        #'''TODO: Define the second convolutional layer'''
        tf.keras.layers.Conv2D(filters=48,kernel_size=(5,5), activation=tf.nn.relu), # TODO

        #'''TODO: Define the second max pooling layer'''
        tf.keras.layers.MaxPool2D(pool_size=(2,2)), # TODO

        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(128, activation=tf.nn.relu),
        #'''TODO: Define the last Dense layer'''
        tf.keras.layers.Dense(10, activation= tf.keras.activations.softmax) # TODO
    ])
    return cnn_model

cnn_model = build_cnn_model()
print(cnn_model.summary())Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (Conv2D)              (None, 24, 24, 32)        832
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 12, 12, 32)        0
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 8, 8, 48)          38448
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 4, 4, 48)          0
_________________________________________________________________
flatten_1 (Flatten)          (None, 768)               0
_________________________________________________________________
dense_2 (Dense)              (None, 128)               98432
_________________________________________________________________
dense_3 (Dense)              (None, 10)                1290
=================================================================
Total params: 139,002
Trainable params: 139,002
Non-trainable params: 0
_________________________________________________________________
None

### Train and test the CNN model

Now, as before, we can define the loss function, optimizer, and metrics through the `compile` method. Compile the CNN model with an optimizer and learning rate of choice:_____no_output_____

'''TODO: Define the compile operation with your optimizer and learning rate of choice.
We recommend beginning with the same optimizer and learning rate as the feed forward model'''
cnn_model.compile(optimizer=tf.train.GradientDescentOptimizer(learning_rate=1e-1), # TODO
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])_____no_output_____

Now we can train our model using the `fit` method via the Keras API:_____no_output_____

'''TODO: Train the CNN model'''
history_cnn = cnn_model.fit(train_images,train_labels,batch_size=BATCH_SIZE, epochs=EPOCHS) # TODOTrain on 60000 samples
Epoch 1/5
60000/60000 [==============================] - 7s 114us/sample - loss: 0.2147 - acc: 0.9319
Epoch 2/5
60000/60000 [==============================] - 2s 39us/sample - loss: 0.0594 - acc: 0.9815
Epoch 3/5
60000/60000 [==============================] - 2s 38us/sample - loss: 0.0414 - acc: 0.9872
Epoch 4/5
60000/60000 [==============================] - 2s 38us/sample - loss: 0.0312 - acc: 0.9898
Epoch 5/5
60000/60000 [==============================] - 2s 38us/sample - loss: 0.0251 - acc: 0.9919
plt.figure(figsize=(15,5))
plt.title("Statistics")
plt.subplot(1,2,1)
plt.xlabel('EPOCHS x LOSS')
plt.plot(history_cnn.history['loss'])
plt.subplot(1,2,2)
plt.xlabel('EPOCHS x ACC')
plt.plot(history_cnn.history['acc'])_____no_output_____

Great! Now that we've trained the model, let's evaluate it on the test dataset using the [`evaluate`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential#evaluate) method:_____no_output_____

'''TODO: Use the evaluate method to test the model!'''
test_loss, test_acc = cnn_model.evaluate(test_images,test_labels) # TODO

print('Test accuracy:', test_acc)10000/10000 [==============================] - 1s 54us/sample - loss: 0.0345 - acc: 0.9884
('Test accuracy:', 0.9884)

What is the highest accuracy you're able to achieve using the CNN model, and how does the accuracy of the CNN model compare to the accuracy of the simple fully connected network? What optimizers and learning rates seem to be optimal for training the CNN model? _____no_output_____### Make predictions with the CNN model

With the model trained, we can use it to make predictions about some images. We'll use the [`predict`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential#predict) function call to generate the output predictions given a set of input samples.
_____no_output_____

predictions = cnn_model.predict(test_images)_____no_output_____

With this function call, the model has predicted the label for each image in the testing set. Let's take a look at the prediction for the first image in the test dataset:_____no_output_____

predictions[0]_____no_output_____

As you can see, a prediction is an array of 10 numbers. Recall that the output of our model is a probability distribution over the 10 digit classes. Thus, these numbers describe the model's "confidence" that the image corresponds to each of the 10 different digits.

Let's look at the digit that has the highest confidence for the first image in the test dataset:_____no_output_____

'''TODO: identify the digit with the highest confidence prediction for the first
    image in the test dataset'''
digit = np.argmax(predictions[0])
digit_____no_output_____

So, the model is most confident that this image is a "7". We can check the test label (remember, this is the true identity of the digit) to see if this prediction is correct:_____no_output_____

test_labels[0]_____no_output_____

It is!

We can define a couple of functions to help visualize the classification results on the MNIST dataset. First, we'll write a function, `plot_image`, to plot images along with their predicted label and the probability of the prediction. Second, we'll also define a function, `plot_value_array`, to plot the prediction probabilities for each of the digits. _____no_output_____

def plot_image(i, predictions_array, true_label, img):
  predictions_array, true_label, img = predictions_array[i], true_label[i], img[i]
  plt.grid(False)
  plt.xticks([])
  plt.yticks([])

  plt.imshow(np.squeeze(img), cmap=plt.cm.binary)

  predicted_label = np.argmax(predictions_array)
  if predicted_label == true_label:
    color = 'blue'
  else:
    color = 'red'

  plt.xlabel("{} {:2.0f}% ({})".format(predicted_label,
                                100*np.max(predictions_array),
                                true_label),
                                color=color)

def plot_value_array(i, predictions_array, true_label):
  predictions_array, true_label = predictions_array[i], true_label[i]
  plt.grid(False)
  plt.xticks([])
  plt.yticks([])
  thisplot = plt.bar(range(10), predictions_array, color="#777777")
  plt.ylim([0, 1])
  predicted_label = np.argmax(predictions_array)

  thisplot[predicted_label].set_color('red')
  thisplot[true_label].set_color('blue')_____no_output_____

Let's use these functions to visualize the model's predictions for the images in the test dataset: _____no_output_____

#@title Change the slider to look at the model's predictions! { run: "auto" }

image_index = 100 #@param {type:"slider", min:0, max:100, step:1}
plt.subplot(1,2,1)
plot_image(image_index, predictions, test_labels, test_images)
plt.subplot(1,2,2)
plot_value_array(image_index, predictions,  test_labels)_____no_output_____

We can also plot several images along with their predictions, where correct prediction labels are blue and incorrect prediction labels are red. The number gives the percent confidence (out of 100) for the predicted label. Note the model can be very confident in an incorrect prediction!_____no_output_____

# Plot the first X test images, their predicted label, and the true label
# Color correct predictions in blue, incorrect predictions in red
num_rows = 5
num_cols = 4
num_images = num_rows*num_cols
plt.figure(figsize=(2*2*num_cols, 2*num_rows))
for i in range(num_images):
  plt.subplot(num_rows, 2*num_cols, 2*i+1)
  plot_image(i, predictions, test_labels, test_images)
  plt.subplot(num_rows, 2*num_cols, 2*i+2)
  plot_value_array(i, predictions, test_labels)
_____no_output_____

## 1.4 Training the model 2.0

Earlier in the lab, we used the [`fit`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential#fit) function call to train the model. This function is quite high-level and intuitive, which is really useful for simpler models. As you may be able to tell, this function abstracts away many details in the training call, and we have less control over training model, which could be useful in other contexts.

As an alternative to this, we can use the [`tf.GradientTape`](https://www.tensorflow.org/api_docs/python/tf/GradientTape) class to record differentiation operations during training, and then call the [`tf.GradientTape.gradient`](https://www.tensorflow.org/api_docs/python/tf/GradientTape#gradient) function to actually compute the gradients.

You may recall seeing this in Lab 1 Part 1, but let's take another look at this here.

We'll use this framework to train our `cnn_model` using stochastic gradient descent._____no_output_____

# Rebuild the CNN model
cnn_model = build_cnn_model()

batch_size = 12
loss_history = util.LossHistory(smoothing_factor=0.99) # to record the evolution of the loss
plotter = util.PeriodicPlotter(sec=2, xlabel='Iterations', ylabel='Loss', scale='semilogy')
optimizer = tf.train.GradientDescentOptimizer(learning_rate=1e-2) # define our optimizer

bar = util.create_progress_bar()
for idx in bar(range(0, train_images.shape[0],batch_size)):
  # First grab a batch of training data and convert the input images to tensors
  (images, labels) = (train_images[idx:idx+batch_size], train_labels[idx:idx+batch_size])
  images = tf.convert_to_tensor(images, dtype=tf.float32)

  # GradientTape to record differentiation operations
  with tf.GradientTape() as tape:
    logits = cnn_model(images) # feed the images into the model
    loss_value = tf.keras.backend.sparse_categorical_crossentropy(labels, logits) # value of the loss

  loss_history.append(loss_value.numpy().mean()) # append the loss to the loss_history record
  plotter.plot(loss_history.get())
  # Backpropagation
  grads = tape.gradient(loss_value, cnn_model.variables)
  optimizer.apply_gradients(zip(grads, cnn_model.variables),
                            global_step=tf.train.get_or_create_global_step())
_____no_output_____

## 1.5 Conclusion
In this part of the lab, you had the chance to play with different MNIST classifiers with different architectures (fully-connected layers only, CNN), and experiment with how different hyperparameters affect accuracy (learning rate, etc.). The next part of the lab explores another application of CNNs, facial detection, and some drawbacks of AI systems in real world applications, like issues of bias. _____no_output_____


# Notebook from hansuribe/graficaspython2
Path: mpld3-notebook.ipynb

# mpld3 notebook demo
Programmer's club 28th September 2017_____no_output_____

import numpy as np
import pandas as pd
import matplotlib
# matplotlib.use('Agg')

import matplotlib.pyplot as plt
import mpld3_____no_output_____x1, x2, y1, y2, z = 'teff', 'mass', 'Vmag', 'par', 'logg'

df = pd.read_table('stars.csv')
columns = df.columns.values
df = df.loc[:, list(set(['Star', x1, x2, y1, y2, z]))].dropna(axis=0)
_____no_output_____

### Standard matplotlib plot_____no_output_____

fig, ax = plt.subplots(2, 2, figsize=(10, 8), sharex='col', sharey='row')
points = ax[0, 0].scatter(df[x1], df[y1], c=df[z], alpha=0.6)
points = ax[1, 0].scatter(df[x1], df[y2], c=df[z], alpha=0.6)
points = ax[0, 1].scatter(df[x2], df[y1], c=df[z], alpha=0.6)
points = ax[1, 1].scatter(df[x2], df[y2], c=df[z], alpha=0.6)
ax[1, 0].set_xlabel(x1)
ax[1, 1].set_xlabel(x2)
ax[0, 0].set_ylabel(y1)
ax[1, 0].set_ylabel(y2)

plt.show()
_____no_output_____

### Using mpld3 for display_____no_output_____

fig, ax = plt.subplots(2, 2, figsize=(10, 8), sharex='col', sharey='row')
points = ax[0, 0].scatter(df[x1], df[y1], c=df[z], alpha=0.6)
points = ax[1, 0].scatter(df[x1], df[y2], c=df[z], alpha=0.6)
points = ax[0, 1].scatter(df[x2], df[y1], c=df[z], alpha=0.6)
points = ax[1, 1].scatter(df[x2], df[y2], c=df[z], alpha=0.6)
ax[1, 0].set_xlabel(x1)
ax[1, 1].set_xlabel(x2)
ax[0, 0].set_ylabel(y1)
ax[1, 0].set_ylabel(y2)

mpld3.display()
_____no_output_____

# Some interactivity_____no_output_____

from mpld3 import plugins

fig, ax = plt.subplots(3, 3, figsize=(6, 6))
fig.subplots_adjust(hspace=0.1, wspace=0.1)
ax = ax[::-1]

fig, ax = plt.subplots(2, 2, figsize=(10, 8), sharex='col', sharey='row')
points = ax[0, 0].scatter(df[x1], df[y1], c=df[z], alpha=0.6)
points = ax[1, 0].scatter(df[x1], df[y2], c=df[z], alpha=0.6)
points = ax[0, 1].scatter(df[x2], df[y1], c=df[z], alpha=0.6)
points = ax[1, 1].scatter(df[x2], df[y2], c=df[z], alpha=0.6)

plugins.connect(fig, plugins.LinkedBrush(points))
mpld3.display()_____no_output_____


# Notebook from SuryaPradeepM/Advanced-ML-Specialization-HSE-Coursera
Path: Intro to Deep Learning/Week 5/POS_task.ipynb

# set tf 1.x for colab
%tensorflow_version 1.x

! wget https://raw.githubusercontent.com/hse-aml/intro-to-dl/master/setup_google_colab.py -O setup_google_colab.py
import setup_google_colab
setup_google_colab.setup_week5()--2020-07-15 12:00:46--  https://raw.githubusercontent.com/hse-aml/intro-to-dl/master/setup_google_colab.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3636 (3.6K) [text/plain]
Saving to: ‘setup_google_colab.py’

setup_google_colab.   0%[                    ]       0  --.-KB/s
setup_google_colab. 100%[===================>]   3.55K  --.-KB/s    in 0s

2020-07-15 12:00:47 (39.5 MB/s) - ‘setup_google_colab.py’ saved [3636/3636]

__This seminar:__ after you're done coding your own recurrent cells, it's time you learn how to train recurrent networks easily with Keras. We'll also learn some tricks on how to use keras layers and model. We also want you to note that this is a non-graded assignment, meaning you are not required to pass it for a certificate.

Enough beatin' around the bush, let's get to the task!_____no_output_____## Part Of Speech Tagging

Unlike our previous experience with language modelling, this time around we learn the mapping between two different kinds of elements.

This setting is common for a range of useful problems:
* Speech Recognition - processing human voice into text
* Part Of Speech Tagging - for morphology-aware search and as an auxuliary task for most NLP problems
* Named Entity Recognition - for chat bots and web crawlers
* Protein structure prediction - for bioinformatics

Our current guest is part-of-speech tagging. As the name suggests, it's all about converting a sequence of words into a sequence of part-of-speech tags. We'll use a reduced tag set for simplicity:

### POS-tags
- ADJ - adjective (new, good, high, ...)
- ADP - adposition	(on, of, at, ...)
- ADV - adverb	(really, already, still, ...)
- CONJ	- conjunction	(and, or, but, ...)
- DET - determiner, article	(the, a, some, ...)
- NOUN	- noun	(year, home, costs, ...)
- NUM - numeral	(twenty-four, fourth, 1991, ...)
- PRT -	particle (at, on, out, ...)
- PRON - pronoun (he, their, her, ...)
- VERB - verb (is, say, told, ...)
- .	- punctuation marks	(. , ;)
- X	- other	(ersatz, esprit, dunno, ...)_____no_output_____

import nltk
import sys
import numpy as np
nltk.download('brown')
nltk.download('universal_tagset')
data = nltk.corpus.brown.tagged_sents(tagset='universal')
all_tags = ['#EOS#','#UNK#','ADV', 'NOUN', 'ADP', 'PRON', 'DET', '.', 'PRT', 'VERB', 'X', 'NUM', 'CONJ', 'ADJ']

data = np.array([ [(word.lower(),tag) for word,tag in sentence] for sentence in data ])[nltk_data] Downloading package brown to /root/nltk_data...
[nltk_data]   Unzipping corpora/brown.zip.
[nltk_data] Downloading package universal_tagset to /root/nltk_data...
[nltk_data]   Unzipping taggers/universal_tagset.zip.
from sklearn.model_selection import train_test_split
train_data,test_data = train_test_split(data,test_size=0.25,random_state=42)
print(f"Training set: {len(train_data)} | Testing set: {len(test_data)}")Training set: 43005 | Testing set: 14335
from IPython.display import HTML, display
def draw(sentence):
    words,tags = zip(*sentence)
    display(HTML('

{tags}
{words}

'.format(
                words = '{}	'.format('	'.join(words)),
                tags = '{}	'.format('	'.join(tags)))))

draw(data[11])
draw(data[10])
draw(data[7])_____no_output_____

### Building vocabularies

Just like before, we have to build a mapping from tokens to integer ids. This time around, our model operates on a word level, processing one word per RNN step. This means we'll have to deal with far larger vocabulary.

Luckily for us, we only receive those words as input i.e. we don't have to predict them. This means we can have a large vocabulary for free by using word embeddings._____no_output_____

from collections import Counter
word_counts = Counter()
for sentence in data:
    words,tags = zip(*sentence)
    word_counts.update(words)

all_words = ['#EOS#','#UNK#']+list(list(zip(*word_counts.most_common(10000)))[0])

#let's measure what fraction of data words are in the dictionary
print("Coverage = %.5f"%(float(sum(word_counts[w] for w in all_words)) / sum(word_counts.values())))Coverage = 0.92876
from collections import defaultdict
word_to_id = defaultdict(lambda:1,{word:i for i,word in enumerate(all_words)})
tag_to_id = {tag:i for i,tag in enumerate(all_tags)}_____no_output_____

convert words and tags into fixed-size matrix_____no_output_____

def to_matrix(lines,token_to_id,max_len=None,pad=0,dtype='int32',time_major=False):
    """Converts a list of names into rnn-digestable matrix with paddings added after the end"""

    max_len = max_len or max(map(len,lines))
    matrix = np.empty([len(lines),max_len],dtype)
    matrix.fill(pad)

    for i in range(len(lines)):
        line_ix = list(map(token_to_id.__getitem__,lines[i]))[:max_len]
        matrix[i,:len(line_ix)] = line_ix

    return matrix.T if time_major else matrix_____no_output_____batch_words,batch_tags = zip(*[zip(*sentence) for sentence in data[-3:]])

print("Word ids:")
print(to_matrix(batch_words,word_to_id))
print("Tag ids:")
print(to_matrix(batch_tags,tag_to_id))Word ids:
[[   2 3057    5    2 2238 1334 4238 2454    3    6   19   26 1070   69
     8 2088    6    3    1    3  266   65  342    2    1    3    2  315
     1    9   87  216 3322   69 1558    4    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0]
 [  45   12    8  511 8419    6   60 3246   39    2    1    1    3    2
   845    1    3    1    3   10 9910    2    1 3470    9   43    1    1
     3    6    2 1046  385   73 4562    3    9    2    1    1 3250    3
    12   10    2  861 5240   12    8 8936  121    1    4]
 [  33   64   26   12  445    7 7346    9    8 3337    3    1 2811    3
     2  463  572    2    1    1 1649   12    1    4    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0    0    0    0
     0    0    0    0    0    0    0    0    0    0    0]]
Tag ids:
[[ 6  3  4  6  3  3  9  9  7 12  4  5  9  4  6  3 12  7  9  7  9  8  4  6
   3  7  6 13  3  4  6  3  9  4  3  7  0  0  0  0  0  0  0  0  0  0  0  0
   0  0  0  0  0]
 [ 5  9  6  9  3 12  6  3  7  6 13  3  7  6 13  3  7 13  7  5  9  6  3  3
   4  6 13  3  7 12  6  3  6 13  3  7  4  6  3  9  3  7  9  4  6 13  3  9
   6  3  2 13  7]
 [ 4  6  5  9 13  4  3  4  6 13  7 13  3  7  6  3  4  6 13  3  3  9  9  7
   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
   0  0  0  0  0]]

### Build model

Unlike our previous lab, this time we'll focus on a high-level keras interface to recurrent neural networks. It is as simple as you can get with RNN, allbeit somewhat constraining for complex tasks like seq2seq.

By default, all keras RNNs apply to a whole sequence of inputs and produce a sequence of hidden states `(return_sequences=True` or just the last hidden state `(return_sequences=False)`. All the recurrence is happening under the hood.

At the top of our model we need to apply a Dense layer to each time-step independently. As of now, by default keras.layers.Dense would apply once to all time-steps concatenated. We use __keras.layers.TimeDistributed__ to modify Dense layer so that it would apply across both batch and time axes._____no_output_____

import keras
import keras.layers as L

model = keras.models.Sequential()
model.add(L.InputLayer([None],dtype='int32'))
model.add(L.Embedding(len(all_words),50))
model.add(L.SimpleRNN(64,return_sequences=True))

#add top layer that predicts tag probabilities
stepwise_dense = L.Dense(len(all_tags),activation='softmax')
stepwise_dense = L.TimeDistributed(stepwise_dense)
model.add(stepwise_dense)Using TensorFlow backend.

__Training:__ in this case we don't want to prepare the whole training dataset in advance. The main cause is that the length of every batch depends on the maximum sentence length within the batch. This leaves us two options: use custom training code as in previous seminar or use generators.

Keras models have a __`model.fit_generator`__ method that accepts a python generator yielding one batch at a time. But first we need to implement such generator:_____no_output_____

from keras.utils.np_utils import to_categorical
BATCH_SIZE=32
def generate_batches(sentences,batch_size=BATCH_SIZE,max_len=None,pad=0):
    assert isinstance(sentences, np.ndarray),"Make sure sentences is a numpy array"

    while True:
        indices = np.random.permutation(np.arange(len(sentences)))
        for start in range(0,len(indices)-1,batch_size):
            batch_indices = indices[start:start+batch_size]
            batch_words,batch_tags = [],[]
            for sent in sentences[batch_indices]:
                words,tags = zip(*sent)
                batch_words.append(words)
                batch_tags.append(tags)

            batch_words = to_matrix(batch_words,word_to_id,max_len,pad)
            batch_tags = to_matrix(batch_tags,tag_to_id,max_len,pad)

            batch_tags_1hot = to_categorical(batch_tags,len(all_tags)).reshape(batch_tags.shape+(-1,))
            yield batch_words,batch_tags_1hot
_____no_output_____

__Callbacks:__ Another thing we need is to measure model performance. The tricky part is not to count accuracy after sentence ends (on padding) and making sure we count all the validation data exactly once.

While it isn't impossible to persuade Keras to do all of that, we may as well write our own callback that does that.
Keras callbacks allow you to write a custom code to be ran once every epoch or every minibatch. We'll define one via LambdaCallback_____no_output_____

def compute_test_accuracy(model):
    test_words,test_tags = zip(*[zip(*sentence) for sentence in test_data])
    test_words,test_tags = to_matrix(test_words,word_to_id),to_matrix(test_tags,tag_to_id)

    #predict tag probabilities of shape [batch,time,n_tags]
    predicted_tag_probabilities = model.predict(test_words,verbose=1)
    predicted_tags = predicted_tag_probabilities.argmax(axis=-1)

    #compute accurary excluding padding
    numerator = np.sum(np.logical_and((predicted_tags == test_tags),(test_words != 0)))
    denominator = np.sum(test_words != 0)
    return float(numerator)/denominator

class EvaluateAccuracy(keras.callbacks.Callback):
    def on_epoch_end(self,epoch,logs=None):
        sys.stdout.flush()
        print("\nMeasuring validation accuracy...")
        acc = compute_test_accuracy(self.model)
        print("\nValidation accuracy: %.5f\n"%acc)
        sys.stdout.flush()
        _____no_output_____model.compile('adam','categorical_crossentropy')

model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,
                    callbacks=[EvaluateAccuracy()], epochs=5,)WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:697: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:2749: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:879: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:602: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:866: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:2289: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

Epoch 1/5
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:154: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:159: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:333: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:341: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

1343/1343 [============================>.] - ETA: 0s - loss: 0.2751
Measuring validation accuracy...
14272/14335 [============================>.] - ETA: 0s
Validation accuracy: 0.93903

1344/1343 [==============================] - 38s - loss: 0.2750
Epoch 2/5
1343/1343 [============================>.] - ETA: 0s - loss: 0.0593
Measuring validation accuracy...
14272/14335 [============================>.] - ETA: 0s
Validation accuracy: 0.94408

1344/1343 [==============================] - 40s - loss: 0.0593
Epoch 3/5
1342/1343 [============================>.] - ETA: 0s - loss: 0.0521
Measuring validation accuracy...
14335/14335 [==============================] - 5s

Validation accuracy: 0.94589

1344/1343 [==============================] - 40s - loss: 0.0521
Epoch 4/5
1341/1343 [============================>.] - ETA: 0s - loss: 0.0473
Measuring validation accuracy...
14272/14335 [============================>.] - ETA: 0s
Validation accuracy: 0.94620

1344/1343 [==============================] - 41s - loss: 0.0473
Epoch 5/5
1342/1343 [============================>.] - ETA: 0s - loss: 0.0431
Measuring validation accuracy...
14304/14335 [============================>.] - ETA: 0s
Validation accuracy: 0.94563

1344/1343 [==============================] - 40s - loss: 0.0431

Measure final accuracy on the whole test set._____no_output_____

acc = compute_test_accuracy(model)
print("Final accuracy: %.5f"%acc)

assert acc>0.94, "Keras has gone on a rampage again, please contact course staff."14240/14335 [============================>.] - ETA: 0sFinal accuracy: 0.94563

### Task I: getting all bidirectional

Since we're analyzing a full sequence, it's legal for us to look into future data.

A simple way to achieve that is to go both directions at once, making a __bidirectional RNN__.

In Keras you can achieve that both manually (using two LSTMs and Concatenate) and by using __`keras.layers.Bidirectional`__.

This one works just as `TimeDistributed` we saw before: you wrap it around a recurrent layer (SimpleRNN now and LSTM/GRU later) and it actually creates two layers under the hood.

Your first task is to use such a layer for our POS-tagger._____no_output_____

#Define a model that utilizes bidirectional SimpleRNN
model = keras.models.Sequential([
        L.InputLayer([None],dtype='int32'),
        L.Embedding(len(all_words),50),
        L.Bidirectional(L.SimpleRNN(64, return_sequences=True))
])

#add top layer that predicts tag probabilities
stepwise_dense = L.Dense(len(all_tags), activation='softmax')
stepwise_dense = L.TimeDistributed(stepwise_dense)
model.add(stepwise_dense)_____no_output_____model.compile('adam','categorical_crossentropy')

model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,
                    callbacks=[EvaluateAccuracy()], epochs=5,)Epoch 1/5
1343/1343 [============================>.] - ETA: 0s - loss: 0.2139
Measuring validation accuracy...
14272/14335 [============================>.] - ETA: 0s
Validation accuracy: 0.95609

1344/1343 [==============================] - 58s - loss: 0.2138
Epoch 2/5
1342/1343 [============================>.] - ETA: 0s - loss: 0.0424
Measuring validation accuracy...
14272/14335 [============================>.] - ETA: 0s
Validation accuracy: 0.96086

1344/1343 [==============================] - 59s - loss: 0.0424
Epoch 3/5
1342/1343 [============================>.] - ETA: 0s - loss: 0.0351
Measuring validation accuracy...
14272/14335 [============================>.] - ETA: 0s
Validation accuracy: 0.96152

1344/1343 [==============================] - 58s - loss: 0.0351
Epoch 4/5
1342/1343 [============================>.] - ETA: 0s - loss: 0.0299
Measuring validation accuracy...
14304/14335 [============================>.] - ETA: 0s
Validation accuracy: 0.96121

1344/1343 [==============================] - 57s - loss: 0.0299
Epoch 5/5
1343/1343 [============================>.] - ETA: 0s - loss: 0.0251
Measuring validation accuracy...
14335/14335 [==============================] - 8s

Validation accuracy: 0.96138

1344/1343 [==============================] - 57s - loss: 0.0251
acc = compute_test_accuracy(model)
print("\nFinal accuracy: %.5f"%acc)

assert acc>0.96, "Bidirectional RNNs are better than this!"
print("Well done!")14304/14335 [============================>.] - ETA: 0s
Final accuracy: 0.96138
Well done!

### Task II: now go and improve it

You guessed it. We're now gonna ask you to come up with a better network.

Here's a few tips:

* __Go beyond SimpleRNN__: there's `keras.layers.LSTM` and `keras.layers.GRU`
  * If you want to use a custom recurrent Cell, read [this](https://keras.io/layers/recurrent/#rnn)
  * You can also use 1D Convolutions (`keras.layers.Conv1D`). They are often as good as recurrent layers but with less overfitting.
* __Stack more layers__: if there is a common motif to this course it's about stacking layers
  * You can just add recurrent and 1dconv layers on top of one another and keras will understand it
  * Just remember that bigger networks may need more epochs to train
* __Gradient clipping__: If your training isn't as stable as you'd like, set `clipnorm` in your optimizer.
  * Which is to say, it's a good idea to watch over your loss curve at each minibatch. Try tensorboard callback or something similar.
* __Regularization__: you can apply dropouts as usuall but also in an RNN-specific way
  * `keras.layers.Dropout` works inbetween RNN layers
  * Recurrent layers also have `recurrent_dropout` parameter
* __More words!__: You can obtain greater performance by expanding your model's input dictionary from 5000 to up to every single word!
  * Just make sure your model doesn't overfit due to so many parameters.
  * Combined with regularizers or pre-trained word-vectors this could be really good cuz right now our model is blind to >5% of words.
* __The most important advice__: don't cram in everything at once!
  * If you stuff in a lot of modiffications, some of them almost inevitably gonna be detrimental and you'll never know which of them are.
  * Try to instead go in small iterations and record experiment results to guide further search.

There's some advanced stuff waiting at the end of the notebook.

Good hunting!_____no_output_____

#Define a model that utilizes bidirectional SimpleRNN
model = keras.models.Sequential()

model.add(L.InputLayer([None],dtype='int32'))
model.add(L.Embedding(len(all_words),50))
model.add(L.Bidirectional(L.GRU(128,return_sequences=True)))
model.add(L.Dropout(0.17))
model.add(L.Bidirectional(L.GRU(64,return_sequences=True)))
model.add(L.Dropout(0.17))

#add top layer that predicts tag probabilities
stepwise_dense = L.Dense(len(all_tags),activation='softmax')
stepwise_dense = L.TimeDistributed(stepwise_dense)
model.add(stepwise_dense)WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:2878: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
#feel free to change anything here

model.compile('adam','categorical_crossentropy')

model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,
                    callbacks=[EvaluateAccuracy()], epochs=5,)Epoch 1/5
1343/1343 [============================>.] - ETA: 0s - loss: 0.1652
Measuring validation accuracy...
14335/14335 [==============================] - 64s

Validation accuracy: 0.96053

1344/1343 [==============================] - 420s - loss: 0.1651
Epoch 2/5
1343/1343 [============================>.] - ETA: 0s - loss: 0.0407
Measuring validation accuracy...
14335/14335 [==============================] - 63s

Validation accuracy: 0.96429

1344/1343 [==============================] - 417s - loss: 0.0407
Epoch 3/5
1343/1343 [============================>.] - ETA: 0s - loss: 0.0341
Measuring validation accuracy...
14335/14335 [==============================] - 64s

Validation accuracy: 0.96685

1344/1343 [==============================] - 416s - loss: 0.0341
Epoch 4/5
1343/1343 [============================>.] - ETA: 0s - loss: 0.0294
Measuring validation accuracy...
14335/14335 [==============================] - 63s

Validation accuracy: 0.96728

1344/1343 [==============================] - 417s - loss: 0.0294
Epoch 5/5
1343/1343 [============================>.] - ETA: 0s - loss: 0.0260
Measuring validation accuracy...
14335/14335 [==============================] - 63s

Validation accuracy: 0.96746

1344/1343 [==============================] - 415s - loss: 0.0260
acc = compute_test_accuracy(model)
print("\nFinal accuracy: %.5f"%acc)

if acc >= 0.99:
    print("Awesome! Sky was the limit and yet you scored even higher!")
elif acc >= 0.98:
    print("Excellent! Whatever dark magic you used, it certainly did it's trick.")
elif acc >= 0.97:
    print("Well done! If this was a graded assignment, you would have gotten a 100% score.")
elif acc > 0.96:
    print("Just a few more iterations!")
else:
    print("There seems to be something broken in the model. Unless you know what you're doing, try taking bidirectional RNN and adding one enhancement at a time to see where's the problem.")14335/14335 [==============================] - 63s

Final accuracy: 0.96746
Just a few more iterations!

#### Some advanced stuff
Here there are a few more tips on how to improve training that are a bit trickier to impliment. We strongly suggest that you try them _after_ you've got a good initial model.
* __Use pre-trained embeddings__: you can use pre-trained weights from [there](http://ahogrammer.com/2017/01/20/the-list-of-pretrained-word-embeddings/) to kickstart your Embedding layer.
  * Embedding layer has a matrix W (layer.W) which contains word embeddings for each word in the dictionary. You can just overwrite them with tf.assign.
  * When using pre-trained embeddings, pay attention to the fact that model's dictionary is different from your own.
  * You may want to switch trainable=False for embedding layer in first few epochs as in regular fine-tuning.
* __More efficient batching__: right now TF spends a lot of time iterating over "0"s
  * This happens because batch is always padded to the length of a longest sentence
  * You can speed things up by pre-generating batches of similar lengths and feeding it with randomly chosen pre-generated batch.
  * This technically breaks the i.i.d. assumption, but it works unless you come up with some insane rnn architectures.
* __Structured loss functions__: since we're tagging the whole sequence at once, we might as well train our network to do so.
  * There's more than one way to do so, but we'd recommend starting with [Conditional Random Fields](http://blog.echen.me/2012/01/03/introduction-to-conditional-random-fields/)
  * You could plug CRF as a loss function and still train by backprop. There's even some neat tensorflow [implementation](https://www.tensorflow.org/api_guides/python/contrib.crf) for you.
_____no_output_____


# Notebook from theghostofc/Exploratory-Data-Analytics
Path: Pandas/5_Merging_Concatenating.ipynb

# Merging and Concatenating Dataframes

In this section, you will merge and concatenate multiple dataframes. Merging is one of the most common operations you will do, since data often comes in various files.

In our case, we have sales data of a retail store spread across multiple files. We will now work with all these data files and learn to:
* Merge multiple dataframes using common columns/keys using ```pd.merge()```
* Concatenate dataframes using ```pd.concat()```

Let's first read all the data files._____no_output_____

# loading libraries and reading the data
import numpy as np
import pandas as pd

market_df = pd.read_csv("../Data/global_sales_data/market_fact.csv")
customer_df = pd.read_csv("../Data/global_sales_data/cust_dimen.csv")
product_df = pd.read_csv("../Data/global_sales_data/prod_dimen.csv")
shipping_df = pd.read_csv("../Data/global_sales_data/shipping_dimen.csv")
orders_df = pd.read_csv("../Data/global_sales_data/orders_dimen.csv")_____no_output_____

### Merging Dataframes Using ```pd.merge()```

There are five data files:
1. The ```market_fact``` table contains the sales data of each order
2. The other 4 files are called 'dimension tables/files' and contain metadata about customers, products, shipping details, order details etc.

If you are familiar with star schemas and data warehouse designs, you will note that we have one fact table and four dimension tables.
_____no_output_____

# Already familiar with market data: Each row is an order
market_df.head()_____no_output_____# Customer dimension table: Each row contains metadata about customers
customer_df.head()_____no_output_____# Product dimension table
product_df.head()_____no_output_____# Shipping metadata
shipping_df.head()_____no_output_____# Orders dimension table
orders_df.head()_____no_output_____

### Merging Dataframes

Say you want to select all orders and observe the ```Sales``` of the customer segment *Corporate*. Since customer segment details are present in the dataframe ```customer_df```, we will first need to merge it with ```market_df```.
_____no_output_____

# Merging the dataframes
# Note that Cust_id is the common column/key, which is provided to the 'on' argument
# how = 'inner' makes sure that only the customer ids present in both dfs are included in the result
df_1 = pd.merge(market_df, customer_df, how='inner', on='Cust_id')
df_1.head()_____no_output_____# Now, you can subset the orders made by customers from 'Corporate' segment
df_1.loc[df_1['Customer_Segment'] == 'CORPORATE', :]_____no_output_____# Example 2: Select all orders from product category = office supplies and from the corporate segment
# We now need to merge the product_df

df_2 = pd.merge(df_1, product_df, how='inner', on='Prod_id')
df_2.head()_____no_output_____# Select all orders from product category = office supplies and from the corporate segment
df_2.loc[(df_2['Product_Category']=='OFFICE SUPPLIES') & (df_2['Customer_Segment']=='CORPORATE'),:]_____no_output_____

Similary, you can merge the other dimension tables - ```shipping_df``` and ```orders_df``` to create a ```master_df``` and perform indexing using any column in the master dataframe.
_____no_output_____

# Merging shipping_df
df_3 = pd.merge(df_2, shipping_df, how='inner', on='Ship_id')
df_3.shape_____no_output_____# Merging the orders table to create a master df
master_df = pd.merge(df_3, orders_df, how='inner', on='Ord_id')
master_df.shape
master_df.head()_____no_output_____

Similary, you can perform left, right and outer merges (joins) by using the argument ```how = 'left' / 'right' / 'outer'```._____no_output_____### Concatenating Dataframes

Concatenation is much more straightforward than merging. It is used when you have dataframes having the same columns and want to append them (pile one on top of the other), or having the same rows and want to append them side-by-side.

#### Concatenating Dataframes Having the Same columns

Say you have two dataframes having the same columns, like so:_____no_output_____

# dataframes having the same columns
df1 = pd.DataFrame({'Name': ['Aman', 'Joy', 'Rashmi', 'Saif'],
                    'Age': ['34', '31', '22', '33'],
                    'Gender': ['M', 'M', 'F', 'M']}
                  )

df2 = pd.DataFrame({'Name': ['Akhil', 'Asha', 'Preeti'],
                    'Age': ['31', '22', '23'],
                    'Gender': ['M', 'F', 'F']}
                  )
df1_____no_output_____df2_____no_output_____# To concatenate them, one on top of the other, you can use pd.concat
# The first argument is a sequence (list) of dataframes
# axis = 0 indicates that we want to concat along the row axis
pd.concat([df1, df2], axis = 0)_____no_output_____# A useful and intuitive alternative to concat along the rows is the append() function
# It concatenates along the rows
df1.append(df2)
_____no_output_____

#### Concatenating Dataframes Having the Same Rows

You may also have dataframes having the same rows but different columns (and having no common columns). In this case, you may want to concat them side-by-side. For e.g.:_____no_output_____

df1 = pd.DataFrame({'Name': ['Aman', 'Joy', 'Rashmi', 'Saif'],
                    'Age': ['34', '31', '22', '33'],
                    'Gender': ['M', 'M', 'F', 'M']}
                  )
df1_____no_output_____df2 = pd.DataFrame({'School': ['RK Public', 'JSP', 'Carmel Convent', 'St. Paul'],
                    'Graduation Marks': ['84', '89', '76', '91']}
                  )
df2_____no_output_____# To join the two dataframes, use axis = 1 to indicate joining along the columns axis
# The join is possible because the corresponding rows have the same indices
pd.concat([df1, df2], axis = 1)_____no_output_____

Note that you can also use the ```pd.concat()``` method to merge dataframes using common keys, though here we will not discuss that. For simplicity, we have used the ```pd.merge()``` method for database-style merging and ```pd.concat()``` for appending dataframes having no common columns._____no_output_____