# Notebook from audiofeature/DeepLearningTutorial_2019
Path: Car_recognition.ipynb

# Tutorial: Deep Learning using Convolutional Neural Networks
# Example: Car Recognition

This tutorial shows how Convolutional Neural Networks are used to recognize cars on images vs. images with no cars on them (binary classification).

The data set used is the [UIUC Image Database for Car Detection](http://cogcomp.cs.illinois.edu/Data/Car/) containing:
* 1050 training images (550 car and 500 non-car images)
* 170 test images, containing 200 cars at roughly the same scale as in the training images
(we do not use the multi-scale test images, containing 139 cars at various scales, here)

This tutorial contains:
* Image Loading and Preprocessing
* Standardization of Data
* Fully Connected Neural Networks
* Convolutional Neural Networks
* Batch Normalization
* ReLU Activation
* Dropout
* Data Augmentation
* Plotting the Training Curve

You can execute the following code blocks by pressing SHIFT+Enter consecutively._____no_output_____

import os

# if you want to use the GPU
#device = 'gpu'
#os.environ['THEANO_FLAGS']='mode=FAST_RUN,device=' + device + ',floatX=float32'

import argparse
import csv
import datetime
import glob
import math
import numpy as np
import sys
import time

from PIL import Image
from sklearn.metrics import accuracy_score

#import keras
from keras.backend import floatx  # abstract default float type for Keras (e.g. float32)
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D, Activation, Dense, Dropout, Flatten
from keras.layers.normalization import BatchNormalization

np.random.seed(1) # we initialize a random seed here to make the experiments repeatable with same resultsUsing TensorFlow backend.

## Load the Images from Training Set_____no_output_____Set the DATA_PATH please:_____no_output_____

# if you use the iDSDL Lab machine
DATA_PATH = '/home/ffg/idsdlmteacher1/Code/DeepLearningTutorial_2019/data'
# if you downloaded the dataset into the tutorial folder (otherwise adjust the DATA_PATH please)
# DATA_PATH = '.'_____no_output_____path = os.path.join(DATA_PATH, 'CarData/TrainImages')
files = glob.glob(os.path.join(path, '*.pgm'))
print("Found %d files" % len(files))Found 1050 files
images = []
image_names = []

for filename in files:
    image_names.append(os.path.basename(filename))
    with Image.open(filename) as img:
        images.append(np.array(img)) # we convert the images to a Numpy array and store them in a list_____no_output_____len(images)_____no_output_____# show a selected image to check
i=700
print(image_names[i])
Image.fromarray(images[i])pos-372.pgm
type(images[i])_____no_output_____images[i].shape   # height x width   (Numpy ordering)_____no_output_____

Note: Color RGB images have an additional dimension of depth 3, e.g. (40, 100, 3)_____no_output_____## Make 1 big array of list of images_____no_output_____

# a list of many 40x100 images is made into 1 big array
# as data type we use "floatx()" from Keras which is imported above (for GPU it is most often float32)
img_array = np.array(images, dtype=floatx())
img_array.shape_____no_output_____

## Create the Groundtruth based on filenames:

In this data set, images with cars start with "pos-" and images with no cars start with "neg-". We create a numeric list here, containing 1 for car images and 0 for non-car images._____no_output_____

classes = []
for name in image_names:
    if name.startswith('neg'):
        classes.append(0)
    else:
        classes.append(1)_____no_output_____# look at the first 25 classes
classes[0:25]_____no_output_____# look classes 490 to 510
classes[490:510]_____no_output_____print("Groundtruth Statistics:")

for c in set(classes):
    print("Class {}: {} instances".format(c, classes.count(c)))Groundtruth Statistics:
Class 0: 500 instances
Class 1: 550 instances
baseline = 550 * 1.0 / len(classes)
print("baseline = %.3f" % baseline)baseline = 0.524

The baseline is 52.4%, i.e. a 'dumb' classifier can assign all predictions to the majority class achieving this accurcay. We aim at building a classifier that performs better than that._____no_output_____## Standardization

Always standardize the data before feeding it into the Neural Network!

Here we use Zero-mean Unit-variance standardization which means we deduct the mean and divide by the standard deviation.

(Note: Here, we do this "flat", i.e. one mean and std.dev. for the whole image is computed over all pixels (not per pixel); in RGB images, standardization can be done e.g. for each colour channel individually; in other/non-image data sets, attribute-wise standardization should be applied)._____no_output_____

img_array.min(), img_array.max()_____no_output_____# compute mean and standard deviation
mean = img_array.mean()
stddev = img_array.std()
mean, stddev_____no_output_____# apply zero mean unit variance standardization on all pixels of all images
img_array = (img_array - mean) / stddev
img_array.mean(), img_array.std()_____no_output_____#check the new min, mean and max
img_array.min(), img_array.mean(), img_array.max()_____no_output_____

# Creating Neural Network Models in Keras_____no_output_____## Fully Connected Networks

For a fully connected neural network, the x and y axis of an image do not play a role at all. All pixels are considered as a completely individual input to the neural network. Therefore the 2D image arrays have to be flattened to a vector._____no_output_____

#  flatten images to vectors
images_flat = img_array.reshape(img_array.shape[0],-1)
images_flat.shape_____no_output_____# find out input shape for NN, which is just a long vector (40x100 = 4000)
input_shape = images_flat.shape[1]
input_shape_____no_output_____

### Creating a Model

In Keras, one can choose between a **Sequential model** and a **Graph model**. Sequential models are the standard case. Graph models are for parallel networks and use the functional API (see Music/Speech tutorial).

Here we create a sequential model with 2 fully connected (a.k.a. 'dense') layers containing 256 units each.

The output unit is a single Sigmoid unit which can predict values between 0 and 1._____no_output_____

# simple Fully-connected network with 2 hidden Dense layers

model = Sequential()

model.add(Dense(256, input_dim=input_shape, activation='sigmoid'))

model.add(Dense(256, activation='sigmoid'))

model.add(Dense(1,activation='sigmoid'))_____no_output_____

Printing the model's summary of layers is always a good idea:_____no_output_____

model.summary()_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense_1 (Dense)              (None, 256)               1024256
_________________________________________________________________
dense_2 (Dense)              (None, 256)               65792
_________________________________________________________________
dense_3 (Dense)              (None, 1)                 257
=================================================================
Total params: 1,090,305
Trainable params: 1,090,305
Non-trainable params: 0
_________________________________________________________________
#from IPython.display import SVG
#from keras.utils.vis_utils import model_to_dot
#SVG(model_to_dot(model).create(prog='dot', format='svg'))_____no_output_____

### Defining Loss Function and Optimizer Strategy: Gradient Descent_____no_output_____

# Define a loss function
loss = 'binary_crossentropy'  # 'categorical_crossentropy' for multi-class problems

# Optimizer = Stochastic Gradient Descent
optimizer = 'sgd'

# Compiling the model: creates the whole model structure in memory
model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])_____no_output_____

### Training the model on the input dataset_____no_output_____We define the number of epochs (= training cycles) and a batch size (= number of images presented to the network for each weight update):_____no_output_____

epochs = 15
model.fit(images_flat, classes, batch_size=32, epochs=epochs) #, validation_data=validation_data) Epoch 1/15
1050/1050 [==============================] - 0s 409us/step - loss: 0.6762 - acc: 0.6362
Epoch 2/15
1050/1050 [==============================] - 0s 170us/step - loss: 0.6216 - acc: 0.7743
Epoch 3/15
1050/1050 [==============================] - 0s 171us/step - loss: 0.5798 - acc: 0.8600
Epoch 4/15
1050/1050 [==============================] - 0s 169us/step - loss: 0.5402 - acc: 0.8914
Epoch 5/15
1050/1050 [==============================] - 0s 171us/step - loss: 0.5026 - acc: 0.8933
Epoch 6/15
1050/1050 [==============================] - 0s 167us/step - loss: 0.4649 - acc: 0.9086
Epoch 7/15
1050/1050 [==============================] - 0s 167us/step - loss: 0.4302 - acc: 0.9076
Epoch 8/15
1050/1050 [==============================] - 0s 168us/step - loss: 0.3989 - acc: 0.9124
Epoch 9/15
1050/1050 [==============================] - 0s 167us/step - loss: 0.3700 - acc: 0.9219
Epoch 10/15
1050/1050 [==============================] - 0s 169us/step - loss: 0.3440 - acc: 0.9248
Epoch 11/15
1050/1050 [==============================] - 0s 169us/step - loss: 0.3211 - acc: 0.9276
Epoch 12/15
1050/1050 [==============================] - 0s 170us/step - loss: 0.3006 - acc: 0.9333
Epoch 13/15
1050/1050 [==============================] - 0s 170us/step - loss: 0.2825 - acc: 0.9324
Epoch 14/15
1050/1050 [==============================] - 0s 169us/step - loss: 0.2661 - acc: 0.9371
Epoch 15/15
1050/1050 [==============================] - 0s 169us/step - loss: 0.2520 - acc: 0.9381
# verify Accuracy on Train set
predictions = model.predict_classes(images_flat)
accuracy_score(classes, predictions)_____no_output_____

### 100% Accuracy - perfect, no?

This is the accuracy on the training set. A (large, especially fully connected network with sufficient number of units) can easily learn the entire training set (especially a small one like here).

This very likely leads to overfitting. That's why we test on an independent test set.
_____no_output_____# Testing with Test Data Set_____no_output_____

path = os.path.join(DATA_PATH, 'CarData/TestImages')
files = glob.glob(os.path.join(path, '*.pgm'))
print("Found %d files" % len(files))Found 170 files

The images to predict might be of different shape and size. We are using a resize and crop function (defined in image_preprocessing.py) to bring them to the same format (40x100) as needed by our neural network._____no_output_____

from image_preprocessing import resize_and_crop

test_images = []

for filename in files:
    with Image.open(filename) as img:
        img_resized = resize_and_crop(img,target_width=100,target_height=40)
        test_images.append(np.array(img_resized))
        print("Image size: original: {} \t reshaped: {}".format(img.size, img_resized.size))Image size: original: (266, 158) 	 reshaped: (100, 40)
Image size: original: (155, 106) 	 reshaped: (100, 40)
Image size: original: (302, 146) 	 reshaped: (100, 40)
Image size: original: (150, 98) 	 reshaped: (100, 40)
Image size: original: (240, 145) 	 reshaped: (100, 40)
Image size: original: (167, 110) 	 reshaped: (100, 40)
Image size: original: (170, 91) 	 reshaped: (100, 40)
Image size: original: (160, 109) 	 reshaped: (100, 40)
Image size: original: (144, 98) 	 reshaped: (100, 40)
Image size: original: (175, 90) 	 reshaped: (100, 40)
Image size: original: (246, 110) 	 reshaped: (100, 40)
Image size: original: (142, 83) 	 reshaped: (100, 40)
Image size: original: (183, 123) 	 reshaped: (100, 40)
Image size: original: (146, 89) 	 reshaped: (100, 40)
Image size: original: (155, 106) 	 reshaped: (100, 40)
Image size: original: (146, 94) 	 reshaped: (100, 40)
Image size: original: (175, 119) 	 reshaped: (100, 40)
Image size: original: (235, 117) 	 reshaped: (100, 40)
Image size: original: (160, 95) 	 reshaped: (100, 40)
Image size: original: (219, 132) 	 reshaped: (100, 40)
Image size: original: (170, 98) 	 reshaped: (100, 40)
Image size: original: (140, 95) 	 reshaped: (100, 40)
Image size: original: (132, 106) 	 reshaped: (100, 40)
Image size: original: (225, 140) 	 reshaped: (100, 40)
Image size: original: (159, 111) 	 reshaped: (100, 40)
Image size: original: (160, 100) 	 reshaped: (100, 40)
Image size: original: (190, 123) 	 reshaped: (100, 40)
Image size: original: (205, 119) 	 reshaped: (100, 40)
Image size: original: (261, 156) 	 reshaped: (100, 40)
Image size: original: (200, 136) 	 reshaped: (100, 40)
Image size: original: (165, 113) 	 reshaped: (100, 40)
Image size: original: (110, 75) 	 reshaped: (100, 40)
Image size: original: (174, 83) 	 reshaped: (100, 40)
Image size: original: (160, 109) 	 reshaped: (100, 40)
Image size: original: (159, 119) 	 reshaped: (100, 40)
Image size: original: (186, 121) 	 reshaped: (100, 40)
Image size: original: (145, 99) 	 reshaped: (100, 40)
Image size: original: (159, 119) 	 reshaped: (100, 40)
Image size: original: (124, 85) 	 reshaped: (100, 40)
Image size: original: (209, 143) 	 reshaped: (100, 40)
Image size: original: (160, 97) 	 reshaped: (100, 40)
Image size: original: (264, 145) 	 reshaped: (100, 40)
Image size: original: (190, 101) 	 reshaped: (100, 40)
Image size: original: (280, 138) 	 reshaped: (100, 40)
Image size: original: (180, 101) 	 reshaped: (100, 40)
Image size: original: (220, 124) 	 reshaped: (100, 40)
Image size: original: (229, 118) 	 reshaped: (100, 40)
Image size: original: (146, 110) 	 reshaped: (100, 40)
Image size: original: (157, 107) 	 reshaped: (100, 40)
Image size: original: (180, 123) 	 reshaped: (100, 40)
Image size: original: (150, 88) 	 reshaped: (100, 40)
Image size: original: (160, 109) 	 reshaped: (100, 40)
Image size: original: (212, 126) 	 reshaped: (100, 40)
Image size: original: (229, 112) 	 reshaped: (100, 40)
Image size: original: (173, 103) 	 reshaped: (100, 40)
Image size: original: (245, 157) 	 reshaped: (100, 40)
Image size: original: (159, 99) 	 reshaped: (100, 40)
Image size: original: (260, 195) 	 reshaped: (100, 40)
Image size: original: (126, 86) 	 reshaped: (100, 40)
Image size: original: (266, 140) 	 reshaped: (100, 40)
Image size: original: (229, 146) 	 reshaped: (100, 40)
Image size: original: (180, 135) 	 reshaped: (100, 40)
Image size: original: (175, 119) 	 reshaped: (100, 40)
Image size: original: (138, 73) 	 reshaped: (100, 40)
Image size: original: (175, 108) 	 reshaped: (100, 40)
Image size: original: (217, 141) 	 reshaped: (100, 40)
Image size: original: (159, 112) 	 reshaped: (100, 40)
Image size: original: (145, 99) 	 reshaped: (100, 40)
Image size: original: (150, 102) 	 reshaped: (100, 40)
Image size: original: (177, 107) 	 reshaped: (100, 40)
Image size: original: (190, 130) 	 reshaped: (100, 40)
Image size: original: (135, 76) 	 reshaped: (100, 40)
Image size: original: (181, 98) 	 reshaped: (100, 40)
Image size: original: (168, 106) 	 reshaped: (100, 40)
Image size: original: (140, 88) 	 reshaped: (100, 40)
Image size: original: (169, 115) 	 reshaped: (100, 40)
Image size: original: (176, 120) 	 reshaped: (100, 40)
Image size: original: (226, 125) 	 reshaped: (100, 40)
Image size: original: (225, 143) 	 reshaped: (100, 40)
Image size: original: (125, 85) 	 reshaped: (100, 40)
Image size: original: (186, 116) 	 reshaped: (100, 40)
Image size: original: (149, 112) 	 reshaped: (100, 40)
Image size: original: (266, 147) 	 reshaped: (100, 40)
Image size: original: (235, 176) 	 reshaped: (100, 40)
Image size: original: (230, 140) 	 reshaped: (100, 40)
Image size: original: (120, 82) 	 reshaped: (100, 40)
Image size: original: (272, 138) 	 reshaped: (100, 40)
Image size: original: (168, 102) 	 reshaped: (100, 40)
Image size: original: (184, 108) 	 reshaped: (100, 40)
Image size: original: (155, 98) 	 reshaped: (100, 40)
Image size: original: (235, 143) 	 reshaped: (100, 40)
Image size: original: (260, 155) 	 reshaped: (100, 40)
Image size: original: (230, 157) 	 reshaped: (100, 40)
Image size: original: (153, 115) 	 reshaped: (100, 40)
Image size: original: (234, 126) 	 reshaped: (100, 40)
Image size: original: (204, 127) 	 reshaped: (100, 40)
Image size: original: (180, 100) 	 reshaped: (100, 40)
Image size: original: (205, 134) 	 reshaped: (100, 40)
Image size: original: (151, 98) 	 reshaped: (100, 40)
Image size: original: (189, 110) 	 reshaped: (100, 40)
Image size: original: (167, 112) 	 reshaped: (100, 40)
Image size: original: (152, 91) 	 reshaped: (100, 40)
Image size: original: (165, 102) 	 reshaped: (100, 40)
Image size: original: (213, 98) 	 reshaped: (100, 40)
Image size: original: (167, 91) 	 reshaped: (100, 40)
Image size: original: (302, 186) 	 reshaped: (100, 40)
Image size: original: (216, 159) 	 reshaped: (100, 40)
Image size: original: (160, 83) 	 reshaped: (100, 40)
Image size: original: (174, 96) 	 reshaped: (100, 40)
Image size: original: (235, 134) 	 reshaped: (100, 40)
Image size: original: (170, 116) 	 reshaped: (100, 40)
Image size: original: (250, 156) 	 reshaped: (100, 40)
Image size: original: (210, 115) 	 reshaped: (100, 40)
Image size: original: (167, 101) 	 reshaped: (100, 40)
Image size: original: (220, 145) 	 reshaped: (100, 40)
Image size: original: (178, 109) 	 reshaped: (100, 40)
Image size: original: (206, 136) 	 reshaped: (100, 40)
Image size: original: (253, 172) 	 reshaped: (100, 40)
Image size: original: (147, 100) 	 reshaped: (100, 40)
Image size: original: (252, 155) 	 reshaped: (100, 40)
Image size: original: (184, 124) 	 reshaped: (100, 40)
Image size: original: (325, 147) 	 reshaped: (100, 40)
Image size: original: (188, 112) 	 reshaped: (100, 40)
Image size: original: (270, 162) 	 reshaped: (100, 40)
Image size: original: (200, 150) 	 reshaped: (100, 40)
Image size: original: (126, 86) 	 reshaped: (100, 40)
Image size: original: (167, 99) 	 reshaped: (100, 40)
Image size: original: (225, 127) 	 reshaped: (100, 40)
Image size: original: (228, 128) 	 reshaped: (100, 40)
Image size: original: (163, 111) 	 reshaped: (100, 40)
Image size: original: (150, 90) 	 reshaped: (100, 40)
Image size: original: (128, 96) 	 reshaped: (100, 40)
Image size: original: (212, 128) 	 reshaped: (100, 40)
Image size: original: (172, 117) 	 reshaped: (100, 40)
Image size: original: (301, 179) 	 reshaped: (100, 40)
Image size: original: (138, 87) 	 reshaped: (100, 40)
Image size: original: (175, 108) 	 reshaped: (100, 40)
Image size: original: (150, 103) 	 reshaped: (100, 40)
Image size: original: (128, 77) 	 reshaped: (100, 40)
Image size: original: (178, 114) 	 reshaped: (100, 40)
Image size: original: (168, 115) 	 reshaped: (100, 40)
Image size: original: (180, 126) 	 reshaped: (100, 40)
Image size: original: (168, 115) 	 reshaped: (100, 40)
Image size: original: (179, 100) 	 reshaped: (100, 40)
Image size: original: (175, 119) 	 reshaped: (100, 40)
Image size: original: (154, 105) 	 reshaped: (100, 40)
Image size: original: (200, 150) 	 reshaped: (100, 40)
Image size: original: (175, 119) 	 reshaped: (100, 40)
Image size: original: (160, 104) 	 reshaped: (100, 40)
Image size: original: (207, 116) 	 reshaped: (100, 40)
Image size: original: (185, 126) 	 reshaped: (100, 40)
Image size: original: (220, 130) 	 reshaped: (100, 40)
Image size: original: (360, 176) 	 reshaped: (100, 40)
Image size: original: (200, 139) 	 reshaped: (100, 40)
# show random image
i=4
Image.fromarray(test_images[i])_____no_output_____# make 1 big array again from list
test_images = np.array(test_images, dtype=floatx())_____no_output_____

### Standardize Test Set

The test data has to be standardized in the same way as the training data for compatibility with the model! That means, we take the mean and standard deviation of the training data to transform also the test data._____no_output_____

# NO! we take the same mean and stddev from the training data above!
#mean = test_images.mean()
#stddev = test_images.std()
#print(mean, stddev)_____no_output_____test_images.mean(), test_images.std()_____no_output_____# re-use values from train set above
test_images = (test_images - mean) / stddev_____no_output_____

### Flatten Images for Full model_____no_output_____

test_images_flat = test_images.reshape(test_images.shape[0],-1)
test_images_flat.shape_____no_output_____

### Make Predictions_____no_output_____

test_pred = model.predict_classes(test_images_flat)
# show 30 first predictions
test_pred[0:30,0]_____no_output_____# Groundtruth:
# this TEST SET contains ONLY CARS on images!
# Thus all the test classes are 1
test_classes = [1] * len(files)_____no_output_____# let's count the number of ones ...
test_pred.sum()_____no_output_____

As ALL our test classes are 1, counting the number of 1's and dividing by number of files gives us the Accuracy:_____no_output_____

len(files)_____no_output_____test_pred.sum() / len(files)_____no_output_____

The real way to do it is to compare the predictions (test_pred) with the ground truth (test_classes) and sum up the correct ones.
This is exactly what the scikit-learn function accuracy_score does:_____no_output_____

acc = accuracy_score(test_classes, test_pred)
acc_____no_output_____

Accuracy on the Test Set is rather low._____no_output_____# Convolutional Neural Networks

A Convolutional Neural Network (ConvNet or CNN) is a type of (deep) Neural Network that is well-suited for 2D axes data, such as images, as it is optimized for learning from spatial proximity. Its core elements are 2D filter kernels which essentially learn the weights of the Neural Network, and downscaling functions such as Max Pooling.

A CNN can have one or more Convolution layers, each of them having an arbitrary number of N filters (which define the depth of the CNN layer), following typically by a pooling step, which groups neighboring pixels together and thus reduces the image resolution by retaining only the maximum values of neighboring pixels._____no_output_____### Input

Our input to the CNN is the standardized version of the original image array.

#### Adding the channel

For CNNs, we need to add a dimension for the color channel to the data. RGB images typically have an 3rd dimension with the color.
For greyscale images we need to add an extra dimension for compatibility with the CNN implementation.

In Tensorflow, the color channel is the last dimension in the image shape.

For greyscale images, we add the number 1 as the depth of the additional dimension of the input shape (for RGB color images, the number of channels is 3)._____no_output_____

n_channels = 1
N, height, width = img_array.shape
train_img = img_array.reshape(N, height, width, n_channels)  # Tensorflow ordering: channnel last
N, height, width = test_images.shape  # height and width should/must be the same as img_array
test_img = test_images.reshape(N, height, width, n_channels)_____no_output_____# NOTE: 2 alternative ways to do the same:
# a) train_img = np.expand_dims(img_array, axis=3)
# b) train_img = img_array[:,:,:,np.newaxis]_____no_output_____img_array.shape_____no_output_____train_img.shape_____no_output_____# we store the new shape of the images in the 'input_shape' variable.
# take all dimensions except the 0th one (which is the number of images)

input_shape = train_img.shape[1:]
input_shape_____no_output_____

## Create a simple CNN model

We start with a simple 2 layer Convolutional Neural Network, defined as a Keras Sequential model:_____no_output_____

def simpleModel():
    model = Sequential()

    # Layer 1
    model.add(Conv2D(filters=16, kernel_size=(3,3), input_shape=input_shape, activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))  # reducing image resolution by half

    # Layer 2
    model.add(Conv2D(filters=32, kernel_size=(3,3), activation='relu'))  # input_shape is only needed in 1st layer
    model.add(MaxPooling2D(pool_size=(2, 2)))

    # the 2D output of a Conv Layer has to be flattened before going into a Dense Layer
    model.add(Flatten()) # Note: Keras does automatic shape inference.

    # Full Layer
    model.add(Dense(64, activation='sigmoid'))

    # Output Layer
    model.add(Dense(1, activation='sigmoid'))

    return model_____no_output_____model = simpleModel()
model.summary()_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 38, 98, 16)        160
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 19, 49, 16)        0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 17, 47, 32)        4640
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 8, 23, 32)         0
_________________________________________________________________
flatten_1 (Flatten)          (None, 5888)              0
_________________________________________________________________
dense_4 (Dense)              (None, 64)                376896
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 65
=================================================================
Total params: 381,761
Trainable params: 381,761
Non-trainable params: 0
_________________________________________________________________

## Training the CNN_____no_output_____

# Compiling the model
loss = 'binary_crossentropy'
optimizer = 'sgd'

model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])_____no_output_____# TRAINING the model
epochs = 15
history = model.fit(train_img, classes, batch_size=32, epochs=epochs)Epoch 1/15
1050/1050 [==============================] - 1s 1ms/step - loss: 0.6910 - acc: 0.5543
Epoch 2/15
1050/1050 [==============================] - 1s 721us/step - loss: 0.6514 - acc: 0.6286
Epoch 3/15
1050/1050 [==============================] - 1s 757us/step - loss: 0.6123 - acc: 0.7771
Epoch 4/15
1050/1050 [==============================] - 1s 757us/step - loss: 0.5548 - acc: 0.8438
Epoch 5/15
1050/1050 [==============================] - 1s 737us/step - loss: 0.4824 - acc: 0.8610
Epoch 6/15
1050/1050 [==============================] - 1s 729us/step - loss: 0.4132 - acc: 0.8895
Epoch 7/15
1050/1050 [==============================] - 1s 723us/step - loss: 0.3549 - acc: 0.8990
Epoch 8/15
1050/1050 [==============================] - 1s 725us/step - loss: 0.3117 - acc: 0.9029
Epoch 9/15
1050/1050 [==============================] - 1s 718us/step - loss: 0.2826 - acc: 0.9114
Epoch 10/15
1050/1050 [==============================] - 1s 724us/step - loss: 0.2569 - acc: 0.9162
Epoch 11/15
1050/1050 [==============================] - 1s 719us/step - loss: 0.2334 - acc: 0.9238
Epoch 12/15
1050/1050 [==============================] - 1s 719us/step - loss: 0.2131 - acc: 0.9333
Epoch 13/15
1050/1050 [==============================] - 1s 730us/step - loss: 0.1940 - acc: 0.9429
Epoch 14/15
1050/1050 [==============================] - 1s 751us/step - loss: 0.1771 - acc: 0.9533
Epoch 15/15
1050/1050 [==============================] - 1s 740us/step - loss: 0.1620 - acc: 0.9581

Again, our Accuracy rises quickly to almost 100%, with the CNN now even faster than with the Fully Connected Network.
But is our model really good at predicting unseen data?_____no_output_____### Training with Validation Data

We split off 10 % of the training data and use it as independend validation set to verify how good we are
on an independent data (not used for training) in each epoch_____no_output_____

# recreate and recompile the model (otherwise we continue learning)
model = simpleModel()
model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

# train with showing accuracy on split off validation data
history = model.fit(train_img, classes, batch_size=32, epochs=epochs, validation_split=0.1) # portion of val. dataTrain on 945 samples, validate on 105 samples
Epoch 1/15
945/945 [==============================] - 1s 1ms/step - loss: 0.6891 - acc: 0.5608 - val_loss: 0.6688 - val_acc: 0.6190
Epoch 2/15
945/945 [==============================] - 1s 765us/step - loss: 0.6514 - acc: 0.6381 - val_loss: 0.6513 - val_acc: 0.8190
Epoch 3/15
945/945 [==============================] - 1s 752us/step - loss: 0.6244 - acc: 0.8296 - val_loss: 0.6194 - val_acc: 0.7048
Epoch 4/15
945/945 [==============================] - 1s 761us/step - loss: 0.5870 - acc: 0.7894 - val_loss: 0.5848 - val_acc: 0.8571
Epoch 5/15
945/945 [==============================] - 1s 757us/step - loss: 0.5371 - acc: 0.8995 - val_loss: 0.5338 - val_acc: 0.8667
Epoch 6/15
945/945 [==============================] - 1s 756us/step - loss: 0.4769 - acc: 0.8995 - val_loss: 0.4763 - val_acc: 0.8762
Epoch 7/15
945/945 [==============================] - 1s 758us/step - loss: 0.4118 - acc: 0.9090 - val_loss: 0.4198 - val_acc: 0.8952
Epoch 8/15
945/945 [==============================] - 1s 765us/step - loss: 0.3531 - acc: 0.9196 - val_loss: 0.3730 - val_acc: 0.8762
Epoch 9/15
945/945 [==============================] - 1s 752us/step - loss: 0.3068 - acc: 0.9153 - val_loss: 0.3372 - val_acc: 0.9048
Epoch 10/15
945/945 [==============================] - 1s 752us/step - loss: 0.2742 - acc: 0.9217 - val_loss: 0.3102 - val_acc: 0.9143
Epoch 11/15
945/945 [==============================] - 1s 760us/step - loss: 0.2469 - acc: 0.9291 - val_loss: 0.2782 - val_acc: 0.9238
Epoch 12/15
945/945 [==============================] - 1s 755us/step - loss: 0.2231 - acc: 0.9354 - val_loss: 0.2548 - val_acc: 0.9238
Epoch 13/15
945/945 [==============================] - 1s 764us/step - loss: 0.2029 - acc: 0.9397 - val_loss: 0.2335 - val_acc: 0.9238
Epoch 14/15
945/945 [==============================] - 1s 747us/step - loss: 0.1842 - acc: 0.9503 - val_loss: 0.2111 - val_acc: 0.9333
Epoch 15/15
945/945 [==============================] - 1s 752us/step - loss: 0.1678 - acc: 0.9534 - val_loss: 0.1931 - val_acc: 0.9429

The results on the split-off validation data are quite high (usually similar, but not as high as on the training data)._____no_output_____### Using Test Set as Validation Set

Note: This usually is not recommended as during experimentation you will overfit also to the test data.

We show it here only for demonstration purposes to see how (bad) the validation accuracy is on our independet test data. The recommended way is to have a separate training, validation and test set (i.e. 3 splits or separate data sets)._____no_output_____

# recreate and recompile the model (otherwise we continue learning)
model = simpleModel()
model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

# show result on Test Data while training
# we use test data as validation data to see direct results (usually NOT RECOMMENDED due to overfitting to the problem!)
validation_data = (test_img, test_classes)

history = model.fit(train_img, classes, batch_size=32, epochs=epochs, validation_data=validation_data) Train on 1050 samples, validate on 170 samples
Epoch 1/15
1050/1050 [==============================] - 1s 1ms/step - loss: 0.6967 - acc: 0.5333 - val_loss: 0.6361 - val_acc: 0.9412
Epoch 2/15
1050/1050 [==============================] - 1s 778us/step - loss: 0.6556 - acc: 0.6400 - val_loss: 0.6766 - val_acc: 0.5647
Epoch 3/15
1050/1050 [==============================] - 1s 771us/step - loss: 0.6134 - acc: 0.7952 - val_loss: 0.6895 - val_acc: 0.4706
Epoch 4/15
1050/1050 [==============================] - 1s 778us/step - loss: 0.5612 - acc: 0.8333 - val_loss: 0.8912 - val_acc: 0.1353
Epoch 5/15
1050/1050 [==============================] - 1s 767us/step - loss: 0.4917 - acc: 0.8886 - val_loss: 1.0221 - val_acc: 0.1176
Epoch 6/15
1050/1050 [==============================] - 1s 782us/step - loss: 0.4177 - acc: 0.9048 - val_loss: 1.1996 - val_acc: 0.1118
Epoch 7/15
1050/1050 [==============================] - 1s 759us/step - loss: 0.3513 - acc: 0.9124 - val_loss: 1.4384 - val_acc: 0.1059
Epoch 8/15
1050/1050 [==============================] - 1s 779us/step - loss: 0.2969 - acc: 0.9229 - val_loss: 1.6691 - val_acc: 0.1059
Epoch 9/15
1050/1050 [==============================] - 1s 769us/step - loss: 0.2581 - acc: 0.9286 - val_loss: 1.6765 - val_acc: 0.1412
Epoch 10/15
1050/1050 [==============================] - 1s 768us/step - loss: 0.2254 - acc: 0.9381 - val_loss: 1.7660 - val_acc: 0.1471
Epoch 11/15
1050/1050 [==============================] - 1s 767us/step - loss: 0.2018 - acc: 0.9400 - val_loss: 1.9118 - val_acc: 0.1471
Epoch 12/15
1050/1050 [==============================] - 1s 762us/step - loss: 0.1812 - acc: 0.9410 - val_loss: 2.0423 - val_acc: 0.1353
Epoch 13/15
1050/1050 [==============================] - 1s 781us/step - loss: 0.1657 - acc: 0.9505 - val_loss: 2.0171 - val_acc: 0.1588
Epoch 14/15
1050/1050 [==============================] - 1s 770us/step - loss: 0.1507 - acc: 0.9552 - val_loss: 1.9424 - val_acc: 0.1765
Epoch 15/15
1050/1050 [==============================] - 1s 765us/step - loss: 0.1404 - acc: 0.9610 - val_loss: 1.8924 - val_acc: 0.1824

On the test set we perform really bad: less than 15% Accuracy!_____no_output_____### Verifying Accuracy on Test Set_____no_output_____

test_pred = model.predict_classes(test_img)
# show 35 first predictions
test_pred[0:35,0]_____no_output_____np.array(test_classes[0:35])_____no_output_____predictions = model.predict_classes(test_img)
accuracy_score(test_classes, predictions)_____no_output_____

## A more advanced CNN model

You may try to change the following to see the impact on the result:
* number of filters
* filter kernel size (e.g. 3 x 3, 5 x 5, ...)
* adding/not adding Batch Normalization
* adding/not adding ReLU Activation
* adding/not adding Max Pooling
* changing Pooling size (e.g. 1 x 2, 2 x 2, 2 x 1, or more)
* adding/changing/removing Dropout_____no_output_____

def advancedModel():
    model = Sequential()

    # this applies n_filters convolution filters of size 3x3 in each of the 2 layers below
    n_filters = 16
    kernel_size = (3, 3)

    # Layer 1
    model.add(Conv2D(n_filters, kernel_size, padding='valid', input_shape=input_shape))
    # input shape: 100x100 images with 3 channels -> input_shape should be (3, 100, 100)
    model.add(BatchNormalization())  # ON/OFF
    model.add(Activation('relu'))  # ReLu activation
    model.add(MaxPooling2D(pool_size=(2, 2))) # reducing image resolution by half
    model.add(Dropout(0.3))  # ON/OFF: random "deletion" of %-portion of units in each batch

    # Layer 2
    model.add(Conv2D(n_filters, kernel_size))  # input_shape is only needed in 1st layer
    #model.add(BatchNormalization()) # ON/OFF
    model.add(Activation('relu'))
    #model.add(MaxPooling2D(pool_size=(2, 2)))  # ON/OFF
    model.add(Dropout(0.3))  # ON/OFF

    # Flatten
    model.add(Flatten())

    # Fully Connected Layer
    model.add(Dense(256))
    model.add(Activation('relu'))
    model.add(Dropout(0.1))

    # Output Layer
    model.add(Dense(1, activation='sigmoid'))

    return model_____no_output_____model = advancedModel()
model.summary()_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_7 (Conv2D)            (None, 38, 98, 16)        160
_________________________________________________________________
batch_normalization_1 (Batch (None, 38, 98, 16)        64
_________________________________________________________________
activation_1 (Activation)    (None, 38, 98, 16)        0
_________________________________________________________________
max_pooling2d_7 (MaxPooling2 (None, 19, 49, 16)        0
_________________________________________________________________
dropout_1 (Dropout)          (None, 19, 49, 16)        0
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 17, 47, 16)        2320
_________________________________________________________________
activation_2 (Activation)    (None, 17, 47, 16)        0
_________________________________________________________________
dropout_2 (Dropout)          (None, 17, 47, 16)        0
_________________________________________________________________
flatten_4 (Flatten)          (None, 12784)             0
_________________________________________________________________
dense_10 (Dense)             (None, 256)               3272960
_________________________________________________________________
activation_3 (Activation)    (None, 256)               0
_________________________________________________________________
dropout_3 (Dropout)          (None, 256)               0
_________________________________________________________________
dense_11 (Dense)             (None, 1)                 257
=================================================================
Total params: 3,275,761
Trainable params: 3,275,729
Non-trainable params: 32
_________________________________________________________________

## Training the CNN_____no_output_____

# Compiling the model
loss = 'binary_crossentropy'
optimizer = 'sgd'
#optimizer = SGD(lr=0.001)  # possibility to adapt the learn rate

model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])_____no_output_____# TRAINING the model
epochs = 15
history = model.fit(train_img, classes, batch_size=32, epochs=epochs)Epoch 1/15
1050/1050 [==============================] - 3s 3ms/step - loss: 0.5270 - acc: 0.8010
Epoch 2/15
1050/1050 [==============================] - 2s 2ms/step - loss: 0.1915 - acc: 0.9219
Epoch 3/15
1050/1050 [==============================] - 2s 2ms/step - loss: 0.1283 - acc: 0.9533
Epoch 4/15
1050/1050 [==============================] - 2s 2ms/step - loss: 0.0873 - acc: 0.9676
Epoch 5/15
1050/1050 [==============================] - 2s 2ms/step - loss: 0.0517 - acc: 0.9876
Epoch 6/15
1050/1050 [==============================] - 2s 2ms/step - loss: 0.0508 - acc: 0.9867
Epoch 7/15
1050/1050 [==============================] - 2s 2ms/step - loss: 0.0443 - acc: 0.9876
Epoch 8/15
1050/1050 [==============================] - 2s 2ms/step - loss: 0.0300 - acc: 0.9924
Epoch 9/15
1050/1050 [==============================] - 2s 2ms/step - loss: 0.0394 - acc: 0.9829
Epoch 10/15
1050/1050 [==============================] - 2s 2ms/step - loss: 0.0253 - acc: 0.9924A: 0s - loss: 0.0244 - acc: 0.992
Epoch 11/15
1050/1050 [==============================] - 2s 2ms/step - loss: 0.0198 - acc: 0.9943
Epoch 12/15
1050/1050 [==============================] - 2s 2ms/step - loss: 0.0184 - acc: 0.9962
Epoch 13/15
1050/1050 [==============================] - 2s 2ms/step - loss: 0.0250 - acc: 0.9914
Epoch 14/15
1050/1050 [==============================] - 2s 2ms/step - loss: 0.0144 - acc: 0.9962
Epoch 15/15
1050/1050 [==============================] - 2s 2ms/step - loss: 0.0186 - acc: 0.9952

### Plotting the Training Curve

The `model.fit` function returns a history including the evolution of training and validation loss and accuracy. We can plot it to see a nice training curve._____no_output_____

hist = history.history
hist.keys()_____no_output_____# this will only work if you have matplotlib installed

import matplotlib.pyplot as plt
%matplotlib inline

colors = {'loss':'r', 'acc':'b', 'val_loss':'m', 'val_acc':'g'}
plt.figure(figsize=(10,6))
plt.title("Training Curve")
plt.xlabel("Epoch")

for measure in hist.keys():
    color = colors[measure]
    plt.plot(range(1,epochs+1), hist[measure], color + '-', label=measure)  # use last 2 values to draw line

plt.legend(loc='upper left', scatterpoints = 1, frameon=False)
_____no_output_____

#### Conclusion: We are completely overfitting!
While we achieve nearly 100% on the Training Set, the generalization to the Test Set is really bad, with an Accuracy of about only 10%._____no_output_____# Data Augmentation

Increase the training set by adding more images: Rotate, shift, flip and scale the original images to generate additional examples that will help the Neural Network to generalize better._____no_output_____

from keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)_____no_output_____# ImageDataGenerator needs the classes as Numpy array instead of normal list
classes_array = np.array(classes)_____no_output_____np.random.seed(0) # enforce repeatable result_____no_output_____batch_size = 16
steps_per_epoch = len(train_img) / batch_size

int(steps_per_epoch)_____no_output_____# recreate and recompile the model (otherwise we continue learning)
model = advancedModel()
model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

batch_size = 32
# steps_per_epoch is the number of batches to draw from the generator at each epoch.
# suggestion: samples_per_epoch(= full train set)/batch_size
steps_per_epoch = len(train_img) / batch_size

# fits the model on batches with real-time data augmentation:
history = model.fit_generator(datagen.flow(train_img, classes_array, batch_size=batch_size),
                    steps_per_epoch=steps_per_epoch, epochs=epochs,
                    validation_data=validation_data) Epoch 1/15
33/32 [==============================] - 3s 101ms/step - loss: 0.6978 - acc: 0.5846 - val_loss: 0.5980 - val_acc: 0.6882
Epoch 2/15
33/32 [==============================] - 2s 71ms/step - loss: 0.5960 - acc: 0.6740 - val_loss: 0.7219 - val_acc: 0.5765
Epoch 3/15
33/32 [==============================] - 2s 71ms/step - loss: 0.5750 - acc: 0.7037 - val_loss: 0.8361 - val_acc: 0.4824
Epoch 4/15
33/32 [==============================] - 2s 72ms/step - loss: 0.5290 - acc: 0.7295 - val_loss: 1.3836 - val_acc: 0.1647
Epoch 5/15
33/32 [==============================] - 2s 70ms/step - loss: 0.5082 - acc: 0.7375 - val_loss: 0.8696 - val_acc: 0.4353
Epoch 6/15
33/32 [==============================] - 2s 70ms/step - loss: 0.4963 - acc: 0.7588 - val_loss: 0.8090 - val_acc: 0.4941
Epoch 7/15
33/32 [==============================] - 2s 72ms/step - loss: 0.4518 - acc: 0.7947 - val_loss: 0.5715 - val_acc: 0.6588
Epoch 8/15
33/32 [==============================] - 2s 72ms/step - loss: 0.4012 - acc: 0.8148 - val_loss: 1.7280 - val_acc: 0.1529
Epoch 9/15
33/32 [==============================] - 2s 69ms/step - loss: 0.3849 - acc: 0.8232 - val_loss: 0.9896 - val_acc: 0.4706
Epoch 10/15
33/32 [==============================] - 2s 71ms/step - loss: 0.3617 - acc: 0.8419 - val_loss: 0.3244 - val_acc: 0.8294
Epoch 11/15
33/32 [==============================] - 2s 70ms/step - loss: 0.3609 - acc: 0.8337 - val_loss: 1.0366 - val_acc: 0.4824
Epoch 12/15
33/32 [==============================] - 2s 70ms/step - loss: 0.3188 - acc: 0.8611 - val_loss: 0.6042 - val_acc: 0.6588
Epoch 13/15
33/32 [==============================] - 2s 69ms/step - loss: 0.2923 - acc: 0.8783 - val_loss: 0.3665 - val_acc: 0.7941
Epoch 14/15
33/32 [==============================] - 2s 70ms/step - loss: 0.2934 - acc: 0.8647 - val_loss: 0.2638 - val_acc: 0.8941
Epoch 15/15
33/32 [==============================] - 2s 69ms/step - loss: 0.3433 - acc: 0.8459 - val_loss: 0.7712 - val_acc: 0.6471

### Verifying Accuracy on Test Set (with Data Augmentation)_____no_output_____

test_pred = model.predict_classes(test_img)
test_pred[0:35,0] # show 35 first predictions_____no_output_____predictions = model.predict_classes(test_img)
accuracy_score(test_classes, predictions)_____no_output_____

Conclusion: With Data Augmentation, the model has more diverse training examples and generalizes much better. The Accuracy on the Test Set increases from less than 10% to more than 55% up to 75%!_____no_output_____### Plotting the Training Curve with Data Augmentation _____no_output_____

hist = history.history

plt.figure(figsize=(10,6))
plt.title("Training Curve")
plt.xlabel("Epoch")

for measure in hist.keys():
    color = colors[measure]
    plt.plot(range(1,epochs+1), hist[measure], color + '-', label=measure)  # use last 2 values to draw line

plt.legend(loc='upper left', scatterpoints = 1, frameon=False)
_____no_output_____

This graph is not ideal, but we see both training (blue) and validation (green) Accuracy going up, the training loss going continuously down and the validation loss fluctuating, but not exploding like previously._____no_output_____# Home Exercises 1

* **Exercise 1.1)** Adjust the parameters mentioned under "A more advanced CNN model" in the code block of "def advancedModel():" above: Tune the parameters and find a model configuration that performs better on the Test Set.
* **Exercise 1.2)** Try to improve the Data Augmentation further by adjusting the parameters of the ImageDataGenerator and/or the batch_size and/or steps_per_epoch.

For both exercises include:
* the code block with the parametrs of the model / augmentation you adapted
* the best achieved validation accuracy + valiation loss
* the plot of the training curve_____no_output_____