# Notebook from AbhishekTyagi404/Kaggle_Competition
Path: Movie_recommendation_system/movie_recommendation_system2.ipynb

<code>
# import pandas library
import pandas as pd

# Get the data
column_names = ['user_id', 'item_id', 'rating', 'timestamp']

path = 'file.tsv'

df = pd.read_csv(path, sep='\t', names=column_names)

# Check the head of the data
df.head()
_____no_output_____# Check out all the movies and their respective IDs
movie_titles = pd.read_csv('Movie_Id_Titles.csv')
movie_titles.head(30)
_____no_output_____data = pd.merge(df, movie_titles, on='item_id')
data.head() _____no_output_____# Calculate mean rating of all movies
data.groupby('title')['rating'].mean().sort_values(ascending=False).head() _____no_output_____# Calculate count rating of all movies
data.groupby('title')['rating'].count().sort_values(ascending=False).head() _____no_output_____# creating dataframe with 'rating' count values
ratings = pd.DataFrame(data.groupby('title')['rating'].mean())

ratings['num of ratings'] = pd.DataFrame(data.groupby('title')['rating'].count())

ratings.head()
_____no_output_____import matplotlib.pyplot as plt
import seaborn as sns

sns.set_style('white')
%matplotlib inline
_____no_output_____# plot graph of 'num of ratings column'
plt.figure(figsize =(10, 4))

ratings['num of ratings'].hist(bins = 70)
_____no_output_____# plot graph of 'ratings' column
plt.figure(figsize =(10, 4))

ratings['rating'].hist(bins = 70)
_____no_output_____# Sorting values according to
# the 'num of rating column'
moviemat = data.pivot_table(index ='user_id',
			columns ='title', values ='rating')

moviemat.head()

ratings.sort_values('num of ratings', ascending = False).head(10)
_____no_output_____# analysing correlation with similar movies
starwars_user_ratings = moviemat['Star Wars (1977)']
liarliar_user_ratings = moviemat['Liar Liar (1997)']

starwars_user_ratings.head()
_____no_output_____# analysing correlation with similar movies
similar_to_starwars = moviemat.corrwith(starwars_user_ratings)
similar_to_liarliar = moviemat.corrwith(liarliar_user_ratings)

corr_starwars = pd.DataFrame(similar_to_starwars, columns =['Correlation'])
corr_starwars.dropna(inplace = True)

corr_starwars.head()
A:\Anaconda3\envs\Vision\lib\site-packages\numpy\lib\function_base.py:2522: RuntimeWarning: Degrees of freedom <= 0 for slice
  c = cov(x, y, rowvar)
A:\Anaconda3\envs\Vision\lib\site-packages\numpy\lib\function_base.py:2451: RuntimeWarning: divide by zero encountered in true_divide
  c *= np.true_divide(1, fact)
# Similar movies like starwars
corr_starwars.sort_values('Correlation', ascending = False).head(10)
corr_starwars = corr_starwars.join(ratings['num of ratings'])

corr_starwars.head()

corr_starwars[corr_starwars['num of ratings']>100].sort_values('Correlation', ascending = False).head()
_____no_output_____# Similar movies as of liarliar
corr_liarliar = pd.DataFrame(similar_to_liarliar, columns =['Correlation'])
corr_liarliar.dropna(inplace = True)

corr_liarliar = corr_liarliar.join(ratings['num of ratings'])
corr_liarliar[corr_liarliar['num of ratings']>100].sort_values('Correlation', ascending = False).head()
_____no_output_____
</code>


# Notebook from DataScienceUB/DeepLearningMaster2019-2020
Path: 9. Convolutional Neural Networks.ipynb

_____no_output_____# 9. Convolutional Neural Networks

> A Convolutional Neural Networks is a NN which has at least one layer (``tf.nn.conv2d`` in ``tf``) that does a convolution between its input $f$ and a configurable kernel $k$, $(f*k)$.

Convolutional Neural Networks have been some of the most influential innovations in the field of **computer vision** in the last years.

When considering the analysis of an image with a computer we must define a computational representation of an image. To this end, images are represented as $n \times m \times 3$ array of numbers, called **pixels**.
The 3 refers to RGB values and $n, m$ refers to the height and width of the image in pixels. Each number in this array is given a value from 0 to 255 which describes the pixel intensity at that point.  These numbers are the only inputs available to the computer.

**What is to classify an image?** The idea is that you give the computer this array of numbers and it must output numbers that describe the probability of the image being a certain class.

Of course, this kind of image representation is suited to be classified by a classical neural network composed of **dense layers**, but this approach has **several limitations**.

The first one is that large images with a high number of pixels will need from **extremely large networks** to be analyzed. If an image has $256 \times 256 = 65,536$ pixels, the first layer of a classical neural network needs to have $65,536 \times 65,536 = 4,294,967,296$ different weights to consider all pixel interactions. Even in the case that this number of weights could be stored in the available memory, learning these weights would be very time consuming. But there are better alternatives.

One possible solution for this problem is to consider **local receptive fields (LRF)** (an inspiration that comes from neuroscience). A local receptive field is a standard MLP with a local view of the image: its input is a submatrix $M \times M$ of the original $N \times N$, where $M << N$.

Using LRF is like decomposing the problem of image analysis in a set of local *experts* that are specialized in certain parts of the image. The outcome of these specialists can be integrated at a high level of the model.

> Could you propose a problem where a LRF architecture could be the best option?

Let's suppose we are dealing with a common task: object detection.

In the most general setting, images represent the field of view of a camera (robot, autonomous car, person, etc.). Hence, we cannot assume any prior information about the presence of the object of interest on the image (i.e. any prior information about its scale, point of view, etc. or even its localization with respect to image coordinates).

> An object may appear at any image coordinates!

In this case LRF are not suited for this task because in order to learn a detector we should provide the model with a lot of examples in order to consider all possible object localitions!

But, what about sharing LRF weights?

Natural images are not a random combination of values in a $N \times M$ array, but present **strong correlations** at different levels.

At the most basic, it is evident that the **value of a pixel** is not independent of the values of its neighboring pixels.

Moreover, natural images present another interesting property: **location invariance**. That means that visual structures, such as a cat or a dog, can be present on any place of the image at any scale. Image location is not important, what is important for attaching a meaning to an image are the **relative positions of geometric and photometric structures**.

All this considerations leaded, partially inspired by biological models, to the proposal of a very special kind of layers: those based on **convolutions**.

> A convolution is a mathematical operation that combines two input images to form a third one. One of the input images is the image we want to process. The other one, that is smaller, is called the kernel.

Let's suppose that our kernel is this one:

$$ Kernel = \left( \begin{array}{rrr} 0 & -1 & 0 \\ -1 & 5 & -1 \\  0 & -1 & 0 \end{array} \right)$$

The output of image convolution is calculated as follows:

- Flip the kernel both horizontally and vertically. As our selected kernel is symmetric, the flipped kernel is equal to the original.
- Put the first element of the kernel at every element of the image matrix. Multiply each element of the kernel with its corresponding element of the image matrix (the one which is overlapped with it).
- Sum up all product outputs and put the result at the same position in the output matrix as the center of kernel in image matrix.
_____no_output_____## What is a convolution?

Mathematically, given a convolution kernel $K$ represented by a $(M \times N)$ array, the convolution of an image $I$ with $K$ is:

$$
output(x,y) = (I \otimes K)(x,y) = \sum_{m=0}^{M-1} \sum_{n=1}^{N-1} K(m,n) I(x-n, y-m)
$$

The output of image convolution is another image that might represent some kind of information that was present in the image in a very subtle way. For example, the kernel we have used is called an **edge detector** because it highlights the edges of visual structures and attenuates smooth regions.

> [Convolution visualization](http://deeplearning.net/software/theano_versions/dev/_images/numerical_no_padding_no_strides.gif)

> Demo: http://setosa.io/ev/image-kernels/

In **convolutional neural networks** the values of the kernel matrix are **free parameters** that must be learned to perform the optimal information extraction in order to classify the image.

Convolutions are linear operators and because of this the application of successive convolutions can always be represented by a single convolution.

$$
((I \otimes K_1) \otimes K_2) = (I \otimes (K_1 \otimes K_2))
$$

But if we apply a non linear activation function after each convolution the application of successive convolution operators makes sense and results in a powerful image feature structure. _____no_output_____

import tensorflow as tf
import numpy as np
from imageio import imread
from skimage.transform import resize
import matplotlib.pyplot as plt
%matplotlib inline

sample_image = imread("https://raw.githubusercontent.com/DataScienceUB/DeepLearningMaster2019/master/images/result.png")
sample_image = sample_image.astype(float)

size = sample_image.shape
print("sample image shape: "+ str(sample_image.shape))

def show(image):
    image = np.squeeze(image.astype("uint8"))
    plt.imshow(image, cmap="gray")

show(sample_image)sample image shape: (256, 209, 4)
sample_image= sample_image[:,:,:-1]
sample_image.shape_____no_output_____

#### Example

Let's build a convolution filter that blurs the image using `tf.nn.depthwise_conv2d`. This function treats each channel separately. If not, it would be a 3D convolution!

Given an input tensor of shape ``[batch,  in_height, in_width, in_channels]`` and a filter tensor of shape ``[filter_height, filter_width, in_channels, channel_multiplier]`` containing ``in_channels`` convolutional filters of depth 1, ``depthwise_conv2d`` applies a different filter to each input channel (expanding from 1 channel to ``channel_multiplier`` channels for each), then concatenates the results together. _____no_output_____

image = tf.placeholder(tf.float32, shape=(None, None, None, 3))
kernel = tf.placeholder(tf.float32, shape=(5, 5, 3))

def conv(x, k):
    k = tf.reshape(k, shape=(5, 5, 3, 1))
    return tf.nn.depthwise_conv2d(x,
                                  k,
                                  strides=(1, 1, 1, 1),
                                  padding='SAME')

kernel_data = np.zeros(shape=(5, 5, 3)).astype(np.float32)
kernel_data[:, :, :] = 1 / 25.0

output_image = conv(image, kernel)

# move the channel dimension to the first dimension to
# make it easy to see the spacial organization of the kernel
# on the last 2 dimensions with print:
print(np.transpose(kernel_data, (2, 0, 1))) [[[0.04 0.04 0.04 0.04 0.04]
  [0.04 0.04 0.04 0.04 0.04]
  [0.04 0.04 0.04 0.04 0.04]
  [0.04 0.04 0.04 0.04 0.04]
  [0.04 0.04 0.04 0.04 0.04]]

 [[0.04 0.04 0.04 0.04 0.04]
  [0.04 0.04 0.04 0.04 0.04]
  [0.04 0.04 0.04 0.04 0.04]
  [0.04 0.04 0.04 0.04 0.04]
  [0.04 0.04 0.04 0.04 0.04]]

 [[0.04 0.04 0.04 0.04 0.04]
  [0.04 0.04 0.04 0.04 0.04]
  [0.04 0.04 0.04 0.04 0.04]
  [0.04 0.04 0.04 0.04 0.04]
  [0.04 0.04 0.04 0.04 0.04]]]
print(kernel_data) [[[0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]]

 [[0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]]

 [[0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]]

 [[0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]]

 [[0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]
  [0.04 0.04 0.04]]]
with tf.Session() as sess:
    feed_dict = {image: [sample_image], kernel: kernel_data}
    conv_img = sess.run(output_image, feed_dict=feed_dict)
    print(conv_img.shape)
    show(conv_img[0])(1, 256, 209, 3)

#### Example

``tf.nn.conv2d``: Given an input tensor of shape ``[batch, in_height, in_width, in_channels]`` and a filter / kernel tensor of shape ``[filter_height, filter_width, in_channels, out_channels]``, this op performs the following:

+ Flattens the filter to a 2-D matrix with shape ``[filter_height * filter_width * in_channels, output_channels]``.
+ Extracts image patches from the input tensor to form a virtual tensor of shape ``[batch, out_height, out_width, filter_height * filter_width * in_channels]``.
+ For each patch, right-multiplies the filter matrix and the image patch vector._____no_output_____

image = tf.placeholder(tf.float32, shape=(None, None, None, 3))
kernel = tf.placeholder(tf.float32, shape=(5, 5, 3))

def conv(x, k):
    k = tf.reshape(k, shape=(5, 5, 3, 1))
    return tf.nn.conv2d(x,
                        k,
                        strides=(1, 1, 1, 1),
                        padding='SAME')

output_image = conv(image, kernel)
kernel_data = np.zeros(shape=(5, 5, 3)).astype(np.float32)
kernel_data[:, :, :] = 1 / 125.0

# move the channel dimension to the first dimension to
# make it easy to see the spacial organization of the kernel
# on the last 2 dimensions with print:
print(np.transpose(kernel_data, (2, 0, 1))) [[[0.008 0.008 0.008 0.008 0.008]
  [0.008 0.008 0.008 0.008 0.008]
  [0.008 0.008 0.008 0.008 0.008]
  [0.008 0.008 0.008 0.008 0.008]
  [0.008 0.008 0.008 0.008 0.008]]

 [[0.008 0.008 0.008 0.008 0.008]
  [0.008 0.008 0.008 0.008 0.008]
  [0.008 0.008 0.008 0.008 0.008]
  [0.008 0.008 0.008 0.008 0.008]
  [0.008 0.008 0.008 0.008 0.008]]

 [[0.008 0.008 0.008 0.008 0.008]
  [0.008 0.008 0.008 0.008 0.008]
  [0.008 0.008 0.008 0.008 0.008]
  [0.008 0.008 0.008 0.008 0.008]
  [0.008 0.008 0.008 0.008 0.008]]]
with tf.Session() as sess:
    feed_dict = {image: [sample_image], kernel: kernel_data}
    conv_img = sess.run(output_image, feed_dict=feed_dict)
    print(conv_img.shape)
    show(conv_img[0])(1, 256, 209, 1)

### Exercise

+ What happens if we build a 3x3 kernel with stride 2? What is the size of the output image?

+ When a kernel is overlapped on an image it should be set to fit within the bounds of the image. Filling the missing area of the image is known as *padding* the image. The padding parameter must be ``SAME``  if we want the convolution output to be the same size as the input, and ``VALID`` if we want only the useful part of the output.
What happens if we change the padding to 'VALID'?_____no_output_____

# your code here

_____no_output_____

## Convolutional Neural Netwoks

The kernel weights can be learn from examples if we define a loss function.

> CNN are composed of repetitive blocks of neurons that are applied across
space (for images) or time (for audio signals).

> For images, these blocks of neurons can be interpreted as
2D convolutional kernels, repeatedly applied over each
patch of the image.

> For speech, they can be seen as the 1D convolutional
kernels applied across time-windows.

At training time, the weights for these repeated blocks are
'shared', i.e. the weight gradients learned over various
image patches are averaged._____no_output_____CNN where originally proposed in early 90's by Y.LeCun:

LeNet architecture used one additional ingredient: **max pooling**. Pooling is a way of sub-sampling, i.e. reducing the dimension of the input. It is usually done after convolutional layers.

It is also useful since it provides a form of translation invariance. _____no_output_____## MNIST Convolutional Network

A simple CNN architecture combines a convolutional layer (``tf.nn.conv2d``), a non-linear activation function (``tf.nn.relu``), a pooling layer (``tf.nn.max_pool``) and a fully connected layer (``tf.matmul``).

Max Pooling. Source: http://cs231n.github.io/convolutional-networks/

The ``tf`` input pipeline is designed to work with multiple images in a batch: `` [image_batch_size, image_height, image_width, image_channels] ``. _____no_output_____

# Adapted notebook from Author: Aymeric Damien
# Project: https://github.com/aymericdamien/TensorFlow-Examples/

# Import MINST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("tmp/data/", one_hot=True)Extracting tmp/data/train-images-idx3-ubyte.gz
Extracting tmp/data/train-labels-idx1-ubyte.gz
Extracting tmp/data/t10k-images-idx3-ubyte.gz
Extracting tmp/data/t10k-labels-idx1-ubyte.gz
import tensorflow as tf

# Parameters
learning_rate = 0.01
training_iters = 100000
batch_size = 128
display_step = 20_____no_output_____# Network Parameters
n_input = 784 # MNIST data input (img shape: 28*28)
n_classes = 10 # MNIST total classes (0-9 digits)
dropout = 0.75 # Dropout, probability to keep units_____no_output_____# tf Graph input
x = tf.placeholder(tf.float32, [None, n_input])
y = tf.placeholder(tf.float32, [None, n_classes])
keep_prob = tf.placeholder(tf.float32) #dropout (keep probability)_____no_output_____

The parameter ``strides`` causes a kernel to skip over pixels of an image and not to include them in the output.

Strides are a way to adjust the dimensionality of input tensors (to reduce processing power). This allows the design of CNN with small kernels but getting information from large parts of the input. _____no_output_____

# Create model
def conv2d(img, w, b):
    return tf.nn.relu(tf.nn.bias_add(tf.nn.conv2d(img, w, strides=[1, 1, 1, 1],
                                                  padding='SAME'),b))

def max_pool(img, k):
    return tf.nn.max_pool(img, ksize=[1, k, k, 1], strides=[1, k, k, 1], padding='SAME')

def conv_net(_X, _weights, _biases, _dropout):
    # Reshape input picture
    _X = tf.reshape(_X, shape=[-1, 28, 28, 1])

    # Convolution Layer
    conv1 = conv2d(_X, _weights['wc1'], _biases['bc1'])
    # Max Pooling (down-sampling)
    conv1 = max_pool(conv1, k=2)
    # Apply Dropout
    conv1 = tf.nn.dropout(conv1, _dropout)

    # Convolution Layer
    conv2 = conv2d(conv1, _weights['wc2'], _biases['bc2'])
    # Max Pooling (down-sampling)
    conv2 = max_pool(conv2, k=2)
    # Apply Dropout
    conv2 = tf.nn.dropout(conv2, _dropout)

    # Fully connected layer
    # Reshape conv2 output to fit dense layer input
    dense1 = tf.reshape(conv2, [-1, _weights['wd1'].get_shape().as_list()[0]])
    # Relu activation
    dense1 = tf.nn.relu(tf.add(tf.matmul(dense1, _weights['wd1']), _biases['bd1']))
    # Apply Dropout
    dense1 = tf.nn.dropout(dense1, _dropout) # Apply Dropout

    # Output, class prediction
    out = tf.add(tf.matmul(dense1, _weights['out']), _biases['out'])
    return out_____no_output_____# Store layers weight & bias
weights = {
    # 5x5 conv, 1 input, 32 outputs
    'wc1': tf.Variable(tf.random_normal([5, 5, 1, 32])),
    # 5x5 conv, 32 inputs, 64 outputs
    'wc2': tf.Variable(tf.random_normal([5, 5, 32, 64])),
    # fully connected, 7*7*64 inputs, 512 outputs
    'wd1': tf.Variable(tf.random_normal([7*7*64, 512])),
    # 512 inputs, 10 outputs (class prediction)
    'out': tf.Variable(tf.random_normal([512, n_classes]))
}

biases = {
    'bc1': tf.Variable(tf.random_normal([32])),
    'bc2': tf.Variable(tf.random_normal([64])),
    'bd1': tf.Variable(tf.random_normal([512])),
    'out': tf.Variable(tf.random_normal([n_classes]))
}_____no_output_____# Construct model
pred = conv_net(x, weights, biases, keep_prob)_____no_output_____# Define loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2
                      (logits= pred, labels= y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)_____no_output_____# Evaluate model
correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))_____no_output_____# Initializing the variables
init = tf.global_variables_initializer()_____no_output_____# Launch the graph
with tf.Session() as sess:
    sess.run(init)
    step = 1
    # Keep training until reach max iterations
    while step * batch_size < training_iters:
        batch_xs, batch_ys = mnist.train.next_batch(batch_size)
        # Fit training using batch data
        sess.run(optimizer,
                 feed_dict={x: batch_xs, y: batch_ys, keep_prob: dropout})
        if step % display_step == 0:
            # Calculate batch accuracy
            acc = sess.run(accuracy,
                           feed_dict={x: batch_xs, y: batch_ys, keep_prob: 1.})
            # Calculate batch loss
            loss = sess.run(cost,
                            feed_dict={x: batch_xs, y: batch_ys, keep_prob: 1.})
            print("Iter ", str(step*batch_size),
                  "| Minibatch Loss= {:.6f}".format(loss)  ,
                  "| Training Accuracy= {:.5f}".format(acc))
        step += 1
    print("Optimization Finished!")
    # Calculate accuracy for 256 mnist test images
    print("Testing Accuracy:",
          sess.run(accuracy,
                   feed_dict={x: mnist.test.images[:256],
                   y: mnist.test.labels[:256],
                   keep_prob: 1.}))Iter  2560 | Minibatch Loss= 2423.555176 | Training Accuracy= 0.58594
Iter  5120 | Minibatch Loss= 689.561096 | Training Accuracy= 0.73438
Iter  7680 | Minibatch Loss= 395.890930 | Training Accuracy= 0.85156
Iter  10240 | Minibatch Loss= 289.860046 | Training Accuracy= 0.85156
Iter  12800 | Minibatch Loss= 243.940140 | Training Accuracy= 0.85156
Iter  15360 | Minibatch Loss= 136.920807 | Training Accuracy= 0.89062
Iter  17920 | Minibatch Loss= 175.171494 | Training Accuracy= 0.89062
Iter  20480 | Minibatch Loss= 90.708237 | Training Accuracy= 0.94531
Iter  23040 | Minibatch Loss= 229.978210 | Training Accuracy= 0.82031
Iter  25600 | Minibatch Loss= 116.338692 | Training Accuracy= 0.90625
Iter  28160 | Minibatch Loss= 278.184509 | Training Accuracy= 0.88281
Iter  30720 | Minibatch Loss= 77.760223 | Training Accuracy= 0.93750
Iter  33280 | Minibatch Loss= 198.099701 | Training Accuracy= 0.89062
Iter  35840 | Minibatch Loss= 89.329224 | Training Accuracy= 0.91406
Iter  38400 | Minibatch Loss= 51.937981 | Training Accuracy= 0.93750
Iter  40960 | Minibatch Loss= 83.168900 | Training Accuracy= 0.96094
Iter  43520 | Minibatch Loss= 104.349327 | Training Accuracy= 0.92969
Iter  46080 | Minibatch Loss= 108.106598 | Training Accuracy= 0.92188
Iter  48640 | Minibatch Loss= 38.267960 | Training Accuracy= 0.96094
Iter  51200 | Minibatch Loss= 45.728611 | Training Accuracy= 0.96094
Iter  53760 | Minibatch Loss= 70.274506 | Training Accuracy= 0.94531
Iter  56320 | Minibatch Loss= 41.208733 | Training Accuracy= 0.97656
Iter  58880 | Minibatch Loss= 53.943806 | Training Accuracy= 0.95312
Iter  61440 | Minibatch Loss= 76.221741 | Training Accuracy= 0.92969
Iter  64000 | Minibatch Loss= 21.152348 | Training Accuracy= 0.96094
Iter  66560 | Minibatch Loss= 23.621267 | Training Accuracy= 0.97656
Iter  69120 | Minibatch Loss= 25.055222 | Training Accuracy= 0.95312
Iter  71680 | Minibatch Loss= 8.120686 | Training Accuracy= 0.97656
Iter  74240 | Minibatch Loss= 54.463661 | Training Accuracy= 0.94531
Iter  76800 | Minibatch Loss= 43.605202 | Training Accuracy= 0.96875
Iter  79360 | Minibatch Loss= 50.153397 | Training Accuracy= 0.96094
Iter  81920 | Minibatch Loss= 45.550892 | Training Accuracy= 0.95312
Iter  84480 | Minibatch Loss= 28.427061 | Training Accuracy= 0.96875
Iter  87040 | Minibatch Loss= 78.017303 | Training Accuracy= 0.92188
Iter  89600 | Minibatch Loss= 29.417255 | Training Accuracy= 0.96094
Iter  92160 | Minibatch Loss= 13.598746 | Training Accuracy= 0.96875
Iter  94720 | Minibatch Loss= 34.922489 | Training Accuracy= 0.98438
Iter  97280 | Minibatch Loss= 26.119286 | Training Accuracy= 0.94531
Iter  99840 | Minibatch Loss= 32.713379 | Training Accuracy= 0.95312
Optimization Finished!
Testing Accuracy: 0.9765625

Some CNN architectures use **normalization** layers.

For example, ``tf.nn.local_response_normalization`` is a function which shapes the output based on a summation operation in the following way: within a given vector, each component is divided by the weighted, squared sum of inputs within a depth radius.

> In neurobiology, there is a concept called “lateral inhibition”. This refers to the capacity of an excited neuron to subdue its neighbors.

In fact, after a convolutional layer there are two kinds of non linear functions that are usually applied: non-linear activation functions such as sigmoids or **ReLU and pooling**. Pooling layers are used with the purpose to progressively reduce the spatial size of the image to achieve scale invariance.The most common layer is the maxpool layer. Basically a maxpool of $2 \times 2$ causes a filter of 2 by 2 to traverse over the entire input array and pick the largest element from the window to be included in the next representation map. Pooling can also be implemented by using other criteria, such as averaging instead of taking the max element.

A convolutional neural network is a neural network that is build by using several convolutional layers, each one formed by the concatenation of three different operators: **convolutions, non-linear activation and pooling**. This kind of networks are able of extracting powerful image descriptors when applied in sequence. The power of the method has been credited to the fact that these descriptors can be seen as hierarchical features that are suited to optimally represent visual structures in natural images. The last layers of a convolutional neural network are classical dense layers, which are connected to a classification or regression loss function.

Finally, it is interesting to point out that convolutional layers are much **lighter**, in terms of number of weights, than fully connected layers, but more computationally demanding (The number of weights we must learn for a $(M \times N)$ convolution kernel is only $(M \times N)$, which is independent of the size of the image).

In some sense, **convolutional layers trade weights for computation when extracting information**.
_____no_output_____## Visualization of learned kernels.

Convolution kernels from the first AlexNet layer.

Source: http://cs231n.github.io/convolutional-networks/
_____no_output_____## Exercise:

**VGGNet**. The runner-up in ILSVRC 2014 was the network from Karen Simonyan and Andrew Zisserman that became known as the VGGNet. Its main contribution was in showing that the depth of the network is a critical component for good performance. Their final best network contains 16 CONV/FC layers and, appealingly, features an extremely homogeneous architecture that only performs $3 \times 3$ convolutions and $2 \times 2$ pooling from the beginning to the end.

The architecture is:

+ INPUT: [224x224x3]
+ CONV3-64: [224x224x64]
+ CONV3-64: [224x224x64]
+ POOL2: [112x112x64]
+ CONV3-128: [112x112x128]
+ CONV3-128: [112x112x128]
+ POOL2: [56x56x128]
+ CONV3-256: [56x56x256]
+ CONV3-256: [56x56x256]
+ CONV3-256: [56x56x256]
+ POOL2: [28x28x256]
+ CONV3-512: [28x28x512]
+ CONV3-512: [28x28x512]
+ CONV3-512: [28x28x512]
+ POOL2: [14x14x512]
+ CONV3-512: [14x14x512]
+ CONV3-512: [14x14x512]
+ CONV3-512: [14x14x512]
+ POOL2: [7x7x512]
+ FC1: [1x1x4096]
+ FC2: [1x1x4096]
+ FC3: [1x1x1000]

Which is the approximate number of parameters of this network? Which is the heaviest layer?_____no_output_____

# Your code here_____no_output_____

## VGG-16 in Keras_____no_output_____

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Flatten, Dense, Dropout
from tensorflow.keras.layers import Conv2D, MaxPooling2D, ZeroPadding2D
from tensorflow.keras.optimizers import SGD

model = Sequential()
model.add(ZeroPadding2D((1,1), input_shape=(224, 224, 3), data_format='channels_last'))
model.add(Conv2D(64, kernel_size=(3, 3), strides=1, activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(64, kernel_size=(3, 3), strides=1, activation='relu'))
model.add(MaxPooling2D((2,2), strides=(2,2), data_format='channels_last'))

model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))
model.add(MaxPooling2D((2,2), strides=(2,2), data_format='channels_last'))

model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(256, kernel_size=(3, 3), activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(256, kernel_size=(3, 3), activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(256, kernel_size=(3, 3), activation='relu'))
model.add(MaxPooling2D((2,2), strides=(2,2), data_format='channels_last'))

model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(512, kernel_size=(3, 3), activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(512, kernel_size=(3, 3), activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(512, kernel_size=(3, 3), activation='relu'))
model.add(MaxPooling2D((2,2), strides=(2,2), data_format='channels_last'))

model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(512, kernel_size=(3, 3), activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(512, kernel_size=(3, 3), activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(512, kernel_size=(3, 3), activation='relu'))
model.add(MaxPooling2D((2,2), strides=(2,2), data_format='channels_last'))

model.add(Flatten())
model.add(Dense(4096, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(4096, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1000, activation='softmax'))_____no_output_____

## Smile Detection using Convolutional Neural Networks_____no_output_____

# Adapted from Kyle McDonald
# Project: https://github.com/kylemcdonald/SmileCNN

!curl -L -O https://github.com/hromi/SMILEsmileD/archive/master.zip
!unzip -q master.zip
!rm master.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   122    0   122    0     0    603      0 --:--:-- --:--:-- --:--:--   600
100 39.6M    0 39.6M    0     0  13.9M      0 --:--:--  0:00:02 --:--:-- 21.8M
#util and imports
import os
import fnmatch

def list_all_files(directory, extensions=None):
    for root, dirnames, filenames in os.walk(directory):
        for filename in filenames:
            base, ext = os.path.splitext(filename)
            joined = os.path.join(root, filename)
            if extensions is None or ext.lower() in extensions:
                yield joined

import numpy as np
import math

def find_rectangle(n, max_ratio=2):
    sides = []
    square = int(math.sqrt(n))
    for w in range(square, max_ratio * square):
        h = n / w
        used = w * h
        leftover = n - used
        sides.append((leftover, (w, h)))
    return sorted(sides)[0][1]

# should work for 1d and 2d images, assumes images are square but can be overriden
def make_mosaic(images, n=None, nx=None, ny=None, w=None, h=None):
    if n is None and nx is None and ny is None:
        nx, ny = find_rectangle(len(images))
    else:
        nx = n if nx is None else nx
        ny = n if ny is None else ny
    images = np.array(images)
    if images.ndim == 2:
        side = int(np.sqrt(len(images[0])))
        h = side if h is None else h
        w = side if w is None else w
        images = images.reshape(-1, h, w)
    else:
        h = images.shape[1]
        w = images.shape[2]
    image_gen = iter(images)
    mosaic = np.empty((h*ny, w*nx))
    for i in range(ny):
        ia = (i)*h
        ib = (i+1)*h
        for j in range(nx):
            ja = j*w
            jb = (j+1)*w
            mosaic[ia:ib, ja:jb] = next(image_gen)
    return mosaic

import numpy as np
from skimage.measure import block_reduce

def crop_and_resize(img, target_size=32, zoom=1):
    small_side = int(np.min(img.shape) * zoom)
    reduce_factor = small_side / target_size
    crop_size = target_size * reduce_factor
    mid = np.array(img.shape) / 2
    half_crop = crop_size / 2
    center = img[mid[0]-half_crop:mid[0]+half_crop,
    	mid[1]-half_crop:mid[1]+half_crop]
    return block_reduce(center, (reduce_factor, reduce_factor), np.mean)

from io import BytesIO
import numpy as np
import PIL.Image
import IPython.display
import shutil

def show_array(a, fmt='png', filename=None):
    a = np.squeeze(a)
    a = np.uint8(np.clip(a, 0, 255))
    image_data = BytesIO()
    PIL.Image.fromarray(a).save(image_data, fmt)
    if filename is None:
        IPython.display.display(IPython.display.Image(data=image_data.getvalue()))
    else:
        with open(filename, 'w') as f:
            image_data.seek(0)
            shutil.copyfileobj(image_data, f)_____no_output_____negative_paths = list(list_all_files('SMILEsmileD-master/SMILEs/negatives/negatives7/', ['.jpg']))
print('loaded', len(negative_paths), 'negative examples')
positive_paths = list(list_all_files('SMILEsmileD-master/SMILEs/positives/positives7/', ['.jpg']))
print('loaded', len(positive_paths), 'positive examples')
examples = [(path, 0) for path in negative_paths] + [(path, 1) for path in positive_paths]loaded 9475 negative examples
loaded 3690 positive examples
import numpy as np
from skimage.measure import block_reduce
from skimage.io import imread

def examples_to_dataset(examples, block_size=2):
    X = []
    y = []
    for path, label in examples:
        img = imread(path, as_grey=True)
        img = block_reduce(img, block_size=(block_size, block_size), func=np.mean)
        X.append(img)
        y.append(label)
    return np.asarray(X), np.asarray(y)

%time X, y = examples_to_dataset(examples)CPU times: user 7.36 s, sys: 488 ms, total: 7.84 s
Wall time: 7.85 s
X = X.astype(np.float32) / 255.
y = y.astype(np.int32)
print(X.dtype, X.min(), X.max(), X.shape)
print(y.dtype, y.min(), y.max(), y.shape)float32 0.0 1.0 (13165, 32, 32)
int32 0 1 (13165,)
show_array(255 * make_mosaic(X[:len(negative_paths)], 8), fmt='jpeg') # negative at the beginning
show_array(255 * make_mosaic(X[-len(positive_paths):], 8), fmt='jpeg') # positive at the end_____no_output_____X = np.expand_dims(X, axis=-1)
np.save('X.npy', X)
np.save('y.npy', y)_____no_output_____import numpy as np
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Flatten, Reshape
from keras.layers.convolutional import Conv2D, MaxPooling2D
from keras.utils import np_utils

# load the data
X = np.load('X.npy')
y = np.load('y.npy')

# convert classes to vector
nb_classes = 2
y = np_utils.to_categorical(y, nb_classes).astype(np.float32)

# shuffle all the data
indices = np.arange(len(X))
np.random.shuffle(indices)
X = X[indices]
y = y[indices]

# prepare weighting for classes since they're unbalanced
class_totals = y.sum(axis=0)
class_weight = class_totals.max() / class_totals

print(X.dtype, X.min(), X.max(), X.shape)
print(y.dtype, y.min(), y.max(), y.shape)float32 0.0 1.0 (13165, 32, 32, 1)
float32 0.0 1.0 (13165, 2)
nb_filters = 32
nb_pool = 2
nb_conv = 3

model = Sequential()

model.add(Conv2D(nb_filters, (nb_conv, nb_conv), activation='relu', input_shape=X.shape[1:]))
model.add(Conv2D(nb_filters, (nb_conv, nb_conv), activation='relu'))
model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(nb_classes, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 30, 30, 32)        320
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 28, 28, 32)        9248
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 14, 14, 32)        0
_________________________________________________________________
dropout_1 (Dropout)          (None, 14, 14, 32)        0
_________________________________________________________________
flatten_1 (Flatten)          (None, 6272)              0
_________________________________________________________________
dense_1 (Dense)              (None, 128)               802944
_________________________________________________________________
dropout_2 (Dropout)          (None, 128)               0
_________________________________________________________________
dense_2 (Dense)              (None, 2)                 258
=================================================================
Total params: 812,770
Trainable params: 812,770
Non-trainable params: 0
_________________________________________________________________
validation_split = 0.10
model.fit(X, y, batch_size=128, class_weight=class_weight, epochs=5, verbose=1, validation_split=validation_split)WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Train on 11848 samples, validate on 1317 samples
Epoch 1/5
11848/11848 [==============================] - 5s 422us/step - loss: 0.4918 - acc: 0.7567 - val_loss: 0.3680 - val_acc: 0.8345
Epoch 2/5
11848/11848 [==============================] - 1s 121us/step - loss: 0.3498 - acc: 0.8498 - val_loss: 0.2942 - val_acc: 0.8853
Epoch 3/5
11848/11848 [==============================] - 1s 121us/step - loss: 0.3156 - acc: 0.8710 - val_loss: 0.2636 - val_acc: 0.8975
Epoch 4/5
11848/11848 [==============================] - 1s 120us/step - loss: 0.2829 - acc: 0.8868 - val_loss: 0.2501 - val_acc: 0.9028
Epoch 5/5
11848/11848 [==============================] - 1s 120us/step - loss: 0.2702 - acc: 0.8937 - val_loss: 0.2392 - val_acc: 0.9081
open('model.json', 'w').write(model.to_json())
model.save_weights('weights.h5')_____no_output_____%matplotlib inline
from matplotlib import pyplot as plt
plt.plot(model.model.history.history['loss'])
plt.plot(model.model.history.history['acc'])
plt.plot(model.model.history.history['val_loss'])
plt.plot(model.model.history.history['val_acc'])
plt.show()/usr/local/lib/python3.6/dist-packages/keras/engine/sequential.py:110: UserWarning: `Sequential.model` is deprecated. `Sequential` is a subclass of `Model`, you can just use your `Sequential` instance directly.
  warnings.warn('`Sequential.model` is deprecated. '
from sklearn.metrics import roc_auc_score
n_validation = int(len(X) * validation_split)
y_predicted = model.predict(X[-n_validation:])
print(roc_auc_score(y[-n_validation:], y_predicted))0.9585530065545739


# Notebook from ShepherdCode/ShepherdML
Path: Workshop/MLP_209.ipynb

# MLP 209
* Operate on 16000 GenCode 34 seqs.
* 5-way cross validation. Save best model per CV.
* Report mean accuracy from final re-validation with best 5.
* Use Adam with a learn rate decay schdule._____no_output_____

NC_FILENAME='ncRNA.gc34.processed.fasta'
PC_FILENAME='pcRNA.gc34.processed.fasta'
DATAPATH=""
try:
    from google.colab import drive
    IN_COLAB = True
    PATH='/content/drive/'
    drive.mount(PATH)
    DATAPATH=PATH+'My Drive/data/'  # must end in "/"
    NC_FILENAME = DATAPATH+NC_FILENAME
    PC_FILENAME = DATAPATH+PC_FILENAME
except:
    IN_COLAB = False
    DATAPATH=""

EPOCHS=200
SPLITS=5
K=1
VOCABULARY_SIZE=4**K+1   # e.g. K=3 => 64 DNA K-mers + 'NNN'
EMBED_DIMEN=16
FILENAME='MLP209'
NEURONS=16 # see model_____no_output_____import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import ShuffleSplit
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedKFold
from sklearn.model_selection import StratifiedKFold
import tensorflow as tf
from tensorflow import keras
from keras.wrappers.scikit_learn import KerasRegressor
from keras.models import Sequential
from keras.layers import Bidirectional
from keras.layers import GRU
from keras.layers import Dense
from keras.layers import LayerNormalization
import time
dt='float32'
tf.keras.backend.set_floatx(dt)_____no_output_____

## Build model_____no_output_____

def compile_model(model):
    adam_default_learn_rate = 0.001
    schedule = tf.keras.optimizers.schedules.ExponentialDecay(
        initial_learning_rate = adam_default_learn_rate*10,
        #decay_steps=100000, decay_rate=0.96, staircase=True)
        decay_steps=10000, decay_rate=0.99, staircase=True)
    # learn rate = initial_learning_rate * decay_rate ^ (step / decay_steps)
    opt = tf.keras.optimizers.Adam(learning_rate=schedule)
    bc=tf.keras.losses.BinaryCrossentropy(from_logits=False)
    print("COMPILE...")
    model.compile(loss=bc, optimizer=opt, metrics=["accuracy"])
    print("...COMPILED")
    return model

def build_model(maxlen):
    act="elu"
    #embed_layer  = keras.layers.Embedding(
    #    VOCABULARY_SIZE,EMBED_DIMEN,input_length=maxlen);
    dense1_layer = keras.layers.Dense(64, activation=act,dtype=dt,
                                      input_dim=VOCABULARY_SIZE)
    dense2_layer = keras.layers.Dense(NEURONS, activation=act,dtype=dt)
    dense3_layer = keras.layers.Dense(NEURONS, activation=act,dtype=dt)
    output_layer = keras.layers.Dense(1,  activation="sigmoid",dtype=dt)
    mlp = keras.models.Sequential()
    #mlp.add(embed_layer)
    mlp.add(dense1_layer)
    mlp.add(dense2_layer)
    mlp.add(dense3_layer)
    mlp.add(output_layer)
    mlpc = compile_model(mlp)
    return mlpc_____no_output_____

## Load and partition sequences_____no_output_____

# Assume file was preprocessed to contain one line per seq.
# Prefer Pandas dataframe but df does not support append.
# For conversion to tensor, must avoid python lists.
def load_fasta(filename,label):
    DEFLINE='>'
    labels=[]
    seqs=[]
    lens=[]
    nums=[]
    num=0
    with open (filename,'r') as infile:
        for line in infile:
            if line[0]!=DEFLINE:
                seq=line.rstrip()
                num += 1   # first seqnum is 1
                seqlen=len(seq)
                nums.append(num)
                labels.append(label)
                seqs.append(seq)
                lens.append(seqlen)
    df1=pd.DataFrame(nums,columns=['seqnum'])
    df2=pd.DataFrame(labels,columns=['class'])
    df3=pd.DataFrame(seqs,columns=['sequence'])
    df4=pd.DataFrame(lens,columns=['seqlen'])
    df=pd.concat((df1,df2,df3,df4),axis=1)
    return df

def separate_X_and_y(data):
    y=   data[['class']].copy()
    X=   data.drop(columns=['class','seqnum','seqlen'])
    return (X,y)

_____no_output_____

## Make K-mers_____no_output_____

def make_kmer_table(K):
    npad='N'*K
    shorter_kmers=['']
    for i in range(K):
        longer_kmers=[]
        for mer in shorter_kmers:
            longer_kmers.append(mer+'A')
            longer_kmers.append(mer+'C')
            longer_kmers.append(mer+'G')
            longer_kmers.append(mer+'T')
        shorter_kmers = longer_kmers
    all_kmers = shorter_kmers
    kmer_dict = {}
    kmer_dict[npad]=0
    value=1
    for mer in all_kmers:
        kmer_dict[mer]=value
        value += 1
    return kmer_dict

KMER_TABLE=make_kmer_table(K)

def strings_to_vectors(data,uniform_len):
    all_seqs=[]
    for seq in data['sequence']:
        i=0
        seqlen=len(seq)
        kmers=[]
        while i < seqlen-K+1 -1:  # stop at minus one for spaced seed
            #kmer=seq[i:i+2]+seq[i+3:i+5]    # SPACED SEED 2/1/2 for K=4
            kmer=seq[i:i+K]
            i += 1
            value=KMER_TABLE[kmer]
            kmers.append(value)
        pad_val=0
        while i < uniform_len:
            kmers.append(pad_val)
            i += 1
        all_seqs.append(kmers)
    pd2d=pd.DataFrame(all_seqs)
    return pd2d   # return 2D dataframe, uniform dimensions_____no_output_____def make_kmers(MAXLEN,train_set):
    (X_train_all,y_train_all)=separate_X_and_y(train_set)
    X_train_kmers=strings_to_vectors(X_train_all,MAXLEN)
    # From pandas dataframe to numpy to list to numpy
    num_seqs=len(X_train_kmers)
    tmp_seqs=[]
    for i in range(num_seqs):
        kmer_sequence=X_train_kmers.iloc[i]
        tmp_seqs.append(kmer_sequence)
    X_train_kmers=np.array(tmp_seqs)
    tmp_seqs=None
    labels=y_train_all.to_numpy()
    return (X_train_kmers,labels)_____no_output_____def make_frequencies(Xin):
    Xout=[]
    VOCABULARY_SIZE= 4**K + 1  # plus one for 'NNN'
    for seq in Xin:
        freqs =[0] * VOCABULARY_SIZE
        total = 0
        for kmerval in seq:
            freqs[kmerval] += 1
            total += 1
        for c in range(VOCABULARY_SIZE):
            freqs[c] = freqs[c]/total
        Xout.append(freqs)
    Xnum = np.asarray(Xout)
    return (Xnum)
def make_slice(data_set,min_len,max_len):
    slice = data_set.query('seqlen <= '+str(max_len)+' & seqlen>= '+str(min_len))
    return slice_____no_output_____

## Cross validation_____no_output_____

def do_cross_validation(X,y,given_model):
    cv_scores = []
    fold=0
    splitter = ShuffleSplit(n_splits=SPLITS, test_size=0.1, random_state=37863)
    for train_index,valid_index in splitter.split(X):
        fold += 1
        X_train=X[train_index] # use iloc[] for dataframe
        y_train=y[train_index]
        X_valid=X[valid_index]
        y_valid=y[valid_index]
        # Avoid continually improving the same model.
        model = compile_model(keras.models.clone_model(given_model))
        bestname=DATAPATH+FILENAME+".cv."+str(fold)+".best"
        mycallbacks = [keras.callbacks.ModelCheckpoint(
            filepath=bestname, save_best_only=True,
            monitor='val_accuracy', mode='max')]
        print("FIT")
        start_time=time.time()
        history=model.fit(X_train, y_train, # batch_size=10, default=32 works nicely
                epochs=EPOCHS, verbose=1,  # verbose=1 for ascii art, verbose=0 for none
                callbacks=mycallbacks,
                validation_data=(X_valid,y_valid) )
        end_time=time.time()
        elapsed_time=(end_time-start_time)
        print("Fold %d, %d epochs, %d sec"%(fold,EPOCHS,elapsed_time))
        pd.DataFrame(history.history).plot(figsize=(8,5))
        plt.grid(True)
        plt.gca().set_ylim(0,1)
        plt.show()
        best_model=keras.models.load_model(bestname)
        scores = best_model.evaluate(X_valid, y_valid, verbose=0)
        print("%s: %.2f%%" % (best_model.metrics_names[1], scores[1]*100))
        cv_scores.append(scores[1] * 100)
    print()
    print("%d-way Cross Validation mean %.2f%% (+/- %.2f%%)" % (fold, np.mean(cv_scores), np.std(cv_scores)))_____no_output_____

## Train on RNA lengths 200-1Kb_____no_output_____

MINLEN=200
MAXLEN=1000
print("Load data from files.")
nc_seq=load_fasta(NC_FILENAME,0)
pc_seq=load_fasta(PC_FILENAME,1)
train_set=pd.concat((nc_seq,pc_seq),axis=0)
nc_seq=None
pc_seq=None
print("Ready: train_set")
#train_set
print ("Compile the model")
model=build_model(MAXLEN)
print ("Summarize the model")
print(model.summary())  # Print this only once
model.save(DATAPATH+FILENAME+'.model')
print ("Data prep")
subset=make_slice(train_set,MINLEN,MAXLEN)# One array to two: X and y
print ("Data reshape")
(X_train,y_train)=make_kmers(MAXLEN,subset)
X_train=make_frequencies(X_train)Load data from files.
Ready: train_set
Compile the model
COMPILE...
...COMPILED
Summarize the model
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense (Dense)                (None, 64)                384
_________________________________________________________________
dense_1 (Dense)              (None, 16)                1040
_________________________________________________________________
dense_2 (Dense)              (None, 16)                272
_________________________________________________________________
dense_3 (Dense)              (None, 1)                 17
=================================================================
Total params: 1,713
Trainable params: 1,713
Non-trainable params: 0
_________________________________________________________________
None
WARNING:tensorflow:From /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py:1813: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
INFO:tensorflow:Assets written to: MLP209.model/assets
Data prep
Data reshape
print ("Cross valiation")
do_cross_validation(X_train,y_train,model)
print ("Done")Cross valiation
COMPILE...
...COMPILED
FIT
Epoch 1/200
453/453 [==============================] - ETA: 0s - loss: 0.5638 - accuracy: 0.7054 ETA: 0s - loss: 0.5709 - accuracy: 0.70 - ETA: 0s - loss: 0.5699 - accuINFO:tensorflow:Assets written to: MLP209.cv.1.best/assets
453/453 [==============================] - 2s 4ms/step - loss: 0.5638 - accuracy: 0.7054 - val_loss: 0.5320 - val_accuracy: 0.7269
Epoch 2/200
431/453 [===========================>..] - ETA: 0s - loss: 0.5513 - accuracy: 0.7172INFO:tensorflow:Assets written to: MLP209.cv.1.best/assets
453/453 [==============================] - 2s 3ms/step - loss: 0.5513 - accuracy: 0.7175 - val_loss: 0.5336 - val_accuracy: 0.7312
Epoch 3/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5480 - accuracy: 0.7186 - val_loss: 0.5353 - val_accuracy: 0.7232
Epoch 4/200
437/453 [===========================>..] - ETA: 0s - loss: 0.5445 - accuracy: 0.7196INFO:tensorflow:Assets written to: MLP209.cv.1.best/assets
453/453 [==============================] - 2s 3ms/step - loss: 0.5448 - accuracy: 0.7195 - val_loss: 0.5254 - val_accuracy: 0.7412
Epoch 5/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5440 - accuracy: 0.7199 - val_loss: 0.5286 - val_accuracy: 0.7306
Epoch 6/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5439 - accuracy: 0.7212 - val_loss: 0.5416 - val_accuracy: 0.7318
Epoch 7/200
453/453 [==============================] - 1s 3ms/step - loss: 0.5441 - accuracy: 0.7233 - val_loss: 0.5259 - val_accuracy: 0.7325
Epoch 8/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5439 - accuracy: 0.7219 - val_loss: 0.5252 - val_accuracy: 0.7374
Epoch 9/200
453/453 [==============================] - 1s 3ms/step - loss: 0.5421 - accuracy: 0.7232 - val_loss: 0.5249 - val_accuracy: 0.7337
Epoch 10/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5432 - accuracy: 0.7242 - val_loss: 0.5340 - val_accuracy: 0.7368
Epoch 11/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5422 - accuracy: 0.7203 - val_loss: 0.5235 - val_accuracy: 0.7362
Epoch 12/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5429 - accuracy: 0.7232 - val_loss: 0.5253 - val_accuracy: 0.7312
Epoch 13/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5438 - accuracy: 0.7229 - val_loss: 0.5263 - val_accuracy: 0.7343
Epoch 14/200
453/453 [==============================] - 1s 3ms/step - loss: 0.5432 - accuracy: 0.7192 - val_loss: 0.5283 - val_accuracy: 0.7368
Epoch 15/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5434 - accuracy: 0.7232 - val_loss: 0.5246 - val_accuracy: 0.7356
Epoch 16/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5420 - accuracy: 0.7239 - val_loss: 0.5251 - val_accuracy: 0.7343
Epoch 17/200
453/453 [==============================] - 2s 3ms/step - loss: 0.5422 - accuracy: 0.7226 - val_loss: 0.5255 - val_accuracy: 0.7349
Epoch 18/200
453/453 [==============================] - 1s 3ms/step - loss: 0.5410 - accuracy: 0.7187 - val_loss: 0.5266 - val_accuracy: 0.7306
Epoch 19/200
453/453 [==============================] - 3s 6ms/step - loss: 0.5401 - accuracy: 0.7220 - val_loss: 0.5313 - val_accuracy: 0.7343
Epoch 20/200
453/453 [==============================] - 3s 6ms/step - loss: 0.5374 - accuracy: 0.7209 - val_loss: 0.5273 - val_accuracy: 0.7275
Epoch 21/200
453/453 [==============================] - 2s 5ms/step - loss: 0.5385 - accuracy: 0.7217 - val_loss: 0.5352 - val_accuracy: 0.7306
Epoch 22/200
453/453 [==============================] - 2s 5ms/step - loss: 0.5381 - accuracy: 0.7228 - val_loss: 0.5251 - val_accuracy: 0.7381
Epoch 23/200
453/453 [==============================] - 2s 4ms/step - loss: 0.5380 - accuracy: 0.7232 - val_loss: 0.5336 - val_accuracy: 0.7312
Epoch 24/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5369 - accuracy: 0.7248 - val_loss: 0.5274 - val_accuracy: 0.7362
Epoch 25/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5388 - accuracy: 0.7235 - val_loss: 0.5251 - val_accuracy: 0.7393
Epoch 26/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5364 - accuracy: 0.7267 - val_loss: 0.5258 - val_accuracy: 0.7294
Epoch 27/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5377 - accuracy: 0.7232 - val_loss: 0.5241 - val_accuracy: 0.7325
Epoch 28/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5350 - accuracy: 0.7266 - val_loss: 0.5259 - val_accuracy: 0.7337
Epoch 29/200
453/453 [==============================] - 1s 3ms/step - loss: 0.5355 - accuracy: 0.7257 - val_loss: 0.5245 - val_accuracy: 0.7263
Epoch 30/200
453/453 [==============================] - 1s 3ms/step - loss: 0.5345 - accuracy: 0.7266 - val_loss: 0.5253 - val_accuracy: 0.7349
Epoch 31/200
453/453 [==============================] - 1s 3ms/step - loss: 0.5343 - accuracy: 0.7255 - val_loss: 0.5260 - val_accuracy: 0.7300
Epoch 32/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5360 - accuracy: 0.7300 - val_loss: 0.5249 - val_accuracy: 0.7244
Epoch 33/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5342 - accuracy: 0.7289 - val_loss: 0.5307 - val_accuracy: 0.7306
Epoch 34/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5346 - accuracy: 0.7263 - val_loss: 0.5304 - val_accuracy: 0.7312
Epoch 35/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5354 - accuracy: 0.7266 - val_loss: 0.5209 - val_accuracy: 0.7374
Epoch 36/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5325 - accuracy: 0.7303 - val_loss: 0.5369 - val_accuracy: 0.7318
Epoch 37/200
453/453 [==============================] - 1s 3ms/step - loss: 0.5339 - accuracy: 0.7266 - val_loss: 0.5261 - val_accuracy: 0.7349
Epoch 38/200
453/453 [==============================] - 1s 3ms/step - loss: 0.5337 - accuracy: 0.7248 - val_loss: 0.5217 - val_accuracy: 0.7325
Epoch 39/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5332 - accuracy: 0.7290 - val_loss: 0.5306 - val_accuracy: 0.7281
Epoch 40/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5334 - accuracy: 0.7253 - val_loss: 0.5222 - val_accuracy: 0.7331
Epoch 41/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5347 - accuracy: 0.7285 - val_loss: 0.5468 - val_accuracy: 0.7269
Epoch 42/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5324 - accuracy: 0.7266 - val_loss: 0.5262 - val_accuracy: 0.7300
Epoch 43/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5350 - accuracy: 0.7268 - val_loss: 0.5333 - val_accuracy: 0.7325
Epoch 44/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5320 - accuracy: 0.7322 - val_loss: 0.5241 - val_accuracy: 0.7412
Epoch 45/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5320 - accuracy: 0.7307 - val_loss: 0.5270 - val_accuracy: 0.7349
Epoch 46/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5302 - accuracy: 0.7300 - val_loss: 0.5204 - val_accuracy: 0.7399
Epoch 47/200
442/453 [============================>.] - ETA: 0s - loss: 0.5307 - accuracy: 0.7316INFO:tensorflow:Assets written to: MLP209.cv.1.best/assets
453/453 [==============================] - 2s 4ms/step - loss: 0.5301 - accuracy: 0.7319 - val_loss: 0.5310 - val_accuracy: 0.7474
Epoch 48/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5313 - accuracy: 0.7299 - val_loss: 0.5297 - val_accuracy: 0.7461
Epoch 49/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5301 - accuracy: 0.7263 - val_loss: 0.5215 - val_accuracy: 0.7430
Epoch 50/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5331 - accuracy: 0.7284 - val_loss: 0.5255 - val_accuracy: 0.7318
Epoch 51/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5291 - accuracy: 0.7330 - val_loss: 0.5185 - val_accuracy: 0.7381
Epoch 52/200
453/453 [==============================] - 1s 2ms/step - loss: 0.5310 - accuracy: 0.7276 - val_loss: 0.5159 - val_accuracy: 0.7418
Epoch 53/200


# Notebook from ptyadana/probability-and-statistics-for-business-and-data-science
Path: Data Science and Machine Learning Bootcamp - JP/22. Natural Language Processing/01-NLP(Natural Language Processing) with Python.ipynb

# NLP (Natural Language Processing) with Python_____no_output_____

import nltk_____no_output_____

**We need to download `stopwords` package using nltk shell.**_____no_output_____

nltk.download_shell()NLTK Downloader
---------------------------------------------------------------------------
    d) Download   l) List    u) Update   c) Config   h) Help   q) Quit
---------------------------------------------------------------------------
Downloader> l

Packages:
  [ ] abc................. Australian Broadcasting Commission 2006
  [ ] alpino.............. Alpino Dutch Treebank
  [ ] averaged_perceptron_tagger Averaged Perceptron Tagger
  [ ] averaged_perceptron_tagger_ru Averaged Perceptron Tagger (Russian)
  [ ] basque_grammars..... Grammars for Basque
  [ ] biocreative_ppi..... BioCreAtIvE (Critical Assessment of Information
                           Extraction Systems in Biology)
  [ ] bllip_wsj_no_aux.... BLLIP Parser: WSJ Model
  [ ] book_grammars....... Grammars from NLTK Book
  [ ] brown............... Brown Corpus
  [ ] brown_tei........... Brown Corpus (TEI XML Version)
  [ ] cess_cat............ CESS-CAT Treebank
  [ ] cess_esp............ CESS-ESP Treebank
  [ ] chat80.............. Chat-80 Data Files
  [ ] city_database....... City Database
  [ ] cmudict............. The Carnegie Mellon Pronouncing Dictionary (0.6)
  [ ] comparative_sentences Comparative Sentence Dataset
  [ ] comtrans............ ComTrans Corpus Sample
  [ ] conll2000........... CONLL 2000 Chunking Corpus
  [ ] conll2002........... CONLL 2002 Named Entity Recognition Corpus
Hit Enter to continue:
  [ ] conll2007........... Dependency Treebanks from CoNLL 2007 (Catalan
                           and Basque Subset)
  [ ] crubadan............ Crubadan Corpus
  [ ] dependency_treebank. Dependency Parsed Treebank
  [ ] dolch............... Dolch Word List
  [ ] europarl_raw........ Sample European Parliament Proceedings Parallel
                           Corpus
  [ ] floresta............ Portuguese Treebank
  [ ] framenet_v15........ FrameNet 1.5
  [ ] framenet_v17........ FrameNet 1.7
  [ ] gazetteers.......... Gazeteer Lists
  [ ] genesis............. Genesis Corpus
  [ ] gutenberg........... Project Gutenberg Selections
  [ ] ieer................ NIST IE-ER DATA SAMPLE
  [ ] inaugural........... C-Span Inaugural Address Corpus
  [ ] indian.............. Indian Language POS-Tagged Corpus
  [ ] jeita............... JEITA Public Morphologically Tagged Corpus (in
                           ChaSen format)
  [ ] kimmo............... PC-KIMMO Data Files
  [ ] knbc................ KNB Corpus (Annotated blog corpus)
  [ ] large_grammars...... Large context-free and feature-based grammars
                           for parser comparison
Hit Enter to continue:
  [ ] lin_thesaurus....... Lin's Dependency Thesaurus
  [ ] mac_morpho.......... MAC-MORPHO: Brazilian Portuguese news text with
                           part-of-speech tags
  [ ] machado............. Machado de Assis -- Obra Completa
  [ ] masc_tagged......... MASC Tagged Corpus
  [ ] maxent_ne_chunker... ACE Named Entity Chunker (Maximum entropy)
  [ ] maxent_treebank_pos_tagger Treebank Part of Speech Tagger (Maximum entropy)
  [ ] moses_sample........ Moses Sample Models
  [ ] movie_reviews....... Sentiment Polarity Dataset Version 2.0
  [ ] mte_teip5........... MULTEXT-East 1984 annotated corpus 4.0
  [ ] mwa_ppdb............ The monolingual word aligner (Sultan et al.
                           2015) subset of the Paraphrase Database.
  [ ] names............... Names Corpus, Version 1.3 (1994-03-29)
  [ ] nombank.1.0......... NomBank Corpus 1.0
  [ ] nonbreaking_prefixes Non-Breaking Prefixes (Moses Decoder)
  [ ] nps_chat............ NPS Chat
  [ ] omw................. Open Multilingual Wordnet
  [ ] opinion_lexicon..... Opinion Lexicon
  [ ] panlex_swadesh...... PanLex Swadesh Corpora
  [ ] paradigms........... Paradigm Corpus
  [ ] pe08................ Cross-Framework and Cross-Domain Parser
                           Evaluation Shared Task
Hit Enter to continue:
  [ ] perluniprops........ perluniprops: Index of Unicode Version 7.0.0
                           character properties in Perl
  [ ] pil................. The Patient Information Leaflet (PIL) Corpus
  [ ] pl196x.............. Polish language of the XX century sixties
  [ ] porter_test......... Porter Stemmer Test Files
  [ ] ppattach............ Prepositional Phrase Attachment Corpus
  [ ] problem_reports..... Problem Report Corpus
  [ ] product_reviews_1... Product Reviews (5 Products)
  [ ] product_reviews_2... Product Reviews (9 Products)
  [ ] propbank............ Proposition Bank Corpus 1.0
  [ ] pros_cons........... Pros and Cons
  [ ] ptb................. Penn Treebank
  [ ] punkt............... Punkt Tokenizer Models
  [ ] qc.................. Experimental Data for Question Classification
  [ ] reuters............. The Reuters-21578 benchmark corpus, ApteMod
                           version
  [ ] rslp................ RSLP Stemmer (Removedor de Sufixos da Lingua
                           Portuguesa)
  [ ] rte................. PASCAL RTE Challenges 1, 2, and 3
  [ ] sample_grammars..... Sample Grammars
  [ ] semcor.............. SemCor 3.0
Hit Enter to continue:
  [ ] senseval............ SENSEVAL 2 Corpus: Sense Tagged Text
  [ ] sentence_polarity... Sentence Polarity Dataset v1.0
  [ ] sentiwordnet........ SentiWordNet
  [ ] shakespeare......... Shakespeare XML Corpus Sample
  [ ] sinica_treebank..... Sinica Treebank Corpus Sample
  [ ] smultron............ SMULTRON Corpus Sample
  [ ] snowball_data....... Snowball Data
  [ ] spanish_grammars.... Grammars for Spanish
  [ ] state_union......... C-Span State of the Union Address Corpus
  [*] stopwords........... Stopwords Corpus
  [ ] subjectivity........ Subjectivity Dataset v1.0
  [ ] swadesh............. Swadesh Wordlists
  [ ] switchboard......... Switchboard Corpus Sample
  [ ] tagsets............. Help on Tagsets
  [ ] timit............... TIMIT Corpus Sample
  [ ] toolbox............. Toolbox Sample Files
  [ ] treebank............ Penn Treebank Sample
  [ ] twitter_samples..... Twitter Samples
  [ ] udhr2............... Universal Declaration of Human Rights Corpus
                           (Unicode Version)
  [ ] udhr................ Universal Declaration of Human Rights Corpus
Hit Enter to continue:
  [ ] unicode_samples..... Unicode Samples
  [ ] universal_tagset.... Mappings to the Universal Part-of-Speech Tagset
  [ ] universal_treebanks_v20 Universal Treebanks Version 2.0
  [ ] vader_lexicon....... VADER Sentiment Lexicon
  [ ] verbnet3............ VerbNet Lexicon, Version 3.3
  [ ] verbnet............. VerbNet Lexicon, Version 2.1
  [ ] webtext............. Web Text Corpus
  [ ] wmt15_eval.......... Evaluation data from WMT15
  [ ] word2vec_sample..... Word2Vec Sample
  [ ] wordnet............. WordNet
  [ ] wordnet_ic.......... WordNet-InfoContent
  [ ] words............... Word Lists
  [ ] ycoe................ York-Toronto-Helsinki Parsed Corpus of Old
                           English Prose

Collections:
  [P] all-corpora......... All the corpora
  [P] all-nltk............ All packages available on nltk_data gh-pages
                           branch
  [P] all................. All packages
  [P] book................ Everything used in the NLTK Book
  [P] popular............. Popular packages
Hit Enter to continue:
  [ ] tests............... Packages for running tests
  [ ] third-party......... Third-party data packages

([*] marks installed packages; [P] marks partially installed collections)

---------------------------------------------------------------------------
    d) Download   l) List    u) Update   c) Config   h) Help   q) Quit
---------------------------------------------------------------------------
Downloader> d

Download which package (l=list; x=cancel)?
  Identifier> stopwords

-----_____no_output_____# SMS Spam Detection

Data: [
SMS Spam Collection Data Set](https://archive.ics.uci.edu/ml/datasets/sms+spam+collection)

The file we are using contains a collection of more than 5 thousand SMS phone messages. You can check out the **readme** file for more info.
_____no_output_____

file_path = 'smsspamcollection/SMSSpamCollection'

messages = [line.rstrip() for line in open(file_path)]_____no_output_____print(len(messages))5574
messages[:5]_____no_output_____

### A collection of texts is also sometimes called "corpus"_____no_output_____

for message_no, message in enumerate(messages[:10]):
    print(message_no, message)
    print('')0 ham	Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...

1 ham	Ok lar... Joking wif u oni...

2 spam	Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's

3 ham	U dun say so early hor... U c already then say...

4 ham	Nah I don't think he goes to usf, he lives around here though

5 spam	FreeMsg Hey there darling it's been 3 week's now and no word back! I'd like some fun you up for it still? Tb ok! XxX std chgs to send, Â£1.50 to rcv

6 ham	Even my brother is not like to speak with me. They treat me like aids patent.

7 ham	As per your request 'Melle Melle (Oru Minnaminunginte Nurungu Vettam)' has been set as your callertune for all Callers. Press *9 to copy your friends Callertune

8 spam	WINNER!! As a valued network customer you have been selected to receivea Â£900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.

9 spam	Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030

### Analysing Data File

Due to the spacing we can tell that this is a [TSV](http://en.wikipedia.org/wiki/Tab-separated_values) ("tab separated values") file, where the first column is a label saying whether the given message is a normal message (commonly known as "ham") or "spam". The second column is the message itself.

Using these labeled ham and spam examples, we'll **train a machine learning model to learn to discriminate between ham/spam automatically**. Then, with a trained model, we'll be able to **classify arbitrary unlabeled messages** as ham or spam._____no_output_____

messages[0]_____no_output_____

------_____no_output_____# Parsing data using pandas_____no_output_____

import pandas as pd_____no_output_____messages = pd.read_csv(file_path, sep='\t', names=['label', 'message']) # as we know which to use as separator_____no_output_____messages.head()_____no_output_____

# Exploratory Data Analysis_____no_output_____

messages.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5572 entries, 0 to 5571
Data columns (total 2 columns):
 #   Column   Non-Null Count  Dtype
---  ------   --------------  -----
 0   label    5572 non-null   object
 1   message  5572 non-null   object
dtypes: object(2)
memory usage: 87.2+ KB
messages.describe()_____no_output_____

From the describe data, we can see total is 5572 messages.

However As unique messages, there are only 5169 unique ones.

It makes sense because some messages like "Sorry, I'll cal later" can be frequently used message and duplicated in multiple times in dataset._____no_output_____#### We want statistics across labels,_____no_output_____

messages.groupby('label').describe()_____no_output_____

We can see that there are 4825 ham messages and 747 spam messages.

Most popular ham message is "Sorry, I'll call later" with appearing 30 times. For spam category, "Please call our customer.. " appears 4 times._____no_output_____

import matplotlib.pyplot as plt
import seaborn as sns_____no_output_____sns.countplot(data=messages, x='label');_____no_output_____

------_____no_output_____# Feature Engineering

As we continue our analysis we want to start thinking about the features we are going to be using. This goes along with the general idea of [feature engineering](https://en.wikipedia.org/wiki/Feature_engineering). The better your domain knowledge on the data, the better your ability to engineer more features from it. Feature engineering is a very large part of spam detection in general._____no_output_____#### add new column : length of the message_____no_output_____

messages['length'] = messages['message'].apply(len)

messages.head()_____no_output_____# let's visualize the distribution of the messages
sns.displot(data=messages, x='length', bins=100, height=6, aspect=2);
plt.xlim(0, 1000)_____no_output_____

### Statistics of Length_____no_output_____

messages['length'].describe()_____no_output_____

It seems like shortest message is 2 with longest one is 910._____no_output_____### Let's check which is the longest 910 message._____no_output_____

messages[messages['length'] == 910]['message'].iloc[0]_____no_output_____

### Distributions of messages length by label type_____no_output_____

messages.hist(column='length', by='label', bins=60, figsize=(12, 5));_____no_output_____

---------
_____no_output_____# Text Pre-processing_____no_output_____* Our main issue with our data is that it is all in text format (strings).
* The classification algorithms that we've learned about so far will need some sort of numerical feature vector in order to perform the classification task.
* There are actually many methods to convert a corpus to a vector format. The simplest is the the **[bag-of-words](http://en.wikipedia.org/wiki/Bag-of-words_model) approach**, where each unique word in a text will be represented by one number.

In this section we'll convert the raw messages (sequence of characters) into vectors (sequences of numbers).

As a first step, let's write a function that will split a message into its individual words and return a list. We'll also remove very common words, ('the', 'a', etc..). To do this we will take advantage of the NLTK library. It's pretty much the standard library in Python for processing text and has a lot of useful features. We'll only use some of the basic ones here._____no_output_____Let's create a function that will process the string in the message column, then we can just use **apply()** in pandas do process all the text in the DataFrame.

First removing punctuation. We can just take advantage of Python's built-in **string** library to get a quick list of all the possible punctuation:_____no_output_____### Remove Punctuation_____no_output_____

import string_____no_output_____string.punctuation_____no_output_____dummy = 'Sample message! Notice: it has punctuation.'_____no_output_____# Check characters to see if they are in punctuation
no_punc = [c for c in dummy if c not in string.punctuation]_____no_output_____# now all the punctuatoin are removed
no_punc_____no_output_____# Join the characters again to form the string.
no_punc = ''.join(no_punc)
no_punc_____no_output_____

### Remove Stopwords by using nltk

We can impot a list of english stopwords from NLTK._____no_output_____

# we an see the common stopwords from NLTK
from nltk.corpus import stopwords

stopwords.words('english')[:10] # Show some stop words_____no_output_____no_punc.split()_____no_output_____# check whether each words belong to common stopwords, if yes then remove
clean_mess = [word for word in no_punc.split() if word.lower() not in stopwords.words('english')]

clean_mess_____no_output_____

We can see that `it` and `has` are removed._____no_output_____## Custom Function to combine above functionalties to remove both
* punctuation
* stopwords_____no_output_____

def text_process(message):
    """
    1. remove punctuation
    2. remove stopwords
    3. returns list of clean text words
    """

    # 1) remove punctuation
    no_punc = [c for c in message if c not in string.punctuation]
    no_punc = ''.join(no_punc)

    # 2) remove stopwords
    clean_message = [word for word in no_punc.split() if word not in stopwords.words('english')]

    # 3) return list clean text words
    return clean_message_____no_output_____# test function
test_msg = 'Sample message! Notice: it has punctuation.'
text_process(test_msg)_____no_output_____

---------_____no_output_____## Tokenization

Now let's "tokenize" these messages.

### Tokenization is just the term used to describe the process of converting the normal text strings in to a list of tokens (words that we actually want).

Let's see an example output on on column:

**Note:**
We may get some warnings or errors for symbols we didn't account for or that weren't in Unicode (like a British pound symbol)_____no_output_____

messages.head()_____no_output_____# now the message are tokenized by using our function
messages['message'].head(5).apply(text_process)_____no_output_____

-------_____no_output_____# Continuing Normalization

There are a lot of ways to continue normalizing this text.
* Such as [Stemming](https://en.wikipedia.org/wiki/Stemming)
* or distinguishing by [part of speech](http://www.nltk.org/book/ch05.html).

NLTK has lots of built-in tools and great documentation on a lot of these methods. Sometimes they don't work well for text-messages due to the way a lot of people tend to use abbreviations or shorthand, For example:

    'Nah dawg, IDK! Wut time u headin to da club?'

versus

    'No dog, I don't know! What time are you heading to the club?'

Some text normalization methods will have trouble with this type of shorthand. To explore those more advanced methods through the [NLTK book online](http://www.nltk.org/book/).

For now we will just focus on using what we have to convert our list of words to an actual vector that SciKit-Learn can use._____no_output_____---------_____no_output_____# Vectorization

Currently, we have the messages as lists of tokens (also known as [lemmas](http://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html)) and now we need to convert each of those messages into a vector the SciKit Learn's algorithm models can work with.

Now we'll convert each message, represented as a list of tokens (lemmas) above, into a vector that machine learning models can understand.

We'll do that in three steps using the **bag-of-words model**:

1. **Count how many times does a word occur in each message (Known as term frequency)**

2. **Weigh the counts, so that frequent tokens get lower weight (inverse document frequency)**

3. **Normalize the vectors to unit length, to abstract from the original text length (L2 norm)**_____no_output_____# 1. Count how many times does a word occur in each message (Known as term frequency)

Let's begin the first step:

Each vector will have as many dimensions as there are unique words in the SMS corpus.  We will first use SciKit Learn's **CountVectorizer**. This model will convert a collection of text documents to a matrix of token counts.

We can imagine this as a 2-Dimensional matrix. Where the 1-dimension is the entire vocabulary (1 row per word) and the other dimension are the actual documents, in this case a column per text message.

For example:

	 Message 1	 Message 2	 ...	 Message N

Word 1 Count	0	1	...	0

Word 2 Count	0	0	...	0

...	 1	2	...	0

Word N Count	 0	1	...	1

Since there are so many messages, we can expect a lot of zero counts for the presence of that word in that document. Because of this, SciKit Learn will output a [Sparse Matrix](https://en.wikipedia.org/wiki/Sparse_matrix)._____no_output_____###  CountVectorizer as Transformer_____no_output_____

from sklearn.feature_extraction.text import CountVectorizer_____no_output_____

There are a lot of arguments and parameters that can be passed to the CountVectorizer. In this case we will just specify the **analyzer** to be our own previously defined function:_____no_output_____

# Bag of Words tansformer (using our defined function)
bow_transformer = CountVectorizer(analyzer=text_process).fit(messages['message'])_____no_output_____# Print total number of vocab words
print(len(bow_transformer.vocabulary_))11617

#### Testing on randomly selected 4th message_____no_output_____

message_4 = messages['message'][3]
message_4_____no_output_____# transform the 4th above message using our bow transformer into Vector
bow_4 = bow_transformer.transform([message_4])_____no_output_____print(bow_4)  (0, 4221)	2
  (0, 4821)	1
  (0, 5453)	1
  (0, 6396)	1
  (0, 6414)	1
  (0, 7378)	1
  (0, 9746)	2
bow_4.shape_____no_output_____

This means that there are **seven unique words** in message number 4 (after removing common stop words).
* Two of them appear twice (index of 4221 and 9746)
* the rest only once.

Let's go ahead and check and confirm which ones appear twice:_____no_output_____

# check which words appear twice in message 4
# we can get the the word from bow transformer features name

print(bow_transformer.get_feature_names()[4221]) # passing the index
print(bow_transformer.get_feature_names()[9746] )U
say

We can see that `U` and `say` appear twice in original message of `U dun say so early hor... U c already then say...`_____no_output_____### Transforming the whole Message dataframe_____no_output_____

messages_bow = bow_transformer.transform(messages['message'])_____no_output_____print('Shape of Sparse Matrix: ', messages_bow.shape)
print('Amount of Non-Zero occurences: ', messages_bow.nnz)Shape of Sparse Matrix:  (5572, 11617)
Amount of Non-Zero occurences:  56024

#### Sparsity
* basically comparing the number of Non-Zeros messages Versus the actual total number of messages
* tells you how many zeros are in your actual matrix_____no_output_____

sparsity = (100.0 * messages_bow.nnz / (messages_bow.shape[0] * messages_bow.shape[1]))
print('sparsity: {}'.format(sparsity))sparsity: 0.08655038742205227

-------
_____no_output_____After the counting, the term weighting and normalization can be done with [TF-IDF](http://en.wikipedia.org/wiki/Tf%E2%80%93idf), using scikit-learn's `TfidfTransformer`.

____

# 2. Weigh the counts, so that frequent tokens get lower weight (inverse document frequency)

## So what is TF-IDF?
TF-IDF stands for *term frequency-inverse document frequency*, and the tf-idf weight is a weight often used in information retrieval and text mining. **This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus.** The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus. Variations of the tf-idf weighting scheme are often used by search engines as a central tool in scoring and ranking a document's relevance given a user query.

One of the simplest ranking functions is computed by summing the tf-idf for each query term; many more sophisticated ranking functions are variants of this simple model.

#### Typically, the tf-idf weight is composed by two terms:
* the first computes the normalized Term Frequency (TF), aka. the number of times a word appears in a document, divided by the total number of words in that document;
* the second term is the Inverse Document Frequency (IDF), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.

**TF: Term Frequency**, which measures how frequently a term occurs in a document. Since every document is different in length, it is possible that a term would appear much more times in long documents than shorter ones. Thus, the term frequency is often divided by the document length (aka. the total number of terms in the document) as a way of normalization:

*TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document).*

**IDF: Inverse Document Frequency**, which measures how important a term is. While computing TF, all terms are considered equally important. However it is known that certain terms, such as "is", "of", and "that", may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scale up the rare ones, by computing the following:

*IDF(t) = log_e(Total number of documents / Number of documents with term t in it).*

See below for a simple example.

**Example:**

Consider a document containing 100 words where in the word cat appears 3 times.

The term frequency (i.e., tf) for cat is then (3 / 100) = 0.03. Now, assume we have 10 million documents and the word cat appears in one thousand of these. Then, the inverse document frequency (i.e., idf) is calculated as log(10,000,000 / 1,000) = 4. Thus, the Tf-idf weight is the product of these quantities: 0.03 * 4 = 0.12.
____

Let's go ahead and see how we can do this in SciKit Learn:_____no_output_____

from sklearn.feature_extraction.text import TfidfTransformer_____no_output_____tfidf_transformer = TfidfTransformer()

tfidf_transformer.fit(messages_bow) # fit on whole message bow_____no_output_____# test run on 4th message, bow
tfidf_4 = tfidf_transformer.transform(bow_4)
print(tfidf_4)  (0, 9746)	0.5385626262927564
  (0, 7378)	0.4389365653379857
  (0, 6414)	0.3187216892949149
  (0, 6396)	0.29953799723697416
  (0, 5453)	0.29729957405868723
  (0, 4821)	0.26619801906087187
  (0, 4221)	0.40832589933384067

#### Example:

We'll go ahead and check what is the IDF (inverse document frequency) of the word `"u"` and of word `"university"`?_____no_output_____

print(tfidf_transformer.idf_[bow_transformer.vocabulary_['u']])

print(tfidf_transformer.idf_[bow_transformer.vocabulary_['university']])3.2800524267409408
8.527076498901426

### tfidf transforming on whole bow messages_____no_output_____To transform the entire bag-of-words corpus into TF-IDF corpus at once:_____no_output_____

messages_tfidf = tfidf_transformer.transform(messages_bow)
print(messages_tfidf.shape)(5572, 11617)

There are many ways the data can be preprocessed and vectorized. These steps involve feature engineering and building a "pipeline". We can check out SciKit Learn's documentation on dealing with text data as well as the expansive collection of available papers and books on the general topic of NLP._____no_output_____--------_____no_output_____# Machine Learning Modelling and Training

With messages represented as vectors, we can finally train our spam/ham classifier. Now we can actually use almost any sort of classification algorithms. For a [variety of reasons](http://www.inf.ed.ac.uk/teaching/courses/inf2b/learnnotes/inf2b-learn-note07-2up.pdf), the Naive Bayes classifier algorithm is a good choice._____no_output_____We'll be using scikit-learn here, choosing the [Naive Bayes](http://en.wikipedia.org/wiki/Naive_Bayes_classifier) classifier to start with:_____no_output_____# Using NaiveBayes_____no_output_____

from sklearn.naive_bayes import MultinomialNB_____no_output_____# vectorized tfidf data, actual label
spam_detect_model = MultinomialNB().fit(messages_tfidf, messages['label'])_____no_output_____

#### Test on single message_____no_output_____

spam_detect_model.predict(tfidf_4)_____no_output_____

It is predicted as `ham`. So what is the actual label?_____no_output_____

print('predicted:', spam_detect_model.predict(tfidf_4)[0])
print('expected:', messages.label[3])predicted: ham
expected: ham

---------_____no_output_____# Model Evaluation_____no_output_____

all_predictions = spam_detect_model.predict(messages_tfidf)
print(all_predictions)['ham' 'ham' 'spam' ... 'ham' 'ham' 'ham']

We can use SciKit Learn's built-in classification report, which returns [precision, recall,](https://en.wikipedia.org/wiki/Precision_and_recall) [f1-score](https://en.wikipedia.org/wiki/F1_score), and a column for support (meaning how many cases supported that classification). Check out the links for more detailed info on each of these metrics and the figure below:_____no_output__________no_output_____

from sklearn.metrics import classification_report
print (classification_report(messages['label'], all_predictions))              precision    recall  f1-score   support

         ham       0.98      1.00      0.99      4825
        spam       1.00      0.84      0.92       747

    accuracy                           0.98      5572
   macro avg       0.99      0.92      0.95      5572
weighted avg       0.98      0.98      0.98      5572

There are quite a few possible metrics for evaluating model performance. Which one is the most important depends on the task and the business effects of decisions based off of the model. For example, the cost of mis-predicting "spam" as "ham" is probably much lower than mis-predicting "ham" as "spam"._____no_output_____In the above "evaluation",we evaluated accuracy on the same data we used for training. **You should never actually evaluate on the same dataset you train on!**

Such evaluation tells us nothing about the true predictive power of our model. If we simply remembered each example during training, the accuracy on training data would trivially be 100%, even though we wouldn't be able to classify any new messages.

A proper way is to split the data into a training/test set, where the model only ever sees the **training data** during its model fitting and parameter tuning. The **test data** is never used in any way. This is then our final evaluation on test data is representative of true predictive performance._____no_output_____---------_____no_output_____# Train Test Split_____no_output_____

from sklearn.model_selection import train_test_split_____no_output_____msg_train, msg_test, label_train, label_test = train_test_split(messages['message'], messages['label'], test_size=0.3, random_state=42)_____no_output_____msg_train.shape, msg_test.shape, label_train.shape, label_test.shape_____no_output_____msg_train.head(2)_____no_output_____

As we need to redo all Text Pre-Processing on the splitted data, we will use Pipeline to redo all the tasks._____no_output_____## Creating a Data Pipeline

Let's run our model again and then predict off the test set. We will use SciKit Learn's [pipeline](http://scikit-learn.org/stable/modules/pipeline.html) capabilities to store a pipeline of workflow. This will allow us to set up all the transformations that we will do to the data for future use. Let's see an example of how it works:_____no_output_____

from sklearn.pipeline import Pipeline_____no_output_____steps = [
    ('bow', CountVectorizer(analyzer=text_process)),
    ('tfidf', TfidfTransformer()),
    ('classifier', MultinomialNB())
]

pipeline = Pipeline(steps)_____no_output_____pipeline.fit(msg_train, label_train)_____no_output_____predictions = pipeline.predict(msg_test)_____no_output_____print(classification_report(label_test, predictions))              precision    recall  f1-score   support

         ham       0.96      1.00      0.98      1448
        spam       1.00      0.73      0.85       224

    accuracy                           0.96      1672
   macro avg       0.98      0.87      0.91      1672
weighted avg       0.97      0.96      0.96      1672

-----------------_____no_output_____# Using RandomForest Classifier_____no_output_____

from sklearn.ensemble import RandomForestClassifier_____no_output_____steps = [
    ('bow', CountVectorizer(analyzer=text_process)),
    ('tfidf', TfidfTransformer()),
    ('classifier', RandomForestClassifier())
]

pipeline = Pipeline(steps)_____no_output_____pipeline.fit(msg_train, label_train)_____no_output_____predictions = pipeline.predict(msg_test)_____no_output_____print(classification_report(label_test, predictions))              precision    recall  f1-score   support

         ham       0.98      1.00      0.99      1448
        spam       1.00      0.84      0.91       224

    accuracy                           0.98      1672
   macro avg       0.99      0.92      0.95      1672
weighted avg       0.98      0.98      0.98      1672

## More Resources

Check out the links below for more info on Natural Language Processing:

[NLTK Book Online](http://www.nltk.org/book/)

[Kaggle Walkthrough](https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-1-for-beginners-bag-of-words)

[SciKit Learn's Tutorial](http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html)_____no_output_____