# Notebook from pshwetank/NIC-2015-Pytorch-
Path: training.ipynb

## In this notebook, the CNN-RNN model will be trained.

---

_____no_output_____
## Step 1: Training Setup

In this step of the notebook, you will customize the training of your CNN-RNN model by specifying hyperparameters and setting other options that are important to the training procedure.  The values you set now will be used when training your model in **Step 2** below.

### Task #1

Begin by setting the following variables:
- `batch_size` - the batch size of each training batch.  It is the number of image-caption pairs used to amend the model weights in each training step.
- `vocab_threshold` - the minimum word count threshold.  Note that a larger threshold will result in a smaller vocabulary, whereas a smaller threshold will include rarer words and result in a larger vocabulary.
- `vocab_from_file` - a Boolean that decides whether to load the vocabulary from file.
- `embed_size` - the dimensionality of the image and word embeddings.
- `hidden_size` - the number of features in the hidden state of the RNN decoder.
- `num_epochs` - the number of epochs to train the model.  We recommend that you set `num_epochs=3`, but feel free to increase or decrease this number as you wish.  [This paper](https://arxiv.org/pdf/1502.03044.pdf) trained a captioning model on a single state-of-the-art GPU for 3 days, but you'll soon see that you can get reasonable results in a matter of a few hours!  (_But of course, if you want your model to compete with current research, you will have to train for much longer._)
- `save_every` - determines how often to save the model weights.  We recommend that you set `save_every=1`, to save the model weights after each epoch.  This way, after the `i`th epoch, the encoder and decoder weights will be saved in the `models/` folder as `encoder-i.pkl` and `decoder-i.pkl`, respectively.
- `print_every` - determines how often to print the batch loss to the Jupyter notebook while training.  Note that you **will not** observe a monotonic decrease in the loss function while training - this is perfectly fine and completely expected!  You are encouraged to keep this at its default value of `100` to avoid clogging the notebook, but feel free to change it.
- `log_file` - the name of the text file containing - for every step - how the loss and perplexity evolved during training.

In order to decide over the above hyperparameters, these papars can be referred - [this paper](https://arxiv.org/pdf/1502.03044.pdf) & [this paper](https://arxiv.org/pdf/1411.4555.pdf) for useful guidance!  Train a single model, and proceed to the next notebook (**inference.ipynb**).  If you are unhappy with your performance, you can return to this notebook to tweak the hyperparameters (and/or the architecture in **model.py**) and re-train your model.

_____no_output_____

import nltk
nltk.download('punkt')[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
import torch
import torch.nn as nn
from torchvision import transforms
import sys
sys.path.append('/opt/cocoapi/PythonAPI')
from pycocotools.coco import COCO
from data_loader import get_loader
from model import EncoderCNN, DecoderRNN
import math

## TODO #1: Select appropriate values for the Python variables below.
batch_size = 128   # batch size
vocab_threshold = 8        # minimum word count threshold
vocab_from_file = True    # if True, load existing vocab file
embed_size = 300           # dimensionality of image and word embeddings
hidden_size = 128        # number of features in hidden state of the RNN decoder
num_epochs = 3             # number of training epochs
save_every = 1             # determines frequency of saving model weights
print_every = 100          # determines window for printing average loss
log_file = 'training_log.txt'       # name of file with saved training loss and perplexity

#2:image transform.
transform_train = transforms.Compose([
    transforms.Resize(256),                          # smaller edge of image resized to 256
    transforms.RandomCrop(224),                      # get 224x224 crop from random location
    transforms.RandomHorizontalFlip(),               # horizontally flip image with probability=0.5
    transforms.ToTensor(),                           # convert the PIL Image to a tensor
    transforms.Normalize((0.485, 0.456, 0.406),      # normalize image for pre-trained model
                         (0.229, 0.224, 0.225))])

# Build data loader.
data_loader = get_loader(transform=transform_train,
                         mode='train',
                         batch_size=batch_size,
                         vocab_threshold=vocab_threshold,
                         vocab_from_file=vocab_from_file)

# The size of the vocabulary.
vocab_size = len(data_loader.dataset.vocab)

# Initialize the encoder and decoder.
encoder = EncoderCNN(embed_size)
decoder = DecoderRNN(embed_size, hidden_size, vocab_size)

# Move models to GPU if CUDA is available.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
encoder.to(device)
decoder.to(device)

# Define the loss function.
criterion = nn.CrossEntropyLoss().cuda() if torch.cuda.is_available() else nn.CrossEntropyLoss()

#Specify the learnable parameters of the model.
params = list(decoder.parameters()) + list(encoder.embed.parameters())

#Define the optimizer.
optimizer = torch.optim.Adam(params, lr = 0.001)

# Set the total number of training steps per epoch.
total_step = math.ceil(len(data_loader.dataset.caption_lengths) / data_loader.batch_sampler.batch_size)loading annotations into memory...
Done (t=0.88s)
creating index...
index created!
[0/414113] Tokenizing captions...
[100000/414113] Tokenizing captions...
[200000/414113] Tokenizing captions...
[300000/414113] Tokenizing captions...
[400000/414113] Tokenizing captions...
loading annotations into memory...
Done (t=0.91s)
creating index...

## Step 2: Train your Model

Once you have executed the code cell in **Step 1**, the training procedure below should run without issue.

It is completely fine to leave the code cell below as-is without modifications to train your model.  However, if you would like to modify the code used to train the model below, you must ensure that your changes are easily parsed by your reviewer.  In other words, make sure to provide appropriate comments to describe how your code works!

You may find it useful to load saved weights to resume training.  In that case, note the names of the files containing the encoder and decoder weights that you'd like to load (`encoder_file` and `decoder_file`).  Then you can load the weights by using the lines below:

```python
# Load pre-trained weights before resuming training.
encoder.load_state_dict(torch.load(os.path.join('./models', encoder_file)))
decoder.load_state_dict(torch.load(os.path.join('./models', decoder_file)))
```

While trying out parameters, make sure to take extensive notes and record the settings that you used in your various training runs.  In particular, you don't want to encounter a situation where you've trained a model for several hours but can't remember what settings you used :).

### A Note on Tuning Hyperparameters

To figure out how well your model is doing, you can look at how the training loss and perplexity evolve during training - and for the purposes of this project.

However, this will not tell you if your model is overfitting to the training data, and, unfortunately, overfitting is a problem that is commonly encountered when training image captioning models.

For now,  Try to train your model for the suggested 3 epochs without worrying about performance; then, you should immediately transition to the next notebook in the sequence (**inference.ipynb**) to see how your model performs on the test data.  If your model needs to be changed, you can come back to this notebook, amend hyperparameters (if necessary), and re-train the model.

That said, if you would like to go above and beyond in this project, you can read about some approaches to minimizing overfitting in section 4.3.1 of [this paper](http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7505636).  In the next (optional) step of this notebook, we provide some guidance for assessing the performance on the validation dataset._____no_output_____

import torch.utils.data as data
import numpy as np
import os
import requests
import time

# Open the training log file.
f = open(log_file, 'w')

old_time = time.time()
response = requests.request("GET",
                            "http://metadata.google.internal/computeMetadata/v1/instance/attributes/keep_alive_token",
                            headers={"Metadata-Flavor":"Google"})

for epoch in range(1, num_epochs+1):

    for i_step in range(1, total_step+1):

        if time.time() - old_time > 60:
            old_time = time.time()
            requests.request("POST",
                             "https://nebula.udacity.com/api/v1/remote/keep-alive",
                             headers={'Authorization': "STAR " + response.text})

        # Randomly sample a caption length, and sample indices with that length.
        indices = data_loader.dataset.get_train_indices()
        # Create and assign a batch sampler to retrieve a batch with the sampled indices.
        new_sampler = data.sampler.SubsetRandomSampler(indices=indices)
        data_loader.batch_sampler.sampler = new_sampler

        # Obtain the batch.
        images, captions = next(iter(data_loader))

        # Move batch of images and captions to GPU if CUDA is available.
        images = images.to(device)
        captions = captions.to(device)

        # Zero the gradients.
        decoder.zero_grad()
        encoder.zero_grad()

        # Pass the inputs through the CNN-RNN model.
        features = encoder(images)
        outputs = decoder(features, captions)

        # Calculate the batch loss.
        loss = criterion(outputs.view(-1, vocab_size), captions.view(-1))

        # Backward pass.
        loss.backward()

        # Update the parameters in the optimizer.
        optimizer.step()

        # Get training statistics.
        stats = 'Epoch [%d/%d], Step [%d/%d], Loss: %.4f, Perplexity: %5.4f' % (epoch, num_epochs, i_step, total_step, loss.item(), np.exp(loss.item()))

        # Print training statistics (on same line).
        print('\r' + stats, end="")
        sys.stdout.flush()

        # Print training statistics to file.
        f.write(stats + '\n')
        f.flush()

        # Print training statistics (on different line).
        if i_step % print_every == 0:
            print('\r' + stats)

    # Save the weights.
    if epoch % save_every == 0:
        torch.save(decoder.state_dict(), os.path.join('./models', 'decoder-%d.pkl' % epoch))
        torch.save(encoder.state_dict(), os.path.join('./models', 'encoder-%d.pkl' % epoch))

# Close the training log file.
f.close()Epoch [1/3], Step [100/4142], Loss: 4.0316, Perplexity: 56.3500
Epoch [1/3], Step [200/4142], Loss: 3.6704, Perplexity: 39.26596
Epoch [1/3], Step [300/4142], Loss: 3.5332, Perplexity: 34.2332
Epoch [1/3], Step [400/4142], Loss: 3.3520, Perplexity: 28.5605
Epoch [1/3], Step [500/4142], Loss: 3.2972, Perplexity: 27.03573
Epoch [1/3], Step [600/4142], Loss: 3.1514, Perplexity: 23.3688
Epoch [1/3], Step [700/4142], Loss: 3.8084, Perplexity: 45.0803
Epoch [1/3], Step [800/4142], Loss: 3.0559, Perplexity: 21.2397
Epoch [1/3], Step [900/4142], Loss: 3.2008, Perplexity: 24.5515
Epoch [1/3], Step [1000/4142], Loss: 3.1258, Perplexity: 22.7787
Epoch [1/3], Step [1100/4142], Loss: 3.1089, Perplexity: 22.39703
Epoch [1/3], Step [1200/4142], Loss: 3.2686, Perplexity: 26.2749
Epoch [1/3], Step [1300/4142], Loss: 4.0019, Perplexity: 54.6999
Epoch [1/3], Step [1360/4142], Loss: 2.9833, Perplexity: 19.7539


# Notebook from safarie1103/pydata-book
Path: ch07.ipynb

# Data Cleaning and Preparation_____no_output_____

import numpy as np
import pandas as pd
PREVIOUS_MAX_ROWS = pd.options.display.max_rows
pd.options.display.max_rows = 20
np.random.seed(12345)
import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))
np.set_printoptions(precision=4, suppress=True)_____no_output_____

## Handling Missing Data_____no_output_____

string_data = pd.Series(['aardvark', 'artichoke', np.nan, 'avocado'])
string_data
string_data.isnull()_____no_output_____string_data[0] = None
string_data.isnull()_____no_output_____

### Filtering Out Missing Data_____no_output_____

from numpy import nan as NA
data = pd.Series([1, NA, 3.5, NA, 7])
data.dropna()_____no_output_____data[data.notnull()]_____no_output_____data = pd.DataFrame([[1., 6.5, 3.], [1., NA, NA],
                     [NA, NA, NA], [NA, 6.5, 3.]])
cleaned = data.dropna()
data
cleaned_____no_output_____data.dropna(how='all')_____no_output_____data[4] = NA
data
data.dropna(axis=1, how='all')_____no_output_____df = pd.DataFrame(np.random.randn(7, 3))
df.iloc[:4, 1] = NA
df.iloc[:2, 2] = NA
df
df.dropna()
df.dropna(thresh=2)_____no_output_____

### Filling In Missing Data_____no_output_____

df.fillna(0)_____no_output_____df.fillna({1: 0.5, 2: 0})_____no_output______ = df.fillna(0, inplace=True)
df_____no_output_____df = pd.DataFrame(np.random.randn(6, 3))
df.iloc[2:, 1] = NA
df.iloc[4:, 2] = NA
df
df.fillna(method='ffill')
df.fillna(method='ffill', limit=2)_____no_output_____data = pd.Series([1., NA, 3.5, NA, 7])
data.fillna(data.mean())_____no_output_____

## Data Transformation_____no_output_____### Removing Duplicates_____no_output_____

data = pd.DataFrame({'k1': ['one', 'two'] * 3 + ['two'],
                     'k2': [1, 1, 2, 3, 3, 4, 4]})
data_____no_output_____data.duplicated()_____no_output_____data.drop_duplicates()_____no_output_____data['v1'] = range(7)
data.drop_duplicates(['k1'])_____no_output_____data.drop_duplicates(['k1', 'k2'], keep='last')_____no_output_____

### Transforming Data Using a Function or Mapping_____no_output_____

data = pd.DataFrame({'food': ['bacon', 'pulled pork', 'bacon',
                              'Pastrami', 'corned beef', 'Bacon',
                              'pastrami', 'honey ham', 'nova lox'],
                     'ounces': [4, 3, 12, 6, 7.5, 8, 3, 5, 6]})
data_____no_output_____meat_to_animal = {
  'bacon': 'pig',
  'pulled pork': 'pig',
  'pastrami': 'cow',
  'corned beef': 'cow',
  'honey ham': 'pig',
  'nova lox': 'salmon'
}_____no_output_____lowercased = data['food'].str.lower()
lowercased
data['animal'] = lowercased.map(meat_to_animal)
data_____no_output_____data['food'].map(lambda x: meat_to_animal[x.lower()])_____no_output_____

### Replacing Values_____no_output_____

data = pd.Series([1., -999., 2., -999., -1000., 3.])
data_____no_output_____data.replace(-999, np.nan)_____no_output_____data.replace([-999, -1000], np.nan)_____no_output_____data.replace([-999, -1000], [np.nan, 0])_____no_output_____data.replace({-999: np.nan, -1000: 0})_____no_output_____

### Renaming Axis Indexes_____no_output_____

data = pd.DataFrame(np.arange(12).reshape((3, 4)),
                    index=['Ohio', 'Colorado', 'New York'],
                    columns=['one', 'two', 'three', 'four'])_____no_output_____transform = lambda x: x[:4].upper()
data.index.map(transform)_____no_output_____data.index = data.index.map(transform)
data_____no_output_____data.rename(index=str.title, columns=str.upper)_____no_output_____data.rename(index={'OHIO': 'INDIANA'},
            columns={'three': 'peekaboo'})_____no_output_____data.rename(index={'OHIO': 'INDIANA'}, inplace=True)
data_____no_output_____

### Discretization and Binning_____no_output_____

ages = [20, 22, 25, 27, 21, 23, 37, 31, 61, 45, 41, 32]_____no_output_____bins = [18, 25, 35, 60, 100]
cats = pd.cut(ages, bins)
cats_____no_output_____cats.codes
cats.categories
pd.value_counts(cats)_____no_output_____pd.cut(ages, [18, 26, 36, 61, 100], right=False)_____no_output_____group_names = ['Youth', 'YoungAdult', 'MiddleAged', 'Senior']
pd.cut(ages, bins, labels=group_names)_____no_output_____data = np.random.rand(20)
pd.cut(data, 4, precision=2)_____no_output_____data = np.random.randn(1000)  # Normally distributed
cats = pd.qcut(data, 4)  # Cut into quartiles
cats
pd.value_counts(cats)_____no_output_____pd.qcut(data, [0, 0.1, 0.5, 0.9, 1.])_____no_output_____

### Detecting and Filtering Outliers_____no_output_____

data = pd.DataFrame(np.random.randn(1000, 4))
data.describe()_____no_output_____col = data[2]
col[np.abs(col) > 3]_____no_output_____data[(np.abs(data) > 3).any(1)]_____no_output_____data[np.abs(data) > 3] = np.sign(data) * 3
data.describe()_____no_output_____np.sign(data).head()_____no_output_____

### Permutation and Random Sampling_____no_output_____

df = pd.DataFrame(np.arange(5 * 4).reshape((5, 4)))
sampler = np.random.permutation(5)
sampler_____no_output_____df
df.take(sampler)_____no_output_____df.sample(n=3)_____no_output_____choices = pd.Series([5, 7, -1, 6, 4])
draws = choices.sample(n=10, replace=True)
draws_____no_output_____

### Computing Indicator/Dummy Variables_____no_output_____

df = pd.DataFrame({'key': ['b', 'b', 'a', 'c', 'a', 'b'],
                   'data1': range(6)})
pd.get_dummies(df['key'])_____no_output_____dummies = pd.get_dummies(df['key'], prefix='key')
df_with_dummy = df[['data1']].join(dummies)
df_with_dummy_____no_output_____mnames = ['movie_id', 'title', 'genres']
movies = pd.read_table('datasets/movielens/movies.dat', sep='::',
                       header=None, names=mnames)
movies[:10]_____no_output_____all_genres = []
for x in movies.genres:
    all_genres.extend(x.split('|'))
genres = pd.unique(all_genres)_____no_output_____genres_____no_output_____zero_matrix = np.zeros((len(movies), len(genres)))
dummies = pd.DataFrame(zero_matrix, columns=genres)_____no_output_____gen = movies.genres[0]
gen.split('|')
dummies.columns.get_indexer(gen.split('|'))_____no_output_____for i, gen in enumerate(movies.genres):
    indices = dummies.columns.get_indexer(gen.split('|'))
    dummies.iloc[i, indices] = 1_____no_output_____movies_windic = movies.join(dummies.add_prefix('Genre_'))
movies_windic.iloc[0]_____no_output_____np.random.seed(12345)
values = np.random.rand(10)
values
bins = [0, 0.2, 0.4, 0.6, 0.8, 1]
pd.get_dummies(pd.cut(values, bins))_____no_output_____

## String Manipulation_____no_output_____### String Object Methods_____no_output_____

val = 'a,b,  guido'
val.split(',')_____no_output_____pieces = [x.strip() for x in val.split(',')]
pieces_____no_output_____first, second, third = pieces
first + '::' + second + '::' + third_____no_output_____'::'.join(pieces)_____no_output_____'guido' in val
val.index(',')
val.find(':')_____no_output_____val.index(':')_____no_output_____val.count(',')_____no_output_____val.replace(',', '::')
val.replace(',', '')_____no_output_____

### Regular Expressions_____no_output_____

import re
text = "foo    bar\t baz  \tqux"
re.split('\s+', text)_____no_output_____regex = re.compile('\s+')
regex.split(text)_____no_output_____regex.findall(text)_____no_output_____text = """Dave dave@google.com
Steve steve@gmail.com
Rob rob@gmail.com
Ryan ryan@yahoo.com
"""
pattern = r'[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}'

# re.IGNORECASE makes the regex case-insensitive
regex = re.compile(pattern, flags=re.IGNORECASE)_____no_output_____regex.findall(text)_____no_output_____m = regex.search(text)
m
text[m.start():m.end()]_____no_output_____print(regex.match(text))_____no_output_____print(regex.sub('REDACTED', text))_____no_output_____pattern = r'([A-Z0-9._%+-]+)@([A-Z0-9.-]+)\.([A-Z]{2,4})'
regex = re.compile(pattern, flags=re.IGNORECASE)_____no_output_____m = regex.match('wesm@bright.net')
m.groups()_____no_output_____regex.findall(text)_____no_output_____print(regex.sub(r'Username: \1, Domain: \2, Suffix: \3', text))_____no_output_____

### Vectorized String Functions in pandas_____no_output_____

data = {'Dave': 'dave@google.com', 'Steve': 'steve@gmail.com',
        'Rob': 'rob@gmail.com', 'Wes': np.nan}
data = pd.Series(data)
data
data.isnull()_____no_output_____data.str.contains('gmail')_____no_output_____pattern
data.str.findall(pattern, flags=re.IGNORECASE)_____no_output_____matches = data.str.match(pattern, flags=re.IGNORECASE)
matches_____no_output_____matches.str.get(1)
matches.str[0]_____no_output_____data.str[:5]_____no_output_____pd.options.display.max_rows = PREVIOUS_MAX_ROWS_____no_output_____

## Conclusion_____no_output_____


# Notebook from TeamMacLean/ruth-effectors-prediction
Path: scripts/python-scripts/heatmaps/0014_visualise_heatmap_cnn_gru_secreted.ipynb

# Heatmap of Convolutional Layer in CNN-GRU model for Bacteria_____no_output_____

# Load the libraries
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from matplotlib.colors import ListedColormap
from tensorflow.compat.v1.keras.models import load_model_____no_output_____import cv2
import glob_____no_output_____# Get the pretarined model and show the summary of the model
cnn_gru_bacteria = load_model("../../../data/secreted_data/saved_models/bacteria/cnn_gru/sequential_1.40-0.10.hdf5")
cnn_gru_bacteria.summary()Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv1d_1 (Conv1D)            (None, 2574, 16)          336
_________________________________________________________________
max_pooling1d_1 (MaxPooling1 (None, 1287, 16)          0
_________________________________________________________________
bidirectional_1 (Bidirection (None, 32)                3168
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 33
=================================================================
Total params: 3,537
Trainable params: 3,537
Non-trainable params: 0
_________________________________________________________________

# Function_____no_output_____

# Load datasets
dataset = np.load('../../../data/secreted_data/ready_to_process/encoded_files/bacteria/x_train.npy')_____no_output_____def get_sum_heatmap_from_files(data_name, class_eff, layer, verbose = True, dataset = dataset):
    npy_loading_pattern = "results_secreted/all_matrices_" + class_eff + "_" + data_name + "_" + layer + "*.npy"
    data_loading_path = glob.glob(npy_loading_pattern)

    if verbose: print("Loading", data_name, "data from", len(data_loading_path) ,"files...")

    # Initalize empty matrix
    all_matrices = np.empty([0 , 1, np.load(data_loading_path[0]).shape[2]])

    # Append matrices from files
    for path in data_loading_path:
        data = np.load(path)
        all_matrices = np.concatenate((all_matrices, data))

    if verbose: print("Loaded", all_matrices.shape[0], "data samples")
    sum_all_matrices = np.sum(all_matrices, axis = 0)

    # Calculate heatmap
    heatmap = np.maximum(sum_all_matrices, 0)
    heatmap /= np.max(heatmap)
    heatmap = cv2.resize(heatmap, (dataset.shape[1], dataset.shape[2]))

    return(sum_all_matrices, heatmap)_____no_output_____def plot_heatmap(data, from_data, to_data, colorbar = None):
    plt.matshow(data[:,from_data:to_data], cmap = 'viridis', origin = "lower", vmin = 0, vmax = 1)
    ax = plt.gca()
    plot = ax.axes.get_yaxis().set_visible(False)
    if colorbar == "Yes":
        plt.colorbar()

    return plot_____no_output_____def plot_fft(data, threshold, length_to_plot):

    # reshape data
    data = data.reshape(data.shape[1])

    # transform real data to the frequency domain (Fourier transform)
    real_ft = np.fft.rfft(data)

    # remove undesired frequencies
    real_ft[threshold:] = 0

    # move back to the time domain. (Inverse fourier transform)
    smooth_data = np.fft.irfft(real_ft)

    # plot the results
    plt.figure(figsize=[30, 12])
    plt.plot(data[:length_to_plot], label='Original')
    plt.plot(smooth_data[0:length_to_plot],'r', linewidth=3, label='Smooth')
    plt.legend(loc=0)
    plt.show()_____no_output_____# Define the function

def plot_bar(data, from_data = None, to_data = None):

    if from_data is not None and to_data is not None:
        data = data[:,from_data:to_data].ravel()
    else:
        data = data.ravel()

    plt.plot(data)
    fig = plt.gcf()
    fig.set_size_inches(30, 10)
    plt.xlabel('Position')
    plt.ylabel('Importance Score')_____no_output_____

## Bacteria_____no_output_____

sum_all_matrices_xtrain, x_train_bacteria_gru = get_sum_heatmap_from_files("x_train", "bacteria", "conv1d_1")Loading x_train data from 1 files...
Loaded 228 data samples
# Reshape the sum of matrices into 1D array
# sum_all_matrices_xtrain = sum_all_matrices_xtrain.reshape(2574,)_____no_output_____sum_all_matrices_xval, x_val_bacteria_gru = get_sum_heatmap_from_files("x_val", "bacteria", "conv1d_1")Loading x_val data from 1 files...
Loaded 76 data samples
sum_all_matrices_xtest, x_test_bacteria_gru = get_sum_heatmap_from_files("x_test", "bacteria", "conv1d_1")Loading x_test data from 1 files...
Loaded 76 data samples

### Plot the training data_____no_output_____

plot_heatmap(x_train_bacteria_gru, x_train_bacteria_gru.shape[0], x_train_bacteria_gru.shape[1])_____no_output_____plot_heatmap(x_train_bacteria_gru, 0, 75)_____no_output_____

### Plot the validation data_____no_output_____

plot_heatmap(x_val_bacteria_gru, x_val_bacteria_gru.shape[0], x_val_bacteria_gru.shape[1])_____no_output_____plot_heatmap(x_val_bacteria_gru, 0, 75)_____no_output_____

### Plot the testing data_____no_output_____

plot_heatmap(x_test_bacteria_gru, x_test_bacteria_gru.shape[0], x_test_bacteria_gru.shape[1])_____no_output_____plot_heatmap(x_test_bacteria_gru, 0, 100)_____no_output_____

## Visualizing the important feature with bar chart_____no_output_____

pd.DataFrame({"x" : sum_all_matrices_xtrain[0].tolist()}).plot.line(figsize=(10,5))_____no_output_____# pd.DataFrame({"x" : sum_all_matrices_xtrain[0,0:175].tolist()}).plot.line(figsize=(4,2))_____no_output_____pd.DataFrame({"x" : sum_all_matrices_xval[0].tolist()}).plot.line(figsize=(10,5))_____no_output_____# pd.DataFrame({"x" : sum_all_matrices_xval[0,0:175].tolist()}).plot.line(figsize=(4,2))_____no_output_____pd.DataFrame({"x" : sum_all_matrices_xtest[0].tolist()}).plot.line(figsize=(10,5))_____no_output_____# pd.DataFrame({"x" : sum_all_matrices_xtest[0,0:175].tolist()}).plot.line(figsize=(4,2))_____no_output_____

### Training data_____no_output_____

rft_train = np.fft.rfft(x_train_bacteria_gru[0])
rft_train[50:] = 0
y_smooth_train = np.fft.irfft(rft_train)_____no_output_____sum_df_train = pd.DataFrame(
    {
        "train" : x_train_bacteria_gru[0].tolist(),
        "train_smooth": y_smooth_train.tolist()
    }
)_____no_output_____sum_df_train[0:1000].plot.line(figsize=(10,5), color = ['salmon', 'steelblue'], linewidth = 2, xticks = np.linspace(0, 1000, num = 21))_____no_output_____

### Validation data_____no_output_____

rft_val = np.fft.rfft(x_val_bacteria_gru[0])
rft_val[50:] = 0
y_smooth_val = np.fft.irfft(rft_val)_____no_output_____sum_df_val = pd.DataFrame(
    {
        "val" :   x_val_bacteria_gru[0].tolist(),
        "val_smooth":   y_smooth_val.tolist()
    }
)_____no_output_____sum_df_val[0:1000].plot.line(figsize=(10,5), color = ['salmon', 'steelblue'], linewidth = 2, xticks = np.linspace(0, 1000, num = 21))_____no_output_____

### Testing data_____no_output_____

rft_test = np.fft.rfft(x_test_bacteria_gru[0])
rft_test[50:] = 0
y_smooth_test = np.fft.irfft(rft_test)_____no_output_____sum_df_test = pd.DataFrame(
    {
        "test" :  x_test_bacteria_gru[0].tolist(),
        "test_smooth":  y_smooth_test.tolist()
    }
)_____no_output_____sum_df_test[0:1000].plot.line(figsize=(10,5), color = ['salmon', 'steelblue'], linewidth = 2, xticks = np.linspace(0, 1000, num = 21))_____no_output_____

### Plot all of the fft results together_____no_output_____

from plotnine import *_____no_output_____df_fft_all_together = pd.DataFrame(
    {
        "index" : np.concatenate((
            np.arange(0, np.shape(y_smooth_train)[0], 1),
            np.arange(0, np.shape(y_smooth_val)[0], 1),
            np.arange(0, np.shape(y_smooth_test)[0], 1)
        )),
        "value" : np.concatenate((
            y_smooth_train.tolist(),
            y_smooth_val.tolist(),
            y_smooth_test.tolist()
        )),
        "dataset" : np.concatenate((
            np.repeat("train", np.shape(y_smooth_train)[0]),
            np.repeat("val", np.shape(y_smooth_val)[0]),
            np.repeat("test", np.shape(y_smooth_test)[0])
        )),
    }
)_____no_output_____bacteria_all_datasets_plot = (
    ggplot(df_fft_all_together.query('index < 1000')) +
    aes(x = 'index', y = 'value', color = 'dataset') +
    geom_line() +
    geom_vline(xintercept = 25, linetype = 'dashed') +
    geom_vline(xintercept = 50, linetype = 'dashed') +
#     geom_vline(xintercept = 80, linetype = 'dashed') +
    scale_color_cmap_d() +
    scale_x_continuous(breaks = np.arange(0, 1000, 50)) +
    labs(x = 'Position') +
    theme_bw() +
    theme(figure_size = [10,4])
)

ggsave(plot = bacteria_all_datasets_plot, filename = "bacteria_all_datasets_plot.png", path = "fig")/anaconda3/envs/tensorflow2/lib/python3.7/site-packages/plotnine/ggplot.py:729: PlotnineWarning: Saving 10 x 4 in image.
  from_inches(height, units), units), PlotnineWarning)
/anaconda3/envs/tensorflow2/lib/python3.7/site-packages/plotnine/ggplot.py:730: PlotnineWarning: Filename: fig/bacteria_all_datasets_plot.png
  warn('Filename: {}'.format(filename), PlotnineWarning)


# Notebook from Felipe-Tommaselli/Sistemas_evolutivos
Path: Bloco3/aula5.ipynb

# BLOCO 3: Aula 5

* link https://drive.google.com/file/d/1-ooVRlmOONFZhiUNCR6dh9enuDSUEzkY/view

* link do quadro com resumão https://gitlab.com/simoesusp/disciplinas/-/blob/master/SSC0713-Sistemas-Evolutivos-Aplicados-a-Robotica/MaterialAulaDistancia/LousaImagens_PrimeiroSemestre2021/Evolutivos_ESTRATEGIAS_AG03.svg_____no_output_____## Algoritmos evolutivos (resumo e aprofundamento)

* Revisão: estabas dos alg evolutivos
    1. Inicia populacao
    2. Avaliação (normalmente a etapa mais lenta)
    3. Seleção
    4. Crossover
    5. Mutação
    6. Rearranjo da população

> 1. Incia populacao

- 100 geracoes e 10 individuos ou 10 geracoes e 100 individuos? DEPENDE
    - A principio a ideia é utilizar o maior numero POSSIVEL de individuos, dessa forma a solução inicial ficaria espalhada pelo conjunto de possibilidades e causaria uma convergencia mais rapida, porem pelo baixo numero de geracoes ela seria grosseira
    - Um alto numero de geracoes pode gerar uma solucao mais refinada e esperta, deve-se atentar aos viese (há a chance de todos os individuos nascerem perto de maximos locais e nao encontrar o max global)
    - Dependendo da aplicação cada caso utilizira uma ideia, porem deve-se sempre optar pelo maior numero de individuos com o maior numero de geracoes possivel

* principalmente na avaliacao, se aumentarmos um gene por exemplo NA HORAD E INICIALIZAR A POPULACAO, nós dobramos todo o trabalho que deve ser fetio considerando que a varaivel é um unsigned char (binario), se for um int ou um double, isso entra em valores absurdos (tomar cuidado com o custo algoritmo e com os tipos escolhidos nas variaveis)
    * fazer a analise do espaco de busca: num de possibilidades = tipo dos genes^num de genes

- bias in evolution x sem bia in evolution
    * testar previamente para onde ta os valores e da uma analisada
    * meu mano simoes é meio contra

> 2. Avaliação

- avaliacao na realidade é uam tarefa mto custosa e demorada, é importante parar o fitness quando a curva estabilizar (aquela curva que parece um log)

* na pratica nao havera funcoes de modelagem boa como uma f(x)
_____no_output_____


# Notebook from Nabizeus/merkmalskonstruktion
Path: Kap2-Logarithmustransformation-28-JAN-2022.ipynb

<code>
import pandas as pd
import json
import matplotlib.pyplot as plt
import numpy as np_____no_output_____# Daten der Geschäfte laden aus JSON Format
biz_file =  open('data/yelp_academic_dataset_business.json')
biz_df = pd.DataFrame([json.loads(x) for x in biz_file.readlines()])
biz_file.close()_____no_output_____biz_df['log_review_count']=np.log10(biz_df['review_count']+1)_____no_output_____fig, (ax1, ax2) = plt.subplots(2,1)
biz_df['review_count'].hist(ax=ax1, bins=100)
ax1.tick_params(labelsize=14)
ax1.set_xlabel('review_count', fontsize=14)
ax1.set_ylabel('Occurrence', fontsize=14)
biz_df['log_review_count'].hist(ax=ax2, bins=100)
ax2.tick_params(labelsize=14)
ax2.set_xlabel('log10(review_count)', fontsize=14)
ax2.set_ylabel('Occurrence', fontsize=14)_____no_output_____# Am Werk

# Beispiel 2.8 - Count vs. Sterne?

# Lerne lineare Regressionsmodelle an, um die mittlere Sternebewertung eines
# Geschäfts anhand des Merkmals review_count mit und ohne
# Logarithmustransformation vorherzusagen.
# Vergleiche die Bewertun beider Modelle nach zehnfacher Kreuzvalidierung.
from sklearn import linear_model
from sklearn.model_selection import cross_val_score
m_orig = linear_model.LinearRegression()
scores_orig = cross_val_score(m_orig, biz_df[['review_count']], biz_df['stars'], cv=10)
m_log = linear_model.LinearRegression()
scores_log = cross_val_score(m_log, biz_df[['log_review_count']], biz_df['stars'], cv=10)

print('R-squared score without log transform: %0.5f (+/- %0.5f)' % (scores_orig.mean(), 2*scores_orig.std()))
print('R-squared score with log transform: %0.5f (+/- %0.5f)' % (scores_log.mean(), 2*scores_log.std()))R-squared score without log transform: 0.00179 (+/- 0.00417)
R-squared score with log transform: 0.00069 (+/- 0.00585)
plt.scatter(biz_df[['review_count']], biz_df['stars'])

# Total bescheuert sowas mit einem linearen Model zu machen_____no_output_____# Beispiel 2.9 - Online News

# Datensatz

df = pd.read_csv('data/ONP_sub_test.csv',delimiter=',', skipinitialspace=True)

df.head()_____no_output_____# Bilde die Logarithmustransformation des Merkmals 'n_tokens_content', das
# die Zahl der Wörter (Tokens) in einer Meldung beschreibt.
df['log_n_tokens_content'] = np.log10(df['n_tokens_content']+1)
#print(df.columns)
#print(df['timedelta'])
#df.head()_____no_output_____# Lerne zwei lineare Regressionsmodelle an, um die Anzahl von
# Weiterverbreitungen eines Artikels vorherzusagen, wobei eines das Merkmal
# selbst und das andere das logarithmierte Merkmal benutzt.

from sklearn import linear_model
from sklearn.model_selection import cross_val_score
m_orig = linear_model.LinearRegression()
scores_orig = cross_val_score(m_orig, df[['n_tokens_content']],
                             df['shares'], cv=10)

m_log = linear_model.LinearRegression()
scores_log = cross_val_score(m_log, df[['log_n_tokens_content']],
                             df['shares'], cv=10)
print('R-squared score without log transform: %0.5f (+/- %0.5f)' % (scores_orig.mean(), 2*scores_orig.std()))
print('R-squared score with log transform: %0.5f (+/- %0.5f)' % (scores_log.mean(), 2*scores_log.std()))
R-squared score without log transform: -0.30116 (+/- 1.01105)
R-squared score with log transform: -0.28651 (+/- 1.04767)
# Beispiel 2-10 Visualisierung der Korrelation

fig2, (ax1, ax2) = plt.subplots(2,1)
ax1.scatter(df['n_tokens_content'], df['shares'])
ax1.tick_params(labelsize=14)
ax1.set_xlabel('Number of Words in Article', fontsize=14)
#ax1.set_ylabel('Number of Shares', fontsize=14)

ax2.scatter(df['log_n_tokens_content'], df['shares'])
ax2.tick_params(labelsize=14)
ax2.set_xlabel('Log of the Number of Words in Article', fontsize=14)
ax2.set_ylabel('Number of Shares', fontsize=14)
_____no_output_____# Beispiel 2-11

fig, (ax1, ax2) = plt.subplots(2,1)
ax1.scatter(biz_df['review_count'], biz_df['stars'])
ax1.tick_params(labelsize=14)
ax1.set_xlabel('Review Count', fontsize=14)
#ax1.set_ylabel('Number of Shares', fontsize=14)

ax2.scatter(biz_df['log_review_count'], biz_df['stars'])
ax2.tick_params(labelsize=14)
ax2.set_xlabel('Log of Review Count', fontsize=14)
ax2.set_ylabel('Average Star Rating', fontsize=14)
_____no_output_____# Next Potenztransformation

# Bespiel 2-12 Box-Cox Transformation

from scipy import stats

# Anschließend an das vorherige Beispiel gehen wir davon aus, dass biz_df die
# Yelp-Daten zu Geschätsrezensionen enthält.
# Die Box-Cox-Transformation setzt positive Eingangsdaten voraus.
# Wir prüfen zur Sicherheit das Minimum.

biz_df['review_count'].min()_____no_output_____# Setzen wir den Eingangsparameter lmbda auf 0, erhalten wir die
# Logarithmustransformation (ohne Verschiebung um eine Konstante).
rc_log = stats.boxcox(biz_df['review_count'], lmbda=0)
 # Von Haus aus bestimmt die scipy-Implementierung der Box-Cox-Transformation
# denjenigen lambda-Parameter, der die Ausgabe am besten an eine
# Normalverteilung annähert.
rc_bc, bc_params = stats.boxcox(biz_df['review_count'])
bc_params_____no_output_____biz_df['rc_log'] = rc_log
biz_df['rc_bc'] = rc_bc_____no_output_____# Beispiel 2-13: Viz

fig, (ax1, ax2, ax3) = plt.subplots(3,1)
# HIstogramm der ursprünglichen Rezensionszahlen
biz_df['review_count'].hist(ax=ax1, bins=100)
ax1.set_yscale('log')
ax1.tick_params(labelsize=14)
ax1.set_title('Review Counts Histogram', fontsize=14)
ax1.set_xlabel('')
ax1.set_ylabel('Occurrence', fontsize=14)

# Rezensionszahlen nach der Logarithmustransformation
biz_df['rc_log'].hist(ax=ax2, bins=100)
ax2.set_yscale('log')
ax2.tick_params(labelsize=14)
ax2.set_title('Log Transformed Counts Histogram', fontsize=14)
ax2.set_xlabel('')
ax2.set_ylabel('Occurrence', fontsize=14)

# Rezensionszahlen nach der optimalen Box-Cox-Transformation
biz_df['rc_bc'].hist(ax=ax3, bins=100)
ax3.set_yscale('log')
ax3.tick_params(labelsize=14)
ax3.set_title('Box-Cox Transformed Counts Histogram', fontsize=14)
ax3.set_xlabel('')
ax3.set_ylabel('Occurrence', fontsize=14)
_____no_output_____# Beispiel 2-14
# Wahrscheinlichkeitplot vs. Normal der ursprünglichen Rezensionszahlen
fig, (ax1, ax2, ax3) = plt.subplots(3,1)
prob1 = stats.probplot(biz_df['review_count'], dist=stats.norm, plot=ax1)

ax1.set_xlabel('')
ax1.set_title('Probplot against normal distribution', fontsize=14)

# Wahrscheinlichkeitplot vs. Normal der log Rezensionszahlen

prob1 = stats.probplot(biz_df['rc_log'], dist=stats.norm, plot=ax2)

ax2.set_xlabel('')
ax2.set_title('Probplot against log transform', fontsize=14)

# Wahrscheinlichkeitplot vs. Normal der boxcox Rezensionszahlen

prob1 = stats.probplot(biz_df['rc_bc'], dist=stats.norm, plot=ax3)

ax3.set_xlabel('Theoretical quantiles')
ax3.set_title('Probplot after Box-Cox transform', fontsize=14)
_____no_output_____
</code>