# Notebook from star14ms/transformer-evolution
Path: t5/t5-02.ipynb

## T5 구현 과정 (2/2)
T5 모델 구현에 대한 설명 입니다.

이 내용을 확인하기 전 아래 내용을 확인하시기 바랍니다.
- [Sentencepiece를 활용해 Vocab 만들기](https://paul-hyun.github.io/vocab-with-sentencepiece/)
- [Naver 영화리뷰 감정분석 데이터 전처리 하기](https://paul-hyun.github.io/preprocess-nsmc/)
- [Transformer (Attention Is All You Need) 구현하기 (1/3)](https://paul-hyun.github.io/transformer-01/)
- [Transformer (Attention Is All You Need) 구현하기 (2/3)](https://paul-hyun.github.io/transformer-02/)
- [Transformer (Attention Is All You Need) 구현하기 (3/3)](https://paul-hyun.github.io/transformer-03/)

[Colab](https://colab.research.google.com/)에서 실행 했습니다._____no_output_____#### 0. Pip Install
필요한 패키지를 pip를 이용해서 설치합니다._____no_output_____

!pip install sentencepiece
!pip install wgetCollecting sentencepiece
  Downloading https://files.pythonhosted.org/packages/74/f4/2d5214cbf13d06e7cb2c20d84115ca25b53ea76fa1f0ade0e3c9749de214/sentencepiece-0.1.85-cp36-cp36m-manylinux1_x86_64.whl (1.0MB)

     |▎                               | 10kB 12.9MB/s eta 0:00:01
     |▋                               | 20kB 5.2MB/s eta 0:00:01
     |█                               | 30kB 7.0MB/s eta 0:00:01
     |█▎                              | 40kB 6.8MB/s eta 0:00:01
     |█▋                              | 51kB 5.7MB/s eta 0:00:01
     |██                              | 61kB 5.9MB/s eta 0:00:01
     |██▏                             | 71kB 6.4MB/s eta 0:00:01
     |██▌                             | 81kB 7.2MB/s eta 0:00:01
     |██▉                             | 92kB 7.6MB/s eta 0:00:01
     |███▏                            | 102kB 7.0MB/s eta 0:00:01
     |███▌                            | 112kB 7.0MB/s eta 0:00:01
     |███▉                            | 122kB 7.0MB/s eta 0:00:01
     |████                            | 133kB 7.0MB/s eta 0:00:01
     |████▍                           | 143kB 7.0MB/s eta 0:00:01
     |████▊                           | 153kB 7.0MB/s eta 0:00:01
     |█████                           | 163kB 7.0MB/s eta 0:00:01
     |█████▍                          | 174kB 7.0MB/s eta 0:00:01
     |█████▊                          | 184kB 7.0MB/s eta 0:00:01
     |██████                          | 194kB 7.0MB/s eta 0:00:01
     |██████▎                         | 204kB 7.0MB/s eta 0:00:01
     |██████▋                         | 215kB 7.0MB/s eta 0:00:01
     |███████                         | 225kB 7.0MB/s eta 0:00:01
     |███████▎                        | 235kB 7.0MB/s eta 0:00:01
     |███████▋                        | 245kB 7.0MB/s eta 0:00:01
     |███████▉                        | 256kB 7.0MB/s eta 0:00:01
     |████████▏                       | 266kB 7.0MB/s eta 0:00:01
     |████████▌                       | 276kB 7.0MB/s eta 0:00:01
     |████████▉                       | 286kB 7.0MB/s eta 0:00:01
     |█████████▏                      | 296kB 7.0MB/s eta 0:00:01
     |█████████▌                      | 307kB 7.0MB/s eta 0:00:01
     |█████████▊                      | 317kB 7.0MB/s eta 0:00:01
     |██████████                      | 327kB 7.0MB/s eta 0:00:01
     |██████████▍                     | 337kB 7.0MB/s eta 0:00:01
     |██████████▊                     | 348kB 7.0MB/s eta 0:00:01
     |███████████                     | 358kB 7.0MB/s eta 0:00:01
     |███████████▍                    | 368kB 7.0MB/s eta 0:00:01
     |███████████▋                    | 378kB 7.0MB/s eta 0:00:01
     |████████████                    | 389kB 7.0MB/s eta 0:00:01
     |████████████▎                   | 399kB 7.0MB/s eta 0:00:01
     |████████████▋                   | 409kB 7.0MB/s eta 0:00:01
     |█████████████                   | 419kB 7.0MB/s eta 0:00:01
     |█████████████▎                  | 430kB 7.0MB/s eta 0:00:01
     |█████████████▌                  | 440kB 7.0MB/s eta 0:00:01
     |█████████████▉                  | 450kB 7.0MB/s eta 0:00:01
     |██████████████▏                 | 460kB 7.0MB/s eta 0:00:01
     |██████████████▌                 | 471kB 7.0MB/s eta 0:00:01
     |██████████████▉                 | 481kB 7.0MB/s eta 0:00:01
     |███████████████▏                | 491kB 7.0MB/s eta 0:00:01
     |███████████████▍                | 501kB 7.0MB/s eta 0:00:01
     |███████████████▊                | 512kB 7.0MB/s eta 0:00:01
     |████████████████                | 522kB 7.0MB/s eta 0:00:01
     |████████████████▍               | 532kB 7.0MB/s eta 0:00:01
     |████████████████▊               | 542kB 7.0MB/s eta 0:00:01
     |█████████████████               | 552kB 7.0MB/s eta 0:00:01
     |█████████████████▎              | 563kB 7.0MB/s eta 0:00:01
     |█████████████████▋              | 573kB 7.0MB/s eta 0:00:01
     |██████████████████              | 583kB 7.0MB/s eta 0:00:01
     |██████████████████▎             | 593kB 7.0MB/s eta 0:00:01
     |██████████████████▋             | 604kB 7.0MB/s eta 0:00:01
     |███████████████████             | 614kB 7.0MB/s eta 0:00:01
     |███████████████████▏            | 624kB 7.0MB/s eta 0:00:01
     |███████████████████▌            | 634kB 7.0MB/s eta 0:00:01
     |███████████████████▉            | 645kB 7.0MB/s eta 0:00:01
     |████████████████████▏           | 655kB 7.0MB/s eta 0:00:01
     |████████████████████▌           | 665kB 7.0MB/s eta 0:00:01
     |████████████████████▉           | 675kB 7.0MB/s eta 0:00:01
     |█████████████████████▏          | 686kB 7.0MB/s eta 0:00:01
     |█████████████████████▍          | 696kB 7.0MB/s eta 0:00:01
     |█████████████████████▊          | 706kB 7.0MB/s eta 0:00:01
     |██████████████████████          | 716kB 7.0MB/s eta 0:00:01
     |██████████████████████▍         | 727kB 7.0MB/s eta 0:00:01
     |██████████████████████▊         | 737kB 7.0MB/s eta 0:00:01
     |███████████████████████         | 747kB 7.0MB/s eta 0:00:01
     |███████████████████████▎        | 757kB 7.0MB/s eta 0:00:01
     |███████████████████████▋        | 768kB 7.0MB/s eta 0:00:01
     |████████████████████████        | 778kB 7.0MB/s eta 0:00:01
     |████████████████████████▎       | 788kB 7.0MB/s eta 0:00:01
     |████████████████████████▋       | 798kB 7.0MB/s eta 0:00:01
     |█████████████████████████       | 808kB 7.0MB/s eta 0:00:01
     |█████████████████████████▏      | 819kB 7.0MB/s eta 0:00:01
     |█████████████████████████▌      | 829kB 7.0MB/s eta 0:00:01
     |█████████████████████████▉      | 839kB 7.0MB/s eta 0:00:01
     |██████████████████████████▏     | 849kB 7.0MB/s eta 0:00:01
     |██████████████████████████▌     | 860kB 7.0MB/s eta 0:00:01
     |██████████████████████████▉     | 870kB 7.0MB/s eta 0:00:01
     |███████████████████████████     | 880kB 7.0MB/s eta 0:00:01
     |███████████████████████████▍    | 890kB 7.0MB/s eta 0:00:01
     |███████████████████████████▊    | 901kB 7.0MB/s eta 0:00:01
     |████████████████████████████    | 911kB 7.0MB/s eta 0:00:01
     |████████████████████████████▍   | 921kB 7.0MB/s eta 0:00:01
     |████████████████████████████▊   | 931kB 7.0MB/s eta 0:00:01
     |█████████████████████████████   | 942kB 7.0MB/s eta 0:00:01
     |█████████████████████████████▎  | 952kB 7.0MB/s eta 0:00:01
     |█████████████████████████████▋  | 962kB 7.0MB/s eta 0:00:01
     |██████████████████████████████  | 972kB 7.0MB/s eta 0:00:01
     |██████████████████████████████▎ | 983kB 7.0MB/s eta 0:00:01
     |██████████████████████████████▋ | 993kB 7.0MB/s eta 0:00:01
     |██████████████████████████████▉ | 1.0MB 7.0MB/s eta 0:00:01
     |███████████████████████████████▏| 1.0MB 7.0MB/s eta 0:00:01
     |███████████████████████████████▌| 1.0MB 7.0MB/s eta 0:00:01
     |███████████████████████████████▉| 1.0MB 7.0MB/s eta 0:00:01
     |████████████████████████████████| 1.0MB 7.0MB/s
Installing collected packages: sentencepiece
Successfully installed sentencepiece-0.1.85
Collecting wget
  Downloading https://files.pythonhosted.org/packages/47/6a/62e288da7bcda82b935ff0c6cfe542970f04e29c756b0e147251b2fb251f/wget-3.2.zip
Building wheels for collected packages: wget
  Building wheel for wget (setup.py) ... done
  Created wheel for wget: filename=wget-3.2-cp36-none-any.whl size=9682 sha256=e6a25da026f33947edcc73153385c40ae8e7758d46d8f60b791be74527c547d7
  Stored in directory: /root/.cache/pip/wheels/40/15/30/7d8f7cea2902b4db79e3fea550d7d7b85ecb27ef992b618f3f
Successfully built wget
Installing collected packages: wget
Successfully installed wget-3.2

#### 1. Google Drive Mount
Colab에서는 컴퓨터에 자원에 접근이 불가능 하므로 Google Drive에 파일을 올려 놓은 후 Google Drive를 mount 에서 로컬 디스크처럼 사용 합니다.
1. 아래 블럭을 실행하면 나타나는 링크를 클릭하세요.
2. Google 계정을 선택 하시고 허용을 누르면 나타나는 코드를 복사하여 아래 박스에 입력한 후 Enter 키를 입력하면 됩니다.

학습관련 [데이터 및 결과 파일](https://drive.google.com/open?id=15XGr-L-W6DSoR5TbniPMJASPsA0IDTiN)을 참고 하세요._____no_output_____

from google.colab import drive
drive.mount('/content/drive')
# data를 저장할 폴더 입니다. 환경에 맞게 수정 하세요.
data_dir = "/content/drive/My Drive/Data/transformer-evolution"Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

Enter your authorization code:
··········
Mounted at /content/drive

#### 2. Imports_____no_output_____

import os
import numpy as np
import math
import matplotlib.pyplot as plt
import json
import pandas as pd
from IPython.display import display
from tqdm import tqdm, tqdm_notebook, trange
import sentencepiece as spm
import wget

import torch
import torch.nn as nn
import torch.nn.functional as F_____no_output_____

#### 3. 폴더의 목록을 확인
Google Drive mount가 잘 되었는지 확인하기 위해 data_dir 목록을 확인 합니다._____no_output_____

for f in os.listdir(data_dir):
  print(f)kowiki.csv.gz
kowiki.model
kowiki.vocab
ratings_train.txt
ratings_test.txt
ratings_train.json
ratings_test.json
kowiki.txt
kowiki_gpt.json
save_gpt_pretrain.pth
kowiki_bert_0.json
save_bert_pretrain.pth
kowiki_t5.model
kowiki_t5.vocab
kowiki_t5_0.json
save_t5_pretrain.pth

#### 4. Vocab 및 입력

[Sentencepiece를 활용해 Vocab 만들기](https://paul-hyun.github.io/vocab-with-sentencepiece/)를 통해 만들어 놓은 vocab을 로딩 합니다._____no_output_____

# vocab loading
vocab_file = f"{data_dir}/kowiki_t5.model"
vocab = spm.SentencePieceProcessor()
vocab.load(vocab_file)_____no_output_____

#### 5. Config

모델에 설정 값을 전달하기 위한 config를 만듭니다._____no_output_____

""" configuration json을 읽어들이는 class """
class Config(dict):
    __getattr__ = dict.__getitem__
    __setattr__ = dict.__setitem__

    @classmethod
    def load(cls, file):
        with open(file, 'r') as f:
            config = json.loads(f.read())
            return Config(config)_____no_output_____config = Config({
    "n_vocab": len(vocab),
    "n_seq": 256,
    "n_layer": 6,
    "d_hidn": 256,
    "i_pad": 0,
    "d_ff": 1024,
    "n_head": 4,
    "d_head": 64,
    "dropout": 0.1,
    "layer_norm_epsilon": 1e-12
})
print(config){'n_vocab': 8033, 'n_seq': 256, 'n_layer': 6, 'd_hidn': 256, 'i_pad': 0, 'd_ff': 1024, 'n_head': 4, 'd_head': 64, 'dropout': 0.1, 'layer_norm_epsilon': 1e-12}

#### 6. T5

T5 Class 및 함수 입니다._____no_output_____

""" attention pad mask """
def get_attn_pad_mask(seq_q, seq_k, i_pad):
    batch_size, len_q = seq_q.size()
    batch_size, len_k = seq_k.size()
    pad_attn_mask = seq_k.data.eq(i_pad).unsqueeze(1).expand(batch_size, len_q, len_k)  # <pad>
    return pad_attn_mask

""" attention decoder mask """
def get_attn_decoder_mask(seq):
    subsequent_mask = torch.ones_like(seq).unsqueeze(-1).expand(seq.size(0), seq.size(1), seq.size(1))
    subsequent_mask = subsequent_mask.triu(diagonal=1) # upper triangular part of a matrix(2-D)
    return subsequent_mask

""" scale dot product attention """
class ScaledDotProductAttention(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.dropout = nn.Dropout(config.dropout)
        self.scale = 1 / (self.config.d_head ** 0.5)
        self.num_buckets = 32
        self.relative_attention_bias = torch.nn.Embedding(self.num_buckets, self.config.n_head)

    def forward(self, Q, K, V, attn_mask, bidirectional=True):
        qlen, klen = Q.size(-2), K.size(-2)
        # (bs, n_head, n_q_seq, n_k_seq)
        scores = torch.matmul(Q, K.transpose(-1, -2)).mul_(self.scale)
        # (1, n_head, n_q_seq, n_k_seq)
        position_bias = self.compute_bias(qlen, klen, bidirectional=bidirectional)
        scores += position_bias
        scores.masked_fill_(attn_mask, -1e9)
        # (bs, n_head, n_q_seq, n_k_seq)
        attn_prob = nn.Softmax(dim=-1)(scores)
        attn_prob = self.dropout(attn_prob)
        # (bs, n_head, n_q_seq, d_v)
        context = torch.matmul(attn_prob, V)
        # (bs, n_head, n_q_seq, d_v), (bs, n_head, n_q_seq, n_v_seq)
        return context, attn_prob

    def compute_bias(self, qlen, klen, bidirectional=True):
        context_position = torch.arange(qlen, dtype=torch.long)[:, None]
        memory_position = torch.arange(klen, dtype=torch.long)[None, :]
        # (qlen, klen)
        relative_position = memory_position - context_position
        # (qlen, klen)
        rp_bucket = self._relative_position_bucket(
            relative_position,  # shape (qlen, klen)
            num_buckets=self.num_buckets,
            bidirectional=bidirectional
        )
        # (qlen, klen)
        rp_bucket = rp_bucket.to(self.relative_attention_bias.weight.device)
        # (qlen, klen, n_head)
        values = self.relative_attention_bias(rp_bucket)
        # (1, n_head, qlen, klen)
        values = values.permute([2, 0, 1]).unsqueeze(0)
        return values

    def _relative_position_bucket(self, relative_position, bidirectional=True, num_buckets=32, max_distance=128):
        ret = 0
        n = -relative_position
        if bidirectional:
            num_buckets //= 2
            ret += (n < 0).to(torch.long) * num_buckets  # mtf.to_int32(mtf.less(n, 0)) * num_buckets
            n = torch.abs(n)
        else:
            n = torch.max(n, torch.zeros_like(n))

        # half of the buckets are for exact increments in positions
        max_exact = num_buckets // 2
        is_small = n < max_exact

        # The other half of the buckets are for logarithmically bigger bins in positions up to max_distance
        val_if_large = max_exact + (
                torch.log(n.float() / max_exact) / math.log(max_distance / max_exact) * (num_buckets - max_exact)
        ).to(torch.long)
        val_if_large = torch.min(val_if_large, torch.full_like(val_if_large, num_buckets - 1))

        ret += torch.where(is_small, n, val_if_large)
        return ret

""" multi head attention """
class MultiHeadAttention(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.W_Q = nn.Linear(self.config.d_hidn, self.config.n_head * self.config.d_head)
        self.W_K = nn.Linear(self.config.d_hidn, self.config.n_head * self.config.d_head)
        self.W_V = nn.Linear(self.config.d_hidn, self.config.n_head * self.config.d_head)
        self.scaled_dot_attn = ScaledDotProductAttention(self.config)
        self.linear = nn.Linear(self.config.n_head * self.config.d_head, self.config.d_hidn)
        self.dropout = nn.Dropout(config.dropout)

    def forward(self, Q, K, V, attn_mask, bidirectional=False):
        batch_size = Q.size(0)
        # (bs, n_head, n_q_seq, d_head)
        q_s = self.W_Q(Q).view(batch_size, -1, self.config.n_head, self.config.d_head).transpose(1,2)
        # (bs, n_head, n_k_seq, d_head)
        k_s = self.W_K(K).view(batch_size, -1, self.config.n_head, self.config.d_head).transpose(1,2)
        # (bs, n_head, n_v_seq, d_head)
        v_s = self.W_V(V).view(batch_size, -1, self.config.n_head, self.config.d_head).transpose(1,2)

        # (bs, n_head, n_q_seq, n_k_seq)
        attn_mask = attn_mask.unsqueeze(1).repeat(1, self.config.n_head, 1, 1)

        # (bs, n_head, n_q_seq, d_head), (bs, n_head, n_q_seq, n_k_seq)
        context, attn_prob = self.scaled_dot_attn(q_s, k_s, v_s, attn_mask, bidirectional=bidirectional)
        # (bs, n_head, n_q_seq, h_head * d_head)
        context = context.transpose(1, 2).contiguous().view(batch_size, -1, self.config.n_head * self.config.d_head)
        # (bs, n_head, n_q_seq, e_embd)
        output = self.linear(context)
        output = self.dropout(output)
        # (bs, n_q_seq, d_hidn), (bs, n_head, n_q_seq, n_k_seq)
        return output, attn_prob

""" feed forward """
class PoswiseFeedForwardNet(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.conv1 = nn.Conv1d(in_channels=self.config.d_hidn, out_channels=self.config.d_ff, kernel_size=1)
        self.conv2 = nn.Conv1d(in_channels=self.config.d_ff, out_channels=self.config.d_hidn, kernel_size=1)
        self.active = F.gelu
        self.dropout = nn.Dropout(config.dropout)

    def forward(self, inputs):
        # (bs, d_ff, n_seq)
        output = self.active(self.conv1(inputs.transpose(1, 2)))
        # (bs, n_seq, d_hidn)
        output = self.conv2(output).transpose(1, 2)
        output = self.dropout(output)
        # (bs, n_seq, d_hidn)
        return output_____no_output_____""" encoder layer """
class EncoderLayer(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.self_attn = MultiHeadAttention(self.config)
        self.layer_norm1 = nn.LayerNorm(self.config.d_hidn, eps=self.config.layer_norm_epsilon)
        self.pos_ffn = PoswiseFeedForwardNet(self.config)
        self.layer_norm2 = nn.LayerNorm(self.config.d_hidn, eps=self.config.layer_norm_epsilon)

    def forward(self, inputs, attn_mask):
        # (bs, n_enc_seq, d_hidn), (bs, n_head, n_enc_seq, n_enc_seq)
        att_outputs, attn_prob = self.self_attn(inputs, inputs, inputs, attn_mask)
        att_outputs = self.layer_norm1(inputs + att_outputs)
        # (bs, n_enc_seq, d_hidn)
        ffn_outputs = self.pos_ffn(att_outputs)
        ffn_outputs = self.layer_norm2(ffn_outputs + att_outputs)
        # (bs, n_enc_seq, d_hidn), (bs, n_head, n_enc_seq, n_enc_seq)
        return ffn_outputs, attn_prob

""" encoder """
class Encoder(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.layers = nn.ModuleList([EncoderLayer(self.config) for _ in range(self.config.n_layer)])

    def forward(self, enc_embd, enc_self_mask):
        # (bs, n_enc_seq, d_hidn)
        enc_outputs = enc_embd

        attn_probs = []
        for layer in self.layers:
            # (bs, n_enc_seq, d_hidn), (bs, n_head, n_enc_seq, n_enc_seq)
            enc_outputs, attn_prob = layer(enc_outputs, enc_self_mask)
            attn_probs.append(attn_prob)
        # (bs, n_enc_seq, d_hidn), [(bs, n_head, n_enc_seq, n_enc_seq)]
        return enc_outputs, attn_probs

""" decoder layer """
class DecoderLayer(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.self_attn = MultiHeadAttention(self.config)
        self.layer_norm1 = nn.LayerNorm(self.config.d_hidn, eps=self.config.layer_norm_epsilon)
        self.dec_enc_attn = MultiHeadAttention(self.config)
        self.layer_norm2 = nn.LayerNorm(self.config.d_hidn, eps=self.config.layer_norm_epsilon)
        self.pos_ffn = PoswiseFeedForwardNet(self.config)
        self.layer_norm3 = nn.LayerNorm(self.config.d_hidn, eps=self.config.layer_norm_epsilon)

    def forward(self, dec_inputs, enc_outputs, self_mask, ende_mask):
        # (bs, n_dec_seq, d_hidn), (bs, n_head, n_dec_seq, n_dec_seq)
        self_att_outputs, self_attn_prob = self.self_attn(dec_inputs, dec_inputs, dec_inputs, self_mask, bidirectional=False)
        self_att_outputs = self.layer_norm1(dec_inputs + self_att_outputs)
        # (bs, n_dec_seq, d_hidn), (bs, n_head, n_dec_seq, n_enc_seq)
        dec_enc_att_outputs, dec_enc_attn_prob = self.dec_enc_attn(self_att_outputs, enc_outputs, enc_outputs, ende_mask)
        dec_enc_att_outputs = self.layer_norm2(self_att_outputs + dec_enc_att_outputs)
        # (bs, n_dec_seq, d_hidn)
        ffn_outputs = self.pos_ffn(dec_enc_att_outputs)
        ffn_outputs = self.layer_norm3(dec_enc_att_outputs + ffn_outputs)
        # (bs, n_dec_seq, d_hidn), (bs, n_head, n_dec_seq, n_dec_seq), (bs, n_head, n_dec_seq, n_enc_seq)
        return ffn_outputs, self_attn_prob, dec_enc_attn_prob

""" decoder """
class Decoder(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.layers = nn.ModuleList([DecoderLayer(self.config) for _ in range(self.config.n_layer)])

    def forward(self, dec_embd, enc_outputs, self_mask, ende_mask):
        # (bs, n_dec_seq, d_hidn)
        dec_outputs = dec_embd

        self_attn_probs, dec_enc_attn_probs = [], []
        for layer in self.layers:
            # (bs, n_dec_seq, d_hidn), (bs, n_dec_seq, n_dec_seq), (bs, n_dec_seq, n_enc_seq)
            dec_outputs, self_attn_prob, dec_enc_attn_prob = layer(dec_outputs, enc_outputs, self_mask, ende_mask)
            self_attn_probs.append(self_attn_prob)
            dec_enc_attn_probs.append(dec_enc_attn_prob)
        # (bs, n_dec_seq, d_hidn), [(bs, n_dec_seq, n_dec_seq)], [(bs, n_dec_seq, n_enc_seq)]S
        return dec_outputs, self_attn_probs, dec_enc_attn_probs

""" t5 """
class T5(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.embedding = nn.Embedding(self.config.n_vocab, self.config.d_hidn)
        self.encoder = Encoder(self.config)
        self.decoder = Decoder(self.config)

        self.projection_lm = nn.Linear(self.config.d_hidn, self.config.n_vocab, bias=False)
        self.projection_lm.weight = self.embedding.weight

    def forward(self, enc_inputs, dec_inputs):
        enc_embd = self.embedding(enc_inputs)
        dec_embd = self.embedding(dec_inputs)

        enc_self_mask = get_attn_pad_mask(enc_inputs, enc_inputs, self.config.i_pad)
        dec_self_mask = self.get_attn_dec_mask(dec_inputs)
        dec_ende_mask = get_attn_pad_mask(dec_inputs, enc_inputs, self.config.i_pad)

        # (bs, n_enc_seq, d_hidn), [(bs, n_head, n_enc_seq, n_enc_seq)]
        enc_outputs, enc_self_attn_probs = self.encoder(enc_embd, enc_self_mask)
        # (bs, n_dec_seq, d_hidn), [(bs, n_head, n_dec_seq, n_dec_seq)], [(bs, n_head, n_dec_seq, n_enc_seq)]
        dec_outputs, dec_self_attn_probs, dec_enc_attn_probs = self.decoder(dec_embd, enc_outputs, dec_self_mask, dec_ende_mask)
        # (bs, n_dec_seq, n_vocab)
        dec_outputs = self.projection_lm(dec_outputs)
        # (bs, n_dec_seq, n_vocab), [(bs, n_head, n_enc_seq, n_enc_seq)], [(bs, n_head, n_dec_seq, n_dec_seq)], [(bs, n_head, n_dec_seq, n_enc_seq)]
        return dec_outputs, enc_self_attn_probs, dec_self_attn_probs, dec_enc_attn_probs

    def get_attn_dec_mask(self, dec_inputs):
         # (bs, n_dec_seq, n_dec_seq)
        dec_pad_mask = get_attn_pad_mask(dec_inputs, dec_inputs, self.config.i_pad)
        # (bs, n_dec_seq, n_dec_seq)
        dec_ahead_mask = get_attn_decoder_mask(dec_inputs)
        # (bs, n_dec_seq, n_dec_seq)
        dec_self_mask = torch.gt((dec_pad_mask + dec_ahead_mask), 0)
        # (bs, n_dec_seq, n_dec_seq)
        return dec_self_mask

    def save(self, epoch, loss, path):
        torch.save({
            "epoch": epoch,
            "loss": loss,
            "state_dict": self.state_dict()
        }, path)

    def load(self, path):
        save = torch.load(path)
        self.load_state_dict(save["state_dict"])
        return save["epoch"], save["loss"]_____no_output_____

#### 7. Naver 영화 분류 모델_____no_output_____

""" naver movie classfication """
class MovieClassification(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.t5 = T5(self.config)

    def forward(self, enc_inputs, dec_inputs):
        # (bs, n_dec_seq, n_vocab), [(bs, n_head, n_enc_seq, n_enc_seq)], [(bs, n_head, n_dec_seq, n_dec_seq)], [(bs, n_head, n_dec_seq, n_enc_seq)]
        logits, enc_self_attn_probs, dec_self_attn_probs, dec_enc_attn_probs = self.t5(enc_inputs, dec_inputs)
        return logits, enc_self_attn_probs, dec_self_attn_probs, dec_enc_attn_probs_____no_output_____

#### 8. 네이버 영화 분류 데이터

T5를 위해 vocab을 새로 만들어서 학습 데이터도 새로 만들 었습니다.

_____no_output_____

""" train data 준비 """
def prepare_train(vocab, infile, outfile):
    df = pd.read_csv(infile, sep="\t", engine="python")
    with open(outfile, "w") as f:
        for index, row in df.iterrows():
            document = row["document"]
            if type(document) != str:
                continue
            instance = { "id": row["id"], "doc": vocab.encode_as_pieces(document), "label": row["label"] }
            f.write(json.dumps(instance, ensure_ascii=False))
            f.write("\n")_____no_output_____prepare_train(vocab, f"{data_dir}/ratings_train.txt", f"{data_dir}/ratings_train_t5.json")
prepare_train(vocab, f"{data_dir}/ratings_test.txt", f"{data_dir}/ratings_test_t5.json")_____no_output_____""" 정답 text """
lable_map = {0: "부", 1: "정"}

""" 영화 분류 데이터셋 """
class MovieDataSet(torch.utils.data.Dataset):
    def __init__(self, vocab, infile, is_valid=False):
        self.vocab = vocab
        self.labels = []
        self.enc_inputs = []
        self.dec_inputs = []

        line_cnt = 0
        with open(infile, "r") as f:
            for line in f:
                line_cnt += 1

        with open(infile, "r") as f:
            for i, line in enumerate(tqdm(f, total=line_cnt, desc="Loading Dataset", unit=" lines")):
                data = json.loads(line)

                enc_input = vocab.encode_as_ids("감정분류:") + [vocab.piece_to_id(p) for p in data["doc"]]
                if is_valid:
                    label = vocab.encode_as_ids(lable_map[data["label"]])
                    dec_input = [vocab.piece_to_id("[BOS]")]
                else:
                    label = vocab.encode_as_ids(lable_map[data["label"]]) + [vocab.piece_to_id("[EOS]")]
                    dec_input = [vocab.piece_to_id("[BOS]")] + vocab.encode_as_ids(lable_map[data["label"]])

                self.labels.append(label)
                self.enc_inputs.append(enc_input)
                self.dec_inputs.append(dec_input)

    def __len__(self):
        assert len(self.labels) == len(self.enc_inputs)
        assert len(self.labels) == len(self.dec_inputs)
        return len(self.labels)

    def __getitem__(self, item):
        return (torch.tensor(self.labels[item]),
                torch.tensor(self.enc_inputs[item]),
                torch.tensor(self.dec_inputs[item]))_____no_output_____""" movie data collate_fn """
def movie_collate_fn(inputs):
    labels, enc_inputs, dec_inputs = list(zip(*inputs))

    enc_inputs = torch.nn.utils.rnn.pad_sequence(enc_inputs, batch_first=True, padding_value=0)
    dec_inputs = torch.nn.utils.rnn.pad_sequence(dec_inputs, batch_first=True, padding_value=0)

    batch = [
        torch.stack(labels, dim=0),
        enc_inputs,
        dec_inputs,
    ]
    return batch_____no_output_____""" 데이터 로더 """
batch_size = 128
train_dataset = MovieDataSet(vocab, f"{data_dir}/ratings_train_t5.json")
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True, collate_fn=movie_collate_fn)
test_dataset = MovieDataSet(vocab, f"{data_dir}/ratings_test_t5.json", is_valid=True)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=False, collate_fn=movie_collate_fn)Loading Dataset: 100%|██████████| 149995/149995 [00:06<00:00, 21670.37 lines/s]
Loading Dataset: 100%|██████████| 49997/49997 [00:01<00:00, 25291.42 lines/s]

#### 9. 네이버 영화 분류 데이터 학습_____no_output_____

""" 모델 epoch 평가 """
def eval_epoch(config, model, data_loader):
    matchs = []
    model.eval()

    n_word_total = 0
    n_correct_total = 0
    with tqdm(total=len(data_loader), desc=f"Valid") as pbar:
        for i, value in enumerate(data_loader):
            labels, enc_inputs, dec_inputs = map(lambda v: v.to(config.device), value)

            outputs = model(enc_inputs, dec_inputs)
            logits = outputs[0]
            _, indices = logits.max(2)

            match = torch.eq(indices, labels).detach()
            matchs.extend(match.cpu())
            accuracy = np.sum(matchs) / len(matchs) if 0 < len(matchs) else 0

            pbar.update(1)
            pbar.set_postfix_str(f"Acc: {accuracy:.3f}")
    return np.sum(matchs) / len(matchs) if 0 < len(matchs) else 0_____no_output_____""" 모델 epoch 학습 """
def train_epoch(config, epoch, model, criterion, optimizer, train_loader):
    losses = []
    model.train()

    with tqdm(total=len(train_loader), desc=f"Train({epoch})") as pbar:
        for i, value in enumerate(train_loader):
            labels, enc_inputs, dec_inputs = map(lambda v: v.to(config.device), value)

            optimizer.zero_grad()
            outputs = model(enc_inputs, dec_inputs)
            logits = outputs[0]

            loss = criterion(logits.view(-1, logits.size(2)), labels.view(-1))

            loss_val = loss.item()
            losses.append(loss_val)

            loss.backward()
            optimizer.step()

            pbar.update(1)
            pbar.set_postfix_str(f"Loss: {loss_val:.3f} ({np.mean(losses):.3f})")
    return np.mean(losses)_____no_output_____config.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(config)

learning_rate = 5e-5
n_epoch = 5{'n_vocab': 8033, 'n_seq': 256, 'n_layer': 6, 'd_hidn': 256, 'i_pad': 0, 'd_ff': 1024, 'n_head': 4, 'd_head': 64, 'dropout': 0.1, 'layer_norm_epsilon': 1e-12, 'device': device(type='cuda')}
def train(model):
    model.to(config.device)

    criterion_cls = torch.nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

    best_epoch, best_loss, best_score = 0, 0, 0
    losses, scores = [], []
    for epoch in range(n_epoch):
        loss = train_epoch(config, epoch, model, criterion_cls, optimizer, train_loader)
        score = eval_epoch(config, model, test_loader)

        losses.append(loss)
        scores.append(score)

        if best_score < score:
            best_epoch, best_loss, best_score = epoch, loss, score
    print(f">>>> epoch={best_epoch}, loss={best_loss:.5f}, socre={best_score:.5f}")
    return losses, scores_____no_output_____

###### Pretrain 없이 학습_____no_output_____

model = MovieClassification(config)

losses_00, scores_00 = train(model)Train(0): 100%|██████████| 1172/1172 [05:07<00:00,  3.72it/s, Loss: 0.244 (0.827)]
Valid: 100%|██████████| 391/391 [01:00<00:00,  4.63it/s, Acc: 0.771]
Train(1): 100%|██████████| 1172/1172 [05:06<00:00,  3.89it/s, Loss: 0.232 (0.253)]
Valid: 100%|██████████| 391/391 [01:02<00:00,  4.59it/s, Acc: 0.794]
Train(2): 100%|██████████| 1172/1172 [05:10<00:00,  4.05it/s, Loss: 0.235 (0.229)]
Valid: 100%|██████████| 391/391 [01:02<00:00,  4.76it/s, Acc: 0.812]
Train(3): 100%|██████████| 1172/1172 [05:10<00:00,  3.88it/s, Loss: 0.193 (0.214)]
Valid: 100%|██████████| 391/391 [01:02<00:00,  4.77it/s, Acc: 0.816]
Train(4): 100%|██████████| 1172/1172 [05:09<00:00,  3.75it/s, Loss: 0.192 (0.201)]
Valid: 100%|██████████| 391/391 [01:01<00:00,  4.84it/s, Acc: 0.814]

###### Pretrain을 한 후 학습_____no_output_____

model = MovieClassification(config)

save_pretrain = f"{data_dir}/save_t5_pretrain.pth"
model.t5.load(save_pretrain)

losses_20, scores_20 = train(model)Train(0): 100%|██████████| 1172/1172 [05:11<00:00,  3.69it/s, Loss: 0.227 (0.288)]
Valid: 100%|██████████| 391/391 [01:01<00:00,  4.86it/s, Acc: 0.769]
Train(1): 100%|██████████| 1172/1172 [05:07<00:00,  3.93it/s, Loss: 0.261 (0.220)]
Valid: 100%|██████████| 391/391 [01:00<00:00,  4.68it/s, Acc: 0.813]
Train(2): 100%|██████████| 1172/1172 [05:06<00:00,  3.94it/s, Loss: 0.173 (0.204)]
Valid: 100%|██████████| 391/391 [01:00<00:00,  4.67it/s, Acc: 0.822]
Train(3): 100%|██████████| 1172/1172 [05:09<00:00,  3.78it/s, Loss: 0.205 (0.191)]
Valid: 100%|██████████| 391/391 [01:03<00:00,  4.86it/s, Acc: 0.828]
Train(4): 100%|██████████| 1172/1172 [05:07<00:00,  3.98it/s, Loss: 0.138 (0.181)]
Valid: 100%|██████████| 391/391 [01:00<00:00,  4.56it/s, Acc: 0.827]

#### 10. Result_____no_output_____

# table
data = {
    "loss_00": losses_00,
    "socre_00": scores_00,
    "loss_20": losses_20,
    "socre_20": scores_20,
}
df = pd.DataFrame(data)
display(df)

# graph
plt.figure(figsize=[12, 4])
plt.plot(scores_00, label="score_00")
plt.plot(scores_20, label="score_20")
plt.legend()
plt.xlabel('Epoch')
plt.ylabel('Value')
plt.show()_____no_output_____


# Notebook from mpbrigham/intro-sleep-scoring
Path: Introduction_to_sleep_scoring_normalization_ans.ipynb

## Morpheo baseline project
# Introduction to sleep scoring - Normalization methods (answers)
[Colaboratory version](https://colab.research.google.com/drive/1WvblQqLAEgWMzlH5rsPUipbJ5APXHgXV)_____no_output_____Please execute the cell bellow to initialize the notebook environment_____no_output_____

import numpy as np
import scipy as sp
import matplotlib.pyplot as plt

fig_w, fig_h = (4.5, 3.5)
plt.rcParams.update({'figure.figsize': (fig_w, fig_h)})_____no_output_____

Please execute the cell bellow to download the dataset._____no_output_____

!rm -rf datasets intro-sleep-scoring
!if [ ! -d datasets ]; then git clone https://github.com/mpbrigham/intro-sleep-scoring; \
                            cp -rf ./intro-sleep-scoring/* ./; fi

import sleep_utils_____no_output_____

## Import dataset
Import the dataset into dictionary `dataset` with keys `channels` and `hypnogram` for PSG and hypnogram data, respectively. The PSG data is stored with shape `(epochs, channels, samples)`.

**Suggestions**
* Open HDF5 database `mesa_sleep_0001_s.h5`
* Print table names and shapes

**Refs**
* h5py: HDF5 for Python (http://docs.h5py.org/en/latest)_____no_output_____

import sleep_utils
import importlib
importlib.reload(sleep_utils)

path = './datasets/mesa_small'
channels_ref = ['EEG1', 'EEG2', 'EEG3']

dataset = sleep_utils.load_dataset(path, channels_ref=channels_ref)processing 6 records in folder ./datasets/mesa_small

mesa_sleep_0014_s
mesa_sleep_0002_s
mesa_sleep_0006_s
mesa_sleep_0021_s
mesa_sleep_0001_s
mesa_sleep_0016_s

Imported 6 records

**EXPECTED OUTPUT**
```
processing 6 records in folder ./datasets/mesa_small

mesa_sleep_0021_s
mesa_sleep_0016_s
mesa_sleep_0014_s
mesa_sleep_0006_s
mesa_sleep_0002_s
mesa_sleep_0001_s

Imported 6 records
```_____no_output_____## Data pre-processing_____no_output_____### Plot histogram of EEG channels
Plot histogram of individual EEG channels in dataset.

**Suggestions**
* Use function `np.linspace()` from Numpy to define array `x_bins` with 100 values from -0.2 to 0.2.
* Use function `plt.hist()` from Matplotlib to plot histogram of array `x`, using keyword `bins` to set bins location to `x_bins`_____no_output_____

import sleep_utils
import importlib
importlib.reload(sleep_utils)

sleep_utils.plot_stats(dataset, x_min=-0.2, x_max=0.2)_____no_output_____

**EXPECTED OUTPUT**

_____no_output_____### Center and scale EEG data
Write function `pre_process()` to remove channel mean and scale channels to one standard deviation or 50th absolute percentile.

Apply function to channel data and plot histograms._____no_output_____

def pre_process(x, scale='std'):
  """Remove channel mean from array x and scale to one standard deviation or
  absolute 50th percentile.
  x: array with shape (batch, channel, data)
  scale: 'std' | 'pct'

  returns x_out:  zero channel mean and scaled array x
  """

  x_out = x

  for idx in range(x.shape[1]):

      x_out[:,idx] = x[:,idx] - np.mean(x[:,idx], keepdims=True)

      if scale=='std':
          alpha = np.std(x_out[:,idx], keepdims=True)

      elif scale=='pct':
          alpha = np.percentile(np.abs(x_out[:,idx]), 50, keepdims=True)*10

      x_out[:,idx] = x_out[:,idx]/alpha

  return x_out_____no_output_____dataset_std = {item: dataset[item] for item in dataset}

for key_idx, key in enumerate(dataset):

    dataset_std[key]['samples'] = pre_process(dataset_std[key]['samples'],
                                          scale='std')

sleep_utils.plot_stats(dataset_std, channels)_____no_output_____dataset_pct = {item: dataset[item] for item in dataset}

for key_idx, key in enumerate(dataset):

    dataset_pct[key]['samples'] = pre_process(dataset_pct[key]['samples'],
                                          scale='pct')

sleep_utils.plot_stats(dataset_pct, channels)_____no_output_____

## Prepare data for training_____no_output_____### Write data import function
Write function `load_data()` to import EEG channels and hypnogram from HDF5 database._____no_output_____

def load_data(path):
  """Import EEG channels and hypnogram from HDF5 database.
  path: filesysWrite function `load_data()` to import EEG channels and hypnogram from HDF5 database.Write function `load_data()` to import EEG channels and hypnogram from HDF5 database.tem path of HDF5 database

  returns x: array containing EEG channels
          y: array containing hypnogram
  """

  # insert your code here
  x = None
  y = None

  return (x, y)_____no_output_____def load_data(path):
  """Import EEG channels and hypnogram from HDF5 database.
  path: filesystem path of HDF5 database

  returns x: array containing EEG channels
          y: array containing hypnogram
  """

  with h5py.File(path, 'r') as f:
    x = np.stack((f['EEG1'][::], f['EEG2'][::], f['EEG3'][::]), axis=1)
    y = np.array([f['stages'][::]]).T

  return (x, y)_____no_output_____path = 'colaboratory-data/mesa-small/mesa-sleep-0002_s.h5'

x, y = load_data(path)

x = pre_process(x)

if x is not None:
  print(x[:2])
  print(y[:2])_____no_output_____

**EXPECTED OUTPUT**
```
[[[ 0.04396349  0.07795955  0.07226041 ...  0.04176991  0.03708698  0.04716769]
  [-0.07195704 -0.11975361 -0.10864799 ... -0.09606183 -0.09113557  -0.092215  ]
  [ 0.02422285  0.05473021  0.05198389 ...  0.01406217  0.0240755   0.0294412 ]]

 [[ 0.03657643  0.04430808  0.03648227 ...  0.03918483  0.04145856  0.04325664]
  [-0.05006481 -0.0881365  -0.07769687 ... -0.07160157 -0.05522351  -0.06842058]
  [ 0.03790487  0.05567357  0.05256429 ...  0.0541604   0.04131374  0.06555403]]]
[[0]
 [0]]
```_____no_output_____### Import test set
Import test set from HDF5 database `mesa-sleep-0021_s` into arrays `x_test` and `y_test`.

Print shapes of the new arrays, and basic EEG stats from array `x_test`._____no_output_____

path = 'colaboratory-data/mesa-small/mesa-sleep-0021_s.h5'

x_test, y_test = load_data(path)

x_test = pre_process(x_test)

# insert your code here_____no_output_____path = 'colaboratory-data/mesa-small/mesa-sleep-0021_s.h5'

x_test, y_test = load_data(path)

x_test = pre_process(x_test)

print('var\t\t shape\n')
for item, item_name in ([[x_test, 'x_test'], [y_test, 'y_test']]):
  print(item_name, '\t', item.shape)

print()
print_stats(x_test)_____no_output_____

**EXPECTED OUTPUT**
```
var		 shape

x_test 	 (1079, 3, 1920)
y_test 	 (1079, 1)

EEG	 min		 max		 mean		 std

0 	 -1.9979 	 0.6084 	 0.0000 	 0.0222
1 	 -0.4698 	 2.0072 	 0.0000 	 0.0380
2 	 -5.0036 	 1.0731 	 0.0000 	 0.0511
 ```_____no_output_____### Split dataset into train and validation sets
Split arrays `x` and `y` into train and validation sets `x_train`, `x_val`, `y_train` and `y_val`. The validation set contains 300 epochs from each HDF5 database.

Print the shapes of the new arrays.

**Note:** the function `np.random.seed(0)` from Numpy is used to replicate the expected output.

**Suggestions**
* Create boolean array `bool_idx` with 1439 elements initialized with `False` values
* Use function `np.random.choice()` to randomly select (without replacement) 300 elements of array `bool_idx` and set them to `True`
* Split `x` into `x_train` and `x_val` using indexes from array `bool_idx`
* Use function `np.random.seed(0)` from Numpy to replicate the expected output_____no_output_____

np.random.seed(0)

x_val = None
y_val = None
x_train = None
y_train = None

# insert your code here_____no_output_____np.random.seed(0)

bool_idx = np.array([False]*y.shape[0])
selected = np.random.choice(len(bool_idx), size=300, replace=False)
bool_idx[selected] = True

x_val = x[bool_idx]
y_val = y[bool_idx]
x_train = x[~bool_idx]
y_train = y[~bool_idx]

print('var\t\t shape\n')
for item, item_name in [[x, 'x      '], [y, 'y      '],
                        [x_train, 'x_train'], [y_train, 'y_train'],
                        [x_val, 'x_val  '], [y_val, 'y_val  ']]:
  print(item_name, '\t', item.shape)_____no_output_____

**EXPECTED OUTPUT**
```
var		 shape

x       	 (1319, 3, 1920)
y       	 (1319, 1)
x_train 	 (1019, 3, 1920)
y_train 	 (1019, 1)
x_val   	 (300, 3, 1920)
y_val   	 (300, 1)
```_____no_output_____### Check train and validation sets are mutually exclusive
Execute the following code to validate that `x_val` contains no elements from `x_train`._____no_output_____

for item_train in x_train:
  for item_val in x_val:
    assert not np.allclose(item_train, item_val), 'Failed.'

print('Passed.')_____no_output_____

**EXPECTED OUTPUT**
```
Passed.
```_____no_output_____### Generate train and validation sets
Create train and validation sets in arrays `x_train`, `y_train`, `x_val` and `y_val` from HDF5 databases `mesa-sleep-0001_s`, `mesa-sleep-0002_s`, `mesa-sleep-0006_s`, `mesa-sleep-0014_s` and `mesa-sleep-0016_s`.

Print the shapes of train and validation datasets. Print basic statistical metrics of array `x_train`._____no_output_____

np.random.seed(0)

paths = ['colaboratory-data/mesa-small/mesa-sleep-0001_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0002_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0006_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0014_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0016_s.h5']

# insert your code here_____no_output_____np.random.seed(0)

paths = ['colaboratory-data/mesa-small/mesa-sleep-0001_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0002_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0006_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0014_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0016_s.h5']

x_train, x_val = ([],[])
y_train, y_val = ([],[])

for path in paths:

  x, y = load_data(path)

  x = pre_process(x)

  idx = np.array([False]*y.shape[0])
  selected = np.random.choice(y.shape[0], size=300, replace=False)
  idx[selected] = True

  x_val += [x[idx]]
  y_val += [y[idx]]
  x_train += [x[~idx]]
  y_train += [y[~idx]]

x_val = np.concatenate(x_val, axis=0)
x_train = np.concatenate(x_train, axis=0)
y_val = np.concatenate(y_val, axis=0)
y_train = np.concatenate(y_train, axis=0)

print('var \t\tshape\n')
for item, item_name in ([x_train, 'x_train'], [y_train, 'y_train'],
                        [x_val, 'x_val  '], [y_val,'y_val  ']):
  print(item_name, '\t', item.shape)

print()
print_stats(x_train)_____no_output_____

**EXPECTED OUTPUT**
```
var 		shape

x_train 	 (5215, 1920, 3)
y_train 	 (5215, 1)
x_val   	 (1500, 1920, 3)
y_val   	 (1500, 1)

EEG	 min		 max		 mean		 std

0 	 -2.7610 	 2.7743 	 0.0002 	 0.1589
1 	 -2.7251 	 2.7513 	 0.0001 	 0.4217
2 	 -6.7197 	 6.6099 	 0.0006 	 0.4857
```_____no_output_____## Model setup_____no_output_____### Write input/output conversion functions
Write function `to_input()` to convert EEG data into 2-dimensional array by concatenating EEG channels on dimension 1.

i.e array `x_train` with shape `(5215, 3, 1920)` is mapped to array with shape `(5215, 5760)`.

Write function `to_output()` to sleep scores into binarized `one-hot-encoding`, using function `keras.utils.to_categorical()` from Keras library. This transformation assigns score `0` to `[1 0 0 0 0]`, score `1` to  `[0 1 0 0 0]`, etc.

i.e array `y_train` with shape `(5215, 1)` is mapped to array with shape `(5215, 5)`._____no_output_____

from keras import backend as K
from keras.layers import Input, Dense, Layer
from keras.models import Sequential

def to_input(x):
  """Convert data array to shape (batch, data).
  x: array with shape (channel, batch, data)

  returns x_out: array x with shape (batch, data)
  """

  # insert your code here
  x_out = None

  return x_out

def to_output(y):
  """Convert label array to one-hot-encoding with shape (batch, data).
  y: label array with shape (1, batch)

  returns: y_out (array with shape (batch, label))
  """

  # insert your code here
  y_out = None

  return y_out_____no_output_____from keras import backend as K
from keras.layers import Input, Dense, Layer
from keras.models import Sequential

def to_input(x):
  """Convert data array to shape (batch, data).
  x: array with shape (batch, data, channel)

  returns x_out: array x with shape (batch, data)
  """
  x_out = x.reshape(x.shape[0], -1)

  return x_out

def to_output(y):
  """Convert label array to one-hot-encoding with shape (batch, data).
  y: label array with shape (batch, data)

  returns: x_out (array with shape (batch, label))
  """

  y_out = keras.utils.to_categorical(y, 5)

  return y_out_____no_output_____if to_input(x_train) is not None:

  print('var\t\t\t shape\n')
  for item, item_name in ([[to_input(x_train), 'to_input(x_train)'],
                           [to_output(y_train), 'to_output(y_train)']]):
    print(item_name, '\t', item.shape)

  print('\n')
  print(to_input(x_train)[:2])
  print(to_output(y_train)[:2])_____no_output_____

**EXPECTED OUTPUT**
```
var			 shape

to_input(x_train) 	 (5215, 5760)
to_output(y_train) 	 (5215, 5)

[[ 0.00099635  0.0048069   0.00776085 ...  0.00423072 -0.00867984
   0.02381653]
 [ 0.00212982  0.00123634 -0.00316261 ...  0.01105075  0.03953509
   0.04564268]]
[[1. 0. 0. 0. 0.]
 [1. 0. 0. 0. 0.]]
 ```_____no_output_____### Convert datasets to network input/output format
Convert datasets into format compatible with network using functions `to_input()` and `to_output()`.

Print shapes of the new arrays._____no_output_____

input_train = to_input(x_train)
input_val = to_input(x_val)
input_test = to_input(x_test)

output_train = to_output(y_train)
output_val = to_output(y_val)
output_test = to_output(y_test)

if input_train is not None:
  input_shape = input_train.shape[1]
  output_shape = output_train.shape[1]

print('var\t\t shape\n')

# insert your code here_____no_output_____input_train = to_input(x_train)
input_val = to_input(x_val)
input_test = to_input(x_test)

output_train = to_output(y_train)
output_val = to_output(y_val)
output_test = to_output(y_test)

if input_train is not None:
    input_shape = input_train.shape[1]
    output_shape = output_train.shape[1]

print('var\t\t shape\n')
for item, item_name in ([[input_train, 'input_train'],
                         [input_val, 'input_val'],
                         [input_test, 'input_test'],
                         [output_train, 'output_train'],
                         [output_val, 'output_val'],
                         [output_test, 'output_test']]):
    print(item_name, '\t', item.shape)_____no_output_____

**EXPECTED OUTPUT**
```
var		 shape

input_train 	 (5215, 5760)
input_val   	 (1500, 5760)
input_test  	 (1079, 5760)
output_train	 (5215, 5)
output_val 	  (1500, 5)
output_test 	 (1079, 5)
```_____no_output_____### CNN 1D model
Implement simple CNN network with 1D convolutions using library Keras (https://keras.io/).

Write function `model_cnn1d()` that returns the compiled model.

Use adadelta optimizer, binary crossentropy loss and categorical accuracy metric._____no_output_____

def model_cnn1d():
  """Define CNN network with 1D convolutions

  returns m: Keras model with 1D convolutions
  """

  # insert your code here
  m= None

  return m_____no_output_____def model_cnn1d():
  """Define CNN network with 1D convolutions

  returns m: Keras model with 1D convolutions
  """

  m = Sequential()
  m.add(Layer(input_shape=(input_shape,), name='input'))
  m.add(Dense(output_shape, activation='softmax', name='output'))

  m.compile(optimizer='adadelta', loss='binary_crossentropy', metrics=['categorical_accuracy'])

  return m_____no_output_____model = model_softmax()

if model is not None:
  model.summary()_____no_output_____

**EXPECTED OUTPUT**
```
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input (Layer)                (None, 5760)              0
_________________________________________________________________
output (Dense)               (None, 5)                 28805
=================================================================
Total params: 28,805
Trainable params: 28,805
Non-trainable params: 0
_________________________________________________________________
```_____no_output_____### Train softmax model
Train softmax network during 5 training epochs, batch size of 32 with sample shuffling.

**Suggestions**
* Use method `fit()` with keywords `epochs`, `batch_size` and `shuffle` to train model
* Use method `evaluate()` to evaluate performance metrics in validation and test sets_____no_output_____

np.random.seed(0)

n_epochs = 5

model = model_softmax()

# insert your code here_____no_output_____np.random.seed(0)

n_epochs = 5

model = model_softmax()

model.fit(input_train, output_train,
          epochs=n_epochs,
          batch_size=32,
          shuffle=True)

val = model.evaluate(input_val, output_val, verbose=False)
test = model.evaluate(input_test, output_test, verbose=False)

print('\nDataset\t loss\t\t accuracy')
print('val\t', '{0:.4f}'.format(val[0]), '\t', '{0:.4f}'.format(val[1]))
print('test\t', '{0:.4f}'.format(test[0]), '\t', '{0:.4f}'.format(test[1]))_____no_output_____

**EXPECTED OUTPUT** (may vary slightly)
```
Epoch 1/5
5215/5215 [==============================] - 2s 295us/step - loss: 0.5193 - categorical_accuracy: 0.3342
...
Epoch 5/5
5215/5215 [==============================] - 1s 162us/step - loss: 0.4389 - categorical_accuracy: 0.4071

Dataset  loss	    accuracy
val 	 0.4664 	 0.3500
test	 0.5069 	 0.1158
```_____no_output_____### Performance evaluation with cross validation
Estimate model performance on unseen data with function `cross_validation()` that implements *leave-one-out cross validation*. In this  performance evaluation scheme the original dataset is split in `K` sets or *folds*. The model is succesfully trained in `K-1` sets and tested on the remaining set. In this case, a set corresponds to a database._____no_output_____

def cross_validation(paths, model_ref, epochs=5, verbose=True):
  """Leave-one-out cross validation scheme at database level
  paths: list containing paths of HDF5 databases
  model_ref: Keras model
  epochs: number of training epochs
  verbose: print intermediate results

  returns models: list with trained Keras models
          metrics: list with train, validation and test metrics

  """

  # insert your code here
  models = []
  metrics = []

  return (models, metrics)_____no_output_____def cross_validation(paths, model_ref, epochs=5, verbose=True):
  """Leave-one-out cross validation scheme at database level
  paths: list containing paths of HDF5 databases
  model_ref: Keras model
  epochs: number of training epochs
  verbose: print intermediate results

  returns models: list with trained Keras models
         metrics: list with train, validation and test metrics

  """

  models, train, val, test = ([], [], [], [])

  n = len(paths)

  if verbose:
    print(n, 'fold cross-validation\n\nacc val   acc test   fold\n')

  for idx_test in range(len(paths)):

    x_test, y_test = load_data(paths[idx_test])
    x_test = pre_process(x_test)

    x_val, y_val = ([],[])
    x_train, y_train = ([],[])

    for idx_train in range(len(paths)):

      if idx_train!=idx_test:

        x, y = load_data(paths[idx_train])
        x = pre_process(x)

        idx = np.array([False]*y.shape[0])
        selected = np.random.choice(y.shape[0], size=300, replace=False)
        idx[selected] = True

        x_val += [x[idx]]
        y_val += [y[idx]]
        x_train += [x[~idx]]
        y_train += [y[~idx]]

    x_val = np.concatenate(x_val, axis=0)
    x_train = np.concatenate(x_train, axis=0)
    y_val = np.concatenate(y_val, axis=0)
    y_train = np.concatenate(y_train, axis=0)

    input_val = to_input(x_val)
    input_train = to_input(x_train)
    input_test = to_input(x_test)

    output_val = to_output(y_val)
    output_train = to_output(y_train)
    output_test = to_output(y_test)

    model = model_ref()

    model_fit = model.fit(input_train, output_train, epochs=epochs,
                          batch_size=32, verbose=False, shuffle=True)
    train += [model_fit.history]
    val += [model.evaluate(input_val, output_val, verbose=False)]
    test += [model.evaluate(input_test, output_test, verbose=False)]

    models += [model]

    if verbose:
      print('{0:.4f}   '.format(val[-1][1]),
            '{0:.4f}    '.format(test[-1][1]),
            idx_test+1)

  metrics = [train, val, test]

  return (models, metrics)_____no_output_____

### Train softmax model with cross validation
Train softmax model with cross validation, 10 training epochs, batch size of 32 with sample shuffling.

Plot loss and accuracy for each fold and their mean._____no_output_____

paths = ['colaboratory-data/mesa-small/mesa-sleep-0001_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0002_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0006_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0014_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0016_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0021_s.h5']

np.random.seed(0)

n_epochs = 5

models_softmax, metrics_softmax = cross_validation(paths, model_softmax,
                                                   epochs=n_epochs)

# insert your code here_____no_output_____paths = ['colaboratory-data/mesa-small/mesa-sleep-0001_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0002_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0006_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0014_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0016_s.h5',
         'colaboratory-data/mesa-small/mesa-sleep-0021_s.h5']

np.random.seed(0)

n_epochs = 5

models_softmax, metrics_softmax = cross_validation(paths, model_softmax,
                                                   epochs=n_epochs)

metrics, metrics_name = [metrics_softmax, 'softmax']

print('\nmin \tmax \tmean \tstd \t(accuracy)\n')
for item, item_name in [[np.array(metrics[1]), 'val'],
                        [np.array(metrics[2]), 'test']]:
  print('{0:.4f}'.format(np.min(item[:,1])),
        '\t{0:.4f}'.format(np.max(item[:,1])),
        '\t{0:.4f}'.format(np.mean(item[:,1])),
        '\t{0:.4f}\t'.format(np.std(item[:,1])), item_name)_____no_output_____

**EXPECTED OUTPUT** (may vary slightly)
```
6 fold cross-validation

acc val   acc test   fold

0.3753    0.3912     1
0.3360    0.3601     2
0.3013    0.3049     3
0.4067    0.3300     4
0.3353    0.2986     5
0.3487    0.1158     6

min   	 max   	 mean   	std 	(accuracy)

0.3013 	0.4067 	0.3506 	0.0332	 val
0.1158 	0.3912 	0.3001 	0.0883	 test
```_____no_output_____### ANN model
Implement ANN model with single hidden layer with 256 ReLU units and softmax output.

Write function `model_ann()` that returns the compiled model.

Use adadelta optimizer, binary crossentropy loss and categorical accuracy metric._____no_output_____

def model_ann():
  """Define shallow ANN model

  returns m: shallow ANN Keras model
  """

  # insert your code here
  m = None

  return m_____no_output_____def model_ann():
  """Define shallow ANN model

  returns m: shallow ANN Keras model
  """

  m = None

  m = Sequential()
  m.add(Layer(input_shape=(input_shape,), name='input'))
  m.add(Dense(256, activation='relu', name='h1'))
  m.add(Dense(output_shape, activation='softmax', name='output'))

  m.compile(optimizer='adadelta', loss='binary_crossentropy', metrics=['categorical_accuracy'])

  return m_____no_output_____model = model_ann()

if model is not None:
  model.summary()_____no_output_____

**EXPECTED OUTPUT**
```
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input (Layer)                (None, 5760)              0
_________________________________________________________________
h1 (Dense)                   (None, 256)               1474816
_________________________________________________________________
output (Dense)               (None, 5)                 1285
=================================================================
Total params: 1,476,101
Trainable params: 1,476,101
Non-trainable params: 0
_________________________________________________________________
```_____no_output_____### Train ANN model
Train ANN model during 5 training epochs, batch size of 32 with sample shuffling.

**Note:** the actual output may be slightly different from the expected output_____no_output_____

np.random.seed(0)

n_epochs = 5

model = model_ann()

# insert your code here_____no_output_____np.random.seed(0)

n_epochs = 5

model = model_ann()

model.fit(input_train, output_train,
          epochs=n_epochs,
          batch_size=32,
          shuffle=True)

val = model.evaluate(input_val, output_val, verbose=False)
test = model.evaluate(input_test, output_test, verbose=False)

print('\nDataset\t loss\t\t accuracy')
print('val\t', '{0:.4f}'.format(val[0]), '\t', '{0:.4f}'.format(val[1]))
print('test\t', '{0:.4f}'.format(test[0]), '\t', '{0:.4f}'.format(test[1]))_____no_output_____

**EXPECTED OUTPUT** (may vary slightly)
```
Epoch 1/5
5215/5215 [==============================] - 2s 328us/step - loss: 0.4492 - categorical_accuracy: 0.4276
...
Epoch 5/5
5215/5215 [==============================] - 1s 252us/step - loss: 0.3614 - categorical_accuracy: 0.5528

Dataset  loss   	 accuracy
val 	 0.4203 	 0.5147
test	 0.4869 	 0.5042
```_____no_output_____### Train ANN model with cross validation
Train ANN model with cross validation, 5 training epochs, batch size of 32 with sample shuffling.

**Note:** the actual output may be slightly different from the expected output_____no_output_____

np.random.seed(0)

n_epochs = 5

models_ann, metrics_ann = cross_validation(paths, model_ann, epochs=n_epochs)

# insert your code here_____no_output_____np.random.seed(0)

n_epochs = 5

models_ann, metrics_ann = cross_validation(paths, model_ann, epochs=n_epochs)

metrics, metrics_name = [metrics_ann, 'ann']

print('\nmin \tmax \tmean \tstd \t(accuracy)\n')
for item, item_name in [[np.array(metrics[1]), 'val'],
                        [np.array(metrics[2]), 'test']]:
  print('{0:.4f}'.format(np.min(item[:,1])),
        '\t{0:.4f}'.format(np.max(item[:,1])),
        '\t{0:.4f}'.format(np.mean(item[:,1])),
        '\t{0:.4f}\t'.format(np.std(item[:,1])), item_name) _____no_output_____

**EXPECTED OUTPUT** (may vary slightly)
```
6 fold cross-validation

acc val   acc test   fold

0.4893    0.4246     1
0.4713    0.4049     2
0.5320    0.4551     3
0.4880    0.4586     4
0.5147    0.3661     5
0.4993    0.5088     6

min   	 max   	 mean   	std 	(accuracy)

0.4713 	0.5320 	0.4991 	0.0196	 val
0.3661 	0.5088 	0.4363 	0.0450	 test
```_____no_output_____## Model comparison_____no_output_____### Plot train loss and accuracy for both model types
Plot train and loss per epoch for each model type. Plot average evolution across model types.

**Note:** the actual output may be slightly different from the expected output_____no_output_____

# insert your code here_____no_output_____x_range = np.array(range(n_epochs)) + 1

for metrics, metrics_name in [[metrics_softmax, 'softmax'],
                              [metrics_ann, 'ann']]:

  loss, acc = ([],[])

  fig = plt.figure(figsize=(2*fig_w,fig_h))

  ax1 = plt.subplot(1, 2, 1)
  plt.title('Train loss ('+metrics_name+')')
  plt.ylabel('Loss')
  plt.xlabel('Epochs')

  ax2 = plt.subplot(1, 2, 2)
  plt.title('Train accuracy ('+metrics_name+')')
  plt.ylabel('Accuracy')
  plt.xlabel('Epochs')

  for item in metrics[0]:

    loss += [item['loss']]
    acc += [item['categorical_accuracy']]

    data = loss[-1]
    ax1.plot(x_range, data, 'k-', alpha=0.4)

    data = acc[-1]
    ax2.plot(x_range, data, 'k-', alpha=0.4)

  ax1.plot(x_range, np.mean(np.array(loss), axis=0), '-',
           alpha=0.8, linewidth=4, label='mean')
  ax1.legend()

  ax2.plot(x_range, np.mean(np.array(acc), axis=0), '-',
           alpha=0.8, linewidth=4, label='mean')
  ax2.legend()

  plt.tight_layout()_____no_output_____

**EXPECTED OUTPUT** (may vary slightly)

_____no_output_____### Compare train performance between both models
Plot train and loss per epoch for each model type. Plot average evolution across model types.

**Note:** the actual output may be slightly different from the expected output_____no_output_____

# insert your code here_____no_output_____loss, acc = ([],[])
x_range = np.array(range(n_epochs)) + 1

fig = plt.figure(figsize=(2*fig_w,fig_h))
ax1 = plt.subplot(1, 2, 1)
plt.title('Train loss')
plt.ylabel('Loss')
plt.xlabel('Epochs')
ax2 = plt.subplot(1, 2, 2)
plt.title('Train accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epochs')

for metrics, metrics_name in [[metrics_softmax, 'softmax'],
                              [metrics_ann, 'ann']]:

  loss, acc = ([],[])
  for item in metrics[0]:

    loss += [item['loss']]
    acc += [item['categorical_accuracy']]

  ax1.plot(x_range, np.mean(loss, axis=0), '-',
           alpha=0.8, linewidth=4, label=metrics_name)

  ax2.plot(x_range, np.mean(acc, axis=0), '-',
           alpha=0.8, linewidth=4, label=metrics_name)

ax1.legend()
ax2.legend()

plt.tight_layout()_____no_output_____

**EXPECTED OUTPUT** (may vary slightly)

_____no_output_____## Additional questions_____no_output_____Make changes to the model and training data and investigate performance impacts.

**Suggestions**
* Try other types of activation units
* Add additional hidden layers
* Use different backprop optimizer
* Change mini-batch size
* Include additional polysomnograph channels
* Use different data pre-processing operations
* Use spectral input representation
* Use models from cross validation to implement committee of networks with majority voting_____no_output_____