# Notebook from alarafat/Image_Captioning
Path: 2_Training.ipynb

# Computer Vision Nanodegree

## Project: Image Captioning

---

In this notebook, you will train your CNN-RNN model.

You are welcome and encouraged to try out many different architectures and hyperparameters when searching for a good model.

This does have the potential to make the project quite messy!  Before submitting your project, make sure that you clean up:
- the code you write in this notebook.  The notebook should describe how to train a single CNN-RNN architecture, corresponding to your final choice of hyperparameters.  You should structure the notebook so that the reviewer can replicate your results by running the code in this notebook.
- the output of the code cell in **Step 2**.  The output should show the output obtained when training the model from scratch.

This notebook **will be graded**.

Feel free to use the links below to navigate the notebook:
- [Step 1](#step1): Training Setup
- [Step 2](#step2): Train your Model
- [Step 3](#step3): (Optional) Validate your Model_____no_output_____
## Step 1: Training Setup

In this step of the notebook, you will customize the training of your CNN-RNN model by specifying hyperparameters and setting other options that are important to the training procedure.  The values you set now will be used when training your model in **Step 2** below.

You should only amend blocks of code that are preceded by a `TODO` statement.  **Any code blocks that are not preceded by a `TODO` statement should not be modified**.

### Task #1

Begin by setting the following variables:
- `batch_size` - the batch size of each training batch.  It is the number of image-caption pairs used to amend the model weights in each training step.
- `vocab_threshold` - the minimum word count threshold.  Note that a larger threshold will result in a smaller vocabulary, whereas a smaller threshold will include rarer words and result in a larger vocabulary.
- `vocab_from_file` - a Boolean that decides whether to load the vocabulary from file.
- `embed_size` - the dimensionality of the image and word embeddings.
- `hidden_size` - the number of features in the hidden state of the RNN decoder.
- `num_epochs` - the number of epochs to train the model.  We recommend that you set `num_epochs=3`, but feel free to increase or decrease this number as you wish.  [This paper](https://arxiv.org/pdf/1502.03044.pdf) trained a captioning model on a single state-of-the-art GPU for 3 days, but you'll soon see that you can get reasonable results in a matter of a few hours!  (_But of course, if you want your model to compete with current research, you will have to train for much longer._)
- `save_every` - determines how often to save the model weights.  We recommend that you set `save_every=1`, to save the model weights after each epoch.  This way, after the `i`th epoch, the encoder and decoder weights will be saved in the `models/` folder as `encoder-i.pkl` and `decoder-i.pkl`, respectively.
- `print_every` - determines how often to print the batch loss to the Jupyter notebook while training.  Note that you **will not** observe a monotonic decrease in the loss function while training - this is perfectly fine and completely expected!  You are encouraged to keep this at its default value of `100` to avoid clogging the notebook, but feel free to change it.
- `log_file` - the name of the text file containing - for every step - how the loss and perplexity evolved during training.

If you're not sure where to begin to set some of the values above, you can peruse [this paper](https://arxiv.org/pdf/1502.03044.pdf) and [this paper](https://arxiv.org/pdf/1411.4555.pdf) for useful guidance!  **To avoid spending too long on this notebook**, you are encouraged to consult these suggested research papers to obtain a strong initial guess for which hyperparameters are likely to work best.  Then, train a single model, and proceed to the next notebook (**3_Inference.ipynb**).  If you are unhappy with your performance, you can return to this notebook to tweak the hyperparameters (and/or the architecture in **model.py**) and re-train your model.

### Question 1

**Question:** Describe your CNN-RNN architecture in detail.  With this architecture in mind, how did you select the values of the variables in Task 1?  If you consulted a research paper detailing a successful implementation of an image captioning model, please provide the reference.

**Answer:** I took the idea about the hyper-parameters from the suggestied paper: Show and Tell: A Neural Image Caption Generator(https://arxiv.org/pdf/1411.4555.pdf). But, I empirically tweaked several parameters and tried to chose the best one. I ran several model at multiple computers with LSTM & GRU with 2 stacked layers and 1 layer. I finally chose GRU with 1 layer model as it performs much better than the LSTM model with same parameters. The vocabulary size has been evaluated between 2 and 5. Vocabulary size 2 increased training time with lot of unfrequent words and 5 seemed works muck fine for my model. For embedding size and hidden layers size, their sizes for each LSTM cell were tweaked between 256 or 512. My GRU model trained with embedding size 512 and hidden size of 256 works fine. 1 layer LSTM model with embedding size 512, hidden size 256 and 300, and trained with Adadelta also worked great (quite similar). But, GRU model seemed performing better to me. Thus, I chose GRU model. To save a model, I checked perplexicity value.

### (Optional) Task #2

Note that we have provided a recommended image transform `transform_train` for pre-processing the training images, but you are welcome (and encouraged!) to modify it as you wish.  When modifying this transform, keep in mind that:
- the images in the dataset have varying heights and widths, and
- if using a pre-trained model, you must perform the corresponding appropriate normalization.

### Question 2

**Question:** How did you select the transform in `transform_train`?  If you left the transform at its provided value, why do you think that it is a good choice for your CNN architecture?

**Answer:** I tried with centercrop but, it didn't work better and thus, I dropped using it and kept the provided image transformation. I thought using centrecrop instead of Randomcrop might work better while it didn't.

### Task #3

Next, you will specify a Python list containing the learnable parameters of the model.  For instance, if you decide to make all weights in the decoder trainable, but only want to train the weights in the embedding layer of the encoder, then you should set `params` to something like:
```
params = list(decoder.parameters()) + list(encoder.embed.parameters())
```

### Question 3

**Question:** How did you select the trainable parameters of your architecture?  Why do you think this is a good choice?

**Answer:** I used the above given equation about which paramters to train. I trained all the decoder parameters and the embedding layer's paramters of encoder. It's because I am using a pre-trained model for encoder which doesn't need to be trained end-to-end. Thus, training only the embedding layer should suffice.

### Task #4

Finally, you will select an [optimizer](http://pytorch.org/docs/master/optim.html#torch.optim.Optimizer).

### Question 4

**Question:** How did you select the optimizer used to train your model?

**Answer:** I tried with Adadelta, Adam, RMSprop, and SGD as optimizer. RMSprop and Adam worked badly with my model. I used SGD with Cosine Annealing Learning Rate scheduler, but I couldn't make it out-perform Adadelta (may be, I couldn't use it properly). My models using Adadelta converged nicely and fast. Although it giggles around as Adadelta is not a smoothing optimizer, it worked great with my model. Thus, I chose Adadelta as optimizer. _____no_output_____

import nltk
nltk.download('punkt')[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
import torch
import torch.nn as nn
from torchvision import transforms
import sys
sys.path.append('/opt/cocoapi/PythonAPI')
from pycocotools.coco import COCO
from data_loader import get_loader
from model import EncoderCNN, DecoderRNN
import math

## TODO #1: Select appropriate values for the Python variables below.
batch_size = 32          # batch size
vocab_threshold = 5        # minimum word count threshold
vocab_from_file = True    # if True, load existing vocab file
embed_size = 512           # dimensionality of image and word embeddings
hidden_size = 256          # number of features in hidden state of the RNN decoder
num_epochs = 3             # number of training epochs
save_every = 1             # determines frequency of saving model weights
print_every = 100          # determines window for printing average loss
log_file = 'training_log.txt'       # name of file with saved training loss and perplexity

# (Optional) TODO #2: Amend the image transform below.
transform_train = transforms.Compose([
    transforms.Resize(256),                          # smaller edge of image resized to 256
    transforms.RandomCrop(224),                      # get 224x224 crop from random location
    transforms.RandomHorizontalFlip(),               # horizontally flip image with probability=0.5
    transforms.ToTensor(),                           # convert the PIL Image to a tensor
    transforms.Normalize((0.485, 0.456, 0.406),      # normalize image for pre-trained model
                         (0.229, 0.224, 0.225))])

# Build data loader.
data_loader = get_loader(transform=transform_train,
                         mode='train',
                         batch_size=batch_size,
                         vocab_threshold=vocab_threshold,
                         vocab_from_file=vocab_from_file)Vocabulary successfully loaded from vocab.pkl file!
loading annotations into memory...
Done (t=0.98s)
creating index...
# The size of the vocabulary.
vocab_size = len(data_loader.dataset.vocab)

# Initialize the encoder and decoder.
encoder = EncoderCNN(embed_size)
decoder = DecoderRNN(embed_size, hidden_size, vocab_size, num_layers=1)

# Move models to GPU if CUDA is available.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
encoder.to(device)
decoder.to(device)

# Define the loss function.
criterion = nn.CrossEntropyLoss().cuda() if torch.cuda.is_available() else nn.CrossEntropyLoss()

# TODO #3: Specify the learnable parameters of the model.
params = list(decoder.parameters()) + list(encoder.embed.parameters())

# TODO #4: Define the optimizer.
optimizer = torch.optim.Adadelta(params)

# learning_rate = 0.1
# weight_decay = 0.1
# optimizer = torch.optim.SGD(params, lr=learning_rate, momentum=0.9)
# learning_rate_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=50, eta_min=0.000001)

# Set the total number of training steps per epoch.
total_step = math.ceil(len(data_loader.dataset.caption_lengths) / data_loader.batch_sampler.batch_size) Downloading: "https://download.pytorch.org/models/resnet50-19c8e357.pth" to /root/.torch/models/resnet50-19c8e357.pth
100%|██████████| 102502400/102502400 [00:01<00:00, 81255462.31it/s]

## Step 2: Train your Model

Once you have executed the code cell in **Step 1**, the training procedure below should run without issue.

It is completely fine to leave the code cell below as-is without modifications to train your model.  However, if you would like to modify the code used to train the model below, you must ensure that your changes are easily parsed by your reviewer.  In other words, make sure to provide appropriate comments to describe how your code works!

You may find it useful to load saved weights to resume training.  In that case, note the names of the files containing the encoder and decoder weights that you'd like to load (`encoder_file` and `decoder_file`).  Then you can load the weights by using the lines below:

```python
# Load pre-trained weights before resuming training.
encoder.load_state_dict(torch.load(os.path.join('./models', encoder_file)))
decoder.load_state_dict(torch.load(os.path.join('./models', decoder_file)))
```

While trying out parameters, make sure to take extensive notes and record the settings that you used in your various training runs.  In particular, you don't want to encounter a situation where you've trained a model for several hours but can't remember what settings you used :).

### A Note on Tuning Hyperparameters

To figure out how well your model is doing, you can look at how the training loss and perplexity evolve during training - and for the purposes of this project, you are encouraged to amend the hyperparameters based on this information.

However, this will not tell you if your model is overfitting to the training data, and, unfortunately, overfitting is a problem that is commonly encountered when training image captioning models.

For this project, you need not worry about overfitting. **This project does not have strict requirements regarding the performance of your model**, and you just need to demonstrate that your model has learned **_something_** when you generate captions on the test data.  For now, we strongly encourage you to train your model for the suggested 3 epochs without worrying about performance; then, you should immediately transition to the next notebook in the sequence (**3_Inference.ipynb**) to see how your model performs on the test data.  If your model needs to be changed, you can come back to this notebook, amend hyperparameters (if necessary), and re-train the model.

That said, if you would like to go above and beyond in this project, you can read about some approaches to minimizing overfitting in section 4.3.1 of [this paper](http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7505636).  In the next (optional) step of this notebook, we provide some guidance for assessing the performance on the validation dataset._____no_output_____

import torch.utils.data as data
import numpy as np
import os
import requests
import time

# Open the training log file.
f = open(log_file, 'w')

old_time = time.time()
response = requests.request("GET",
                            "http://metadata.google.internal/computeMetadata/v1/instance/attributes/keep_alive_token",
                            headers={"Metadata-Flavor":"Google"})

for epoch in range(1, num_epochs+1):

    for i_step in range(1, total_step+1):

        if time.time() - old_time > 60:
            old_time = time.time()
            requests.request("POST",
                             "https://nebula.udacity.com/api/v1/remote/keep-alive",
                             headers={'Authorization': "STAR " + response.text})

        # Randomly sample a caption length, and sample indices with that length.
        indices = data_loader.dataset.get_train_indices()
        # Create and assign a batch sampler to retrieve a batch with the sampled indices.
        new_sampler = data.sampler.SubsetRandomSampler(indices=indices)
        data_loader.batch_sampler.sampler = new_sampler

        # Obtain the batch.
        images, captions = next(iter(data_loader))

        # Move batch of images and captions to GPU if CUDA is available.
        images = images.to(device)
        captions = captions.to(device)

        # Zero the gradients.
        decoder.zero_grad()
        encoder.zero_grad()

        # Pass the inputs through the CNN-RNN model.
        features = encoder(images)
        outputs = decoder(features, captions)

        # Calculate the batch loss.
        loss = criterion(outputs.view(-1, vocab_size), captions.view(-1))

        # Backward pass.
        loss.backward()

        # Update the parameters in the optimizer.
        optimizer.step()

        # Get training statistics.
        perplexity_val = np.exp(loss.item())
        stats = 'Epoch [%d/%d], Step [%d/%d], Loss: %.4f, Perplexity: %5.4f' % (epoch, num_epochs, i_step, total_step, loss.item(), perplexity_val)

        # Print training statistics (on same line).
        print('\r' + stats, end="")
        sys.stdout.flush()

        # Print training statistics to file.
        f.write(stats + '\n')
        f.flush()

        # Print training statistics (on different line).
        if i_step % print_every == 0:
            print('\r' + stats)
        if perplexity_val < 6.5:
            torch.save(decoder.state_dict(), os.path.join('./models', 'decoder-%d-%d-%d.pkl' % (epoch, i_step, perplexity_val)))
            torch.save(encoder.state_dict(), os.path.join('./models', 'encoder-%d-%d-%d.pkl' % (epoch, i_step, perplexity_val)))

    # Save the weights.
    if epoch % save_every == 0:
        torch.save(decoder.state_dict(), os.path.join('./models', 'decoder-%d.pkl' % epoch))
        torch.save(encoder.state_dict(), os.path.join('./models', 'encoder-%d.pkl' % epoch))
    #         learning_rate_scheduler.step()

# Close the training log file.
f.close()Epoch [1/3], Step [100/12942], Loss: 3.1850, Perplexity: 24.1672
Epoch [1/3], Step [200/12942], Loss: 3.3418, Perplexity: 28.2709
Epoch [1/3], Step [300/12942], Loss: 3.2090, Perplexity: 24.7550
Epoch [1/3], Step [400/12942], Loss: 3.4699, Perplexity: 32.1342
Epoch [1/3], Step [500/12942], Loss: 2.8323, Perplexity: 16.98537
Epoch [1/3], Step [600/12942], Loss: 3.3636, Perplexity: 28.8936
Epoch [1/3], Step [700/12942], Loss: 3.0057, Perplexity: 20.20070
Epoch [1/3], Step [800/12942], Loss: 2.8693, Perplexity: 17.6253
Epoch [1/3], Step [900/12942], Loss: 2.7482, Perplexity: 15.6139
Epoch [1/3], Step [1000/12942], Loss: 2.9931, Perplexity: 19.9472
Epoch [1/3], Step [1100/12942], Loss: 3.5998, Perplexity: 36.59139
Epoch [1/3], Step [1200/12942], Loss: 2.9444, Perplexity: 18.9997
Epoch [1/3], Step [1300/12942], Loss: 2.9473, Perplexity: 19.0550
Epoch [1/3], Step [1400/12942], Loss: 2.9772, Perplexity: 19.6325
Epoch [1/3], Step [1500/12942], Loss: 2.5626, Perplexity: 12.9700
Epoch [1/3], Step [1600/12942], Loss: 3.0036, Perplexity: 20.1578
Epoch [1/3], Step [1700/12942], Loss: 3.0148, Perplexity: 20.3849
Epoch [1/3], Step [1800/12942], Loss: 2.8055, Perplexity: 16.5354
Epoch [1/3], Step [1900/12942], Loss: 2.3978, Perplexity: 10.9993
Epoch [1/3], Step [2000/12942], Loss: 2.8052, Perplexity: 16.5306
Epoch [1/3], Step [2100/12942], Loss: 2.5947, Perplexity: 13.39274
Epoch [1/3], Step [2200/12942], Loss: 3.0825, Perplexity: 21.8137
Epoch [1/3], Step [2300/12942], Loss: 2.9475, Perplexity: 19.0579
Epoch [1/3], Step [2400/12942], Loss: 2.8246, Perplexity: 16.8546
Epoch [1/3], Step [2500/12942], Loss: 2.8106, Perplexity: 16.6207
Epoch [1/3], Step [2600/12942], Loss: 2.5033, Perplexity: 12.2228
Epoch [1/3], Step [2700/12942], Loss: 2.7994, Perplexity: 16.4343
Epoch [1/3], Step [2800/12942], Loss: 2.4734, Perplexity: 11.8629
Epoch [1/3], Step [2900/12942], Loss: 2.7596, Perplexity: 15.7936
Epoch [1/3], Step [3000/12942], Loss: 2.6414, Perplexity: 14.0324
Epoch [1/3], Step [3100/12942], Loss: 2.7657, Perplexity: 15.8901
Epoch [1/3], Step [3200/12942], Loss: 2.3883, Perplexity: 10.8950
Epoch [1/3], Step [3300/12942], Loss: 2.3676, Perplexity: 10.6714
Epoch [1/3], Step [3400/12942], Loss: 3.6534, Perplexity: 38.6050
Epoch [1/3], Step [3500/12942], Loss: 3.1287, Perplexity: 22.8437
Epoch [1/3], Step [3600/12942], Loss: 2.8553, Perplexity: 17.3801
Epoch [1/3], Step [3700/12942], Loss: 2.6268, Perplexity: 13.8299
Epoch [1/3], Step [3800/12942], Loss: 2.3221, Perplexity: 10.1966
Epoch [1/3], Step [3900/12942], Loss: 2.2740, Perplexity: 9.71854
Epoch [1/3], Step [4000/12942], Loss: 2.9177, Perplexity: 18.4980
Epoch [1/3], Step [4100/12942], Loss: 2.5359, Perplexity: 12.62749
Epoch [1/3], Step [4200/12942], Loss: 2.4573, Perplexity: 11.6732
Epoch [1/3], Step [4300/12942], Loss: 2.5088, Perplexity: 12.2903
Epoch [1/3], Step [4400/12942], Loss: 2.6142, Perplexity: 13.6559
Epoch [1/3], Step [4500/12942], Loss: 2.4259, Perplexity: 11.3121
Epoch [1/3], Step [4600/12942], Loss: 2.5985, Perplexity: 13.4429
Epoch [1/3], Step [4700/12942], Loss: 2.7271, Perplexity: 15.2888
Epoch [1/3], Step [4800/12942], Loss: 3.0229, Perplexity: 20.5501
Epoch [1/3], Step [4900/12942], Loss: 2.4728, Perplexity: 11.85615
Epoch [1/3], Step [5000/12942], Loss: 2.5280, Perplexity: 12.5286
Epoch [1/3], Step [5100/12942], Loss: 2.3979, Perplexity: 10.9999
Epoch [1/3], Step [5200/12942], Loss: 2.6083, Perplexity: 13.5755
Epoch [1/3], Step [5300/12942], Loss: 2.5703, Perplexity: 13.0700
Epoch [1/3], Step [5400/12942], Loss: 2.1637, Perplexity: 8.70322
Epoch [1/3], Step [5500/12942], Loss: 2.4114, Perplexity: 11.1491
Epoch [1/3], Step [5600/12942], Loss: 2.5349, Perplexity: 12.6156
Epoch [1/3], Step [5700/12942], Loss: 2.4035, Perplexity: 11.0618
Epoch [1/3], Step [5800/12942], Loss: 2.3624, Perplexity: 10.6162
Epoch [1/3], Step [5900/12942], Loss: 2.4224, Perplexity: 11.2728
Epoch [1/3], Step [6000/12942], Loss: 2.2377, Perplexity: 9.37149
Epoch [1/3], Step [6100/12942], Loss: 2.5252, Perplexity: 12.4938
Epoch [1/3], Step [6200/12942], Loss: 2.9313, Perplexity: 18.7528
Epoch [1/3], Step [6300/12942], Loss: 3.1049, Perplexity: 22.30723
Epoch [1/3], Step [6400/12942], Loss: 2.5903, Perplexity: 13.3337
Epoch [1/3], Step [6500/12942], Loss: 2.9624, Perplexity: 19.3437
Epoch [1/3], Step [6600/12942], Loss: 2.7676, Perplexity: 15.9197
Epoch [1/3], Step [6700/12942], Loss: 2.5104, Perplexity: 12.3103
Epoch [1/3], Step [6800/12942], Loss: 2.1605, Perplexity: 8.67511
Epoch [1/3], Step [6900/12942], Loss: 2.3311, Perplexity: 10.2894
Epoch [1/3], Step [7000/12942], Loss: 2.9559, Perplexity: 19.2197
Epoch [1/3], Step [7100/12942], Loss: 2.6480, Perplexity: 14.1259
Epoch [1/3], Step [7200/12942], Loss: 2.4188, Perplexity: 11.2322
Epoch [1/3], Step [7300/12942], Loss: 2.5698, Perplexity: 13.0627
Epoch [1/3], Step [7400/12942], Loss: 2.5138, Perplexity: 12.35146
Epoch [1/3], Step [7500/12942], Loss: 2.7744, Perplexity: 16.0293
Epoch [1/3], Step [7600/12942], Loss: 2.3397, Perplexity: 10.3783
Epoch [1/3], Step [7700/12942], Loss: 2.3387, Perplexity: 10.3681
Epoch [1/3], Step [7800/12942], Loss: 2.5058, Perplexity: 12.2537
Epoch [1/3], Step [7900/12942], Loss: 2.2406, Perplexity: 9.39946
Epoch [1/3], Step [8000/12942], Loss: 2.0303, Perplexity: 7.61610
Epoch [1/3], Step [8100/12942], Loss: 2.3152, Perplexity: 10.1274
Epoch [1/3], Step [8200/12942], Loss: 2.5095, Perplexity: 12.2988
Epoch [1/3], Step [8300/12942], Loss: 2.7810, Perplexity: 16.1346
Epoch [1/3], Step [8400/12942], Loss: 2.3190, Perplexity: 10.1655
Epoch [1/3], Step [8500/12942], Loss: 2.3240, Perplexity: 10.2166
Epoch [1/3], Step [8600/12942], Loss: 2.4326, Perplexity: 11.3881
Epoch [1/3], Step [8700/12942], Loss: 2.3060, Perplexity: 10.0344
Epoch [1/3], Step [8800/12942], Loss: 2.5523, Perplexity: 12.8361
Epoch [1/3], Step [8900/12942], Loss: 2.4200, Perplexity: 11.2462
Epoch [1/3], Step [9000/12942], Loss: 2.2527, Perplexity: 9.51359
Epoch [1/3], Step [9100/12942], Loss: 2.4060, Perplexity: 11.0896
Epoch [1/3], Step [9200/12942], Loss: 2.6539, Perplexity: 14.2094
Epoch [1/3], Step [9300/12942], Loss: 2.4767, Perplexity: 11.9021
Epoch [1/3], Step [9400/12942], Loss: 2.2394, Perplexity: 9.38736
Epoch [1/3], Step [9500/12942], Loss: 2.6827, Perplexity: 14.6246
Epoch [1/3], Step [9600/12942], Loss: 2.2588, Perplexity: 9.57209
Epoch [1/3], Step [9700/12942], Loss: 3.0631, Perplexity: 21.3937
Epoch [1/3], Step [9800/12942], Loss: 2.7218, Perplexity: 15.20782
Epoch [1/3], Step [9900/12942], Loss: 2.5079, Perplexity: 12.2792
Epoch [1/3], Step [10000/12942], Loss: 2.1879, Perplexity: 8.9167
Epoch [1/3], Step [10100/12942], Loss: 2.8951, Perplexity: 18.0857
Epoch [1/3], Step [10200/12942], Loss: 3.0161, Perplexity: 20.4108
Epoch [1/3], Step [10300/12942], Loss: 2.4440, Perplexity: 11.5196
Epoch [1/3], Step [10400/12942], Loss: 2.6257, Perplexity: 13.8145
Epoch [1/3], Step [10500/12942], Loss: 2.3354, Perplexity: 10.3334
Epoch [1/3], Step [10600/12942], Loss: 2.7558, Perplexity: 15.7336
Epoch [1/3], Step [10700/12942], Loss: 2.7958, Perplexity: 16.3762
Epoch [1/3], Step [10800/12942], Loss: 2.2188, Perplexity: 9.19650
Epoch [1/3], Step [10900/12942], Loss: 2.7284, Perplexity: 15.3087
Epoch [1/3], Step [11000/12942], Loss: 2.2165, Perplexity: 9.17495
Epoch [1/3], Step [11100/12942], Loss: 2.3171, Perplexity: 10.1460
Epoch [1/3], Step [11200/12942], Loss: 2.1527, Perplexity: 8.60783
Epoch [1/3], Step [11300/12942], Loss: 2.0431, Perplexity: 7.71437
Epoch [1/3], Step [11400/12942], Loss: 2.6310, Perplexity: 13.8872
Epoch [1/3], Step [11500/12942], Loss: 1.9649, Perplexity: 7.13429
Epoch [1/3], Step [11600/12942], Loss: 2.2900, Perplexity: 9.87504
Epoch [1/3], Step [11700/12942], Loss: 2.0755, Perplexity: 7.96885
Epoch [1/3], Step [11800/12942], Loss: 2.4710, Perplexity: 11.8342
Epoch [1/3], Step [11900/12942], Loss: 2.3640, Perplexity: 10.6333
Epoch [1/3], Step [12000/12942], Loss: 2.4146, Perplexity: 11.1857
Epoch [1/3], Step [12100/12942], Loss: 1.9962, Perplexity: 7.36128
Epoch [1/3], Step [12200/12942], Loss: 2.2638, Perplexity: 9.61940
Epoch [1/3], Step [12300/12942], Loss: 2.0943, Perplexity: 8.11943
Epoch [1/3], Step [12400/12942], Loss: 2.6329, Perplexity: 13.9136
Epoch [1/3], Step [12500/12942], Loss: 2.3799, Perplexity: 10.8033
Epoch [1/3], Step [12600/12942], Loss: 2.5686, Perplexity: 13.0478
Epoch [1/3], Step [12700/12942], Loss: 2.4502, Perplexity: 11.5906
Epoch [1/3], Step [12800/12942], Loss: 2.1854, Perplexity: 8.89384
Epoch [1/3], Step [12900/12942], Loss: 2.3417, Perplexity: 10.3994
Epoch [2/3], Step [100/12942], Loss: 2.4341, Perplexity: 11.406049
Epoch [2/3], Step [200/12942], Loss: 2.2879, Perplexity: 9.85391
Epoch [2/3], Step [300/12942], Loss: 2.3779, Perplexity: 10.7825
Epoch [2/3], Step [400/12942], Loss: 2.5610, Perplexity: 12.9484
Epoch [2/3], Step [500/12942], Loss: 2.4512, Perplexity: 11.6024
Epoch [2/3], Step [600/12942], Loss: 2.5768, Perplexity: 13.1552
Epoch [2/3], Step [700/12942], Loss: 2.7439, Perplexity: 15.5482
Epoch [2/3], Step [800/12942], Loss: 2.2659, Perplexity: 9.63994
Epoch [2/3], Step [900/12942], Loss: 2.4865, Perplexity: 12.0192
Epoch [2/3], Step [1000/12942], Loss: 2.7440, Perplexity: 15.5493
Epoch [2/3], Step [1100/12942], Loss: 2.2043, Perplexity: 9.06429
Epoch [2/3], Step [1200/12942], Loss: 2.3276, Perplexity: 10.2532
Epoch [2/3], Step [1300/12942], Loss: 2.6515, Perplexity: 14.1750
Epoch [2/3], Step [1400/12942], Loss: 2.1412, Perplexity: 8.50993
Epoch [2/3], Step [1500/12942], Loss: 2.8590, Perplexity: 17.4441
Epoch [2/3], Step [1600/12942], Loss: 2.1226, Perplexity: 8.35274
Epoch [2/3], Step [1700/12942], Loss: 2.2656, Perplexity: 9.63669
Epoch [2/3], Step [1800/12942], Loss: 2.2466, Perplexity: 9.45571
Epoch [2/3], Step [1900/12942], Loss: 2.4939, Perplexity: 12.1085
Epoch [2/3], Step [2000/12942], Loss: 2.2115, Perplexity: 9.12901
Epoch [2/3], Step [2100/12942], Loss: 2.1800, Perplexity: 8.84632
Epoch [2/3], Step [2200/12942], Loss: 2.1141, Perplexity: 8.28175
Epoch [2/3], Step [2300/12942], Loss: 2.3247, Perplexity: 10.2235
Epoch [2/3], Step [2400/12942], Loss: 2.3146, Perplexity: 10.1205
Epoch [2/3], Step [2500/12942], Loss: 2.0785, Perplexity: 7.99211
Epoch [2/3], Step [2600/12942], Loss: 2.6282, Perplexity: 13.8494
Epoch [2/3], Step [2700/12942], Loss: 2.9884, Perplexity: 19.8540
Epoch [2/3], Step [2800/12942], Loss: 2.5953, Perplexity: 13.4010
Epoch [2/3], Step [2900/12942], Loss: 2.3273, Perplexity: 10.2498
Epoch [2/3], Step [3000/12942], Loss: 2.1653, Perplexity: 8.71736
Epoch [2/3], Step [3100/12942], Loss: 1.9582, Perplexity: 7.086340
Epoch [2/3], Step [3200/12942], Loss: 2.3253, Perplexity: 10.2294
Epoch [2/3], Step [3300/12942], Loss: 2.3692, Perplexity: 10.6886
Epoch [2/3], Step [3400/12942], Loss: 2.3245, Perplexity: 10.2215
Epoch [2/3], Step [3500/12942], Loss: 2.0768, Perplexity: 7.97863
Epoch [2/3], Step [3600/12942], Loss: 2.1986, Perplexity: 9.01273
Epoch [2/3], Step [3700/12942], Loss: 2.2580, Perplexity: 9.56358
Epoch [2/3], Step [3800/12942], Loss: 2.2813, Perplexity: 9.78947
Epoch [2/3], Step [3900/12942], Loss: 2.1406, Perplexity: 8.50486
Epoch [2/3], Step [4000/12942], Loss: 2.1295, Perplexity: 8.41092
Epoch [2/3], Step [4100/12942], Loss: 2.4609, Perplexity: 11.7151
Epoch [2/3], Step [4200/12942], Loss: 2.3300, Perplexity: 10.2779
Epoch [2/3], Step [4300/12942], Loss: 2.3054, Perplexity: 10.0279
Epoch [2/3], Step [4400/12942], Loss: 2.4988, Perplexity: 12.1678
Epoch [2/3], Step [4500/12942], Loss: 2.3234, Perplexity: 10.2103
Epoch [2/3], Step [4600/12942], Loss: 2.2894, Perplexity: 9.86917
Epoch [2/3], Step [4700/12942], Loss: 2.2360, Perplexity: 9.35570
Epoch [2/3], Step [4800/12942], Loss: 2.5469, Perplexity: 12.7675
Epoch [2/3], Step [4900/12942], Loss: 2.4091, Perplexity: 11.1237
Epoch [2/3], Step [5000/12942], Loss: 2.1921, Perplexity: 8.95385
Epoch [2/3], Step [5100/12942], Loss: 2.1900, Perplexity: 8.93483
Epoch [2/3], Step [5200/12942], Loss: 2.6108, Perplexity: 13.6106
Epoch [2/3], Step [5300/12942], Loss: 2.4999, Perplexity: 12.1818
Epoch [2/3], Step [5400/12942], Loss: 2.2434, Perplexity: 9.42574
Epoch [2/3], Step [5500/12942], Loss: 3.2286, Perplexity: 25.2451
Epoch [2/3], Step [5600/12942], Loss: 1.8471, Perplexity: 6.34160
Epoch [2/3], Step [5681/12942], Loss: 2.1994, Perplexity: 9.01954

## Step 3: (Optional) Validate your Model

To assess potential overfitting, one approach is to assess performance on a validation set.  If you decide to do this **optional** task, you are required to first complete all of the steps in the next notebook in the sequence (**3_Inference.ipynb**); as part of that notebook, you will write and test code (specifically, the `sample` method in the `DecoderRNN` class) that uses your RNN decoder to generate captions.  That code will prove incredibly useful here.

If you decide to validate your model, please do not edit the data loader in **data_loader.py**.  Instead, create a new file named **data_loader_val.py** containing the code for obtaining the data loader for the validation data.  You can access:
- the validation images at filepath `'/opt/cocoapi/images/train2014/'`, and
- the validation image caption annotation file at filepath `'/opt/cocoapi/annotations/captions_val2014.json'`.

The suggested approach to validating your model involves creating a json file such as [this one](https://github.com/cocodataset/cocoapi/blob/master/results/captions_val2014_fakecap_results.json) containing your model's predicted captions for the validation images.  Then, you can write your own script or use one that you [find online](https://github.com/tylin/coco-caption) to calculate the BLEU score of your model.  You can read more about the BLEU score, along with other evaluation metrics (such as TEOR and Cider) in section 4.1 of [this paper](https://arxiv.org/pdf/1411.4555.pdf).  For more information about how to use the annotation file, check out the [website](http://cocodataset.org/#download) for the COCO dataset._____no_output_____

# (Optional) TODO: Validate your model._____no_output_____


# Notebook from BioSysDesign/E164
Path: hw/hw3.ipynb

# Modeling the central dogma of molecular biology
Resources:

1. [week3_intro_ode.ipynb](https://pages.hmc.edu/pandey/reading/week3_intro_ode.ipynb): This notebook introduces how to model systems using differential equations in Python. The notebook also shows how to use `odeint` to integrate the ODEs to create a simulation of the model.

3. Python tutorials online: You are free to use any tutorials from the Internet on Numpy, Scipy, or any other Python package that you may use to solve the homework problems.

4. Submit your homework to GradeScope by downloading the jupyter notebook as a PDF. Go to Files -> Download as -> PDF. If that does not work, you can go to File -> Print Preview -> Ctrl + P (to print) -> Save as PDF.

Due date: 8th February on GradeScope._____no_output_____## Problem 1: Transcription and translation mechanisms

[BFS 2.6] Consider the transcription and translation reactions incorporating the elongation process as considered in this chapter (Chapter 2, BFS) in equations (2.10)–(2.11). Modify them to the case in which an mRNA molecule can be translated to a polypeptide chain even while it is still elongating._____no_output_____Answer:_____no_output_____## Problem 2: Modeling transcription and translation

(a) Write a simplified two-state model for transcription and translation that models the concentration of protein (X) and mRNA (T) as the two states. Describe what the different parameters mean in your model and how they relate to the various mechanisms of transcription and translation.

(b) Choose biologically meaningful parameter values for your model in (a) assuming that we are modeling transcription and translation processes in a bacteria. Report the parameter values and show your work on how you estimate the parameter values.

(c) What are the steady-state concentrations for the two states in your transcription-translation model. Report the steady-state values in terms of the parameters of your model.

(d) Simulate your model and plot: (1) the mRNA concentration with time, (2) the protein concentration with time. Also, plot the analytically computed steady-state on this graph.

(e) Using your mathematical model comment on how we can control the processes of transcription and translation. What would be the different ways in which we can control the protein expression?_____no_output_____

# Solutions:_____no_output_____

## Problem 3: Mathematical models from SBOL diagrams

Synthetic Biology Open Language (SBOL) is the standard way to draw biological circuit diagrams. A Python library that can automatically create these diagrams is called `dnaplotlib`. This library can be used to specify the circuit design and it renders the diagram. In this problem, you will be given one such diagram created using `dnaplotlib`. You are asked to show your understanding of these diagrams by writing the mechanistic interactions that this diagram describes and a simple mathematical model that quantifies that understanding.

(a) From the design diagram given below for expression of two genes, write the chemical reactions that model the system design.

(b) Write a differential equation model for this system and simulate this model numerically to plot YFP and RFP with time. Choose biologically relevant parameters that represent pA, RBS1, and RBS2.

Hint: You can write a phenomenological model by writing down the rates for the production and degradation of the two proteins. Think about how the two proteins are produced/degraded and how that affects the different parameter choices in your model.

Note that to solve this problem, you do not need to install `dnaplotlib` and generate the diagram again, but you are free to do so and explore this library to create biological circuit diagrams._____no_output_____

#  Uncomment to install
# !pip install dnaplotlib_____no_output_____import dnaplotlib as dpl

promoter_1 = {'type':'Promoter', 'name':'pA', 'fwd':True, 'opts': {'color':[0.38, 0.82, 0.32],
                                                                   'label':'pA', 'label_size':20,
                                                                   'label_y_offset':-8}}
rbs_1 = {'type':'RBS', 'name':'rbs_f1', 'fwd':True, 'opts': {'color':[0.58, 0.52, 0.12],
                                                            'label':'RBS1', 'label_size':10,
                                                            'label_y_offset':8}}
cds_1 = {'type':'CDS', 'name':'YFP',  'fwd':True, 'opts': {'color':[0.98, 0.82, 0.32],
                                                          'label':'YFP', 'label_size':20}}
rbs_2 = {'type':'RBS', 'name':'rbs_f2', 'fwd':True, 'opts': {'color':[0.58, 0.52, 0.12],
                                                            'label':'RBS2', 'label_size':10,
                                                            'label_y_offset':8}}
cds_2  = {'type':'CDS', 'name':'RFP',  'fwd':True, 'opts': {'color':[0.98, 0.02, 0.32],
                                                           'label':'RFP', 'label_size':20}}
term = {'type':'Terminator', 'name':'t0', 'fwd':True}
design1 = [promoter_1, rbs_1, cds_1, rbs_2, cds_2, term]

import matplotlib.pyplot as plt
%matplotlib inline
fig, ax = plt.subplots()
dpl.plot_sbol_designs([ax], [design1]);_____no_output_____# Solution_____no_output_____

## Feedback
Please write here the number of hours you needed to finish this problem set and if you have any other feedback on this course until now._____no_output_____Answer:_____no_output_____


# Notebook from saint1729/in-learning
Path: numpy-data-science-essential-training/Ex_Files_NumPy_Data_EssT/Exercise Files/Ch 4/04_01/Starting/Inline.ipynb

%matplotlib inline_____no_output_____import numpy as np
import matplotlib.pyplot as plt

mu, sigma = 100, 15
data_set = mu + sigma * np.random.randn(10000)

# the histogram of the data
n, bins, patches = plt.hist(data_set, 50, density=True, facecolor='g', alpha=0.75)

plt.xlabel('Smarts')
plt.ylabel('Probability')
plt.title('Histogram of IQ')
plt.text(60, .025, r'$\mu=100,\ \sigma=15$')
plt.axis([40, 160, 0, 0.03])
plt.grid(True)
plt.show()_____no_output_____

Available Colors:

code       color
________________________
'k'         black
'b'         blue
'c'         cyan
'g'         green
'm'         magenta
'r'         red
'w'         white
'y'         yellow

_____no_output_____
Available Markers:

<colgroup>
<col width="38%" />
<col width="62%" />
</colgroup>

marker
description

”.”
point

”,”
pixel

“o”
circle

“v”
triangle_down

“^”
triangle_up

“<”
triangle_left

“>”
triangle_right

“1”
tri_down

“2”
tri_up

“3”
tri_left

“4”
tri_right

“8”
octagon

“s”
square

“p”
pentagon

“*”
star

“h”
hexagon1

“H”
hexagon2

“+”
plus

“x”
x

“D”
diamond

“d”
thin_diamond

“|”
vline

“_”
hline

TICKLEFT
tickleft

TICKRIGHT
tickright

TICKUP
tickup

TICKDOWN
tickdown

CARETLEFT
caretleft

CARETRIGHT
caretright

CARETUP
caretup

CARETDOWN
caretdown

“None”
nothing

None
nothing

” “
nothing

“”
nothing

'$...$'
render the string using mathtext.

verts
a list of (x, y) pairs used for Path vertices.
The center of the marker is located at (0,0) and
the size is normalized.

path
a Path instance.

(numsides, style, angle)
see below

_____no_output_____


# Notebook from alexfmsu/pyquantum
Path: stuff/old/TC.ipynb

# TC
from PyQuantum.TC.Cavity import Cavity
from PyQuantum.TC.Hamiltonian import Hamiltonian

from PyQuantum.TC.WaveFunction import WaveFunction
from PyQuantum.TC.DensityMatrix import *

from PyQuantum.TC.Evolution import *_____no_output_____# Common
from PyQuantum.Common.LoadPackage import *
from PyQuantum.Common.STR import *

from PyQuantum.Common.Tools import mkdir
from PyQuantum.Common.Print import *
from PyQuantum.Common.PyPlot import PyPlot3D_____no_output_____import PyQuantum.TC.config as config

mkdir(config.path)_____no_output_____cavity = Cavity(n=config.n, wc=config.wc, wa=config.wa, g=config.g)

cavity.print()

print("T:", config.T)
print("nt:", config.nt)
print("dt:", config.dt)

print()_____no_output_____H = Hamiltonian(capacity=config.capacity, cavity=cavity)

H.print_states()

H.iprint_symb()

H.df_____no_output_____w_0 = WaveFunction(states=H.states, init_state=config.init_state) - \
    WaveFunction(states=H.states, init_state=config.init_state2)

w_0.normalize()

w_0.print()_____no_output_____run_w(w_0, H, dt=config.dt, nt=config.nt, config=config, fidelity_mode=True)_____no_output_____from PyQuantum.Common.PyPlot import PlotBuilder

plt = PlotBuilder()

plt.set_width(950)
plt.set_height(650)

# ---------------------------------------------- TITLE --------------------------------------------
title = ""
title += ""
title += "n = " + str(config.n)
if config.capacity - config.n > 0:
    title += "
" + str(config.capacity - config.n) + \
        " фотонов в полости"
# else:
# title += "
" + "empty cavity"

# title += "
atoms state: |Ψ0 i = |11...1>A0 |00...0>A1 |vakip" + \
#     str(config.init_state)
title += "
"
title += "
wc = " + wc_str(config.wc)
title += "
wa = " + wa_str(config.wa)
title += "
 g/hwc = " + str(config.g/config.wc)
title += "
"
title += "
"
title += ""

plt.set_title(title)
# ---------------------------------------------- TITLE --------------------------------------------

# ---------------------------------------------- SCALE --------------------------------------------
y_scale = 1

if config.T < 0.25 * config.mks:
    y_scale = 0.1
elif config.T <= 0.5 * config.mks:
    y_scale = 0.025
elif config.T == 0.5 * config.mks:
    y_scale = 0.01
elif config.T == 1 * config.mks:
    y_scale = 7.5
    # y_scale = 10
elif config.T == 5 * config.mks:
    y_scale = 1

plt.set_yscale(y_scale)

plt.set_xaxis("states")
plt.set_yaxis("time, " + T_str_mark(config.T))
plt.set_zaxis("prob.\t\t\t\t\t\t.")
# ---------------------------------------------- SCALE --------------------------------------------

# -------------------------------------------------------------------------------------------------
plt.PyPlot3D(
    x_csv=config.path + "/" + "x.csv",
    y_csv=config.path + "/" + "t.csv",
    z_csv=config.path + "/" + "z.csv",
    # t_coeff=20000 / 1000 * (config.T / 1e-6),
    online=False,
    path=config.path,
    filename="Bipartite",
)
# -------------------------------------------------------------------------------------------------_____no_output_____


# Notebook from pllim/notebooks
Path: notebooks/MAST/K2/K2_TPF/k2_tpf.ipynb

# Beginner: Read and Display A K2 Target Pixel File

This notebook tutorial demonstrates how to load and display the contents of a K2 target pixel (tp) file.  We will display the flux values from a given cadence, and identify the pixels used in the photometric aperture._____no_output_____

%matplotlib inline
from astropy.io import fits
import matplotlib.pyplot as plt
import numpy as np_____no_output_____

## Introduction

A target pixel file contains the raw and calibrated fluxes for the pixels downloaded at either 1-minute (short) or 30-minute (long) cadence.  It also contains information about the aperture, including which pixels were used to calculate the total flux, which pixels were used to estimate the background flux, and which pixels were used to calculate the flux weighted or pixel response function centroids.  The data shown here will be for the star TRAPPIST-1 (the standard readout is EPIC 246199087, a larger readout for this star was also obtained with a K2 ID of EPIC 200164267), which is known to host a system of seven Earth-sized planets.

This tutorial will refer to a couple TESS-related terms that we define here.
* Campaign = During the K2 mission, the Kepler telescope observed the sky in a given pointing along the ecliptic plane for approximately 80 days at a time. Each of these regions is referred to as a "Campaign", starting with Campaign 0 and ending with Campaign 19. There was also a special "Engineering" Campaign before Campaign 0 that lasted ~10 days.
* HDU = Header Data Unit.  A FITS file is made up of HDUs that contain data and metadata relating to the file. The first HDU is called the primary HDU, and anything that follows is considered an "extension", e.g., "the first FITS extension", "the second FITS extension", etc.
* BJD = Barycentric Julian Date, the Julian Date that has been corrected for differences in the Earth's position with respect to the Solar System center of mass.
* BKJD = Barycentric Kepler Julian Date, the timestamp measured in BJD, but offset by 2454833.0. I.e., BKJD = BJD - 2454833.0
* Cadence = The interval between flux measurements, either one minute for short cadence or 30 minutes for long cadence._____no_output_____## Obtaining The Target Pixel File

We will read the target pixel file from Campaign 12 using the MAST URL location.  So that we can get started with understanding the file contents without reviewing how to automatically search for and retrieve K2 files, we won't show how to search and retrieve K2 target pixel files in this tutorial._____no_output_____

# For the purposes of this tutorial, we just know the MAST URL location of the file we want to examine.
fits_file = "https://archive.stsci.edu/missions/k2/target_pixel_files/c12/246100000/99000/ktwo246199087-c12_lpd-targ.fits.gz"_____no_output_____

## Understanding The Target Pixel File Structure

The K2 target pixel files contain a primary HDU with metadata stored in the header.  The first extension HDU contains more metadata in the header and stores arrays of data in a binary FITS table, which include the timestamps, fluxes, and background fluxes for each cadence of the pixels read out and downloaded by the spacecraft.  The second extension HDU contains an image that stores the pixels that were read out, and records information such as which pixels were used in the optimal photometric aperture to create the SAP fluxes.  Let's examine the structure of the FITS file using the astropy.fits `info` function, which shows the FITS file format in more detail._____no_output_____

fits.info(fits_file)_____no_output_____

Let's examine the binary table in the first FITS extension, since that contains the arrays of timestamps and fluxes for each cadence.  We will use the astropy.fits `getdata` function to access the table from the first extension HDU, and then show the columns of the table.  We can see included in the table are columns for the timestamps in K2 BJD format (**TIME**), raw counts (**RAW_CNTS**), and calibrated fluxes (**FLUX**)._____no_output_____

fits.getdata(fits_file, ext=1).columns_____no_output_____

If you examine the structure of the data, the table is more complicated than just rows and columns of values.  You can see the dimensions of the raw counts, flux, flux errors, flux backgrounds, and flux background errors are 9x10 (the size of the pixel stamp read out for this particular target).  We'll show how to examine the fluxes from a particular cadence below._____no_output_____## Reading the timestamps and fluxes.

Now that we have the target pixel file, let's store the timestamps and fluxes for use later._____no_output_____

with fits.open(fits_file, mode="readonly") as hdulist:
    k2_bjds = hdulist[1].data['TIME']
    raw_counts = hdulist[1].data['RAW_CNTS']
    calibrated_fluxes = hdulist[1].data['FLUX']_____no_output_____

Let's examine one of these to see how Python stores the data._____no_output_____

print(type(calibrated_fluxes))
print(calibrated_fluxes.shape)_____no_output_____

So these data are stored as a numpy array.  The first dimension is the number of cadences, followed by the size of the readout.  Thus, accessing `calibrated_fluxes[4,:,:]` should return to us the 9x10 grid of pixels from the **fifth** cadence (remember Python is zero-indexed, so the first cadence is stored in the zeroth index, the second in the first index, etc.)_____no_output_____

calibrated_fluxes[4,:,:].shape_____no_output_____

## Show the calibrated pixel values.

Let's display the calibrated fluxes for one of the cadences in our target pixel file.  For now, we'll pick the fifth cadence (index number 4).  You could of course decide to look at multiple cadences by looping over each index in the first dimension of the structure (e.g., loop over `calibrated_fluxes[i,:,:]`).  You could even make an animated .gif of the fluxes and see how they change from cadence to cadence.  To get you started though, let's just display one of the cadence's flux values._____no_output_____

# Start figure and axis.
fig, ax = plt.subplots()
fig.set_size_inches(12., 8.)

# Display the calibrated fluxes as an image for the fifth cadence.
cax = ax.imshow(calibrated_fluxes[4,:,:], cmap=plt.cm.YlGnBu_r, origin="lower")

# Let's define a title for the figure.
fig.suptitle("TRAPPIST-1 Calibrated Fluxes - Campaign 12, Fifth Cadence")

# Add a color bar.
cbar = fig.colorbar(cax)
plt.show()_____no_output_____

## Displaying The Aperture Pixel Information

Let's read in the second FITS extension HDU to display the aperture information.  First, let's read in the aperture pixels from the HDU._____no_output_____

with fits.open(fits_file, mode="readonly") as hdulist:
    aperture = hdulist[2].data_____no_output_____

Let's plot the pixels as an image._____no_output_____

# Start figure and axis.
fig, ax = plt.subplots()
fig.set_size_inches(12., 8.)

# Display the pixels as an image.
cax = ax.imshow(aperture, cmap=plt.cm.YlGnBu_r, origin="lower")

# Add a color bar.
cbar = fig.colorbar(cax)

# Add a title to the plot.
fig.suptitle("TRAPPIST-1 Aperture - Campaign 12")
plt.show()_____no_output_____

## Understanding The Aperture Pixel Values

We see the pixel values are integers, but what do they mean?  The pixels are bitmasks that encode information about each pixel.  You can find a summary of what the different values mean [here](https://archive.stsci.edu/kepler/manuals/archive_manual.pdf#page=20).  For example, a pixel in the aperture that might have a value of 15 can be broken down into power of 2 like: 8+4+2+1 = 15.  Referencing the table of values, this means this particular pixel was in was used to calculate the Pixel Response Function (PRF) centroid, was used to calculate the flux weighted centroid, was part of the optimal photometric aperture, and was collected by the spacecraft.  Numpy has a bulit-in function that can convert an integer into a binary bit mask.  Let's use that now one one of the common values we see in our displayed image above._____no_output_____

# Break down a pixel value of 3 (yellow pixels displayed above) into its constituent bits.
bits = np.binary_repr(3)
print(bits)_____no_output_____

Binary bits start from the right and end at the left, so the bit farthest on the right is the Least Significant Bit (LSB, equal to 2^0 = 1), the second from the right is 2^1=2, the third from the right is 2^2=4, etc. This bit mask has bit values of 1 (2^0) and 2 (2^1) set. From our look-up table, these values mean the pixels have been: collected by the spacecraft (value = 1) and used in the optimal photometric aperture (value = 2), so these pixels were used from this cadence to create the flux measurement in the mission-produced light curve files._____no_output_____## Marking All Pixels Used In The Photometric Aperture

Let's display the pixels in the aperture FITS extension again, but this time, let's only mark those that are used in the optimal photometric aperture (bit mask value of 2 set).  While it is possible to do this in fewer lines, we'll show more of the steps involved for clarity._____no_output_____

# Create an array that will keep track of which pixels have a bitmask
# value of 2 set. To start, everything is set to 0 = not set.  We make it
# the same dimension as the aperture pixel table, for clarity.
bitmask2_set = np.zeros(aperture.shape)

# Identify which pixels have the bit mask value of 2 set.  We'll first
# loop over each row (i) in the 9x10 table, and then each column (j).
for i, row in enumerate(aperture):
    for j, pix in enumerate(row):

        # Get the bitmask as a string of zeros and ones.
        this_bitmask = np.binary_repr(pix)

        # Is the bitmask value of 2 set?  If so, the 2nd integer from the
        # right should be set to 1.  We'll use Python negative indexes to
        # access the second integer from the right. But, we also need to
        # check that the length of the returned string is at least two!
        # If the pixel mask is 0 or 1 then it won't even have two digits
        # to check, and would result in an error.
        if len(this_bitmask) > 1:
            if this_bitmask[-2] == '1':
                # Then record this pixel as having a bitmask value of 2 set.
                bitmask2_set[i,j] = 1_____no_output_____

Now we can display the aperture, but this time we will only display the bitmask2_set table we've created, where a zero means the bitmask value of 2 is NOT set (thus these pixels aren't used in the optimal photometric aperture) and a one means the bitmask value of 2 is set (and thus these pixels were used in the optimal photometric aperture)._____no_output_____

# Start figure and axis.
fig, ax = plt.subplots()
fig.set_size_inches(12., 8.)

# Display, as an image, the 11x11 table that records the bitmask value of 2 being set.
cax = ax.imshow(bitmask2_set, cmap=plt.cm.YlGnBu_r, origin="lower")

# Add a color bar.
cbar = fig.colorbar(cax)

# Add a title to the plot.
fig.suptitle("TRAPPIST-1 Aperture - Campaign 12 - Pixels Used In Phot. Ap.")
plt.show()_____no_output_____

## About this Notebook

**Author:** Scott W. Fleming, STScI Archive Scientist

**Updated On:** 2019-02-01_____no_output_____[Top of Page](#title_ID)
 _____no_output_____