# Notebook from ahalfpen727/-Downloads-Genomic-Analyses-w-Python
Path: Pandas/Basic Pandas.ipynb

# Basic Pandas
#### A short workshop run by the Library Bioinformatics Service
Based on the Data Carpentry curriculum for [Data Visualization in Python](http://www.datacarpentry.org/python-ecology-lesson/)
( © Data Carpentry under [Creative Commons Attribution
license](https://creativecommons.org/licenses/by/4.0/) )

## The Basics of Python
Python is a general purpose programming language that supports rapid development of scripts and applications.

Python’s main advantages:

* Open Source software, supported by Python Software Foundation
* Available on all platforms
* It is a general-purpose programming language
* Supports multiple programming paradigms
* Very large community with a rich ecosystem of third-party packages_____no_output_____## LOADING THIS JUPYTER NOTEBOOK
The following instructions will show you how to download and open this jupyter notebook interactively, with the notes and exercises for the introductory workshop.

1. Download the notebook from  https://github.com/oxpeter/library_bioinformatics_service/archive/master.zip
2. Unzip the folder that downloads

3. Download Anaconda from https://www.anaconda.com/download/  (select your operating system, then click 'Download'
4. Run the Anaconda installer you just downloaded
5. Open Anaconda
6. Open Jupyterlab (click 'Launch')

7. In Jupyterlab, navigate to where you downloaded the notebook folder
8. Navigate to "library_bioinformatics_service-master > Pandas"
9. Double click on "Basic Pandas.ipynb"

10. The notebook should now load!_____no_output_____## Using Python in Jupyterlab
This Jupyter notebook has been built to run on a python 3 'kernel'. That means you can enter python code into the code cells, and it will be able to execute as python.

To execute code, select the cell(s), then either:
    * Choose "Run Selected Cells" from the Command tab on the left
    * Click the run button (triangle) at the top of the notebook
    * Press "Shift + Enter"

Try to execute the code in the cell below:_____no_output_____

# Like this first line, anything following a hash character (for the rest of that line) is considered a comment, and won't be run as code
text_str = "Congratulations, you've just run some Python code!"
print(text_str)_____no_output_____

You will notice that the output of the cell is placed immediately underneath the cell, and that a number appears to the left of the cell to indicate the order in which the cell was run :

```In [1]:```

You can modify and execute each cell in this notebook as often as you wish, in any order that you wish. Any variables you create in one cell are available for use in another. For example:_____no_output_____

print(text_str)_____no_output_____

You should also notice, that if you modify the variable in one cell, it will only update in another cell when you re-run it.
> ### Challenge question - running Jupyterlab
> Try to modify the ```text_str``` variable in the above cell, then re-execute the cells to see how they update._____no_output_____## Introduction to Python built-in data types

### Strings, integers and floats

One of the most basic things we can do in Python is assign values to variables:_____no_output_____

text = "Weill Cornell Medicine"     # An example of a string
number = 42                         # An example of an integer
pi_value = 3.1415                   # An example of a float_____no_output_____

Here we've assigned data to the variables `text`, `number` and `pi_value`,
using the assignment operator `=`. To review the value of a variable, we
can type the name of the variable into the cell and press `Shift Enter`:_____no_output_____

text_____no_output_____

Everything in Python has a type. To get the type of something, we can pass it
to the built-in function `type`:_____no_output_____

type(text)_____no_output_____type(number)_____no_output_____type(6.02)_____no_output_____

The variable `text` is of type `str`, short for "string". Strings hold
sequences of characters, which can be letters, numbers, punctuation
or more exotic forms of text (even emoji!)._____no_output_____### Operators

We can perform mathematical calculations in Python using the basic operators
 `+, -, /, *, %`:_____no_output_____

2 + 2  # Addition_____no_output_____6 * 7  # Multiplication_____no_output_____2 ** 16  # Power_____no_output_____13 % 5  # Modulo_____no_output_____

We can also use comparison and logic operators:
`<, >, ==, !=, <=, >=` and statements of identity such as
`and, or, not`. The data type returned by this is
called a _boolean_._____no_output_____

3 > 4_____no_output_____True and True_____no_output_____True or False_____no_output_____

## Sequential types: Lists and Tuples

### Lists

**Lists** are a common data structure to hold an ordered sequence of
elements. Each element can be accessed by an index.  Note that Python
indexes start with 0 instead of 1:_____no_output_____

numbers = [1, 2, 3]
numbers[0]_____no_output_____

A `for` loop can be used to access the elements in a list or other Python data
structure one at a time:_____no_output_____

for num in numbers:
    print(num)_____no_output_____

**Indentation** is very important in Python. Note that the second line in the
example above is indented. You can set an indent either by pressing the `tab` key, or by entering three spaces._____no_output_____To find out what methods are available for an object in jupyterlab, press `shift + tab` when the cursor is at the end of the object (or inside the parentheses, for a function)_____no_output_____### Tuples

A tuple is similar to a list in that it's an ordered sequence of elements.
However, tuples can not be changed once created (they are "immutable"). Tuples
are created by placing comma-separated values inside parentheses `()`._____no_output_____

# Tuples use parentheses
a_tuple= (1, 2, 3)
another_tuple = ('blue', 'green', 'red')

# Note: lists use square brackets
a_list = [1, 2, 3]_____no_output_____

> ## Challenge - Tuples
> 1. What happens when you type `a_tuple[2]=5` vs `a_list[1]=5` ?
> 2. Type `type(a_tuple)` into python - what is the object type?_____no_output_____

# 1:
_____no_output_____# 2:
_____no_output_____

## Dictionaries

A **dictionary** is a container that holds pairs of objects - keys and values._____no_output_____

translation = {'one': 1, 'two': 2}
translation['one']_____no_output_____

Dictionaries work a lot like lists - except that you index them with *keys*.
You can think about a key as a name for or a unique identifier for a set of values
in the dictionary. Keys can only have particular types - they have to be
"hashable". Strings and numeric types are acceptable, but lists aren't._____no_output_____

rev = {1: 'one', 2: 'two'}
rev[1]_____no_output_____bad = {[1, 2, 3]: 3}_____no_output_____

In Python, a "Traceback" is a multi-line error block printed out for the
user.

To add an item to the dictionary we assign a value to a new key:_____no_output_____

rev = {1: 'one', 2: 'two'}
rev[3] = 'three'
rev_____no_output_____

Using `for` loops with dictionaries is a little more complicated. We can do
this in two ways:_____no_output_____

for key, value in rev.items():
   print(key, '->', value)_____no_output_____

or_____no_output_____

for key in rev.keys():
    print(key, '->', rev[key])_____no_output_____

> ## Challenge - Can you do reassignment in a dictionary?
>
> 1. First create a dictionary `challenge_dict` with the following key:value pairs:
>    * 1 : 'uno'
>    * 2 : 'dos'
>    * 3 : 'tres'
> 2. Display the keys and values in the dictionary
> 2. Try to reassign the second value (in the *key value pair*) so that it no longer reads "dos" but instead reads "apple-sauce".
>
> 3. Now display your dictionary again to see if it has changed.
>_____no_output_____

# 1
challenge_dict = _____no_output_____# 2
_____no_output_____# 3
_____no_output_____# 4
_____no_output_____

It is important to note that dictionaries are "unordered" and do not remember
the sequence of their items (i.e. the order in which key:value pairs were
added to the dictionary). Because of this, the order in which items are
returned from loops over dictionaries might appear random and can even change
with time._____no_output_____---_____no_output_____# Working With Pandas DataFrames in Python

We can automate the process of performing data manipulations in Python. It's efficient to spend time
building the code to perform these tasks because once it's built, we can use it
over and over on different datasets that use a similar format. This makes our
methods easily reproducible. We can also easily share our code with colleagues
and they can replicate the same analysis._____no_output_____### Our Data

For this lesson, we will be using NCD Risk Factor Collaboration (NDC-RisC)  data, from `Worldwide trends in body-mass index, underweight, overweight, and obesity from 1975 to 2016: a pooled analysis of 2416 population-based measurement studies in 128.9 million children, adolescents, and adults.` [Lancet 2017, published online 11 October 2017](http://www.ncdrisc.org/index.html)

We will be using the adult BMI dataset [country-specific data](http://www.ncdrisc.org/downloads/bmi/NCD_RisC_Lancet_2017_BMI_age_standardised_country.csv).

The downloaded filename is `NCD_RisC_Lancet_2017_BMI_age_standardised_country.csv`, but we have saved a simplified version of the file here as `NCD_RisC_bmi.csv`.

We are studying the body-mass index (BMI) of individuals from 200 countries over a number of years. The dataset is stored as a `.csv` file: each row holds information for a single measurement period, and the columns represent:

| Column           | Description                        |
|------------------|------------------------------------|
| country          | Country                            |
| iso              | Country code                       |
| sex              | Gender of the group                |
| year             | Year measurements were taken       |
| bmi              | Mean BMI                           |
| lower95ci        | lower 95% confidence interval      |
| upper95ci        | upper 95% confidence interval      |
|obese_pc          | prevalance of BMI>30 (obese)       |
|obese_lower95ci   | lower 95% confidence interval      |
|obese_upper95ci   | upper 95% confidence interval      |_____no_output_____The first few rows of our first file look like this:

```
country,iso,sex,year,bmi,lower95ci,upper95ci,obese_pc,obese_lower95ci,obese_upper95ci
Afghanistan,AFG,Men,1975,18.9994401470858,16.3311193806968,21.7630768066895,0.00207754471275783,0.000383525249039564,0.00651548709746884
Afghanistan,AFG,Men,1976,19.1055182251737,16.5000772402149,21.8011089747524,0.00223682902494332,0.000437170207444467,0.0068203531416254105
Afghanistan,AFG,Men,1977,19.2120802193391,16.6465068023822,21.8420206008889,0.0024101143094182,0.000497358014754166,0.00717898800855567
Afghanistan,AFG,Men,1978,19.3190525301356,16.7868986048326,21.8765159552146,0.00259867365703349,0.000558764696829654,0.007567109958970071
```_____no_output_____---

## About Libraries
A library in Python contains a set of tools (called functions) that perform
tasks on our data. Importing a library is like getting a piece of lab equipment
out of a storage locker and setting it up on the bench for use in a project.
Once a library is set up, it can be used or called to perform many tasks.

## Pandas in Python
One of the best options for working with tabular data in Python is to use the
[Python Data Analysis Library](http://pandas.pydata.org/) (a.k.a. Pandas). The
Pandas library provides data structures, produces high quality plots with
[matplotlib](http://matplotlib.org/) and integrates nicely with other libraries
that use [NumPy](http://www.numpy.org/) (which is another Python library) arrays.

Python doesn't load all of the libraries available to it by default. We have to
add an `import` statement to our code in order to use library functions. To import
a library, we use the syntax `import libraryName`. If we want to give the
library a nickname to shorten the command, we can add `as nickNameHere`.  An
example of importing the pandas library using the common nickname `pd` is below._____no_output_____

import pandas as pd_____no_output_____

Each time we call a function that's in a library, we use the syntax
`LibraryName.FunctionName`. Adding the library name with a `.` before the
function name tells Python where to find the function. In the example above, we
have imported Pandas as `pd`. This means we don't have to type out `pandas` each
time we call a Pandas function.

# Reading CSV Data Using Pandas

We will begin by locating and reading our survey data which are in CSV format. CSV stands for Comma-Separated Values and is a common way store formatted data. Other symbols my also be used, so you might see tab-separated, colon-separated or space separated files. It is quite easy to replace one separator with another, to match your application. The first line in the file often has headers to explain what is in each column. CSV (and other separators) make it easy to share data, and can be imported and exported from many applications, including Microsoft Excel. For more details on CSV files, see the [Data Organisation in Spreadsheets](http://www.datacarpentry.org/spreadsheet-ecology-lesson/05-exporting-data/) lesson.
We can use Pandas' `read_csv` function to pull the file directly into a
[DataFrame](http://pandas.pydata.org/pandas-docs/stable/dsintro.html#dataframe).

## So What's a DataFrame?

A DataFrame is a 2-dimensional data structure that can store data of different
types (including characters, integers, floating point values, factors and more)
in columns. It is similar to a spreadsheet or an SQL table or the `data.frame` in
R. A DataFrame always has an index (0-based). An index refers to the position of
an element in the data structure._____no_output_____

# Note that pd.read_csv is used because we imported pandas as pd
pd.read_csv("data/NCD_RisC_bmi.csv")_____no_output_____

We can see that there were 16,800 rows parsed. Each row has 10
columns. The first column is the index of the DataFrame. The index is used to
identify the position of the data, but it is not an actual column of the DataFrame.
It looks like  the `read_csv` function in Pandas  read our file properly. However,
we haven't saved any data to memory so we can work with it. We need to assign the
DataFrame to a variable. Remember that a variable is a name for a value, such as `x`,
or  `data`. We can create a new  object with a variable name by assigning a value to it using `=`.

Let's call the imported survey data `df_surveys`:_____no_output_____

df_surveys = pd.read_csv("data/NCD_RisC_bmi.csv")_____no_output_____

Notice when you assign the imported DataFrame to a variable, Python does not
produce any output on the screen. We can view the value of the `df_surveys`
object by typing its name into the Python command prompt._____no_output_____

df_surveys_____no_output_____

which prints contents like above.

Note: if the output is too wide to print on your narrow terminal window, you may see something
slightly different as the large set of data scrolls past. You may see simply the last column
of data:
```python
17        NaN
18        NaN
19        NaN
20        NaN
21        NaN
22        NaN
23        NaN
24        NaN
25        NaN
26        NaN
27        NaN
28        NaN
29        NaN
...       ...
35519    36.0
35520    48.0
35521    45.0
35522    44.0
35523    27.0
35524    26.0
35525    24.0
35526    43.0
35527     NaN
35528    25.0
35529     NaN
35530     NaN
35531    43.0
35532    48.0
35533    56.0
35534    53.0
35535    42.0
35536    46.0
35537    31.0
35538    68.0
35539    23.0
35540    31.0
35541    29.0
35542    34.0
35543     NaN
35544     NaN
35545     NaN
35546    14.0
35547    51.0
35548     NaN

[35549 rows x 9 columns]
```
Never fear, all the data is there, if you scroll up. Selecting just a few rows, so it is
easier to fit on one window, you can see that pandas has neatly formatted the data to fit
our screen:_____no_output_____

df_surveys.head() # The head() function displays the first several lines of a file. It is discussed below._____no_output_____

## Exploring Our Obesity Survey Data

Again, we can use the `type` function to see what kind of thing `df_surveys` is:_____no_output_____

type(df_surveys)_____no_output_____

As expected, it's a DataFrame (or, to use the full name that Python uses to refer
to it internally, a `pandas.core.frame.DataFrame`).

What kind of things does `df_surveys` contain? DataFrames have an attribute
called `dtypes` that answers this:_____no_output_____

df_surveys.dtypes_____no_output_____

All the values in a column have the same type. For example, year has type
`int64`, which is a kind of integer. Cells in the year column cannot have
fractional values, but the bmi and obese_pc columns can, because they
have type `float64`. The `object` type doesn't have a very helpful name, but in
this case it represents strings (such as 'Men' and 'Women' in the case of sex).

We'll talk a bit more about what the different formats mean in a different lesson.

### Useful Ways to View DataFrame objects in Python

There are many ways to summarize and access the data stored in DataFrames,
using attributes and methods provided by the DataFrame object.

To access an attribute, use the DataFrame object name followed by the attribute
name `df_object.attribute`. Using the DataFrame `df_surveys` and attribute
`columns`, an index of all the column names in the DataFrame can be accessed
with `df_surveys.columns`._____no_output_____

df_surveys.columns_____no_output_____

Methods are called in a similar fashion using the syntax `df_object.method()`.
As an example, `df_surveys.head()` gets the first few rows in the DataFrame
`df_surveys` using **the `head()` method**. With a method, we can supply extra
information in the parentheses to control behaviour.

Let's look at the data using these.

> ## Challenge - DataFrames
>
> Using our DataFrame `df_surveys`, try out the attributes & methods below to see
> what they return.
>
> 1. `df_surveys.columns`
> 2. `df_surveys.shape` Take note of the output of `shape` - what format does it
>    return the shape of the DataFrame in?
>
>    HINT: [More on tuples, here](https://docs.python.org/3/tutorial/datastructures.html#tuples-and-sequences).
> 3. `df_surveys.head()` Also, what does `df_surveys.head(15)` do?
> 4. `df_surveys.tail()`
_____no_output_____## Calculating Statistics From Data In A Pandas DataFrame

We've read our data into Python. Next, let's perform some quick summary
statistics to learn more about the data that we're working with. We can perform summary stats quickly using groups. But
first we need to figure out what we want to group by.

Let's begin by exploring our data:_____no_output_____

# Look at the column names
df_surveys.columns_____no_output_____

Let's get a list of all the years. The `pd.unique` function tells us all of
the unique values in the `year` column._____no_output_____

pd.unique(df_surveys['year'])_____no_output_____

> ## Challenge - Statistics
>
> 1. Create a list of unique countries found in the surveys data. Call it
>   `country_names`. How many unique countries are there in the data? How many unique
>   years are in the data?
>
> 2. What is the difference between `len(country_names)` and `df_surveys['country'].nunique()`?_____no_output_____# Groups in Pandas

We often want to calculate summary statistics grouped by subsets or attributes
within fields of our data. For example, we might want to calculate the average
bmi of all countries per year.

We can calculate basic statistics for all records in a single column using the
syntax below:_____no_output_____

df_surveys['bmi'].describe()_____no_output_____

We can also extract one specific metric if we wish:_____no_output_____

df_surveys['bmi'].min()_____no_output_____df_surveys['bmi'].max()_____no_output_____df_surveys['bmi'].mean()_____no_output_____df_surveys['bmi'].std()_____no_output_____df_surveys['bmi'].count()_____no_output_____

But if we want to summarize by one or more variables, for example sex, we can
use **Pandas' `.groupby` method**. Once we've created a groupby DataFrame, we
can quickly calculate summary statistics by a group of our choice._____no_output_____

# Group data by sex
grouped_data = df_surveys.groupby('sex')_____no_output_____

The **pandas function `describe`** will return descriptive stats including: mean,
median, max, min, std and count for a particular column in the data. Pandas'
`describe` function will only return summary values for columns containing
numeric data._____no_output_____

# Summary statistics for all numeric columns by sex
grouped_data.describe()_____no_output_____# Provide the mean for each numeric column by sex
grouped_data.mean()_____no_output_____

The `groupby` command is powerful in that it allows us to quickly generate
summary stats.

> ## Challenge - Summary Data
>
> 1. How many recorded individuals are women and how many are men?
> 2. What happens when you group by two columns using the following syntax and
>    then grab mean values:
>	- `grouped_data2 = df_surveys.groupby(['country','sex'])`
>	- `grouped_data2.mean()`
> 3. Summarize bmi values for each country in your data. HINT: you can use the
>   following syntax to only create summary statistics for one column in your data
>   `by_country['bmi'].describe()`
>_____no_output_____## Quickly Creating Summary Counts in Pandas

Let's next count the number of years measured for each country. We can do this in a few
ways, but we'll use `groupby` combined with **a `count()` method**._____no_output_____

# Count the number of samples by species
year_counts = df_surveys.groupby('country')['year'].count()
print(year_counts)_____no_output_____

Or, we can also count just the rows with the USA:_____no_output_____

df_surveys.groupby('country')['year'].count()['United States of America']_____no_output_____

> ## Challenge - Make a list
>
>  What's another way to create a list of countries and an associated `count` of the
>  records in the data? Hint: you can perform `count`, `min`, etc functions on
>  groupby DataFrames in the same way you can perform them on regular DataFrames.
_____no_output_____## Basic Math Functions

If we wanted to, we could perform math on an entire column of our data. For
example let's multiply all weight values by 2. A more practical use of this might
be to normalize the data according to a mean, area, or some other value
calculated from our data._____no_output_____

# Multiply all weight values by 2
df_surveys_doubled = df_surveys['bmi'] * 2_____no_output_____

# Quick & Easy Plotting Data Using Pandas

We can plot our summary stats using Pandas, too._____no_output_____

# Make sure figures appear inline in Jupyter Notebook
%matplotlib inline_____no_output_____# Create a quick bar chart
year_bmi = df_surveys.groupby("year").mean()["bmi"]
year_bmi.plot(kind='bar');_____no_output_____

We can also look at how many countries were measured in each year:_____no_output_____

total_count = df_surveys.groupby('year')['country'].nunique()
# Let's plot that too
total_count.plot(kind='bar');_____no_output_____

(Perhaps a boring graph, but a good way to check that each year is consistent)_____no_output_____> ## Challenge - Plots
>
> 1. Create a plot of average bmi across all years per country. (to resize your graph, try using the keyword `figsize=(x-inches,y-inches)`). Can you determine the highest average bmi is for a country? What is the lowest? To sort a dataframe by the values in a particular column, you can use the df.sort_values( *column_name* )
> 2. Create a plot of average male versus average female obesity prevalence for the entire dataset._____no_output_____

df_surveys.groupby("country").mean()['bmi'].sort_values()_____no_output_____

> ## Summary Plotting Challenge
>
> Create a stacked bar plot, with obesity prevalence on the Y axis, and the stacked variable
> being sex. The plot should show mean obesity prevalence by sex for each year. Some
> tips are below to help you solve this challenge:
>
> * [For more on Pandas plots, visit this link.](http://pandas.pydata.org/pandas-docs/stable/visualization.html#basic-plotting-plot)
> * You can use the code that follows to create a stacked bar plot but the data to stack
>  need to be in individual columns.  Here's a simple example with some data where
>  'a', 'b', and 'c' are the groups, and 'one' and 'two' are the subgroups.
>_____no_output_____

d = {'one' : pd.Series([1., 2., 3.], index=['a', 'b', 'c']),'two' : pd.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}
pd.DataFrame(d)_____no_output_____

> We can plot the above with_____no_output_____

# Plot stacked data so columns 'one' and 'two' are stacked
my_df = pd.DataFrame(d)
my_df.plot(kind='bar',stacked=True,title="The title of my graph")_____no_output_____

> * You can use the `.unstack()` method to transform grouped data into columns
> for each plotting.  Try running `.unstack()` on some DataFrames above and see
> what it yields.
>
> Start by transforming the grouped data (by plot and sex) into an unstacked layout, then create
> a stacked plot.                                             _____no_output_____> ## Your solution:_____no_output_____>> ## Solution to Summary Challenge_____no_output_____>>
>> First we group data by year and by sex, and then calculate a mean for each year._____no_output_____

by_year_sex = df_surveys.groupby(['year','sex'])
year_sex_mean = by_year_sex['obese_pc'].mean()_____no_output_____

>> This calculates the mean of bmis for each sex within each year as a table_____no_output_____

year_sex_mean.head()_____no_output_____

>>
>> Below we'll use `.unstack()` on our grouped data to figure out the BMI that each sex was measured at each year.
>>_____no_output_____

spc = year_sex_mean.unstack()
spc_____no_output_____

>> Now, create a stacked bar plot with that data where the BMi for each sex are stacked by year.
>>
>> Rather than display it as a table, we can plot the above data by stacking the values of each sex as follows:
>>_____no_output_____

s_plot = spc.plot(kind='bar',stacked=True,title="Mean BMI by year and sex")

s_plot.set_ylabel("Prevalence of obesity")
s_plot.set_xlabel("Year")_____no_output_____


# Notebook from meeseeksmachine/bqplot
Path: examples/Applications/Logs Analytics/Logs Dashboard.ipynb

## Logs Analytics Dashboard
This dashboard can be used for performing visual analytics on the server log files. You can interact with any chart in the dashboard, like so:

1. Click on a bar in any bar chart to filter the whole dataset by that bar's value
2. Click on a slice of any pie chart to filter the whole dataset by that pie slices value

Fun things to try:
1. Select any specific day in the `Daily Events` bar chart (for e.g. why do first and last days have fewer events?)
2. Find out which product queries resulted in server error (click on `Server Error` slice of `Events By Status` pie chart)
_____no_output_____

<code>
import numpy as np
import pandas as pd

import ipywidgets as widgets
import bqplot as bq
import bqplot.pyplot as plt_____no_output_____def get_status_code(x):
    """
    map integer http status to  string status code
    """
    if x >= 200 and x < 300:
        return 'SUCCESS'
    elif x >= 300 and x < 400:
        return 'REDIRECT'
    elif x >= 400 and x < 500:
        return 'CLIENT ERROR'
    elif x >= 500:
        return 'SERVER ERROR'_____no_output_____def get_events_by_hour(log_data_slice):
    """
    get event counts by hour
    """
    return log_data_slice['status_code']\
    .groupby(lambda x: x.hour)\
    .count()\
    .reindex(np.arange(24))\
    .fillna(0)_____no_output_____# load data into pandas and do some munging
log_data = pd.read_csv('access.log', sep=' ', header=None)
log_data.drop([1, 2], axis=1, inplace=True)
log_data.columns = ['ip_address', 'timestamp', 'request', 'status', 'col1', 'url', 'agent', 'col2']

log_data.index = pd.to_datetime(log_data['timestamp'].str.replace('[\[\]]', ''),
                                format='%d/%b/%Y:%H:%M:%S')
log_data.drop('timestamp', axis=1, inplace=True)

# add extra columns for easy querying
log_data['status_code'] = log_data['status'].map(get_status_code)
log_data['category'] = log_data['request'].str.extract('categoryId=(.*)&', expand=False)
log_data['product'] = log_data['request'].str.extract('productId=(.*)&', expand=False)
log_data['hour'] = log_data.index.map(lambda x: x.hour)
log_data['day'] = log_data.index.strftime('%Y-%m-%d')_____no_output_____all_days = log_data['day'].value_counts().index
all_hours = np.arange(24)
all_categories = list(log_data['category'].value_counts().index)
all_products = list(log_data['product'].value_counts().index)
all_status_codes = list(log_data['status_code'].value_counts().index)

category_colors = dict(zip(all_categories, bq.CATEGORY10))
status_label_colors = dict([('SUCCESS', '#006d2c'),
                            ('CLIENT ERROR', '#fc8d59'),
                            ('SERVER ERROR', '#a63603')])_____no_output_____# daily events bar chart
daily_events_fig = plt.figure(title='Daily Events',
                              animation_duration=1000,
                              layout=widgets.Layout(width='900px', height='500px'))

plt.scales(scales={'x': bq.DateScale()})

# intsel = BrushIntervalSelector(scale=, marks=[hist])
bar_axes_options = {'x': {'grid_lines': 'none'},
                    'y': {'tick_format': ','}}
common_bar_options = dict(interactions={'click': 'select'},
                          selected_style={'stroke': 'Red',
                                          'stroke-width': 4},
                          axes_options=bar_axes_options)
daily_events_bar = plt.bar(pd.to_datetime(all_days), [],
                           colors=['dodgerblue'],
                           opacities=[.8] * len(all_days),
                           **common_bar_options)

num_days = len(all_days)
filtered_daily_events_bar = plt.bar(pd.to_datetime(all_days),
                                    np.zeros(num_days),
                                    colors=['lightgreen'],
                                    opacities=[.8] * num_days,
                                    axes_options=bar_axes_options)

# hourly events line chart
hourly_events_fig = plt.figure(title='Hourly Events',
                               animation_duration=1000,
                               layout=widgets.Layout(width='600px', height='500px'))

plt.scales(scales={'x': bq.OrdinalScale()})
hourly_events_bar = plt.bar(all_hours,[],
                            colors=['goldenrod'],
                            opacities=[.8] * 24,
                            padding=.2,
                            **common_bar_options)

filtered_hourly_events_bar = plt.bar(all_hours, [],
                                     colors=['lightgreen'],
                                     opacities=[.8] * 24,
                                     padding=.2,
                                     axes_options=bar_axes_options)

products_fig = plt.figure(title='Events By Product',
                          animation_duration=1000,
                          fig_margin=dict(top=60, bottom=20, left=100, right=40),
                          layout=widgets.Layout(width='550px', height='500px'))
products_bar = plt.bar(all_products, [],
                       colors=['salmon'],
                       orientation='horizontal',
                       opacities=[.8] * len(all_products),
                       padding=.2,
                       **common_bar_options)

common_pie_args = dict(display_labels='outside',
                       interactions={'click': 'select'},
                       selected_style={'stroke': 'white',
                                       'stroke-width': 3},
                       inner_radius=80,
                       apply_clip=False)
# categories pie chart
categories_fig = plt.figure(title='Events By Category', animation_duration=1000,
                           layout=widgets.Layout(width='550px', height='500px'))
categories_pie = plt.pie([], labels=all_categories, **common_pie_args)

# status codes pie chart
status_codes_fig = plt.figure(title='Events By Status', animation_duration=1000,
                              layout=widgets.Layout(width='550px', height='500px'))
status_codes_pie = plt.pie([], labels=all_status_codes, **common_pie_args)

# buttons for updating and resetting filters
update_btn = widgets.Button(description='Update', button_style='Success')
reset_btn = widgets.Button(description='Reset', button_style='Success')_____no_output_____fields = ['day', 'hour', 'product', 'category', 'status_code']
field_vals = [all_days, all_hours, all_products, all_categories, all_status_codes]
plots = [daily_events_bar, hourly_events_bar, products_bar, categories_pie, status_codes_pie]_____no_output_____def generate_filters():
    filters_dict = {}
    for field, field_val, plot in zip(fields, field_vals, plots):
        if plot.selected:
            selected_vals = [field_val[i] for i in plot.selected]
            filters_dict[field] = selected_vals
    return filters_dict_____no_output_____def apply_filters(df, filters):
    filtered_df = df

    for k, v in filters.items():
        if v and len(v) > 0:
            filtered_df = filtered_df[filtered_df[k].isin(v)]
    return filtered_df_____no_output_____def update_plots(*args):
    global filters, filtered_log_data
    filters = generate_filters()
    filtered_log_data = apply_filters(log_data, filters)

    daily_events = filtered_log_data\
        .resample('D')\
        .count()['status_code']\
        .reindex(pd.to_datetime(all_days))\
        .fillna(0)
    daily_events_bar.y = daily_events

    hourly_events_bar.y = get_events_by_hour(filtered_log_data)

    products_bar.y = filtered_log_data['product']\
        .value_counts()\
        .reindex(all_products)\
        .fillna(0)

    events_by_category = filtered_log_data['category'].value_counts()

    with categories_pie.hold_sync():
        categories_pie.labels = list(events_by_category.index)
        categories_pie.sizes = events_by_category
        categories_pie.colors = [category_colors[d] for d in categories_pie.labels]

    events_by_status_code = filtered_log_data['status_code'].value_counts()

    with status_codes_pie.hold_sync():
        status_codes_pie.labels = list(events_by_status_code.index)
        status_codes_pie.sizes = events_by_status_code.values
        status_codes_pie.colors = [status_label_colors[d] for d in status_codes_pie.labels]_____no_output_____def reset_filters(*args):
    for plot in plots:
        plot.selected = None
    update_plots(None)

update_btn.on_click(lambda btn: update_plots(None))
reset_btn.on_click(lambda btn: reset_filters())
btns_layout = widgets.VBox([update_btn, reset_btn], layout=widgets.Layout(overflow_x='hidden'))_____no_output_____update_plots(None)

parent_plots = widgets.HBox([daily_events_fig, hourly_events_fig])
child_plots = widgets.HBox([products_fig, categories_fig, status_codes_fig])
widgets.VBox([widgets.HBox([parent_plots, btns_layout]),child_plots])_____no_output_____
</code>


# Notebook from JulienPascal/KoopmanObservableSubspaces
Path: Koopman_Control_Python_PartII.ipynb

# The linear–quadratic regulator Part II_____no_output_____This notebook builds upon what has been described in Part I. In Part I, we introduced the linear–quadratic regulator (LQR) framework in Python. We solved the linearized control problem. In this notebook, we will see that we can do better. The basic idea is to follow the the evolution of "observables" — functions of the state space — instead of the evolution of the state itself using the **Koopman operator**. In the space of observables, the differential equation is linear. Thus, we can solve for the optimal control in the this transformed space, without having to linearize the system around its steady-state.

#### In this notebook, you will learn:

* the basics on the Koopman operator
* how to solve for the optimal control in the Koopman subspace

#### Acknowledgements

* The linear system studied in this notebook is based on the [paper](https://arxiv.org/abs/1510.03007): Brunton, Steven L., et al. "Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control." PloS one 11.2 (2016). For details and more references, I strongly invite you to read it.
_____no_output_____# I. The Koopman operator 101_____no_output_____Let us remember that in Part I, we analyzed the evolution of the following dynamical system:

$$ \begin{cases} \frac{d}{dt} x = \mu x \\ \frac{d}{dt} y = \lambda (y - x^2)  \end{cases} $$ _____no_output_____Let's make the observation that we can defined a new vector $z$ defined by:

$$ z \equiv \begin{pmatrix} z_1 \\ z_2 \\ z_3 \end{pmatrix} = \begin{pmatrix} x \\ y \\ x^2 \end{pmatrix}$$
_____no_output_____The original nonlinear dynamical system is linear when considering the evolution of $z$:

$$ \frac{d}{dt} \begin{pmatrix} z_1 \\ z_2 \\ z_3 \end{pmatrix} = \begin{pmatrix} \mu & 0 & 0 \\ 0 & \lambda & -\lambda \\ 0 & 0 & 2 \mu \end{pmatrix}  \begin{pmatrix} z_1 \\ z_2 \\ z_3 \end{pmatrix}  $$

    _____no_output_____What we just did — writing the evolution the dynamical system using some observables — is the central idea of the Koopman operator. Here we are lucky because our new variable $z$ is of finite dimension. In the general case, $z$ is infinite-dimensional:

$$ \frac{d}{dt} \begin{pmatrix} z_1 \\ z_2 \\ ...  \end{pmatrix} = \begin{pmatrix} a_{11} & a_{12} & ... \\ a_{21} & a_{22} & ... \\ ... & ... & ...  \end{pmatrix}  \begin{pmatrix} z_1 \\ z_2 \\ ... \end{pmatrix}  $$

and we have no guarantee that the truncation of z by a finite dimensional counterpart will produce a good approximation of the system. For more detail on this problem, I invite you to read [this paper](https://arxiv.org/abs/1510.03007)._____no_output_____Because now the dynamical system is linear, we can directly use the results from LQR framework and solve for the optimal control that minimizes the cost $J$:

$$ J = \int_{0}^{\infty} x'(t) Q x(t) + u'(t) R u(t) dt $$  _____no_output_____We already know that the optimal control from the controller is a linear function of the current state of the system:

$$ u = - C_z z$$
_____no_output_____When the system is controlled optimally, the equation governing the evolution of the system writes:

$$ \frac{d}{dt} z = A_z z - B C_z z $$_____no_output_____At a later stage, because we want to compare the Koopman controller to the linearized controller, we do not want to penalize the variable $z_3$ for being away from its steady state. The rationale is the variable $z_3$ is "fictive".
We still have in mind that we want to control the "real" variables $x$ and $y$. As a result, the matrix $Q$ we consider
is:

$$ \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix} $$

_____no_output_____## II. Simulating forward the dynamical system_____no_output_____For what follows, you will need the following packages:_____no_output_____

import os
import matplotlib.pyplot as plt
plt.style.use('ggplot')
%matplotlib inline
import numpy as np
from control.matlab import *  # MATLAB-like functions
#to solve ODE
from scipy import integrate
#show the version of Python I am using:
!python3 --versionPython 3.5.2

Let's study optimal control for the following differential equation:_____no_output_____$$ \frac{d}{dt} \begin{pmatrix} z_1 \\ z_2 \\ z_3 \end{pmatrix} = \begin{pmatrix} \mu & 0 & 0 \\ 0 & \lambda & -\lambda \\ 0 & 0 & 2 \mu \end{pmatrix}  \begin{pmatrix} z_1 \\ z_2 \\ z_3 \end{pmatrix}  $$

_____no_output_____The following block of code define the parameter values, the matrix A for the linear part of the differential equation and the matrix B specifying that the controller can only act on x. We also specify the time span during which we want to simulate forward the trajectories:_____no_output_____

# Parameters and matrices A and B:
mu = -0.05
llambda = -1.0
# Matrices for the orginial system
A = np.array([[mu, 0], [0, llambda]])
B = np.array([[0], [1]])
R = np.array([1])
Q = np.eye(2)
# Matrices for the transformed system:
A_z = np.array([[mu, 0, 0], [0, llambda, -llambda], [0, 0, 2*mu]])
B_z = np.array([[0], [1], [0]])
R_z = np.array([1])
Q_z = np.eye(3)
# Time span
t0, t1 = 0, 100 # start and end
t = np.arange(t0, t1, 0.01)

# Function that defines the dynamic system:
def vdp0(t, y):
    # linear part + nonlinear part:
    x = A.dot(y) + np.array([0, -llambda*(y[0]**2)])
    return x

# Function that defines the dynamic system in the Koopman subspace:
def vdp0z(t, y):
    x = A_z.dot(y)
    return x _____no_output_____

We then define 4 different starting values and simulate forward the system using the `scipy.integrate` toolkit:_____no_output_____

# Set of starting values:
y0A = np.array([1.5, -1, (1.5)**2])
y0B = np.array([-1.5, -1, (-1.5)**2])
y0C = np.array([5, 5, 25])
y0D = np.array([-5, 5, 25])

# To store the different trajectories
list_y_z = []

# Method for the ODE:
# This is an explicit runge-kutta method of order (4)5 due to Dormand & Prince
integrator = "dopri5"

# Loop over the different starting values and calculate trajectories:
for y0 in [y0A, y0B, y0C,y0D]:
    # initialize an array to store the solution
    y = np.zeros((len(t), len(y0)))   # array for solution
    r0 = integrate.ode(vdp0z).set_integrator(integrator)
    r0.set_initial_value(y0, t0)   # initial values
    for i in range(1, t.size):
       y[i, :] = r0.integrate(t[i]) # get one more value, add it to the array
       if not r0.successful():
           raise RuntimeError("Could not integrate")
    # append the list of solution
    list_y_z.append(y)_____no_output_____

The next graph shows the trajectory of x and y using the augmented linear system corresponds to the trajectory
we found using the nonlinear dynamical system, as expected:_____no_output_____

# Plot the different paths:
fig, ax = plt.subplots(figsize=(10, 5))

for index,y0 in enumerate([y0A, y0B, y0C, y0D]):
    ax.plot(list_y_z[index][1:-1, 0], list_y_z[index][1:-1, 1], label = str(y0))
    plt.xlabel("x")
    plt.ylabel("y")
plt.title("Trajectories for different starting values")
plt.legend()
plt.show()_____no_output_____

The next graph shows that the dynamical system moves along a nice parabola in the third dimension:_____no_output_____

# Plot the different paths:
fig, ax = plt.subplots(figsize=(10, 5))

for index,y0 in enumerate([y0A, y0B, y0C, y0D]):
    ax.plot(list_y_z[index][1:-1, 0], list_y_z[index][1:-1, 2], label = str(y0))
    plt.xlabel("x")
    plt.ylabel("z")
plt.title("Trajectories for different starting values")
plt.legend()
plt.show()_____no_output_____

## III. Solving for the optimal control _____no_output_____Let us remember that our aim is find the matrix $C_z$ defining the optimal control:

$$ u = - C_z z $$

Interestingly, while the optimal $u$ is linear when considering $z$, it is quadratic when considering the original vector of state $x$:

$$ u = - (C_{z,1} C_{z,2}) \begin{pmatrix} x \\ y  \end{pmatrix} -  C_{z,3} x^2  $$

_____no_output_____We will see in a minute that having a non-linear control allows us to outperform the linear system obtained
in Part I._____no_output_____

# Solve for C:
(C, X, E) = lqr(A, B, Q, R)
print("Feedback matrix C : {}".format(C))

# Solve for C_z:
(C_z, X_z, E_z) = lqr(A_z, B_z, Q_z, R_z)
print("Feedback matrix C_z : {}".format(C_z))Feedback matrix C : [[0.         0.41421356]]
Feedback matrix C_z : [[0.         0.41421356 0.27355029]]

We can now proceed as previously to simulate forward the differential equation. The only difference is that
now we have to take into account the optimal control applied each period on the system. The optimal control is
taken into consideration in the function `vdp1(t, y)`:_____no_output_____

def vdp1(t, y):
    # Ay - B*X*y + Cy
    x = A.dot(y)  - np.matmul(B,C).dot(y)
    return x + np.array([0, -llambda*(y[0]**2)])

def vdp1z(t, y):
    return A_z.dot(y)  - np.matmul(B_z,C_z).dot(y)

y0 = [-5, 5]                   # initial value
y0_z = [-5, 5, 25]                   # initial value

y = np.zeros((len(t), len(y0))) # array for solution
y_z = np.zeros((len(t), len(y0_z))) # array for solution
y[0, :] = y0
y_z[0, :] = y0_z_____no_output_____# Controlled trajectory using the linearized system
r = integrate.ode(vdp1).set_integrator(integrator)
r.set_initial_value(y0, t0)   # initial values

for i in range(1, t.size):
   y[i, :] = r.integrate(t[i]) # get one more value, add it to the array
   if not r.successful():
       raise RuntimeError("Could not integrate")

# Controlled trajectory using the linearized system
r = integrate.ode(vdp1z).set_integrator(integrator)
r.set_initial_value(y0_z, t0)   # initial values

for i in range(1, t.size):
   y_z[i, :] = r.integrate(t[i]) # get one more value, add it to the array
   if not r.successful():
       raise RuntimeError("Could not integrate")_____no_output_____

We can now compare the controlled and Koopman controlled trajectories. Interestingly, the Koopman controlled
trajectory moves along a trajectory involving lower values for $y$:_____no_output_____

index = 3 #choose the last trajectory
fig, ax = plt.subplots(figsize=(10, 5))
plt.plot(list_y_z[index][1:-1, 0], list_y_z[index][1:-1, 1], label = "Uncontrolled")
plt.plot(y[:, 0], y[:, 1], label = "Controlled")
plt.plot(y_z[:, 0], y_z[:, 1], label = "Koopman controlled")
plt.xlabel("x")
plt.ylabel("y")
plt.legend()
plt.show()_____no_output_____

The next logical question is whether or not the Koopman controller is better than the controller based on the
linearization of the dynamical system around its steady state. The next plot shows that indeed the Koopman controller outperforms the latter. _____no_output_____

# Controlled
JLQR = np.cumsum( np.square(y[:, 0]) + np.square(y[:, 1]) + np.square(np.matmul(C, y.T)) ).T
# Koopman controlled
JLQRz = np.cumsum( np.square(y_z[:, 0]) + np.square(y_z[:, 1]) + np.square(np.matmul(C_z, y_z.T)) ).T
# Uncontrolled
JLQR0 = np.cumsum( np.square(list_y_z[index][:, 0]) + np.square(list_y_z[index][:, 1]) )

fig, ax = plt.subplots(figsize=(10, 5))
plt.plot(t, JLQR0, label = "Uncontrolled")
plt.plot(t, JLQR, label = "Controlled")
plt.plot(t, JLQRz, label = "Koopman controlled")
plt.xlabel("t")
plt.ylabel("JLQR")
plt.legend()
plt.show()_____no_output_____

## Conclusion_____no_output_____This notebook illustrated the idea behind the Koopman operator in a very simple setting. We saw that by solving for the optimal control in the space of observables, in which the system is exactly linear, we find a controller that outperforms the one obtained by linearizing the system around its steady-state.

The Koopman LQR control drew my attention in the first place because many techniques, especially in Economics, are based on the linearization of dynamic systems around its steady-state. While these linearization techniques are accurate when the economy is close to its steady-state (in a "business as usual" situation), it is hard to know how they perform when the economy is far away from normal. This point is certainly relevant given the current economic context._____no_output_____## References

* The linear system studied in this notebook is based on the paper: Brunton, Steven L., et al. "Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control." PloS one 11.2 (2016).

* Koopman, Bernard O. "Hamiltonian systems and transformation in Hilbert space." Proceedings of the national academy of sciences of the united states of america 17.5 (1931): 315._____no_output_____


# Notebook from yookyungkho/Dacon2020_Guide-for-your-safe-homecoming
Path: 안전한 귀향을 위한 안내서.ipynb

# 안전한 귀향을 위한 안내서

아래 셀들을 전체 실행 하시고 마지막 셀에서 출발지와 도착지를 자유롭게 입력하시면 됩니다.

더욱 자세한 분석과 시각화 과정을 보고 싶으시다면 데이콘 사이트의 **[포스트 코로나, 귀향길이 걱정되는 당신을 위한 <안전한 귀향 안내서>](https://dacon.io/competitions/official/235622/codeshare/1621?page=2&dtype=recent&ptype=pub)** 를 참고해주시기 바랍니다._____no_output_____## 1. 전처리_____no_output_____### 1.0. 모듈 import_____no_output_____

import pandas as pd
import numpy as np
import datetime as dt
import matplotlib as mpl
from matplotlib import pyplot as plt
import seaborn as sns
import warnings
import folium
from folium import plugins
import plotly.express as px
import plotly.offline as pyo
import plotly.graph_objs as go
from plotly.subplots import make_subplots
from IPython.display import display, Image
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException,StaleElementReferenceException
import json
import time
warnings.filterwarnings(action='ignore')
warnings.filterwarnings("ignore", category=DeprecationWarning)

%matplotlib inline
plt.style.use('seaborn')
plt.rc('font', family='Malgun Gothic')
plt.rc('axes', unicode_minus=False)_____no_output_____#지도를 시각화하기 위해 다운받아야 할 모듈
!pip install git+https://github.com/python-visualization/branca.git@master_____no_output_____

### 1.1 . 발권OD.dat와 지역코드.dat 병합_____no_output_____

#발권OD 로드
bo = pd.read_csv("발권 OD.dat", sep='|', header=None)
columns = ["년도", "년월", "운행 일자", "요일 구분", "출발 시도 코드", "출발 시군구 코드", "출발 지역 코드", "도착 시도 코드", "도착 시군구 코드",
           "도착 지역 코드", "철도 통행 인원", "시외버스 통행 인원", "고속버스 통행 인원"]
bo.columns = columns
#지역코드 로드
jc = pd.read_csv("지역코드.dat", sep='|', header=None)
columns = ["지역 구분", "시도 코드", "시군구 코드", "이용 지역 코드", "시도 명", "시군구 명", "읍면동 명"]
jc.columns = columns
#지역코드에서 지역명 추출
sido = {}
for i in list(jc['시도 코드'].unique()):
    sido[i] = jc.loc[jc['시도 코드'] == i, '시도 명'].iloc[0]
sigungu = {}
for i in list(jc['시군구 코드'].unique()):
    sigungu[i] = jc.loc[jc['시군구 코드'] == i, '시군구 명'].iloc[0]
jiyeok = {}
for i in list(jc['이용 지역 코드'].unique()):
    jiyeok[i] = jc.loc[jc['이용 지역 코드'] == i, '읍면동 명'].iloc[0]
#발권데이터에서 코드에 따른 지역명 변수 추가 위해 함수 생성
def sido_change(x):
    try: return sido[int(x)]
    except: return np.nan
def sigungu_change(x):
    try: return sigungu[int(x)]
    except: return np.nan
def jiyeok_change(x):
    try: return jiyeok[int(x)]
    except: return np.nan
#발권데이터에 지역명 추가
bo['출발 시도 명'] = bo['출발 시도 코드'].apply(lambda x: sido_change(x))
bo['도착 시도 명'] = bo['도착 시도 코드'].apply(lambda x: sido_change(x))
bo['출발 시군구 명'] = bo['출발 시군구 코드'].apply(lambda x: sigungu_change(x))
bo['도착 시군구 명'] = bo['도착 시군구 코드'].apply(lambda x: sigungu_change(x))
bo['출발 읍면동 명'] = bo['출발 지역 코드'].apply(lambda x: jiyeok_change(x))
bo['도착 읍면동 명'] = bo['도착 지역 코드'].apply(lambda x: jiyeok_change(x))
bo = bo[["년도", "년월", "운행 일자", "요일 구분", "출발 시도 명", "출발 시군구 명","출발 읍면동 명", "도착 시도 명", "도착 시군구 명", "도착 읍면동 명",
         "철도 통행 인원", "시외버스 통행 인원", "고속버스 통행 인원", "출발 시도 코드", "출발 시군구 코드", "출발 지역 코드", "도착 시도 코드", "도착 시군구 코드", "도착 지역 코드"]]_____no_output_____

### 1.2. 안내서를 위한 추가 전처리_____no_output_____

bo["버스 통행 인원"] = bo["시외버스 통행 인원"] + bo["고속버스 통행 인원"]
#요일 구분 변수 생성
def weekday(x):
    days = {1:"일", 2:"월", 3:"화", 4:"수", 5:"목", 6:"금", 7:"토"}
    return days[x]
bo['요일 구분'] = bo['요일 구분'].apply(lambda x: weekday(x))
#20년도 데이터 추출
bo_2020 = bo[bo['년도'] == 2020]
bo_2020 = bo_2020[bo_2020["운행 일자"]>=20200119]
#운행일자 dtype 변환
def to_datetime(x):
    date = dt.datetime.strptime(str(x), "%Y%m%d").date()
    return date
bo_2020['운행 일자'] = bo_2020['운행 일자'].apply(lambda x: to_datetime(x))
#출발지 서울, 전국 데이터 분할
bo_2020_drop = bo_2020.copy()
bo_2020 = bo_2020[bo_2020['출발 시도 명'] == "서울특별시"]
bo_2020_drop.dropna(subset=['도착 시도 명', '도착 시군구 명'], inplace=True)_____no_output_____

## 2. 안전한 귀향을 위한 안내서_____no_output_____### 2.1. 안내서 제작을 위한 함수 생성
함수 목록은 다음과 같습니다.

1) **to_query** : 인자로 전달한 플랫폼의 출발지역코드를 가진 행을 필터링하기 위하여 쿼리를 작성해주는 함수

2) **covid_count_crawler**: 당일 서울 확진자수와 전국 확진자 수를 반환해주는 함수
  - 크롤링 코드 실행을 위해 [이 곳](https://sites.google.com/a/chromium.org/chromedriver/downloads) 에서 버전에 맞는 **chromedriver.exe를 설치**해주시고 경로를 올바르게 설정해주시길 바랍니다.

3) **find_res_plat** : 도착지로 가는 플랫폼과 출발지에서 가까운 플랫폼을 인자로 받아, 도착지까지 운행하면서 출발지에서 가장 가까운 플랫폼을 출력해주는 함수

4) **where** : 도착지로 도시명이 입력되었을때, 해당 시도 명을 반환해주는 함수

5) **view_map** : 최종 플랫폼으로 선정된 데이터와 출발지(구)를 folium 지도상으로 시각화하는 함수

6) **find_platform** : 출발지, 도착지를 인자로 전달하며, 도착지까지 운행하면서 출발지에서 가장 가까운 플랫폼(철도, 고속버스, 시외버스 각각 1개)을 선정하며, 이 플랫폼들에 해당되는 출발지역코드를 가진 행들만을 필터링 해주는 함수

7) **trans_graph** : 도착지까지 가는 각 플랫폼(철도, 고속버스, 시외버스)의 요일별 일평균 통행량을 그래프로 시각화하는 함수

8) **dest_graph** : 도착지를 기준으로 세부 시군구에 따른 일평균 통행 수단별 통행 인원을 그래프로 시각화하는 함수

9) **pie_plot** : 출발지와 도착지를 모두 고려한 철도와 버스터미널의 통행인원비율을 그래프로 시각화하는 함수

10) **final_guide** : 위 함수들을 모두 종합하여 안내서를 반환하는 함수
_____no_output_____

# 1) to_query
def to_query(location, loc):
  start = "출발지역코드 == "
  col = "'"
  loc_code = location[loc]
  return (start + col + loc_code + col), loc_code
# 2) covid_count_crawler #chromedriver 경로 확인해주세요!
def covid_count_crawler():
    driver = webdriver.Chrome(executable_path="C:\chromedriver_win32\chromedriver.exe")
    driver.get("https://www.seoul.go.kr/coronaV/coronaStatus.do")
    try:
        covid_seoul = driver.find_element_by_class_name("status-seoul")
        count_seoul = covid_seoul.find_element_by_class_name("counter")
        confirmed_seoul = count_seoul.text
        covid_korea = driver.find_element_by_class_name("status-korea")
        count_korea = covid_korea.find_element_by_class_name("counter")
        confirmed_korea = count_korea.text
        time = covid_seoul.text[4:22]
    except: print("Error")
    driver.close()
    return confirmed_seoul, confirmed_korea, time
# 3) find_res_plat
def find_res_plat(dest, near):
  res = ""
  for i in range(len(near)):
    if near[i] in dest:
      res += near[i]
      break
  return res
# 4) where
def where(destination):
    areas = ["부산광역시", "대구광역시", "광주광역시", "대전광역시", "울산광역시", "세종특별자치시", "인천광역시",
            "경기도", "충청북도", "충청남도", "전라북도", "전라남도", "경상북도", "경상남도", "강원도"]
    for a in areas:
        if destination in a:
            destin = a
            return destin
    destin = bo_2020_drop[bo_2020_drop["도착 시군구 명"].str.contains(destination)].iloc[0,7]
    return destin
# 5) view_map
def view_map(start_region, res_plat):
  gps_plat = pd.read_csv("서울시_주요_기차역+터미널_좌표.csv")
  gps_gu = pd.read_csv("서울시 행정구역 시군구 정보 (좌표계_ WGS1984).csv", encoding='cp949')
  geo_path = '02. skorea_municipalities_geo_simple.json'
  geo_str = json.load(open(geo_path, encoding='utf-8'))
  f = folium.Figure(width=700, height=400)
  map = folium.Map(location=[37.5502, 126.982], zoom_start=11).add_to(f)
  map.choropleth(geo_data=geo_str, fill_color = '#BEEFFF')
  gu_xy = gps_gu[gps_gu['시군구명_한글'] == start_region][['위도', '경도']].values
  gu_x = gu_xy[0][0]
  gu_y = gu_xy[0][1]
  folium.Marker([gu_x, gu_y], popup=start_region, tooltip=start_region, icon=folium.Icon(color='red', icon='star')).add_to(map)
  if True:
    for i in range(len(res_plat)):
        plat_xy = gps_plat[gps_plat['이름'] == res_plat[i]][['위도', '경도']].values
        try:
            plat_x = plat_xy[0][0]
            plat_y = plat_xy[0][1]
            folium.Marker([plat_x, plat_y], tooltip=res_plat[i], popup=res_plat[i]).add_to(map)
        except: pass
  else:
     for i in range(len(res_plat)):
        plat_xy = gps_plat[gps_plat['이름'] == res_plat[i]][['위도', '경도']].values
        plat_x = plat_xy[0][0]
        plat_y = plat_xy[0][1]
        folium.Marker([plat_x, plat_y], tooltip=res_plat[i], popup=res_plat[i]).add_to(map)
  display(map)
  return map
# 6) find_platform
def find_platform(df, start_region, destination):
  destins = where(destination)
  df.rename(columns={"출발 지역 코드":"출발지역코드"}, inplace=True)
  near = pd.read_csv("인접플랫폼.csv")
  dest = pd.read_csv("도착터미널구분.csv")
  location = {'서울역' : '1117010700', '용산역' : '1117012800', '수서역' : '1168011500', '청량리역' : '1123010400', '영등포역' : '1156010100', '상봉역' : '1126010200',
            '서울고속버스터미널' : '1165010700', '서울남부터미널' : '1165010800', '동서울종합터미널' : '1121510300', '센트럴시티터미널' : '1165010700'}
  print("다음으로, 출발지 <" + start_region + ">에서 도착지 <" + destination + ">까지 가는 안전한 귀향길을 안내해 드리겠습니다.")
  dest_list = dest[dest['도착지'].str.contains(destins)]
  try: dest_express = dest_list['고속버스'].values[0].split(",")
  except: dest_express = []
  try: dest_inter = dest_list['시외버스'].values[0].split(",")
  except: dest_inter = []
  try: dest_train = dest_list['기차'].values[0].split(",")
  except: dest_train = []
  dest_ava_list = []
  if len(dest_express) >= 1: dest_ava_list.append("고속버스")
  if len(dest_inter) >= 1: dest_ava_list.append("시외버스")
  if len(dest_train) >= 1: dest_ava_list.append("기차")
  near_list = near[near['이름'] == start_region]
  near_terminal = near_list['터미널'].values[0].split(",")
  near_train = near_list['기차역'].values[0].split(",")
  res_plat = []
  temp_plat = find_res_plat(dest_train, near_train)
  if temp_plat == "": pass
  else: res_plat.append(temp_plat)
  temp_plat = find_res_plat(dest_express, near_terminal)
  if temp_plat == "": pass
  else: res_plat.append(temp_plat)
  temp_plat = find_res_plat(dest_inter, near_terminal)
  if temp_plat == "": pass
  else: res_plat.append(temp_plat)
  print("목적지인 "+ destination + "까지 운행하면서, 출발지와 가장 가까운 플랫폼은 다음과 같습니다.\n")
  if len(res_plat) == 3:
    print(" 1. 기차:", res_plat[0], "\n", "2. 고속버스:", res_plat[1], "\n", "3. 시외버스:", res_plat[2],"\n"*2)
  else:
      if destins == "대구광역시": #데이터 상으로 운행하는 통행 수단이 기차(철도) 하나뿐인 대구광역시 예외처리
          print(" 1. 기차:", res_plat[0],"\n"*2)
      elif destins == "인천광역시": #데이터 상으로 운행하는 통행 수단이 시외버스 하나뿐인 인천광역시 예외처리
          print(" 3. 시외버스:", res_plat[0],"\n"*2)
  query = ""
  count = 0
  loc_code = []
  for i in range(len(res_plat)):
    if res_plat[i] != '':
      q, code = to_query(location, res_plat[i])
      query += q
      count += 1
      loc_code.append(code)
      if (i != len(res_plat)-1):
        query += " or "
  if query[-3:] == 'or ':
    query = query[:-4]
  print("한 눈에 보시기 편하게 플랫폼 별 위치를 지도로 보여드리겠습니다.", "\n")
  map = view_map(start_region, res_plat)
  return df.query(query), map, loc_code, res_plat
# 7) trans_graph
def trans_graph(df, loc_code, res_plat, destins):
    nums = ["철도 통행 인원", "고속버스 통행 인원", "시외버스 통행 인원"]
    names = ["(기차)", "(고속버스)", "(시외버스)"]
    days = ["월", "화", "수", "목", "금", "토", "일"]
    colors = ["#009770", "#FFDF65", "#E91E63"]
    if destins == "대구광역시":#데이터 상으로 운행하는 통행 수단이 기차(철도) 하나뿐인 대구광역시 예외처리
        df_1 = df.groupby('출발지역코드').get_group(loc_code[0])
        df_1 = df_1[df_1[nums[0]]!=0]
        avgs = []
        for d in range(len(days)):
            new = df_1.groupby("요일 구분").get_group(days[d])
            avg = new[nums[0]].sum()//len(new["운행 일자"].unique())
            avgs.append(avg)
        fig = go.Figure(data=[go.Bar(x=days, y=avgs, marker_color="#009770")])
        fig.update_layout(width=700, template='plotly_dark',title_text="2020년 "+res_plat[0]+names[0]+ " 요일별 일평균 통행인원", )
        fig.show()
    elif destins == "인천광역시":#데이터 상으로 운행하는 통행 수단이 시외버스 하나뿐인 인천광역시 예외처리
        df_1 = df.groupby('출발지역코드').get_group(loc_code[0])
        df_1 = df_1[df_1[nums[2]]!=0]
        avgs = []
        for d in range(len(days)):
            new = df_1.groupby("요일 구분").get_group(days[d])
            avg = new[nums[2]].sum()//len(new["운행 일자"].unique())
            avgs.append(avg)
        fig = go.Figure(data=[go.Bar(x=days, y=avgs, marker_color="#E91E63")])
        fig.update_layout(width=700, template='plotly_dark',title_text="2020년 "+res_plat[0]+names[2]+ " 요일별 일평균 통행인원", )
        fig.show()
    else:
        fig = make_subplots(rows=1, cols=3, subplot_titles=(res_plat[0]+names[0],res_plat[1]+names[1],res_plat[2]+names[2]))
        for c in range(len(loc_code)):
            df_1 = df.groupby('출발지역코드').get_group(loc_code[c])
            df_1 = df_1[df_1[nums[c]]!=0]
            avgs = []
            for d in range(len(days)):
                new = df_1.groupby("요일 구분").get_group(days[d])
                avg = new[nums[c]].sum()//len(new["운행 일자"].unique())
                avgs.append(avg)
            fig.add_trace(go.Bar(x=days, y=avgs, name=nums[c], marker_color=colors[c] ),row=1, col=c+1)
        fig.update_layout(template='plotly_dark',title_text="2020년 요일별 일평균 통행인원")
        fig.show()
# 8) dest_graph
def dest_graph(destination):
    areas = ["부산광역시", "대구광역시", "광주광역시", "대전광역시", "울산광역시", "세종특별자치시", "인천광역시",
            "경기도", "충청북도", "충청남도", "전라북도", "전라남도", "경상북도", "경상남도", "강원도"]
    sigungus = []
    for s in range(7):
        area = bo_2020_drop.groupby('도착 시도 명').get_group(areas[s])
        sigungus.append(list(area["도착 시군구 명"].unique()))
    sigungus.append(["수원시", "평택시", "광명시", "성남시", "화성시", "안산시"])
    sigungus.append(["청주시", "충주시", "제천시", "진천군", "영동군", "음성군"])
    sigungus.append(["천안시", "아산시", "서산시", "당진시", "공주시", "논산시"])
    sigungus.append(["전주시", "익산시", "군산시", "정읍시", "남원시", "임실군"])
    sigungus.append(["순천시", "목포시", "여수시", "나주시", "광양시", "고흥군"])
    sigungus.append(["경주시", "구미시", "포항시", "김천시", "경산시", "안동시"])
    sigungus.append(["진주시", "창원시", "밀양시", "김해시", "통영시", "거제시"])
    sigungus.append(["원주시", "춘천시", "강릉시", "속초시", "동해시", "홍천군"])
    a = areas.index(where(destination))
    kk_20 = bo_2020_drop.groupby('도착 시도 명').get_group(where(destination))
    all_kk = pd.DataFrame(index=range(0,6), columns = ["년도", "도착 시군구 명", "철도 통행 인원", "시외버스 통행 인원", "고속버스 통행 인원"])
    nums = ["철도 통행 인원", "시외버스 통행 인원", "고속버스 통행 인원"]
    for i in range(len(sigungus[a])):
        all_kk["도착 시군구 명"][i] = sigungus[a][i]
        all_kk["년도"][i] = 2020
        kk_20_1 = kk_20[kk_20['도착 시군구 명'].str.contains(sigungus[a][i])]
        for n in range(len(nums)):
            new_20 = kk_20_1[kk_20_1[nums[n]]!=0]
            avg_20 = new_20[nums[n]].sum()//len(new_20["운행 일자"].unique())
            all_kk[nums[n]][i] = avg_20
    trace1 = go.Bar(x=all_kk["도착 시군구 명"], y=all_kk["철도 통행 인원"], name='철도', marker_color="#009770")
    trace2 = go.Bar(x=all_kk["도착 시군구 명"], y=all_kk["시외버스 통행 인원"], name='시외버스', marker_color="#E91E63")
    trace3 = go.Bar(x=all_kk["도착 시군구 명"], y=all_kk["고속버스 통행 인원"], name='고속버스', marker_color="#FFDF65")
    data = [trace1, trace2, trace3]
    title = "2020년 "+areas[a]+" 일평균 통행 수단 별 통행 인원 분포"
    layout = go.Layout(template='plotly_dark',width=700, title=title, barmode='stack')
    fig = go.Figure(data=data, layout=layout)
    pyo.iplot(fig)
# 9) pie_plot
def pie_plot(destination, result):
    dict_train = {'서울역':'1117010700', '용산역':'1117012800', '수서역': '1168011500', '영등포역':'1156010100', '청량리역': '1123010400'}
    dict_bus = {'서울고속버스터미널':'1165010700', '센트럴시티터미널':'1165010700','동서울종합터미널':'1121510300', '서울남부터미널': '1165010800'}
    result_train = bo[(bo['출발 지역 코드'] == dict_train[result[0]]) & (bo['도착 시도 명'] == destination)]
    result_bus1 = bo[(bo['출발 지역 코드'] == dict_bus[result[1]]) & (bo['도착 시도 명'] == destination)]
    result_bus2 = bo[(bo['출발 지역 코드'] == dict_bus[result[2]]) & (bo['도착 시도 명'] == destination)]
    means = [result_train['철도 통행 인원'].mean(), result_bus1['버스 통행 인원'].mean(), result_bus2['버스 통행 인원'].mean()]
    if dict_bus[result[1]] != dict_bus[result[2]]:
        ratio = [means[0], means[1], means[2]]
        labels = [result[0], result[1], result[2]]
        df = px.data.tips()
        fig = px.pie(df, values=ratio, names=labels, color_discrete_sequence=["#009770", "#FFDF65", "#E91E63"])
        fig.update_layout(width=600, height=500, template='plotly_dark')
        fig.update_layout(title={'text':'출발지와 목적지 모두 동일한 경우의 철도/버스 평균 인원 비율', 'y':0.95,'x':0.5,'xanchor': 'center','yanchor': 'top'})
        fig.show()
    else:
        ratio = [means[0], means[1]]
        labels = [result[0], result[1]]
        df = px.data.tips()
        fig = px.pie(df, values=ratio, names=labels, color_discrete_sequence=["#009770", "#E91E63"])
        fig.update_layout(width=600, height=500, template='plotly_dark')
        fig.update_layout(title={'text':'출발지와 목적지 모두 동일한 경우의 철도/버스 평균 인원 비율', 'y':0.95,'x':0.5,'xanchor': 'center','yanchor': 'top'})
        fig.show()
# 10) final_guide
def final_guide(start_region, destination):
    new_destin = where(destination)
    print("\n"*2," "*9, "---"*35,"\n")
    print(" "*42, "<< 당신의 안전한 귀향을 위한 안내서 >> \n")
    print(" "*10, "---"*35,"\n"*3)
    print("안녕하세요. 당신의 안전한 귀향을 위한 안내서입니다.\n")
    print("저희 안내서는 사용자가 출발지부터 도착지까지 가는 다양한 경로 중, \n타인과의 접촉을 최소화할 수 있는 '안전한 경로'를 선택하는 데 도움을 줄 수 있습니다.\n\n")
    print("먼저, 오늘의 코로나 확진자 현황을 살펴보겠습니다.\n")
    cf_seoul, cf_korea, time = covid_count_crawler()
    print(time, "코로나 확진자는 다음과 같습니다.\n")
    print("    1. 전국 확진자 : ", cf_korea, "명")
    print("    2. 서울 확진자 : ", cf_seoul, "명\n")
    print("이용에 참고하시길 바라겠습니다. :) \n\n")
    df, map, loc_code, res_plat = find_platform(bo_2020, start_region, destination)
    print("\n출발지는 빨간 표식, 플랫폼은 파란 표식입니다. 플랫폼의 대략적인 위치를 파악하십시오.\n지도는 확대, 축소가 자유롭게 가능하며, 마우스를 마커에 올리면 플랫폼명도 확인하실 수 있습니다.")
    print("\n"*3+ "다음으로 보여드릴 세 그래프는 선정된 플랫폼들의 혼잡도 정보를 제공해드립니다.\n이때 혼잡도는 각 플랫폼의 통행량 정도로 정의하였습니다.")
    print("하단 그래프들이 귀향을 위한 통행 수단 선택에 도움이 되길 바랍니다.\n")
    print("\n첫번째 그래프는 위에서 선정된 기차역/고속버스터미널/시외버스터미널 플랫폼 각각의 혼잡도를 요일별로 비교합니다.\n요일별 평균으로 세분화해서 나타내었으니 참고하시길 바랍니다.\n")
    print("마우스를 그래프에 가져다 대시면 보다 세부적인 정보를 확인하실 수 있습니다.")
    trans_graph(df, loc_code, res_plat, new_destin)
    if ("서울남부터미널" not in res_plat) and (new_destin != "인천광역시") and (new_destin != "대구광역시"):
        print("\n\n\n두 번째 그래프는 출발지가 <", start_region, "> 이고 도착지가 <", destination, "> 인 철도와 버스터미널의 통행인원비율입니다.\n")
        print("이 그래프는 입력받은 출발지와 도착지가 모두 동일한 경우의 통행 수단 별 인원 비율을 제공합니다. \n이 차트에서 자신과 동일한 출발/도착지를 가진 다른 사람들은 어떤 통행 수단을 선택했는가에 대한 비율을 살펴볼 수 있습니다.\n")
        print("마우스를 그래프에 가져다 대시면 보다 세부적인 정보를 확인하실 수 있습니다.")
        pie_plot(new_destin, res_plat)
    print("\n\n\n마지막으로 도착지인 <", destination, "> 을 기준으로, 도착지 인접지역의 통행 수단 별 통행 인원 분포를 보여드리겠습니다.\n")
    print("이는 추천드린 플랫폼들 뿐만 아니라 전국을 출발지로 두고 도착지가 <", destination, ">와 인접 지역인 통행량 및 통행수단 별 비율을 나타냅니다. \n이를 통해, 고향의 인접지역까지 이동하는 사람들은 얼마나 있는지, 또 통행수단별 비율은 어느 정도인지 확인하실 수 있습니다.\n")
    print("마우스를 그래프에 가져다 대시면 보다 세부적인 정보를 확인하실 수 있습니다.")
    dest_graph(destination)
    print("\n\n저희가 제공하는 안내서는 여기까지입니다.\n안내서가 제공하는 그래프들을 통해 안전한 통행 수단을 선택할 수 있길 바랍니다 :)")
    print("\n\n\n")
    print("\n"*2," "*9, "---"*35,"\n")
    print(" "*24,"<< 저희의 안내서를 통해 목적지인", destination, "까지 안전한 귀향길 되시길 바랍니다. >> \n")
    print(" "*10, "---"*35,"\n"*3)_____no_output_____

### 2.2. 안전한 귀향을 위한 안내서
아래 셀을 실행하여 확인하고 싶은 출발지와 도착지를 입력하시면 됩니다.

**[유의할 점]**
1. 코로나 확진자 현황을 크롤링하기 때문에 로딩하는데에 약간의 시간이 소요될 수 있는 점 양해 부탁드립니다.
2. 고속버스터미널과 시외버스터미널이 **동서울종합터미널**로 동일한 경우, 출발지가 같기 때문에 원형 차트에서 두 통행 수단의 인원이 합산되어 나타납니다.
3. 시외버스터미널이 **서울남부터미널**인 경우, 데이터상에서 도착지가 나타나있지 않아 원형 차트가 출력되지 않습니다.
4. **대구광역시**의 경우 기차노선만 존재하여 요일별 차트가 기차역 1개만 출력되며, 원형차트가 출력되지 않습니다.
5. **인천광역시**의 경우 시외버스 노선만 존재하여 요일별 차트가 시외버스터미널 1개만 출력되며, 원형차트가 출력되지 않습니다.
6. **출발지**는 **서울 시내 25개 지역구명**으로 입력하시고, **목적지**는 **시도 단위**로 **아래 예시와 같이 입력**해주시기 바랍니다._____no_output_____

start= input("출발지를 지역구 단위로 입력해주세요.(예시: 종로구) : ")
end = input("목적지를 시 단위로 입력해주세요. (예시: 속초) : ")
final_guide(start, end)출발지를 지역구 단위로 입력해주세요.(예시: 종로구) : 노원구
목적지를 시 단위로 입력해주세요. (예시: 속초) : 천안

           ---------------------------------------------------------------------------------------------------------

                                           << 당신의 안전한 귀향을 위한 안내서 >>

           ---------------------------------------------------------------------------------------------------------

안녕하세요. 당신의 안전한 귀향을 위한 안내서입니다.

저희 안내서는 사용자가 출발지부터 도착지까지 가는 다양한 경로 중,
타인과의 접촉을 최소화할 수 있는 '안전한 경로'를 선택하는 데 도움을 줄 수 있습니다.

먼저, 오늘의 코로나 확진자 현황을 살펴보겠습니다.

('20.09.04.00시 기준) 코로나 확진자는 다음과 같습니다.

    1. 전국 확진자 :  198 명
    2. 서울 확진자 :  68 명

이용에 참고하시길 바라겠습니다. :)

다음으로, 출발지 <노원구>에서 도착지 <천안>까지 가는 안전한 귀향길을 안내해 드리겠습니다.
목적지인 천안까지 운행하면서, 출발지와 가장 가까운 플랫폼은 다음과 같습니다.

 1. 기차: 서울역
 2. 고속버스: 서울고속버스터미널
 3. 시외버스: 동서울종합터미널

한 눈에 보시기 편하게 플랫폼 별 위치를 지도로 보여드리겠습니다.

안전한 귀향을 위한 안내서.ipynb


# Notebook from mariabardon/nanodegree_computer_vision
Path: Project_2_Image_Captioning/2_Training.ipynb

# Computer Vision Nanodegree

## Project: Image Captioning

---

In this notebook, you will train your CNN-RNN model.

You are welcome and encouraged to try out many different architectures and hyperparameters when searching for a good model.

This does have the potential to make the project quite messy!  Before submitting your project, make sure that you clean up:
- the code you write in this notebook.  The notebook should describe how to train a single CNN-RNN architecture, corresponding to your final choice of hyperparameters.  You should structure the notebook so that the reviewer can replicate your results by running the code in this notebook.
- the output of the code cell in **Step 2**.  The output should show the output obtained when training the model from scratch.

This notebook **will be graded**.

Feel free to use the links below to navigate the notebook:
- [Step 1](#step1): Training Setup
- [Step 2](#step2): Train your Model
- [Step 3](#step3): (Optional) Validate your Model_____no_output_____
## Step 1: Training Setup

In this step of the notebook, you will customize the training of your CNN-RNN model by specifying hyperparameters and setting other options that are important to the training procedure.  The values you set now will be used when training your model in **Step 2** below.

You should only amend blocks of code that are preceded by a `TODO` statement.  **Any code blocks that are not preceded by a `TODO` statement should not be modified**.

### Task #1

Begin by setting the following variables:
- `batch_size` - the batch size of each training batch.  It is the number of image-caption pairs used to amend the model weights in each training step.
- `vocab_threshold` - the minimum word count threshold.  Note that a larger threshold will result in a smaller vocabulary, whereas a smaller threshold will include rarer words and result in a larger vocabulary.
- `vocab_from_file` - a Boolean that decides whether to load the vocabulary from file.
- `embed_size` - the dimensionality of the image and word embeddings.
- `hidden_size` - the number of features in the hidden state of the RNN decoder.
- `num_epochs` - the number of epochs to train the model.  We recommend that you set `num_epochs=3`, but feel free to increase or decrease this number as you wish.  [This paper](https://arxiv.org/pdf/1502.03044.pdf) trained a captioning model on a single state-of-the-art GPU for 3 days, but you'll soon see that you can get reasonable results in a matter of a few hours!  (_But of course, if you want your model to compete with current research, you will have to train for much longer._)
- `save_every` - determines how often to save the model weights.  We recommend that you set `save_every=1`, to save the model weights after each epoch.  This way, after the `i`th epoch, the encoder and decoder weights will be saved in the `models/` folder as `encoder-i.pkl` and `decoder-i.pkl`, respectively.
- `print_every` - determines how often to print the batch loss to the Jupyter notebook while training.  Note that you **will not** observe a monotonic decrease in the loss function while training - this is perfectly fine and completely expected!  You are encouraged to keep this at its default value of `100` to avoid clogging the notebook, but feel free to change it.
- `log_file` - the name of the text file containing - for every step - how the loss and perplexity evolved during training.

If you're not sure where to begin to set some of the values above, you can peruse [this paper](https://arxiv.org/pdf/1502.03044.pdf) and [this paper](https://arxiv.org/pdf/1411.4555.pdf) for useful guidance!  **To avoid spending too long on this notebook**, you are encouraged to consult these suggested research papers to obtain a strong initial guess for which hyperparameters are likely to work best.  Then, train a single model, and proceed to the next notebook (**3_Inference.ipynb**).  If you are unhappy with your performance, you can return to this notebook to tweak the hyperparameters (and/or the architecture in **model.py**) and re-train your model.

### Question 1

**Question:** Describe your CNN-RNN architecture in detail.  With this architecture in mind, how did you select the values of the variables in Task 1?  If you consulted a research paper detailing a successful implementation of an image captioning model, please provide the reference.

**Answer:**
My CNN-RNN consists on a CNN encoder and a RNN decoder. For the CNN encoder I used a pretrained resnet(50) (without the last layer) and a linear layer that outputs a tensor of size embed_size. This last fully connected layer will be trained. For a decoder I used an embedding layer that takes word tensors of size vocab_size and returns tensors of size embed_size. Then the vectors will go through a recurrent LSTM with 512 layers. Then I add a droput layer, and finaly a fully connected layer that outputs a tensor of size vocab_size.

### (Optional) Task #2

Note that we have provided a recommended image transform `transform_train` for pre-processing the training images, but you are welcome (and encouraged!) to modify it as you wish.  When modifying this transform, keep in mind that:
- the images in the dataset have varying heights and widths, and
- if using a pre-trained model, you must perform the corresponding appropriate normalization.

### Question 2

**Question:** How did you select the transform in `transform_train`?  If you left the transform at its provided value, why do you think that it is a good choice for your CNN architecture?

**Answer:**
I used the transform as its provided value because it is the set of transforms used to train all pretrained [resnet models](https://pytorch.org/hub/pytorch_vision_resnet/). Since we are using a pretrained resnet model, it only makes sense to use the same transforms.

### Task #3

Next, you will specify a Python list containing the learnable parameters of the model.  For instance, if you decide to make all weights in the decoder trainable, but only want to train the weights in the embedding layer of the encoder, then you should set `params` to something like:
```
params = list(decoder.parameters()) + list(encoder.embed.parameters())
```

### Question 3

**Question:** How did you select the trainable parameters of your architecture?  Why do you think this is a good choice?

**Answer:** I used the embedding dimention of 256 also used [here](https://arxiv.org/pdf/1603.02814.pdf). I also use a simmilar batch size of 128. I decided to use 512 layers in the lstm, with the hope it would perform better.
Using 5 repetitions in the as vocab_threshold seemed a reasonalbe trade off number between the vocab_size not being too large, and how usefull the word would be when creating captions.

### Task #4

Finally, you will select an [optimizer](http://pytorch.org/docs/master/optim.html#torch.optim.Optimizer).

### Question 4

**Question:** How did you select the optimizer used to train your model?

**Answer:** I used Adam, a variation of Stochatic Gradient Descent that has been reported to work well in [this paper](https://arxiv.org/pdf/1603.02814.pdf), for example..._____no_output_____

!pip install nltk

import nltk

nltk.download('punkt')Requirement already satisfied: nltk in /opt/conda/lib/python3.6/site-packages (3.5)
Requirement already satisfied: tqdm in /opt/conda/lib/python3.6/site-packages (from nltk) (4.11.2)
Requirement already satisfied: regex in /opt/conda/lib/python3.6/site-packages (from nltk) (2020.11.13)
Requirement already satisfied: joblib in /opt/conda/lib/python3.6/site-packages (from nltk) (0.11)
Requirement already satisfied: click in /opt/conda/lib/python3.6/site-packages (from nltk) (6.7)
import torch
import torch.nn as nn
from torchvision import transforms
import sys
sys.path.append('/opt/cocoapi/PythonAPI')
from pycocotools.coco import COCO
from data_loader import get_loader
from model import EncoderCNN, DecoderRNN
import math

## TODO #1: Select appropriate values for the Python variables below.
batch_size = 128          # batch size
vocab_threshold = 5        # minimum word count threshold
vocab_from_file = False    # if True, load existing vocab file
embed_size = 256           # dimensionality of image and word embeddings
hidden_size = 512          # number of features in hidden state of the RNN decoder
num_epochs = 3             # number of training epochs
save_every = 1             # determines frequency of saving model weights
print_every = 100          # determines window for printing average loss
log_file = 'training_log.txt'       # name of file with saved training loss and perplexity

# (Optional) TODO #2: Amend the image transform below.
transform_train = transforms.Compose([
    transforms.Resize(256),                          # smaller edge of image resized to 256
    transforms.RandomCrop(224),                      # get 224x224 crop from random location
    transforms.RandomHorizontalFlip(),               # horizontally flip image with probability=0.5
    transforms.ToTensor(),                           # convert the PIL Image to a tensor
    transforms.Normalize((0.485, 0.456, 0.406),      # normalize image for pre-trained model
                         (0.229, 0.224, 0.225))])

# Build data loader.
data_loader = get_loader(transform=transform_train,
                         mode='train',
                         batch_size=batch_size,
                         vocab_threshold=vocab_threshold,
                         vocab_from_file=vocab_from_file)

# The size of the vocabulary.
vocab_size = len(data_loader.dataset.vocab)

# Initialize the encoder and decoder.
encoder = EncoderCNN(embed_size)
decoder = DecoderRNN(embed_size, hidden_size, vocab_size)

# Move models to GPU if CUDA is available.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
encoder.to(device)
decoder.to(device)

# Define the loss function.
criterion = nn.CrossEntropyLoss().cuda() if torch.cuda.is_available() else nn.CrossEntropyLoss()

# TODO #3: Specify the learnable parameters of the model.
params = list(decoder.parameters()) + list(encoder.embed.parameters())
# TODO #4: Define the optimizer.
optimizer = torch.optim.Adam(params = params, lr = 0.001)
# Set the total number of training steps per epoch.
total_step = math.ceil(len(data_loader.dataset.caption_lengths) / data_loader.batch_sampler.batch_size)loading annotations into memory...
Done (t=0.89s)
creating index...
index created!
[0/414113] Tokenizing captions...
[100000/414113] Tokenizing captions...
[200000/414113] Tokenizing captions...
[300000/414113] Tokenizing captions...
[400000/414113] Tokenizing captions...
loading annotations into memory...
Done (t=0.94s)
creating index...

## Step 2: Train your Model

Once you have executed the code cell in **Step 1**, the training procedure below should run without issue.

It is completely fine to leave the code cell below as-is without modifications to train your model.  However, if you would like to modify the code used to train the model below, you must ensure that your changes are easily parsed by your reviewer.  In other words, make sure to provide appropriate comments to describe how your code works!

You may find it useful to load saved weights to resume training.  In that case, note the names of the files containing the encoder and decoder weights that you'd like to load (`encoder_file` and `decoder_file`).  Then you can load the weights by using the lines below:

```python
# Load pre-trained weights before resuming training.
encoder.load_state_dict(torch.load(os.path.join('./models', encoder_file)))
decoder.load_state_dict(torch.load(os.path.join('./models', decoder_file)))
```

While trying out parameters, make sure to take extensive notes and record the settings that you used in your various training runs.  In particular, you don't want to encounter a situation where you've trained a model for several hours but can't remember what settings you used :).

### A Note on Tuning Hyperparameters

To figure out how well your model is doing, you can look at how the training loss and perplexity evolve during training - and for the purposes of this project, you are encouraged to amend the hyperparameters based on this information.

However, this will not tell you if your model is overfitting to the training data, and, unfortunately, overfitting is a problem that is commonly encountered when training image captioning models.

For this project, you need not worry about overfitting. **This project does not have strict requirements regarding the performance of your model**, and you just need to demonstrate that your model has learned **_something_** when you generate captions on the test data.  For now, we strongly encourage you to train your model for the suggested 3 epochs without worrying about performance; then, you should immediately transition to the next notebook in the sequence (**3_Inference.ipynb**) to see how your model performs on the test data.  If your model needs to be changed, you can come back to this notebook, amend hyperparameters (if necessary), and re-train the model.

That said, if you would like to go above and beyond in this project, you can read about some approaches to minimizing overfitting in section 4.3.1 of [this paper](http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7505636).  In the next (optional) step of this notebook, we provide some guidance for assessing the performance on the validation dataset._____no_output_____

import torch.utils.data as data
import numpy as np
import os
import requests
import time

# Open the training log file.
f = open(log_file, 'w')

old_time = time.time()
response = requests.request("GET",
                            "http://metadata.google.internal/computeMetadata/v1/instance/attributes/keep_alive_token",
                            headers={"Metadata-Flavor":"Google"})

for epoch in range(1, num_epochs+1):

    for i_step in range(1, total_step+1):

        if time.time() - old_time > 60:
            old_time = time.time()
            requests.request("POST",
                             "https://nebula.udacity.com/api/v1/remote/keep-alive",
                             headers={'Authorization': "STAR " + response.text})

        # Randomly sample a caption length, and sample indices with that length.
        indices = data_loader.dataset.get_train_indices()
        # Create and assign a batch sampler to retrieve a batch with the sampled indices.
        new_sampler = data.sampler.SubsetRandomSampler(indices=indices)
        data_loader.batch_sampler.sampler = new_sampler

        # Obtain the batch.
        images, captions = next(iter(data_loader))

        # Move batch of images and captions to GPU if CUDA is available.
        images = images.to(device)
        captions = captions.to(device)

        # Zero the gradients.
        decoder.zero_grad()
        encoder.zero_grad()

        # Pass the inputs through the CNN-RNN model.
        features = encoder(images)
        outputs = decoder(features, captions)

        # Calculate the batch loss.
        loss = criterion(outputs.view(-1, vocab_size), captions.view(-1))

        # Backward pass.
        loss.backward()

        # Update the parameters in the optimizer.
        optimizer.step()

        # Get training statistics.
        stats = 'Epoch [%d/%d], Step [%d/%d], Loss: %.4f, Perplexity: %5.4f' % (epoch, num_epochs, i_step, total_step, loss.item(), np.exp(loss.item()))

        # Print training statistics (on same line).
        print('\r' + stats, end="")
        sys.stdout.flush()

        # Print training statistics to file.
        f.write(stats + '\n')
        f.flush()

        # Print training statistics (on different line).
        if i_step % print_every == 0:
            print('\r' + stats)

    # Save the weights.
    if epoch % save_every == 0:
        torch.save(decoder.state_dict(), os.path.join('./models', 'decoder-%d.pkl' % epoch))
        torch.save(encoder.state_dict(), os.path.join('./models', 'encoder-%d.pkl' % epoch))

# Close the training log file.
f.close()Epoch [1/3], Step [100/3236], Loss: 3.8390, Perplexity: 46.4799
Epoch [1/3], Step [200/3236], Loss: 3.4494, Perplexity: 31.4823
Epoch [1/3], Step [300/3236], Loss: 3.2700, Perplexity: 26.3108
Epoch [1/3], Step [400/3236], Loss: 3.0931, Perplexity: 22.0449
Epoch [1/3], Step [500/3236], Loss: 3.0239, Perplexity: 20.5721
Epoch [1/3], Step [600/3236], Loss: 3.0887, Perplexity: 21.9482
Epoch [1/3], Step [700/3236], Loss: 2.9499, Perplexity: 19.1047
Epoch [1/3], Step [800/3236], Loss: 2.8563, Perplexity: 17.3962
Epoch [1/3], Step [900/3236], Loss: 2.9503, Perplexity: 19.1123
Epoch [1/3], Step [1000/3236], Loss: 2.7734, Perplexity: 16.0132
Epoch [1/3], Step [1100/3236], Loss: 2.7129, Perplexity: 15.0727
Epoch [1/3], Step [1200/3236], Loss: 2.5628, Perplexity: 12.9715
Epoch [1/3], Step [1300/3236], Loss: 2.5414, Perplexity: 12.6970
Epoch [1/3], Step [1400/3236], Loss: 2.4386, Perplexity: 11.4570
Epoch [1/3], Step [1500/3236], Loss: 2.4895, Perplexity: 12.0547
Epoch [1/3], Step [1600/3236], Loss: 2.5316, Perplexity: 12.5736
Epoch [1/3], Step [1700/3236], Loss: 2.8642, Perplexity: 17.5350
Epoch [1/3], Step [1800/3236], Loss: 2.5344, Perplexity: 12.6084
Epoch [1/3], Step [1900/3236], Loss: 2.4011, Perplexity: 11.0350
Epoch [1/3], Step [2000/3236], Loss: 2.4330, Perplexity: 11.3935
Epoch [1/3], Step [2100/3236], Loss: 2.6013, Perplexity: 13.4815
Epoch [1/3], Step [2200/3236], Loss: 2.2781, Perplexity: 9.75854
Epoch [1/3], Step [2300/3236], Loss: 2.4647, Perplexity: 11.7598
Epoch [1/3], Step [2400/3236], Loss: 2.6162, Perplexity: 13.6835
Epoch [1/3], Step [2500/3236], Loss: 2.2839, Perplexity: 9.81479
Epoch [1/3], Step [2600/3236], Loss: 2.1956, Perplexity: 8.98502
Epoch [1/3], Step [2700/3236], Loss: 2.2682, Perplexity: 9.66225
Epoch [1/3], Step [2800/3236], Loss: 2.8015, Perplexity: 16.4686
Epoch [1/3], Step [2900/3236], Loss: 2.4408, Perplexity: 11.4822
Epoch [1/3], Step [3000/3236], Loss: 2.5244, Perplexity: 12.4830
Epoch [1/3], Step [3100/3236], Loss: 2.6787, Perplexity: 14.5667
Epoch [1/3], Step [3200/3236], Loss: 2.4625, Perplexity: 11.7339
Epoch [2/3], Step [100/3236], Loss: 2.3479, Perplexity: 10.46377
Epoch [2/3], Step [200/3236], Loss: 2.1224, Perplexity: 8.35108
Epoch [2/3], Step [300/3236], Loss: 2.2875, Perplexity: 9.85051
Epoch [2/3], Step [400/3236], Loss: 2.6672, Perplexity: 14.3993
Epoch [2/3], Step [500/3236], Loss: 2.2870, Perplexity: 9.84552
Epoch [2/3], Step [600/3236], Loss: 2.2645, Perplexity: 9.62621
Epoch [2/3], Step [700/3236], Loss: 2.2281, Perplexity: 9.28209
Epoch [2/3], Step [800/3236], Loss: 2.2273, Perplexity: 9.27458
Epoch [2/3], Step [900/3236], Loss: 2.2126, Perplexity: 9.13976
Epoch [2/3], Step [1000/3236], Loss: 2.8609, Perplexity: 17.4770
Epoch [2/3], Step [1100/3236], Loss: 2.6687, Perplexity: 14.4216
Epoch [2/3], Step [1200/3236], Loss: 2.2865, Perplexity: 9.84061
Epoch [2/3], Step [1300/3236], Loss: 2.3748, Perplexity: 10.7487
Epoch [2/3], Step [1400/3236], Loss: 2.2569, Perplexity: 9.55366
Epoch [2/3], Step [1500/3236], Loss: 2.0397, Perplexity: 7.68824
Epoch [2/3], Step [1600/3236], Loss: 2.1990, Perplexity: 9.01611
Epoch [2/3], Step [1700/3236], Loss: 2.1670, Perplexity: 8.73173
Epoch [2/3], Step [1800/3236], Loss: 2.2186, Perplexity: 9.19427
Epoch [2/3], Step [1900/3236], Loss: 2.3181, Perplexity: 10.1563
Epoch [2/3], Step [2000/3236], Loss: 2.2425, Perplexity: 9.416921
Epoch [2/3], Step [2100/3236], Loss: 2.1352, Perplexity: 8.45853
Epoch [2/3], Step [2200/3236], Loss: 2.2006, Perplexity: 9.03095
Epoch [2/3], Step [2300/3236], Loss: 2.0420, Perplexity: 7.70576
Epoch [2/3], Step [2400/3236], Loss: 2.0264, Perplexity: 7.58688
Epoch [2/3], Step [2500/3236], Loss: 2.2574, Perplexity: 9.55861
Epoch [2/3], Step [2600/3236], Loss: 2.1347, Perplexity: 8.45444
Epoch [2/3], Step [2700/3236], Loss: 2.6327, Perplexity: 13.9115
Epoch [2/3], Step [2800/3236], Loss: 1.9200, Perplexity: 6.82082
Epoch [2/3], Step [2900/3236], Loss: 2.1046, Perplexity: 8.20368
Epoch [2/3], Step [3000/3236], Loss: 2.2361, Perplexity: 9.35702
Epoch [2/3], Step [3100/3236], Loss: 2.0557, Perplexity: 7.81267
Epoch [2/3], Step [3200/3236], Loss: 2.1574, Perplexity: 8.64851
Epoch [3/3], Step [100/3236], Loss: 3.1367, Perplexity: 23.02743
Epoch [3/3], Step [200/3236], Loss: 2.0603, Perplexity: 7.84802
Epoch [3/3], Step [300/3236], Loss: 2.2221, Perplexity: 9.22648
Epoch [3/3], Step [400/3236], Loss: 2.4032, Perplexity: 11.0580
Epoch [3/3], Step [443/3236], Loss: 2.0651, Perplexity: 7.88596

## Step 3: (Optional) Validate your Model

To assess potential overfitting, one approach is to assess performance on a validation set.  If you decide to do this **optional** task, you are required to first complete all of the steps in the next notebook in the sequence (**3_Inference.ipynb**); as part of that notebook, you will write and test code (specifically, the `sample` method in the `DecoderRNN` class) that uses your RNN decoder to generate captions.  That code will prove incredibly useful here.

If you decide to validate your model, please do not edit the data loader in **data_loader.py**.  Instead, create a new file named **data_loader_val.py** containing the code for obtaining the data loader for the validation data.  You can access:
- the validation images at filepath `'/opt/cocoapi/images/train2014/'`, and
- the validation image caption annotation file at filepath `'/opt/cocoapi/annotations/captions_val2014.json'`.

The suggested approach to validating your model involves creating a json file such as [this one](https://github.com/cocodataset/cocoapi/blob/master/results/captions_val2014_fakecap_results.json) containing your model's predicted captions for the validation images.  Then, you can write your own script or use one that you [find online](https://github.com/tylin/coco-caption) to calculate the BLEU score of your model.  You can read more about the BLEU score, along with other evaluation metrics (such as TEOR and Cider) in section 4.1 of [this paper](https://arxiv.org/pdf/1411.4555.pdf).  For more information about how to use the annotation file, check out the [website](http://cocodataset.org/#download) for the COCO dataset._____no_output_____

# (Optional) TODO: Validate your model._____no_output_____