# Notebook from rayzkaunda/datarigs.github.io
Path: images/d-patients/Final-Diabetic Hospital Stays Prediction-Copy1.ipynb

# Predicting how many days a Diabetic patient will stay in a hospital
_____no_output_____

from IPython.display import Image
from IPython.core.display import HTML
Image(url= "https://cdn-images-1.medium.com/max/1600/1*X7lC_0LS71Jx_kjMKxCNug.jpeg")_____no_output_____

# Contents

1. [Objective](#Objective)
1. [Setup](#Setup)
1. [Data](#Data)
1. [Train](#Train)
1. [Host](#Host)
1. [Predict](#Predict)
1. [Extensions](#Extensions)

---

# Business Problem:
Using the given dataset, you  will develop a modern scalable full stack application on AWS cloud using a predictive model to predict how many days an admitted diabetic patient will spend 5 days or more in a hospital. We have framed our problem as a binary classification problem but it can also be a regression problem.

### Project Overview

In 2012, there were about 36.5 million hospital stays with an average length of stay of 4.5 days
and an average cost of \\$10,400 per stay summing up to \\$377 billion('https://www.hcup-us.ahrq.gov/reports/statbriefs/sb180-Hospitalizations-United-States-2012.pdf').

In 2012, there were approximately 36.5 million hospital stays in the United States, representing a
hospitalization rate of 116.2 stays per 1,000 population. Across all types of stays, the average length
of a hospital stay was 4.5 days.

Predictive analytics is an increasingly important tool in the healthcare field since modern machine learning (ML) methods can use large amounts of available data to predict individual outcomes for patients. For example, ML predictions can help healthcare providers determine likelihoods of disease, aid in diagnosis, recommend treatment, and predict future wellness. For this project, I chose to focus on a more logistical metric of healthcare, hospital length-of-stay (LOS). LOS is defined as the time between hospital admission and discharge measured in days.

### Cost of Diabetis in US
The total estimated cost of diagnosed diabetes in 2017 is  \\$327 billion, including  \\$237 billion in direct medical costs and \\$90 billion in reduced productivity. For the cost categories analyzed, care for people with diagnosed diabetes accounts for 1 in 4 health care dollars in the U.S., and more than half of that expenditure is directly attributable to diabetes. People with diagnosed diabetes incur average medical expenditures of \\$16,750 per year, of which \\$9,600 is attributed to diabetes. People with diagnosed diabetes, on average, have medical expenditures ∼2.3 times higher than what expenditures would be in the absence of diabetes. Indirect costs include increased absenteeism (\\$3.3 billion) and reduced productivity while at work (\\$26.9 billion) for the employed population, reduced productivity for those not in the labor force (\\$2.3 billion), inability to work because of disease-related disability (\\$37.5 billion), and lost productivity due to 277,000 premature deaths attributed to diabetes (\\$19.9 billion).('http://care.diabetesjournals.org/content/early/2018/03/20/dci18-0007.full-text.pdf')

U.S. hospital stays cost the health system at least \\$377.5 billion per year and recent Medicare legislation standardizes payments for procedures performed, regardless of the number of days a patient spends in the hospital. This incentivizes hospitals to identify patients of high LOS risk at the time of admission. Once identified, patients with high LOS risk can have their treatment plan optimized to minimize LOS and lower the chance of getting a hospital-acquired condition such as staph infection. Another benefit is that prior knowledge of LOS can aid in logistics such as room and bed allocation planning.

### Solution overview
_____no_output_____

from IPython.display import Image
from IPython.core.display import HTML
Image(url= "https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2018/06/22/serverless-frontend-sagemaker-1-w.gif")_____no_output_____

# Data and Materials
This study used the Health Facts database
(Cerner Corporation, Kansas City, MO), a national data
warehouse that collects comprehensive clinical records across
hospitals throughout the United States. Health Facts is a
voluntary program offered to organizations which use the
Cerner Electronic Health Record System. The database contains data systematically collected from participating institutions electronic medical records and includes encounter data
(emergency, outpatient, and inpatient), provider specialty,
demographics (age, sex, and race), diagnoses and in-hospital
procedures documented by ICD-9-CM codes, laboratory
data, pharmacy data, in-hospital mortality, and hospital
characteristics. All data were deidentified in compliance with
the Health Insurance Portability and Accountability Act of
1996 before being provided to the investigators. Continuity
of patient encounters within the same health system (EHR
system) is preserved.
The Health Facts data we used was an extract representing
10 years (1999–2008) of clinical care at 130 hospitals and
integrated delivery networks throughout the United States:
Midwest (18 hospitals), Northeast (58), South (28), and West
(16). Most of the hospitals (78) have bed size between 100 and
499, 38 hospitals have bed size less than 100, and bed size of
14 hospitals is greater than 500.
The database consists of 41 tables in a fact-dimension
schema and a total of 117 features. The database includes
74,036,643 unique encounters (visits) that correspond to
17,880,231 unique patients and 2,889,571 providers. Because
this data represents integrated delivery network health systems in addition to stand-alone hospitals, the data contains
both inpatient and outpatient data, including emergency
department, for the same group of patients. However, data
from out-of-network providers is not captured.
_____no_output_____Transformed/Wrangled data: Comma Separated Values(csv)
* Basic setup for using SageMaker.
* converting datasets to protobuf format used by the Amazon SageMaker algorithms and uploading to S3.
* Training SageMaker's linear learner on the data set.
* Hosting the trained model.
* Scoring using the trained model.

_____no_output_____## Setup

Let's start by specifying:

* The SageMaker role arn used to give learning and hosting access to your data. The snippet below will use the same role used by your SageMaker notebook instance, if you're using other.  Otherwise, specify the full ARN of a role with the SageMakerFullAccess policy attached.
* The S3 bucket that you want to use for training and storing model objects._____no_output_____

import os
import boto3
import re
from sagemaker import get_execution_role

role = get_execution_role()

bucket = 'datastorez'# enter your s3 bucket where you will copy data and model artifacts
prefix = 'sagemaker/Diabetic-prediction' # place to upload training files within the bucket_____no_output_____

Now we'll import the Python libraries we'll need._____no_output_____

import pandas as pd
pd.set_option('display.max_columns',100)
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import io
import time
import json
import sagemaker.amazon.common as smac_____no_output_____

---
## Load the Data

In this case the data is in csv format and stored on a data lake on AWS's S3. In order to perform implement cloud based models and take advantage od the low costs the data should be uploaded to S3.

Let's download the data and save it in the local folder with the name data.csv and take a look at it._____no_output_____

df = pd.read_csv('s3://datastorez/diabetic_data.csv', header = 0, sep=',')
df.head(5)
_____no_output_____df.shape_____no_output_____df['readmitted'].value_counts()_____no_output_____

### Data Prepocessing and Cleanup
---
I used to hate cleaning and wrangling data but now it's the love of my life. I simply love it. I know many of you don't believe me. Dirty data is the norm and as a former machinist I love to get my hands dirty without the oil smell. The key to cleaning data is simple. Love it. To have a sucessful data project one must clean and explore the data intensively in order to ask and answer the right questions.

The data contains some columns with missing values and non-numeric columns and will need alot of clean up and prepocessing before feeding it to our model._____no_output_____#### Me when I come across dirty data_____no_output_____

Image(url= "https://i.imgur.com/oqfhOOE.gif?noredirect")_____no_output_____

First we are going to  convert  the categorical columns to numerical columns and then all the other columns we are going to use in our model._____no_output_____

#clean age column to 10 categories
df['age_group'] = df['age'].replace({'[0-10)': 0, '[10-20)': 1, '[20-30)': 2, '[30-40)': 3, '[40-50)': 4, '[50-60)': 5, '[60-70)': 6, '[70-80)': 7, '[80-90)': 8, '[90-100)': 9})
#will leave age column for now for visualization but will drop it later  _____no_output_____#clean race column and will leave race column for now for visualization but will drop it later
df['ethinicity'] = df.apply(lambda row: 0 if (row['race'] == '?') else -1, axis=1)
df['ethinicity'] = df.apply(lambda row: 1 if (row['race'] == 'AfricanAmerican') else row['ethinicity'], axis=1)
df['ethinicity'] = df.apply(lambda row: 2 if (row['race'] == 'Asian') else row['ethinicity'], axis=1)
df['ethinicity'] = df.apply(lambda row: 3 if (row['race'] == 'Caucasian') else row['ethinicity'], axis=1)
df['ethinicity'] = df.apply(lambda row: 4 if (row['race'] == 'Hispanic') else row['ethinicity'], axis=1)
df['ethinicity'] = df.apply(lambda row: 5 if (row['race'] == 'Other') else row['ethinicity'], axis=1)
df.head(5)_____no_output_____

### Clean medications columns and other columns

We will convert the gender, readimitted columns to numerical to 0 and 1 and the medications column to 0 if the patient did not receive the medications and 1 if the patient recieved the medications. In the original dataset, the  medications columns also had other attributes like steady but for this model we are going to categorize our columns into 2 attributes which in production should be given alot of consideration according to the  problem statement and how we are going to evaluate our model._____no_output_____

df['gender'] = df['gender'].apply(lambda x: 0 if x=='Female' else 1)
df['troglitazone'] = df['troglitazone'].apply(lambda x: 0 if x=='No' else 1)
df['metformin'] = df['metformin'].apply(lambda x: 0 if x=='No' else 1)
df['repaglinide'] = df['repaglinide'].apply(lambda x: 0 if x=='No' else 1)
df['nateglinide'] = df['nateglinide'].apply(lambda x: 0 if x=='No' else 1)
df['chlorpropamide'] = df['chlorpropamide'].apply(lambda x: 0 if x=='No' else 1)
df['chlorpropamide'] = df['chlorpropamide'].apply(lambda x: 0 if x=='No' else 1)
df['glimepiride'] = df['glimepiride'].apply(lambda x: 0 if x=='No' else 1)
df['acetohexamide'] =df['acetohexamide'].apply(lambda x: 0 if x=='No' else 1)
df['glyburide'] = df['glyburide'].apply(lambda x: 0 if x=='No' else 1)
df['tolbutamide'] =df['tolbutamide'].apply(lambda x: 0 if x=='No' else 1)
df['pioglitazone'] = df['pioglitazone'].apply(lambda x: 0 if x=='No' else 1)
df['rosiglitazone'] = df['rosiglitazone'].apply(lambda x: 0 if x=='No' else 1)
df['acarbose'] = df['acarbose'].apply(lambda x: 0 if x=='No' else 1)
df['miglitol'] = df['miglitol'].apply(lambda x: 0 if x=='No' else 1)
df['tolazamide'] = df['tolazamide'].apply(lambda x: 0 if x=='No' else 1)
df['examide'] = df['examide'].apply(lambda x: 0 if x=='No' else 1)
df['citoglipton'] = df['citoglipton'].apply(lambda x: 0 if x=='No' else 1)
df['glyburide-metformin'] = df['glyburide-metformin'].apply(lambda x: 0 if x=='No' else 1)
df['glipizide-metformin'] = df['glipizide-metformin'].apply(lambda x: 0 if x=='No' else 1)
df['glimepiride-pioglitazone'] = df['glimepiride-pioglitazone'].apply(lambda x: 0 if x=='No' else 1)
df['metformin-rosiglitazone'] = df['metformin-rosiglitazone'].apply(lambda x: 0 if x=='No' else 1)
df['metformin-pioglitazone'] = df['metformin-pioglitazone'].apply(lambda x: 0 if x=='No' else 1)
df['insulin'] = df['insulin'].apply(lambda x: 0 if x=='No' else 1)
df['change'] = df['change'].apply(lambda x: 0 if x=='No' else 1)
df['diabetesMed'] = df['diabetesMed'].apply(lambda x: 0 if x=='No' else 1)
df['readmitted'] = df['readmitted'].apply(lambda x: 0 if x=='NO' else 1)_____no_output_____#check for missing values and NaN
#print (df.info ()) _____no_output_____

### Dealing with diagonosis columns
----
Initialy I excluded these columns from the dataset but my model only got 67% accuracy. After researching to improve my model accuracy I found this paper which grouped the columns. _____no_output_____

#only primary diagonosis
df['d1'] = df['diag_1']
df['d2'] = df['diag_2']
df['d3'] = df['diag_3']_____no_output_____

Next we are grouping the digonosis columns based on the ICD-10 codes. _____no_output_____

# Regrouping the main diagnosis
df['d1'] = df.apply(lambda row: 1 if (row['diag_1'][0:3].zfill(3) >= '390') and (row['diag_1'][0:3].zfill(3) <= '459' ) or  (row['diag_1'][0:3].zfill(3) == '785' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 2 if (row['diag_1'][0:3].zfill(3) >= '460') and (row['diag_1'][0:3].zfill(3) <= '519' ) or  (row['diag_1'][0:3].zfill(3) == '786' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 3 if (row['diag_1'][0:3].zfill(3) >= '520') and (row['diag_1'][0:3].zfill(3) <= '579' ) or  (row['diag_1'][0:3].zfill(3) == '787' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 4 if (row['diag_1'][0:3].zfill(3) == '250') else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 5 if (row['diag_1'][0:3].zfill(3) >= '800') and (row['diag_1'][0:3].zfill(3) <= '999' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 6 if (row['diag_1'][0:3].zfill(3) >= '710') and (row['diag_1'][0:3].zfill(3) <= '739' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 7 if (row['diag_1'][0:3].zfill(3) >= '580') and (row['diag_1'][0:3].zfill(3) <= '629' ) or  (row['diag_1'][0:3].zfill(3) == '788' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 8 if (row['diag_1'][0:3].zfill(3) >= '140') and (row['diag_1'][0:3].zfill(3) <= '239' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 9 if (row['diag_1'][0:3].zfill(3) >= '790') and (row['diag_1'][0:3].zfill(3) <= '799' ) or  (row['diag_1'][0:3].zfill(3) == '780' ) or  (row['diag_1'][0:3].zfill(3) == '781' ) or  (row['diag_1'][0:3].zfill(3) == '784' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 10 if (row['diag_1'][0:3].zfill(3) >= '240') and (row['diag_1'][0:3].zfill(3) <= '249' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 10 if (row['diag_1'][0:3].zfill(3) >= '251') and (row['diag_1'][0:3].zfill(3) <= '279' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 11 if (row['diag_1'][0:3].zfill(3) >= '680') and (row['diag_1'][0:3].zfill(3) <= '709' ) or  (row['diag_1'][0:3].zfill(3) == '782' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 12 if (row['diag_1'][0:3].zfill(3) >= '001') and (row['diag_1'][0:3].zfill(3) <= '139' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 13 if (row['diag_1'][0:3].zfill(3) >= '290') and (row['diag_1'][0:3].zfill(3) <= '319' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 13 if (row['diag_1'][0:1] >= 'E') and (row['diag_1'][0:1] <= 'V' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 13 if (row['diag_1'][0:3].zfill(3) >= '280') and (row['diag_1'][0:3].zfill(3) <= '289' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 13 if (row['diag_1'][0:3].zfill(3) >= '320') and (row['diag_1'][0:3].zfill(3) <= '359' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 13 if (row['diag_1'][0:3].zfill(3) >= '630') and (row['diag_1'][0:3].zfill(3) <= '679' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 13 if (row['diag_1'][0:3].zfill(3) >= '360') and (row['diag_1'][0:3].zfill(3) <= '389' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 13 if (row['diag_1'][0:3].zfill(3) >= '740') and (row['diag_1'][0:3].zfill(3) <= '759' ) else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: 0 if (row['diag_1'][0:3].zfill(3)  == '783' or row['diag_1'][0:3].zfill(3)  == '789') else row['d1'], axis=1)
df['d1'] = df.apply(lambda row: -1 if (row['diag_1'][0:1] == '?') else row['d1'], axis=1)                           _____no_output_____# Regrouping of the first secondary diagnosis
df['d2'] = df.apply(lambda row: 1 if (row['diag_2'][0:3].zfill(3) >= '390') and (row['diag_2'][0:3].zfill(3) <= '459' ) or  (row['diag_2'][0:3].zfill(3) == '785' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 2 if (row['diag_2'][0:3].zfill(3) >= '460') and (row['diag_2'][0:3].zfill(3) <= '519' ) or  (row['diag_2'][0:3].zfill(3) == '786' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 3 if (row['diag_2'][0:3].zfill(3) >= '520') and (row['diag_2'][0:3].zfill(3) <= '579' ) or  (row['diag_2'][0:3].zfill(3) == '787' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 4 if (row['diag_2'][0:3].zfill(3) == '250') else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 5 if (row['diag_2'][0:3].zfill(3) >= '800') and (row['diag_2'][0:3].zfill(3) <= '999' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 6 if (row['diag_2'][0:3].zfill(3) >= '710') and (row['diag_2'][0:3].zfill(3) <= '739' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 7 if (row['diag_2'][0:3].zfill(3) >= '580') and (row['diag_2'][0:3].zfill(3) <= '629' ) or  (row['diag_2'][0:3].zfill(3) == '788' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 8 if (row['diag_2'][0:3].zfill(3) >= '140') and (row['diag_2'][0:3].zfill(3) <= '239' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 9 if (row['diag_2'][0:3].zfill(3) >= '790') and (row['diag_2'][0:3].zfill(3) <= '799' ) or  (row['diag_2'][0:3].zfill(3) == '780' ) or  (row['diag_2'][0:3].zfill(3) == '781' ) or  (row['diag_2'][0:3].zfill(3) == '784' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 10 if (row['diag_2'][0:3].zfill(3) >= '240') and (row['diag_2'][0:3].zfill(3) <= '249' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 10 if (row['diag_2'][0:3].zfill(3) >= '251') and (row['diag_2'][0:3].zfill(3) <= '279' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 11 if (row['diag_2'][0:3].zfill(3) >= '680') and (row['diag_2'][0:3].zfill(3) <= '709' ) or  (row['diag_2'][0:3].zfill(3) == '782' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 12 if (row['diag_2'][0:3].zfill(3) >= '001') and (row['diag_2'][0:3].zfill(3) <= '139' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 13 if (row['diag_2'][0:3].zfill(3) >= '290') and (row['diag_2'][0:3].zfill(3) <= '319' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 13 if (row['diag_2'][0:1] >= 'E') and (row['diag_2'][0:1] <= 'V' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 13 if (row['diag_2'][0:3].zfill(3) >= '280') and (row['diag_2'][0:3].zfill(3) <= '289' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 13 if (row['diag_2'][0:3].zfill(3) >= '320') and (row['diag_2'][0:3].zfill(3) <= '359' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 13 if (row['diag_2'][0:3].zfill(3) >= '630') and (row['diag_2'][0:3].zfill(3) <= '679' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 13 if (row['diag_2'][0:3].zfill(3) >= '360') and (row['diag_2'][0:3].zfill(3) <= '389' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 13 if (row['diag_2'][0:3].zfill(3) >= '740') and (row['diag_2'][0:3].zfill(3) <= '759' ) else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: 0 if (row['diag_2'][0:3].zfill(3)  == '783' or row['diag_2'][0:3].zfill(3)  == '789') else row['d2'], axis=1)
df['d2'] = df.apply(lambda row: -1 if (row['diag_2'][0:1] == '?') else row['d2'], axis=1)                           _____no_output_____# Regrouping the second secondary diagnosis
df['d3'] = df.apply(lambda row: 1 if (row['diag_3'][0:3].zfill(3) >= '390') and (row['diag_3'][0:3].zfill(3) <= '459' ) or  (row['diag_3'][0:3].zfill(3) == '785' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 2 if (row['diag_3'][0:3].zfill(3) >= '460') and (row['diag_3'][0:3].zfill(3) <= '519' ) or  (row['diag_3'][0:3].zfill(3) == '786' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 3 if (row['diag_3'][0:3].zfill(3) >= '520') and (row['diag_3'][0:3].zfill(3) <= '579' ) or  (row['diag_3'][0:3].zfill(3) == '787' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 4 if (row['diag_3'][0:3].zfill(3) == '250') else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 5 if (row['diag_3'][0:3].zfill(3) >= '800') and (row['diag_3'][0:3].zfill(3) <= '999' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 6 if (row['diag_3'][0:3].zfill(3) >= '710') and (row['diag_3'][0:3].zfill(3) <= '739' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 7 if (row['diag_3'][0:3].zfill(3) .zfill(3)>= '580') and (row['diag_3'][0:3].zfill(3) <= '629' ) or  (row['diag_3'][0:3].zfill(3) == '788' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 8 if (row['diag_3'][0:3].zfill(3) >= '140') and (row['diag_3'][0:3].zfill(3) <= '239' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 9 if (row['diag_3'][0:3].zfill(3) >= '790') and (row['diag_3'][0:3].zfill(3) <= '799' ) or  (row['diag_3'][0:3].zfill(3) == '780' ) or  (row['diag_3'][0:3].zfill(3) == '781' ) or  (row['diag_3'][0:3].zfill(3) == '784' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 10 if (row['diag_3'][0:3].zfill(3) >= '240') and (row['diag_3'][0:3].zfill(3) <= '249' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 10 if (row['diag_3'][0:3].zfill(3) >= '251') and (row['diag_3'][0:3].zfill(3) <= '279' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 11 if (row['diag_3'][0:3].zfill(3) >= '680') and (row['diag_3'][0:3].zfill(3) <= '709' ) or  (row['diag_3'][0:3].zfill(3) == '782' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 12 if (row['diag_3'][0:3].zfill(3) >= '001') and (row['diag_3'][0:3].zfill(3) <= '139' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 13 if (row['diag_3'][0:3].zfill(3) >= '290') and (row['diag_3'][0:3].zfill(3) <= '319' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 13 if (row['diag_3'][0:1] >= 'E') and (row['diag_3'][0:1] <= 'V' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 13 if (row['diag_3'][0:3].zfill(3) >= '280') and (row['diag_3'][0:3].zfill(3) <= '289' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 13 if (row['diag_3'][0:3].zfill(3) >= '320') and (row['diag_3'][0:3].zfill(3) <= '359' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 13 if (row['diag_3'][0:3].zfill(3) >= '630') and (row['diag_3'][0:3].zfill(3) <= '679' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 13 if (row['diag_3'][0:3].zfill(3) >= '360') and (row['diag_3'][0:3].zfill(3) <= '389' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 13 if (row['diag_3'][0:3].zfill(3) >= '740') and (row['diag_3'][0:3].zfill(3) <= '759' ) else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: 0 if (row['diag_3'][0:3].zfill(3)  == '783' or row['diag_3'][0:3].zfill(3)  == '789') else row['d3'], axis=1)
df['d3'] = df.apply(lambda row: -1 if (row['diag_3'][0:1] == '?') else row['d3'], axis=1)                           _____no_output_____df.head(5)_____no_output_____

# Exploratory Data Analysis

Next we are going to perform an exploratory data analysis on the dataset to gain more insight and visualize some features of the the dataset._____no_output_____

# find the counts for each unique category
patient_counts = df['time_in_hospital'].value_counts()
patient_counts_____no_output_____# Distribution of number of hospital stays
fig = plt.figure(figsize=(16,8)) # define plot area
ax = fig.gca() # define axis
patient_counts.plot.bar(ax = ax) # Use the plot.bar method on the counts data frame
ax.set_title('Number of days in Hospital') # Give the plot a main title
ax.set_xlabel('Number of days in Hospital') # Set text for the x axis
ax.set_ylabel('Total Days')# Set text for y axis_____no_output_____

### Plot for counts of patients who are readmitted:
Plot for counts of patients who are readmitted or not readmited after an encounter. Note in the dataset the readmitted column has 3 labels readmitted within 30 days(<30), (>30) and No. We are going to group the readmitted patients together but in production this could be a business decision about how to group our patients based on cost of whether a patient is readmitted in less than 30 days or not. _____no_output_____

# find the counts for readmitted column(0 for no 1 for yes)
readmitted_counts = df['readmitted'].value_counts()
readmitted_counts_____no_output_____fig = plt.figure(figsize=(16,8))
ax = fig.gca()
readmitted_counts.plot.bar(ax = ax)
ax.set_title('Patient Readmissions')
ax.set_xlabel('Was patient Readmitted(0 for No 1 for Yes)')
ax.set_ylabel('Total Readmitted Patients')_____no_output_____

### How is the data distributed?
The plot tells us that of the total patient encounters 54864 were not readmitted and 46902 were readmitted. Since our model will be predicting only the days spent in the hospital we are going to drop all rows where a patient was not readmitted._____no_output_____

#drop patients who were not readmitted
df.drop(df[df.readmitted == 0].index, inplace=True)_____no_output_____

### How is the data distributed?
Plot some of the features distribution and write a summary of what the plot tells you._____no_output_____

print(df.shape)(46902, 55)

### Observation:
This is a bi-model distribution and the data can reveal a shift in the process.For Processes that display this distribution, it is normally understood that there are 2 independent sources of Variation that result in Peaks within the data._____no_output_____### Observation:
The total number of asteroid which are not hazardous are 34000 and hazardous are 4000

_____no_output_____

#sum all patients by ethinicity
df_ethinicity_counts = df["ethinicity"].value_counts()
df_ethinicity_counts.head()_____no_output_____fig = plt.figure(figsize=(12,8))
ax = fig.gca()
df['ethinicity'].plot.hist(ax = ax, bins=5, rwidth=.8, range=(0.5, 5.5))
ax.set_title('Histogram of ethinicity')
ax.set_xlabel('Ethinicity')
ax.set_ylabel('Total Patients')_____no_output_____

### Observation:
The plot shows that the caucasian group had highest number of LOS, followed by african-american and hispanics. suprising here as this is the normal distributing of our dataset._____no_output_____

# find the counts for each age group
age_group_counts = df['age_group'].value_counts()
age_group_counts_____no_output_____fig = plt.figure(figsize=(16,8))
ax = fig.gca()
age_group_counts.plot.bar(ax = ax)
ax.set_title('Number of Patients in each Age group')
ax.set_xlabel('Age Groups')
ax.set_ylabel('Total patients')_____no_output_____

### Observation:
The plot shows the distributing of age groups. The lower the group number the younger the age group._____no_output_____### Plot Length of Stay
Now Lets plot our label column which is the length of stay(LOS)_____no_output_____

# find the counts for each unique star rating
day_counts = df['time_in_hospital'].value_counts()
day_counts_____no_output_____fig = plt.figure(figsize=(15,8)) # define plot area
ax = fig.gca() # define axis
df['time_in_hospital'].plot.hist(ax = ax, bins=14, rwidth=.8, range=(0.5, 14.5))
ax.set_title('Histogram of LOS')
ax.set_xlabel('Days in  Hospital')
ax.set_ylabel('Total Days of Length of Stay(LOS)')_____no_output_____# average length of stay(los)
df_mean_los = df['time_in_hospital'].mean()
df_mean_los_____no_output_____

#### Key observations:
* Data has 569 observations and 32 columns.
* First field is 'id'.
* Second field, 'diagnosis', is an indicator of the actual diagnosis ('M' = Malignant; 'B' = Benign).
* There are 30 other numeric features available for prediction._____no_output_____

fig = plt.figure(figsize=(15,8))
ax = fig.gca()
df['num_procedures'].plot.hist(ax = ax, bins=6, rwidth=.8, range=(0.5, 6.5))
ax.set_title('Histogram of Procedures')
ax.set_xlabel('Number of procedures')
ax.set_ylabel('Total number of procedures')_____no_output_____fig = plt.figure(figsize=(18,10))
ax = fig.gca()
df[['number_diagnoses','time_in_hospital']].boxplot(by = 'time_in_hospital', ax = ax)
ax.set_title('Box plots of auto engine size by rating')
ax.set_xlabel('Number of days in Hospital')
ax.set_ylabel('Number of diagnoses')_____no_output_____fig = plt.figure(figsize=(20,12)) # define plot area
sns.set_style("darkgrid")
sns.boxplot(data = df, x = 'time_in_hospital', y = 'number_diagnoses', hue="age");_____no_output_____#plot length of stay with number of diagonoses with gender
fig = plt.figure(figsize=(20,12)) # define plot area
sns.set_style("darkgrid")
sns.boxplot(data = df, x = 'time_in_hospital', y = 'number_diagnoses', hue="gender");_____no_output_____

### Correlation between data:
Are any of the columns correlated?_____no_output_____

fig = plt.figure(figsize=(15,8))
cmap = sns.diverging_palette(0, 255, sep=1, n=258, as_cmap=True)
correlations = df[['time_in_hospital', 'num_lab_procedures', 'num_medications', 'number_outpatient', 'number_emergency', 'number_inpatient', 'number_diagnoses', 'd1', 'd2', 'd3','change',  'readmitted', 'age_group']].corr()
sns.heatmap(correlations, cmap=cmap)
plt.show()_____no_output_____

### Define the label column
Here we split the time in hospital column which will be our label. The original dataset has time_in_hospital up to 14 days but we are going to simplyify our model to 2 classes. Since the average days spent in hospital is 4.6 we are going to classify the patients into 2 classes; whether the patient will stay less than 5 days in in the hospital or more. Therefore our model will be a binary classifier. This can vary depending on the business problem our model is tying to solve. For example if we plan on using the model in a hospital setting we would probably want to know the approximate number of days the patient will stay so we can plan room availability accordingly. For insurance purposes we probaly have already planned the expenses for the average hospital stay and we just want want to know if a patient will stay more days. You get the idea, the model can be changed to a regression model depending on business problem.   _____no_output_____

#clean time_in_hospital column we give it 2 classes 0 for 0-5, 1 more than 5 days spent in hospital
df['label'] = df['time_in_hospital'].replace({0: 0, 1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 1, 7: 1, 8: 1, 9: 1, 10: 1, 11: 1, 12: 1, 13: 1, 14: 1})
  _____no_output_____

Here we drop the columns we are not going to use for our model._____no_output_____

df.drop(['time_in_hospital', 'patient_nbr','weight', 'race', 'encounter_id', 'age','admission_type_id','discharge_disposition_id', 'admission_source_id', 'payer_code', 'medical_specialty', 'diag_1', 'diag_2', 'diag_3', 'max_glu_serum', 'A1Cresult', 'glipizide'], axis=1, inplace=True)
#data = data.drop(['weight'],  axis=1, inplace=True)_____no_output_____# list the columns in the df
datalist = df.columns.tolist()
# Assign new position for label column(hazardous) as required by Sagemaker's algorithms
datalist.insert(0, datalist.pop(datalist.index('label')))
df = df.reindex(columns= datalist)
df.head(5)_____no_output_____df.isnull().sum()_____no_output_____print(df.shape)(46902, 39)
df.head(5)_____no_output_____df['d1'].replace('[^A-Z]', '')
df['d2'].replace('[^A-Z]', '')
df['d3'].replace('[^A-Z]', '')_____no_output_____

## Create Features and Labels
#### Split the data into 80% training, 10% validation and 10% testing._____no_output_____

# Randomly sort the data then split out first 60%, second 20%, and last 20% into training, validation, and test sets

rand_split = np.random.rand(len(df))
train_list = rand_split < 0.6
val_list = (rand_split >= 0.6) & (rand_split < 0.8)
test_list = rand_split >= 0.8
_____no_output_____data_train = df[train_list]
data_val = df[val_list]
data_test = df[test_list]
data_train.head()_____no_output_____#change dataframe to array types
train_y = data_train.iloc[:,0].as_matrix();
train_X = data_train.iloc[:,1:].as_matrix();

val_y = data_val.iloc[:,0].as_matrix();
val_X = data_val.iloc[:,1:].as_matrix();

test_y = data_test.iloc[:,0].as_matrix();
test_X = data_test.iloc[:,1:].as_matrix();
_____no_output_____

Now, we'll convert the datasets to the recordIO-wrapped protobuf format used by the Amazon SageMaker algorithms, and then upload this data to S3.  We'll start with training data. Note we change the data to float32 which is required by the Linear Learner algorithm._____no_output_____

train_file = 'linear_train.data'

f = io.BytesIO()
smac.write_numpy_to_dense_tensor(f, train_X.astype('float32'), train_y.astype('float32'))
f.seek(0)

boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'train', train_file)).upload_fileobj(f)_____no_output_____#verify the data
train_X[0]_____no_output_____

Next we'll convert and upload the validation dataset._____no_output_____

validation_file = 'linear_validation.data'

f = io.BytesIO()
smac.write_numpy_to_dense_tensor(f, val_X.astype('float32'), val_y.astype('float32'))
f.seek(0)

boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'validation', validation_file)).upload_fileobj(f)_____no_output_____

---
## Train

Now we can begin to specify our linear model.  Amazon SageMaker's Linear Learner actually fits many models in parallel, each with slightly different hyperparameters, and then returns the one with the best fit.  This functionality is automatically enabled.  We can influence this using parameters like:

- `num_models` to increase to total number of models run.  The specified parameters will always be one of those models, but the algorithm also chooses models with nearby parameter values in order to find a solution nearby that may be more optimal.  In this case, we're going to use the max of 32.
- `loss` which controls how we penalize mistakes in our model estimates.  For this case, let's use absolute loss as we haven't spent much time cleaning the data, and absolute loss will be less sensitive to outliers.
- `wd` or `l1` which control regularization.  Regularization can prevent model overfitting by preventing our estimates from becoming too finely tuned to the training data, which can actually hurt generalizability.  In this case, we'll leave these parameters as their default "auto" though.
- `binary_classifier` Here we set the type of model to use therefore we dont have to specify number of classes._____no_output_____### Specify container images used for training and hosting SageMaker's linear-learner_____no_output_____

# See 'Algorithms Provided by Amazon SageMaker: Common Parameters' in the SageMaker documentation for an explanation of these values.
from sagemaker.amazon.amazon_estimator import get_image_uri
container = get_image_uri(boto3.Session().region_name, 'linear-learner')_____no_output_____linear_job = 'DIABETIC-linear-' + time.strftime("%Y-%m-%d-%H-%M-%S", time.gmtime())

print("Job name is:", linear_job)

linear_training_params = {
    "RoleArn": role,
    "TrainingJobName": linear_job,
    "AlgorithmSpecification": {
        "TrainingImage": container,
        "TrainingInputMode": "File"
    },
    "ResourceConfig": {
        "InstanceCount": 1,
        "InstanceType": "ml.c4.2xlarge",
        "VolumeSizeInGB": 10
    },
    "InputDataConfig": [
        {
            "ChannelName": "train",
            "DataSource": {
                "S3DataSource": {
                    "S3DataType": "S3Prefix",
                    "S3Uri": "s3://{}/{}/train/".format(bucket, prefix),
                    "S3DataDistributionType": "ShardedByS3Key"
                }
            },
            "CompressionType": "None",
            "RecordWrapperType": "None"
        },
        {
            "ChannelName": "validation",
            "DataSource": {
                "S3DataSource": {
                    "S3DataType": "S3Prefix",
                    "S3Uri": "s3://{}/{}/validation/".format(bucket, prefix),
                    "S3DataDistributionType": "FullyReplicated"
                }
            },
            "CompressionType": "None",
            "RecordWrapperType": "None"
        }

    ],
    "OutputDataConfig": {
        "S3OutputPath": "s3://{}/{}/".format(bucket, prefix)
    },
    "HyperParameters": {
        "feature_dim": "38",
        "mini_batch_size": "250",
        "predictor_type": "binary_classifier",
        "epochs": "200",
        "num_models": "32",
        "loss": "logistic"
    },
    "StoppingCondition": {
        "MaxRuntimeInSeconds": 60 * 60
    }
}Job name is: DIABETIC-linear-2019-03-14-22-03-14

Now let's kick off our training job in SageMaker's distributed, managed training, using the parameters we just created.  Because training is managed, we don't have to wait for our job to finish to continue, but for this case, let's use boto3's 'training_job_completed_or_stopped' waiter so we can ensure that the job has been started._____no_output_____

%%time

region = boto3.Session().region_name
sm = boto3.client('sagemaker')

sm.create_training_job(**linear_training_params)

status = sm.describe_training_job(TrainingJobName=linear_job)['TrainingJobStatus']
print(status)
sm.get_waiter('training_job_completed_or_stopped').wait(TrainingJobName=linear_job)
if status == 'Failed':
    message = sm.describe_training_job(TrainingJobName=linear_job)['FailureReason']
    print('Training failed with the following error: {}'.format(message))
    raise Exception('Training job failed')InProgress
CPU times: user 161 ms, sys: 1.65 ms, total: 163 ms
Wall time: 16min 1s

---
## Host

Now that we've trained the linear algorithm on our data, let's setup a model which can later be hosted.  We will:
1. Point to the scoring container
1. Point to the model.tar.gz that came from training
1. Create the hosting model_____no_output_____

linear_hosting_container = {
    'Image': container,
    'ModelDataUrl': sm.describe_training_job(TrainingJobName=linear_job)['ModelArtifacts']['S3ModelArtifacts']
}

create_model_response = sm.create_model(
    ModelName=linear_job,
    ExecutionRoleArn=role,
    PrimaryContainer=linear_hosting_container)

print(create_model_response['ModelArn'])arn:aws:sagemaker:us-east-1:238416803993:model/diabetic-linear-2019-03-14-22-03-14

Once we've setup a model, we can configure what our hosting endpoints should be.  Here we specify:
1. EC2 instance type to use for hosting
1. Initial number of instances
1. Our hosting model name_____no_output_____

linear_endpoint_config = 'Diabetic-linear-endpoint-config-' + time.strftime("%Y-%m-%d-%H-%M-%S", time.gmtime())
print(linear_endpoint_config)
create_endpoint_config_response = sm.create_endpoint_config(
    EndpointConfigName=linear_endpoint_config,
    ProductionVariants=[{
        'InstanceType': 'ml.m4.xlarge',
        'InitialInstanceCount': 1,
        'ModelName': linear_job,
        'VariantName': 'AllTraffic'}])

print("Endpoint Config Arn: " + create_endpoint_config_response['EndpointConfigArn'])Diabetic-linear-endpoint-config-2019-03-14-22-19-18
Endpoint Config Arn: arn:aws:sagemaker:us-east-1:238416803993:endpoint-config/diabetic-linear-endpoint-config-2019-03-14-22-19-18

Now that we've specified how our endpoint should be configured, we can create them.  This can be done in the background, but for now let's run a loop that updates us on the status of the endpoints so that we know when they are ready for use._____no_output_____

%%time

linear_endpoint = 'Diabetic-linear-endpoint-config-' + time.strftime("%Y%m%d%H%M", time.gmtime())
print(linear_endpoint)
create_endpoint_response = sm.create_endpoint(
    EndpointName=linear_endpoint,
    EndpointConfigName=linear_endpoint_config)
print(create_endpoint_response['EndpointArn'])

resp = sm.describe_endpoint(EndpointName=linear_endpoint)
status = resp['EndpointStatus']
print("Status: " + status)

sm.get_waiter('endpoint_in_service').wait(EndpointName=linear_endpoint)

resp = sm.describe_endpoint(EndpointName=linear_endpoint)
status = resp['EndpointStatus']
print("Arn: " + resp['EndpointArn'])
print("Status: " + status)

if status != 'InService':
    raise Exception('Endpoint creation did not succeed')Diabetic-linear-endpoint-config-201903142219
arn:aws:sagemaker:us-east-1:238416803993:endpoint/diabetic-linear-endpoint-config-201903142219
Status: Creating
Arn: arn:aws:sagemaker:us-east-1:238416803993:endpoint/diabetic-linear-endpoint-config-201903142219
Status: InService
CPU times: user 199 ms, sys: 3.84 ms, total: 203 ms
Wall time: 7min 32s

## Predict
### Predict on Test Data

Now that we have our hosted endpoint, we can generate statistical predictions from it.  Let's predict on our test dataset to understand how accurate our model is.

There are many metrics to measure classification accuracy.  Common examples include include:
- Precision
- Recall
- F1 measure
- Area under the ROC curve - AUC
- Total Classification Accuracy
- Mean Absolute Error

For our example, we'll keep things simple and use total classification accuracy as our metric of choice. We will also evaluate  Mean Absolute  Error (MAE) as the linear-learner has been optimized using this metric, not necessarily because it is a relevant metric from an application point of view. We'll compare the performance of the linear-learner against a naive benchmark prediction which uses majority class observed in the training data set for prediction on the test data.

_____no_output_____### Function to convert an array to a csv_____no_output_____

def np2csv(arr):
    csv = io.BytesIO()
    np.savetxt(csv, arr, delimiter=',', fmt='%g')
    return csv.getvalue().decode().rstrip()_____no_output_____

Next, we'll invoke the endpoint to get predictions._____no_output_____

runtime= boto3.client('runtime.sagemaker')

payload = np2csv(test_X)
response = runtime.invoke_endpoint(EndpointName=linear_endpoint,
                                   ContentType='text/csv',
                                   Body=payload)
result = json.loads(response['Body'].read().decode())
test_pred = np.array([r['score'] for r in result['predictions']])
test_label = np.array([r['predicted_label'] for r in result['predictions']])_____no_output_____result_____no_output_____

Let's compare linear learner based mean absolute prediction errors from a baseline prediction which uses majority class to predict every instance._____no_output_____

test_mae_linear = np.mean(np.abs(test_y - test_pred))
test_mae_baseline = np.mean(np.abs(test_y - np.median(train_y))) ## training median as baseline predictor

print("Test MAE Baseline :", round(test_mae_baseline, 3))
print("Test MAE Linear:", round(test_mae_linear,3))
Test MAE Baseline : 0.306
Test MAE Linear: 0.331

Let's compare predictive accuracy using a classification threshold of 0.5 for the predicted and compare against the majority class prediction from training data set. This threshold can be adjusted later according to costs assigned to predicted labels for the model. This is alot of consideration should be given to provide business value._____no_output_____

test_pred_class = (test_pred > 0.5)+0;
test_pred_baseline = np.repeat(np.median(train_y), len(test_y))

prediction_accuracy = np.mean((test_y == test_pred_class))*100
baseline_accuracy = np.mean((test_y == test_pred_baseline))*100

print("Prediction Accuracy:", round(prediction_accuracy,1), "%")
print("Baseline Accuracy:", round(baseline_accuracy,1), "%")Prediction Accuracy: 75.9 %
Baseline Accuracy: 69.4 %

Plot the confusion matrix_____no_output_____

import json
from pandas.io.json import json_normalize

#create the scores and labels dataframes
dataset1 = pd.DataFrame({'score':test_pred[:]})
dataset2 = pd.DataFrame({'predictions':test_label[:]})
#merge the scores and labels dataframes
scores_labels = pd.concat([dataset1, dataset2], axis=1)
scores_labels.count()_____no_output_____confusion_matrix = pd.crosstab(np.round(scores_labels.score), scores_labels.predictions, margins=True)
confusion_matrix_____no_output_____

Note, due to randomized elements of the algorithm, you results may differ slightly.

Of the 2301 of patients, we've correctly predicted 2301 of them (true positives). And, we incorrectly predicted 1258 patients would stay more than 5 days who then ended up not doing so (false positives). There are also 0 patients who  that we predicted would not (false negatives).

An important point here is that because of the np.round() function above we are using a simple threshold (or cutoff) of 0.5. Our predictions from xgboost come out as continuous values between 0 and 1 and we force them into the binary classes that we began with. However, because a customer that churns is expected to cost the company more than proactively trying to retain a customer who we think might churn, we should consider adjusting this cutoff. That will almost certainly increase the number of false positives, but it can also be expected to increase the number of true positives and reduce the number of false negatives.

To get a rough intuition here, let's look at the continuous values of our predictions._____no_output_____

def plot_confusion_matrix(confusion_matrix, title='Confusion matrix', cmap=plt.cm.cool):
    plt.matshow(confusion_matrix, cmap=cmap) # imshow
    #plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(confusion_matrix.columns))
    plt.xticks(tick_marks, confusion_matrix.columns, rotation=45)
    plt.yticks(tick_marks, confusion_matrix.index)
    #plt.tight_layout()
    plt.ylabel(confusion_matrix.index.name)
    plt.xlabel(confusion_matrix.columns.name)

plot_confusion_matrix(confusion_matrix)_____no_output_____plt.hist(scores_labels.score, bins=6, rwidth=.9, range=(0.0, 1.0))
plt.show()_____no_output_____

###### Run the cell below to delete endpoint once you are done._____no_output_____

sm.delete_endpoint(EndpointName=linear_endpoint)_____no_output_____

---
## Conclusion

- Our  model does a average job of predicting length of hospital stays for admitted diabetic patients has an overall accuracy of close to 75%. We can re-run the model with different values of the hyper-parameters, loss functions etc and see if we get improved prediction. Re-running the model with further tweaks to these hyperparameters may provide more accurate out-of-sample predictions.
- We also did not do much feature engineering. We can create additional features by considering cross-product/intreaction of multiple features, squaring or raising higher powers of the features to induce non-linear effects, etc. If we expand the features using non-linear terms and interactions, we can then tweak the regulaization parameter to optimize the expanded model and hence generate improved forecasts.
- As a further extension, we can use many of non-linear models available through SageMaker such as XGBoost, MXNet etc.
- The main goal of the project will be part 2 where we deploy our model to provide real time predictions at any scale using AWS Lambda, S3, API Gateway and Chalice the flask like serverless framework from AWS.

## Part 2: Deploying Our Model for real time predictations

In part two we are going to set up our model using a AWS Lambda and S3 for serveless front end to offer predictions for a given patient in real time. _____no_output_____

from IPython.display import Image
from IPython.core.display import HTML
Image(url= "https://s3.amazonaws.com/imagescope/Diamed.jpeg")_____no_output_____


# Notebook from NLM-Reproducibility-Project/ReAnalysis-of-gene-pathways-between-TB-and-RA
Path: scripts/.ipynb_checkpoints/Meta Analysis-Yanqin-checkpoint.ipynb

# Read Matrix Data
Read in the gene expression data sets._____no_output_____

## Download microarray matrix files from GEO_____no_output_____setwd("~/NLM_Reproducibility_Workshop/tb_and_arthritis/data")_____no_output_____url <- c("ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE54nnn/GSE54992/matrix/GSE54992_series_matrix.txt.gz",
        "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE19nnn/GSE19435/matrix/GSE19435_series_matrix.txt.gz",
        "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE15nnn/GSE15573/matrix/GSE15573_series_matrix.txt.gz",
        "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE19nnn/GSE19444/matrix/GSE19444_series_matrix.txt.gz",
        "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE19nnn/GSE19435/matrix/GSE19435_series_matrix.txt.gz",
        "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE65nnn/GSE65517/matrix/GSE65517_series_matrix.txt.gz")_____no_output_____dest.file <- c("GSE54992_series_matrix.txt.gz","GSE19435_series_matrix.txt.gz","GSE15573_series_matrix.txt.gz",
              "GSE19444_series_matrix.txt.gz","GSE19435_series_matrix.txt.gz","GSE65517_series_matrix.txt.gz")_____no_output_____for (i in 1:length(url)){
utils::download.file(url[i], destfile=dest.file[i], mode="wb")
}_____no_output_____## Download and install different types of microarray annotation database files_____no_output_____install.packages("BiocManager")
BiocManager::install("plyr")
BiocManager::install("annotate")
BiocManager::install("illuminaHumanv4.db")
BiocManager::install("hgu133plus2.db")
BiocManager::install("illuminaHumanv2.db")
BiocManager::install("illuminaHumanv3.db")
_____no_output_____library(annotate)
library(illuminaHumanv3.db)
library(illuminaHumanv2.db)
library(hgu133plus2.db)
library(illuminaHumanv4.db)
library(plyr)

## setup working directory
working.dir <- "~/NLM_Reproducibility_Workshop/tb_and_arthritis/working"
setwd(working.dir)

## read into matrix which was downloaded from GEO
dat.v2 <- read.delim("GSE15573_series_matrix_networkanalyst.txt")

## annotate microarray problesets
id.v2 <- select(illuminaHumanv2.db, as.character(dat.v2[2:nrow(dat.v2),1]),
                c("SYMBOL","ENTREZID", "GENENAME"))

dat.v3.1 <- read.delim("GSE19435_series_matrix_networkanalyst.txt")
id.v3.1 <- select(illuminaHumanv3.db, as.character(dat.v3.1[2:nrow(dat.v3.1),1]),
                c("SYMBOL","ENTREZID", "GENENAME"))

dat.v3.2 <- read.delim("GSE19444_series_matrix_networkanalyst.txt")
id.v3.2 <- select(illuminaHumanv3.db, as.character(dat.v3.2[2:nrow(dat.v3.2),1]),
                  c("SYMBOL","ENTREZID", "GENENAME"))

dat.v4 <- read.delim("GSE65517_series_matrix_networkanalyst.txt")
id.v4 <- select(illuminaHumanv4.db, as.character(dat.v4[2:nrow(dat.v4),1]),
                c("SYMBOL","ENTREZID", "GENENAME"))

dat.plus2 <- read.delim("GSE4588_series_matrix_networkanalyst.txt")
id.plus2 <- select(hgu133plus2.db, as.character(dat.plus2[2:nrow(dat.plus2),1]),
                c("SYMBOL","ENTREZID", "GENENAME"))

dat.plus2.2 <- read.delim("GSE54992_series_matrix_networkanalyst.txt")
id.plus2.2 <- select(hgu133plus2.db, as.character(dat.plus2.2[2:nrow(dat.plus2.2),1]),
                c("SYMBOL","ENTREZID", "GENENAME"))

## merge data matrix with its annotation table by probeID
colnames(dat.v2)[1]=colnames(id.v2)[1]
dat.v2.all <- join(dat.v2,id.v2,by="PROBEID")

colnames(dat.v3.1)[1]=colnames(id.v3.1)[1]
dat.v3.1.all <- join(dat.v3.1,id.v3.1,by="PROBEID")

colnames(dat.v3.2)[1]=colnames(id.v3.2)[1]
dat.v3.2all <- join(dat.v3.2,id.v3.2,by="PROBEID")

colnames(dat.v4)[1]=colnames(id.v4)[1]
dat.v4.all <- join(dat.v4,id.v4,by="PROBEID")

colnames(dat.plus2)[1]=colnames(id.plus2)[1]
dat.plus2.all <- join(dat.plus2,id.plus2,by="PROBEID")

colnames(dat.plus2.2)[1]=colnames(id.plus2.2)[1]
dat.plus2.2.all <- join(dat.plus2.2,id.plus2.2,by="PROBEID")Loading required package: AnnotationDbi
Loading required package: stats4
Loading required package: BiocGenerics
Loading required package: parallel

Attaching package: ‘BiocGenerics’

The following objects are masked from ‘package:parallel’:

    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
    clusterExport, clusterMap, parApply, parCapply, parLapply,
    parLapplyLB, parRapply, parSapply, parSapplyLB

The following objects are masked from ‘package:stats’:

    IQR, mad, sd, var, xtabs

The following objects are masked from ‘package:base’:

    anyDuplicated, append, as.data.frame, basename, cbind, colnames,
    dirname, do.call, duplicated, eval, evalq, Filter, Find, get, grep,
    grepl, intersect, is.unsorted, lapply, Map, mapply, match, mget,
    order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
    rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply,
    union, unique, unsplit, which, which.max, which.min

Loading required package: Biobase
Welcome to Bioconductor

    Vignettes contain introductory material; view with
    'browseVignettes()'. To cite Bioconductor, see
    'citation("Biobase")', and for packages 'citation("pkgname")'.

Loading required package: IRanges
Loading required package: S4Vectors

Attaching package: ‘S4Vectors’

The following object is masked from ‘package:base’:

    expand.grid

Loading required package: XML
Loading required package: org.Hs.eg.db

Attaching package: ‘plyr’

The following object is masked from ‘package:IRanges’:

    desc

The following object is masked from ‘package:S4Vectors’:

    rename

'select()' returned 1:many mapping between keys and columns
'select()' returned 1:many mapping between keys and columns
'select()' returned 1:many mapping between keys and columns
'select()' returned 1:many mapping between keys and columns
'select()' returned 1:many mapping between keys and columns
'select()' returned 1:many mapping between keys and columns
'select()' returned 1:many mapping between keys and columns

# Sample Filtering
The paper used some inclusion criteria to select samples from each study. Samples without class labels are thus removed from further analyses._____no_output_____

datasets <- list(dat.v2.all,dat.v3.1.all,dat.v3.2.all,dat.v4.all,dat.plus2.all,dat.plus2.2.all)
for (i in 1:length(datasets)) {
    dataset <- datasets[1]
    dataset <- dataset[which(is.na(dataset[1,])),]
    print(ncol(datasets[i]))
}_____no_output_____getwd()_____no_output_____setwd("/home/ubuntu/NLM_Reproducibility_Workshop/tb_and_arthritis/data")_____no_output_____library(limma)_____no_output_____source("http://bioconductor.org/biocLite.R")
biocLite("affy","vsn")Bioconductor version 3.7 (BiocInstaller 1.30.0), ?biocLite for help
A newer version of Bioconductor is available for this version of R,
  ?BiocUpgrade for help
BioC_mirror: https://bioconductor.org
Using Bioconductor 3.7 (BiocInstaller 1.30.0), R 3.6.0 (2019-04-26).
Installing package(s) ‘affy’
installation path not writeable, unable to update packages: boot, class,
  cluster, codetools, foreign, KernSmooth, lattice, MASS, Matrix, mgcv, nlme,
  nnet, spatial, survival
library(vsn)Loading required package: Biobase
Loading required package: BiocGenerics
Loading required package: parallel

Attaching package: ‘BiocGenerics’

The following objects are masked from ‘package:parallel’:

    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
    clusterExport, clusterMap, parApply, parCapply, parLapply,
    parLapplyLB, parRapply, parSapply, parSapplyLB

The following object is masked from ‘package:limma’:

    plotMA

The following objects are masked from ‘package:stats’:

    IQR, mad, sd, var, xtabs

The following objects are masked from ‘package:base’:

    anyDuplicated, append, as.data.frame, basename, cbind, colnames,
    dirname, do.call, duplicated, eval, evalq, Filter, Find, get, grep,
    grepl, intersect, is.unsorted, lapply, Map, mapply, match, mget,
    order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
    rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply,
    union, unique, unsplit, which, which.max, which.min

Welcome to Bioconductor

    Vignettes contain introductory material; view with
    'browseVignettes()'. To cite Bioconductor, see
    'citation("Biobase")', and for packages 'citation("pkgname")'.

Registered S3 methods overwritten by 'ggplot2':
  method         from
  [.quosures     rlang
  c.quosures     rlang
  print.quosures rlang
library(ggplot2)_____no_output_____# Read into meta files
dat <- read.delim("merged.dataset.txt")
dim(dat)
head(dat)
class(dat)_____no_output_____dat.num <- apply(dat[-1,],2,as.numeric)
dim(dat.num)_____no_output_____grp <- c("Healthy","TB","RA")
grp.all <- c(which(dat[1,]==grp[1]),which(dat[1,]==grp[2]),which(dat[1,]==grp[3]))
length(grp.all)
dat.num.rord <- dat.num[,grp.all]
rownames(dat.num.rord)=rownames(dat)[-1]
col <- rep(c('green','red','blue'),c(length(which(dat[1,]==grp[1])),
length(which(dat[1,]==grp[2])),length(which(dat[1,]==grp[3]))))_____no_output_____# normalize crossing all batches samples
dat.norm <- normalizeVSN(dat.num.rord)_____no_output_____head(dat.norm)_____no_output_____boxplot(dat.norm)_____no_output_____BiocManager::install("affyPLM")

library(affyPLM)Bioconductor version 3.9 (BiocManager 1.30.4), R 3.6.0 (2019-04-26)
Installing package(s) 'affyPLM'
also installing the dependencies ‘XVector’, ‘Biostrings’, ‘gcrma’

installation path not writeable, unable to update packages: boot, class,
  cluster, codetools, foreign, KernSmooth, lattice, MASS, Matrix, mgcv, nlme,
  nnet, spatial, survival
Update old packages: 'affy', 'affyio', 'vsn', 'zlibbioc'
Loading required package: affy
Loading required package: gcrma
Loading required package: preprocessCore
cl <- rep(c("green","red","blue"),c(length(which(dat[1,]==grp[1])),
length(which(dat[1,]==grp[2])),length(which(dat[1,]==grp[3]))))_____no_output_____par(mfrow=c(2,1))
boxplot(dat.num.rord, col=cl)
boxplot(dat.norm, col=cl)_____no_output_____dat.norm.pca <- prcomp(dat.norm)_____no_output_____summary(dat.norm.pca)_____no_output_____str(dat.norm.pca)List of 5
 $ sdev    : num [1:141] 18.65 6.96 5 1.95 1.77 ...
 $ rotation: num [1:141, 1:141] 0.0856 0.0873 0.0879 0.0874 0.0895 ...
  ..- attr(*, "dimnames")=List of 2
  .. ..$ : chr [1:141] "GSM389705" "GSM389707" "GSM389709" "GSM389710" ...
  .. ..$ : chr [1:141] "PC1" "PC2" "PC3" "PC4" ...
 $ center  : Named num [1:141] -0.783 -0.794 -0.791 -0.793 -0.791 ...
  ..- attr(*, "names")= chr [1:141] "GSM389705" "GSM389707" "GSM389709" "GSM389710" ...
 $ scale   : logi FALSE
 $ x       : num [1:18205, 1:141] -20.8 -21.9 -20.8 -19.3 -15.2 ...
  ..- attr(*, "dimnames")=List of 2
  .. ..$ : chr [1:18205] "A1CF" "A2M" "A2ML1" "A4GALT" ...
  .. ..$ : chr [1:141] "PC1" "PC2" "PC3" "PC4" ...
 - attr(*, "class")= chr "prcomp"
library(devtools)
install_github("vqv/ggbiplot")
library(ggbiplot)_____no_output_____autoplot(prcomp(dat.norm), scale = 0)_____no_output_____ggbiplot(dat.norm.pca)_____no_output_____dat.num.rord.pca <- prcomp(dat.num.rord, center = TRUE, scale. = TRUE)_____no_output_____ggbiplot(dat.num.rord.pca)_____no_output_____dim(dat.num.rord)_____no_output_____BiocManager::install("FactoMineR")
library("FactoMineR")Bioconductor version 3.9 (BiocManager 1.30.4), R 3.6.0 (2019-04-26)
Installing package(s) 'FactoMineR'
also installing the dependencies ‘ellipsis’, ‘rematch’, ‘forcats’, ‘cellranger’, ‘zip’, ‘SparseM’, ‘MatrixModels’, ‘sp’, ‘haven’, ‘readxl’, ‘openxlsx’, ‘carData’, ‘abind’, ‘pbkrtest’, ‘quantreg’, ‘maptools’, ‘rio’, ‘car’, ‘ellipse’, ‘flashClust’, ‘leaps’, ‘scatterplot3d’

installation path not writeable, unable to update packages: boot, class,
  cluster, codetools, foreign, KernSmooth, lattice, MASS, Matrix, mgcv, nlme,
  nnet, spatial, survival
Update old packages: 'affy', 'affyio', 'vsn', 'zlibbioc'
dat.pca <- PCA(t(dat.num.rord),graph=FALSE)_____no_output_____dat.norm.pca <- PCA(t(dat.norm), graph = FALSE)_____no_output_____fviz_pca_ind(dat.norm.pca,  label="none")_____no_output_____plot(dat.pca)_____no_output_____plot(dat.norm.pca)_____no_output_____## Combat normalization
BiocManager::install("sva")Bioconductor version 3.9 (BiocManager 1.30.4), R 3.6.0 (2019-04-26)
Installing package(s) 'sva'
also installing the dependencies ‘formatR’, ‘lambda.r’, ‘futile.options’, ‘futile.logger’, ‘snow’, ‘genefilter’, ‘BiocParallel’, ‘matrixStats’

installation path not writeable, unable to update packages: boot, class,
  cluster, codetools, foreign, KernSmooth, lattice, MASS, Matrix, mgcv, nlme,
  nnet, spatial, survival
Update old packages: 'affy', 'affyio', 'vsn', 'zlibbioc'
library(sva)Loading required package: mgcv
Loading required package: nlme
This is mgcv 1.8-28. For overview type 'help("mgcv-package")'.
Loading required package: genefilter
Loading required package: BiocParallel
#modcombat = model.matrix(~1, data=dat.num.rord)
_____no_output_____head(dat)[,1:9]_____no_output_____newdata = ComBat(dat=dat[-1,], batch=grp, par.prior = TRUE)Found3batches
Adjusting for0covariate(s) or covariate level(s)


# Notebook from IsabelAverill/Scipy2018---Tensorflow
Path: extras/amld/notebooks/solutions/1_qd_data.ipynb

# QuickDraw Data

If machine learning is rocket science then data is your fuel! So before
doing anything we will have a close look at the data available and spend
some time bringing it into the "right" form (i.e.
[tf.train.Example](https://www.tensorflow.org/versions/r1.0/api_docs/python/tf/train/Example)).

That's why we start by spending quite a lot on this notebook, downloading
the data, understanding it, and transforming it into the right format for
Tensorflow.

The data used in this workshop is taken from Google's quickdraw (click on
the images to see loads of examples):

https://quickdraw.withgoogle.com/data_____no_output_____

from __future__ import division, print_function

import base64, io, itertools, functools, json, os, random, re, textwrap, time
import numpy as np
import tensorflow as tf
from matplotlib import pyplot
from PIL import Image, ImageDraw
from IPython import display
from six.moves.urllib import request
from xml.dom import minidom

!mkdir -p _derived  # (Needed for running in Colab environment.)
%matplotlib inline
# Always make sure you are using running the expected version.
# There are considerable differences between versions...
# Tested with 1.4.1 and 1.7.0
tf.__version_______no_output_____

# 1 Get the data

In this section we download a set of raw data files from the web._____no_output_____

# Retrieve list of classes.

def list_bucket(bucket, regexp='.*'):
    """Returns a (filtered) list of Keys in specified GCE bucket."""
    keys = []
    fh = request.urlopen('https://storage.googleapis.com/%s' % bucket)
    content = minidom.parseString(fh.read())
    for e in content.getElementsByTagName('Contents'):
        key = e.getElementsByTagName('Key')[0].firstChild.data
        if re.match(regexp, key):
            keys.append(key)
    return keys

all_ndjsons = list_bucket('quickdraw_dataset', '.*ndjson$')
print('available: (%d)' % len(all_ndjsons))
print('\n'.join(textwrap.wrap(
    ' '.join([key.split('/')[-1].split('.')[0] for key in all_ndjsons]),
    width=100)))available: (309)
The Eiffel Tower The Great Wall of China The Mona Lisa aircraft carrier airplane alarm clock
ambulance angel animal migration ant anvil apple arm asparagus axe backpack banana bandage barn
baseball bat baseball basket basketball bat bathtub beach bear beard bed bee belt bench bicycle
binoculars bird birthday cake blackberry blueberry book boomerang bottlecap bowtie bracelet brain
bread bridge broccoli broom bucket bulldozer bus bush butterfly cactus cake calculator calendar
camel camera camouflage campfire candle cannon canoe car carrot castle cat ceiling fan cell phone
cello chair chandelier church circle clarinet clock cloud coffee cup compass computer cookie cooler
couch cow crab crayon crocodile crown cruise ship cup diamond dishwasher diving board dog dolphin
donut door dragon dresser drill drums duck dumbbell ear elbow elephant envelope eraser eye
eyeglasses face fan feather fence finger fire hydrant fireplace firetruck fish flamingo flashlight
flip flops floor lamp flower flying saucer foot fork frog frying pan garden hose garden giraffe
goatee golf club grapes grass guitar hamburger hammer hand harp hat headphones hedgehog helicopter
helmet hexagon hockey puck hockey stick horse hospital hot air balloon hot dog hot tub hourglass
house plant house hurricane ice cream jacket jail kangaroo key keyboard knee knife ladder lantern
laptop leaf leg light bulb lighter lighthouse lightning line lion lipstick lobster lollipop mailbox
map marker matches megaphone mermaid microphone microwave monkey moon mosquito motorbike mountain
mouse moustache mouth mug mushroom nail necklace nose ocean octagon octopus onion oven owl paint can
paintbrush palm tree panda pants paper clip parachute parrot passport peanut pear peas pencil
penguin piano pickup truck picture frame pig pillow pineapple pizza pliers police car pond pool
popsicle postcard potato power outlet purse rabbit raccoon radio rain rainbow rake remote control
rhinoceros rifle river roller coaster rollerskates sailboat sandwich saw saxophone school bus
scissors scorpion screwdriver sea turtle see saw shark sheep shoe shorts shovel sink skateboard
skull skyscraper sleeping bag smiley face snail snake snorkel snowflake snowman soccer ball sock
speedboat spider spoon spreadsheet square squiggle squirrel stairs star steak stereo stethoscope
stitches stop sign stove strawberry streetlight string bean submarine suitcase sun swan sweater
swing set sword syringe t-shirt table teapot teddy-bear telephone
# Store all data locally in this directory.
data_path = '../data'

# Mini group of two animals.
pets = ['cat', 'dog']

# Somewhat larger group of zoo animals.
zoo = ['elephant', 'giraffe', 'kangaroo', 'lion', 'monkey', 'panda',
       'penguin', 'rhinoceros', 'tiger', 'zebra']

# Even larger group of all animals.
animals = ['bat', 'bird', 'butterfly', 'camel', 'cat', 'cow', 'crab',
           'crocodile', 'dog', 'dolphin', 'duck', 'elephant', 'fish',
           'frog', 'giraffe', 'hedgehog', 'horse', 'kangaroo', 'lion',
           'lobster', 'monkey', 'mosquito', 'mouse', 'octopus', 'owl',
           'panda', 'parrot', 'penguin', 'pig', 'rabbit', 'raccoon',
           'rhinoceros', 'scorpion', 'sea turtle', 'shark', 'sheep',
           'snail', 'spider', 'squirrel', 'teddy-bear', 'tiger',
           'whale', 'zebra']

# Create your own group -- the more classes you include the more challenging
# the classification task will be...

# Choose one of above groups for remainder of workshop.
# Note: This will result in ~100MB of download per class.
classes, classes_name = zoo, 'zoo'_____no_output_____# Download above chosen group.

def valid_ndjson(filename):
    """Checks presence + completeness of .ndjson file."""
    try:
        json.loads(open(filename).readlines()[-1])
        return True
    except (ValueError, IOError):
        return False

def retrieve(bucket, key, filename):
    """Returns a file specified by its Key from a GCE bucket."""
    url = 'https://storage.googleapis.com/%s/%s' % (bucket, key)
    if not os.path.isfile(filename):
        request.urlretrieve(url=url, filename=filename)
    while not valid_ndjson(filename):
        print('*** Corrupted download (%.2f MB), retrying...' % (os.path.getsize(filename) / 2.**20))
        request.urlretrieve(url=url, filename=filename)

if not os.path.exists(data_path):
    os.mkdir(data_path)

print('\n%d classes:' % len(classes))

for name in classes:
    print(name, end=' ')
    dst = '%s/%s.ndjson' % (data_path, name)
    retrieve('quickdraw_dataset', 'full/simplified/%s.ndjson' % name, dst)
    print('%.2f MB' % (os.path.getsize(dst) / 2.**20))

print('\nDONE :)')
10 classes:
elephant 70.86 MB
giraffe 58.27 MB
kangaroo 96.06 MB
lion 90.47 MB
monkey 91.05 MB
panda 97.04 MB
penguin 137.90 MB
rhinoceros 92.20 MB
tiger 86.09 MB
zebra 92.10 MB

DONE :)

# 2 Inspect the data

What is the format of the downloaded files?_____no_output_____

# So let's check out the downloaded files...
!ls -lh $data_pathtotal 1867888
-rw-r--r--   1 andreassteiner  staff   322B Dec 20 07:36 README.md
drwxr-xr-x  33 andreassteiner  staff   1.1K Jan 21 15:56 dataset_img
drwxr-xr-x  33 andreassteiner  staff   1.1K Jan 22 12:10 dataset_img_70k
drwxr-xr-x  33 andreassteiner  staff   1.1K Jan 21 16:40 dataset_stroke
-rw-r--r--   1 andreassteiner  staff    71M Dec 19 07:43 elephant.ndjson
-rw-r--r--   1 andreassteiner  staff    58M Dec 19 07:43 giraffe.ndjson
-rw-r--r--   1 andreassteiner  staff    96M Dec 19 07:43 kangaroo.ndjson
-rw-r--r--   1 andreassteiner  staff    90M Dec 19 07:44 lion.ndjson
-rw-r--r--   1 andreassteiner  staff    91M Dec 19 07:44 monkey.ndjson
-rw-r--r--   1 andreassteiner  staff    97M Dec 19 07:44 panda.ndjson
-rw-r--r--   1 andreassteiner  staff   138M Dec 19 07:47 penguin.ndjson
-rw-r--r--   1 andreassteiner  staff    92M Dec 19 07:47 rhinoceros.ndjson
-rw-r--r--   1 andreassteiner  staff    86M Dec 19 07:48 tiger.ndjson
-rw-r--r--   1 andreassteiner  staff    92M Dec 19 07:48 zebra.ndjson
drwxr-xr-x  33 andreassteiner  staff   1.1K Jan 21 15:47 zoo_img_01
drwxr-xr-x  33 andreassteiner  staff   1.1K Dec 19 08:01 zoo_img_bytes_70k
drwxr-xr-x  33 andreassteiner  staff   1.1K Jan 21 15:38 zoo_img_bytes_7k
drwxr-xr-x  33 andreassteiner  staff   1.1K Jan 21 15:31 zoo_img_int64_7k
# What is the NDJSON file format?
# Seems to be one JSON dictionary per line...
path = sorted(tf.gfile.Glob(os.path.join(data_path, '*.ndjson')))[1]
print(open(path).read()[:1000] + '...'){"word":"giraffe","countrycode":"BR","timestamp":"2017-01-26 17:23:19.3785 UTC","recognized":true,"key_id":"5807561943023616","drawing":[[[25,9,0,1,13,35],[32,46,59,65,67,47]],[[64,47,36,25,25,41,53,66,69],[14,15,21,37,42,56,59,41,24]],[[20,39],[8,23]],[[51,52],[0,11]],[[56,46,46,55],[55,91,127,183]],[[74,70,73],[58,95,172]],[[53,79,100,126,149,150,138,131,96,47,39,40],[187,178,174,173,178,190,203,207,215,203,190,182]],[[54,36,22],[204,228,253]],[[73,49],[220,255]],[[118,119,132],[205,223,252]],[[142,143,159],[195,206,238]],[[143,161,165,169,166],[181,168,169,200,214]]]}
{"word":"giraffe","countrycode":"FR","timestamp":"2017-01-29 13:59:05.33086 UTC","recognized":true,"key_id":"5747753718120448","drawing":[[[34,35,45,59,65,67,69,74,79,81,79,81,88,96,99,99,103,101,104,121,129,127,137,151,158,166,163,155,177,198,191,161,115,82,65,56,53,56],[101,249,246,231,218,184,179,179,181,186,220,244,245,230,215,183,199,225,249,255,251,187,203,242,246,225,207,186,148,132,129,133,117,98,85,71,54,20]],...
# Parse single line.
data_json = json.loads(open(path).readline())
data_json.keys()_____no_output_____# So we have some meta information...
for k, v in data_json.items():
    if k != 'drawing':
        print('%20s   ->   %s' % (k, v))                word   ->   giraffe
         countrycode   ->   BR
           timestamp   ->   2017-01-26 17:23:19.3785 UTC
          recognized   ->   True
              key_id   ->   5807561943023616
# ...and the actual drawing.
drawing = data_json['drawing']
# The drawing consists of a series of strokes:
[np.array(stroke).shape for stroke in drawing]_____no_output_____# Draw the image -- the strokes all have have shape (2, n)
# so the first index seems to be x/y coordinate:
for stroke in drawing:
    pyplot.plot(np.array(stroke[0]), -np.array(stroke[1]))
# Would YOU recognize this drawing successfully?_____no_output_____%%time
# Some more code to load many sketches at once.
# Let's ignore the difficult "unrecognized" sketches for now...
# (i.e. unrecognized by the official quickdraw classifier)

def convert(line):
    """Converts single line to JSON + converts 'drawing' to list of np.array."""
    d = json.loads(line)
    d['drawing'] = [np.array(stroke) for stroke in d['drawing']]
    return d

def loaditer(name, unrecognized=False):
    """Returns iterable of drawings in specified file.

    Args:
      name: Name of the downloaded object (e.g. "elephant").
      unrecognized: Whether to include drawings that were not recognized
          by Google AI (i.e. the hard ones).
    """
    for line in open('%s/%s.ndjson' % (data_path, name)):
        d = convert(line)
        if d['recognized'] or unrecognized:
            yield d

def loadn(name, n, unrecognized=False):
    """Returns list of drawings.

    Args:
      name: Name of the downloaded object (e.g. "elephant").
      n: Number of drawings to load.
      unrecognized: Whether to include drawings that were not recognized
          by Google AI (i.e. the hard ones).
    """
    it = loaditer(name, unrecognized=unrecognized)
    return list(itertools.islice(it, 0, n))

print('loading some "%s"...' % classes[0])
sample = loadn(classes[0], 100)loading some "elephant"...
CPU times: user 5.81 ms, sys: 1.76 ms, total: 7.57 ms
Wall time: 6.15 ms
# Some more drawings...
pyplot.figure(figsize=(10, 10))
n = 3
for x in range(n):
    for y in range(n):
        i = x * n + y
        pyplot.subplot(n, n, i + 1)
        for stroke in sample[i]['drawing']:
            pyplot.plot(np.array(stroke[0]), -np.array(stroke[1]))_____no_output_____

# 3 Rasterize

Idea: After converting the raw drawing data into rasterized images, we can
use [MNIST](https://www.tensorflow.org/get_started/mnist/beginners)-like
image processing to classify the drawings._____no_output_____

%%writefile _derived/1_json_to_img.py
# (Written into separate file for sharing between notebooks.)

# Function that converts drawing (specified by individual strokes)
# to a rendered black/white image.

def json_to_img(drawing, img_sz=64, lw=3, maximize=True):
    img = Image.new('L', (img_sz, img_sz))
    draw = ImageDraw.Draw(img)
    lines = np.array([
        stroke[0:2, i:i+2]
        for stroke in drawing['drawing']
        for i in range(stroke.shape[1] - 1)
    ], dtype=np.float32)
    if maximize:
        for i in range(2):
            min_, max_ = lines[:,i,:].min() * 0.95, lines[:,i,:].max() * 1.05
            lines[:,i,:] = (lines[:,i,:] - min_) / max(max_ - min_, 1)
    else:
        lines /= 1024
    for line in lines:
        draw.line(tuple(line.T.reshape((-1,)) * img_sz), fill='white', width=lw)
    return imgOverwriting _derived/1_json_to_img.py
# (Load code from previous cell -- make sure to have executed above cell first.)
%run -i _derived/1_json_to_img.py

# Show some examples.

def showimg(img):
    if isinstance(img, np.ndarray):
        img = Image.fromarray(img, 'L')
    b = io.BytesIO()
    img.convert('RGB').save(b, format='png')
    enc = base64.b64encode(b.getvalue()).decode('utf-8')
    display.display(display.HTML(
        '' % enc))

# Fetch some images + shuffle order.
rows, cols = 10, 10
n_per_class = rows * cols // len(classes) + 1
drawings_matrix = [loadn(name, rows*cols) for name in classes]
drawings_list = functools.reduce(lambda x, y: x + y, drawings_matrix, [])
drawings_list = np.random.permutation(drawings_list)

# Create mosaic of rendered images.
lw = 4
img_sz = 64
tableau = np.zeros((img_sz * rows, img_sz * cols), dtype=np.uint8)
for y in range(rows):
    for x in range(cols):
        i = y * rows + x
        img = json_to_img(drawings_list[i], img_sz=img_sz, lw=lw, maximize=True)
        tableau[y*img_sz:(y+1)*img_sz, x*img_sz:(x+1)*img_sz] = np.asarray(img)

showimg(tableau);_____no_output_____

# 4 tf.train.Example data format

Tensorflow's "native" format for data storage is the `tf.train.Example`
[protocol buffer](https://en.wikipedia.org/wiki/Protocol_Buffers).

In this section we briefly explore the API needed to access the data
inside the `tf.train.Example` protocol buffer. It's **not necessary** to read
through the
[Python API documentation](https://developers.google.com/protocol-buffers/docs/pythontutorial)._____no_output_____

# Create a new (empty) instance.
example = tf.train.Example()
# (empty example will print nothing)
print(example)
# An example contains a map from feature name to "Feature".
# Every "Feature" contains a list of elements of the same
# type, which is one of:
# - bytes_list (similar to Python's "str")
# - float_list (float number)
# - int64_list (integer number)

# These values can be accessed as follows (no need to understand
# details):

# Add float value "3.1416" to feature "magic_numbers"
example.features.feature['magic_numbers'].float_list.value.append(3.1416)
# Add some more values to the float list "magic_numbers".
example.features.feature['magic_numbers'].float_list.value.extend([2.7183, 1.4142, 1.6180])

### YOUR ACTION REQUIRED:
# Create a second feature named "adversaries" and add the elements
# "Alice" and "Bob".
example.features.feature['adversaries'].bytes_list.value.extend([b'Alice', b'Bob']) #example.features.feature['adversaries'].

# This will now print a serialized representation of our protocol buffer
# with features "magic_numbers" and "adversaries" set...
print(example)

# .. et voila : that's all you need to know about protocol buffers
# for this workshop.features {
  feature {
    key: "adversaries"
    value {
      bytes_list {
        value: "Alice"
        value: "Bob"
      }
    }
  }
  feature {
    key: "magic_numbers"
    value {
      float_list {
        value: 3.1415998935699463
        value: 2.7183001041412354
        value: 1.414199948310852
        value: 1.6180000305175781
      }
    }
  }
}

# 5 Create dataset

Now let's create a "dataset" of `tf.train.Example`
[protocol buffers](https://developers.google.com/protocol-buffers/) ("protos").

A single example contains all the information for a drawing (i.e. rasterized
image, label, and meta information).

A dataset consists of non-overlapping sets of examples that will be used for
training and evaluation of the classifier (the "test" set will be used for the
final evaluation). Because these files can quickly become very large, we
"shard" them into multiple smaller files of equal size._____no_output_____

# Let's first check how many [recognized=True] examples we have in each class.
# Depending on your choice of classes you could generate up to 200k examples...
for name in classes:
    print(name, len(list(open('%s/%s.ndjson' % (data_path, name)))), 'recognized', len(list(loaditer(name))))elephant 126969 recognized 104986
giraffe 127182 recognized 122396
kangaroo 174470 recognized 152511
lion 120949 recognized 98976
monkey 127633 recognized 105650
panda 113613 recognized 94814
penguin 253791 recognized 231826
rhinoceros 188484 recognized 166539
tiger 121067 recognized 110177
zebra 144608 recognized 127159
# Helper code to create sharded recordio files.
# (No need to read through this.)

# Well... Since you continue to read through this cell, I could as
# well explain in more detail what it is about :-)
# Because we work with large amounts of data, we will create "sharded"
# files, that is, we split a single dataset into a number of files, like
# train-00000-of-00005, ..., train-00004-of-00005 (if we're using 5 shards).
# This way we have smaller individual files, and we can also easily access
# e.g. 20% of all data, or have 5 threads reading through the data
# simultaneously.

# The code in this cell simply takes a list of iterators and then
# randomly distributes the values returned by these iterators into sharded
# datasets (e.g. a train/eval/test split).

def rand_key(counts):
    """Returns a random key from "counts", using values as distribution."""
    r = random.randint(0, sum(counts.values()))
    for key, count in counts.items():
        if r > count or count == 0:
            r -= count
        else:
            counts[key] -= 1
            return key

def make_sharded_files(make_example, path, classes, iters, splits,
                       shards=10, overwrite=False, report_dt=10):
    """Create sharded files from "iters".

    Args:
      make_example: Converts object returned by elements of "iters"
          to tf.train.Example() proto.
      path: Directory that will contain recordio files.
      classes: Names of classes, will be written to "labels.txt".
      splits: Dictionary mapping filename to number of examples (of
          every class).
      shards: Number of files to be created per split.
      overwrite: Whether a pre-existing directory should be overwritten.
      report_dt: Number of seconds between status updates (0=no updates).

    Returns:
      Total number of examples written to disk (this should be equal to
      the number of classes times the sum of the number of examples of
      all the splits).
    """
    assert len(iters) == len(classes)
    if not os.path.exists(path):
        os.makedirs(path)
    paths = {
        split: ['%s/%s-%05d-of-%05d' % (path, split, i, shards)
                for i in range(shards)]
        for split in splits
    }
    assert overwrite or not os.path.exists(paths.values()[0][0])
    writers = {
        split: [tf.python_io.TFRecordWriter(ps[i]) for i in range(shards)]
        for split, ps in paths.items()
    }
    t0 = time.time()
    n = sum(splits.values())
    examples = 0
    for i in range(n):
        split = rand_key(splits)
        writer = writers[split][splits[split] % shards]
        for j in range(len(classes)):
            example = make_example(j, next(iters[j]))
            writer.write(example.SerializeToString())
            examples += 1
        remaining = sum(splits.values())
        if report_dt > 0 and time.time() - t0 > report_dt:
            print('processed %d/%d (%.2f%%)' % (i, n, 100. * i / n))
            t0 = time.time()
    for split in splits:
        for writer in writers[split]:
            writer.close()
    with open('%s/labels.txt' % path, 'w') as f:
        f.write('\n'.join(classes))
    return examples_____no_output_____%%writefile _derived/1_make_example_img.py
# (Written into separate file for sharing between notebooks.)

# Convert drawing tf.train.Example proto.
# Uses json_to_img() from previous cell to create raster image.

def make_example_img(label, drawing):
    example = tf.train.Example()
    example.features.feature['label'].int64_list.value.append(label)
    img_64 = np.asarray(json_to_img(drawing, img_sz=64, lw=4, maximize=True)).reshape(-1)
    example.features.feature['img_64'].int64_list.value.extend(img_64)
    example.features.feature['countrycode'].bytes_list.value.append(drawing['countrycode'].encode())
    example.features.feature['recognized'].int64_list.value.append(drawing['recognized'])
    example.features.feature['word'].bytes_list.value.append(drawing['word'].encode())
    ts = drawing['timestamp']
    ts = time.mktime(time.strptime(ts[:ts.index('.')], '%Y-%m-%d %H:%M:%S'))
    example.features.feature['timestamp'].int64_list.value.append(int(ts))
    example.features.feature['key_id'].int64_list.value.append(int(drawing['key_id']))
    return exampleOverwriting _derived/1_make_example_img.py
# (Load code from previous cell -- make sure to have executed above cell first.)
%run -i _derived/1_make_example_img.py

# Create the (rasterized) dataset.

path = '%s/dataset_img' % data_path
t0 = time.time()
n = make_sharded_files(
    make_example=make_example_img,
    path=path,
    classes=classes,
    iters=[loaditer(name) for name in classes],
    # Note: We only generate few examples here so you won't be
    # blocked for too long while waiting for this cell to finish.
    # You can re-run the cell with larger values (don't forget to
    # update "path" above) in the background to get a larger
    # dataset...
    splits=dict(train=5000, eval=1000, test=1000),
    overwrite=True,
)

print('stored data to "%s"' % path)
print('generated %d examples in %d seconds' % (n, time.time() - t0))processed 512/7000 (7.31%)
processed 1056/7000 (15.09%)
processed 1684/7000 (24.06%)
processed 2303/7000 (32.90%)
processed 2929/7000 (41.84%)
processed 3510/7000 (50.14%)
processed 4047/7000 (57.81%)
processed 4594/7000 (65.63%)
processed 5109/7000 (72.99%)
processed 5660/7000 (80.86%)
processed 6244/7000 (89.20%)
processed 6814/7000 (97.34%)
stored data to "../data/dataset_img"
generated 70000 examples in 123 seconds

# 6 Prepare dataset for RNN – bonus!

This section creates another dataset of example protos that contain the raw
stroke data, suitable for usage with a recurrent neural network.

Note that later notebooks will have a "bonus" section that uses this dataset,
but the "non-bonus" parts can be worked through without executing below
cells..._____no_output_____

%%writefile _derived/1_json_to_stroke.py
# (Written into separate file for sharing between notebooks.)

# Convert stroke coordinates into normalized relative coordinates,
# one single list, and add a "third dimension" that indicates when
# a new stroke starts.

def json_to_stroke(d):
    norm = lambda x: (x - x.min()) / max(1, (x.max() - x.min()))
    xy = np.concatenate([np.array(s, dtype=np.float32) for s in d['drawing']], axis=1)
    z = np.zeros(xy.shape[1])
    if len(d['drawing']) > 1:
        z[np.cumsum(np.array(list(map(lambda x: x.shape[1], d['drawing'][:-1]))))] = 1
    dxy = np.diff(norm(xy))
    return np.concatenate([dxy, z.reshape((1, -1))[:, 1:]])Overwriting _derived/1_json_to_stroke.py
# (Load code from previous cell -- make sure to have executed above cell first.)
%run -i _derived/1_json_to_stroke.py

# Visualize / control output of json_to_stroke().

stroke = json_to_stroke(sample[3])
# First 2 dimensions are normalized dx/dy coordinates
# third dimension indicates "new stroke".
xy = stroke[:2, :].cumsum(axis=1)
pyplot.plot(*xy)
pxy = xy[:, stroke[2] != 0]
pyplot.plot(pxy[0], pxy[1], 'ro')_____no_output_____%%writefile _derived/1_make_example_stroke.py
# (Written into separate file for sharing between notebooks.)

# Convert drawing tf.train.Example proto.
# Uses json_to_stroke() from previous cell to create raster image.

def make_example_stroke(label, drawing):
    example = tf.train.Example()
    example.features.feature['label'].int64_list.value.append(label)
    stroke = json_to_stroke(drawing)
    example.features.feature['stroke_x'].float_list.value.extend(stroke[0, :])
    example.features.feature['stroke_y'].float_list.value.extend(stroke[1, :])
    example.features.feature['stroke_z'].float_list.value.extend(stroke[2, :])
    example.features.feature['stroke_len'].int64_list.value.append(stroke.shape[1])
    example.features.feature['countrycode'].bytes_list.value.append(drawing['countrycode'].encode())
    example.features.feature['recognized'].int64_list.value.append(drawing['recognized'])
    example.features.feature['word'].bytes_list.value.append(drawing['word'].encode())
    ts = drawing['timestamp']
    ts = time.mktime(time.strptime(ts[:ts.index('.')], '%Y-%m-%d %H:%M:%S'))
    example.features.feature['timestamp'].int64_list.value.append(int(ts))
    example.features.feature['key_id'].int64_list.value.append(int(drawing['key_id']))
    return exampleOverwriting _derived/1_make_example_stroke.py
# (Load code from previous cell -- make sure to have executed above cell first.)
%run -i _derived/1_make_example_stroke.py

# Create the (stroke) dataset.

path = '%s/dataset_stroke' % data_path
t0 = time.time()
n = make_sharded_files(
    make_example=make_example_stroke,
    path=path,
    classes=classes,
    iters=[loaditer(name) for name in classes],
    splits=dict(train=50000, eval=10000, test=10000),
    overwrite=True,
)

print('stored examples to "%s"' % path)
print('generated %d examples in %d seconds' % (n, time.time() - t0))processed 2363/70000 (3.38%)
processed 4977/70000 (7.11%)
processed 7413/70000 (10.59%)
processed 9744/70000 (13.92%)
processed 12645/70000 (18.06%)
processed 15708/70000 (22.44%)
processed 18466/70000 (26.38%)
processed 21321/70000 (30.46%)
processed 24161/70000 (34.52%)
processed 27029/70000 (38.61%)
processed 29769/70000 (42.53%)
processed 32459/70000 (46.37%)
processed 35332/70000 (50.47%)
processed 38213/70000 (54.59%)
processed 40995/70000 (58.56%)
processed 43693/70000 (62.42%)
processed 46311/70000 (66.16%)
processed 49223/70000 (70.32%)
processed 51939/70000 (74.20%)
processed 54672/70000 (78.10%)
processed 57510/70000 (82.16%)
processed 60445/70000 (86.35%)
processed 63033/70000 (90.05%)
processed 65995/70000 (94.28%)
processed 68858/70000 (98.37%)
stored examples to "../data/dataset_stroke"
generated 700000 examples in 253 seconds


# Notebook from kylesha/MLearn210
Path: MLearn210-exercise04.ipynb

Lesson 4. Model selection: Car mileage
_____no_output_____

In this exercise you will predict car mileage based on its properties. The data is taken from ISLR website, but requires few transformations to make it usable for model building. These transformations will include: missing value imputation and normalizing the variables.

This dataset contains 392 car entries with following features:

mpg miles per gallon (what we are trying to predict)

cylinders Number of cylinders between 4 and 8

displacement Engine displacement (cu. inches)

horsepower Engine horsepower

weight Vehicle weight (lbs.)

acceleration Time to accelerate from 0 to 60 mph (sec.)

year Model year (modulo 100)

origin Origin of car (1. American, 2. European, 3. Japanese)

name Vehicle name

_____no_output_____

library(ggplot2)
library(magrittr)
library(xtable)
library(MASS)_____no_output_____# Load the data
auto_url = "http://www-bcf.usc.edu/~gareth/ISL/Auto.csv"
auto = read.csv(auto_url, na.strings="?")
# Note that you can also obtain the data from ISLR package:
# library(help = ISLR)
# We will name the rows using the 'name' column.
# However, assigning the column to rows, will throw an error, because they are not unique.
#
# Let's make them unique by combining it with the row number:
row.names(auto) = paste(auto$name, 1:nrow(auto))
# After this operation, you can delete the name from our dataset:
auto$name <- NULL

head(auto)
str(auto)
plot(auto)_____no_output_____

Q.1. Write a command that will calculate the number of missing values in this dataset
_____no_output_____

# Command(s)
features <- colnames(auto)
sapply(X = features, FUN = function(x) {sum(is.na(auto[,colnames(auto) == x]))})_____no_output_____

Q.2. Write a command that will only show the rows with missing values. Can you comment on why the values might be missing (e.g. excessive prevalence of specific feature)?
_____no_output_____

# Command:
#rows with missing values ('NA')
rows_with_NA <- auto[is.na(auto$horsepower),]
rows_with_NA

# diagnostic boxplots
ggplot2::ggplot(data= auto %>% dplyr::mutate(., missing = ifelse(is.na(auto$horsepower), "yes", "no")) %>%
       tidyr::gather(data = ., key = features,
                     features = mpg:origin,
                     value = value,
                     factor_key = TRUE,
                     na.rm = FALSE)) +
    ggplot2::geom_boxplot(aes(x=missing, y=value, fill=missing)) +
    ggplot2::facet_wrap(facets = ~ features, scales = "free_y") +
    ggplot2::labs(x = "features",
                  legend = "missing values",
                  caption = "Auto data set divided into two groups: one contains missing values and the other does not.
It appears all missing values are in variable \"horsepower\".") +
    ggplot2::theme(plot.caption = element_text(size = 7))_____no_output_____

> **ANSWER Q2:**
> The data set contains 5 missing values, all in variable _horsepower_._____no_output_____
Q.3. Impute the missing values using median for the column. Given answer to Q.2, is this a good approach to fix the missing value problem?
_____no_output_____

# Command(s):
hp.imputed <- ifelse(is.na(auto$horsepower),
                     median(x = auto$horsepower, na.rm = TRUE),
                     auto$horsepower)

auto$hp.imputed <- hp.imputed

# sanity check
auto[which(!(auto$horsepower %in% auto$hp.imputed)),]
auto$horsepower <- NULL #remove original horsepower column
head(auto)_____no_output_____

> **ANSWER Q3:**
> It does make sense to impute missing _horsepower_ values using the median. As can be seen in the "cylinder" boxplot in Q2, the median cylinder for the entire auto data set is 4. Since cylinder count determines a car's horsepower, imputing on the median is a reasonable choice._____no_output_____
Q.4. Create the contrast variables (origin.American, origin.European) for the origin of the car. Remove the numerical 'origin' column from the data.frame. Why didn't we create a variable for 'origin.Japanese'?
_____no_output_____

# Commands:
# Origin of car (1. American, 2. European, 3. Japanese)
auto$origin.American <- ifelse(auto$origin == 1, 1, 0)
auto$origin.European <- ifelse(auto$origin == 2, 1, 0)

# Remove origin
auto$origin <- NULL
tail(auto)_____no_output_____

> **ANSWER Q4:**
> The number of levels/contrasts needed to represent k categories is k-1. Thus, in our re-coded _origin_ variable...
>
| | origin.American | origin.European |
 |-:|--:|---:|
 |American|1|0|
 |European|0|1|
 |Japanese|0|0|
 _____no_output_____
Q.5. Develop a model that predicts mpg using all the available features in the data.frame. Summarize it and comment on its quality.
_____no_output_____

# Develop the model
modQ5 <- lm(formula = mpg ~., data = auto)
summary(modQ5)_____no_output_____

> **ANSWER Q5:**
> The multivariate linear regression model reveals at least one feature is associated with _mpg_ (F-statistics = 224.5, with p-value < 2.2e-16)._____no_output_____
Q.6. Show histogram of engine displacement to reveal its non-normal distribution. Calculate optimal 'lambda' value for this feature and apply it to displacement to create 'displacement.boxcox' feature. Plot this new feature to see whether it is normally distributed.
_____no_output_____

# Histogram of engine displacement
par(mfrow = c(2,2))
hist(auto$displacement, main = "displacement: No Transformation")
qqnorm(auto$displacement, main = "QQ-plot: No Transformation")
qqline(auto$displacement, lty=2)

# Calculate optimal 'lambda'
bc <- MASS::boxcox(object = modQ5,
                   lambda = seq(from = -5, to = 5, length.out = nrow(auto)),
                   plot = FALSE)
lambda <- bc$x[which.max(bc$y)]
# lambda

# Create 'displacement.boxcox'
auto$displacement.boxcox <- bc$y
auto$displacement <- NULL
# head(auto)

# Plot histogram of this new feature to reveal normal-like distribution
hist(auto$displacement.boxcox, main = "displacement: boxcox transformed")
qqnorm(auto$displacement.boxcox, main = "QQ-plot: boxcox transformed")
qqline(auto$displacement.boxcox, lty=2)_____no_output_____

> **ANSWER Q6:**
> $\lambda_{optimmal}=-0.3535$
> It appears boxcox transformation makes the variable _displacement_ take more of a normal distribution. However, the transformed version is heavily left-skewed (histogram, QQ-plot)_____no_output_____
Q.7. Recalculate the model to predict mpg with the new feature and comment on its performance in comparison to the original model.
_____no_output_____

# Calculate then new model
modQ7 <- lm(formula = mpg ~., data = auto)
compare <- rbind(Q5=broom::glance(modQ5), Q7=broom::glance(modQ7))
compare <- compare[, -c(1,6:11)]
colnames(compare) <- c("adj R^2", "sigma", "F-statistic", "p-value")
compare
# Comment on its performance
#_____no_output_____

> **ANSWER Q7**
> A comparison of model metrics from Q5 and Q7 is tabulated above, where sigma is Residual Standard Error and other columns are self-explanatory. Overall, both models are highly similar, as both indicate that there is at least one predictor variable that is associated with _mpg_. The Q7 model (with boxcox transformation) is slightly more robust (larger adjusted R^2 and F-statistic, smaller sigma and p-value)._____no_output_____
Q.8. What is the mean and standard deviation for all the features?
_____no_output_____

# Command(s) that shows mean for all the features
features <- colnames(auto)
features.mean <- sapply(X = features, FUN = function(x) {mean(auto[,colnames(auto) == x])})

# Command(s) that shows standard deviation for all the features
features.sd <- sapply(X = features, FUN = function(x) {sd(auto[,colnames(auto) == x])})

# sanity check
round(cbind(features.mean, features.sd),2)_____no_output_____

> **ANSWER Q8**
> The mean and standard deviation for each feature is tabulated above. It is unclear to me how to interpret the mean and SD of a categorical variable such as _origin.American, origina.European_, or _cylinders_. However, the question asks to calculate those statistics for all features; so I did!_____no_output_____
Q.9. Center all the features, so that their mean is 0 and scale them, so that their standard deviation is 1. Recalculate the model, comment on its performance.
_____no_output_____

###### Center and scale

### NOTE: centering around mean, as described in van den Berg et al. (BMC Genomics 2006, 7:142)
auto_centered <- sapply(X = colnames(auto),
                        FUN = function(x) {auto[,colnames(auto) == x] - features.mean[names(features.mean) == x]})
auto_centered <- as.data.frame(auto_centered)

### NOTE: using "auto-scaling" method as described in van den Berg et al. (BMC Genomics 2006, 7:142)
auto_centered_scaled <- sapply(X = colnames(auto_centered),
                               FUN = function(x)
                               {auto_centered[,colnames(auto_centered) == x] / features.sd[names(features.sd) == x]})
auto_centered_scaled <- as.data.frame(auto_centered_scaled)
rownames(auto_centered_scaled) <- rownames(auto)

# sanity checks
# head(auto)
# head(auto_centered)
# head(auto_centered_scaled)

# Recalculate the model
modQ9 <- lm(formula = mpg ~ ., data = auto_centered_scaled)
summary(modQ9)

# Comment on its performance
compare <- rbind(Q5=broom::glance(modQ5), Q7=broom::glance(modQ7), Q9=broom::glance(modQ9))
compare <- compare[, -c(1,6:11)]
colnames(compare) <- c("adj R^2", "sigma", "F-statistic", "p-value")
compare_____no_output_____

> **ANSWER Q9**
> Metrics from the models in Q5, Q7, and Q9 are tabulated above.
> The differences between Q7 and Q9 models are negligible in terms of their adjusted R^2 values, F-statistics, and p-values. However, the Residual Standard Error (sigma) of Q9 model is approximately 1/8 that of Q5 and Q7 models, a significant difference. Because the RSE measures the average deviation of the model from the true regression line, a smaller RSE indicates a tighter model fit. If one were interested in using the models for prediction, I'd say Q9 is at least as good as the other two. But if we're interested in inference, I'm not sure Q9 is any better, since the data transformation makes the model difficult to interpret._____no_output_____