# Notebook from ykhiari/Patient-Selection-for-Diabetes-Drug-Testing
Path: starter_code/student_project.ipynb

# Overview_____no_output_____1. Project Instructions & Prerequisites
2. Learning Objectives
3. Data Preparation
4. Create Categorical Features with TF Feature Columns
5. Create Continuous/Numerical Features with TF Feature Columns
6. Build Deep Learning Regression Model with Sequential API and TF Probability Layers
7. Evaluating Potential Model Biases with Aequitas Toolkit
_____no_output_____#  1. Project Instructions & Prerequisites_____no_output_____## Project Instructions_____no_output_____**Context**: EHR data is becoming a key source of real-world evidence (RWE) for the pharmaceutical industry and regulators to [make decisions on clinical trials](https://www.fda.gov/news-events/speeches-fda-officials/breaking-down-barriers-between-clinical-trials-and-clinical-care-incorporating-real-world-evidence). You are a data scientist for an exciting unicorn healthcare startup that has created a groundbreaking diabetes drug that is ready for clinical trial testing. It is a very unique and sensitive drug that requires administering the drug over at least 5-7 days of time in the hospital with frequent monitoring/testing and patient medication adherence training with a mobile application. You have been provided a patient dataset from a client partner and are tasked with building a predictive model that can identify which type of patients the company should focus their efforts testing this drug on. Target patients are people that are likely to be in the hospital for this duration of time and will not incur significant additional costs for administering this drug to the patient and monitoring.

In order to achieve your goal you must build a regression model that can predict the estimated hospitalization time for a patient and use this to select/filter patients for your study.
_____no_output_____**Expected Hospitalization Time Regression Model:** Utilizing a synthetic dataset(denormalized at the line level augmentation) built off of the UCI Diabetes readmission dataset, students will build a regression model that predicts the expected days of hospitalization time and then convert this to a binary prediction of whether to include or exclude that patient from the clinical trial.

This project will demonstrate the importance of building the right data representation at the encounter level, with appropriate filtering and preprocessing/feature engineering of key medical code sets. This project will also require students to analyze and interpret their model for biases across key demographic groups.

Please see the project rubric online for more details on the areas your project will be evaluated._____no_output_____### Dataset_____no_output_____Due to healthcare PHI regulations (HIPAA, HITECH), there are limited number of publicly available datasets and some datasets require training and approval. So, for the purpose of this exercise, we are using a dataset from UC Irvine(https://archive.ics.uci.edu/ml/datasets/Diabetes+130-US+hospitals+for+years+1999-2008) that has been modified for this course. Please note that it is limited in its representation of some key features such as diagnosis codes which are usually an unordered list in 835s/837s (the HL7 standard interchange formats used for claims and remits)._____no_output_____**Data Schema**
The dataset reference information can be https://github.com/udacity/nd320-c1-emr-data-starter/blob/master/project/data_schema_references/
. There are two CSVs that provide more details on the fields and some of the mapped values._____no_output_____## Project Submission _____no_output_____When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "student_project_submission.ipynb" and save another copy as an HTML file by clicking "File" -> "Download as.."->"html". Include the "utils.py" and "student_utils.py" files in your submission. The student_utils.py should be where you put most of your code that you write and the summary and text explanations should be written inline in the notebook. Once you download these files, compress them into one zip file for submission._____no_output_____## Prerequisites _____no_output_____- Intermediate level knowledge of Python
- Basic knowledge of probability and statistics
- Basic knowledge of machine learning concepts
- Installation of Tensorflow 2.0 and other dependencies(conda environment.yml or virtualenv requirements.txt file provided)_____no_output_____## Environment Setup_____no_output_____For step by step instructions on creating your environment, please go to https://github.com/udacity/nd320-c1-emr-data-starter/blob/master/project/README.md._____no_output_____# 2.  Learning Objectives_____no_output_____By the end of the project, you will be able to
   - Use the Tensorflow Dataset API to scalably extract, transform, and load datasets and build datasets aggregated at the line, encounter, and patient data levels(longitudinal)
   - Analyze EHR datasets to check for common issues (data leakage, statistical properties, missing values, high cardinality) by performing exploratory data analysis.
   - Create categorical features from Key Industry Code Sets (ICD, CPT, NDC) and reduce dimensionality for high cardinality features by using embeddings
   - Create derived features(bucketing, cross-features, embeddings) utilizing Tensorflow feature columns on both continuous and categorical input features
   - SWBAT use the Tensorflow Probability library to train a model that provides uncertainty range predictions that allow for risk adjustment/prioritization and triaging of predictions
   - Analyze and determine biases for a model for key demographic groups by evaluating performance metrics across groups by using the Aequitas framework
_____no_output_____# 3. Data Preparation_____no_output_____

# from __future__ import absolute_import, division, print_function, unicode_literals
import os
import numpy as np
import tensorflow as tf
from tensorflow.keras import layers
import tensorflow_probability as tfp
import matplotlib.pyplot as plt
import pandas as pd
import aequitas as ae
import seaborn as sns
# Put all of the helper functions in utils
from sklearn.metrics import roc_auc_score, accuracy_score, f1_score, classification_report, precision_score, recall_score
from utils import build_vocab_files, show_group_stats_viz, aggregate_dataset, preprocess_df, df_to_dataset, posterior_mean_field, prior_trainable
pd.set_option('display.max_columns', 500)
# this allows you to make changes and save in student_utils.py and the file is reloaded every time you run a code block
%load_ext autoreload
%autoreload_____no_output_____#OPEN ISSUE ON MAC OSX for TF model training
import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'_____no_output_____

## Dataset Loading and Schema Review_____no_output_____Load the dataset and view a sample of the dataset along with reviewing the schema reference files to gain a deeper understanding of the dataset. The dataset is located at the following path https://github.com/udacity/nd320-c1-emr-data-starter/blob/master/project/starter_code/data/final_project_dataset.csv. Also, review the information found in the data schema https://github.com/udacity/nd320-c1-emr-data-starter/blob/master/project/data_schema_references/_____no_output_____

dataset_path = "./data/final_project_dataset.csv"
df = pd.read_csv(dataset_path)_____no_output_____df.head()_____no_output_____assert len(np.unique(df["encounter_id"])) < df.shape[0]
print("The dataset is at the line level")The dataset is at the line level

## Determine Level of Dataset (Line or Encounter)_____no_output_____**Question 1**: Based off of analysis of the data, what level is this dataset? Is it at the line or encounter level? Are there any key fields besides the encounter_id and patient_nbr fields that we should use to aggregate on? Knowing this information will help inform us what level of aggregation is necessary for future steps and is a step that is often overlooked. _____no_output_____Student Response:
- From the previous cell we can conclude that the dataset is at the line level.
- The other key filed that we should aggregate on is "primary_diagnosis_code"_____no_output_____## Analyze Dataset_____no_output_____**Question 2**: Utilizing the library of your choice (recommend Pandas and Seaborn or matplotlib though), perform exploratory data analysis on the dataset. In particular be sure to address the following questions:
    - a. Field(s) with high amount of missing/zero values
    - b. Based off the frequency histogram for each numerical field, which numerical field(s) has/have a Gaussian(normal) distribution shape?
    - c. Which field(s) have high cardinality and why (HINT: ndc_code is one feature)
    - d. Please describe the demographic distributions in the dataset for the age and gender fields.

_____no_output_____**OPTIONAL**: Use the Tensorflow Data Validation and Analysis library to complete.
- The Tensorflow Data Validation and Analysis library(https://www.tensorflow.org/tfx/data_validation/get_started) is a useful tool for analyzing and summarizing dataset statistics. It is especially useful because it can scale to large datasets that do not fit into memory.
- Note that there are some bugs that are still being resolved with Chrome v80 and we have moved away from using this for the project. _____no_output_____### Answer to question 2.a
the fields with a high missing/zero values are: weight, max_glu_serum, A1Cresult, edical_speciality,prayer_code and ndc_code_____no_output_____

df = df.replace('?', np.nan).replace('None', np.nan)_____no_output_____df.isnull().mean().sort_values(ascending=False)_____no_output_____

### Answer to question 2.b
numerical fileds with gaussian distribution are : num_lab_procedures, num_medications_____no_output_____

df.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 143424 entries, 0 to 143423
Data columns (total 26 columns):
 #   Column                    Non-Null Count   Dtype
---  ------                    --------------   -----
 0   encounter_id              143424 non-null  int64
 1   patient_nbr               143424 non-null  int64
 2   race                      140115 non-null  object
 3   gender                    143424 non-null  object
 4   age                       143424 non-null  object
 5   weight                    4302 non-null    object
 6   admission_type_id         143424 non-null  int64
 7   discharge_disposition_id  143424 non-null  int64
 8   admission_source_id       143424 non-null  int64
 9   time_in_hospital          143424 non-null  int64
 10  payer_code                89234 non-null   object
 11  medical_specialty         73961 non-null   object
 12  primary_diagnosis_code    143391 non-null  object
 13  other_diagnosis_codes     143424 non-null  object
 14  number_outpatient         143424 non-null  int64
 15  number_inpatient          143424 non-null  int64
 16  number_emergency          143424 non-null  int64
 17  num_lab_procedures        143424 non-null  int64
 18  number_diagnoses          143424 non-null  int64
 19  num_medications           143424 non-null  int64
 20  num_procedures            143424 non-null  int64
 21  ndc_code                  119962 non-null  object
 22  max_glu_serum             7015 non-null    object
 23  A1Cresult                 25774 non-null   object
 24  change                    143424 non-null  object
 25  readmitted                143424 non-null  object
dtypes: int64(13), object(13)
memory usage: 28.5+ MB
numeric_field = [c for c in df.columns if df[c].dtype == "int64"]
numeric_field_____no_output_____for c in numeric_field:
    sns.distplot(df[c], kde=False)
    plt.title(c)
    plt.show()_____no_output_____

### Answer to question 2.c
The fields with high cardinality are: 'other_diagnosis_codes', 'primary_diagnosis_code', 'ndc_code'_____no_output_____

# identify categorical columns
cat_col = list(df.select_dtypes(['object']).columns)
cat_col.extend(['admission_type_id','discharge_disposition_id', 'admission_source_id'])
for col in cat_col:
    df[col] = df[col].astype(str)
cat_col_____no_output_____pd.DataFrame({'cardinality': df[cat_col].nunique()})_____no_output_____

### Answer to question 2.d
according to the representations below it is shown that the majority of ages are between 50 to 90 and that a small advantage of the femal number of hospitalization_____no_output_____

plt.figure(figsize=(8, 5))
sns.countplot(x = 'age', data = df)_____no_output_____plt.figure(figsize=(8, 5))
sns.countplot(x = 'gender', data = df)_____no_output_____plt.figure(figsize=(8, 5))
sns.countplot(x = 'age', hue = 'gender', data = df)_____no_output_____# !pip install tensorflow-data-validation
# !pip install apache-beam[interactive]
#import tensorflow_data_validation as tfdv_____no_output_____######NOTE: The visualization will only display in Chrome browser. ########
#full_data_stats = tfdv.generate_statistics_from_csv(data_location='./data/final_project_dataset.csv')
#tfdv.visualize_statistics(full_data_stats)_____no_output_____

## Reduce Dimensionality of the NDC Code Feature_____no_output_____**Question 3**: NDC codes are a common format to represent the wide variety of drugs that are prescribed for patient care in the United States. The challenge is that there are many codes that map to the same or similar drug. You are provided with the ndc drug lookup file https://github.com/udacity/nd320-c1-emr-data-starter/blob/master/project/data_schema_references/ndc_lookup_table.csv derived from the National Drug Codes List site(https://ndclist.com/). Please use this file to come up with a way to reduce the dimensionality of this field and create a new field in the dataset called "generic_drug_name" in the output dataframe. _____no_output_____

#NDC code lookup file
ndc_code_path = "./medication_lookup_tables/final_ndc_lookup_table"
ndc_code_df = pd.read_csv(ndc_code_path)_____no_output_____ndc_code_df.head()_____no_output_____from student_utils import reduce_dimension_ndc_____no_output_____reduce_dim_df = reduce_dimension_ndc(df, ndc_code_df)
reduce_dim_df.head()_____no_output_____# Number of unique values should be less for the new output field
assert df['ndc_code'].nunique() > reduce_dim_df['generic_drug_name'].nunique()_____no_output_____

## Select First Encounter for each Patient _____no_output_____**Question 4**: In order to simplify the aggregation of data for the model, we will only select the first encounter for each patient in the dataset. This is to reduce the risk of data leakage of future patient encounters and to reduce complexity of the data transformation and modeling steps. We will assume that sorting in numerical order on the encounter_id provides the time horizon for determining which encounters come before and after another._____no_output_____

from student_utils import select_first_encounter
first_encounter_df = select_first_encounter(reduce_dim_df)_____no_output_____# unique patients in transformed dataset
unique_patients = first_encounter_df['patient_nbr'].nunique()
print("Number of unique patients:{}".format(unique_patients))

# unique encounters in transformed dataset
unique_encounters = first_encounter_df['encounter_id'].nunique()
print("Number of unique encounters:{}".format(unique_encounters))

original_unique_patient_number = reduce_dim_df['patient_nbr'].nunique()
# number of unique patients should be equal to the number of unique encounters and patients in the final dataset
assert original_unique_patient_number == unique_patients
assert original_unique_patient_number == unique_encounters
print("Tests passed!!")Number of unique patients:71518
Number of unique encounters:71518
Tests passed!!

## Aggregate Dataset to Right Level for Modeling _____no_output_____In order to provide a broad scope of the steps and to prevent students from getting stuck with data transformations, we have selected the aggregation columns and provided a function to build the dataset at the appropriate level. The 'aggregate_dataset" function that you can find in the 'utils.py' file can take the preceding dataframe with the 'generic_drug_name' field and transform the data appropriately for the project.

To make it simpler for students, we are creating dummy columns for each unique generic drug name and adding those are input features to the model. There are other options for data representation but this is out of scope for the time constraints of the course._____no_output_____

exclusion_list = ['generic_drug_name', 'ndc_code']
grouping_field_list = [c for c in first_encounter_df.columns if c not in exclusion_list]
agg_drug_df, ndc_col_list = aggregate_dataset(first_encounter_df, grouping_field_list, 'generic_drug_name')/home/workspace/starter_code/utils.py:10: FutureWarning: Indexing with multiple keys (implicitly converted to a tuple of keys) will be deprecated, use a list instead.
  array_field].apply(lambda x: x[array_field].values.tolist()).reset_index().rename(columns={
assert len(agg_drug_df) == agg_drug_df['patient_nbr'].nunique() == agg_drug_df['encounter_id'].nunique()_____no_output_____

## Prepare Fields and Cast Dataset _____no_output_____### Feature Selection_____no_output_____**Question 5**: After you have aggregated the dataset to the right level, we can do feature selection (we will include the ndc_col_list, dummy column features too). In the block below, please select the categorical and numerical features that you will use for the model, so that we can create a dataset subset.

For the payer_code and weight fields, please provide whether you think we should include/exclude the field in our model and give a justification/rationale for this based off of the statistics of the data. Feel free to use visualizations or summary statistics to support your choice._____no_output_____

plt.figure(figsize=(8, 5))
sns.countplot(x = 'payer_code',  data = agg_drug_df)_____no_output_____plt.figure(figsize=(8, 5))
sns.countplot(x = 'number_emergency',  data = agg_drug_df)_____no_output_____

Student response: We should exclude both payer_code and weight in our model due to the big amount of missing values_____no_output_____

'''
Please update the list to include the features you think are appropriate for the model
and the field that we will be using to train the model. There are three required demographic features for the model
and I have inserted a list with them already in the categorical list.
These will be required for later steps when analyzing data splits and model biases.
'''
required_demo_col_list = ['race', 'gender', 'age']
student_categorical_col_list = [ 'change', 'primary_diagnosis_code'] + required_demo_col_list + ndc_col_list
student_numerical_col_list = [ 'number_inpatient', 'number_emergency', 'num_lab_procedures', 'number_diagnoses','num_medications','num_procedures']
PREDICTOR_FIELD = 'time_in_hospital'_____no_output_____def select_model_features(df, categorical_col_list, numerical_col_list, PREDICTOR_FIELD, grouping_key='patient_nbr'):
    selected_col_list = [grouping_key] + [PREDICTOR_FIELD] + categorical_col_list + numerical_col_list
    return agg_drug_df[selected_col_list]
_____no_output_____selected_features_df = select_model_features(agg_drug_df, student_categorical_col_list, student_numerical_col_list,
                                            PREDICTOR_FIELD)_____no_output_____

### Preprocess Dataset - Casting and Imputing  _____no_output_____We will cast and impute the dataset before splitting so that we do not have to repeat these steps across the splits in the next step. For imputing, there can be deeper analysis into which features to impute and how to impute but for the sake of time, we are taking a general strategy of imputing zero for only numerical features.

OPTIONAL: What are some potential issues with this approach? Can you recommend a better way and also implement it?_____no_output_____

processed_df = preprocess_df(selected_features_df, student_categorical_col_list,
        student_numerical_col_list, PREDICTOR_FIELD, categorical_impute_value='nan', numerical_impute_value=0)/home/workspace/starter_code/utils.py:29: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df[predictor] = df[predictor].astype(float)
/home/workspace/starter_code/utils.py:31: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df[c] = cast_df(df, c, d_type=str)
/home/workspace/starter_code/utils.py:33: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df[numerical_column] = impute_df(df, numerical_column, numerical_impute_value)

## Split Dataset into Train, Validation, and Test Partitions_____no_output_____**Question 6**: In order to prepare the data for being trained and evaluated by a deep learning model, we will split the dataset into three partitions, with the validation partition used for optimizing the model hyperparameters during training. One of the key parts is that we need to be sure that the data does not accidently leak across partitions.

Please complete the function below to split the input dataset into three partitions(train, validation, test) with the following requirements.
- Approximately 60%/20%/20%  train/validation/test split
- Randomly sample different patients into each data partition
- **IMPORTANT** Make sure that a patient's data is not in more than one partition, so that we can avoid possible data leakage.
- Make sure that the total number of unique patients across the splits is equal to the total number of unique patients in the original dataset
- Total number of rows in original dataset = sum of rows across all three dataset partitions_____no_output_____

def patient_dataset_splitter(df, patient_key='patient_nbr'):
    '''
    df: pandas dataframe, input dataset that will be split
    patient_key: string, column that is the patient id

    return:
     - train: pandas dataframe,
     - validation: pandas dataframe,
     - test: pandas dataframe,
    '''
    df[student_numerical_col_list] = df[student_numerical_col_list].astype(float)
    train_val_df = df.sample(frac = 0.8, random_state=3)
    train_df = train_val_df.sample(frac = 0.8, random_state=3)
    val_df = train_val_df.drop(train_df.index)
    test_df = df.drop(train_val_df.index)
    return train_df.reset_index(drop = True), val_df.reset_index(drop = True), test_df.reset_index(drop = True)_____no_output_____#from student_utils import patient_dataset_splitter
d_train, d_val, d_test = patient_dataset_splitter(processed_df, 'patient_nbr')/opt/conda/lib/python3.7/site-packages/pandas/core/frame.py:2963: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self[k1] = value[k2]
assert len(d_train) + len(d_val) + len(d_test) == len(processed_df)
print("Test passed for number of total rows equal!")Test passed for number of total rows equal!
assert (d_train['patient_nbr'].nunique() + d_val['patient_nbr'].nunique() + d_test['patient_nbr'].nunique()) == agg_drug_df['patient_nbr'].nunique()
print("Test passed for number of unique patients being equal!")Test passed for number of unique patients being equal!

## Demographic Representation Analysis of Split_____no_output_____After the split, we should check to see the distribution of key features/groups and make sure that there is representative samples across the partitions. The show_group_stats_viz function in the utils.py file can be used to group and visualize different groups and dataframe partitions._____no_output_____### Label Distribution Across Partitions_____no_output_____Below you can see the distributution of the label across your splits. Are the histogram distribution shapes similar across partitions?_____no_output_____

show_group_stats_viz(processed_df, PREDICTOR_FIELD)time_in_hospital
1.0     10717
2.0     12397
3.0     12701
4.0      9567
5.0      6839
6.0      5171
7.0      3999
8.0      2919
9.0      1990
10.0     1558
11.0     1241
12.0      955
13.0      795
14.0      669
dtype: int64
AxesSubplot(0.125,0.125;0.775x0.755)
show_group_stats_viz(d_train, PREDICTOR_FIELD)time_in_hospital
1.0     6904
2.0     7978
3.0     8116
4.0     6141
5.0     4341
6.0     3294
7.0     2542
8.0     1846
9.0     1259
10.0     989
11.0     788
12.0     659
13.0     500
14.0     414
dtype: int64
AxesSubplot(0.125,0.125;0.775x0.755)
show_group_stats_viz(d_test, PREDICTOR_FIELD)time_in_hospital
1.0     2159
2.0     2486
3.0     2576
4.0     1842
5.0     1364
6.0     1041
7.0      814
8.0      584
9.0      404
10.0     307
11.0     242
12.0     182
13.0     165
14.0     138
dtype: int64
AxesSubplot(0.125,0.125;0.775x0.755)

## Demographic Group Analysis_____no_output_____We should check that our partitions/splits of the dataset are similar in terms of their demographic profiles. Below you can see how we might visualize and analyze the full dataset vs. the partitions._____no_output_____

# Full dataset before splitting
patient_demo_features = ['race', 'gender', 'age', 'patient_nbr']
patient_group_analysis_df = processed_df[patient_demo_features].groupby('patient_nbr').head(1).reset_index(drop=True)
show_group_stats_viz(patient_group_analysis_df, 'gender')gender
Female             38025
Male               33490
Unknown/Invalid        3
dtype: int64
AxesSubplot(0.125,0.125;0.775x0.755)
# Training partition
show_group_stats_viz(d_train, 'gender')gender
Female             24197
Male               21572
Unknown/Invalid        2
dtype: int64
AxesSubplot(0.125,0.125;0.775x0.755)
# Test partition
show_group_stats_viz(d_test, 'gender')gender
Female             7631
Male               6672
Unknown/Invalid       1
dtype: int64
AxesSubplot(0.125,0.125;0.775x0.755)

## Convert Dataset Splits to TF Dataset_____no_output_____We have provided you the function to convert the Pandas dataframe to TF tensors using the TF Dataset API.
Please note that this is not a scalable method and for larger datasets, the 'make_csv_dataset' method is recommended -https://www.tensorflow.org/api_docs/python/tf/data/experimental/make_csv_dataset._____no_output_____

# Convert dataset from Pandas dataframes to TF dataset
batch_size = 128
diabetes_train_ds = df_to_dataset(d_train, PREDICTOR_FIELD, batch_size=batch_size)
diabetes_val_ds = df_to_dataset(d_val, PREDICTOR_FIELD, batch_size=batch_size)
diabetes_test_ds = df_to_dataset(d_test, PREDICTOR_FIELD, batch_size=batch_size)_____no_output_____# We use this sample of the dataset to show transformations later
diabetes_batch = next(iter(diabetes_train_ds))[0]
def demo(feature_column, example_batch):
    feature_layer = layers.DenseFeatures(feature_column)
    print(feature_layer(example_batch))_____no_output_____

# 4. Create Categorical Features with TF Feature Columns_____no_output_____## Build Vocabulary for Categorical Features_____no_output_____Before we can create the TF categorical features, we must first create the vocab files with the unique values for a given field that are from the **training** dataset. Below we have provided a function that you can use that only requires providing the pandas train dataset partition and the list of the categorical columns in a list format. The output variable 'vocab_file_list' will be a list of the file paths that can be used in the next step for creating the categorical features._____no_output_____

vocab_file_list = build_vocab_files(d_train, student_categorical_col_list)_____no_output_____

## Create Categorical Features with Tensorflow Feature Column API_____no_output_____**Question 7**: Using the vocab file list from above that was derived fromt the features you selected earlier, please create categorical features with the Tensorflow Feature Column API, https://www.tensorflow.org/api_docs/python/tf/feature_column. Below is a function to help guide you._____no_output_____

from student_utils import create_tf_categorical_feature_cols
tf_cat_col_list = create_tf_categorical_feature_cols(student_categorical_col_list)INFO:tensorflow:vocabulary_size = 3 in change is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/change_vocab.txt.
INFO:tensorflow:vocabulary_size = 654 in primary_diagnosis_code is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/primary_diagnosis_code_vocab.txt.
INFO:tensorflow:vocabulary_size = 7 in race is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/race_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in gender is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/gender_vocab.txt.
INFO:tensorflow:vocabulary_size = 11 in age is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/age_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Acarbose is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Acarbose_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Glimepiride is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Glimepiride_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Glipizide is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Glipizide_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Glipizide_And_Metformin_Hcl is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Glipizide_And_Metformin_Hcl_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Glipizide_And_Metformin_Hydrochloride is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Glipizide_And_Metformin_Hydrochloride_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Glyburide is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Glyburide_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Glyburide_And_Metformin_Hydrochloride is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Glyburide_And_Metformin_Hydrochloride_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Glyburide-metformin_Hydrochloride is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Glyburide-metformin_Hydrochloride_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Human_Insulin is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Human_Insulin_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Insulin_Human is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Insulin_Human_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Metformin_Hcl is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Metformin_Hcl_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Metformin_Hydrochloride is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Metformin_Hydrochloride_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Miglitol is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Miglitol_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Nateglinide is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Nateglinide_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Pioglitazole_And_Metformin_Hydrochloride is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Pioglitazole_And_Metformin_Hydrochloride_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Pioglitazone is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Pioglitazone_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Pioglitazone_Hydrochloride_And_Glimepiride is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Pioglitazone_Hydrochloride_And_Glimepiride_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Repaglinide is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Repaglinide_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Rosiglitazone_Maleate is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Rosiglitazone_Maleate_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Tolazamide is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Tolazamide_vocab.txt.
INFO:tensorflow:vocabulary_size = 4 in Tolbutamide is inferred from the number of elements in the vocabulary_file ./diabetes_vocab/Tolbutamide_vocab.txt.
test_cat_var1 = tf_cat_col_list[0]
print("Example categorical field:\n{}".format(test_cat_var1))
demo(test_cat_var1, diabetes_batch)Example categorical field:
IndicatorColumn(categorical_column=VocabularyFileCategoricalColumn(key='change', vocabulary_file='./diabetes_vocab/change_vocab.txt', vocabulary_size=3, num_oov_buckets=1, dtype=tf.string, default_value=-1))
WARNING:tensorflow:Layer dense_features is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.

If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.

To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.

WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4267: IndicatorColumn._variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4322: VocabularyFileCategoricalColumn._num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
tf.Tensor(
[[0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 1. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]
 [0. 1. 0. 0.]], shape=(128, 4), dtype=float32)

# 5. Create Numerical Features with TF Feature Columns_____no_output_____**Question 8**: Using the TF Feature Column API(https://www.tensorflow.org/api_docs/python/tf/feature_column/), please create normalized Tensorflow numeric features for the model. Try to use the z-score normalizer function below to help as well as the 'calculate_stats_from_train_data' function._____no_output_____

from student_utils import create_tf_numeric_feature_____no_output_____

For simplicity the create_tf_numerical_feature_cols function below uses the same normalizer function across all features(z-score normalization) but if you have time feel free to analyze and adapt the normalizer based off the statistical distributions. You may find this as a good resource in determining which transformation fits best for the data https://developers.google.com/machine-learning/data-prep/transform/normalization.
_____no_output_____

def calculate_stats_from_train_data(df, col):
    mean = df[col].describe()['mean']
    std = df[col].describe()['std']
    return mean, std

def create_tf_numerical_feature_cols(numerical_col_list, train_df):
    tf_numeric_col_list = []
    for c in numerical_col_list:
        mean, std = calculate_stats_from_train_data(train_df, c)
        tf_numeric_feature = create_tf_numeric_feature(c, mean, std)
        tf_numeric_col_list.append(tf_numeric_feature)
    return tf_numeric_col_list_____no_output_____tf_cont_col_list = create_tf_numerical_feature_cols(student_numerical_col_list, d_train)_____no_output_____test_cont_var1 = tf_cont_col_list[0]
print("Example continuous field:\n{}\n".format(test_cont_var1))
demo(test_cont_var1, diabetes_batch)Example continuous field:
NumericColumn(key='number_inpatient', shape=(1,), default_value=(0,), dtype=tf.float64, normalizer_fn=functools.partial(<function create_tf_numeric_feature.<locals>.<lambda> at 0x7f8b0dcff290>, m=0.17600664176006642, s=0.6009985590232482))

WARNING:tensorflow:Layer dense_features_1 is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.

If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.

To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.

tf.Tensor(
[[-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [ 1.3710403]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [ 3.0349379]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [ 1.3710403]
 [-0.292857 ]
 [ 1.3710403]
 [-0.292857 ]
 [-0.292857 ]
 [ 3.0349379]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [ 1.3710403]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [ 3.0349379]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [ 1.3710403]
 [-0.292857 ]
 [-0.292857 ]
 [ 1.3710403]
 [ 3.0349379]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [ 1.3710403]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [ 1.3710403]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [ 1.3710403]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [ 1.3710403]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [ 1.3710403]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [ 4.6988354]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [-0.292857 ]
 [ 1.3710403]
 [-0.292857 ]
 [-0.292857 ]], shape=(128, 1), dtype=float32)

# 6. Build Deep Learning Regression Model with Sequential API and TF Probability Layers_____no_output_____## Use DenseFeatures to combine features for model_____no_output_____Now that we have prepared categorical and numerical features using Tensorflow's Feature Column API, we can combine them into a dense vector representation for the model. Below we will create this new input layer, which we will call 'claim_feature_layer'._____no_output_____

claim_feature_columns = tf_cat_col_list + tf_cont_col_list
claim_feature_layer = tf.keras.layers.DenseFeatures(claim_feature_columns)_____no_output_____

## Build Sequential API Model from DenseFeatures and TF Probability Layers_____no_output_____Below we have provided some boilerplate code for building a model that connects the Sequential API, DenseFeatures, and Tensorflow Probability layers into a deep learning model. There are many opportunities to further optimize and explore different architectures through benchmarking and testing approaches in various research papers, loss and evaluation metrics, learning curves, hyperparameter tuning, TF probability layers, etc. Feel free to modify and explore as you wish._____no_output_____**OPTIONAL**: Come up with a more optimal neural network architecture and hyperparameters. Share the process in discovering the architecture and hyperparameters._____no_output_____

def build_sequential_model(feature_layer):
    model = tf.keras.Sequential([
        feature_layer,
        tf.keras.layers.Dense(150, activation='relu'),
        tf.keras.layers.Dense(75, activation='relu'),
        tfp.layers.DenseVariational(1+1, posterior_mean_field, prior_trainable),
        tfp.layers.DistributionLambda(
            lambda t:tfp.distributions.Normal(loc=t[..., :1],
                                             scale=1e-3 + tf.math.softplus(0.01 * t[...,1:])
                                             )
        ),
    ])
    return model

def build_diabetes_model(train_ds, val_ds,  feature_layer,  epochs=5, loss_metric='mse'):
    model = build_sequential_model(feature_layer)
    model.compile(optimizer='rmsprop', loss=loss_metric, metrics=[loss_metric])
    early_stop = tf.keras.callbacks.EarlyStopping(monitor=loss_metric, patience=3)
    history = model.fit(train_ds, validation_data=val_ds,
                        callbacks=[early_stop],
                        epochs=epochs)
    return model, history _____no_output_____diabetes_model, history = build_diabetes_model(diabetes_train_ds, diabetes_val_ds,  claim_feature_layer,  epochs=100)Train for 358 steps, validate for 90 steps
Epoch 1/100
358/358 [==============================] - 8s 22ms/step - loss: 24.6293 - mse: 24.4528 - val_loss: 18.8390 - val_mse: 18.0638
Epoch 2/100
358/358 [==============================] - 5s 14ms/step - loss: 16.2279 - mse: 15.5325 - val_loss: 13.9956 - val_mse: 12.7708
Epoch 3/100
358/358 [==============================] - 5s 13ms/step - loss: 12.7957 - mse: 11.8131 - val_loss: 12.6461 - val_mse: 11.9815
Epoch 4/100
358/358 [==============================] - 5s 13ms/step - loss: 11.1345 - mse: 10.1729 - val_loss: 9.6361 - val_mse: 8.7062
Epoch 5/100
358/358 [==============================] - 5s 13ms/step - loss: 10.5669 - mse: 9.6203 - val_loss: 8.7706 - val_mse: 7.6288
Epoch 6/100
358/358 [==============================] - 5s 14ms/step - loss: 9.7915 - mse: 8.9645 - val_loss: 10.0533 - val_mse: 9.2921
Epoch 7/100
358/358 [==============================] - 5s 13ms/step - loss: 9.3032 - mse: 8.3535 - val_loss: 9.9639 - val_mse: 9.1109
Epoch 8/100
358/358 [==============================] - 5s 13ms/step - loss: 9.1580 - mse: 8.3823 - val_loss: 9.1492 - val_mse: 8.6072
Epoch 9/100
358/358 [==============================] - 5s 15ms/step - loss: 8.5008 - mse: 7.6392 - val_loss: 9.1765 - val_mse: 8.3628
Epoch 10/100
358/358 [==============================] - 5s 14ms/step - loss: 8.6621 - mse: 7.9144 - val_loss: 8.8989 - val_mse: 8.1129
Epoch 11/100
358/358 [==============================] - 5s 13ms/step - loss: 8.4136 - mse: 7.5674 - val_loss: 9.0078 - val_mse: 8.0410
Epoch 12/100
358/358 [==============================] - 4s 12ms/step - loss: 8.4214 - mse: 7.5767 - val_loss: 8.2852 - val_mse: 7.1796
Epoch 13/100
358/358 [==============================] - 5s 13ms/step - loss: 8.0986 - mse: 7.2194 - val_loss: 8.4048 - val_mse: 7.3981
Epoch 14/100
358/358 [==============================] - 5s 13ms/step - loss: 8.0938 - mse: 7.1935 - val_loss: 8.5188 - val_mse: 7.6221
Epoch 15/100
358/358 [==============================] - 5s 13ms/step - loss: 8.1018 - mse: 7.1305 - val_loss: 8.0634 - val_mse: 7.2399
Epoch 16/100
358/358 [==============================] - 4s 13ms/step - loss: 7.8850 - mse: 6.9900 - val_loss: 8.3079 - val_mse: 7.2268
Epoch 17/100
358/358 [==============================] - 5s 14ms/step - loss: 7.7499 - mse: 6.9222 - val_loss: 8.0496 - val_mse: 7.0870
Epoch 18/100
358/358 [==============================] - 5s 14ms/step - loss: 7.7917 - mse: 6.8462 - val_loss: 8.0874 - val_mse: 7.1399
Epoch 19/100
358/358 [==============================] - 7s 19ms/step - loss: 7.6744 - mse: 6.7074 - val_loss: 7.8026 - val_mse: 6.6841
Epoch 20/100
358/358 [==============================] - 5s 13ms/step - loss: 7.5486 - mse: 6.6938 - val_loss: 8.2209 - val_mse: 7.1531
Epoch 21/100
358/358 [==============================] - 4s 12ms/step - loss: 7.5367 - mse: 6.6350 - val_loss: 7.5655 - val_mse: 6.8027
Epoch 22/100
358/358 [==============================] - 5s 13ms/step - loss: 7.4362 - mse: 6.5466 - val_loss: 7.7240 - val_mse: 6.7320
Epoch 23/100
358/358 [==============================] - 5s 13ms/step - loss: 7.5003 - mse: 6.6328 - val_loss: 8.3950 - val_mse: 7.0638
Epoch 24/100
358/358 [==============================] - 5s 13ms/step - loss: 7.5862 - mse: 6.5718 - val_loss: 7.7765 - val_mse: 6.9072
Epoch 25/100
358/358 [==============================] - 5s 13ms/step - loss: 7.5311 - mse: 6.5044 - val_loss: 7.5655 - val_mse: 6.4988
Epoch 26/100
358/358 [==============================] - 5s 15ms/step - loss: 7.3392 - mse: 6.4640 - val_loss: 8.1907 - val_mse: 7.2635
Epoch 27/100
358/358 [==============================] - 4s 12ms/step - loss: 7.4509 - mse: 6.4358 - val_loss: 7.7066 - val_mse: 6.7276
Epoch 28/100
358/358 [==============================] - 5s 13ms/step - loss: 7.3021 - mse: 6.4213 - val_loss: 7.8700 - val_mse: 6.6681
Epoch 29/100
358/358 [==============================] - 4s 12ms/step - loss: 7.1892 - mse: 6.3133 - val_loss: 7.7775 - val_mse: 6.7267
Epoch 30/100
358/358 [==============================] - 4s 12ms/step - loss: 7.1800 - mse: 6.3027 - val_loss: 7.2485 - val_mse: 6.6883
Epoch 31/100
358/358 [==============================] - 4s 12ms/step - loss: 7.2975 - mse: 6.2679 - val_loss: 7.4930 - val_mse: 6.4465
Epoch 32/100
358/358 [==============================] - 5s 13ms/step - loss: 7.0811 - mse: 6.1155 - val_loss: 7.6528 - val_mse: 6.6499
Epoch 33/100
358/358 [==============================] - 5s 13ms/step - loss: 7.1447 - mse: 6.2611 - val_loss: 7.2010 - val_mse: 6.3405
Epoch 34/100
358/358 [==============================] - 5s 13ms/step - loss: 7.1626 - mse: 6.2342 - val_loss: 7.8166 - val_mse: 6.9278
Epoch 35/100
358/358 [==============================] - 5s 14ms/step - loss: 7.0655 - mse: 6.0895 - val_loss: 7.7267 - val_mse: 6.6891
Epoch 36/100
358/358 [==============================] - 5s 13ms/step - loss: 6.9679 - mse: 6.0368 - val_loss: 7.6034 - val_mse: 6.5356
Epoch 37/100
358/358 [==============================] - 5s 14ms/step - loss: 6.9613 - mse: 6.0795 - val_loss: 7.5522 - val_mse: 6.7222
Epoch 38/100
358/358 [==============================] - 4s 12ms/step - loss: 7.0079 - mse: 6.0620 - val_loss: 7.3254 - val_mse: 6.7018
Epoch 39/100
358/358 [==============================] - 5s 13ms/step - loss: 6.9752 - mse: 5.9678 - val_loss: 7.6082 - val_mse: 6.8405
Epoch 40/100
358/358 [==============================] - 4s 12ms/step - loss: 6.8712 - mse: 5.9475 - val_loss: 7.6724 - val_mse: 6.6324
Epoch 41/100
358/358 [==============================] - 5s 13ms/step - loss: 6.9053 - mse: 5.9111 - val_loss: 8.5137 - val_mse: 7.7260
Epoch 42/100
358/358 [==============================] - 5s 13ms/step - loss: 6.7378 - mse: 5.8993 - val_loss: 7.2622 - val_mse: 6.6142
Epoch 43/100
358/358 [==============================] - 4s 12ms/step - loss: 6.8369 - mse: 5.9022 - val_loss: 7.5613 - val_mse: 6.8527
Epoch 44/100
358/358 [==============================] - 5s 14ms/step - loss: 6.7802 - mse: 5.8444 - val_loss: 7.3737 - val_mse: 6.4707
Epoch 45/100
358/358 [==============================] - 4s 12ms/step - loss: 6.9552 - mse: 6.0110 - val_loss: 7.8480 - val_mse: 6.7347
Epoch 46/100
358/358 [==============================] - 5s 13ms/step - loss: 6.8904 - mse: 5.8933 - val_loss: 7.7557 - val_mse: 6.8180
Epoch 47/100
358/358 [==============================] - 4s 12ms/step - loss: 6.8171 - mse: 5.8959 - val_loss: 7.1823 - val_mse: 6.5033

## Show Model Uncertainty Range with TF Probability_____no_output_____**Question 9**: Now that we have trained a model with TF Probability layers, we can extract the mean and standard deviation for each prediction. Please fill in the answer for the m and s variables below. The code for getting the predictions is provided for you below._____no_output_____

feature_list = student_categorical_col_list + student_numerical_col_list
diabetes_x_tst = dict(d_test[feature_list])
diabetes_yhat = diabetes_model(diabetes_x_tst)
preds = diabetes_model.predict(diabetes_test_ds)_____no_output_____from student_utils import get_mean_std_from_preds
m, s = get_mean_std_from_preds(diabetes_yhat)_____no_output_____

## Show Prediction Output _____no_output_____

prob_outputs = {
    "pred": preds.flatten(),
    "actual_value": d_test['time_in_hospital'].values,
    "pred_mean": m.numpy().flatten(),
    "pred_std": s.numpy().flatten()
}
prob_output_df = pd.DataFrame(prob_outputs)_____no_output_____prob_output_df.head()_____no_output_____

## Convert Regression Output to Classification Output for Patient Selection_____no_output_____**Question 10**: Given the output predictions, convert it to a binary label for whether the patient meets the time criteria or does not (HINT: use the mean prediction numpy array). The expected output is a numpy array with a 1 or 0 based off if the prediction meets or doesnt meet the criteria._____no_output_____

from student_utils import get_student_binary_prediction
student_binary_prediction = get_student_binary_prediction(prob_output_df, 'pred_mean')_____no_output_____

### Add Binary Prediction to Test Dataframe_____no_output_____Using the student_binary_prediction output that is a numpy array with binary labels, we can use this to add to a dataframe to better visualize and also to prepare the data for the Aequitas toolkit. The Aequitas toolkit requires that the predictions be mapped to a binary label for the predictions (called 'score' field) and the actual value (called 'label_value')._____no_output_____

def add_pred_to_test(test_df, pred_np, demo_col_list):
    for c in demo_col_list:
        test_df[c] = test_df[c].astype(str)
    test_df['score'] = pred_np
    test_df['label_value'] = test_df['time_in_hospital'].apply(lambda x: 1 if x >=5 else 0)
    return test_df

pred_test_df = add_pred_to_test(d_test, student_binary_prediction, ['race', 'gender'])_____no_output_____pred_test_df[['patient_nbr', 'gender', 'race', 'time_in_hospital', 'score', 'label_value']].head()_____no_output_____

## Model Evaluation Metrics _____no_output_____**Question 11**: Now it is time to use the newly created binary labels in the 'pred_test_df' dataframe to evaluate the model with some common classification metrics. Please create a report summary of the performance of the model and be sure to give the ROC AUC, F1 score(weighted), class precision and recall scores. _____no_output_____For the report please be sure to include the following three parts:
- With a non-technical audience in mind, explain the precision-recall tradeoff in regard to how you have optimized your model.

- What are some areas of improvement for future iterations?_____no_output_____

# AUC, F1, precision and recall
print(classification_report(pred_test_df['label_value'], pred_test_df['score']))              precision    recall  f1-score   support

           0       0.84      0.72      0.77      9063
           1       0.61      0.77      0.68      5241

    accuracy                           0.74     14304
   macro avg       0.73      0.74      0.73     14304
weighted avg       0.76      0.74      0.74     14304

f1_score(pred_test_df['label_value'], pred_test_df['score'], average='weighted')_____no_output_____accuracy_score(pred_test_df['label_value'], pred_test_df['score'])_____no_output_____roc_auc_score(pred_test_df['label_value'], pred_test_df['score'])_____no_output_____precision_score(pred_test_df['label_value'], pred_test_df['score'])_____no_output_____recall_score(pred_test_df['label_value'], pred_test_df['score'])_____no_output_____

# 7. Evaluating Potential Model Biases with Aequitas Toolkit_____no_output_____## Prepare Data For Aequitas Bias Toolkit _____no_output_____Using the gender and race fields, we will prepare the data for the Aequitas Toolkit._____no_output_____

# Aequitas
from aequitas.preprocessing import preprocess_input_df
from aequitas.group import Group
from aequitas.plotting import Plot
from aequitas.bias import Bias
from aequitas.fairness import Fairness

ae_subset_df = pred_test_df[['race', 'gender', 'score', 'label_value']]
ae_df, _ = preprocess_input_df(ae_subset_df)
g = Group()
xtab, _ = g.get_crosstabs(ae_df)
absolute_metrics = g.list_absolute_metrics(xtab)
clean_xtab = xtab.fillna(-1)
aqp = Plot()
b = Bias()
/opt/conda/lib/python3.7/site-packages/aequitas/group.py:143: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df['score'] = df['score'].astype(float)

## Reference Group Selection_____no_output_____Below we have chosen the reference group for our analysis but feel free to select another one._____no_output_____

# test reference group with Caucasian Male
bdf = b.get_disparity_predefined_groups(clean_xtab,
                    original_df=ae_df,
                    ref_groups_dict={'race':'Caucasian', 'gender':'Male'
                                     },
                    alpha=0.05,
                    check_significance=False)

f = Fairness()
fdf = f.get_group_value_fairness(bdf)get_disparity_predefined_group()

## Race and Gender Bias Analysis for Patient Selection_____no_output_____**Question 12**: For the gender and race fields, please plot two metrics that are important for patient selection below and state whether there is a significant bias in your model across any of the groups along with justification for your statement._____no_output_____

# Plot two metrics
aqp.plot_group_metric(clean_xtab, 'fpr', min_group_size=0.05)
# Is there significant bias in your model for either race or gender?_____no_output_____aqp.plot_group_metric(clean_xtab, 'tpr', min_group_size=0.05)_____no_output_____aqp.plot_group_metric(clean_xtab, 'fnr', min_group_size=0.05)_____no_output_____aqp.plot_group_metric(clean_xtab, 'tnr', min_group_size=0.05)_____no_output_____

## Fairness Analysis Example - Relative to a Reference Group _____no_output_____**Question 13**: Earlier we defined our reference group and then calculated disparity metrics relative to this grouping. Please provide a visualization of the fairness evaluation for this reference group and analyze whether there is disparity._____no_output_____

# Reference group fairness plot
aqp.plot_fairness_disparity(bdf, group_metric='fnr', attribute_name='race', significance_alpha=0.05, min_group_size=0.05)_____no_output_____aqp.plot_fairness_disparity(fdf, group_metric='fnr', attribute_name='gender', significance_alpha=0.05, min_group_size=0.05)_____no_output_____aqp.plot_fairness_disparity(fdf, group_metric='fpr', attribute_name='race', significance_alpha=0.05, min_group_size=0.05)_____no_output_____aqp.plot_fairness_group(fdf, group_metric='fpr', title=True, min_group_size=0.05)_____no_output_____aqp.plot_fairness_group(fdf, group_metric='fnr', title=True)_____no_output_____

According to the graphs there is no any huge tendancy to a specific race or gender.
No specic bias in the model as well._____no_output_____


# Notebook from BrownDwarf/subsub
Path: notebooks/01_initial_exploratory/subsub_01-06_HDF5_by_order.ipynb

# subsub
## 06 Batch reduce everything!!
### part I- Generate per-order HDF5 files needed for chunking, also trim off the order ends.

Michael Gully-Santiago and Natalie Gosnell at Houndstooth in Austin, TX!!
Thursday, June 8, 2017

Updated on April 6 and April 11, 2018.

Let's do it._____no_output_____

import numpy as np
import matplotlib.pyplot as plt
% matplotlib inline
% config InlineBackend.figure_format = 'retina'
import seaborn as sns_____no_output_____from astropy.io import fits
import h5py/Users/obsidian/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters

## H-Band_____no_output_____- 64 is the target
- 72 is the A0V standard_____no_output_____

hdu = fits.open('../data/raw/SDCH_20150425_0064.spec_a0v.fits')
a0v_var = fits.open('../data/raw/SDCH_20150425_0072.variance.fits')
tar_var = fits.open('../data/raw/SDCH_20150425_0064.variance.fits')
hdr = hdu[0].header_____no_output_____hdu.info()Filename: ../data/raw/SDCH_20150425_0064.spec_a0v.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  SPEC_DIVIDE_A0V    1 PrimaryHDU     183   (2048, 28)   float64
  1  WAVELENGTH    1 ImageHDU       184   (2048, 28)   float64
  2  TGT_SPEC      1 ImageHDU       184   (2048, 28)   float32
  3  A0V_SPEC      1 ImageHDU       184   (2048, 28)   float32
  4  VEGA_SPEC     1 ImageHDU       184   (2048, 28)   float64

What are the entries in hdu?

| index | interpretation |
|--|--|
|0 | Telluric corrected target spectrum|
|1 | Wavelength in micron|
|2 | Raw 1D spectrum of target|
|3 | Raw 1D spectrum of A0V|
|4 | Model 1D spectrum of A0V|_____no_output_____The variance files contain only the variance from that individual target._____no_output_____

a0v_var.info()Filename: ../data/raw/SDCH_20150425_0072.variance.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU     181   (2048, 28)   float32

# Do the telluric correction

Here are the equations to compute the spectrum and its uncertainty. We need to compute the uncertainty manually because the IGRINS pipeline does not propagate the uncertainty attributable to the A0V star, and likelihood-based inference methods, like Starfish, work better when you have accurate noise models.

### propagating the uncertainty

"hats" are placed on the estimators-- models for the A0V star.
`std` refers to standard,
$\star$ refers to the target of interest, and
`raw` refers to the pipeline-processed-but-not-A0V-corrected raw flux.

$ f_{\star} = \frac{f_{raw, \star}}{f_{raw, std}} \times \hat{f}_{std}$

$ \sigma_{\star} = f_{\star} \sqrt{ \left( \frac{\sigma_{raw, std}}{f_{raw, std}}\right)^2 + \left( \frac{\sigma_{raw, \star}}{f_{raw,\star}} \right)^2 } \times \hat{f}_{std}$_____no_output_____

f_hat_std = hdu[4].data/np.nanmean(hdu[4].data)
f_star = hdu[2].data/hdu[3].data*f_hat_std
term1 = a0v_var[0].data / hdu[3].data**2
term2 = tar_var[0].data / hdu[2].data**2
sig_star = f_star * np.sqrt( term1 + term2 ) * f_hat_std_____no_output_____o=10
wls_out = hdu[1].data[o,:]*10000.0
fls_out = f_star[o,:] / np.nanmedian(f_star[o,:])

pix = np.arange(2048)
plt.plot(pix, fls_out, 'k-')
plt.ylim(0, 1.5*np.nanpercentile(fls_out, 85))_____no_output_____

## Plot it_____no_output_____We will trim between pixel 450 and 1950._____no_output_____

o=16
fls_out = f_star[o,:]
sig_out = sig_star[o,:]
wls_out = hdu[1].data[o,:]*10000.0
sig = sig_out

pix = np.arange(2048)
plt.plot(pix, fls_out, 'k.', ms=3)
plt.xlabel('pixel position')

plt.fill_between(pix, fls_out-sig, fls_out+sig, color='r')

plt.vlines(450, 0, 2, linestyles='--')
plt.vlines(1950, 0, 2, linestyles='--')
plt.ylim(0.0, 1.2);_____no_output_____lft = 450
rgt = 1950

for o in range(28):
    m = o + 98
    out_name = '../data/reduced/subsub_IGRINS_m{:03d}.hdf5'.format(m)
    fls_out = f_star[o,lft:rgt]
    sig_out = sig_star[o,lft:rgt]
    wls_out = hdu[1].data[o,lft:rgt]*10000.0
    msk_out = np.ones(len(wls_out), dtype=int)
    bad_ids = (sig_out < 0) | (sig_out != sig_out) | (fls_out != fls_out)
    good_ids = ~bad_ids
    f_new = h5py.File(out_name, 'w')
    f_new.create_dataset('fls', data=fls_out[good_ids])
    f_new.create_dataset('wls', data=wls_out[good_ids])
    f_new.create_dataset('sigmas', data=sig_out[good_ids])
    f_new.create_dataset('masks', data=msk_out[good_ids])
    print("{:03d}: {:.0f}  -  {:.0f}   {} {}".format(m, wls_out[0],
                                                     wls_out[-1], out_name, good_ids.sum()))
    f_new.close()098: 18154  -  18336   ../data/reduced/subsub_IGRINS_m098.hdf5 1197
099: 17975  -  18155   ../data/reduced/subsub_IGRINS_m099.hdf5 1500
100: 17800  -  17978   ../data/reduced/subsub_IGRINS_m100.hdf5 1500
101: 17628  -  17805   ../data/reduced/subsub_IGRINS_m101.hdf5 1500
102: 17460  -  17635   ../data/reduced/subsub_IGRINS_m102.hdf5 1500
103: 17295  -  17468   ../data/reduced/subsub_IGRINS_m103.hdf5 1500
104: 17133  -  17305   ../data/reduced/subsub_IGRINS_m104.hdf5 1500
105: 16975  -  17145   ../data/reduced/subsub_IGRINS_m105.hdf5 1500
106: 16819  -  16988   ../data/reduced/subsub_IGRINS_m106.hdf5 1500
107: 16667  -  16834   ../data/reduced/subsub_IGRINS_m107.hdf5 1500
108: 16517  -  16682   ../data/reduced/subsub_IGRINS_m108.hdf5 1500
109: 16370  -  16534   ../data/reduced/subsub_IGRINS_m109.hdf5 1500
110: 16226  -  16388   ../data/reduced/subsub_IGRINS_m110.hdf5 1500
111: 16084  -  16245   ../data/reduced/subsub_IGRINS_m111.hdf5 1500
112: 15945  -  16105   ../data/reduced/subsub_IGRINS_m112.hdf5 1500
113: 15809  -  15967   ../data/reduced/subsub_IGRINS_m113.hdf5 1500
114: 15675  -  15832   ../data/reduced/subsub_IGRINS_m114.hdf5 1500
115: 15543  -  15699   ../data/reduced/subsub_IGRINS_m115.hdf5 1500
116: 15414  -  15568   ../data/reduced/subsub_IGRINS_m116.hdf5 1500
117: 15287  -  15439   ../data/reduced/subsub_IGRINS_m117.hdf5 1500
118: 15162  -  15313   ../data/reduced/subsub_IGRINS_m118.hdf5 1500
119: 15039  -  15189   ../data/reduced/subsub_IGRINS_m119.hdf5 1500
120: 14919  -  15067   ../data/reduced/subsub_IGRINS_m120.hdf5 1500
121: 14800  -  14947   ../data/reduced/subsub_IGRINS_m121.hdf5 1500
122: 14683  -  14829   ../data/reduced/subsub_IGRINS_m122.hdf5 1500
123: 14569  -  14713   ../data/reduced/subsub_IGRINS_m123.hdf5 1500
124: 14456  -  14599   ../data/reduced/subsub_IGRINS_m124.hdf5 1390
125: 14345  -  14487   ../data/reduced/subsub_IGRINS_m125.hdf5 133

## K-band_____no_output_____Skipping K-band because it's low signal-to-noise!_____no_output_____### The end!_____no_output_____


# Notebook from ofou/course-content
Path: tutorials/W1D2_ModelingPractice/W1D2_Tutorial1.ipynb

[![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/tutorials/W1D2_ModelingPractice/W1D2_Tutorial1.ipynb)_____no_output_____# Tutorial 1: Framing the Question
**Week 1, Day 2: Modeling Practice**

**By Neuromatch Academy**

__Content creators:__ Marius 't Hart, Megan Peters, Paul Schrater, Gunnar Blohm

__Content reviewers:__ Eric DeWitt, Tara van Viegen, Marius Pachitariu

__Production editors:__ Ella Batty _____no_output_____**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

_____no_output_____---
# Tutorial objectives
Yesterday you gained some understanding of what models can buy us in neuroscience. But how do you build a model? Today, we will try to clarify the process of computational modeling, by thinking through the logic of modeling based on your project ideas.

We assume that you have a general idea of a project in mind, i.e. a preliminary question, and/or phenomenon you would like to understand. You should have started developing a project idea yesterday with [this brainstorming demo](https://youtu.be/H6rSlZzlrgQ). Maybe you have a goal in mind. We will now work through the 4 first steps of modeling ([Blohm et al., 2019](https://doi.org/10.1523/ENEURO.0352-19.2019)):

**Framing the question**

1. finding a phenomenon and a question to ask about it
2. understanding the state of the art
3. determining the basic ingredients
4. formulating specific, mathematically defined hypotheses

The remaining steps 5-10 will be covered in a second notebook that you can consult throughout the modeling process when you work on your projects.

**Importantly**, we will guide you through Steps 1-4 today. After you do more work on projects, you likely have to revite some or all of these steps *before* you move on the the remaining steps of modeling.

**Note**: there will be no coding today. It's important that you think through the different steps of this how-to-model tutorial to maximize your chance of succeeding in your group projects. **Also**: "Models" here can be data analysis pipelines, not just computational models...

**Think! Sections**: All activities you should perform are labeled with **Think!**. These are discussion based exercises and can be found in the Table of Content on the left side of the notebook. Make sure you complete all within a section before moving on!

### Demos
We will demo the modeling process to you based on the train illusion. The introductory video will explain the phenomenon to you. Then we will do roleplay to showcase some common pitfalls to you based on a computational modeling project around the train illusion. In addition to the computational model, we will also provide a data neuroscience project example to you so you can appreciate similarities and differences.

Enjoy!
_____no_output_____

# @title Video 1: Introduction to tutorial
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1Mf4y1b7xS", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="GyGNs1fLIYQ", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)_____no_output_____

# Setup_____no_output_____

# Imports

import numpy as np
import matplotlib.pyplot as plt

# for random distributions:
from scipy.stats import norm, poisson

# for logistic regression:
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score_____no_output_____# @title Plotting Functions

def rasterplot(spikes,movement,trial):

  [movements, trials, neurons, timepoints] = np.shape(spikes)

  trial_spikes = spikes[movement,trial,:,:]

  trial_events = [((trial_spikes[x,:] > 0).nonzero()[0]-150)/100 for x in range(neurons)]

  plt.figure()
  dt=1/100
  plt.eventplot(trial_events, linewidths=1);
  plt.title('movement: %d - trial: %d'%(movement, trial))
  plt.ylabel('neuron')
  plt.xlabel('time [s]')

def plotCrossValAccuracies(accuracies):
  f, ax = plt.subplots(figsize=(8, 3))
  ax.boxplot(accuracies, vert=False, widths=.7)
  ax.scatter(accuracies, np.ones(8))
  ax.set(
    xlabel="Accuracy",
    yticks=[],
    title=f"Average test accuracy: {accuracies.mean():.2%}"
  )
  ax.spines["left"].set_visible(False)_____no_output_____#@title Generate Data

def generateSpikeTrains():

  gain        = 2
  neurons     = 50
  movements   = [0,1,2]
  repetitions = 800

  np.random.seed(37)

  # set up the basic parameters:
  dt = 1/100
  start, stop = -1.5, 1.5
  t = np.arange(start, stop+dt, dt)                                            # a time interval
  Velocity_sigma = 0.5                                                         # std dev of the velocity profile
  Velocity_Profile = norm.pdf(t,0,Velocity_sigma)/norm.pdf(0,0,Velocity_sigma) # The Gaussian velocity profile, normalized to a peak of 1

  # set up the neuron properties:
  Gains = np.random.rand(neurons) * gain          # random sensitivity between 0 and `gain`
  FRs   = (np.random.rand(neurons) * 60 ) - 10    # random base firing rate between -10 and 50

  # output matrix will have this shape:
  target_shape = [len(movements), repetitions, neurons, len(Velocity_Profile)]

  # build matrix for spikes, first, they depend on the velocity profile:
  Spikes = np.repeat(Velocity_Profile.reshape([1,1,1,len(Velocity_Profile)]),len(movements)*repetitions*neurons,axis=2).reshape(target_shape)

  # multiplied by gains:
  S_gains = np.repeat(np.repeat(Gains.reshape([1,1,neurons]), len(movements)*repetitions, axis=1).reshape(target_shape[:3]), len(Velocity_Profile)).reshape(target_shape)
  Spikes = Spikes * S_gains

  # and multiplied by the movement:
  S_moves = np.repeat( np.array(movements).reshape([len(movements),1,1,1]), repetitions*neurons*len(Velocity_Profile), axis=3 ).reshape(target_shape)
  Spikes = Spikes * S_moves

  # on top of a baseline firing rate:
  S_FR = np.repeat(np.repeat(FRs.reshape([1,1,neurons]), len(movements)*repetitions, axis=1).reshape(target_shape[:3]), len(Velocity_Profile)).reshape(target_shape)
  Spikes = Spikes + S_FR

  # can not run the poisson random number generator on input lower than 0:
  Spikes = np.where(Spikes < 0, 0, Spikes)

  # so far, these were expected firing rates per second, correct for dt:
  Spikes = poisson.rvs(Spikes * dt)

  return(Spikes)

def subsetPerception(spikes):

  movements = [0,1,2]
  split     = 400
  subset    = 40
  hwin      = 3

  [num_movements, repetitions, neurons, timepoints] = np.shape(spikes)

  decision = np.zeros([num_movements, repetitions])

  # ground truth for logistic regression:
  y_train = np.repeat([0,1,1],split)
  y_test  = np.repeat([0,1,1],repetitions-split)

  m_train = np.repeat(movements, split)
  m_test  = np.repeat(movements, split)

  # reproduce the time points:
  dt = 1/100
  start, stop = -1.5, 1.5
  t = np.arange(start, stop+dt, dt)

  w_idx = list( (abs(t) < (hwin*dt)).nonzero()[0] )
  w_0 = min(w_idx)
  w_1 = max(w_idx)+1  # python...

  # get the total spike counts from stationary and movement trials:
  spikes_stat = np.sum( spikes[0,:,:,:], axis=2)
  spikes_move = np.sum( spikes[1:,:,:,:], axis=3)

  train_spikes_stat = spikes_stat[:split,:]
  train_spikes_move = spikes_move[:,:split,:].reshape([-1,neurons])

  test_spikes_stat = spikes_stat[split:,:]
  test_spikes_move = spikes_move[:,split:,:].reshape([-1,neurons])

  # data to use to predict y:
  x_train = np.concatenate((train_spikes_stat, train_spikes_move))
  x_test  = np.concatenate(( test_spikes_stat,  test_spikes_move))

  # this line creates a logistics regression model object, and immediately fits it:
  population_model = LogisticRegression(solver='liblinear', random_state=0).fit(x_train, y_train)

  # solver, one of: 'liblinear', 'newton-cg', 'lbfgs', 'sag', and 'saga'
  # some of those require certain other options
  #print(population_model.coef_)       # slope
  #print(population_model.intercept_)  # intercept

  ground_truth = np.array(population_model.predict(x_test))
  ground_truth = ground_truth.reshape([3,-1])

  output = {}
  output['perception'] = ground_truth
  output['spikes']     = spikes[:,split:,:subset,:]

  return(output)

def getData():

  spikes = generateSpikeTrains()

  dataset = subsetPerception(spikes=spikes)

  return(dataset)

dataset = getData()
perception = dataset['perception']
spikes = dataset['spikes']_____no_output_____

----
# Step 1: Finding a phenomenon and a question to ask about it
_____no_output_____

# @title Video 2: Asking a question
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1VK4y1M7dc", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="4Gl8X_y_uoA", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)_____no_output_____# @title Example projects step 1
from ipywidgets import widgets
from IPython.display import Markdown

markdown1 = '''

## Step 1

The train illusion occurs when sitting on a train and viewing another train outside the window. Suddenly, the other train *seems* to move, i.e. you experience visual motion of the other train relative to your train. But which train is actually moving?

Often people have the wrong percept. In particular, they think their own train might be moving when it's the other train that moves; or vice versa. The illusion is usually resolved once you gain vision of the surroundings that lets you disambiguate the relative motion; or if you experience strong vibrations indicating that it is indeed your own train that is in motion.

We asked the following (arbitrary) question for our demo project: "How do noisy vestibular estimates of motion lead to illusory percepts of self motion?"

'''

markdown2 = '''
## Step 1

The train illusion occurs when sitting on a train and viewing another train outside the window. Suddenly, the other train *seems* to move, i.e. you experience visual motion of the other train relative to your train. But which train is actually moving?

Often people mix this up. In particular, they think their own train might be moving when it's the other train that moves; or vice versa. The illusion is usually resolved once you gain vision of the surroundings that lets you disambiguate the relative motion; or if you experience strong vibrations indicating that it is indeed your own train that is in motion.

We assume that we have build the train illusion model (see the other example project colab). That model predicts that accumulated sensory evidence from vestibular signals determines the decision of whether self-motion is experienced or not. We now have vestibular neuron data (simulated in our case, but let's pretend) and would like to see if that prediction holds true.

The data contains *N* neurons and *M* trials for each of 3 motion conditions: no self-motion, slowly accelerating self-motion and faster accelerating self-motion. In our data,
*N* = 40 and *M* = 400.

**So we can ask the following question**: "Does accumulated vestibular neuron activity correlate with self-motion judgements?"

'''

out2 = widgets.Output()
with out2:
  display(Markdown(markdown2))

out1 = widgets.Output()
with out1:
  display(Markdown(markdown1))

out = widgets.Tab([out1, out2])
out.set_title(0, 'Computational Model')
out.set_title(1, 'Data Analysis')

display(out)_____no_output_____

## Think! 1: Asking your own question

*Please discuss the following for about 25 min*

You should already have a project idea from your brainstorming yesterday. **Write down the phenomenon, question and goal(s) if you have them.**

As a reminder, here is what you should discuss and write down:

* What exact aspect of data needs modeling?
  * Answer this question clearly and precisely!
Otherwise you will get lost (almost guaranteed)
  * Write everything down!
  * Also identify aspects of data that you do not want to address (yet)
* Define an evaluation method!
  * How will you know your modeling is good?
  * E.g. comparison to specific data (quantitative method of comparison?)
* For computational models: think of an experiment that could test your model
  * You essentially want your model to interface with this experiment, i.e. you want to simulate this experiment

You can find interesting questions by looking for phenomena that differ from your expectations. In *what* way does it differ? *How* could that be explained (starting to think about mechanistic questions and structural hypotheses)? *Why* could it be the way it is? What experiment could you design to investigate this phenomenon? What kind of data would you need?_____no_output_____**Make sure to avoid the pitfalls!**

Click here for a recap on pitfalls

Question is too general

Remember: science advances one small step at the time. Get the small step right…

Precise aspect of phenomenon you want to model is unclear

You will fail to ask a meaningful question

You have already chosen a toolkit

This will prevent you from thinking deeply about the best way to answer your scientific question

You don’t have a clear goal

What do you want to get out of modeling?

You don’t have a potential experiment in mind

This will help concretize your objectives and think through the logic behind your goal

_____no_output_____**Note**

The hardest part is Step 1. Once that is properly set up, all other should be easier. **BUT**: often you think that Step 1 is done only to figure out in later steps (anywhere really) that you were not as clear on your question and goal than you thought. Revisiting Step 1 is frequent necessity. Don't feel bad about it. You can revisit Step 1 later; for now, let's move on to the nest step._____no_output_____----
# Step 2: Understanding the state of the art & background
_____no_output_____Here you will do a literature review (**to be done AFTER this tutorial!**)._____no_output_____

# @title Video 3: Literature Review & Background Knowledge
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1by4y1M7TZ", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="d8zriLaMc14", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)_____no_output_____# @title Example projects step 2
from ipywidgets import widgets
from IPython.display import Markdown

markdown1 = '''

## Step 2

You have learned all about the vestibular system in the Intro video. This is also where you would do a literature search to learn more about what's known about self-motion perception and vestibular signals. You would also want to examine any attempts to model self-motion, perceptual decision making and vestibular processing.
'''

markdown21 = '''
## Step 2

While it seems a well-known fact that vestibular signals are noisy, we should check if we can also find this in the literature.

Let's also see what's in our data, there should be a 4d array called `spikes` that has spike counts (positive integers), a 2d array called `perception` with self-motion judgements (0=no motion or 1=motion). Let's see what this data looks like:

'''

markdown22 = '''

In the `spikes` array, we see our 3 acceleration conditions (first dimension), with 400 trials each (second dimensions) and simultaneous recordings from 40 neurons (third dimension), across 3 seconds in 10 ms bins (fourth dimension). The first two dimensions are also there in the `perception` array.

Perfect perception would have looked like [0, 1, 1]. The average judgements are far from correct (lots of self-motion illusions) but they do make some sense: it's closer to 0 in the no-motion condition and closer to 1 in both of the real-motion conditions.

The idea of our project is that the vestibular signals are noisy so that they might be mis-interpreted by the brain. Let's see if we can reproduce the stimuli from the data:

'''

markdown23 = '''

Blue is the no-motion condition, and produces flat average spike counts across the 3 s time interval. The orange and green line do show a bell-shaped curve that corresponds to the acceleration profile. But there also seems to be considerable noise: exactly what we need. Let's see what the spike trains for a single trial look like:

'''

markdown24 = '''

You can change the trial number in the bit of code above to compare what the rasterplots look like in different trials. You'll notice that they all look kind of the same: the 3 conditions are very hard (impossible?) to distinguish by eye-balling.

Now that we have seen the data, let's see if we can extract self-motion judgements from the spike counts.

'''

display(Markdown(r""))

out2 = widgets.Output()
with out2:
  display(Markdown(markdown21))
  print(f'The shape of `spikes` is: {np.shape(spikes)}')
  print(f'The shape of `perception` is: {np.shape(perception)}')
  print(f'The mean of `perception` is: {np.mean(perception, axis=1)}')
  display(Markdown(markdown22))

  for move_no in range(3):
    plt.plot(np.arange(-1.5,1.5+(1/100),(1/100)),np.mean(np.mean(spikes[move_no,:,:,:], axis=0), axis=0), label=['no motion', '$1 m/s^2$', '$2 m/s^2$'][move_no])
  plt.xlabel('time [s]');
  plt.ylabel('averaged spike counts');
  plt.legend()
  plt.show()

  display(Markdown(markdown23))

  for move in range(3):
    rasterplot(spikes = spikes, movement = move, trial = 0)
  plt.show()

  display(Markdown(markdown24))

out1 = widgets.Output()
with out1:
  display(Markdown(markdown1))

out = widgets.Tab([out1, out2])
out.set_title(0, 'Computational Model')
out.set_title(1, 'Data Analysis')

display(out)_____no_output_____

Here you will do a literature review (**to be done AFTER this tutorial!**). For the projects, do not spend too much time on this. A thorough literature review could take weeks or months depending on your prior knowledge of the field...

The important thing for your project here is not to exhaustively survey the literature but rather to learn the process of modeling. 1-2 days of digging into the literature should be enough!

**Here is what you should get out of it**:
* Survey the literature
  * What’s known?
  * What has already been done?
  * Previous models as a starting point?
  * What hypotheses have been emitted in the field?
  * Are there any alternative / complementary modeling approaches?
* What skill sets are required?
  * Do I need learn something before I can start?
  * Ensure that no important aspect is missed
* Potentially provides specific data sets / alternative modeling approaches for comparison
_____no_output_____**Do this AFTER the tutorial**_____no_output_____----
# Step 3: Determining the basic ingredients
_____no_output_____

# @title Video 4: Determining basic ingredients
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1Mq4y1x77s", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="XpEj-p7JkFE", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)_____no_output_____# @title Example projects step 3
from ipywidgets import widgets
from IPython.display import Markdown, Math

markdown1 = r'''

## Step 3

We determined that we probably needed the following ingredients for our model:

* Vestibular input: *v(t)*

* Binary decision output: *d* - time dependent?

* Decision threshold: θ

* A filter (maybe running average?): *f*

* An integration mechanism to get from vestibular acceleration to sensed velocity: ∫

'''

markdown2 = '''
## Step 3

In order to address our question we need to design an appropriate computational data analysis pipeline. We did some brainstorming and think that we need to somehow extract the self-motion judgements from the spike counts of our neurons. Based on that, our algorithm needs to make a decision: was there self motion or not? This is a classical 2-choice classification problem. We will have to transform the raw spike data into the right input for the algorithm (spike pre-processing).

So we determined that we probably needed the following ingredients:

* spike trains *S* of 3-second trials (10ms spike bins)
* ground truth movement *mr* (real) and perceived movement *mp*
* some form of classifier *C* giving us a classification *c*
* spike pre-processing

'''

# No idea why this is necessary but math doesn't render properly without it
display(Markdown(r""))

out2 = widgets.Output()
with out2:
  display(Markdown(markdown2))

out1 = widgets.Output()
with out1:
  display(Markdown(markdown1))

out = widgets.Tab([out1, out2])
out.set_title(0, 'Computational Model')
out.set_title(1, 'Data Analysis')

display(out)_____no_output_____

## Think! 3: Determine your basic ingredients

*Please discuss the following for about 25 min*

This will allow you to think deeper about what your modeling project will need. It's a crucial step before you can formulate hypotheses because you first need to understand what your modeling approach will need. There are 2 aspects you want to think about:

1.   What parameters / variables are needed?]
      * Constants?
      * Do they change over space, time, conditions…?
      * What details can be omitted?
      * Constraints, initial conditions?
      * Model inputs / outputs?
2.   Variables needed to describe the process to be modelled?
      * Brainstorming!
      * What can be observed / measured?  latent variables?
      * Where do these variables come from?
      * Do any abstract concepts need to be instantiated as variables?
        * E.g. value, utility, uncertainty, cost, salience, goals, strategy, plant, dynamics
        * Instantiate them so that they relate to potential measurements!

This is a step where your prior knowledge and intuition is tested. You want to end up with an inventory of *specific* concepts and/or interactions that need to be instantiated. _____no_output_____**Make sure to avoid the pitfalls!**

Click here for a recap on pitfalls

I’m experienced, I don’t need to think about ingredients anymore

Or so you think…

I can’t think of any ingredients

Think about the potential experiment. What are your stimuli? What parameters? What would you control? What do you measure?

I have all inputs and outputs

Good! But what will link them? Thinking about that will start shaping your model and hypotheses

I can’t think of any links (= mechanisms)

You will acquire a library of potential mechanisms as you keep modeling and learning

But the literature will often give you hints through hypotheses

If you still can't think of links, then maybe you're missing ingredients?

_____no_output_____----
# Step 4: Formulating specific, mathematically defined hypotheses

_____no_output_____

# @title Video 5: Formulating a hypothesis
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1fh411h7aX", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="nHXMSXLcd9A", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)_____no_output_____# @title Example projects step 4
from ipywidgets import widgets
from IPython.display import Markdown

# Not writing in latex because that didn't render in jupyterbook

markdown1 = r'''

## Step 4

Our main hypothesis is that the strength of the illusion has a linear relationship to the amplitude of vestibular noise.

Mathematically, this would write as

S = k ⋅ N

where *S* is the illusion strength and *N* is the noise level, and *k* is a free parameter.
>we could simply use the frequency of occurance across repetitions as the "strength of the illusion"

We would get the noise as the standard deviation of *v(t)*, i.e.

N = E[v(t)2],

 where **E** stands for the expected value.

Do we need to take the average across time points?
> doesn't really matter because we have the generative process, so we can just use the σ that we define

'''

markdown2 = '''
## Step 4

We think that noise in the signal drives whether or not people perceive self motion. Maybe the brain uses the strongest signal at peak acceleration to decide on self motion, but we actually think it is better to accumulate evidence over some period of time. We want to test this. The noise idea also means that when the signal-to-noise ratio is higher, the brain does better, and this would be in the faster acceleration condition. We want to test this too.

We came up with the following hypotheses focussing on specific details of our overall research question:

* Hyp 1: Accumulated vestibular spike rates explain self-motion judgements better than average spike rates around peak acceleration.
* Hyp 2: Classification performance should be better for faster vs slower self-motion.

> There are many other hypotheses you could come up with, but for simplicity, let's go with those.

Mathematically, we can write our hypotheses as follows (using our above ingredients):
* Hyp 1: **E**(caccum) > **E**(cwin)
* Hyp 2: **E**(cfast) > **E**(cslow)

Where **E** denotes taking the expected value (in this case the mean) of its argument: classification outcome in a given trial type.

'''

# No idea why this is necessary but math doesn't render properly without it
display(Markdown(r""))

out2 = widgets.Output()
with out2:
  display(Markdown(markdown2))

out1 = widgets.Output()
with out1:
  display(Markdown(markdown1))

out = widgets.Tab([out1, out2])
out.set_title(0, 'Computational Model')
out.set_title(1, 'Data Analysis')

display(out)_____no_output_____

## Think! 4: Formulating your hypothesis

*Please discuss the following for about 25 min*

Once you have your question and goal lines up, you have done a literature review (let's assume for now) and you have thought about ingredients needed for your model, you're now ready to start thinking about *specific* hypotheses.

Formulating hypotheses really consists of two consecutive steps:

1.   You think about the hypotheses in words by relating ingredients identified in Step 3
      * What is the model mechanism expected to do?
      * How are different parameters expected to influence model results?
2.   You then express these hypotheses in mathematical language by giving the ingredients identified in Step 3 specific variable names.
      * Be explicit, e.g. $y(t)=f(x(t),k)$ but $z(t)$ doesn’t influence $y$

There are also "structural hypotheses" that make assumptions on what model components you hypothesize will be crucial to capture the phenomenon at hand.

**Important**: Formulating the hypotheses is the last step before starting to model. This step determines the model approach and ingredients. It provides a more detailed description of the question / goal from Step 1. The more precise the hypotheses, the easier the model will be to justify.
_____no_output_____**Make sure to avoid the pitfalls!**

Click here for a recap on pitfalls

I don’t need hypotheses, I will just play around with the model

Hypotheses help determine and specify goals. You can (and should) still play…

My hypotheses don’t match my question (or vice versa)

This is a normal part of the process!

You need to loop back to Step 1 and revisit your question / phenomenon / goals

I can’t write down a math hypothesis

Often that means you lack ingredients and/or clarity on the hypothesis

OR: you have a “structural” hypothesis, i.e. you expect a certain model component to be crucial in explaining the phenomenon / answering the question

_____no_output_____----
# Summary

In this tutorial, we worked through some steps of the process of modeling.

- We defined a phenomenon and formulated a question (step 1)
- We collected information the state-of-the-art about the topic (step 2)
- We determined the basic ingredients (step 3), and used these to formulate a specific mathematically defined hypothesis (step 4)

You are now in a position that you could start modeling without getting lost. But remember: you might have to work through steps 1-4 again after doing a literature review and/or if there were other pitfalls you identified along the way (which is totally normal)._____no_output_____----
# Next steps
In [a follow-up notebook](https://compneuro.neuromatch.io/projects/modelingsteps/ModelingSteps_5through10.html), we will continue with the steps 5-10 to guide you through the implementation and completion stages of the projects. You can also find this in the Modeling Steps section of the Project Booklet._____no_output_____----
# Reading
Blohm G, Kording KP, Schrater PR (2020). _A How-to-Model Guide for Neuroscience_. eNeuro, 7(1) ENEURO.0352-19.2019. https://doi.org/10.1523/ENEURO.0352-19.2019

Kording KP, Blohm G, Schrater P, Kay K (2020). _Appreciating the variety of goals in computational neuroscience_. Neurons, Behavior, Data Analysis, and Theory 3(6). https://nbdt.scholasticahq.com/article/16723-appreciating-the-variety-of-goals-in-computational-neuroscience

Schrater PR, Peters MK, Kording KP, Blohm G (2019). _Modeling in Neuroscience as a Decision Process_. OSF pre-print. https://osf.io/w56vt/_____no_output_____