# Notebook from Warlock-bit/Zomato-Rating-Prediction
Path: zomato_bangalore_report2.ipynb

# Predicting Restaurant Rating Of New Restaurant on Zomato
### Machine Learning capstone project

# Project Overview
Zomato is a restaurant search and discovery website/application which operates in many major cities spread across 24 countries.
Zomato helps land customers over the restaurants through many search options based on restaurants location, restaurant type, cuisines they serve and helps discover better using filters and sort by options for advance searches.
    in this project dataset for restaurants in **Bengaluru,India** is used [dataset link](https://www.kaggle.com/himanshupoddar/zomato-bangalore-restaurants)

> **The dataset contains ratings marked as "NEW" for 2000+ rows. These rows will be used for predicting after the model is trained and tested.**

# Problem Statement
**Rating System of Zomato**

The restaurants on Zomato have ratings between 1-5 with an interval of .1. example 1.1,3.2,3.4. The customers gives ratings which is denoted as stars, a customer reviews/rates a restaurant by giving a score/star between 1 to 5. **The final rating is not the average** of the ratings given by the customers but the ratings of restaurants are distributed over a **normal distribution curve for a city** and can change by ratings of other restaurants.[Zomato rating system](https://www.zomato.com/blog/project-warp-fighting-bias-on-zomato)

src: https://www.zomato.com/bangalore/indiranagar-restaurants

**Example to understand better**

For this reason the ratings may change in time based on ratings/reviews given on another restaurant. That is the ratings are relative to each other.

For example if a restaurant A gets 5 star ratings 5 times, it does not mean that the final rating will be 5. It can be 3.2, 4.5. If another restaurant B gets 3, 5 stars and 1, 4 star and gets 2 reviews will likely get higher rating than A because it got 2 positive reviews.

> **Problem:**
 If a New restaurant wants to get on Zomato it will be placed on the normal curve. The normal curve is centered around 3.8,3.9 and most of the restaurants fight to be on the right side of this curve i.e on the right side of 3.8. Even if the customer rating for the resrtaurant is good the overall rating may say otherwise based on complicated Zomato rating algorithms which compares the rating/review frequency and uses NLP for reviews.

 **Rating bins**
There are 8 bins in Zomato between 1-5 with and interval of .5. Each of the rating is color coded. **This project is a classification problem, which predicts the bin, the rating of the NEW restaurant will fall into**

**The Aim of this project** is to make a Deep Learning model which predicts the bin of the NEW restaurant will fall into.

src: https://www.zomato.com/bangalore/indiranagar-restaurants

**Why there is a need of better rating?**

-	Zomato charges less commission for higher rating rstaurants
-	Zomato has filter options(rating 3.5+) which displays better rated restaurants in the search
-	Restaurant shows on top over the other insearch
- Attracts more footfall to the Restaurants.

_____no_output_____## Table of Contents

Data Exploration

Exploring visulization

Book table and Online order

Visulazing Rating

Listed in Location

List in Restaurant Type

Visualizing different Cuisines

Data Pre-Processing

Assessing columns missing

Assessing Row missing

Preprocessing columns

Imputing and Feature scaling

Training and Testing set

Training and Testing Benchmarck Model

Training, Optimizing and Testing Neural Network Model

Predicting New Restaurant Color Bin

Limitation,Conclusion and Improvement

> After each Visualization or process there is an **Inference** section
After Each Section There is a **Summary** section_____no_output_____

#importing all the libraries here
#imorting numpty and pandas to handle dataframe
import numpy as np
import pandas as pd
#import matplotlib to plot graphs
import matplotlib.pyplot as plt

#splting data into train and test data
from sklearn.model_selection import train_test_split
#importing simple imputer to deal with missing values
from sklearn.impute import SimpleImputer
#Used Minax Scalar for scaling values between [0,1]
from sklearn.preprocessing import MinMaxScaler

#bencahmarck model and evaluration metrics
from sklearn.linear_model import LinearRegression
from sklearn import metrics

#Librabries for creating and training neural Networks
import torch

#helper function
from helper import *

#functions for exploratory visualizations
import visuals as vs

#functions for training,processing and predicitng
import train as tr
import predict as pr

%matplotlib inline _____no_output_____#copying data in csv file to dataframe
bengaluru_res_df = pd.read_csv("zomato/zomato.csv")_____no_output_____

## 1. Exploring Data_____no_output_____

# displaying the first two rows
bengaluru_res_df.head(2)_____no_output_____bengaluru_res_df.shape_____no_output_____bengaluru_res_df.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 51717 entries, 0 to 51716
Data columns (total 17 columns):
url                            51717 non-null object
address                        51717 non-null object
name                           51717 non-null object
online_order                   51717 non-null object
book_table                     51717 non-null object
rate                           43942 non-null object
votes                          51717 non-null int64
phone                          50509 non-null object
location                       51696 non-null object
rest_type                      51490 non-null object
dish_liked                     23639 non-null object
cuisines                       51672 non-null object
approx_cost(for two people)    51371 non-null object
reviews_list                   51717 non-null object
menu_item                      51717 non-null object
listed_in(type)                51717 non-null object
listed_in(city)                51717 non-null object
dtypes: int64(1), object(16)
memory usage: 6.7+ MB
bengaluru_res_df.isnull().sum()_____no_output_____bengaluru_res_df.isnull().sum().sum()_____no_output_____bengaluru_res_df.describe()_____no_output_____#total number of unique restaurants
bengaluru_res_df['name'].value_counts().shape[0]_____no_output_____bengaluru_res_df.duplicated().sum()_____no_output_____

**About Data set**
- Dataset size: 51,717 rows x 17 columns
- All columns are of type "object" except for votes
- 7775 ratings are missin in the rating column
- 1208 phone numbers are missing in the phone column
- 21 are mising location
- 28,078 dished liked are missing
- 45 cuisines are missing
- 346 approx cost for two persons is missing
- No duplicated row
- Basic statistics is discussed after processing the data click here for stats_____no_output_____**Columns details**
1.	**url(object)** – This is simply a string with tells the URL which is almost common for all as it is starting with Zomato.com/xyz/abc.
2.	**Address(object)** – Complete address of the restaurant. Contains the full address which is not required for the analysis, can be deleted.
3.	**Name(object)** – Name of the restaurant, The name is unique and is different languages such as Gabar(Hindi Word), Asha(Hindi word), Twist (Englih) – it will be difficult to categories 51,717 values in the ANme column. For these reasons the column can be dropped.
4.	**Online order(Object)** – Yes or No, useful for predicting by converting it to binary.
5.	**Book Table(Object)** – Restaurant provides online table booking service (yes or no), useful for predicting by converting it to binary.
6.	**Rate (Target)**– Overall rating of the restaurant between” 1 – 5” upto one decimal point example 4.2. “New” for new restaurants.
7.	**Votes(int)** – Total number of reviews/votes, not useful for predicting the target variable, needs to be dropped.
8.	**Phone(object)** – Phone number of the restaurant. The counts of phone numbers provided could help in predicting target.
9.	**Location(object)** – Neighborhood of the Restaurant in Bengaluru, important for predicting rating. One- hot encode to change it to dummies.
10.	**Rest_type(object)** – Type of the restaurant, important for predicting rating. One- hot encode to change it to dummies.
11.	**Dish_liked(object)** – Dish liked by people in the restaurant,
12.	**Cuisines(object)** – Cuisines types in the re restaurants. Types are comma separated
13.	**Approx_cost(object)** – expected cost for meal of two in Rupees, important for prediction
14.	 **Reviews_list(object)** - list of tuples containing reviews for the restaurant, each tuple consists of two values, rating and review by the customer. Needs to be dropped for the prediction for new restaurant.
15.	**Menu_item(Object)** – List of menus available in restaurants. Contains more than 54% missing values, needs to be dropped.
16.	**Listed_in(type)** -  type of meal
17.	**Listed_in(city)** - contains the neighborhood in which the restaurant is listed - duplicate column can be dropped
_____no_output_____
## 2. Exploring visulization
Visualizing columns using matplotlib_____no_output_____
### 2.1 Book table and Online order _____no_output_____

#References for visualizations used in visuals.py
#https://stackoverflow.com/questions/22408237/named-colors-in-matplotlib
#https://stackoverflow.com/questions/11927715/how-to-give-a-pandas-matplotlib-bar-graph-custom-colors
#https://www.geeksforgeeks.org/append-extend-python/
#https://stackoverflow.com/questions/22408237/named-colors-in-matplotlib
#https://stackoverflow.com/questions/11927715/how-to-give-a-pandas-matplotlib-bar-graph-custom-colors
#https://matplotlib.org/3.1.1/tutorials/text/text_intro.html -
#https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.text.html
#https://www.geeksforgeeks.org/python-pandas-series-sort_values/
#https://stackoverflow.com/questions/18837262/convert-python-dict-into-a-dataframe_____no_output_____#Storing uniques values of Book table in series
series_book_table = bengaluru_res_df.groupby('book_table').count()['online_order']
print("REstaurants with book table options:\n Yes:",series_book_table['Yes'],"\nNo:", series_book_table['No'])
print("Total Number of restaurants:",bengaluru_res_df.shape[0]," total Yes + No:",(series_book_table['Yes']+series_book_table['No']))REstaurants with book table options:
 Yes: 6449
No: 45268
Total Number of restaurants: 51717  total Yes + No: 51717
vs.bar_booktable(bengaluru_res_df)_____no_output_____

>**Interpretation:**
out of 51,717 records 87.5% of restaurants do not have options to book table online using Zomato_____no_output_____

#counting the unique values and storing it in series
series_online_order = bengaluru_res_df.groupby('online_order').count()['book_table']
print("Restaurants with online order options:\n Yes:",series_online_order['Yes'],"\nNo:", series_online_order['No'])
print("Total Number of restaurants:",bengaluru_res_df.shape[0]," total Yes + No:",(series_online_order['Yes']+series_online_order['No']))Restaurants with online order options:
 Yes: 30444
No: 21273
Total Number of restaurants: 51717  total Yes + No: 51717
vs.bar_onlineorder(bengaluru_res_df)_____no_output_____

> **Interpretation:** out of 51,717 records 41.1% of restaurants do not have options to deliver using Zomato application_____no_output_____
### 2.2 Rating bar plot

#### 2.2.1 Ratings Between 1-5 with interval .1_____no_output_____In this section I have
> - Visualized the ratings, normal curve formded by the count of unique ratings using bar graph

> -  Visualized the color bins for the ratings of the restaurant using bar graph

The ratings are in the form of string '3.5 /5' containing spaces in some and have few missing values in form of NaNs

The Ratngs are in the form:
    - *'3.5/5'* -> rating out of 5
    - *'-'* -> no ratings given for this restaurant
    - *NaNs* -> missing ratings in the column_____no_output_____

vs.rating_curve(bengaluru_res_df)_____no_output_____

> **Interpretation:** The rating count is fitted inside a normal curve centered around 3.8 rating. .2% of ratings are not goven yet to the restauants and 5% of the restaurants are new on zomato. _____no_output_____### 2.2.2 Ratings Color bins

As discussed above, Zomato has 8 colored bins with dark green containing the highest top 6 ratings 4.5,4.6,4.7,4.8,4.9 and 5 and red containing the lowest rating 1,1.1,1.2,1.3,1.4... visualizing the ratios for each bin.
 There are two more bins marked as
- "NEW" for new restaurants on Zomato
- "-" restaurants not rated yet on Zomato
>**Note**: for visualization "New" and "-" bins are also considered

> **For this project, the model is trained to predict these colored bins**_____no_output_____

vs.color_bins(bengaluru_res_df)_____no_output_____

** Ratios of the Restaurants **
- 34.35% of the restaurants in Bengaluru have ratings between 3.5 - 3.9.
- 21.57% of the restaurants have ratings between 4.0 - 4.4.
- Only 2.41% of the restaurants have ratings between 4.5 - 5.
- Only 23.98% of the restaurant falls on the right of the curve_____no_output_____
### 2.3 Location and Location listed in city

#### 2.3.1 Listed in 30 Location

Zomato displays 30 popular localities in Bengaluru. The 30 results changes with the selected city.
 Top 30 localities in Bengaluru on 17th - jun- 2019 is

src: https://www.zomato.com/bangalore_____no_output_____The 30 locations are stored in "listed_in(city)" columns. the visualization will show the locations and the counts of restaurants present there.

>Note: The visualization may differ from the image above as the data considered is from another date._____no_output_____

vs.listed_in_city(bengaluru_res_df)_____no_output_____

#### 2.3.2 Locations of the restaurant i.e closest location/area from the restaurant
The 30 locations have restaurants within 10-12 kms. There are about 93 locations which gives restaurant results closer than the listed_in_city locations._____no_output_____

vs.location(bengaluru_res_df)Total Locations: 93
Locations: Index(['BTM', 'HSR', 'Koramangala 5th Block', 'JP Nagar', 'Whitefield',
       'Indiranagar', 'Jayanagar', 'Marathahalli', 'Bannerghatta Road',
       'Bellandur', 'Electronic City', 'Koramangala 1st Block', 'Brigade Road',
       'Koramangala 7th Block', 'Koramangala 6th Block', 'Sarjapur Road',
       'Ulsoor', 'Koramangala 4th Block', 'MG Road', 'Banashankari',
       'Kalyan Nagar', 'Richmond Road', 'Frazer Town', 'Malleshwaram',
       'Basavanagudi', 'Residency Road', 'Banaswadi', 'Brookefield',
       'New BEL Road', 'Kammanahalli', 'Rajajinagar', 'Church Street',
       'Lavelle Road', 'Shanti Nagar', 'Shivajinagar', 'Domlur',
       'Cunningham Road', 'Old Airport Road', 'Ejipura', 'Commercial Street',
       'St. Marks Road', 'Koramangala 8th Block', 'Vasanth Nagar',
       'Jeevan Bhima Nagar', 'Wilson Garden', 'Bommanahalli',
       'Koramangala 3rd Block', 'Kumaraswamy Layout', 'Thippasandra',
       'Basaveshwara Nagar', 'Nagawara', 'Seshadripuram', 'Hennur', 'Majestic',
       'HBR Layout', 'Infantry Road', 'Race Course Road', 'City Market',
       'Yeshwantpur', 'Varthur Main Road, Whitefield',
       'ITPL Main Road, Whitefield', 'South Bangalore',
       'Koramangala 2nd Block', 'Hosur Road', 'Kaggadasapura',
       'CV Raman Nagar', 'Vijay Nagar', 'RT Nagar', 'Sanjay Nagar',
       'Sadashiv Nagar', 'Sahakara Nagar', 'Koramangala', 'East Bangalore',
       'Jalahalli', 'Magadi Road', 'Rammurthy Nagar', 'Langford Town',
       'Sankey Road', 'Old Madras Road', 'Mysore Road', 'Kanakapura Road',
       'KR Puram', 'Uttarahalli', 'Hebbal', 'North Bangalore', 'Nagarbhavi',
       'Kengeri', 'Central Bangalore', 'West Bangalore', 'Yelahanka', 'Jakkur',
       'Rajarajeshwari Nagar', 'Peenya'],
      dtype='object')

 Top 50 Locations with restaurant Counts

### 2.4 restaurant type and listed in type_____no_output_____#### 2.4.1 Resraurant Listed In
For quick searhces Zomato has categorized all the Restaurant types into 7 main categories. The ratios of all the  51,717  restaurants and there types is shown below:

src: https://www.zomato.com/bangalore/indiranagar-restaurants_____no_output_____

vs.rest_type_listedin(bengaluru_res_df)_____no_output_____

#### 2.4.2 Restaurant Types as mentioned by the Restaurants
Restaurants can mention two types which are comma separated. From the code block below, the 3rd row, the types are Cafe and Casual Dining. this column has mixed nominal categorical values which are comma separated. _____no_output_____

bengaluru_res_df.loc[2]['rest_type']_____no_output_____

- '1' example 'Cafe'
- '2' example 'Cafe,casual'
- '0' examle 'nan'_____no_output_____

unique_rest = vs.rest_mixed_type(bengaluru_res_df)types counts: restaurant counts {1: 43763, 2: 7727, 3: 0, 0: 227}
print("Total Number ofUnique Restaurant types",len(unique_rest))
print("Unique Restaurant Types:")
unique_rest.keys()Total Number ofUnique Restaurant types 25
Unique Restaurant Types:
vs.unique_rest_types(bengaluru_res_df, unique_rest)_____no_output_____

>**Inference**
More than 20,000 restaurants are of quick bites type,

followed by casual dinining places, followed by cafe and delivery types
_____no_output_____
### 2.5 Cuisines_____no_output_____Cuisines is a mixed categorical column eg. 'Cafe, Mexican, Italian'. A restaurant can have upto 8 cuisines mentioned, which are comma separated.
Some restaurants have more than one cusines.
Number of cuisines: restaurant counts.  _____no_output_____

bengaluru_res_df.loc[2]['cuisines']_____no_output_____

The graph below shows the count of restaurants with 1 or more cuisines_____no_output_____

unique_cuisines = vs.unique_cuisines(bengaluru_res_df)_____no_output_____vs.unique_cuisines_top30(bengaluru_res_df,unique_cuisines)Total Number of Unique cuisines: 107

**Inference**
There are more than 20,000 restaurants which servies North Indian, which makes it the top served cusine in Bengaluru, followed by chinese and southIndian Food._____no_output_____

vs.unique_cuisines_lowest30(bengaluru_res_df,unique_cuisines)_____no_output_____

**Inference:** Malwani, Panasian, hotdogs,sindhi, australian and few more cusines are the least served cuisines with only 2-10 places serving them.  _____no_output_____**Summary**
- 12.5% of restaurants out of 51,717 have book table options  and  58.9% have order online option,
- Ratings are distributed on normal curve centered around 3.8-3.9.
- Approximately 24% of restaurants have ratings between 4-5. That is only 24% of them are on the right side of the normal curve
- There are 30 listed locations in Bengaluru for quick searches of restaurants, BTM containing the most number of Restaurants
- There are 7 listed restauranted types by Zomato for quick searches, DElivery being on top.
- There are 25 unique Restaurant types as listed by the restaurants themsevles. Some have mentioned two types. There are more number of Quickbites and casual dinings types of restaurants in Bengaluru.
- There are 107 unique cuisines served. Each restaurant serves minimum of 1 cuisine and maximum of 8. NorthIndian, chinease, southIndian and fastfood is the most popular. Malwani, Panasian, hotdogs,sindhi, australian and few more cusines are the least served cuisines with only 2-10 places serving them._____no_output_____
## 3. Data Pre-Processing_____no_output_____
### 3.1 Assessing columns
- 3.1.1 Renaming columns - columns have spaces and few have charcters (,), renaming these columns with lower underscore separated columns
- 3.1.2 Assessing Columns - replacing empty lists '[]' and nans
- 3.1.3 Droping unwated columns

#### 3.1.1 Renaming columns_____no_output_____Renaming columns because of bad column name
- approx_cost(for two people) to cost_for_two
- listed_in(type) to listed_in_type
- listed_in(city) to listed_in_city_____no_output_____

bengaluru_res_df = bengaluru_res_df.rename(columns={"approx_cost(for two people)": "cost_for_two", "listed_in(type)": "listed_in_type","listed_in(city)":"listed_in_city"})_____no_output_____

#### 3.1.2 Assessing colunmns_____no_output_____Converting menu_item empty list '[]' to NaN_____no_output_____

print(bengaluru_res_df["menu_item"].unique()[0])[]
bengaluru_res_df.replace(['[]'],np.nan,inplace=True)_____no_output_____vs.column_nan_ratios(bengaluru_res_df)_____no_output_____

**Inference**
- More than 75% of menu_items are missing
- More than 50% of Dish_liked is missing
- close to 15% of ratings are missing
- 15% of reviews are missing
- less tahn 5% of phone numbers, cost for two and restaurant types are missing_____no_output_____#### 3.1.3 Dropping columns _____no_output_____URL, Address, name
Columns - URL, Address and name needs to be dropped
 - The URL
 - Name - most of the names have their short address with their names. The number of words can not be used to analyse the sdata
 - Address - The locality and city in Bengalure is mentioned in another column.
 - Votes - The new restaurants will not have any prior vote counts
 - dish_liked - More than 50% of the data is missing, hence can not be used for nanylysis and machine learning
 - menu_item - More than 75% of data is not provided, This column can not be used for analysis or prediction
 - review_list - The new restautrant won't have a review list as it is a new restaurant on Zomato_____no_output_____

bengaluru_res_df.drop(columns=['url', 'address','name','votes','menu_item','dish_liked','reviews_list'],inplace = True, errors ='ignore')_____no_output_____bengaluru_res_df.head(2)_____no_output_____

Missing values after Dropping unwanted columns_____no_output_____

vs.column_nan_ratios(bengaluru_res_df)_____no_output_____

### 3.2 Assessing Rows
_____no_output_____- 3.2.1 Assessing target column
- 3.2.2 Assessing phone column¶
- 3.2.3 threshold for missing in rows

#### 3.2.1 Assessing target column_____no_output_____

(bengaluru_res_df['rate'].isnull().sum()/bengaluru_res_df['rate'].shape[0])*100_____no_output_____bengaluru_res_df[bengaluru_res_df['rate'] == 'NEW'].shape_____no_output_____

>15 percent of target values are missing, droping Rows with missing ratings as these rows cannot be used to train the model.

**2208 restaurants are listed as NEW on omato and is stored in "bengaluru_res_df_new"** for prediction after training the model._____no_output_____

bengaluru_res_df.dropna(subset=['rate'], inplace = True)_____no_output_____bengaluru_res_df.drop(bengaluru_res_df[bengaluru_res_df.rate == '-'].index, axis=0, inplace=True)_____no_output_____bengaluru_res_df_new = bengaluru_res_df[bengaluru_res_df['rate'] == 'NEW']
bengaluru_res_df.drop(bengaluru_res_df[bengaluru_res_df.rate == 'NEW'].index, axis=0, inplace=True)_____no_output_____vs.column_nan_ratios(bengaluru_res_df)_____no_output_____

The Rate column has zero missing values now_____no_output_____#### 3.2.2 Assessing phone column
More than 2% of percent of phone number is missing for this project I am **assuming the missing phone numbers as no phone numbers** provided and **replacing NaN values with 0**_____no_output_____

bengaluru_res_df['phone'].fillna(0, inplace = True)_____no_output_____vs.column_nan_ratios(bengaluru_res_df)_____no_output_____

#### 3.2.3 threshold for missing in rows

For this project I am dropping rows with more than 1 missing values_____no_output_____

threshold = 1
ls_threshold = bengaluru_res_df.loc[((bengaluru_res_df.isnull().sum(axis = 1)) <= threshold) & ((bengaluru_res_df.isnull().sum(axis = 1)) != 0) ]
#Rows count for NaNs count more than threshold
gr_threshold = bengaluru_res_df.loc[(bengaluru_res_df.isnull().sum(axis = 1)) > threshold]_____no_output_____print("Number of Restaurants before dropping rows having missing valuemore than 1:",bengaluru_res_df.shape[0])
print(ls_threshold.shape[0])
gr_threshold.shape[0]
Number of Restaurants before dropping rows having missing valuemore than 1: 41665
397
bengaluru_res_df.dropna(thresh = len(bengaluru_res_df.columns)-threshold,inplace = True)
print("Number of Restaurants after dropping rows having missing valuemore than 1:",bengaluru_res_df.shape[0])Number of Restaurants after dropping rows having missing valuemore than 1: 41660
vs.column_nan_ratios(bengaluru_res_df)_____no_output_____

### 3.3 Reencoding columns

In this section the columns are preprossed from string values to integer/binary values. All the columns are of object type and needs to be converted. the remaining 10 columns are:
-  non numerical binary categorical
- Numerical data in form of string
- nominal categorical data
- mixed nominal categorical data_____no_output_____

bengaluru_res_df.info()<class 'pandas.core.frame.DataFrame'>
Int64Index: 41660 entries, 0 to 51716
Data columns (total 10 columns):
online_order      41660 non-null object
book_table        41660 non-null object
rate              41660 non-null object
phone             41660 non-null object
location          41660 non-null object
rest_type         41513 non-null object
cuisines          41652 non-null object
cost_for_two      41418 non-null object
listed_in_type    41660 non-null object
listed_in_city    41660 non-null object
dtypes: object(10)
memory usage: 3.5+ MB

**About Columns:**
1. Non Numerical Binary columns,
    - `online_order` - yes or no for oline order booking
    - `book_table` - yes or no for table booking

2. Numerical,float
    - `Rate` - 1,1.1,1.2,....,4.9,5
    - `phone`
        - 0 for no phone numbers
        - 1 for one phonr number provided
        - 2 for two phone number provided
    - `cost_for_two` - Approx cost for two in the restaurants

> **Note:** For this project I am counting the number of phone numbers provided on Zomato

3. Multi level categorical columns
    - `location`  - nearest location of the restaurant in the city eg. Marathahalli, Shivajinagar - 93 unique names
    - `listed_in_city` - part of the city in Bengaluru eg Church Street, BTM - 30 unique names
    - `listed_in_type` - Type of the restaurant on zomato - 7 unqiue types

4. Mixed categorical columns
    - `rest_type` - restaurant type as mentioned by the restaurant - few have two types
    - `cuisines` - cuisines type as menstioned by the restaurant - restaurant may have cuisines type between 1-8.

_____no_output_____#### 3.3.1 Non Numerical Binary column to dummy variables

Created dummy variables for both the columns which gave 4 columns with binary values. Few restaurants types don’t have tables example few quick bites, cafes don’t really need booking tables hence did not binaries the columns to 1 for yes and 0 for No. _____no_output_____

print("Columns count:",len(bengaluru_res_df.columns))Columns count: 10
#yes_no_1_0 = {'Yes':1,'No':0}
#bengaluru_res_df['online_order'] = bengaluru_res_df.apply(lambda x: yes_no_1_0[x.online_order], axis=1)
bengaluru_res_df = pd.get_dummies(bengaluru_res_df, columns=['online_order'])_____no_output_____#bengaluru_res_df['book_table'] = bengaluru_res_df.apply(lambda x: yes_no_1_0[x.book_table], axis=1)
bengaluru_res_df = pd.get_dummies(bengaluru_res_df, columns=['book_table'])_____no_output_____#converting the new columns to lower case
bengaluru_res_df.columns = [x.lower() for x in bengaluru_res_df.columns]_____no_output_____#Displaying columns after a dummy columns were created
bengaluru_res_df.columns_____no_output_____print("Columns count:",len(bengaluru_res_df.columns))Columns count: 12

#### 3.3.2 Numerical,float

- Ratings are given in the form of a string “1.5 /5”. A function is defined in helper.py which removes spaces separates rates where“/”  and returns 1.5 in form of float.
- Phones are in the form of str “+91 9865745477\n\r 080 67577578” which are either 8 digit for landline or 10 digits for mobile.
- Cost for two is given in string with commas “1,700”. A function on the column removes comma and returns the integer value example “1,700” to 1700. returns numpy nan for misisng value._____no_output_____

bengaluru_res_df.loc[2]['rate']_____no_output_____#function replaces any space
#returns float value separated by "/" and np.nan for missing value
bengaluru_res_df['rate'] = bengaluru_res_df.apply(lambda x: rates(x.rate), axis=1)_____no_output_____bengaluru_res_df.loc[2]['phone']_____no_output_____#Function checks 8 continuos integers and increases the phone count by 1
#returns number of phone numbers and 0 for missing value
bengaluru_res_df['phone'] = bengaluru_res_df.apply(lambda x: phone_num_count(x.phone), axis=1)_____no_output_____bengaluru_res_df.loc[2]['cost_for_two']_____no_output_____#function str_to_float replaces "," and " "-space and returns the float value of the cost.
# returns np.nan for missing value
bengaluru_res_df['cost_for_two'] = bengaluru_res_df.apply(lambda x: str_to_float(x.cost_for_two), axis=1)_____no_output_____

_____no_output__________no_output_____

bengaluru_res_df.describe()_____no_output_____bengaluru_res_df['cost_for_two'].median()_____no_output_____bengaluru_res_df['cost_for_two'].mean()_____no_output_____bengaluru_res_df.corr()_____no_output_____

**Inference:**

- small correlation can be seen between rates and cost_for_two which was expected
- a positive correlation can be seen between booking table and rates._____no_output_____

bengaluru_res_df.columns_____no_output_____

#### 3.3.3 Multi level categorical columns_____no_output_____#### 1. methods to deal with multi level categorical values
https://www.analyticsvidhya.com/blog/2015/11/easy-methods-deal-categorical-variables-predictive-modeling/

The location columns is nominal as the name there are total 93 distinct location for Restaurants in Bengaluru
- Creating dummy values
- creating new levels based on frequency
- Converting to numeric values - label encoding
- Hashing
there are other complicated encoders such as Hermet, sum, backward, polynomial encoders
>For my project **I used creating dummy variables** because it saves all the information in form of binary input. Giving labels to more than 30 values will takeout all the information and it will be difficult to process the data for prediction._____no_output_____**i) listed_in_type column**
Types of restaurants listed by Zomato for quick searches - 7 unqiue types.
- The types are converted into lower case with space replaced by "_" undercolumn
- dummies are created for these 7 types._____no_output_____

listed_rest_types = bengaluru_res_df['listed_in_type'].unique()
listed_rest_types_____no_output_____print("Columns count before dummies:",len(bengaluru_res_df.columns))Columns count before dummies: 12
#replacing space with "_" and string to lower case
bengaluru_res_df['listed_in_type'] = bengaluru_res_df.apply(lambda x: x.listed_in_type.replace(" ","_").lower(), axis = 1)_____no_output_____#Getting dummies for listed_in_type
bengaluru_res_df = pd.get_dummies(bengaluru_res_df, columns=['listed_in_type'])_____no_output_____print("Columns count after dummies:",len(bengaluru_res_df.columns))Columns count after dummies: 18
bengaluru_res_df.columns_____no_output_____

**ii) listed_in_city column**
these are the city location in Bengaluru that Zomato has listed all its restaurants:
- There are 30 locations
- The types are converted into lower case with space replaced by "_" undercolumn_____no_output_____

#printiing unique 30 locations listed by zomato
listed_rest_city_loc = bengaluru_res_df['listed_in_city'].unique()
listed_rest_city_loc_____no_output_____bengaluru_res_df['listed_in_city'] = bengaluru_res_df.apply(lambda x: x.listed_in_city.replace(" ","_").lower(), axis = 1)_____no_output_____print("Columns count before dummies:",len(bengaluru_res_df.columns))Columns count before dummies: 18
bengaluru_res_df = pd.get_dummies(bengaluru_res_df, columns=['listed_in_city'])_____no_output_____print("Columns count after dummies:",len(bengaluru_res_df.columns))Columns count after dummies: 47
bengaluru_res_df.columns_____no_output_____

**iii) location column**

there are 93 unique locations, for this project I am considering "listed_in_city" for location and droping "location" column_____no_output_____

bengaluru_res_df['location'].unique()_____no_output_____#bengaluru_res_df['location'] = le.transform(bengaluru_res_df['location'])
bengaluru_res_df.drop(columns = ['location'],inplace = True, axis = 1) _____no_output_____len(bengaluru_res_df.columns)_____no_output_____

#### 3.3.4 mixed Multi level categorical columns
- rest_type - restaurant type as mentioned by the restaurant - few have two types, **converting rest_type** the
creating two columns for rest_type which separates the two restaurant types **in rest_type1 and rest_type2**. the value is converted to lower case and the space is removed. example

| rest_type   |
|------------|
|   Cafe,Casual Dining|

| rest_type1 | rest_type2   |
|------|------|
|   cafe  | casualdining|

- cuisines - cuisines type as menstioned by the restaurant - restaurant may have cuisines type between 1-8. **converting cuisines** to two columns **cuisines_type_1 and cuisines_type_2** which contains first two cuisines in two columns.the value is converted to lower case and the space is removed. example

| cuisines   |
|------------|
|   North Indian, Fast Food, South Indian|

| cuisines_type1 | cuisines_type2   |
|------|------|
|   northindian  | fastfood|_____no_output_____**i) rest_type**_____no_output_____

#before processing
bengaluru_res_df['rest_type'][2]_____no_output_____#converting to lower case and replacing space with no space
bengaluru_res_df['rest_type'] = bengaluru_res_df.apply(lambda x: lower_(x.rest_type), axis=1)_____no_output_____#after processin
bengaluru_res_df['rest_type'][2]_____no_output_____bengaluru_res_df['rest_type1'] = bengaluru_res_df.apply(lambda x: type_x(x.rest_type,1), axis=1)
bengaluru_res_df['rest_type2'] = bengaluru_res_df.apply(lambda x: type_x(x.rest_type,2), axis=1)_____no_output_____print(len(bengaluru_res_df['rest_type1'].unique()))
bengaluru_res_df['rest_type1'].unique()23
print(len(bengaluru_res_df['rest_type2'].unique()))
bengaluru_res_df['rest_type2'].unique()18

**Inference**

There are 25 unique restaurants listed by them, it was seen that the first column created  **rest_type1** has only 23 unique types out of 25 and ** rest_type2** has only 18 unique types out of 25.
> **Note:**The dummies cannot be created using pd.dummies, **the below code creates dummy columns for all the unique rest type  (for rest in unique_rest) for the columns rest_type1,rest_type2**

 **unique_rest** - contain unique restaurants types created during exploration of rest_type_____no_output_____print("Restaurant type unique mentioned in rest_type:" len(unique_rest.keys()))
vs.unique_rest_types(bengaluru_res_df, unique_rest)_____no_output_____

print("columns before creating dummies:")
len(bengaluru_res_df.columns)columns before creating dummies:
#code which creates dummy variable for both the column rest_type1 and rest_type2
for rest in unique_rest:
    column_name1 = 'rest_type1_'+ rest
    column_name2 = 'rest_type2_' + rest
    bengaluru_res_df[column_name1] = bengaluru_res_df.apply(lambda x: 1 if x.rest_type1 == rest else 0, axis=1)
    bengaluru_res_df[column_name2] = bengaluru_res_df.apply(lambda x: 1 if x.rest_type2 == rest else 0, axis=1)
    #print(column_name)_____no_output_____#dropping columns after creating columns
bengaluru_res_df.drop(columns =['rest_type1','rest_type2','rest_type'],inplace = True)_____no_output_____print("columns after creating dummies:")
len(bengaluru_res_df.columns)columns after creating dummies:
#https://stackoverflow.com/questions/36808434/label-encoder-encoding-missing-values
#bengaluru_res_df['rest_type1'] = bengaluru_res_df.apply(lambda x: le.transform([x.rest_type1])[0] if type(x.rest_type1) == str else x.rest_type1,axis = 1)
#bengaluru_res_df['rest_type2'] = bengaluru_res_df.apply(lambda x: le.transform([x.rest_type2])[0] if type(x.rest_type2) == str else x.rest_type2,axis = 1)_____no_output_____

**ii). cuisines**_____no_output_____

bengaluru_res_df['cuisines'] = bengaluru_res_df.apply(lambda x: lower_(x.cuisines), axis=1)_____no_output_____bengaluru_res_df['cuisines'].head()_____no_output_____bengaluru_res_df['cuisines_1'] = bengaluru_res_df.apply(lambda x: type_x(x.cuisines,1), axis=1)
bengaluru_res_df['cuisines_2'] = bengaluru_res_df.apply(lambda x: type_x(x.cuisines,2), axis=1)_____no_output_____

> Creating dummy variable using for loop, because columns cuisines_1, cuisines_2 may not contain all unique cuisine type

**unique_cuisines** - contains unique cuisines created during exploration_____no_output_____

print("columns before creating dummies:")
len(bengaluru_res_df.columns)columns before creating dummies:
for cuisine in unique_cuisines:
    cuisine = cuisine.lower()
    column_name1 = 'cuisines_1_'+ cuisine
    column_name2 = 'cuisines_2_' + cuisine
    bengaluru_res_df[column_name1] = bengaluru_res_df.apply(lambda x: 1 if x.cuisines_1 == cuisine else 0, axis=1)
    bengaluru_res_df[column_name2] = bengaluru_res_df.apply(lambda x: 1 if x.cuisines_2 == cuisine else 0, axis=1)_____no_output_____bengaluru_res_df.drop(columns =['cuisines_1','cuisines_2','cuisines'],inplace = True)_____no_output_____print("columns after creating dummies:")
len(bengaluru_res_df.columns)columns after creating dummies:
bengaluru_res_df.head()_____no_output_____

**Dataframe after processing all the data**:_____no_output_____

bengaluru_res_df.shape_____no_output_____

- The processed dataframe contains _____no_output_____### 3.3.5 Processing Target variable

the target bin is labelled here for training the model.

as the diagram suggests the ratings will be labelled as which bin it beongs to for the model to train.
After training the model the model will predit which label/bin the NEW restaurant belongs to.
- ratings between 1-1.4 - labelled as 0
- ratings between 1.5-1.9 - labelled as 1
- ratings between 2-2.4 - labelled as 2
- ratings between 2.5-2.9.4 - labelled as 3
- ratings between 3-3.4 - labelled as 4
- ratings between 3.5-3.9 - labelled as 5
- ratings between 4-4.4 - labelled as 6
- ratings between 4.4-5 - labelled as 7
>**A new columns rate_color is created to store the labels**.
Creating **target** variables and **feature** columns_____no_output_____

bengaluru_res_df['rate'].isnull().sum()_____no_output_____bengaluru_res_df['rate_color'] = bengaluru_res_df.apply(lambda x: target_rates_to_color(x.rate),axis= 1)_____no_output_____bengaluru_res_df['rate_color'].unique()_____no_output_____#rate_to_label = {}
#label = 0
#for i in np.linspace(1,5,41):
    #print(round(i,1))
    #rate_to_label[np.round(i,1)] = label
    #label += 1

#bengaluru_res_df['rate'] = bengaluru_res_df.apply(lambda x: rate_to_label[x.rate],axis= 1)_____no_output_____#bengaluru_res_df['rate'].unique()_____no_output_____bengaluru_res_df['rate_color'].head()_____no_output_____#bengaluru_res_df['rate'].head()_____no_output_____

>Creating target variables and feature columns_____no_output_____

target = bengaluru_res_df[['rate','rate_color']]
features = bengaluru_res_df.drop(columns = ['rate','rate_color'],axis = 1)_____no_output_____

### 3.3.4 imputing and Feature Scaling_____no_output_____Only cost_for_two has 242 missing values out of 41,660. The missing value will be replaced by median. Median is less than the mean. all the other columns are converted to binary hence will have 0 for missing or 1 for present._____no_output_____

print(bengaluru_res_df.shape)
bengaluru_res_df['cost_for_two'].isnull().sum()(41660, 309)
bengaluru_res_df['cost_for_two'].median()_____no_output_____bengaluru_res_df['cost_for_two'].mean()_____no_output_____imp_median = SimpleImputer(missing_values=np.NaN, strategy='median')_____no_output_____

bengaluru_res_df_imp has no missing values and can be used for training_____no_output_____

bengaluru_res_df_imp = pd.DataFrame(imp_median.fit_transform(features))_____no_output_____bengaluru_res_df_imp.isnull().sum().sum()_____no_output_____bengaluru_res_df_imp.columns = features.columns_____no_output_____bengaluru_res_df_imp.head(2)_____no_output_____

For this project most of the columns are binary, to preserve there value, I am going to using MinMax Scaller

$$x_new = \frac{x - x_min}{x_max - x_min} $$

$$x_max = maximum \ value \ in \ space \ the \ space \ column$$
$$x_min = minimum \ value \ in \ space \ the \ space \ column$$_____no_output_____

scaler = MinMaxScaler()
bengaluru_res_df_sca = pd.DataFrame(scaler.fit_transform(bengaluru_res_df_imp))
bengaluru_res_df_sca.columns = bengaluru_res_df_imp.columns
bengaluru_res_df_sca.head(2)_____no_output_____

### 3.5 Training and Test _____no_output_____>There are 41660 restaurants for training 95% of the data is used. Since there are alot of rows only **5% of the data is reserved for testing** the benchmarck model and optimized Neural network model_____no_output_____

X_train, X_test, y_train, y_test = train_test_split(bengaluru_res_df_sca, target,test_size=0.05, random_state=42)_____no_output_____

## 4. Training and testing Bench mark model_____no_output_____from sklearn linear model LinearRegression is used for training the becch mark model. Linear model is underfitting because it is assuming the problem is linear and that the prediction follows a linear equation.
The Aim of this is to form a threshold for the final optimised model which should work better than the linear model.
$$\bar y = Wx + b $$

$$ W = [W_1, W_2, W_3,...,W_n] $$
$$ x = [x_1, x_2, x_3,...,x_n] $$

**Accuracy score:** is the metric used which calculates the catergory correctly predicted by the model

$$ Accuracy = \frac{ Correctly \ predicted \ Class}{Total \ Number \ of \ Predictions}  $$_____no_output_____

LR = LinearRegression()_____no_output_____LR.fit(X_train, y_train['rate_color'])_____no_output_____y_hat = np.round(LR.predict(X_test))_____no_output_____y_test['rate_color'].shape_____no_output_____y_hat.shape_____no_output_____score = metrics.accuracy_score(y_test['rate_color'], y_hat)_____no_output_____score_____no_output_____y_hat_____no_output_____

> **Accuracy Score of the Linear model is 54.78%**_____no_output_____
## 5. NN Model

**The Model is a Neural network with 307 inputs and 3 hidden layers and 8 categories/bins: **

- The neural Network Architechture is created using torch nn module, which has 8 output. Log_softmax is performed to get the label.
- nn.dropout(.02) is used so that the network does not over fit.
- nn.NLLLoss is used to calcuate the loss. since the output is a softmax which is in the form of exponential, Natural Log loss will be best to get the loss
- loss.backward() performs the backward pass
- optimizer.step() updates the new weight
- optimizer.zero_grad() sets the grdient to zero for next backword propagation
- model.eval() freezes the gradient and removes the dropout for evaluation/validation during training

**Minibatch:** - defined in train.py

src:https://stackoverflow.com/questions/38157972/how-to-implement-mini-batch-gradient-descent-in-python

src:https://stackoverflow.com/questions/50307707/convert-pandas-dataframe-to-pytorch-tensor

- there are 39,577 rows for each epoch 4 mini batches are randomly created out of which 3 mini batches are used for training and 1 mini batch is used for validation

**Parameter tuning for the model:**
-	Epochs
-	Learning rate
-	Mini Batch size
-	Number of Hidden layers
-	Number of Hidden nodes

**NN Architecture**

The model is created and trained using **train.py**_____no_output_____

model = tr.create_train_model(X_train,y_train)Epoch: 2/450..  Training Loss: 1.422..  validation Loss: 1.379..  validation Accuracy: 0.515
Epoch: 7/450..  Training Loss: 1.135..  validation Loss: 1.106..  validation Accuracy: 0.543
Epoch: 12/450..  Training Loss: 1.064..  validation Loss: 1.047..  validation Accuracy: 0.556
Epoch: 17/450..  Training Loss: 0.996..  validation Loss: 0.997..  validation Accuracy: 0.575
Epoch: 22/450..  Training Loss: 0.929..  validation Loss: 0.903..  validation Accuracy: 0.617
Epoch: 27/450..  Training Loss: 0.865..  validation Loss: 0.848..  validation Accuracy: 0.647
Epoch: 32/450..  Training Loss: 0.817..  validation Loss: 0.770..  validation Accuracy: 0.683
Epoch: 37/450..  Training Loss: 0.780..  validation Loss: 0.725..  validation Accuracy: 0.704
Epoch: 42/450..  Training Loss: 0.740..  validation Loss: 0.687..  validation Accuracy: 0.719
Epoch: 47/450..  Training Loss: 0.710..  validation Loss: 0.656..  validation Accuracy: 0.733
Epoch: 52/450..  Training Loss: 0.682..  validation Loss: 0.639..  validation Accuracy: 0.743
Epoch: 57/450..  Training Loss: 0.667..  validation Loss: 0.594..  validation Accuracy: 0.759
Epoch: 62/450..  Training Loss: 0.648..  validation Loss: 0.571..  validation Accuracy: 0.766
Epoch: 67/450..  Training Loss: 0.629..  validation Loss: 0.565..  validation Accuracy: 0.770
Epoch: 72/450..  Training Loss: 0.613..  validation Loss: 0.541..  validation Accuracy: 0.780
Epoch: 77/450..  Training Loss: 0.604..  validation Loss: 0.513..  validation Accuracy: 0.792
Epoch: 82/450..  Training Loss: 0.588..  validation Loss: 0.504..  validation Accuracy: 0.792
Epoch: 87/450..  Training Loss: 0.579..  validation Loss: 0.496..  validation Accuracy: 0.801
Epoch: 92/450..  Training Loss: 0.568..  validation Loss: 0.488..  validation Accuracy: 0.798
Epoch: 97/450..  Training Loss: 0.557..  validation Loss: 0.479..  validation Accuracy: 0.805
Epoch: 102/450..  Training Loss: 0.549..  validation Loss: 0.461..  validation Accuracy: 0.806
Epoch: 107/450..  Training Loss: 0.536..  validation Loss: 0.467..  validation Accuracy: 0.809
Epoch: 112/450..  Training Loss: 0.527..  validation Loss: 0.455..  validation Accuracy: 0.809
Epoch: 117/450..  Training Loss: 0.527..  validation Loss: 0.447..  validation Accuracy: 0.815
Epoch: 122/450..  Training Loss: 0.516..  validation Loss: 0.447..  validation Accuracy: 0.811
Epoch: 127/450..  Training Loss: 0.514..  validation Loss: 0.437..  validation Accuracy: 0.813
Epoch: 132/450..  Training Loss: 0.506..  validation Loss: 0.442..  validation Accuracy: 0.810
Epoch: 137/450..  Training Loss: 0.496..  validation Loss: 0.429..  validation Accuracy: 0.814
Epoch: 142/450..  Training Loss: 0.494..  validation Loss: 0.418..  validation Accuracy: 0.826
Epoch: 147/450..  Training Loss: 0.491..  validation Loss: 0.419..  validation Accuracy: 0.825
Epoch: 152/450..  Training Loss: 0.487..  validation Loss: 0.403..  validation Accuracy: 0.832
Epoch: 157/450..  Training Loss: 0.479..  validation Loss: 0.402..  validation Accuracy: 0.826
Epoch: 162/450..  Training Loss: 0.477..  validation Loss: 0.392..  validation Accuracy: 0.833
Epoch: 167/450..  Training Loss: 0.472..  validation Loss: 0.404..  validation Accuracy: 0.828
Epoch: 172/450..  Training Loss: 0.476..  validation Loss: 0.382..  validation Accuracy: 0.839
Epoch: 177/450..  Training Loss: 0.466..  validation Loss: 0.386..  validation Accuracy: 0.840
Epoch: 182/450..  Training Loss: 0.467..  validation Loss: 0.386..  validation Accuracy: 0.834
Epoch: 187/450..  Training Loss: 0.461..  validation Loss: 0.378..  validation Accuracy: 0.840
Epoch: 192/450..  Training Loss: 0.461..  validation Loss: 0.375..  validation Accuracy: 0.843
Epoch: 197/450..  Training Loss: 0.452..  validation Loss: 0.378..  validation Accuracy: 0.840
Epoch: 202/450..  Training Loss: 0.456..  validation Loss: 0.365..  validation Accuracy: 0.843
Epoch: 207/450..  Training Loss: 0.450..  validation Loss: 0.372..  validation Accuracy: 0.845
Epoch: 212/450..  Training Loss: 0.437..  validation Loss: 0.375..  validation Accuracy: 0.840
Epoch: 217/450..  Training Loss: 0.440..  validation Loss: 0.364..  validation Accuracy: 0.843
Epoch: 222/450..  Training Loss: 0.442..  validation Loss: 0.362..  validation Accuracy: 0.845
Epoch: 227/450..  Training Loss: 0.439..  validation Loss: 0.355..  validation Accuracy: 0.850
Epoch: 232/450..  Training Loss: 0.432..  validation Loss: 0.347..  validation Accuracy: 0.850
Epoch: 237/450..  Training Loss: 0.428..  validation Loss: 0.349..  validation Accuracy: 0.849
Epoch: 242/450..  Training Loss: 0.429..  validation Loss: 0.343..  validation Accuracy: 0.855
Epoch: 247/450..  Training Loss: 0.426..  validation Loss: 0.348..  validation Accuracy: 0.855
Epoch: 252/450..  Training Loss: 0.418..  validation Loss: 0.353..  validation Accuracy: 0.851
Epoch: 257/450..  Training Loss: 0.418..  validation Loss: 0.344..  validation Accuracy: 0.856
Epoch: 262/450..  Training Loss: 0.417..  validation Loss: 0.338..  validation Accuracy: 0.856
Epoch: 267/450..  Training Loss: 0.420..  validation Loss: 0.331..  validation Accuracy: 0.857
Epoch: 272/450..  Training Loss: 0.411..  validation Loss: 0.338..  validation Accuracy: 0.856
Epoch: 277/450..  Training Loss: 0.410..  validation Loss: 0.328..  validation Accuracy: 0.861
Epoch: 282/450..  Training Loss: 0.408..  validation Loss: 0.331..  validation Accuracy: 0.854
Epoch: 287/450..  Training Loss: 0.404..  validation Loss: 0.323..  validation Accuracy: 0.866
Epoch: 292/450..  Training Loss: 0.401..  validation Loss: 0.337..  validation Accuracy: 0.852
Epoch: 297/450..  Training Loss: 0.407..  validation Loss: 0.321..  validation Accuracy: 0.861
Epoch: 302/450..  Training Loss: 0.396..  validation Loss: 0.324..  validation Accuracy: 0.860
Epoch: 307/450..  Training Loss: 0.397..  validation Loss: 0.324..  validation Accuracy: 0.861
Epoch: 312/450..  Training Loss: 0.399..  validation Loss: 0.314..  validation Accuracy: 0.867
Epoch: 317/450..  Training Loss: 0.394..  validation Loss: 0.312..  validation Accuracy: 0.864
Epoch: 322/450..  Training Loss: 0.391..  validation Loss: 0.315..  validation Accuracy: 0.862
Epoch: 327/450..  Training Loss: 0.391..  validation Loss: 0.310..  validation Accuracy: 0.868
Epoch: 332/450..  Training Loss: 0.387..  validation Loss: 0.305..  validation Accuracy: 0.871
Epoch: 337/450..  Training Loss: 0.384..  validation Loss: 0.306..  validation Accuracy: 0.868
Epoch: 342/450..  Training Loss: 0.381..  validation Loss: 0.310..  validation Accuracy: 0.867
Epoch: 347/450..  Training Loss: 0.382..  validation Loss: 0.307..  validation Accuracy: 0.868
Epoch: 352/450..  Training Loss: 0.381..  validation Loss: 0.306..  validation Accuracy: 0.870
Epoch: 357/450..  Training Loss: 0.379..  validation Loss: 0.295..  validation Accuracy: 0.871
Epoch: 362/450..  Training Loss: 0.376..  validation Loss: 0.299..  validation Accuracy: 0.873
Epoch: 367/450..  Training Loss: 0.373..  validation Loss: 0.302..  validation Accuracy: 0.871
Epoch: 372/450..  Training Loss: 0.376..  validation Loss: 0.287..  validation Accuracy: 0.878
Epoch: 377/450..  Training Loss: 0.374..  validation Loss: 0.294..  validation Accuracy: 0.870
Epoch: 382/450..  Training Loss: 0.375..  validation Loss: 0.288..  validation Accuracy: 0.873
Epoch: 387/450..  Training Loss: 0.372..  validation Loss: 0.294..  validation Accuracy: 0.872
Epoch: 392/450..  Training Loss: 0.363..  validation Loss: 0.298..  validation Accuracy: 0.870
Epoch: 397/450..  Training Loss: 0.364..  validation Loss: 0.289..  validation Accuracy: 0.874
Epoch: 402/450..  Training Loss: 0.365..  validation Loss: 0.278..  validation Accuracy: 0.883
Epoch: 407/450..  Training Loss: 0.364..  validation Loss: 0.284..  validation Accuracy: 0.877
Epoch: 412/450..  Training Loss: 0.364..  validation Loss: 0.288..  validation Accuracy: 0.878
Epoch: 417/450..  Training Loss: 0.360..  validation Loss: 0.285..  validation Accuracy: 0.876
Epoch: 422/450..  Training Loss: 0.357..  validation Loss: 0.286..  validation Accuracy: 0.877
Epoch: 427/450..  Training Loss: 0.357..  validation Loss: 0.281..  validation Accuracy: 0.881
Epoch: 432/450..  Training Loss: 0.354..  validation Loss: 0.281..  validation Accuracy: 0.878
Epoch: 437/450..  Training Loss: 0.353..  validation Loss: 0.284..  validation Accuracy: 0.875
Epoch: 442/450..  Training Loss: 0.359..  validation Loss: 0.274..  validation Accuracy: 0.880
Epoch: 447/450..  Training Loss: 0.354..  validation Loss: 0.274..  validation Accuracy: 0.882

### Testing accuracy of the optimized model on test set_____no_output_____

#converting the X_test into tensor type
X_test_t = torch.tensor(X_test.values).type(torch.float)_____no_output_____#Predicting for X_test
with torch.no_grad():
    model.eval()
    prediction = model(X_test_t)_____no_output_____ps = torch.exp(prediction)
ps.shape_____no_output_____top_p, top_class = ps.topk(1, dim=1)_____no_output_____#converting tensor type to Series
top_class = pd.Series(top_class.numpy().ravel())_____no_output_____# calculating the score
score = metrics.accuracy_score(y_test['rate_color'], top_class)_____no_output_____score_____no_output_____

The accuracy score for the optimzed model is 76.33% _____no_output_____
## 6. Predicitng Color Bin for New Restaurant

- **bengaluru_res_df_new dataframe containing information for the New restaurants** needs to be preocessed similarly as bengaluru_res_df to get 307 columns.
- The total number of rows is for predicting is 2208.
- the **min max scaler** is perfomed manually by passing the minimum and maximum of phone and cost_for_two in a list.phone_min_max = [0,2]  cost_min_max = [40,6000]
- **pr.process()**  - perform preprocessing on the New restaurant data to get the 307 columns in the same order as order used for trainning
- **pr.predict()** - Performs prediction and returns top 3 classes in with probabilty showed in form of bar graph_____no_output_____

#Printing the first two rows of the dataframe
bengaluru_res_df_new.head(2)_____no_output_____#printing rows and columns
bengaluru_res_df_new.shape_____no_output_____

describe_____no_output_____

#putting the max and min of phone and cost_for_two to scale the
#min max filled using the describe()
phone_min_max = [0,2]
cost_min_max = [40,6000]_____no_output_____bengaluru_res_df_new_p = bengaluru_res_df_new.copy()_____no_output_____X_predict = pr.process(bengaluru_res_df_new_p,listed_rest_types,listed_rest_city_loc,unique_rest,unique_cuisines,phone_min_max,cost_min_max)_____no_output_____

>** Prints if all the columns names and orders are same after processing the dataframe containng the NEW data set**

**Checking if the columns of for X_test and X_predict have the same number of columns which are equally ordered and have the same name**
 prints True if matches the criteria else false_____no_output_____

#check if the X_predict is processed correctly
print((X_test.columns.values == X_predict.columns.values).sum() == 307)True

#### The index_ is used to do the prediction for index n in bengaluru_res_df_new dataframe containing the restaurant detail for the NEW one.

From below we can see the selected restaurant at index_ and its processed form in the next code line

### Please change the index_ below to see results for other restaurants_____no_output_____

#Predict bin for index
index_ = 2110
bengaluru_res_df_new[index_:index_+1]_____no_output_____X_predict[index_:index_+1]_____no_output_____

VErifying signals which will be used to predict the bin_____no_output_____

X_predict_t = torch.tensor(X_predict[index_:index_+1].values).type(torch.float)_____no_output_____X_predict_t.shape_____no_output_____

### Predicting for selected restaurant index_____no_output_____

print("The Top 3 Classes for the restaurant below with probabilities are:")
display(bengaluru_res_df_new[index_:index_+1])
pr.predict(model,X_predict_t)The Top 3 Classes for the restaurant below with probabilities are:

## 7. Limitation and  Conclusion_____no_output_____**Limitaions:**

- The model can predict scores only for the REstaurants provided in csv and have rate value as "NEW"
- Assuming that missing numbers as count 0 i.e no phone number provided for restaurant
- 346 restaurants hav  cost for two missing which is replaced by median
- The model is only limited to predict Bengaluru Data
- Due to computation time only two columns were prepared from cuisines to store first two in "cuisine_type1" and cuisine_type2" respectively. There are restaurants with cuisines types upto 8.

**conclusion**
- The model is able to predict with 76% of accuracy  the colored bin the New restaurnt will fall into
- only 12.7% of restaurants have book table options in Bengaluru
- Only 23% of restaurants have ratings between 4-5.

**Improvement**
- User interface can be made which takes inputs for a new restuaunts
- Neural network can undergo pruning to remove unwanted excess nodes
- creating more columns for cuisine types to separate the comma separated cuisines
- XGBoost and other algorithms can be used to check if it performs better than NN

_____no_output_____


# Notebook from Galadirith/positive
Path: docs/mvfits_paper/issues/issue3_gmvr_toy_problem.ipynb

# Toy Problem for GMVR_____no_output_____

# Setup ipython environment
%load_ext autoreload
%autoreload 2
%matplotlib inline
from numpy import *
import numpy as np
from positive import *
from matplotlib.pyplot import *(positive)>> Applying custom matplotlib settings.

### 2D Example with noise _____no_output_____

################################################################################
h = 3
Q = 25
x = h*linspace(-1,1,Q)
y = h*linspace(-1,1,Q)
X,Y = meshgrid(x,y)

# Optionally add noise to domain
# X += np.random.random( X.shape )-0.5
# Y += np.random.random( X.shape )-0.5

zfun = lambda xx,yy: 50 + (1.0 + (xx+0.5)*(yy+0.2) - 0.01*xx*xx) / ( 0.9 + xx**2 + yy**2 )
numerator_symbols, denominator_symbols = ['01','0','1'], ['00','11']

# zfun = lambda xx,yy: 50 + (1.0 + 2*xx*yy - (xx+0.1)*(yy+0.2))
# numerator_symbols, denominator_symbols = ['01','0','1'], ['K']

np.random.seed(42)
ns = 0.1*(np.random.random( X.shape )-0.5)

# print mean(ns)
# ns = ns-mean(ns)

Z = zfun(X,Y) + ns
Z_validation = []
for k in range(500):
    ns = 0.1*(np.random.random( X.shape )-0.5)
    Z_validation.append( zfun(X,Y) + ns )

domain,scalar_range = ndflatten( [X,Y], Z )
################################################################################_____no_output_____

### Use Greedy Algorithm -- GMVR_____no_output_____

star = gmvrfit( domain, scalar_range, verbose=True, tol=0.0001 )(gmvrfit)>> Now working deg = 1
&& The estimator has changed by -inf
&& Degree tempering will continue.
&& The current boundary is [('1', True)]
&& The current estimator value is 0.961341

(gmvrfit)>> Now working deg = 2
&& The estimator has changed by -0.888200
&& Degree tempering will continue.
&& The current boundary is [('01', True), ('00', True), ('11', True), ('00', False), ('11', False), ('1', True), ('0', True)]
&& The current estimator value is 0.073141

(gmvrfit)>> Now working deg = 3
&& The estimator has changed by 0.000000
&& Degree tempering will continue.
&& The current boundary is [('01', True), ('00', True), ('11', True), ('00', False), ('11', False), ('1', True), ('0', True)]
&& The current estimator value is 0.073141

(gmvrfit)>> Now working deg = 4
&& The estimator has changed by -0.000404
&& Degree tempering has completed becuase the estimator has changed by |-0.000404| < 0.010000. The results of the last iteration will be kept.
&& The Final boundary is [('01', True), ('00', True), ('11', True), ('00', False), ('11', False), ('1', True), ('0', True)]
&& The Final estimator value is 0.073141

========================================
# Degree Tempered Positive Greedy Solution:
========================================
f = lambda x0,x1: 5.02327690e+01  +  4.08687092e-01 * (  5.22040531e-01*(x0) + -6.31846320e-01*(x0*x0) + 2.62667238e+00*(x0*x1) + 1.32291203e+00*(x1) + -6.29847437e-01*(x1*x1) + 2.37830672e+00 ) / ( 1.0 +  1.06042339e+00*(x0*x0) + 1.06672305e+00*(x1*x1) )

############################################
# Applying a Negative Greedy Algorithm
############################################

Iteration #1 (Negative Greedy)
------------------------------------
>> min_estimator = 1.5096e-01
>> The current boundary = [('01', True), ('00', True), ('11', True), ('00', False), ('11', False), ('1', True), ('0', True)]
>> Exiting because |min_est-initial_estimator_value| = |0.150958-0.073141| = |0.077817| > 0.077039.
>> NOTE that the result of the previous iteration will be kept.

========================================
# Negative Greedy Solution:
========================================
f = lambda x0,x1: 5.02327690e+01  +  4.08687092e-01 * (  5.22040531e-01*(x0) + -6.31846320e-01*(x0*x0) + 2.62667238e+00*(x0*x1) + 1.32291203e+00*(x1) + -6.29847437e-01*(x1*x1) + 2.37830672e+00 ) / ( 1.0 +  1.06042339e+00*(x0*x0) + 1.06672305e+00*(x1*x1) )

Fit Information:
----------------------------------------
f = lambda x0,x1: 5.02327690e+01  +  4.08687092e-01 * (  5.22040531e-01*(x0) + -6.31846320e-01*(x0*x0) + 2.62667238e+00*(x0*x1) + 1.32291203e+00*(x1) + -6.29847437e-01*(x1*x1) + 2.37830672e+00 ) / ( 1.0 +  1.06042339e+00*(x0*x0) + 1.06672305e+00*(x1*x1) )
star.plot()
savefig('/Users/book/KOALA/positive_dev/docs/mvfits_paper/fig/issue3_summary.pdf',pad_inches=0, bbox_inches='tight')_____no_output_____# Plot convergence of greedy process
fig = star.bin['ngreedy_result'].plot()
title('')
xlabel('Iteration Number, $k$',size=18)
ylabel(r'Estimator, $\epsilon$',size=18)
savefig('/Users/book/KOALA/positive_dev/docs/mvfits_paper/fig/issue3_pgreedy.pdf',pad_inches=0, bbox_inches='tight')_____no_output_____
counts = []
counts0 = []
N = 50
for k in range(len(Z_validation)):

    domain,ZZ = ndflatten( [X,Y], Z_validation[k] )
    domain,Z0 = ndflatten( [X,Y], zfun(X,Y) )
    res0 = Z0 - ZZ
    res = star.eval(star.domain) - ZZ

    c,bins,other = hist(res0,N,histtype='step',color='r',alpha=0.01)
    counts.append(c)
    c,bins0,other = hist(res,bins=bins,histtype='step',color='k',alpha=0.01)
    counts0.append(c)_____no_output_____# plot(sum(counts)/len(counts),'-o')
a = sum(counts,axis=0)/sum(sum(counts,axis=0))
b = sum(counts0,axis=0)/sum(sum(counts0,axis=0))

plot( bins[1:], sum(counts,axis=0)/sum(sum(counts,axis=0)), '-o' )
plot( bins[1:], sum(counts0,axis=0)/sum(sum(counts0,axis=0)), '-o' )_____no_output_____vals = xcorr(a,b,normed=True)
print max(vals[1])
print argmax(vals[1])0.994610867251
10

### Notes

 * The greedy tolerance is the sole hyper, and has direct impact on which basis symbols are selected.
 * The negative greedy tolerance is a function of the late stage behavior of the greedy tolerance, and may result in a representation error slightly larger than the input tolerance
 * the accuracy of the fit may be framed as how well the model residuals recover the input noise distribution; in this case the "match" is 0.994015838784_____no_output_____

print star.__str_latex__()f(\vec{x}) \; &= \; 5.02327690e+01 \; + 0.40868709 \; \frac{   5.2204e-01\,x0 \; + \; (-6.3185e-01)\,{x0}^{2} \; + \; (2.6267e+00)\,x0\,x1 \; + \; (1.3229e+00)\,x1 \; + \; (-6.2985e-01)\,{x1}^{2} \; + \; (2.3783e+00)  }{ 1 \; + \;   -1.0604e+00\,{x0}^{2} \; + \; (-1.0667e+00)\,{x1}^{2} }

$
f(\vec{x}) \; = \; 5.02408308e+01 \; + 0.40781196 \; \frac{   5.2285e-01\,x0 \; + \; (-6.0155e-01)\,{x0}^{2} \; + \; (2.6306e+00)\,x0\,x1 \; + \; (1.3250e+00)\,x1 \; + \; (-6.5200e-01)\,{x1}^{2} \; + \; (2.3632e+00)  }{ 1 \; + \;   -1.0597e+00\,{x0}^{2} \; + \; (-1.0660e+00)\,{x1}^{2} }
$_____no_output_____

print star.__str__()f = lambda x0,x1: 5.02327690e+01  +  4.08687092e-01 * (  5.22040531e-01*(x0) + -6.31846320e-01*(x0*x0) + 2.62667238e+00*(x0*x1) + 1.32291203e+00*(x1) + -6.29847437e-01*(x1*x1) + 2.37830672e+00 ) / ( 1.0 +  1.06042339e+00*(x0*x0) + 1.06672305e+00*(x1*x1) )
star.bin.keys()_____no_output_____print mean( star.scalar_range )
print std( star.scalar_range )
print star.numerator_symbols
print star.denominator_symbols
print star.numerator_coeffs * std( star.scalar_range )
print star.denominator_coeffs 50.2327689793
0.40868709196
['0', '00', '01', '1', '11', 'K']
['00', '11']
[ 0.21335123 -0.25822744  1.0734871   0.54065707 -0.25741052  0.97198326]
[-1.06042339 -1.06672305]


# Notebook from MIDOSS/analysis-rachael
Path: notebooks/monte_carlo_dev/.ipynb_checkpoints/make_ATB_fill_fraction-checkpoint.ipynb

## Estimate the amount of oil cargo carried by ATBs by comparing DOE cargo transfers to tank capacities_____no_output_____This notebook is a compliment to [ATB_FullEmptyRatio_AISplot.ipynb](https://github.com/MIDOSS/analysis-rachael/blob/main/notebooks/monte_carlo/ATB_FullEmptyRatio_AISplot.ipynb) and [ATB_FullEmptyRatio.ipynb](https://github.com/MIDOSS/analysis-rachael/blob/main/notebooks/monte_carlo/ATB_FullEmptyRatio.ipynb). See also [Susan's spreadsheet](https://docs.google.com/spreadsheets/d/1nyp_gtYGsc4xvN7eLj48I6eWmitgb-Plut-vmX79UMM/edit#gid=0) on oil transfer

Use `analysis-rachael/env/monte_carlo.yaml` to create an environment for this notebook:
```
 conda env create -f [analysis-rachael/env/monte_carlo.yaml]
 ```
or, to activate this environment, use
```
 conda activate monte_carlo
 ```
To deactivate an active environment, use
```
 conda deactivate
 ```_____no_output_____

import pandas
import numpy
import matplotlib.pyplot as plt
import yaml
from pathlib import Path
import datetime

# import functions for querying DOE and monte-carlo dataframes
from monte_carlo_utils import get_DOE_df, get_DOE_atb_____no_output_____# Dept. of Ecology data files
DOE_dir  = Path('/Users/rmueller/Data/MIDOSS/DeptOfEcology/')
DOE_2018_xlsx = DOE_dir/'MuellerTrans4-30-20.xlsx'

# ATB input spreadsheet
atb_capacity_xlsx = Path(
    '/Users/rmueller/Data/MIDOSS/marine_transport_data/ATB_oil_capacity.xlsx'
)
# ATB output spreadsheet
atb_fullness_xlsx = Path(
    '/Users/rmueller/Data/MIDOSS/marine_transport_data/ATB_fullness.xlsx'
)

# yaml output directory
yaml_out_dir = Path(
    '/Users/rmueller/Data/MIDOSS/marine_transport_data/'
)

# Facility names and lat/lon information file
facilities_xlsx = Path(
    '/Users/rmueller/Data/MIDOSS/marine_transport_data/'
    'Oil_Transfer_Facilities.xlsx'
)

# Constants
bbl2gal = 42  #barrel to gallons_____no_output_____

### Create histogram of percent fill for each transfer, by each vessel, for import and export transfers combined
- Create dictionary `capacity` with tank barge and oil capacity as key:value pairs
- Load DOE data and create dataframes for each tank barge
- Create dictionary of cargo fullness by ATB tank barge
    - Group dataframes by AntID
    - Estimate fullness per transfer based on cargo capacity information. NOTE: Assumption here is that ATBs fill at one location rather than, e.g., two separate locations
    - (save transfer dates and tank barge names in each bin? )
- Create a histogram of percent full for imports, exports and combined for each vessel
- Add up the combined histograms across all vessels_____no_output_____#### Create dictionary `capacity` with tank barge and oil capacity as key:value pairs_____no_output_____

atb_df = pandas.read_excel(
    atb_capacity_xlsx,
    sheet_name='Cargo Capacities',
    usecols="A,H"
    )
capacity = {}
for idx,name in enumerate(atb_df['DOE name']):
    capacity[name] = atb_df['95% or 96% Capacity (bbls)'][idx] * bbl2gal
_____no_output_____capacity_____no_output_____

#### Load DOE data _____no_output_____

# load data
[imports, exports]=get_DOE_atb(
    DOE_2018_xlsx,
    facilities_xlsx,
    transfer_type = 'cargo',
    facilities='selected'
)
imports.Deliverer = imports.Deliverer.replace('ATB BARGE DBL185','ATB BARGE DBL 185')
exports.Receiver = exports.Receiver.replace('ATB BARGE DBL185','ATB BARGE DBL 185')_____no_output_____imports.head(1)_____no_output_____

#### Create an ordered list of ATB names sorted by highest to lowest number of transfers_____no_output_____

# extract the important stuff
imports_sm = imports[['AntID','Deliverer','TransferQtyInGallon']]
exports_sm = exports[['AntID','Receiver','TransferQtyInGallon']]
# define way to group dataframe
aggregate_in = {
    'AntID': 'first',
    'Deliverer': 'first',
    'TransferQtyInGallon': 'sum'
}
aggregate_out = {
    'AntID': 'first',
    'Receiver': 'first',
    'TransferQtyInGallon': 'sum'
}_____no_output_____count_in = {}
count_out = {}
count = {}
# count exports
exports_sm_AndID = exports_sm.groupby(exports['AntID']).aggregate(aggregate_out)
count_out =  exports_sm_AndID[['Receiver','TransferQtyInGallon']].groupby(
    'Receiver').count().rename(columns={'TransferQtyInGallon':'Count'})
count_out.index.name = 'Name'
count_out.reset_index(inplace=True)
# count imports
imports_sm_AndID = imports_sm.groupby(imports['AntID']).aggregate(aggregate_in)
count_in =  imports_sm_AndID[['Deliverer','TransferQtyInGallon']].groupby(
    'Deliverer').count().rename(columns={'TransferQtyInGallon':'Count'})
count_in.index.name = 'Name'
count_in.reset_index(inplace=True)
#create new df
count = pandas.DataFrame(
    {'Name':list(count_out['Name']) + list(set(count_in['Name']) - set(count_out['Name']))})
count = pandas.merge(
    left = count,
    right = count_out,
    how = 'left',
    on = 'Name'
).fillna(0)
# Add import transfers to export transfers
count = count.set_index('Name').add(count_in.set_index('Name'), fill_value=0).reset_index()
# order list by total number of transfers
count = count.sort_values(by='Count', ascending=False)
count_____no_output_____

#### Create dictionary of cargo fullness by tank barge _____no_output_____

# extract the important stuff
imports = imports[
    ['AntID','StartDateTime','Deliverer','Receiver',
     'Product','TransferQtyInGallon']
]
exports = exports[
    ['AntID','StartDateTime','Deliverer','Receiver',
     'Product','TransferQtyInGallon']
]
# define way to group dataframe
aggregation_functions = {
    'AntID': 'first',
    'StartDateTime': 'first',
    'Deliverer': 'first',
    'Receiver': 'first',
    'Product': lambda x: x.tolist(),
    'TransferQtyInGallon': 'sum'
}_____no_output_____imports_fill = {}
exports_fill = {}
fullness = {}

# first group by transfer ID (AntID) and add `TransferQtyInGallon` for each AntID
imports_byAntID = imports.groupby(imports['AntID']).aggregate(aggregation_functions)
exports_byAntID = exports.groupby(exports['AntID']).aggregate(aggregation_functions)

# loop through ordered list of vessel names and create excel sheet with %fullness
# values for each AntID, by vessel
with pandas.ExcelWriter(atb_fullness_xlsx) as writer:
    for idx,name in enumerate(count['Name']):
        # ~~~ imports ~~~
        imports_fill[name] = imports_byAntID.loc[
            imports_byAntID['Deliverer']==name,
            ['StartDateTime','TransferQtyInGallon','Receiver','Product']
        ].rename(columns={'Receiver':'Facility'})
        # divide total gallons transferred by cargo capacity to get fullness
        imports_fill[name]['FillFraction'] = numpy.zeros(
            len(imports_fill[name]['TransferQtyInGallon']))
        imports_fill[name]['FillFraction'] = numpy.asarray(
            [x / capacity[name]
             for x in imports_fill[name]['TransferQtyInGallon']]
        )
        imports_fill[name]['direction'] = 'import'
        # ~~~ exports ~~~
        exports_fill[name] = exports_byAntID.loc[
            exports_byAntID['Receiver']==name,
            ['StartDateTime','TransferQtyInGallon','Deliverer','Product']
        ].rename(columns={'Deliverer':'Facility'})
        # divide total gallons transferred by cargo capacity to get fullness
        exports_fill[name]['FillFraction'] = numpy.zeros(
            len(exports_fill[name]['TransferQtyInGallon']))
        exports_fill[name]['FillFraction'] = numpy.asarray([
            x / capacity[name]
            for x in exports_fill[name]['TransferQtyInGallon']
        ])
        exports_fill[name]['direction'] = 'export'

        # ~~~ combine imports and exports ~~~
        frames = [imports_fill[name], exports_fill[name]]
        fullness[name] = pandas.concat(frames).reset_index()

        # ~~~ save to excel with new tab for each ATB ~~~
        fullness[name].to_excel(writer, sheet_name=name)_____no_output_____

### Create a histogram of percent full for imports, exports and combined for each vessel_____no_output_____

count['Name'][0]_____no_output_____nbins = 14
max_full = 1.4
bin_values = numpy.arange(0,max_full,max_full/nbins)
binned_transfers = numpy.zeros(nbins-1)
# select barge for testing
#name = count['Name'][0]
# n={}
# bins={}
# patches={}
for idx,name in enumerate(count['Name']):
    # the histogram of the data
    fig, ax = plt.subplots(figsize = (3,3))
    n, bins, patches = ax.hist(
        fullness[name]['FillFraction'],
        bins = bin_values)
    ax.set_xlabel('Full fraction')
    ax.set_ylabel('Number of transfers')
    ax.set_ylim(0,30)
    ax.set_title(f'{name}')

    # combine all transfers into one
    binned_transfers = numpy.add(binned_transfers,n)_____no_output_____

### Add up and plot combined histograms across all vessels
- add all transfers with fill values > 1.0 to 1.0.  _____no_output_____

len(bins)_____no_output_____bins_____no_output_____binned_transfers_____no_output_____binned_transfers[9] += binned_transfers[-3:].sum()
binned_transfers = numpy.delete(binned_transfers,[10,11,12])
binned_transfers_____no_output_____len(binned_transfers)_____no_output_____# set bins to upper bound of range for fill capacity
bins = numpy.arange(.1,1.1,.1)_____no_output_____len(bins)_____no_output_____#centered_bins=[(bins[i] + bins[i+1])/2 for i in numpy.arange(0,nbins-1)]
fig, ax = plt.subplots(figsize = (3,3))
ax.bar(bins, binned_transfers)
ax.set_title('Histogram of transfers by fraction of fullness')
ax.set_ylabel('Number of transfers')
ax.set_xlabel('Fullness fraction')_____no_output_____binned_transfers_____no_output_____binned_transfers.sum()_____no_output_____fullness_weight = numpy.array([transfers/binned_transfers.sum() for transfers in binned_transfers])_____no_output_____fullness_weight_____no_output_____fullness_weight.sum()_____no_output_____

### Output to .yaml_____no_output_____

fullness = {}
fullness['weights'] = fullness_weight.tolist()
fullness['fill_fraction'] = bins.tolist()_____no_output_____fullness['fill_fraction']_____no_output_____with open(yaml_out_dir/'fill_fraction.yaml', 'w') as file:
    documents = yaml.safe_dump(fullness, file)_____no_output_____


# Notebook from d-diaz/Lidar_Plot_Registration
Path: notebooks/02_Visualize a Single Tree in 3D.ipynb

# How we are modeling individual trees
Several geometric models have been described for approximating the shape of a tree crown.

In this project, we utilize an asymmetric hull model, described originally by Koop (1989) and shown in the graphic to the right, and applied later by researchers including Cescatti (1997), Sheng et al. (2001), and Pradal et al. (2009).

This flexibility of this model allows specification of trees that are conical, ellipsoidal, or concave, and with mixtures of these shapes in the top and bottom of the crown. The flexibility of this model was illustrated well by Pradal et al. (2007), below.

_____no_output_____# Make your own tree
Using the sliders below, you can adjust the parameters which determine the location, size, and shape of the tree crown._____no_output_____

from forest3d.visualize import plot_tree_with_widgets
plot_tree_with_widgets()_____no_output_____

### References
Cescatti, A. (1997). "Modelling the radiative transfer in discontinuous canopies of asymmetric crowns. I. Model structure and algorithms." Ecological Modelling 101: 263–274.

Koop, H. (1989). Forest dynamics: SILVI-STAR: a comprehensive monitoring system. Springer: New York.

Pradal, C.; Boudon, F.; Nouguier, C.; Chopard, J.; Godin, C. (2007). "PlantGL: a Python-based geometric library for 3D plant modelling at different scales." Institut National de Recherche en Informatique et en Automatique: Sophia Antipolis, France. 39pp.

Pradal, C.; Boudon, F.; Nouguier, C.; Chopard, J.; Godin, C. (2009). "PlantGL: A Python-based geometric library for 3D plant modelling at different scales." Graphical Models 71: 1–21.

Sheng, Yongwei; Gong, P.; Biging, G. S. (2001). "Model-Based Conifer-Crown Surface Reconstruction from High-Resolution Aerial Images." Photogrammetric Engineering & Remote Sensing 67: 957–965.  _____no_output_____


# Notebook from Renelvon/fsic
Path: tutorial/nfsic_locs_surface.ipynb

This notebook is to investigate the value of NFSIC as a function of the two test location $v$ and $w$._____no_output_____

%load_ext autoreload
%autoreload 2
%matplotlib inline
#%config InlineBackend.figure_format = 'svg'
#%config InlineBackend.figure_format = 'pdf'
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import fsic.util as util
import fsic.data as data
import fsic.kernel as kernel
import fsic.indtest as it
import scipy.stats as stats
_____no_output_____# font options
font = {
    #'family' : 'normal',
    #'weight' : 'bold',
    'size'   : 16
}

plt.rc('font', **font)
plt.rc('lines', linewidth=2)
#matplotlib.rc('text', usetex=True)
#matplotlib.rcParams['text.latex.preamble']=[r"\usepackage{amsmath}"]_____no_output_____def get_med_kernels(X, Y):
    """Return kernels k and l by computing the median heuristic
    for X, and Y."""
    # data and parameters
    medx2 = util.median_distance(X)**2
    medy2 = util.median_distance(Y)**2
    # print
    print('med x: %.3f'%medx2**0.5)
    print('med y: %.3f'%medy2**0.5)
    k = kernel.KGauss(medx2)
    l = kernel.KGauss(medy2)
    return k, l
_____no_output_____def nfsic_stat(pdata, k, l, V, W, reg=0):
    nfsic = it.NFSIC(k, l, V, W, alpha=0.01, reg=reg)
    s = nfsic.compute_stat(pdata)
    return s

def nfsic_u(pdata, k, l, V, W, reg=0):
    X, Y = pdata.xy()
    _, u, _ = it.nfsic(X, Y, k, l, V, W, reg=reg)
    return u

def nfsic_joint_embed(pdata, k, l, V, W, reg=0):
    X, Y = pdata.xy()
    n = X.shape[0]
    J = V.shape[0]

    K = k.eval(X, V) # n x J
    L = l.eval(Y, W) # n x J
    # mean
    mean_k = np.mean(K, 0)
    mean_l = np.mean(L, 0)
    return np.mean(K*L, 0)

def nfsic_product_embeds(pdata, k, l, V, W, reg=0):
    X, Y = pdata.xy()
    n = X.shape[0]
    J = V.shape[0]

    K = k.eval(X, V) # n x J
    L = l.eval(Y, W) # n x J
    # mean
    mean_k = np.mean(K, 0)
    mean_l = np.mean(L, 0)
    return np.mean(K, 0)*np.mean(L, 0)

def nfsic_variance(pdata, k, l, V, W, reg=0):
    X, Y = pdata.xy()
    _, _, Sig = it.nfsic(X, Y, k, l, V, W, reg=reg)
    return Sig
_____no_output_____
def generic_contourf(pdata, k, l, func, title=None):
    """
    func: (pdata, k, l, V, W) |-> value. A function computing the values to plot.
    """
    X, Y = pdata.xy()
    max1, max2 = np.max(X, 0), np.max(Y, 0)
    min1, min2 = np.min(X, 0), np.min(Y, 0)
    #sd1, sd2 = np.std(XY, 0)
    sd1, sd2 = ((max1-min1)*0.3, (max2-min2)*0.3)
    # form a frequency grid to try
    nd1 = 40
    nd2 = 40
    loc1_cands = np.linspace(min1-sd1/2, max1+sd1/2, nd1)
    loc2_cands = np.linspace(min2-sd2/2, max2+sd2/2, nd2)
    lloc1, lloc2 = np.meshgrid(loc1_cands, loc2_cands)
    # nd2 x nd1 x 2
    loc3d = np.dstack((lloc1, lloc2))
    # #candidates x 2
    all_loc2s = np.reshape(loc3d, (-1, 2) )

    # all_locs = #candidates x J x 2
    #all_locs = np.array( [np.vstack((c, loc1)) for c in all_loc2s] )

    # evaluate Lambda(T) on each candidate T on the grid. Size = (#candidates, )
    stat_grid = np.array([func(pdata, k, l, np.array([T[0]])[np.newaxis, :],
                                     np.array([T[1]])[np.newaxis, :]) for T in all_loc2s])
    stat_grid = np.reshape(stat_grid, (nd2, nd1) )

    #ax = fig.gca(projection='3d')
    #ax.plot_surface(lloc1, lloc2, stat_grid, rstride=8, cstride=8, alpha=0.3)
    #cset = ax.contourf(lloc1, lloc2, stat_grid, zdir='z', offset=0, cmap=cm.coolwarm)
    plt.figure(figsize=(8, 3))
    plt.contourf(lloc1, lloc2, stat_grid, alpha=0.6)
    plt.gca().get_xaxis().set_visible(False)
    plt.gca().get_yaxis().set_visible(False)
    #plt.axis('off')
    #plt.colorbar()

    max_stat = np.max(stat_grid)
    #plt.xlabel('')
    #plt.ylabel('')
    n = X.shape[0]
    if title is not None:
        pass
        #plt.title(title)
    #ax.view_init(elev=max_stat*2, azim=90)

    # plot the data
    plt.plot(X, Y, '.k', markeredgecolor='k', markersize=5, alpha=0.9)
    #plt.xlabel('$X$')
    #plt.ylabelel('$Y$')

    # return the locations V, W
    ind2d = np.unravel_index(np.argmax(stat_grid), stat_grid.shape)
    V = np.array([ loc1_cands[ind2d[1]] ])[np.newaxis, :]
    W = np.array([ loc2_cands[ind2d[0]] ])[np.newaxis, :]
    print('V: %s'%V)
    print('W: %s'%W)

    # put a star at the highest location
    #plt.plot(V[0, 0], W[0, 0], 'r*', markersize=25)
    return V, W

def nfsic_contourf(pdata, k, l):
    """return V, W corresponding to the points with the highest objective"""

    n = pdata.sample_size()
    title = '$(\mathbf{v}, \mathbf{w}) \mapsto \hat{\lambda}_{n}((\mathbf{v}, \mathbf{w}))$. $n=%d$. $J=1$'%n
    V, W = generic_contourf(pdata, k, l, nfsic_stat, title)
    return V, W

def nfsic_mean_embeds_contourf(pdata, k, l):
    """
    Plot the difference of the joint mean embedding to the product of mean embeddings
    of the marginals.
    """
    n = pdata.sample_size()
    title = r'$\hat{\mu}_{xy}(\mathbf{v}, \mathbf{w}) - \hat{\mu}_x(\mathbf{v}) \hat{\mu}_y(\mathbf{w})$. $n=%d$. $J=1$'%n
    V, W = generic_contourf(pdata, k, l, nfsic_u, title)
    return V, W

def nfsic_var_contourf(pdata, k, l):
    """
    Plot the variance that is used to normalize the statistic.
    """
    n = pdata.sample_size()
    title = r'$\hat{\mathbf{\Sigma}}((\mathbf{v}, \mathbf{w}))$. $n=%d$. $J=1$'%n
    V, W = generic_contourf(pdata, k, l, nfsic_variance, title)
    return V, W

def nfsic_joint_embed_contourf(pdata, k, l):
    """
    Plot the joint mean embedding.
    """
    n = pdata.sample_size()
    title = r'$\hat{\mu}_{xy}(\mathbf{v}, \mathbf{w})$. $n=%d$. $J=1$'%n
    V, W = generic_contourf(pdata, k, l, nfsic_joint_embed, title)
    return V, W

def nfsic_product_embeds_contourf(pdata, k, l):
    """
    Plot the product of the embeddings of the marginals of X and Y.
    """
    n = pdata.sample_size()
    title = r'$\hat{\mu}_x(\mathbf{v}) \hat{\mu}_y(\mathbf{w})$. $n=%d$. $J=1$'%n
    V, W = generic_contourf(pdata, k, l, nfsic_product_embeds, title)
    return V, W
    _____no_output_____def get_quad_psfunc():
    """
    Return a PairedSource to generate y = x^2 + Gaussian noise.
    """
    mean = 2
    width = 6
    px = lambda n: stats.uniform.rvs(loc=mean-width/2, scale=width, size=n)[:, np.newaxis]
    f = lambda x: 0.3*(x-1)**2 + 0.3*np.random.randn(x.shape[0], 1)
    #f = lambda x: x
    return data.PSFunc(f, px)

def plot_locs_surface(pairedSource, n, seed=29, fname_prefix=''):
    pdata = pairedSource.sample(n, seed=seed)
    tr, te = pdata.split_tr_te(tr_proportion=0.5, seed=10)
    xtr, ytr = tr.xy()
    k, l = get_med_kernels(xtr, ytr)

    # plot the normalized statistics
    V, W = nfsic_contourf(tr, k, l)
    plt.savefig('%s_obj.pdf'%fname_prefix)
    # plot the unnormalized statistics
    nfsic_mean_embeds_contourf(tr, k, l)
    plt.savefig('%s_embed_diff.pdf'%fname_prefix)
    # plot the normalizing variance
    nfsic_var_contourf(tr, k, l)
    plt.savefig('%s_var.pdf'%fname_prefix)
    # plot the joint embedding
    nfsic_joint_embed_contourf(tr, k, l)
    plt.savefig('%s_joint_embed.pdf'%fname_prefix)
    # plot the product of marginal embeddings
    nfsic_product_embeds_contourf(tr, k, l)
    plt.savefig('%s_prod_embed.pdf'%fname_prefix)

    # also do testing
    nfsic = it.NFSIC(k, l, V, W)
    test_result = nfsic.perform_test(te)
    print(test_result)
    _____no_output_____ps_sin = data.PS2DSinFreq(freq=1)
plot_locs_surface(ps_sin, n=1500, seed=33, fname_prefix='sin')_____no_output_____def get_lin_psfunc():
    """
    Return a PairedSource to generate y = -x + Gaussian noise.
    """
    mean = 2
    width = 10
    px = lambda n: stats.uniform.rvs(loc=mean-width/2, scale=width, size=n)[:, np.newaxis]
    f = lambda x: -x + 0.3*np.random.randn(x.shape[0], 1)
    return data.PSFunc(f, px)

ps_lin = get_lin_psfunc()
plot_locs_surface(ps_lin, n=1000, seed=32, fname_prefix='lin')_____no_output_____# paired source
alpha = 0.01
n = 500
seed = 14

ps = get_quad_psfunc()
#ps = data.PSIndUnif(xlb=[0, 3], xub=[1, 10], ylb=[-5, 5], yub=[8, 10])
plot_locs_surface(ps, n)_____no_output_____dx = 1
dy = 1
ps_sg = data.PSIndSameGauss(dx, dy)
plot_locs_surface(ps_sg, n=400)_____no_output_____

When $X$ and $Y$ are independent, the regions that maximize the objective keep changing arbitrarily with $n$._____no_output_____

def get_sin_psfunc(freq = 1/3.0):
    """
    Return a PairedSource to generate y = sin(x*f) + Gaussian noise.
    """
    px = lambda n: np.random.randn(n, 1)*8
    f = lambda x: np.sin(freq*x) + 0.1*np.random.randn(x.shape[0], 1)
    return data.PSFunc(f, px)

ps_sin = get_sin_psfunc()
plot_locs_surface(ps_sin, n=600)
_____no_output_____

This is the first example where the regions with high objective values keep moving as $n$ changes, even when $X$ and $Y$ are dependent. The test fails to reject $H_0$. It seems the test does not like a dependency given by a periodic function. **Update**: the test can handle it if the frequency is not too high._____no_output_____

ps_rotate = data.PS2DUnifRotate(angle=np.pi*(1.0/4), xlb=-1, xub=1, ylb=-1, yub=1)
plot_locs_surface(ps_rotate, n=800)_____no_output_____rot_data = ps_rotate.sample(3000, seed=3)
x, y = rot_data.xy()
plt.hist(x)_____no_output_____


# Notebook from Young-won/deepbiome
Path: examples/preprocessing_phylogenetic_tree_sim1.ipynb

# Preprocessing for simulation 1

## Effects at phylum level and order level

@ Aug. 16, Youngwon (youngwon08@gmail.com)_____no_output_____

import pandas as pd
import numpy as np
from sklearn.utils.extmath import softmax as softmax
from scipy.special import expit as sigmoid
from sklearn.preprocessing import MinMaxScaler

import matplotlib.pyplot as plt
%matplotlib inline_____no_output_____path = "./data/simulation/s1"
path_genus = "./data/genus48"

def relu(x):
    x[x <= 0.] = 0.
    return x_____no_output_____

## Phylogenetic Tree information_____no_output_____

## TODO: Should we start with genus48.csv?

phylogenetic_tree_info = pd.read_csv('%s/genus48_dic.csv' % path_genus)
# tree_level_list = ['Genus', 'Family', 'Order', 'Class', 'Phylum']
tree_level_list = phylogenetic_tree_info.columns[:-1].tolist()
phylogenetic_tree_info_____no_output_____print('------------------------------------------------------------------------------------------')
print('Phylogenetic tree level list: %s' % tree_level_list)
print('------------------------------------------------------------------------------------------')
phylogenetic_tree_dict = {'Number':{}}
for tree_lvl in tree_level_list:
    lvl_category = phylogenetic_tree_info[tree_lvl].unique()
    lvl_num = lvl_category.shape[0]
    print('%6s: %d' % (tree_lvl, lvl_num))
    phylogenetic_tree_dict[tree_lvl] = dict(zip(lvl_category, np.arange(lvl_num)))
    phylogenetic_tree_dict['Number'][tree_lvl]=lvl_num
print('------------------------------------------------------------------------------------------')
print('Phylogenetic_tree_dict info: %s' % list(phylogenetic_tree_dict.keys()))
print('------------------------------------------------------------------------------------------')------------------------------------------------------------------------------------------
Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']
------------------------------------------------------------------------------------------
 Genus: 48
Family: 40
 Order: 23
 Class: 17
Phylum: 9
------------------------------------------------------------------------------------------
Phylogenetic_tree_dict info: ['Phylum', 'Number', 'Family', 'Genus', 'Order', 'Class']
------------------------------------------------------------------------------------------
phylogenetic_tree = phylogenetic_tree_info.iloc[:,:-1]
for tree_lvl in tree_level_list:
    phylogenetic_tree[tree_lvl] = phylogenetic_tree[tree_lvl].map(phylogenetic_tree_dict[tree_lvl])
phylogenetic_tree = np.array(phylogenetic_tree)
phylogenetic_tree/usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py:3: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  This is separate from the ipykernel package so we can avoid doing imports until
/usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py:3: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  This is separate from the ipykernel package so we can avoid doing imports until
/usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py:3: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  This is separate from the ipykernel package so we can avoid doing imports until
/usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py:3: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  This is separate from the ipykernel package so we can avoid doing imports until
/usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py:3: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  This is separate from the ipykernel package so we can avoid doing imports until
tree_weight_list = []
tree_weight_noise_list = []
num_dict = phylogenetic_tree_dict['Number']
for i in range(len(tree_level_list)-1):
    print('Build edge weights between [%6s, %6s]'%(tree_level_list[i],tree_level_list[i+1]))
    lower = phylogenetic_tree[:,i]
    upper = phylogenetic_tree[:,i+1]
    n_lower = num_dict[tree_level_list[i]]
    n_upper = num_dict[tree_level_list[i+1]]

    tree_w = np.zeros((n_lower,n_upper))
    tree_w_n = np.zeros_like(tree_w) + 0.01
    for j in range(n_upper):
        tree_w[lower[j==upper],j] = 1.
        tree_w_n[lower[j==upper],j] = 1.
    tree_weight_list.append(tree_w)
    tree_weight_noise_list.append(tree_w_n)Build edge weights between [ Genus, Family]
Build edge weights between [Family,  Order]
Build edge weights between [ Order,  Class]
Build edge weights between [ Class, Phylum]

### Check Error_____no_output_____

maskw_l1 = np.array(pd.read_csv('%s/mw1.csv' % path))
maskw_l2 = np.array(pd.read_csv('%s/mw2.csv' % path))
maskw_l3 = np.array(pd.read_csv('%s/mw3.csv' % path))
maskw_l4 = np.array(pd.read_csv('%s/mw4.csv' % path))
maskw_list = [maskw_l1,maskw_l2,maskw_l3,maskw_l4]
print([np.sum(np.abs(tw-mw)) for tw, mw in zip(tree_weight_noise_list, maskw_list)])[0.0, 0.0, 0.0, 0.0]

# Regression effected with order level and phylum level selections

Aug. 10, 2019

## Experiments with 1000 replications

## Generating X: microbiome abundance data
* Using the data that Jing made, described in detail by Zhai et al. (2018a,b).
    * $n \times p$ OTU count matrix
    * Aggregating $p_0=2964$ OTUs to $p=48$ genus
* Sample size for each replication: $n = 1000$
    * Training $75%$; Test $25%$

* Phylogenetic tree levels:
    1. Genus: $m^0 = 48$
    1. Family: $m^1 = 40$
    1. Order: $m^2 = 23$
    1. Class: $m^3 = 17$
    1. Phylum: $m^4 = 9$_____no_output_____## Generation Y: a forward propagation approach

###  Ver 0: same weights with same true connection for each repeatition

#### (similar to the original version)

$x^0 = x^{input} \in \mathbb{R}^{n\times p}$ (input genus abundance data)

#### Before repeatition,

* Select the true connection to disease (based on the phylogenetic tree information)
    1. Choose 4 indexes from the order nodes
    1. Construct the disease path according to the chosen order nodes.

* Construct the true weights.
    1. For $i = 1,2,...,5$, $w^{i} \in \mathbb{R}^{m^{i-1} \times m^{i}}, b^{i} \in \mathbb{R}^{m^{i} \times 1},$
       $$w^{i}_{j,k} \sim \left\{\begin{matrix}
       \text{Uniform}(-4.5,3) \quad\quad  \text{associated with the disease} \\
%        \mathcal{N}(0,0.0001) \quad\quad  \text{not associaated with the disease} \\
       0 \quad\quad  \text{not associaated with the disease} \\
       0 \quad\quad\quad \text{not associaated with the phylogenetic trees}
       \end{matrix}\right.$$
       $$b^{i}_{k} \sim \mathcal{N}(0,4)$$

#### For each repeatition,

* For $i = 1,2,...,4$,
    1. $h^i = w^{i} x^{i-1} + b^i$
    1. $x^{i} = \text{ReLU}(h^i) \in \mathbb{R}^{m^{i}}$
* For the last layer for the classification with the K classes:
    1.  $y=x^{5} = w^{5}x^{4}+b^{5} + \epsilon$ where $$\epsilon \sim \mathcal{N}(0, 1)$$_____no_output_____

verbose=False
count_path = 'data/simulation/count/'
x_list = np.array(pd.read_csv('data/simulation/gcount_list.csv', header=None)[0])
np.random.seed(10)
print('-------------------------------------------------------------------------------')
print('Generation True Connection to disease')
order_idx = np.array([5,4,1,9]) # 5(0) and 4(2) are effect at order level, 1(1) and 9(1) are effects at phylum level
phylum_idx = np.array([1,0,2]) # 1(1, 9) are effect at phylum level, 1(6) and 3(5) are effects at order level

true_tree_idx_list = []
for i in range(5):
    idx_order = np.unique(np.concatenate([phylogenetic_tree[:,i][phylogenetic_tree[:,2]==k] for k in order_idx]))
    idx_phylum = np.unique(np.concatenate([phylogenetic_tree[:,i][phylogenetic_tree[:,4]==k] for k in phylum_idx]))
    idx = np.intersect1d(idx_order, idx_phylum)
    print("%6s idx: %s"% (tree_level_list[i], idx))
    true_tree_idx_list.append(idx)
print('------------------------------------------------------------------------------------------')
true_tree_weight_list = []
num_dict = phylogenetic_tree_dict['Number']
for i in range(len(tree_level_list)-1):
    print('Build true edge weights between [%6s, %6s]'%(tree_level_list[i],tree_level_list[i+1]))
    lower = phylogenetic_tree[:,i]
    upper = phylogenetic_tree[:,i+1]
    n_lower = num_dict[tree_level_list[i]]
    n_upper = num_dict[tree_level_list[i+1]]
    tree_w = np.zeros((n_lower,n_upper), dtype=np.float32)
    for j in true_tree_idx_list[i]:
        tree_w[j,upper[lower==j]] = 1.
    true_tree_weight_list.append(tree_w)
    np.save('%s/ver0/tw_%d.npy'%(path,i+1), np.repeat([tree_w], x_list.shape[0], axis=0))
    print('Saved true edge weights between [%6s, %6s] to %s/ver0/tw_%d.npy'%(tree_level_list[i],tree_level_list[i+1],path,i+1))

print('-------------------------------------------------------------------------------')
print('Generation y')

yweight_list = []
ybias_list = []
for i in range(len(true_tree_weight_list)):
    yw = np.random.uniform(-4.5,3., true_tree_weight_list[i].shape).astype(np.float32) * true_tree_weight_list[i] # left only the weights on the true connections to the disease
    yb = np.random.normal(0,4, true_tree_weight_list[i].shape[-1]).astype(np.float32)
    yw = np.repeat([yw], x_list.shape[0], axis=0)
    yb = np.repeat([yb], x_list.shape[0], axis=0)
    yweight_list.append(yw)
    ybias_list.append(yb)
    np.save('%s/ver0/solw_%d.npy'%(path,i), yw)
    np.save('%s/ver0/solb_%d.npy'%(path,i), yb)

ywc = np.zeros((true_tree_weight_list[3].shape[-1],1), dtype=np.float32)
ywc[:3,0] = np.random.uniform(-4.5,3., 3).astype(np.float32)
# ywc = np.expand_dims(np.array([-0.025, 0.15, -0.35, 0, 0, 0, 0, 0, 0]), axis=-1)
ywc[:,0] = ywc[:,0] * (np.sum(true_tree_weight_list[-1], axis=0) > 0).astype(np.float32)
ywc = np.repeat([ywc], x_list.shape[0], axis=0)
ybc = np.random.normal(0,4, ywc.shape[-1]).astype(np.float32)
ybc = np.repeat([ybc], x_list.shape[0], axis=0)
np.save('%s/ver0/solw_%d.npy'%(path,len(true_tree_weight_list)), ywc)
# np.save('%s/ver0/solb_%d.npy'%(path,len(true_tree_weight_list)), ybc)

newy_all = []
for fold in range(x_list.shape[0]):
    x = pd.read_csv('%s/%s'%(count_path, x_list[fold])) # input x
    mat = np.matrix(x)
    prepro = MinMaxScaler()
    prepro.fit(mat)
    x = pd.DataFrame(prepro.transform(mat), columns = list(x.columns))
    h = np.array(x, dtype=np.float32)
    for i, (yw, yb) in enumerate(zip(yweight_list,ybias_list)):
        yw_noise = yw[fold]
#         yw_noise += np.random.normal(0,0.0001, true_tree_weight_list[i].shape) \
#                     *(1.-true_tree_weight_list[i])*(tree_weight_list[i]) # add noise on the tree
        h = relu(np.dot(h, yw_noise) + np.repeat([yb[fold]], h.shape[0], axis=0))
    h = np.dot(h, ywc[fold])
    p = h + np.repeat([ybc[fold]], h.shape[0], axis=0)
    p += np.random.normal(0,1)
    newy = p[:,0]
    newy_all.append(newy)
newy_all = pd.DataFrame(np.stack(newy_all).T)
newy_all.to_csv('%s/ver0/y.csv'%path, index=False)
np.save('%s/ver0/solb_%d.npy'%(path,len(true_tree_weight_list)), ybc)-------------------------------------------------------------------------------
Generation True Connection to disease
 Genus idx: [ 1  3  5  6  7  9 11 16 17 21 24 25 32 37 38 41 44 46 47]
Family idx: [ 1  3  5  6  7  9 11 15 16 21 27 32 34 38 39]
 Order idx: [1 4 5 9]
 Class idx: [1 4 5]
Phylum idx: [0 1 2]
------------------------------------------------------------------------------------------
Build true edge weights between [ Genus, Family]
Saved true edge weights between [ Genus, Family] to ./data/simulation/s1/ver0/tw_1.npy
Build true edge weights between [Family,  Order]
Saved true edge weights between [Family,  Order] to ./data/simulation/s1/ver0/tw_2.npy
Build true edge weights between [ Order,  Class]
Saved true edge weights between [ Order,  Class] to ./data/simulation/s1/ver0/tw_3.npy
Build true edge weights between [ Class, Phylum]
Saved true edge weights between [ Class, Phylum] to ./data/simulation/s1/ver0/tw_4.npy
-------------------------------------------------------------------------------
Generation y
newy_all_____no_output_____plt.hist(np.array(newy_all).flatten())_____no_output_____