# Notebook from amrhwanis22/DSND-Capstone-Starbucks
Path: Starbucks_Capstone_notebook.ipynb

# Starbucks Capstone Challenge

### Introduction

This data set contains simulated data that mimics customer behavior on the Starbucks rewards mobile app. Once every few days, Starbucks sends out an offer to users of the mobile app. An offer can be merely an advertisement for a drink or an actual offer such as a discount or BOGO (buy one get one free). Some users might not receive any offer during certain weeks.

Not all users receive the same offer, and that is the challenge to solve with this data set.

Your task is to combine transaction, demographic and offer data to determine which demographic groups respond best to which offer type. This data set is a simplified version of the real Starbucks app because the underlying simulator only has one product whereas Starbucks actually sells dozens of products.

Every offer has a validity period before the offer expires. As an example, a BOGO offer might be valid for only 5 days. You'll see in the data set that informational offers have a validity period even though these ads are merely providing information about a product; for example, if an informational offer has 7 days of validity, you can assume the customer is feeling the influence of the offer for 7 days after receiving the advertisement.

You'll be given transactional data showing user purchases made on the app including the timestamp of purchase and the amount of money spent on a purchase. This transactional data also has a record for each offer that a user receives as well as a record for when a user actually views the offer. There are also records for when a user completes an offer.

Keep in mind as well that someone using the app might make a purchase through the app without having received an offer or seen an offer.

### Example

To give an example, a user could receive a discount offer buy 10 dollars get 2 off on Monday. The offer is valid for 10 days from receipt. If the customer accumulates at least 10 dollars in purchases during the validity period, the customer completes the offer.

However, there are a few things to watch out for in this data set. Customers do not opt into the offers that they receive; in other words, a user can receive an offer, never actually view the offer, and still complete the offer. For example, a user might receive the "buy 10 dollars get 2 dollars off offer", but the user never opens the offer during the 10 day validity period. The customer spends 15 dollars during those ten days. There will be an offer completion record in the data set; however, the customer was not influenced by the offer because the customer never viewed the offer.

### Cleaning

This makes data cleaning especially important and tricky.

You'll also want to take into account that some demographic groups will make purchases even if they don't receive an offer. From a business perspective, if a customer is going to make a 10 dollar purchase without an offer anyway, you wouldn't want to send a buy 10 dollars get 2 dollars off offer. You'll want to try to assess what a certain demographic group will buy when not receiving any offers.

### Final Advice

Because this is a capstone project, you are free to analyze the data any way you see fit. For example, you could build a machine learning model that predicts how much someone will spend based on demographics and offer type. Or you could build a model that predicts whether or not someone will respond to an offer. Or, you don't need to build a machine learning model at all. You could develop a set of heuristics that determine what offer you should send to each customer (i.e., 75 percent of women customers who were 35 years old responded to offer A vs 40 percent from the same demographic to offer B, so send offer A)._____no_output_____# Data Sets

The data is contained in three files:

* portfolio.json - containing offer ids and meta data about each offer (duration, type, etc.)
* profile.json - demographic data for each customer
* transcript.json - records for transactions, offers received, offers viewed, and offers completed

Here is the schema and explanation of each variable in the files:

**portfolio.json**
* id (string) - offer id
* offer_type (string) - type of offer ie BOGO, discount, informational
* difficulty (int) - minimum required spend to complete an offer
* reward (int) - reward given for completing an offer
* duration (int) - time for offer to be open, in days
* channels (list of strings)

**profile.json**
* age (int) - age of the customer
* became_member_on (int) - date when customer created an app account
* gender (str) - gender of the customer (note some entries contain 'O' for other rather than M or F)
* id (str) - customer id
* income (float) - customer's income

**transcript.json**
* event (str) - record description (ie transaction, offer received, offer viewed, etc.)
* person (str) - customer id
* time (int) - time in hours since start of test. The data begins at time t=0
* value - (dict of strings) - either an offer id or transaction amount depending on the record

**Note:** If you are using the workspace, you will need to go to the terminal and run the command `conda update pandas` before reading in the files. This is because the version of pandas in the workspace cannot read in the transcript.json file correctly, but the newest version of pandas can. You can access the termnal from the orange icon in the top left of this notebook.

You can see how to access the terminal and how the install works using the two images below.  First you need to access the terminal:

Then you will want to run the above command:

Finally, when you enter back into the notebook (use the jupyter icon again), you should be able to run the below cell without any errors._____no_output_____## Importing needed libraries
_____no_output_____

import pandas as pd
import numpy as np
import math
import json
% matplotlib inline
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import LabelEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report
from sklearn.svm import SVC
from sklearn.svm import LinearSVC

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestRegressor
# read in the json files
portfolio = pd.read_json('data/portfolio.json', orient='records', lines=True)
profile = pd.read_json('data/profile.json', orient='records', lines=True)
transcript = pd.read_json('data/transcript.json', orient='records', lines=True)_____no_output_____

### Viewing data before any cleaning
*All the data checking is made on profile, portfolio and  transcript datasets*
* Check on columns types and data types
* Check on numerical values
* Check on null rows
* Check ids used in offer types
* Plot histogram of income amounts
* Plot gender transactions
* Check on extremes in data
_____no_output_____

portfolio.head()_____no_output_____len(portfolio)_____no_output_____portfolio.describe()_____no_output_____np.sum(portfolio.isna())_____no_output_____portfolio.groupby('offer_type')['id'].value_counts()_____no_output_____profile.head()_____no_output_____profile.describe()_____no_output_____len(profile)_____no_output_____np.sum(profile.isna())_____no_output_____profile['id'].nunique()_____no_output_____profile['gender'].value_counts()_____no_output_____profile.groupby('gender')['gender'].value_counts().sort_values(ascending=False).plot(kind='bar')_____no_output_____profile['income'].hist()_____no_output_____profile[profile['age']==118].count()_____no_output_____profile[(profile['gender']=='M')&(profile['age']!=118)]['income'].plot(kind='hist')_____no_output_____profile[(profile['gender']=='F')&(profile['age']!=118)]['income'].plot(kind='hist')_____no_output_____

## Transcript dataset viewing
_____no_output_____

transcript.head()_____no_output_____transcript.describe()_____no_output_____transcript.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 306534 entries, 0 to 306533
Data columns (total 4 columns):
event     306534 non-null object
person    306534 non-null object
time      306534 non-null int64
value     306534 non-null object
dtypes: int64(1), object(3)
memory usage: 9.4+ MB
transcript.isna().sum()_____no_output_____transcript['event'].value_counts()_____no_output_____df_transaction = transcript[transcript['event'] == 'transaction']_____no_output_____df_transaction.head()_____no_output_____

# Starting data cleaning
### Protfolio dataset:
* Converting duration columns from days to hours
* Use MinMaxScaler on difficulty and duration columns
* Create 4 new columns based on channels used in the offer,  1 if the channels columns else 0
* Create a map for offer_id it could be done with two methods first with dicts, the final one with LabelEncoder
_____no_output_____

portfolio_cleaned=portfolio.copy()_____no_output_____portfolio_cleaned.head()_____no_output_____portfolio_cleaned['duration']=portfolio_cleaned['duration']*24_____no_output_____portfolio_cleaned.head()_____no_output_____minmax_scaler=MinMaxScaler()
portfolio_cleaned[['difficulty','reward']]=minmax_scaler.fit_transform(portfolio_cleaned[['difficulty','reward']])
portfolio_cleaned.head()_____no_output_____portfolio_cleaned['channel_email']=portfolio_cleaned['channels'].apply(lambda x:1 if 'email' in x else 0)_____no_output_____portfolio_cleaned['channel_web']=portfolio_cleaned['channels'].apply(lambda x:1 if 'web' in x else 0)_____no_output_____portfolio_cleaned['channel_mobile']=portfolio_cleaned['channels'].apply(lambda x:1 if 'mobile' in x else 0)_____no_output_____portfolio_cleaned['channel_social']=portfolio_cleaned['channels'].apply(lambda x:1 if 'social' in x else 0)_____no_output_____portfolio_cleaned.drop('channels',inplace=True,axis=1)
portfolio_cleaned.head()_____no_output_____portfolio_cleaned.rename(columns={'id':'offer_id'},inplace=True)
_____no_output_____labels_offer_id = portfolio_cleaned['offer_id'].astype('category').cat.categories.tolist()
replace_map_comp_offer_id = {'offer_id' : {k: v for k,v in zip(labels_offer_id,list(range(1,len(labels_offer_id)+1)))}}_____no_output_____replace_map_comp_offer_id_____no_output_____portfolio_cleaned.replace(replace_map_comp_offer_id, inplace=True)
portfolio_cleaned.head()_____no_output_____labels_offer_type = portfolio_cleaned['offer_type'].astype('category').cat.categories.tolist()
replace_map_comp_offer_type = {'offer_type' : {k: v for k,v in zip(labels_offer_type,list(range(1,len(labels_offer_type)+1)))}}
_____no_output_____portfolio_cleaned.replace(replace_map_comp_offer_type, inplace=True)_____no_output_____replace_map_comp_offer_type_____no_output_____portfolio_cleaned.head()_____no_output_____portfolio_cleaned.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10 entries, 0 to 9
Data columns (total 9 columns):
difficulty        10 non-null float64
duration          10 non-null int64
offer_id          10 non-null int64
offer_type        10 non-null int64
reward            10 non-null float64
channel_email     10 non-null int64
channel_web       10 non-null int64
channel_mobile    10 non-null int64
channel_social    10 non-null int64
dtypes: float64(2), int64(7)
memory usage: 800.0 bytes
portfolio_cleaned.rename(columns={'id':'offer_id'},inplace=True)_____no_output_____

### Profile dataset
* Rename id to cust_id to help in joining process later
* Clean age 118 values
* Create bins for each age group
* Create bins for each income range
* Create new features from date column as Year,Month,Week,Day,Dayofweek,Dayofyear,Is_month_end,Is_month_start,Is_quarter_end,Is_quarter_start,Is_year_end and Is_year_start
* Calculate how long the user is a member of starbucks
* Remove containing null rows for age 118

_____no_output_____

profile_cleaned = profile.copy()_____no_output_____profile_cleaned.head()_____no_output_____profile_cleaned.rename(columns={'id':'cust_id'},inplace=True)_____no_output_____profile_cleaned.columns_____no_output_____profile_cleaned=profile_cleaned.reindex(columns=['cust_id','age','income','became_member_on','gender'])_____no_output_____profile_cleaned.head()_____no_output_____labels_customer=profile_cleaned['cust_id'].astype('category').cat.categories.tolist()

replace_map_label_customer = {'cust_id':{k: v for k,v in zip(labels_customer,list(range(1,len(labels_customer)+1)))}} _____no_output_____profile_cleaned.replace(replace_map_label_customer,inplace=True)_____no_output_____profile_cleaned.head()_____no_output_____profile_cleaned['age']=profile_cleaned['age'].apply(lambda x: np.nan if x==118 else x)_____no_output_____profile_cleaned.head()_____no_output_____profile_cleaned[profile_cleaned['age']==118]_____no_output_____profile_cleaned.dropna(axis=0,how='any',inplace=True)_____no_output_____profile_cleaned[['age','income']]=profile_cleaned[['age','income']].astype(int)_____no_output_____profile_cleaned_____no_output_____profile_cleaned['age_group']=pd.cut(profile_cleaned['age'],bins=[17, 22, 35, 60, 103],
                                    labels=['Teenager','Youth','Adult','Elderly'])
profile_cleaned['age_group'].head()_____no_output_____labels_age_grp=profile_cleaned['age_group'].astype('category').cat.categories.tolist()
replace_map_age_grp={'age_group':{k:v for k,v in zip(labels_age_grp,list(range(1,len(labels_age_grp)+1)))}}_____no_output_____replace_map_age_grp_____no_output_____profile_cleaned.replace(replace_map_age_grp,inplace=True)_____no_output_____profile_cleaned.head()_____no_output_____profile_cleaned['income_range'] = pd.cut(profile_cleaned['income'], bins=[29999, 60000, 90000, 120001],
                                         labels=['average', 'above-average', 'high'])_____no_output_____labels_income_range = profile_cleaned['income_range'].astype('category').cat.categories.tolist()
replace_map_comp_income_range = {'income_range' : {k: v for k,v in zip(labels_income_range,
                                                                       list(range(1,len(labels_income_range)+1)))}}_____no_output_____replace_map_comp_income_range_____no_output_____profile_cleaned['income_range'].value_counts()_____no_output_____profile_cleaned.replace(replace_map_comp_income_range,inplace=True)_____no_output_____profile_cleaned.head()_____no_output_____le=LabelEncoder()
_____no_output_____profile_cleaned['gender_cleaned']=le.fit_transform(profile_cleaned['gender'])
np.save('classes_gender.npy', le.classes_)_____no_output_____profile_cleaned.head()_____no_output_____profile_cleaned['became_member_on']=pd.to_datetime(profile_cleaned['became_member_on'],format='%Y%m%d')_____no_output_____profile_cleaned.head()_____no_output_____import re
def add_datepart(df, fldname, drop=True):
    '''
    INPUT:
        df DataFrame : the containg datafframe that will add new date features to it
        dlfname (str): string containg date column that will extract features from
        drop (bool) : drop date feature after cleaning dataset
    OUTPUT:
        the input df with new date feature as year star, day of week etc.
    '''
    fld = df[fldname]
    if not np.issubdtype(fld.dtype, np.datetime64):
        df[fldname] = fld = pd.to_datetime(fld, infer_datetime_format=True)
    targ_pre = re.sub('[Dd]ate$', '', fldname)
    for n in ('Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Dayofyear',
            'Is_month_end', 'Is_month_start', 'Is_quarter_end', 'Is_quarter_start', 'Is_year_end', 'Is_year_start'):
        df[targ_pre+n] = getattr(fld.dt,n.lower())
    df[targ_pre+'Elapsed'] = fld.astype(np.int64) // 10**9
    if drop: df.drop(fldname, axis=1, inplace=True)
add_datepart(profile_cleaned,'became_member_on',drop=False)
profile_cleaned.head()_____no_output_____import datetime
profile_cleaned['member_ship_days']=datetime.datetime.today().date() - profile_cleaned['became_member_on'].dt.date_____no_output_____profile_cleaned.head()_____no_output_____profile_cleaned['member_ship_days']=profile_cleaned['member_ship_days'].dt.days_____no_output_____profile_cleaned.head()_____no_output_____profile_cleaned['member_type'] = pd.cut(profile_cleaned['member_ship_days'],
                                        bins=[390, 1000, 1600, 2500],labels=['new', 'regular', 'loyal'])_____no_output_____

# replacing the 'member_type' categorical labels by numerical labels
labels_member_type = profile_cleaned['member_type'].astype('category').cat.categories.tolist()
replace_map_comp_member_type = {'member_type' : {k: v for k,v in zip(labels_member_type,list(range(1,len(labels_member_type)+1)))}}

_____no_output_____replace_map_comp_member_type_____no_output_____profile_cleaned.replace(replace_map_comp_member_type, inplace=True)_____no_output_____profile_cleaned.head()_____no_output_____profile_cleaned.drop(columns = ['age','income','became_member_on', 'member_ship_days','became_member_onElapsed'], axis=1, inplace=True)_____no_output_____profile_cleaned.head()_____no_output_____profile_cleaned.drop(columns = ['gender'], axis=1, inplace=True)_____no_output_____

### Transcript Dataset

* First creat a copy of original dataset
* Replace column name person with cust_id for mergeing process
* Iterate over keys of value column to splir between offer_id, offer id, amount ,reward
* Create 3 new columns based on amount and offer_id and reward
* Transfrom event column and save it's transformation
* Remove reward and transction values from event column as we will develop model to predict if the user will view offer or consume offer
* Transform offer values

_____no_output_____

transcript_cleaned=transcript.copy()_____no_output_____transcript_cleaned.head()_____no_output_____transcript_cleaned.rename(columns={'person':'cust_id'},inplace=True)_____no_output_____transcript_cleaned.replace(replace_map_label_customer,inplace=True)_____no_output_____transcript_cleaned.head()_____no_output_____keys=[]

for idx,row in transcript_cleaned.iterrows():
    for k in row['value']:
        if k in keys:
            continue
        else:
            keys.append(k)
            _____no_output_____keys_____no_output_____transcript_cleaned['offer_id']=''
transcript_cleaned['amount']=0
transcript_cleaned['reward']=0_____no_output_____for idx,row in transcript_cleaned.iterrows():
    for j in row['value']:
        if j=='offer id' or j=='offer_id':
            transcript_cleaned.at[idx,'offer_id']=row['value'][j]

        elif j=='amount':
            transcript_cleaned.at[idx,'amount']=row['value'][j]

        elif j=='reward':
            transcript_cleaned.at[idx,'reward']=row['value'][j]

            _____no_output_____transcript_cleaned['offer_id']=transcript_cleaned['offer_id'].apply(lambda x: 'N/A' if x=='' else x)_____no_output_____transcript_cleaned.head()_____no_output_____transcript_cleaned.drop('value',axis=1,inplace=True)_____no_output_____transcript_cleaned.groupby('offer_id').count()_____no_output_____transcript_cleaned['event'].value_counts()_____no_output_____transcript_cleaned=transcript_cleaned[transcript_cleaned['event']!='transaction']
transcript_cleaned=transcript_cleaned[transcript_cleaned['event']!='offer received']
_____no_output_____transcript_cleaned['event'].value_counts()_____no_output_____transcript_cleaned['event_cleaned']=le.fit_transform(transcript_cleaned['event'])
np.save('classes_event.npy', le.classes_)_____no_output_____transcript_cleaned.head()_____no_output_____transcript_cleaned.replace(replace_map_comp_offer_id,inplace=True)_____no_output_____transcript_cleaned.head()_____no_output_____

### Merged Dataframe
* Create a new df merged between transcript and portfolio based on offer_id
* Merge the new df with profile to get user data
* Check for null values
* Remove null values
* Transform event as it will be our target and save values from this transformation
* Data visualization plot count of transction based on income , gender, offer type and year of joining
* Transform age-group column and save it for later use
* Calculate how many offer compeleted for females and males
* Generate predictive model to predict if the user will complete an offer or will just view it_____no_output_____

merged_df=transcript_cleaned.merge(portfolio_cleaned,how='left',on='offer_id')_____no_output_____merged_df=merged_df.merge(profile_cleaned,how='left',on='cust_id')_____no_output_____merged_df.head()_____no_output_____merged_df.info()<class 'pandas.core.frame.DataFrame'>
Int64Index: 91304 entries, 0 to 91303
Data columns (total 31 columns):
event                               91304 non-null object
cust_id                             91304 non-null int64
time                                91304 non-null int64
offer_id                            91304 non-null int64
amount                              91304 non-null int64
reward_x                            91304 non-null int64
event_cleaned                       91304 non-null int64
difficulty                          91304 non-null float64
duration                            91304 non-null int64
offer_type                          91304 non-null int64
reward_y                            91304 non-null float64
channel_email                       91304 non-null int64
channel_web                         91304 non-null int64
channel_mobile                      91304 non-null int64
channel_social                      91304 non-null int64
age_group                           82304 non-null float64
income_range                        82304 non-null float64
gender_cleaned                      82304 non-null float64
became_member_onYear                82304 non-null float64
became_member_onMonth               82304 non-null float64
became_member_onWeek                82304 non-null float64
became_member_onDay                 82304 non-null float64
became_member_onDayofweek           82304 non-null float64
became_member_onDayofyear           82304 non-null float64
became_member_onIs_month_end        82304 non-null object
became_member_onIs_month_start      82304 non-null object
became_member_onIs_quarter_end      82304 non-null object
became_member_onIs_quarter_start    82304 non-null object
became_member_onIs_year_end         82304 non-null object
became_member_onIs_year_start       82304 non-null object
member_type                         82304 non-null float64
dtypes: float64(12), int64(12), object(7)
memory usage: 22.3+ MB
np.sum(merged_df.isna())_____no_output_____merged_df.dropna(how='any',axis=0,inplace=True)_____no_output_____merged_df.groupby(['offer_id','event'])['difficulty'].head()_____no_output_____np.sum(merged_df.isna())_____no_output_____len(merged_df)_____no_output_____le.classes_=np.load('classes_event.npy')_____no_output_____le.classes______no_output_____le.transform(['offer completed','offer viewed'])_____no_output_____{'offer_type': {'bogo': 1, 'discount': 2, 'informational': 3}}_____no_output_____merged_df['event_cleaned']=merged_df['event_cleaned'].map({0:'offer completed',1:'offer viewed'})
merged_df['offer_type']=merged_df['offer_type'].map( {1: 'bogo', 2: 'discount',3: 'informational'})
merged_df['income_range'] = merged_df['income_range'].map({1: 'average', 2: 'above-average', 3:'high'})
merged_df['age_group'] = merged_df['age_group'].map({1: 'teenager', 2: 'young-adult', 3:'adult', 4:'elderly'})_____no_output_____merged_df.head()_____no_output_____import seaborn as sns
import matplotlib.pyplot as plt_____no_output_____plt.figure(figsize=(10,10))
g = sns.countplot(x="age_group", hue="offer_type", data=merged_df)
plt.title('Most Popular Offers to Each Age Group')
plt.ylabel('Total')
plt.xlabel('Age Group')
xlabels = ['teenager','young-adult','adult','elderly']
g.set_xticklabels(xlabels)
plt.xticks(rotation = 0)
plt.legend(title='Offer Type')
plt.show();_____no_output_____le.classes_=np.load('classes_gender.npy')
le.classes_
_____no_output_____plt.figure(figsize=(10,10))
pt=sns.countplot(x='gender_cleaned',hue='income_range',data=merged_df[merged_df['gender_cleaned']!=2])
plt.ylabel('Income')
xlabels=['Female','Male']
pt.set_xticklabels(xlabels)
plt.xlabel('Gender')
plt.xticks(rotation=0)
plt.show()_____no_output_____plt.figure(figsize=(12,5))

pt=sns.countplot(x='gender_cleaned',hue='event',data=merged_df[merged_df['gender_cleaned']!=2])
plt.title('Most Popular Offer to consume based on each gender')
plt.ylabel('Total')
xlabels=['Female','Male']
plt.xlabel('Gender')
pt.set_xticklabels(xlabels)
plt.legend()
plt.show()

_____no_output_____avg_offer_consumed_gender=merged_df.groupby(['gender_cleaned','offer_id'])['time'].mean().reset_index()
print(avg_offer_consumed_gender[avg_offer_consumed_gender['gender_cleaned']==0]['time'].values[0],avg_offer_consumed_gender[avg_offer_consumed_gender['gender_cleaned']==0].values[0]/24)
print(avg_offer_consumed_gender[avg_offer_consumed_gender['gender_cleaned']==1]['time'].values[0],avg_offer_consumed_gender[avg_offer_consumed_gender['gender_cleaned']==1].values[0]/24)406.024581006 [  0.           0.04166667  16.91769088]
405.108252947 [  0.04166667   0.04166667  16.87951054]
merged_df['became_member_onYear']=merged_df['became_member_onYear'].astype(int)_____no_output_____plt.figure(figsize=(14, 5))
sns.countplot(merged_df['became_member_onYear'])
plt.title('Number of Profiles In Each Year')
plt.ylabel('Number of Profiles')
plt.xlabel('Year')
plt.xticks()
plt.show()_____no_output_____(merged_df['became_member_onYear'].value_counts()/len(merged_df['became_member_onYear']))*100_____no_output_____plt.figure(figsize=(14,5))
pt=sns.countplot(x='gender_cleaned',hue='offer_type',data=merged_df[merged_df['gender_cleaned']!=2])
plt.xlabel('Gender')
plt.ylabel('Count')
pt.set_xticklabels(['Female','Male'])
plt.legend(title='Offer Type')
plt.show()_____no_output_____trans_avg_time=merged_df.groupby(['cust_id','offer_id'])['time'].mean().reset_index()
trans_avg_time['time'].mean(),trans_avg_time['time'].mean()/24_____no_output_____total_offers_types=merged_df[merged_df['gender_cleaned']!=2].groupby(['gender_cleaned','offer_type']).count()
total_offers_consumed=merged_df[merged_df['gender_cleaned']!=2].groupby(['gender_cleaned','event']).count()

total_offers_consumed_____no_output_____print('Females Offers Completed from offer viewed:',
      total_offers_consumed.loc[0].loc['offer completed'].values[0]
      /(total_offers_consumed.loc[0].loc['offer viewed'].values[0])*100,'%')Females Offers Completed from offer viewed: 74.4587703262 %
print('Males Offers Completed from offer viewed:',
      total_offers_consumed.loc[1].loc['offer completed'].values[0]/
      (total_offers_consumed.loc[1].loc['offer viewed'].values[0])*100,'%')Males Offers Completed from offer viewed: 58.1816896929 %
merged_df['event']_____no_output_____labels_event_mrgd = merged_df['event'].astype('category').cat.categories.tolist()
#generate offers to users_____no_output_____labels_event1 = merged_df['event'].astype('category').cat.categories.tolist()
replace_map_comp_event1 = {'event' : {k: v for k,v in zip(labels_event1,list(range(1,len(labels_event1)+1)))}}

labels_income1 = merged_df['income_range'].astype('category').cat.categories.tolist()
replace_map_comp_income_range1 = {'income_range' : {k: v for k,v in zip(labels_income1,list(range(1,len(labels_income1)+1)))}}

labels_offer_type1 = merged_df['offer_type'].astype('category').cat.categories.tolist()
replace_map_comp_offer_type1 = {'offer_type' : {k: v for k,v in zip(labels_offer_type1,list(range(1,len(labels_offer_type1)+1)))}}
_____no_output_____merged_df.replace(replace_map_comp_event1, inplace=True)
merged_df.replace(replace_map_comp_offer_type1, inplace=True)
merged_df.replace(replace_map_comp_income_range1, inplace=True)
_____no_output_____merged_df['age_group']=le.fit_transform(merged_df['age_group'])

np.save('age_cleaned.npy',le.classes_)_____no_output_____merged_df.columns_____no_output_____x=merged_df[['time', 'offer_id', 'amount', 'reward_x',
        'difficulty', 'duration', 'offer_type', 'reward_y',
       'channel_email', 'channel_web', 'channel_mobile', 'channel_social',
       'age_group', 'income_range', 'gender_cleaned', 'became_member_onYear',
       'became_member_onMonth', 'became_member_onWeek', 'became_member_onDay',
       'became_member_onDayofweek', 'became_member_onDayofyear',
       'became_member_onIs_month_end', 'became_member_onIs_month_start',
       'became_member_onIs_quarter_end', 'became_member_onIs_quarter_start',
       'became_member_onIs_year_end', 'became_member_onIs_year_start',
       'member_type']]

y=merged_df[['event']]_____no_output_____pd.set_option('display.max_columns',None)
x.head()
np.sum(x.isna())
_____no_output_____X_train, X_test, y_train, y_test = train_test_split(x, y, random_state=42)_____no_output_____clf = RandomForestClassifier()

param_dist = {'n_estimators': [50, 100, 200],
                  'criterion': ['gini', 'entropy'],
                  'max_depth': [5, 7, 10, 15, None],
                  'min_samples_split': [2, 3, 5, 7]
                 }

grid_clf = GridSearchCV(clf, param_dist, cv=5, return_train_score=False)

grid_clf.fit(X_train, y_train)
preds=grid_clf.predict(X_test)
print(f'Accuracy of Random Forest classifier on testing set: {round(grid_clf.score(X_test, y_test)*100,2)}%.')
print(classification_report(y_test,preds))/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)

sv_clf=SVC()

sv_clf.fit(X_train,y_train)
preds=sv_clf.predict(X_test)
classification_report(y_test,preds)

/opt/conda/lib/python3.6/site-packages/sklearn/utils/validation.py:578: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)
sv_clf_lin=LinearSVC()

sv_clf_lin.fit(X_train,y_train)
preds=sv_clf_lin.predict(X_test)
classification_report(y_test,preds)
/opt/conda/lib/python3.6/site-packages/sklearn/utils/validation.py:578: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)

### Second model
* Create a new df containg empty columns to calculate from it total_trans, trans_with_off and trans_num
* Iterate over cust_id and calulate the value for each new column
* Use get_trans_recs() to pull user data
* Add 1 to the related column, once there is an offer received, offer viewed and offer completed for the same offer with type BOGO or discount
* Add 0 to the related column, if all offers received, offer viewed are not completed at the end for the same offer with type BOGO or discount
* Add 1 to the related column, once there is an offer received, offer viewed and transaction happened within valid period of an offer with offer type informational
* Add 0 to the related column, if all offer received, offer viewed and no transaction happened within valid period of an offer with offer type informational
* Save new dataframe
* Train models_____no_output_____

from collections import defaultdict
_____no_output_____portfolio = pd.read_json('data/portfolio.json', orient='records', lines=True)
profile = pd.read_json('data/profile.json', orient='records', lines=True)
transcript = pd.read_json('data/transcript.json', orient='records', lines=True)_____no_output_____cols = portfolio.id.values
profile_trans = pd.DataFrame(np.nan, index = profile_cleaned.index.values, columns = cols)
cols_2 = ['trans_with_off','trans_num','trans_tot']
profile_trans2 = pd.DataFrame(np.zeros((profile.shape[0], 3)), index = profile.index.values, columns = cols_2)
profile_trans = pd.concat([profile_trans, profile_trans2], axis = 1)
profile_trans.head()_____no_output_____bogo_disc_offer_ids = portfolio[portfolio.offer_type.isin(['bogo', 'discount'])].id.values
bogo_disc_offer_ids_____no_output_____# function to pull each customer's transcripts
def get_trans_recs(customer_id):
    '''get personal transaction records in transcript data with personal id from profile data
    INPUT:
        customer_id(in): id from profile id

    OUTPUT:
        trans_recs(DataFrame): Personal transaction record
    '''
    trans_recs = transcript_cleaned[transcript_cleaned['cust_id'] == customer_id].reset_index(drop=True)
    return trans_recs_____no_output_____profile.head()
profile.shape[0]_____no_output_____transcript_cleaned.head()_____no_output_____profile_cleaned.head()_____no_output_____tran_profile=transcript_cleaned.merge(profile_cleaned,how='left',on='cust_id')_____no_output_____tran_profile.head()_____no_output_____recs=get_trans_recs(3729).head()_____no_output_____recs['amount'].sum()
for offer in recs['event']:
    if offer =='offer completed':
        print(1)
    else:
        print(0)0
0
0
1
0
cols = portfolio.id.values
profile_trans = pd.DataFrame(np.nan, index = profile.index.values, columns = cols)
profile_trans2 = pd.DataFrame(np.zeros((profile.shape[0], 3)), index = profile.index.values, columns = cols_2)
profile_trans=pd.concat([profile_trans,profile_trans2],axis=1)
profile_trans.head()_____no_output_____curr_ind = 0
for cus_ind in profile_cleaned['cust_id']:
    trans_recs=get_trans_recs(cus_ind)
    off_comp_times = defaultdict(int) # record times of offer complete after viewed
    off_view_times = defaultdict(int) # record times of offer viewed before complete
    receive_idx = trans_recs[trans_recs.event == 'offer received'].index.values # save all offer received event index
    trans_row = trans_recs[trans_recs.event == 'transaction'] # save all transaction events
    profile_trans.loc[curr_ind, 'trans_num'] = trans_row.shape[0] # save the number of transactions
    profile_trans.loc[curr_ind, 'trans_tot'] = trans_row.amount.sum() # save the total transaction value
    trans_offer_idx = []
    for receive_ind  in receive_idx:
        off_id=trans_recs.loc[receive_ind,'offer_id']
        off_hours=int(portfolio[portfolio['id']==off_id].duration)*24
        off_start = trans_recs.loc[receive_ind, 'time']
        off_end = off_start + off_hours
        view_row=trans_recs[(trans_recs['offer_id']==off_id)&
                           (trans_recs['event']=='offer viewed')
                           &(trans_recs['time']>=off_start)&(trans_recs['time']<=off_end)]
        if view_row.shape[0] < 1:
            continue
        else:
            if off_id in bogo_disc_offer_ids:
                comp_off=trans_recs[(trans_recs['offer_id']==off_id)&
                           (trans_recs['event']=='offer completed')
                           &(trans_recs['time']>=off_start)&(trans_recs['time']<=off_end)]
                if comp_off.shape[0]<1:
                    off_view_times[off_id] +=1
                    off_comp_times[off_id] +=0
                else:
                    view_ind = view_row.index[0]
                    comp_ind = comp_row.index[0]

                    if view_ind < comp_ind: # complete after view validly count
                        off_view_times[off_id] += 1
                        off_comp_times[off_id] += 1 # viewed and completed the offer
                        trans_offer_idx.extend(trans_row.loc[view_ind:comp_ind].index.values)

                    else: # complete before view not count
                        continue
            else: # Informational offer
                view_time = view_row.iloc[0].time
                off_view_times[off_id] += 1
                info_trans_row = trans_recs[(trans_recs.time >= view_time) &
                                            (trans_recs.time <= off_end) &
                                            (trans_recs.event == 'transaction')]

                if info_trans_row.shape[0] > 0: # if there is transaction after view and before offer duration
                    off_comp_times[off_id] += 1 # effective offer
                else:
                    off_comp_times[off_id] += 0 # ineffective offer

    for i in off_view_times.keys():
        # calculate effective rate of the offer on that customer
        profile_trans.loc[curr_ind, i] = round(off_comp_times[i] / off_view_times[i], 2)
    if len(trans_offer_idx) > 0: # calculate the sum of transaction driven by offer
        profile_trans.loc[curr_ind, 'trans_with_off'] = trans_row.loc[list(set(trans_offer_idx))].amount.sum()
    curr_ind+=1
_____no_output_____profile_trans.head()_____no_output_____profile_trans.to_csv('profile_trans.csv')_____no_output_____profile_trans = pd.read_csv('profile_trans.csv', index_col = 'Unnamed: 0')_____no_output_____profile_trans['trans_avg'] = round(profile_trans['trans_tot'] / (profile_trans['trans_num']+0.001), 2)
profile_trans['off_drive_ratio'] = round(profile_trans['trans_with_off'] / (profile_trans['trans_tot']+0.001), 2)
profile_trans.drop(['trans_with_off', 'trans_num'], axis=1, inplace=True)

# keep gender in is not for training but for further illustration with figure
profile_trans = pd.concat([profile.loc[:, ['age', 'income', 'gender']], profile_trans], axis=1)
profile_trans.columns = ['age', 'income', 'gender', 'offer_1', 'offer_2', \
                         'offer_3', 'offer_4', 'offer_5', 'offer_6', 'offer_7', 'offer_8',\
                         'offer_9', 'offer_10', 'trans_tot', 'trans_avg', 'off_drive_ratio']_____no_output_____profile_trans.head()_____no_output_____profile_trans=profile_trans[profile_trans['age']!=118]_____no_output_____profile_trans.iloc[0:5,-1]_____no_output_____le= LabelEncoder()
le.classes_=np.load('classes_gender.npy')_____no_output_____le.classes______no_output_____from sklearn.ensemble import RandomForestClassifier_____no_output_____profile_trans['gender']=le.transform(profile_trans['gender'])_____no_output_____def classify_train_data(test_col, profile_trans = profile_trans):
    '''filter profile_trans data of customer with 1 or 0 score under dedicated offer id
    INPUT:
        test_col(str) - test data column name
        profile_trans(pandas DataFrame) - profile_trans dataframe
    OUTPUT:
        X(pandas DataFrame) - training data features
        y(pandas DataFrame) - training data label
    '''
    train_data = profile_trans[(profile_trans.loc[:,test_col] == 1) | \
                                  (profile_trans.loc[:,test_col] == 0)].loc[:, ['age','income','gender']]
    X, y = train_data.iloc[:,:3], train_data.iloc[:,-1]
    return X, y

def classify_grid_search(X, y):
    '''grid search and cross validation with estimator of random forest classifier
    INPUT:
        X(pandas DataFrame) - training data features
        y(pandas DataFrame) - training data label
    OUTPUT:
        grid_clf(sklearn model instance) - trained model
    '''
    clf = RandomForestClassifier()

    param_dist = {'n_estimators': [50, 100, 200],
                  'criterion': ['gini', 'entropy'],
                  'max_depth': [5, 7, 10, 15, None],
                  'min_samples_split': [2, 3, 5, 7]
                 }

    grid_clf = GridSearchCV(clf, param_dist,refit = 'F1', cv=5, return_train_score=False)
    grid_clf.fit(X, y)

    return grid_clf_____no_output_____from sklearn.metrics import classification_report
clfs=[]
offer_list = profile_trans.columns[5:15]
for i, offer_num in enumerate(offer_list):

    X, y = classify_train_data(offer_num)
    x_train,x_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=42)
    grid_clf = classify_grid_search(X, y)
    clfs.append(grid_clf)
    preds=grid_clf.predict(x_test)
    print(classification_report(y_test,preds))

             precision    recall  f1-score   support

          0       1.00      1.00      1.00       187
          1       1.00      1.00      1.00       271
          2       1.00      1.00      1.00        12

avg / total       1.00      1.00      1.00       470

             precision    recall  f1-score   support

          0       1.00      1.00      1.00        85
          1       1.00      1.00      1.00       120
          2       1.00      1.00      1.00         4

avg / total       1.00      1.00      1.00       209

             precision    recall  f1-score   support

          0       1.00      1.00      1.00        72
          1       1.00      1.00      1.00        87
          2       1.00      1.00      1.00         3

avg / total       1.00      1.00      1.00       162

             precision    recall  f1-score   support

          0       1.00      1.00      1.00       180
          1       1.00      1.00      1.00       211
          2       1.00      1.00      1.00         5

avg / total       1.00      1.00      1.00       396

             precision    recall  f1-score   support

          0       1.00      1.00      1.00       171
          1       1.00      1.00      1.00       232
          2       1.00      1.00      1.00         7

avg / total       1.00      1.00      1.00       410

             precision    recall  f1-score   support

          0       1.00      1.00      1.00       305
          1       1.00      1.00      1.00       432
          2       1.00      1.00      1.00        15

avg / total       1.00      1.00      1.00       752

             precision    recall  f1-score   support

          0       1.00      1.00      1.00       189
          1       1.00      1.00      1.00       275
          2       1.00      1.00      1.00         8

avg / total       1.00      1.00      1.00       472

             precision    recall  f1-score   support

          0       1.00      1.00      1.00        90
          1       1.00      1.00      1.00       115
          2       1.00      1.00      1.00         7

avg / total       1.00      1.00      1.00       212

             precision    recall  f1-score   support

          0       1.00      1.00      1.00       170
          1       1.00      1.00      1.00       270
          2       1.00      1.00      1.00         3

avg / total       1.00      1.00      1.00       443

             precision    recall  f1-score   support

          0       1.00      1.00      1.00       206
          1       1.00      1.00      1.00       251
          2       1.00      1.00      1.00         1

avg / total       1.00      1.00      1.00       458

def trans_personal_info_to_list(age, income, gender):
    '''Convert personal info to needed data shape for model predict
    INPUT:
        age(int): age between 18 to 101
        income(float): income between 30000 to 120000
        gender(str): 2 for female, 1 for male, 2 for other
    OUTPUT:
        person_info(numpy array): data structure fit to model predict
    '''

    p = [age, income]
    p.extend([gender])
    person_info = np.array(p).reshape(1,-1)
    return person_info_____no_output_____

def pred_top_n_offer_with_person_info(age, income, gender, n=10, clfs=clfs):
    '''predict probability and return top n appropriate offer type with personal info
    INPUT:
        age(int): age between 18 to 101
        income(float): income between 30000 to 120000
        gender(str): 'F' for female, 'M' for male, 'O' for other
        n(int): the number of top appropriate offer id in the return, from 1 to 10, default 10
        clfs(sklearn models): classifiers used to predict each offer's probability
    OUTPUT:
        top_n(list): top n offer ids with highest probability
    '''
    person_info = trans_personal_info_to_list(age, income, gender)
    pred_probs = []
    for i in range(10):
        prob = clfs[i].predict_proba(person_info)[0][1]
        pred_probs.append(prob)

    offer_prob = pd.Series(pred_probs, index=portfolio.id)
    if n > 10:
        n = 10

    sorted_offer_prob = offer_prob.sort_values(ascending=False)
    top_n = sorted_offer_prob.index.values[:n]
    return top_n

def pred_top_n_offer_with_cus_id(cus_id, clfs=clfs, n=10, profile_trans=profile_trans):
    '''
    predict probability and return top n appropriate offer type with customer id
    INPUT:
        cus_id(str): customer id in profile_trans dataframe
        n(int): the number of top appropriate offer id in the return, default 10
        profile_trans(pandas DataFrame): used to get the customer's personal info
    OUTPUT:
        top_n(list): top n offer ids with highest probability
    '''
    if cus_id in profile_trans.index:
        age, income, gender = profile_trans.loc[cus_id][0:3]
        top_n = pred_top_n_offer_with_person_info(age, income, gender, n)
        return top_n
    else:
        print('Sorry no info for this user')
_____no_output_____top_3 = pred_top_n_offer_with_person_info(55, 39,gender=0, n=3, clfs=clfs)
top_3_____no_output_____x_train_reg,x_test_reg,y_train_reg,y_test_reg=train_test_split(profile_trans.loc[:,['age','income','gender'
                                                                                ]],profile_trans.loc[:,['trans_avg']],
                                                               test_size=0.2)
_____no_output_____param_grid={
                    'n_estimators': [50, 100, 200],
                  'max_depth': [5, 7, 10, None],
                  'min_samples_split': [2, 5, 7, 10]

}

regressor=RandomForestRegressor()
cv=GridSearchCV(regressor,param_grid,n_jobs=-1)
cv.fit(x_train_reg,y_train_reg)

preds=cv.predict(x_test_reg)

/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
/opt/conda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:458: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  estimator.fit(X_train, y_train, **fit_params)
def pred_avg_ratio(age,income,gender,rgsr=cv):
    '''
    INPUT:
        age (int) :  age of the user between 18 and 101
        income (int) :  income of the user between 30000 to 120000
        gender (int) :  0 if user is Female 1 if the user is Male 2 if the user is Other
    OUTPUT:
        avg, ratio of user transactions

    '''
    user_info = trans_personal_info_to_list(age,income,gender)
    pred_results=[]
    pred=round(rgsr.predict(user_info)[0],2)
    pred_results.append(pred)
    return pred_results

def search_avg_ratio(cust_id,profile_trans=profile_trans):
    '''
    INPUT:
        CUST_ID, profile dataframe
    OUTPUT:
        avg ratio, ratio
    '''
    if cust_id in profile_trans.index:
        avg,ratio=profile_trans.loc[cust_id][-2:].values
        return [avg,ratio]
    else:
        print('Sorry no info for this user')

    _____no_output_____profile=profile[profile['age']!=118]
profile.head()_____no_output_____m_68_70000 = pred_avg_ratio(68, 70000,1)
m_68_70000_____no_output_____

### Finally combine two models
_____no_output_____

def offer_suggest(age,income, gender,n=5):
    '''
    INPUT:
        age (int) :  age of the user between 18 and 101
        income (int) :  income of the user between 30000 to 120000
        gender (int) :  0 if user is Female 1 if the user is Male 2 if the user is Other
        n (int) : number of offers to generate
    Output:
        offers : dataframe for the suggested offers based on user
        preds (list): customer transaction amount and offer drive ratio
    '''
    top_n = pred_top_n_offer_with_person_info(age,income,gender,n)
    offer = portfolio.set_index('id',drop=True).loc[top_n]
    preds = pred_avg_ratio(age,income,gender)

    return offer,preds
def offer_suggest_with_cus_id(cust_id,n=5):
    '''
    INPUT:
        cust_id (int) : User id transformed
        n (int) : number of offers to generate
    Output:
        offers : dataframe for the suggested offers based on user
    '''
    if cust_id in profile_trans.index:
        top_n = pred_top_n_offer_with_person_info(cust_id,n)
        offer = portfolio.set_index('id',drop=True).loc[top_n]
        preds = search_avg_ratio(cust_id)
        return offer , preds

    else:
        print('Sorry no info for this user')

    _____no_output_____offers,preds = offer_suggest(68,70000,1,n=5)
offers_____no_output_____

# 3.Conclusion

### Application

* In this project we started by data visualization, and data cleaneing the combining datasets to create predicion models.

* A series of models was made to generate predictions on data
    * the first model is to predict weather the user will compelete an offer or will view it
    * the second model was created to estimate the user average transactions
    * the third and final model to predict most appropriate offer to generate for the user

### Flows

* The first model has a great accuracy with feature extracted from date feature. In addtion of some other feature as age, gender, income also the model must be feeded with new data every now and then as customer base change. And another data may have great significance on the model

* The second should be updated with new data evey nw and then since users age and income change the 3 selected features gives us great insight for the prediction model.

* The regression models perform not as well as classification models. The reason may come from that the data has not scattered well on the features provided. as the off_drive_ratio may depend on other values as occupation and other feature to make it more accurate

### Imporvements to add

* Adding how long the customer has been a user it will have some great results to maximize profit.

* Imporve regression model based on new features as date , weekday, weekend days etc.

* Push customer offer based on how long he/she been a client for starbucks

    _____no_output_____


# Notebook from rodolforicardotech/datadiscoveryone
Path: notebooks/02-pre-processamento.ipynb

import pandas as pd_____no_output_____# importando o arquivo csv e lendo com o PANDAS
df = pd.read_csv("/Users/rodolforicardo/Downloads/datadiscoveryone/data/raw/olist_order_reviews_dataset.csv")_____no_output_____df.head()_____no_output_____# selecionando as colunas válidas para o projeto
selected = df[['review_score', 'review_comment_message']]_____no_output_____selected_na = selected.dropna()_____no_output_____dataframe = pd.DataFrame(selected_na)_____no_output_____# passando as palavras pa minúsculo
dataframe['review_comment_message'] = dataframe['review_comment_message'].str.lower()_____no_output_____dataframe_____no_output_____import unidecode
import unicodedata_____no_output_____# passando a coluna para lista
lista = list(dataframe['review_comment_message'])_____no_output_____# removendo acentuação com o unicodedata
novalista = []
for item in lista:
    item = unicodedata.normalize("NFD", item)
    novalista.append(item.encode("ascii", 'ignore').decode("utf-8"))_____no_output_____# removendo os números
import re
padrao = re.compile(r'\d+')
novalista2 = []
for item in novalista:
    novalista2.append(padrao.sub('', item))_____no_output_____# removendo os caracteres especiais
novalista3 = []
for item in novalista2:
    novalista3.append(re.sub(r'[^a-zA-Z0-9]', ' ', item))_____no_output_____# removendo o nome das Casas de Game of Thrones
# casas = ['lannister','stark','targaryen','greyjoy','baratheon','martell','tyrell','bolton']

padrao = re.compile(r'lannister|stark|targaryen|greyjoy|baratheon|martell|tyrell|bolton')
novalista4 = []
for item in novalista3:
    novalista4.append(padrao.sub('', item))_____no_output_____# importando a biblioteca NLTK para utilizar a lista de stopwords e stemming
import nltk
#nltk.download('stopwords')
from nltk.corpus import stopwords
stopwords = nltk.corpus.stopwords.words('portuguese')
#nltk.download('rslp')_____no_output_____# função para remover as stopwords
def removerStopwords (frases):
    novalista5 = []
    for item in frases:
        add = [p for p in item.split() if p not in stopwords]
        novalista5.append(add)
    return novalista4

novalista5 = removerStopwords(novalista4)_____no_output_____# removendo stopwords e fazendo o STEMMING (retirada dos sufixos)
def aplicaStemmer(texto):
    stemmer = nltk.stem.RSLPStemmer()
    semsufixo = []
    for frases in texto:
        comsufixo = [str(stemmer.stem(p)) for p in frases.split() if p not in stopwords]
        semsufixo.append(comsufixo)
    return semsufixo

novalista6 = aplicaStemmer(novalista4)_____no_output_____# criando uma lista com as frases completas
novalista7 = []
for item in novalista6:
    item = ' '.join(item)
    novalista7.append(item)_____no_output_____# criando um dataframe a partir da última lista tratada
toDataframe = pd.DataFrame(novalista7)_____no_output_____# adicionando uma nova coluna com os dados tratados
dataframe['dados tratados'] = toDataframe.values_____no_output_____# removendo as linhas com score 3
dataframe = dataframe[dataframe['review_score'] != 3]_____no_output_____dataframe_____no_output_____#criando as categorias, positivo e negativo
dataframe['target'] = dataframe['review_score'].apply(lambda x: 0 if x < 3 else 1) #[msgs['review_score']>= 4]/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

import string
from tqdm.auto import tqdm
cont = 0
x_data = []
y_data = []
for star,review,target,_ in tqdm((dataframe).values):
    x_data.append(review)
    y_data.append(target)
    cont += 1_____no_output_____

Bag of Words_____no_output_____

from sklearn.feature_extraction.text import CountVectorizer_____no_output_____vectorizer = CountVectorizer()_____no_output_____X = vectorizer.fit_transform(x_data)_____no_output_____(vectorizer.get_feature_names())_____no_output_____X.toarray()#[0]_____no_output_____# from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
X_train, X_test, y_train, y_test = train_test_split(X.toarray(), y_data, test_size=0.7, random_state=0)
gnb = GaussianNB()
y_pred = gnb.fit(X_train, y_train).predict(X_test)
#print("Number of mislabeled points out of a total %d points : %d"
#       % (X_test.shape[0], (y_test != y_pred).sum()))_____no_output_____from sklearn.metrics import accuracy_score
accuracy_score(y_test, y_pred)_____no_output_____


# Notebook from v-i-s-h/ib-dnn
Path: notebooks/plot_figures.ipynb

# Plot results_____no_output_____

<code>
import os, json
import matplotlib.pyplot as plt
import numpy as np_____no_output_____def plot_results(mi_data):
    sm = plt.cm.ScalarMappable(cmap='gnuplot', norm=plt.Normalize(vmin=0, vmax=config['epochs']))
    sm._A = []

    # plot infoplane evolution
    n_layer_types = len(mi_data)
    fig, ax = plt.subplots(nrows=1, ncols=n_layer_types, figsize=(5*n_layer_types, 5))
    if n_layer_types == 1:
        ax = [ax]
    ax = dict(zip(mi_data.keys(), ax))
    for layer_type, layer_data in mi_data.items():
        for layer_name, mi_values in layer_data.items():
            c = [sm.to_rgba(int(epoch)) for epoch in mi_values.keys()]

            mi = np.stack([mi_val for (_, mi_val) in mi_values.items()])
            ax[layer_type].scatter(mi[:,0], mi[:,1], c=c)

        epochs = list(layer_data[next(iter(layer_data))].keys())
        for epoch_idx in epochs:
            x_data = []
            y_data = []
            for layer_name, mi_values in layer_data.items():
                x_data.append(mi_values[epoch_idx][0])
                y_data.append(mi_values[epoch_idx][1])
            ax[layer_type].plot(x_data, y_data, c='k', alpha=0.1)

        ax[layer_type].set_title(layer_type)
        ax[layer_type].grid()

    cbaxes = fig.add_axes([1.0, 0.10, 0.05, 0.85])
    plt.colorbar(sm, label='Epoch', cax=cbaxes)
    plt.tight_layout()

    # plot layerwise
    for layer_type, layer_data in  mi_data.items():
        if layer_data:
            n_layers = len(layer_data)
            fig, ax = plt.subplots(nrows=1, ncols=n_layers, figsize=(3*n_layers, 3))
            ax = dict(zip(layer_data.keys(), ax))
            for (layer_name, mi_values) in  layer_data.items():
                c = [sm.to_rgba(int(epoch)) for epoch in mi_values.keys()]

                mi = np.stack([mi_val for (_, mi_val) in mi_values.items()])
                ax[layer_name].scatter(mi[:,0], mi[:,1], c=c)
                ax[layer_name].set_title(layer_name)
                ax[layer_name].set_xlabel("I(T;X)")
                ax[layer_name].set_ylabel("I(T;Y)")
                ax[layer_name].grid()
            cbaxes = fig.add_axes([1.0, 0.1, 0.01, 0.80])
            plt.colorbar(sm, label='Epoch', cax=cbaxes)
            plt.tight_layout()_____no_output_____exp_folder = "./../zoo/"_____no_output_____config_file = os.path.join(exp_folder, "config.json")
config = {}
with open(config_file, "r") as f:
    config = json.load(f)_____no_output_____epochs_run = 0
with open(os.path.join(exp_folder, "stats.json"), "r") as f:
    epochs_run = json.load(f)['epoch']_____no_output_____metrics = {}
with open(os.path.join(exp_folder, "metrics.json"), "r") as f:
    metrics = json.load(f)_____no_output_____log_file = os.path.join(exp_folder, "mi_data.json")
mi_data = {}
with open(log_file, "r") as f:
    mi_data = json.load(f)_____no_output_____if "test" in mi_data:
    plot_results(mi_data["test"])_____no_output_____if "train" in mi_data:
    plot_results(mi_data["train"])_____no_output_____epochs = metrics['epoch']
metric_labels = [label for label in metrics.keys() if label != 'epoch' and not label.startswith("val_")]

fig, ax = plt.subplots(nrows=1, ncols=len(metric_labels), figsize=(5*len(metric_labels), 4))
if len(ax) == 1:
    ax = [ax]
ax = dict(zip(metric_labels, ax))
for label in metric_labels:
    ax[label].plot(epochs, metrics[label])
    if "val_"+label in metrics:
        ax[label].plot(metrics["val_"+label])

    ax[label].set_xlabel('epochs')
    ax[label].set_title(label)
    ax[label].grid()_____no_output_____
</code>